[2025-12-27 13:49:41,463][ INFO][PID:1615618] Marking build as starting [2025-12-27 13:49:41,565][ INFO][PID:1615618] Checking that /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le/repodata/repomd.xml exists [2025-12-27 13:49:41,567][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:49:41,567][ INFO][PID:1615618] VM allocation process starts [2025-12-27 13:49:41,578][ INFO][PID:1615618] Trying to allocate VM: ResallocHost, ticket_id=534502, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-12-27 13:49:44,590][ INFO][PID:1615618] Allocated host ResallocHost, ticket_id=534502, hostname=2620:52:6:1161:dead:beef:cafe:c156, name=vmhost_p09_03_prod_00790227_20251227_133934, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-12-27 13:49:44,591][ INFO][PID:1615618] Allocating ssh connection to builder [2025-12-27 13:49:44,591][ INFO][PID:1615618] Checking that builder machine is OK [2025-12-27 13:49:44,617][ ERROR][PID:1615618] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c156 port 22: Permission denied [2025-12-27 13:49:54,618][ ERROR][PID:1615618] Re-try request for task on 'ResallocHost, ticket_id=534502, hostname=2620:52:6:1161:dead:beef:cafe:c156, name=vmhost_p09_03_prod_00790227_20251227_133934, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts [2025-12-27 13:49:54,619][ INFO][PID:1615618] Releasing VM back to pool [2025-12-27 13:49:54,627][ INFO][PID:1615618] Retry #1 (on other host) [2025-12-27 13:49:54,628][ INFO][PID:1615618] Checking that /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le/repodata/repomd.xml exists [2025-12-27 13:49:54,628][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:49:54,629][ INFO][PID:1615618] VM allocation process starts [2025-12-27 13:49:54,663][ INFO][PID:1615618] Trying to allocate VM: ResallocHost, ticket_id=534509, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-12-27 13:49:57,705][ INFO][PID:1615618] Allocated host ResallocHost, ticket_id=534509, hostname=2620:52:6:1161:dead:beef:cafe:c190, name=vmhost_p09_04_prod_00790261_20251227_134226, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-12-27 13:49:57,706][ INFO][PID:1615618] Allocating ssh connection to builder [2025-12-27 13:49:57,706][ INFO][PID:1615618] Checking that builder machine is OK [2025-12-27 13:49:58,800][ INFO][PID:1615618] Running remote command: copr-builder-ready fedora-42-ppc64le [2025-12-27 13:49:59,338][ INFO][PID:1615618] Red Hat subscription not needed for fedora-42-ppc64le Builder is ready to be used [2025-12-27 13:49:59,339][ INFO][PID:1615618] Filling build.info file with builder info [2025-12-27 13:49:59,339][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:49:59,341][ INFO][PID:1615618] Not re-notifying FE and msg buses for the new host. [2025-12-27 13:49:59,341][ INFO][PID:1615618] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9953520-fedora-42-ppc64le --chroot fedora-42-ppc64le --detached [2025-12-27 13:50:00,321][ INFO][PID:1615618] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c190' (ED25519) to the list of known hosts. [2025-12-27 13:50:00,322][ INFO][PID:1615618] Downloading the builder-live.log file, attempt 1 [2025-12-27 13:50:00,324][ INFO][PID:1615618] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c190 copr-rpmbuild-log [2025-12-27 13:50:05,330][ INFO][PID:1615618] Periodic builder liveness probe: alive [2025-12-27 13:50:05,331][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:50:10,332][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:50:15,333][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:50:20,334][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:50:25,335][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:50:30,336][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:50:35,337][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:50:40,338][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:50:45,339][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:50:50,340][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:50:55,341][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:51:00,342][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:51:05,343][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:51:10,344][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:51:15,345][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:51:20,346][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:51:25,347][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:51:30,348][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:51:35,349][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:51:40,350][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:51:45,351][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:51:50,353][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:51:55,354][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:52:00,355][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:52:05,356][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:52:10,357][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:52:15,359][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:52:20,360][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:52:25,361][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:52:30,362][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:52:35,363][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:52:40,364][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:52:45,365][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:52:50,366][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:52:55,366][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:53:00,367][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:53:05,368][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:53:10,369][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:53:15,370][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:53:20,371][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:53:24,351][ ERROR][PID:1615618] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-27 13:53:25,372][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:53:30,375][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:53:34,351][WARNING][PID:1615618] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-27 13:53:35,378][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:53:40,379][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:53:44,417][ INFO][PID:1615618] Downloading the builder-live.log file, attempt 2 [2025-12-27 13:53:44,418][ INFO][PID:1615618] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c190 copr-rpmbuild-log [2025-12-27 13:53:45,380][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:53:50,381][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:53:55,382][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:54:00,383][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:54:05,384][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:54:10,385][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:54:15,386][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:54:20,387][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:54:25,388][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:54:30,389][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:54:35,390][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:54:40,391][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:54:44,479][ ERROR][PID:1615618] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-27 13:54:45,392][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:54:50,393][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:54:54,488][WARNING][PID:1615618] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-27 13:54:55,394][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:55:00,395][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:55:04,489][ INFO][PID:1615618] Downloading the builder-live.log file, attempt 3 [2025-12-27 13:55:04,490][ INFO][PID:1615618] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c190 copr-rpmbuild-log [2025-12-27 13:55:05,396][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:55:10,397][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:55:15,398][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:55:20,399][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:55:25,400][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:55:30,401][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:55:35,402][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:55:40,402][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:55:45,403][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:55:50,404][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:55:55,414][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:56:00,417][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:56:04,557][ ERROR][PID:1615618] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-27 13:56:05,418][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:56:10,420][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:56:14,571][WARNING][PID:1615618] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-27 13:56:15,421][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:56:20,422][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:56:24,571][ INFO][PID:1615618] Downloading the builder-live.log file, attempt 4 [2025-12-27 13:56:24,573][ INFO][PID:1615618] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c190 copr-rpmbuild-log [2025-12-27 13:56:25,423][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:56:30,425][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:56:35,426][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:56:40,427][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:56:45,429][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:56:50,430][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:56:55,431][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:57:00,432][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:57:05,433][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:57:10,434][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:57:15,436][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:57:20,436][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:57:25,115][ ERROR][PID:1615618] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-27 13:57:25,437][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:57:30,438][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:57:35,127][ ERROR][PID:1615618] Giving up for unstable SSH, failures: 1766843614.3516865, 1766843694.488575, 1766843774.5710244, 1766843855.1267855 [2025-12-27 13:57:35,127][ ERROR][PID:1615618] Re-try request for task on 'ResallocHost, ticket_id=534509, hostname=2620:52:6:1161:dead:beef:cafe:c190, name=vmhost_p09_04_prod_00790261_20251227_134226, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-12-27 13:57:35,128][ INFO][PID:1615618] Releasing VM back to pool [2025-12-27 13:57:35,136][ INFO][PID:1615618] Retry #2 (on other host) [2025-12-27 13:57:35,137][ INFO][PID:1615618] Checking that /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le/repodata/repomd.xml exists [2025-12-27 13:57:35,137][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:57:35,138][ INFO][PID:1615618] VM allocation process starts [2025-12-27 13:57:35,167][ INFO][PID:1615618] Trying to allocate VM: ResallocHost, ticket_id=534604, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-12-27 13:57:38,198][ INFO][PID:1615618] Allocated host ResallocHost, ticket_id=534604, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_00790334_20251227_135148, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-12-27 13:57:38,199][ INFO][PID:1615618] Allocating ssh connection to builder [2025-12-27 13:57:38,199][ INFO][PID:1615618] Checking that builder machine is OK [2025-12-27 13:57:39,255][ INFO][PID:1615618] Running remote command: copr-builder-ready fedora-42-ppc64le [2025-12-27 13:57:39,824][ INFO][PID:1615618] Red Hat subscription not needed for fedora-42-ppc64le Builder is ready to be used [2025-12-27 13:57:39,824][ INFO][PID:1615618] Filling build.info file with builder info [2025-12-27 13:57:39,828][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:57:39,829][ INFO][PID:1615618] Not re-notifying FE and msg buses for the new host. [2025-12-27 13:57:39,829][ INFO][PID:1615618] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9953520-fedora-42-ppc64le --chroot fedora-42-ppc64le --detached [2025-12-27 13:57:40,817][ INFO][PID:1615618] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c191' (ED25519) to the list of known hosts. [2025-12-27 13:57:40,818][ INFO][PID:1615618] Downloading the builder-live.log file, attempt 1 [2025-12-27 13:57:40,819][ INFO][PID:1615618] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c191 copr-rpmbuild-log [2025-12-27 13:57:45,818][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:57:50,819][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:57:55,820][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:58:00,821][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:58:05,822][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:58:10,823][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:58:15,824][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:58:20,825][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:58:25,826][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:58:30,827][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:58:35,828][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:58:40,829][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:58:45,830][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:58:50,831][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:58:55,831][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:59:00,833][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:59:05,834][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:59:10,835][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:59:15,836][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:59:20,836][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:59:25,837][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:59:30,838][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:59:35,839][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:59:40,840][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:59:45,841][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:59:50,842][ INFO][PID:1615618] Checking for cancel request [2025-12-27 13:59:55,843][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:00:00,844][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:00:05,849][ INFO][PID:1615618] Periodic builder liveness probe: alive [2025-12-27 14:00:05,850][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:00:10,851][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:00:15,852][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:00:20,852][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:00:25,853][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:00:30,854][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:00:35,855][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:00:40,856][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:00:45,857][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:00:50,858][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:00:55,859][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:01:00,860][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:01:05,861][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:01:10,862][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:01:15,863][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:01:20,864][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:01:25,865][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:01:30,866][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:01:35,867][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:01:40,868][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:01:45,869][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:01:50,870][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:01:56,229][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:02:01,230][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:02:06,231][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:02:11,232][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:02:16,233][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:02:21,234][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:02:26,235][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:02:31,236][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:02:36,237][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:02:41,237][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:02:46,238][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:02:51,240][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:02:56,241][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:03:01,242][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:03:06,244][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:03:11,245][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:03:16,246][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:03:21,247][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:03:26,248][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:03:31,249][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:03:36,250][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:03:40,996][ ERROR][PID:1615618] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-27 14:03:41,251][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:03:46,252][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:03:50,997][WARNING][PID:1615618] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-27 14:03:51,495][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:03:56,497][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:04:00,997][ INFO][PID:1615618] Downloading the builder-live.log file, attempt 2 [2025-12-27 14:04:01,287][ INFO][PID:1615618] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c191 copr-rpmbuild-log [2025-12-27 14:04:01,499][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:04:06,500][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:04:11,501][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:04:16,502][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:04:21,503][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:04:26,505][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:04:31,507][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:04:36,508][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:04:41,509][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:04:46,510][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:04:51,511][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:04:56,512][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:05:01,353][ ERROR][PID:1615618] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-27 14:05:01,513][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:05:06,513][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:05:11,364][WARNING][PID:1615618] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-27 14:05:11,514][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:05:16,518][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:05:21,366][ INFO][PID:1615618] Downloading the builder-live.log file, attempt 3 [2025-12-27 14:05:21,371][ INFO][PID:1615618] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c191 copr-rpmbuild-log [2025-12-27 14:05:21,520][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:05:26,522][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:05:31,523][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:05:36,523][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:05:41,524][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:05:46,525][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:05:51,526][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:05:56,527][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:06:01,528][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:06:06,529][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:06:11,530][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:06:16,531][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:06:21,532][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:06:26,533][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:06:31,534][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:06:36,535][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:06:41,537][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:06:46,539][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:06:51,540][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:06:56,544][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:07:01,547][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:07:06,554][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:07:11,565][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:07:16,568][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:07:21,582][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:07:26,590][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:07:31,601][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:07:36,604][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:07:41,615][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:07:46,623][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:07:51,630][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:07:56,650][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:08:01,665][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:08:06,673][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:08:11,683][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:08:16,713][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:08:21,745][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:08:26,758][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:08:31,765][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:08:36,772][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:08:41,775][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:08:46,782][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:08:51,783][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:08:56,797][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:09:01,803][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:09:06,824][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:09:11,826][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:09:16,828][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:09:21,831][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:09:26,833][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:09:31,836][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:09:36,838][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:09:41,843][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:09:46,846][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:09:51,847][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:09:56,851][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:10:01,852][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:10:06,866][ INFO][PID:1615618] Periodic builder liveness probe: alive [2025-12-27 14:10:06,866][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:10:11,871][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:10:16,889][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:10:21,893][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:10:26,899][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:10:31,902][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:10:36,904][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:10:41,905][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:10:46,906][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:10:51,907][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:10:57,047][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:11:02,049][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:11:07,053][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:11:12,055][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:11:17,057][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:11:22,060][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:11:27,061][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:11:32,062][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:11:37,063][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:11:42,074][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:11:47,075][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:11:52,076][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:11:57,077][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:12:02,078][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:12:07,079][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:12:12,079][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:12:17,080][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:12:22,081][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:12:27,082][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:12:32,083][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:12:37,084][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:12:42,085][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:12:47,086][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:12:52,087][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:12:57,088][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:13:02,089][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:13:07,090][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:13:12,091][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:13:17,092][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:13:22,093][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:13:27,094][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:13:32,095][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:13:37,096][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:13:42,097][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:13:47,137][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:13:52,138][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:13:57,139][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:14:02,140][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:14:07,176][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:14:12,177][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:14:17,178][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:14:22,179][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:14:27,180][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:14:32,181][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:14:37,182][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:14:42,197][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:14:47,198][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:14:52,199][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:14:57,200][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:15:02,201][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:15:07,202][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:15:12,203][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:15:17,204][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:15:22,205][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:15:27,207][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:15:32,208][ INFO][PID:1615618] Checking for cancel request [2025-12-27 14:15:33,502][ INFO][PID:1615618] Downloading results from builder [2025-12-27 14:15:33,503][ INFO][PID:1615618] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le/09953520-cosmic-ext-applet-minimon started [2025-12-27 14:15:33,506][ INFO][PID:1615618] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le/09953520-cosmic-ext-applet-minimon/ &> /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le/09953520-cosmic-ext-applet-minimon/build-09953520.rsync.log [2025-12-27 14:15:34,901][ INFO][PID:1615618] rsyncing finished. [2025-12-27 14:15:34,902][ INFO][PID:1615618] Releasing VM back to pool [2025-12-27 14:15:34,952][ INFO][PID:1615618] Searching for 'success' file in resultdir [2025-12-27 14:15:34,953][ INFO][PID:1615618] Going to sign pkgs from source: 9953520-fedora-42-ppc64le in chroot: /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le [2025-12-27 14:15:34,954][ INFO][PID:1615618] Calling '/bin/sign -u ligenix#cosmic-ext@copr.fedorahosted.org -p' (attempt #1) [2025-12-27 14:15:35,227][ INFO][PID:1615618] Calling '/bin/sign -4 -h sha256 -u ligenix#cosmic-ext@copr.fedorahosted.org -r /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le/09953520-cosmic-ext-applet-minimon/cosmic-ext-applet-minimon-1.0.0-2.fc42.src.rpm' (attempt #1) [2025-12-27 14:15:37,585][ INFO][PID:1615618] signed rpm: /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le/09953520-cosmic-ext-applet-minimon/cosmic-ext-applet-minimon-1.0.0-2.fc42.src.rpm [2025-12-27 14:15:37,586][ INFO][PID:1615618] Calling '/bin/sign -4 -h sha256 -u ligenix#cosmic-ext@copr.fedorahosted.org -r /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le/09953520-cosmic-ext-applet-minimon/cosmic-ext-applet-minimon-1.0.0-2.fc42.ppc64le.rpm' (attempt #1) [2025-12-27 14:15:38,627][ INFO][PID:1615618] signed rpm: /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le/09953520-cosmic-ext-applet-minimon/cosmic-ext-applet-minimon-1.0.0-2.fc42.ppc64le.rpm [2025-12-27 14:15:38,628][ INFO][PID:1615618] Sign done [2025-12-27 14:15:38,628][ INFO][PID:1615618] Incremental createrepo run, adding 09953520-cosmic-ext-applet-minimon into https://download.copr.fedorainfracloud.org/results/ligenix/cosmic-ext/fedora-42-ppc64le, (auto-create-repo=True) [2025-12-27 14:15:38,631][ INFO][PID:1615618] Running command 'copr-repo --batched /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le --add 09953520-cosmic-ext-applet-minimon --no-appstream-metadata' as PID 1732025 [2025-12-27 14:15:39,989][ INFO][PID:1615618] Finished after 1 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le --add 09953520-cosmic-ext-applet-minimon --no-appstream-metadata) [2025-12-27 14:15:39,990][ INFO][PID:1615618] Getting build details [2025-12-27 14:15:39,990][ INFO][PID:1615618] Listing built binary packages in /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le/09953520-cosmic-ext-applet-minimon [2025-12-27 14:15:39,990][ INFO][PID:1615618] Built packages: ['cosmic-ext-applet-minimon 1.0.0'] [2025-12-27 14:15:39,991][ INFO][PID:1615618] build details: {'built_packages': 'cosmic-ext-applet-minimon 1.0.0'} [2025-12-27 14:15:39,991][ INFO][PID:1615618] Retrieving pubkey [2025-12-27 14:15:39,991][ INFO][PID:1615618] Calling '/bin/sign -u ligenix#cosmic-ext@copr.fedorahosted.org -p' (attempt #1) [2025-12-27 14:15:40,276][ INFO][PID:1615618] Added pubkey for user ligenix project cosmic-ext into: /var/lib/copr/public_html/results/ligenix/cosmic-ext/pubkey.gpg [2025-12-27 14:15:40,276][ INFO][PID:1615618] Finished build: id=9953520 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/ligenix/cosmic-ext chroot=fedora-42-ppc64le [2025-12-27 14:15:40,278][ INFO][PID:1615618] Worker succeeded build, took 1080.4489839076996 [2025-12-27 14:15:40,278][ INFO][PID:1615618] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "ligenix", "project_name": "cosmic-ext", "project_dirname": "cosmic-ext", "submitter": "ligenix", "ended_on": 1766844940.2780309, "started_on": 1766843859.829047, "submitted_on": null, "status": 1, "chroot": "fedora-42-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9953520-fedora-42-ppc64le", "build_id": 9953520, "package_name": "cosmic-ext-applet-minimon", "package_version": "1.0.0-2", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/ligenix/cosmic-ext/cosmic-ext-applet-minimon", "git_hash": "5acf782ba42e1094fb7fc92618a7a7bf2f3c8a4e", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "ligenix/cosmic-ext--ligenix", "results": { "packages": [ { "name": "cosmic-ext-applet-minimon", "epoch": null, "version": "1.0.0", "release": "2.fc42", "arch": "ppc64le" }, { "name": "cosmic-ext-applet-minimon", "epoch": null, "version": "1.0.0", "release": "2.fc42", "arch": "src" } ] }, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/ligenix/cosmic-ext/fedora-42-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/ligenix/cosmic-ext", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/ligenix/cosmic-ext", "result_dir": "09953520-cosmic-ext-applet-minimon", "built_packages": "cosmic-ext-applet-minimon 1.0.0", "id": 9953520, "mockchain_macros": { "copr_username": "ligenix", "copr_projectname": "cosmic-ext", "vendor": "Fedora Project COPR (ligenix/cosmic-ext)" } } ] } [2025-12-27 14:15:40,349][ INFO][PID:1615618] Sending fedora-messaging bus message in build.end [2025-12-27 14:15:40,830][ INFO][PID:1615618] Compressing /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le/09953520-cosmic-ext-applet-minimon/builder-live.log by gzip [2025-12-27 14:15:40,831][ INFO][PID:1615618] Running command 'gzip /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le/09953520-cosmic-ext-applet-minimon/builder-live.log' as PID 1732115 [2025-12-27 14:15:40,892][ INFO][PID:1615618] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le/09953520-cosmic-ext-applet-minimon/builder-live.log) [2025-12-27 14:15:40,893][ INFO][PID:1615618] Compressing /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le/09953520-cosmic-ext-applet-minimon/backend.log by gzip [2025-12-27 14:15:40,894][ INFO][PID:1615618] Running command 'gzip /var/lib/copr/public_html/results/ligenix/cosmic-ext/fedora-42-ppc64le/09953520-cosmic-ext-applet-minimon/backend.log' as PID 1732120