[2026-02-12 21:40:02,140][ INFO][PID:3531224] Marking build as starting [2026-02-12 21:40:02,304][ INFO][PID:3531224] Checking that https://download.copr.fedorainfracloud.org/results/sicherha/katarakt/epel-7-ppc64le//repodata/repomd.xml exists [2026-02-12 21:40:02,789][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:40:02,790][ INFO][PID:3531224] VM allocation process starts [2026-02-12 21:40:02,820][ INFO][PID:3531224] Trying to allocate VM: ResallocHost, ticket_id=1308450, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-02-12 21:40:05,858][ INFO][PID:3531224] Allocated host ResallocHost, ticket_id=1308450, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_02240720_20260212_213354, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-02-12 21:40:05,859][ INFO][PID:3531224] Allocating ssh connection to builder [2026-02-12 21:40:05,859][ INFO][PID:3531224] Checking that builder machine is OK [2026-02-12 21:40:06,990][ INFO][PID:3531224] Running remote command: copr-builder-ready epel-7-ppc64le [2026-02-12 21:40:08,026][ INFO][PID:3531224] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2026-02-12 21:40:08,026][ INFO][PID:3531224] Filling build.info file with builder info [2026-02-12 21:40:08,027][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:40:08,028][ INFO][PID:3531224] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "sicherha", "project_name": "katarakt", "project_dirname": "katarakt", "submitter": "sicherha", "ended_on": null, "started_on": 1770932408.028651, "submitted_on": null, "status": 3, "chroot": "epel-7-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "10126060-epel-7-ppc64le", "build_id": 10126060, "package_name": "katarakt", "package_version": "0.3-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/sicherha/katarakt/katarakt", "git_hash": "9f90ed8f3f7e4c079eb330efa781e2884aec482e", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "sicherha/katarakt--sicherha", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 1, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/sicherha/katarakt/epel-7-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/sicherha/katarakt", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/sicherha/katarakt", "result_dir": "10126060-katarakt", "built_packages": "", "id": 10126060, "mockchain_macros": { "copr_username": "sicherha", "copr_projectname": "katarakt", "vendor": "Fedora Project COPR (sicherha/katarakt)" } } ] } [2026-02-12 21:40:08,113][ INFO][PID:3531224] Sending fedora-messaging bus message in build.start [2026-02-12 21:40:08,699][ INFO][PID:3531224] Sending fedora-messaging bus message in chroot.start [2026-02-12 21:40:08,728][ INFO][PID:3531224] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10126060-epel-7-ppc64le --chroot epel-7-ppc64le --detached [2026-02-12 21:40:09,889][ INFO][PID:3531224] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c191' (ED25519) to the list of known hosts. [2026-02-12 21:40:09,889][ INFO][PID:3531224] Downloading the builder-live.log file, attempt 1 [2026-02-12 21:40:09,890][ INFO][PID:3531224] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c191 copr-rpmbuild-log [2026-02-12 21:40:14,897][ INFO][PID:3531224] Periodic builder liveness probe: alive [2026-02-12 21:40:14,961][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:40:19,963][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:40:24,963][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:40:29,964][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:40:34,975][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:40:39,976][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:40:44,977][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:40:49,978][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:40:54,980][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:40:59,981][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:41:04,983][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:41:09,984][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:41:14,985][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:41:19,986][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:41:24,987][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:41:29,988][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:41:34,989][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:41:39,990][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:41:45,035][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:41:50,053][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:41:55,078][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:42:00,099][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:42:05,100][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:42:06,865][ INFO][PID:3531224] Downloading results from builder [2026-02-12 21:42:06,866][ INFO][PID:3531224] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/sicherha/katarakt/epel-7-ppc64le/10126060-katarakt started [2026-02-12 21:42:06,867][ INFO][PID:3531224] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/sicherha/katarakt/epel-7-ppc64le/10126060-katarakt/ &> /var/lib/copr/public_html/results/sicherha/katarakt/epel-7-ppc64le/10126060-katarakt/build-10126060.rsync.log [2026-02-12 21:43:06,907][ INFO][PID:3531224] rsyncing finished. [2026-02-12 21:43:06,907][ ERROR][PID:3531224] Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-02-12 21:43:06,908][ ERROR][PID:3531224] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-02-12 21:43:16,908][ INFO][PID:3531224] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/sicherha/katarakt/epel-7-ppc64le/10126060-katarakt started [2026-02-12 21:43:16,910][ INFO][PID:3531224] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/sicherha/katarakt/epel-7-ppc64le/10126060-katarakt/ &> /var/lib/copr/public_html/results/sicherha/katarakt/epel-7-ppc64le/10126060-katarakt/build-10126060.rsync.log [2026-02-12 21:44:16,995][ INFO][PID:3531224] rsyncing finished. [2026-02-12 21:44:17,011][ ERROR][PID:3531224] Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-02-12 21:44:17,011][ ERROR][PID:3531224] SSH connection lost on #2 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-02-12 21:44:27,012][ INFO][PID:3531224] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/sicherha/katarakt/epel-7-ppc64le/10126060-katarakt started [2026-02-12 21:44:27,013][ INFO][PID:3531224] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/sicherha/katarakt/epel-7-ppc64le/10126060-katarakt/ &> /var/lib/copr/public_html/results/sicherha/katarakt/epel-7-ppc64le/10126060-katarakt/build-10126060.rsync.log [2026-02-12 21:45:27,086][ INFO][PID:3531224] rsyncing finished. [2026-02-12 21:45:27,102][ ERROR][PID:3531224] Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-02-12 21:45:27,102][ ERROR][PID:3531224] SSH connection lost on #3 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-02-12 21:45:37,103][ ERROR][PID:3531224] Re-try request for task on 'ResallocHost, ticket_id=1308450, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_02240720_20260212_213354, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 3 SSH attempts [2026-02-12 21:45:37,103][ INFO][PID:3531224] Releasing VM back to pool [2026-02-12 21:45:37,112][ INFO][PID:3531224] Retry #1 (on other host) [2026-02-12 21:45:37,112][ INFO][PID:3531224] Checking that https://download.copr.fedorainfracloud.org/results/sicherha/katarakt/epel-7-ppc64le//repodata/repomd.xml exists [2026-02-12 21:45:39,491][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:45:39,492][ INFO][PID:3531224] VM allocation process starts [2026-02-12 21:45:39,504][ INFO][PID:3531224] Trying to allocate VM: ResallocHost, ticket_id=1308513, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-02-12 21:45:42,520][ INFO][PID:3531224] Allocated host ResallocHost, ticket_id=1308513, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_02240720_20260212_213354, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-02-12 21:45:42,520][ INFO][PID:3531224] Allocating ssh connection to builder [2026-02-12 21:45:42,520][ INFO][PID:3531224] Checking that builder machine is OK [2026-02-12 21:46:42,588][ ERROR][PID:3531224] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c191 port 22: Connection timed out [2026-02-12 21:46:52,604][ ERROR][PID:3531224] Re-try request for task on 'ResallocHost, ticket_id=1308513, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_02240720_20260212_213354, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts [2026-02-12 21:46:52,604][ INFO][PID:3531224] Releasing VM back to pool [2026-02-12 21:46:52,613][ INFO][PID:3531224] Retry #2 (on other host) [2026-02-12 21:46:52,614][ INFO][PID:3531224] Checking that https://download.copr.fedorainfracloud.org/results/sicherha/katarakt/epel-7-ppc64le//repodata/repomd.xml exists [2026-02-12 21:46:54,183][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:46:54,184][ INFO][PID:3531224] VM allocation process starts [2026-02-12 21:46:54,232][ INFO][PID:3531224] Trying to allocate VM: ResallocHost, ticket_id=1308529, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-02-12 21:46:57,268][ INFO][PID:3531224] Allocated host ResallocHost, ticket_id=1308529, hostname=2620:52:6:1161:dead:beef:cafe:c18b, name=vmhost_p09_04_prod_02240798_20260212_213859, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-02-12 21:46:57,269][ INFO][PID:3531224] Allocating ssh connection to builder [2026-02-12 21:46:57,269][ INFO][PID:3531224] Checking that builder machine is OK [2026-02-12 21:46:58,424][ INFO][PID:3531224] Running remote command: copr-builder-ready epel-7-ppc64le [2026-02-12 21:46:58,987][ INFO][PID:3531224] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2026-02-12 21:46:58,988][ INFO][PID:3531224] Filling build.info file with builder info [2026-02-12 21:46:59,001][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:46:59,002][ INFO][PID:3531224] Not re-notifying FE and msg buses for the new host. [2026-02-12 21:46:59,002][ INFO][PID:3531224] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10126060-epel-7-ppc64le --chroot epel-7-ppc64le --detached [2026-02-12 21:47:00,393][ INFO][PID:3531224] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c18b' (ED25519) to the list of known hosts. [2026-02-12 21:47:00,394][ INFO][PID:3531224] Downloading the builder-live.log file, attempt 1 [2026-02-12 21:47:00,396][ INFO][PID:3531224] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18b copr-rpmbuild-log [2026-02-12 21:47:05,395][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:47:10,396][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:47:15,397][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:47:20,398][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:47:25,399][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:47:30,401][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:47:35,404][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:47:40,405][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:47:45,406][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:47:50,407][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:47:56,033][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:48:01,034][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:48:06,035][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:48:11,036][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:48:16,037][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:48:21,038][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:48:26,040][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:48:31,042][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:48:36,042][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:48:41,043][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:48:46,044][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:48:51,045][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:48:56,046][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:49:01,047][ INFO][PID:3531224] Checking for cancel request [2026-02-12 21:49:02,504][ INFO][PID:3531224] Downloading results from builder [2026-02-12 21:49:02,505][ INFO][PID:3531224] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18b]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/sicherha/katarakt/epel-7-ppc64le/10126060-katarakt started [2026-02-12 21:49:02,505][ INFO][PID:3531224] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18b]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/sicherha/katarakt/epel-7-ppc64le/10126060-katarakt/ &> /var/lib/copr/public_html/results/sicherha/katarakt/epel-7-ppc64le/10126060-katarakt/build-10126060.rsync.log [2026-02-12 21:49:03,372][ INFO][PID:3531224] rsyncing finished. [2026-02-12 21:49:03,372][ INFO][PID:3531224] Releasing VM back to pool [2026-02-12 21:49:03,381][ INFO][PID:3531224] Searching for 'success' file in resultdir [2026-02-12 21:49:03,382][ ERROR][PID:3531224] Build failed: Backend process error: No success file => build failure [2026-02-12 21:49:03,382][ INFO][PID:3531224] Finished build: id=10126060 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/sicherha/katarakt chroot=epel-7-ppc64le [2026-02-12 21:49:03,384][ ERROR][PID:3531224] Backend process error: No success file => build failure [2026-02-12 21:49:03,384][ INFO][PID:3531224] Worker failed build, took 124.3825376033783 [2026-02-12 21:49:03,385][ INFO][PID:3531224] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "sicherha", "project_name": "katarakt", "project_dirname": "katarakt", "submitter": "sicherha", "ended_on": 1770932943.3847592, "started_on": 1770932819.0022216, "submitted_on": null, "status": 0, "chroot": "epel-7-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "10126060-epel-7-ppc64le", "build_id": 10126060, "package_name": "katarakt", "package_version": "0.3-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/sicherha/katarakt/katarakt", "git_hash": "9f90ed8f3f7e4c079eb330efa781e2884aec482e", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "sicherha/katarakt--sicherha", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 1, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/sicherha/katarakt/epel-7-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/sicherha/katarakt", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/sicherha/katarakt", "result_dir": "10126060-katarakt", "built_packages": "", "id": 10126060, "mockchain_macros": { "copr_username": "sicherha", "copr_projectname": "katarakt", "vendor": "Fedora Project COPR (sicherha/katarakt)" } } ] } [2026-02-12 21:49:03,455][ INFO][PID:3531224] Sending fedora-messaging bus message in build.end [2026-02-12 21:49:03,482][ INFO][PID:3531224] Compressing /var/lib/copr/public_html/results/sicherha/katarakt/epel-7-ppc64le/10126060-katarakt/builder-live.log by gzip [2026-02-12 21:49:03,483][ INFO][PID:3531224] Running command 'gzip /var/lib/copr/public_html/results/sicherha/katarakt/epel-7-ppc64le/10126060-katarakt/builder-live.log' as PID 3571812 [2026-02-12 21:49:03,496][ INFO][PID:3531224] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/sicherha/katarakt/epel-7-ppc64le/10126060-katarakt/builder-live.log) [2026-02-12 21:49:03,497][ INFO][PID:3531224] Compressing /var/lib/copr/public_html/results/sicherha/katarakt/epel-7-ppc64le/10126060-katarakt/backend.log by gzip [2026-02-12 21:49:03,498][ INFO][PID:3531224] Running command 'gzip /var/lib/copr/public_html/results/sicherha/katarakt/epel-7-ppc64le/10126060-katarakt/backend.log' as PID 3571813