[2026-02-04 21:15:09,907][ INFO][PID:74861] Marking build as starting [2026-02-04 21:15:09,956][ INFO][PID:74861] Checking that /var/lib/copr/public_html/results/ufven/podman/epel-9-ppc64le/repodata/repomd.xml exists [2026-02-04 21:15:09,957][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:15:09,958][ INFO][PID:74861] VM allocation process starts [2026-02-04 21:15:09,979][ INFO][PID:74861] Trying to allocate VM: ResallocHost, ticket_id=1174624, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-02-04 21:15:14,980][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:15:16,023][ INFO][PID:74861] Allocated host ResallocHost, ticket_id=1174624, hostname=2620:52:6:1161:dead:beef:cafe:c18a, name=vmhost_p09_04_prod_01941139_20260204_205735, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-02-04 21:15:16,024][ INFO][PID:74861] Allocating ssh connection to builder [2026-02-04 21:15:16,025][ INFO][PID:74861] Checking that builder machine is OK [2026-02-04 21:15:17,082][ INFO][PID:74861] Running remote command: copr-builder-ready epel-9-ppc64le [2026-02-04 21:15:17,641][ INFO][PID:74861] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2026-02-04 21:15:17,642][ INFO][PID:74861] Filling build.info file with builder info [2026-02-04 21:15:17,643][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:15:17,646][ INFO][PID:74861] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "ufven", "project_name": "podman", "project_dirname": "podman", "submitter": null, "ended_on": null, "started_on": 1770239717.6461892, "submitted_on": null, "status": 3, "chroot": "epel-9-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "10094424-epel-9-ppc64le", "build_id": 10094424, "package_name": "container-selinux", "package_version": "102:2.245.0-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/ufven/podman/container-selinux", "git_hash": "97586f1c92754694944ef129ef1f439042a2ad65", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "ufven/podman--1e012b8d-70da-4964-8dfa-49b6d7106c0f", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/ufven/podman/epel-9-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le", "arch_power9" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/ufven/podman", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/ufven/podman", "result_dir": "10094424-container-selinux", "built_packages": "", "id": 10094424, "mockchain_macros": { "copr_username": "ufven", "copr_projectname": "podman", "vendor": "Fedora Project COPR (ufven/podman)" } } ] } [2026-02-04 21:15:17,691][ INFO][PID:74861] Sending fedora-messaging bus message in build.start [2026-02-04 21:15:18,308][ INFO][PID:74861] Sending fedora-messaging bus message in chroot.start [2026-02-04 21:15:18,335][ INFO][PID:74861] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10094424-epel-9-ppc64le --chroot epel-9-ppc64le --detached [2026-02-04 21:15:19,364][ INFO][PID:74861] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c18a' (ED25519) to the list of known hosts. [2026-02-04 21:15:19,365][ INFO][PID:74861] Downloading the builder-live.log file, attempt 1 [2026-02-04 21:15:19,367][ INFO][PID:74861] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18a copr-rpmbuild-log [2026-02-04 21:15:24,371][ INFO][PID:74861] Periodic builder liveness probe: alive [2026-02-04 21:15:24,371][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:15:29,374][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:15:34,379][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:15:39,381][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:15:44,382][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:15:49,407][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:15:54,408][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:15:59,424][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:16:04,456][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:16:09,457][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:16:14,469][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:16:19,459][ ERROR][PID:74861] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-04 21:16:19,471][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:16:24,472][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:16:29,460][WARNING][PID:74861] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-04 21:16:29,473][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:16:34,474][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:16:39,461][ INFO][PID:74861] Downloading the builder-live.log file, attempt 2 [2026-02-04 21:16:39,462][ INFO][PID:74861] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18a copr-rpmbuild-log [2026-02-04 21:16:39,475][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:16:44,477][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:16:49,478][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:16:54,479][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:16:59,485][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:17:04,486][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:17:09,487][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:17:14,488][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:17:19,489][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:17:24,490][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:17:29,491][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:17:34,492][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:17:39,493][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:17:44,494][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:17:49,495][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:17:54,496][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:17:59,497][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:18:04,521][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:18:09,522][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:18:14,523][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:18:19,526][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:18:24,529][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:18:29,530][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:18:34,531][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:18:39,532][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:18:44,534][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:18:49,536][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:18:54,537][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:18:59,541][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:19:04,543][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:19:09,549][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:19:14,564][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:19:19,567][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:19:24,571][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:19:29,573][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:19:34,574][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:19:39,576][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:19:44,577][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:19:49,578][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:19:54,579][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:19:59,580][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:20:04,581][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:20:05,646][ INFO][PID:74861] Downloading results from builder [2026-02-04 21:20:05,647][ INFO][PID:74861] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18a]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/ufven/podman/epel-9-ppc64le/10094424-container-selinux started [2026-02-04 21:20:05,648][ INFO][PID:74861] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18a]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/ufven/podman/epel-9-ppc64le/10094424-container-selinux/ &> /var/lib/copr/public_html/results/ufven/podman/epel-9-ppc64le/10094424-container-selinux/build-10094424.rsync.log [2026-02-04 21:21:05,729][ INFO][PID:74861] rsyncing finished. [2026-02-04 21:21:05,730][ ERROR][PID:74861] Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-02-04 21:21:05,730][ ERROR][PID:74861] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-02-04 21:21:15,731][ INFO][PID:74861] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18a]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/ufven/podman/epel-9-ppc64le/10094424-container-selinux started [2026-02-04 21:21:15,732][ INFO][PID:74861] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18a]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/ufven/podman/epel-9-ppc64le/10094424-container-selinux/ &> /var/lib/copr/public_html/results/ufven/podman/epel-9-ppc64le/10094424-container-selinux/build-10094424.rsync.log [2026-02-04 21:22:15,793][ INFO][PID:74861] rsyncing finished. [2026-02-04 21:22:15,793][ ERROR][PID:74861] Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-02-04 21:22:15,794][ ERROR][PID:74861] SSH connection lost on #2 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-02-04 21:22:25,794][ INFO][PID:74861] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18a]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/ufven/podman/epel-9-ppc64le/10094424-container-selinux started [2026-02-04 21:22:25,795][ INFO][PID:74861] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18a]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/ufven/podman/epel-9-ppc64le/10094424-container-selinux/ &> /var/lib/copr/public_html/results/ufven/podman/epel-9-ppc64le/10094424-container-selinux/build-10094424.rsync.log [2026-02-04 21:23:25,880][ INFO][PID:74861] rsyncing finished. [2026-02-04 21:23:25,880][ ERROR][PID:74861] Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-02-04 21:23:25,880][ ERROR][PID:74861] SSH connection lost on #3 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-02-04 21:23:35,881][ ERROR][PID:74861] Re-try request for task on 'ResallocHost, ticket_id=1174624, hostname=2620:52:6:1161:dead:beef:cafe:c18a, name=vmhost_p09_04_prod_01941139_20260204_205735, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 3 SSH attempts [2026-02-04 21:23:35,882][ INFO][PID:74861] Releasing VM back to pool [2026-02-04 21:23:35,915][ INFO][PID:74861] Retry #1 (on other host) [2026-02-04 21:23:35,921][ INFO][PID:74861] Checking that /var/lib/copr/public_html/results/ufven/podman/epel-9-ppc64le/repodata/repomd.xml exists [2026-02-04 21:23:35,923][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:23:35,925][ INFO][PID:74861] VM allocation process starts [2026-02-04 21:23:35,973][ INFO][PID:74861] Trying to allocate VM: ResallocHost, ticket_id=1174726, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-02-04 21:23:39,014][ INFO][PID:74861] Allocated host ResallocHost, ticket_id=1174726, hostname=2620:52:6:1161:dead:beef:cafe:c10a, name=vmhost_p09_02_prod_01941246_20260204_210816, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-02-04 21:23:39,015][ INFO][PID:74861] Allocating ssh connection to builder [2026-02-04 21:23:39,016][ INFO][PID:74861] Checking that builder machine is OK [2026-02-04 21:24:39,086][ ERROR][PID:74861] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c10a port 22: Connection timed out [2026-02-04 21:24:49,087][ ERROR][PID:74861] Re-try request for task on 'ResallocHost, ticket_id=1174726, hostname=2620:52:6:1161:dead:beef:cafe:c10a, name=vmhost_p09_02_prod_01941246_20260204_210816, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 1 SSH attempts [2026-02-04 21:24:49,087][ INFO][PID:74861] Releasing VM back to pool [2026-02-04 21:24:49,097][ INFO][PID:74861] Retry #2 (on other host) [2026-02-04 21:24:49,098][ INFO][PID:74861] Checking that /var/lib/copr/public_html/results/ufven/podman/epel-9-ppc64le/repodata/repomd.xml exists [2026-02-04 21:24:49,098][ INFO][PID:74861] Checking for cancel request [2026-02-04 21:24:49,099][ INFO][PID:74861] VM allocation process starts [2026-02-04 21:24:49,108][ INFO][PID:74861] Trying to allocate VM: ResallocHost, ticket_id=1174747, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-02-04 21:24:52,125][ INFO][PID:74861] Allocated host ResallocHost, ticket_id=1174747, hostname=2620:52:6:1161:dead:beef:cafe:c14f, name=vmhost_p09_03_prod_01941320_20260204_210957, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-02-04 21:24:52,125][ INFO][PID:74861] Allocating ssh connection to builder [2026-02-04 21:24:52,126][ INFO][PID:74861] Checking that builder machine is OK [2026-02-04 21:25:52,173][ ERROR][PID:74861] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c14f port 22: Connection timed out [2026-02-04 21:26:02,174][ ERROR][PID:74861] Re-try request for task on 'ResallocHost, ticket_id=1174747, hostname=2620:52:6:1161:dead:beef:cafe:c14f, name=vmhost_p09_03_prod_01941320_20260204_210957, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 1 SSH attempts [2026-02-04 21:26:02,175][ INFO][PID:74861] Releasing VM back to pool [2026-02-04 21:26:02,183][ ERROR][PID:74861] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c14f', '2620:52:6:1161:dead:beef:cafe:c18a', '2620:52:6:1161:dead:beef:cafe:c10a'} [2026-02-04 21:26:02,183][WARNING][PID:74861] Switching not-finished job state to 'failed' [2026-02-04 21:26:02,184][ INFO][PID:74861] Worker failed build, took 644.5376083850861 [2026-02-04 21:26:02,184][ INFO][PID:74861] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "ufven", "project_name": "podman", "project_dirname": "podman", "submitter": null, "ended_on": 1770240362.1837976, "started_on": 1770239717.6461892, "submitted_on": null, "status": 0, "chroot": "epel-9-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "10094424-epel-9-ppc64le", "build_id": 10094424, "package_name": "container-selinux", "package_version": "102:2.245.0-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/ufven/podman/container-selinux", "git_hash": "97586f1c92754694944ef129ef1f439042a2ad65", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "ufven/podman--1e012b8d-70da-4964-8dfa-49b6d7106c0f", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/ufven/podman/epel-9-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le", "arch_power9" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/ufven/podman", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/ufven/podman", "result_dir": "10094424-container-selinux", "built_packages": "", "id": 10094424, "mockchain_macros": { "copr_username": "ufven", "copr_projectname": "podman", "vendor": "Fedora Project COPR (ufven/podman)" } } ] } [2026-02-04 21:26:02,230][ INFO][PID:74861] Sending fedora-messaging bus message in build.end [2026-02-04 21:26:02,255][ INFO][PID:74861] Compressing /var/lib/copr/public_html/results/ufven/podman/epel-9-ppc64le/10094424-container-selinux/builder-live.log by gzip [2026-02-04 21:26:02,256][ INFO][PID:74861] Running command 'gzip /var/lib/copr/public_html/results/ufven/podman/epel-9-ppc64le/10094424-container-selinux/builder-live.log' as PID 132932 [2026-02-04 21:26:02,302][ INFO][PID:74861] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/ufven/podman/epel-9-ppc64le/10094424-container-selinux/builder-live.log) [2026-02-04 21:26:02,303][ INFO][PID:74861] Compressing /var/lib/copr/public_html/results/ufven/podman/epel-9-ppc64le/10094424-container-selinux/backend.log by gzip [2026-02-04 21:26:02,304][ INFO][PID:74861] Running command 'gzip /var/lib/copr/public_html/results/ufven/podman/epel-9-ppc64le/10094424-container-selinux/backend.log' as PID 132934