[2026-01-07 13:53:35,971][ INFO][PID:2917725] Marking build as starting [2026-01-07 13:53:36,032][ INFO][PID:2917725] Checking that https://download.copr.fedorainfracloud.org/results/@copr/copr-pull-requests/fedora-42-ppc64le//repodata/repomd.xml exists [2026-01-07 13:53:36,360][ INFO][PID:2917725] Checking that https://download.copr.fedorainfracloud.org/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le//repodata/repomd.xml exists [2026-01-07 13:53:36,747][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:53:36,752][ INFO][PID:2917725] VM allocation process starts [2026-01-07 13:53:36,772][ INFO][PID:2917725] Trying to allocate VM: ResallocHost, ticket_id=691262, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-01-07 13:53:39,855][ INFO][PID:2917725] Allocated host ResallocHost, ticket_id=691262, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_00986951_20260107_134534, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-01-07 13:53:39,891][ INFO][PID:2917725] Allocating ssh connection to builder [2026-01-07 13:53:39,895][ INFO][PID:2917725] Checking that builder machine is OK [2026-01-07 13:53:40,821][ INFO][PID:2917725] Running remote command: copr-builder-ready fedora-42-ppc64le [2026-01-07 13:53:41,397][ INFO][PID:2917725] Red Hat subscription not needed for fedora-42-ppc64le Builder is ready to be used [2026-01-07 13:53:41,399][ INFO][PID:2917725] Filling build.info file with builder info [2026-01-07 13:53:41,401][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:53:41,404][ INFO][PID:2917725] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@copr", "project_name": "copr-pull-requests", "project_dirname": "copr-pull-requests:pr:4106", "submitter": null, "ended_on": null, "started_on": 1767794021.4041495, "submitted_on": null, "status": 3, "chroot": "fedora-42-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9981752-fedora-42-ppc64le", "build_id": 9981752, "package_name": "copr-rpmbuild", "package_version": "1.6.post1-1.git.89.822ccd0", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@copr/copr-pull-requests:pr:4106/copr-rpmbuild", "git_hash": "35233e32d7b7848287b37d434584d13dcd8032b3", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@copr/copr-pull-requests--aa0cdbc1-3580-4423-92c7-7377f22e8922", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 1, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@copr/copr-pull-requests/fedora-42-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/", "id": "copr_coprdir", "name": "Coprdir repository", "priority": null }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/@copr/copr-dev/fedora-42-ppc64le/", "id": "copr_copr_copr_dev", "name": "Additional repo copr_copr_copr_dev" } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@copr/copr-pull-requests:pr:4106", "result_dir": "09981752-copr-rpmbuild", "built_packages": "", "id": 9981752, "mockchain_macros": { "copr_username": "@copr", "copr_projectname": "copr-pull-requests", "vendor": "Fedora Project COPR (@copr/copr-pull-requests)" } } ] } [2026-01-07 13:53:41,477][ INFO][PID:2917725] Sending fedora-messaging bus message in build.start [2026-01-07 13:53:42,324][ INFO][PID:2917725] Sending fedora-messaging bus message in chroot.start [2026-01-07 13:53:42,399][ INFO][PID:2917725] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9981752-fedora-42-ppc64le --chroot fedora-42-ppc64le --detached [2026-01-07 13:54:42,497][ ERROR][PID:2917725] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c191 port 22: Connection timed out [2026-01-07 13:54:52,499][ ERROR][PID:2917725] Re-try request for task on 'ResallocHost, ticket_id=691262, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_00986951_20260107_134534, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts [2026-01-07 13:54:52,500][ INFO][PID:2917725] Releasing VM back to pool [2026-01-07 13:54:52,511][ INFO][PID:2917725] Retry #1 (on other host) [2026-01-07 13:54:52,512][ INFO][PID:2917725] Checking that https://download.copr.fedorainfracloud.org/results/@copr/copr-pull-requests/fedora-42-ppc64le//repodata/repomd.xml exists [2026-01-07 13:54:53,241][ INFO][PID:2917725] Checking that https://download.copr.fedorainfracloud.org/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le//repodata/repomd.xml exists [2026-01-07 13:54:54,203][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:54:54,204][ INFO][PID:2917725] VM allocation process starts [2026-01-07 13:54:54,215][ INFO][PID:2917725] Trying to allocate VM: ResallocHost, ticket_id=691299, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-01-07 13:54:57,230][ INFO][PID:2917725] Allocated host ResallocHost, ticket_id=691299, hostname=2620:52:6:1161:dead:beef:cafe:c18c, name=vmhost_p09_04_prod_00987532_20260107_135016, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-01-07 13:54:57,231][ INFO][PID:2917725] Allocating ssh connection to builder [2026-01-07 13:54:57,231][ INFO][PID:2917725] Checking that builder machine is OK [2026-01-07 13:54:58,130][ INFO][PID:2917725] Running remote command: copr-builder-ready fedora-42-ppc64le [2026-01-07 13:54:58,634][ INFO][PID:2917725] Red Hat subscription not needed for fedora-42-ppc64le Builder is ready to be used [2026-01-07 13:54:58,634][ INFO][PID:2917725] Filling build.info file with builder info [2026-01-07 13:54:58,644][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:54:58,645][ INFO][PID:2917725] Not re-notifying FE and msg buses for the new host. [2026-01-07 13:54:58,646][ INFO][PID:2917725] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9981752-fedora-42-ppc64le --chroot fedora-42-ppc64le --detached [2026-01-07 13:54:59,628][ INFO][PID:2917725] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c18c' (ED25519) to the list of known hosts. [2026-01-07 13:54:59,629][ INFO][PID:2917725] Downloading the builder-live.log file, attempt 1 [2026-01-07 13:54:59,630][ INFO][PID:2917725] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18c copr-rpmbuild-log [2026-01-07 13:55:04,638][ INFO][PID:2917725] Periodic builder liveness probe: alive [2026-01-07 13:55:04,639][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:55:09,640][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:55:14,641][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:55:19,642][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:55:24,643][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:55:29,644][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:55:34,645][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:55:39,646][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:55:44,647][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:55:49,648][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:55:54,650][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:55:59,653][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:56:04,654][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:56:09,655][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:56:14,656][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:56:19,657][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:56:24,658][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:56:29,662][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:56:34,663][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:56:39,665][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:56:44,667][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:56:49,669][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:56:54,671][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:56:59,675][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:57:04,678][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:57:09,679][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:57:14,681][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:57:19,683][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:57:24,684][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:57:29,685][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:57:34,686][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:57:39,688][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:57:44,693][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:57:49,698][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:57:54,700][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:57:59,702][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:58:04,706][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:58:09,708][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:58:14,711][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:58:19,714][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:58:24,720][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:58:29,721][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:58:34,724][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:58:39,725][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:58:44,734][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:58:49,737][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:58:54,739][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:58:59,741][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:59:04,742][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:59:09,747][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:59:14,749][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:59:19,753][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:59:24,757][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:59:29,758][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:59:34,763][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:59:39,766][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:59:44,767][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:59:49,769][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:59:54,770][ INFO][PID:2917725] Checking for cancel request [2026-01-07 13:59:59,772][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:00:04,776][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:00:09,778][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:00:14,779][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:00:19,780][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:00:24,781][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:00:29,783][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:00:34,784][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:00:39,786][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:00:44,787][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:00:49,788][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:00:54,789][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:00:59,791][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:01:04,792][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:01:09,795][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:01:14,800][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:01:19,802][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:01:24,804][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:01:29,806][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:01:34,807][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:01:38,370][ ERROR][PID:2917725] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-07 14:01:39,808][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:01:44,811][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:01:48,372][WARNING][PID:2917725] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-07 14:01:49,818][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:01:54,819][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:01:58,375][ INFO][PID:2917725] Downloading the builder-live.log file, attempt 2 [2026-01-07 14:01:58,377][ INFO][PID:2917725] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18c copr-rpmbuild-log [2026-01-07 14:01:59,820][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:02:04,825][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:02:09,829][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:02:14,830][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:02:19,831][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:02:24,833][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:02:29,835][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:02:34,837][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:02:39,854][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:02:44,855][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:02:49,856][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:02:54,859][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:02:58,457][ ERROR][PID:2917725] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-07 14:02:59,859][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:03:04,861][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:03:08,467][WARNING][PID:2917725] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-07 14:03:09,863][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:03:14,865][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:03:18,468][ INFO][PID:2917725] Downloading the builder-live.log file, attempt 3 [2026-01-07 14:03:18,470][ INFO][PID:2917725] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18c copr-rpmbuild-log [2026-01-07 14:03:19,878][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:03:24,879][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:03:29,881][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:03:34,883][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:03:39,884][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:03:44,885][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:03:49,886][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:03:54,887][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:03:59,889][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:04:04,893][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:04:09,896][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:04:14,898][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:04:18,541][ ERROR][PID:2917725] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-07 14:04:19,899][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:04:24,900][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:04:28,558][WARNING][PID:2917725] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-07 14:04:29,902][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:04:34,903][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:04:38,559][ INFO][PID:2917725] Downloading the builder-live.log file, attempt 4 [2026-01-07 14:04:38,561][ INFO][PID:2917725] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18c copr-rpmbuild-log [2026-01-07 14:04:39,904][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:04:44,906][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:04:49,909][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:04:54,911][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:04:59,913][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:05:04,926][ INFO][PID:2917725] Periodic builder liveness probe: alive [2026-01-07 14:05:04,927][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:05:09,930][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:05:14,937][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:05:19,940][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:05:24,941][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:05:29,945][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:05:34,951][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:05:38,631][ ERROR][PID:2917725] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-07 14:05:39,952][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:05:44,954][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:05:48,649][ ERROR][PID:2917725] Giving up for unstable SSH, failures: 1767794508.3727343, 1767794588.467667, 1767794668.5589097, 1767794748.649554 [2026-01-07 14:05:48,650][ ERROR][PID:2917725] Re-try request for task on 'ResallocHost, ticket_id=691299, hostname=2620:52:6:1161:dead:beef:cafe:c18c, name=vmhost_p09_04_prod_00987532_20260107_135016, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-01-07 14:05:48,651][ INFO][PID:2917725] Releasing VM back to pool [2026-01-07 14:05:48,665][ INFO][PID:2917725] Retry #2 (on other host) [2026-01-07 14:05:48,666][ INFO][PID:2917725] Checking that https://download.copr.fedorainfracloud.org/results/@copr/copr-pull-requests/fedora-42-ppc64le//repodata/repomd.xml exists [2026-01-07 14:05:49,473][ INFO][PID:2917725] Checking that https://download.copr.fedorainfracloud.org/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le//repodata/repomd.xml exists [2026-01-07 14:05:50,375][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:05:50,388][ INFO][PID:2917725] VM allocation process starts [2026-01-07 14:05:50,434][ INFO][PID:2917725] Trying to allocate VM: ResallocHost, ticket_id=691462, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-01-07 14:05:53,513][ INFO][PID:2917725] Allocated host ResallocHost, ticket_id=691462, hostname=2620:52:6:1161:dead:beef:cafe:c148, name=vmhost_p09_03_prod_00988516_20260107_135728, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-01-07 14:05:53,515][ INFO][PID:2917725] Allocating ssh connection to builder [2026-01-07 14:05:53,516][ INFO][PID:2917725] Checking that builder machine is OK [2026-01-07 14:05:54,440][ INFO][PID:2917725] Running remote command: copr-builder-ready fedora-42-ppc64le [2026-01-07 14:05:54,935][ INFO][PID:2917725] Red Hat subscription not needed for fedora-42-ppc64le Builder is ready to be used [2026-01-07 14:05:54,936][ INFO][PID:2917725] Filling build.info file with builder info [2026-01-07 14:05:54,942][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:05:54,943][ INFO][PID:2917725] Not re-notifying FE and msg buses for the new host. [2026-01-07 14:05:54,944][ INFO][PID:2917725] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9981752-fedora-42-ppc64le --chroot fedora-42-ppc64le --detached [2026-01-07 14:05:55,928][ INFO][PID:2917725] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c148' (ED25519) to the list of known hosts. [2026-01-07 14:05:55,929][ INFO][PID:2917725] Downloading the builder-live.log file, attempt 1 [2026-01-07 14:05:55,930][ INFO][PID:2917725] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c148 copr-rpmbuild-log [2026-01-07 14:06:00,929][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:06:05,931][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:06:10,932][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:06:15,934][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:06:20,939][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:06:25,941][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:06:30,944][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:06:35,948][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:06:40,956][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:06:45,961][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:06:50,966][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:06:55,969][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:07:00,973][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:07:05,979][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:07:10,987][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:07:15,990][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:07:21,001][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:07:26,007][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:07:31,008][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:07:36,012][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:07:41,018][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:07:46,020][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:07:51,027][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:07:56,045][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:08:01,055][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:08:06,056][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:08:11,060][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:08:16,061][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:08:21,067][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:08:26,070][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:08:31,074][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:08:36,076][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:08:41,080][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:08:46,081][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:08:51,083][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:08:56,086][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:09:01,087][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:09:06,092][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:09:11,094][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:09:16,100][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:09:21,101][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:09:26,102][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:09:31,103][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:09:36,104][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:09:41,106][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:09:46,108][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:09:51,109][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:09:56,110][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:10:01,111][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:10:06,114][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:10:11,118][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:10:16,119][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:10:21,121][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:10:26,123][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:10:31,125][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:10:36,128][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:10:41,132][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:10:46,133][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:10:51,134][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:10:56,135][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:11:01,137][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:11:06,140][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:11:11,143][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:11:16,144][ INFO][PID:2917725] Checking for cancel request [2026-01-07 14:11:20,708][ INFO][PID:2917725] Downloading results from builder [2026-01-07 14:11:20,708][ INFO][PID:2917725] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c148]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild started [2026-01-07 14:11:20,712][ INFO][PID:2917725] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c148]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/ &> /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/build-09981752.rsync.log [2026-01-07 14:12:20,884][ INFO][PID:2917725] rsyncing finished. [2026-01-07 14:12:20,898][ ERROR][PID:2917725] Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-01-07 14:12:20,900][ ERROR][PID:2917725] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-01-07 14:12:30,901][ INFO][PID:2917725] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c148]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild started [2026-01-07 14:12:30,903][ INFO][PID:2917725] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c148]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/ &> /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/build-09981752.rsync.log [2026-01-07 14:12:32,128][ INFO][PID:2917725] rsyncing finished. [2026-01-07 14:12:32,131][ INFO][PID:2917725] Releasing VM back to pool [2026-01-07 14:12:32,150][ INFO][PID:2917725] Searching for 'success' file in resultdir [2026-01-07 14:12:32,151][ INFO][PID:2917725] Going to sign pkgs from source: 9981752-fedora-42-ppc64le in chroot: /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le [2026-01-07 14:12:32,152][ INFO][PID:2917725] Calling '/bin/sign -u @copr#copr-pull-requests@copr.fedorahosted.org -p' (attempt #1) [2026-01-07 14:12:32,425][ INFO][PID:2917725] Calling '/bin/sign -4 -h sha256 -u @copr#copr-pull-requests@copr.fedorahosted.org -r /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/copr-rpmbuild-1.6.post1-1.git.89.822ccd0.fc42.ppc64le.rpm' (attempt #1) [2026-01-07 14:12:33,167][ INFO][PID:2917725] signed rpm: /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/copr-rpmbuild-1.6.post1-1.git.89.822ccd0.fc42.ppc64le.rpm [2026-01-07 14:12:33,169][ INFO][PID:2917725] Calling '/bin/sign -4 -h sha256 -u @copr#copr-pull-requests@copr.fedorahosted.org -r /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/copr-rpmbuild-1.6.post1-1.git.89.822ccd0.fc42.src.rpm' (attempt #1) [2026-01-07 14:12:33,907][ INFO][PID:2917725] signed rpm: /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/copr-rpmbuild-1.6.post1-1.git.89.822ccd0.fc42.src.rpm [2026-01-07 14:12:33,908][ INFO][PID:2917725] Calling '/bin/sign -4 -h sha256 -u @copr#copr-pull-requests@copr.fedorahosted.org -r /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/copr-builder-1.6.post1-1.git.89.822ccd0.fc42.ppc64le.rpm' (attempt #1) [2026-01-07 14:12:34,646][ INFO][PID:2917725] signed rpm: /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/copr-builder-1.6.post1-1.git.89.822ccd0.fc42.ppc64le.rpm [2026-01-07 14:12:34,647][ INFO][PID:2917725] Sign done [2026-01-07 14:12:34,648][ INFO][PID:2917725] Pulp: create_content: /api/v3/content/rpm/packages/upload/ /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/copr-rpmbuild-1.6.post1-1.git.89.822ccd0.fc42.ppc64le.rpm [2026-01-07 14:12:35,408][ INFO][PID:2917725] Pulp: create_content: /api/v3/content/rpm/packages/upload/ /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/copr-rpmbuild-1.6.post1-1.git.89.822ccd0.fc42.src.rpm [2026-01-07 14:12:35,410][ INFO][PID:2917725] [1/3] Uploaded to Pulp: /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/copr-rpmbuild-1.6.post1-1.git.89.822ccd0.fc42.ppc64le.rpm [2026-01-07 14:12:36,013][ INFO][PID:2917725] Pulp: create_content: /api/v3/content/rpm/packages/upload/ /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/copr-builder-1.6.post1-1.git.89.822ccd0.fc42.ppc64le.rpm [2026-01-07 14:12:36,017][ INFO][PID:2917725] [2/3] Uploaded to Pulp: /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/copr-rpmbuild-1.6.post1-1.git.89.822ccd0.fc42.src.rpm [2026-01-07 14:12:36,570][ INFO][PID:2917725] [3/3] Uploaded to Pulp: /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/copr-builder-1.6.post1-1.git.89.822ccd0.fc42.ppc64le.rpm [2026-01-07 14:12:36,580][ INFO][PID:2917725] Pulp: get_repository: /api/v3/repositories/rpm/rpm/?name=%40copr%2Fcopr-pull-requests%3Apr%3A4106%2Ffedora-42-ppc64le&offset=0&limit=1 [2026-01-07 14:12:37,120][ INFO][PID:2917725] Pulp: polling task status: /api/pulp/public-copr/api/v3/tasks/019b98cd-6821-7a17-9c70-1aaa39ef3ab9/ [2026-01-07 14:12:37,124][ INFO][PID:2917725] Pulp: get_by_href: /api/pulp/public-copr/api/v3/tasks/019b98cd-6821-7a17-9c70-1aaa39ef3ab9/ [2026-01-07 14:12:42,443][ INFO][PID:2917725] Pulp: polling task status: /api/pulp/public-copr/api/v3/tasks/019b98cd-6821-7a17-9c70-1aaa39ef3ab9/ [2026-01-07 14:12:42,444][ INFO][PID:2917725] Pulp: get_by_href: /api/pulp/public-copr/api/v3/tasks/019b98cd-6821-7a17-9c70-1aaa39ef3ab9/ [2026-01-07 14:12:42,750][ INFO][PID:2917725] Successfully modified Pulp repository content /api/pulp/public-copr/api/v3/repositories/rpm/rpm/019b93d9-bc50-7f6a-a9ba-776500fa1596/ [2026-01-07 14:12:42,755][ INFO][PID:2917725] Pulp: publishing /api/v3/publications/rpm/rpm/ /api/pulp/public-copr/api/v3/repositories/rpm/rpm/019b93d9-bc50-7f6a-a9ba-776500fa1596/ [2026-01-07 14:12:43,004][ INFO][PID:2917725] Pulp: polling task status: /api/pulp/public-copr/api/v3/tasks/019b98cd-7f18-7dc9-9146-d167007394ef/ [2026-01-07 14:12:43,006][ INFO][PID:2917725] Pulp: get_by_href: /api/pulp/public-copr/api/v3/tasks/019b98cd-7f18-7dc9-9146-d167007394ef/ [2026-01-07 14:12:48,359][ INFO][PID:2917725] Pulp: polling task status: /api/pulp/public-copr/api/v3/tasks/019b98cd-7f18-7dc9-9146-d167007394ef/ [2026-01-07 14:12:48,360][ INFO][PID:2917725] Pulp: get_by_href: /api/pulp/public-copr/api/v3/tasks/019b98cd-7f18-7dc9-9146-d167007394ef/ [2026-01-07 14:12:48,755][ INFO][PID:2917725] Successfully created Pulp publication of repository /api/pulp/public-copr/api/v3/repositories/rpm/rpm/019b93d9-bc50-7f6a-a9ba-776500fa1596/ [2026-01-07 14:12:48,760][ INFO][PID:2917725] Pulp: add_content: /api/pulp/public-copr/api/v3/repositories/rpm/rpm/019b93d9-bc50-7f6a-a9ba-776500fa1596/ (['/api/pulp/public-copr/api/v3/content/rpm/packages/019b98cd-6175-7218-8ae1-bb6e80dfa730/', '/api/pulp/public-copr/api/v3/content/rpm/packages/019b98cd-63de-7a75-9bed-78bbb4037ffd/', '/api/pulp/public-copr/api/v3/content/rpm/packages/019b98cd-65ff-7eb6-8c57-a01473bb0016/']) [2026-01-07 14:12:48,762][ INFO][PID:2917725] Getting build details [2026-01-07 14:12:48,762][ INFO][PID:2917725] Listing built binary packages in /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild [2026-01-07 14:12:48,763][ INFO][PID:2917725] Built packages: ['copr-rpmbuild 1.6.post1', 'copr-builder 1.6.post1'] [2026-01-07 14:12:48,763][ INFO][PID:2917725] build details: {'built_packages': 'copr-rpmbuild 1.6.post1\ncopr-builder 1.6.post1'} [2026-01-07 14:12:48,764][ INFO][PID:2917725] Retrieving pubkey [2026-01-07 14:12:48,765][ INFO][PID:2917725] Calling '/bin/sign -u @copr#copr-pull-requests@copr.fedorahosted.org -p' (attempt #1) [2026-01-07 14:12:49,052][ INFO][PID:2917725] Added pubkey for user @copr project copr-pull-requests into: /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/pubkey.gpg [2026-01-07 14:12:49,053][ INFO][PID:2917725] Removing /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/copr-rpmbuild-1.6.post1-1.git.89.822ccd0.fc42.ppc64le.rpm, it is stored in Pulp [2026-01-07 14:12:49,054][ INFO][PID:2917725] Removing /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/copr-rpmbuild-1.6.post1-1.git.89.822ccd0.fc42.src.rpm, it is stored in Pulp [2026-01-07 14:12:49,055][ INFO][PID:2917725] Removing /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/copr-builder-1.6.post1-1.git.89.822ccd0.fc42.ppc64le.rpm, it is stored in Pulp [2026-01-07 14:12:49,056][ INFO][PID:2917725] Finished build: id=9981752 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106 chroot=fedora-42-ppc64le [2026-01-07 14:12:49,059][ INFO][PID:2917725] Worker succeeded build, took 414.1155467033386 [2026-01-07 14:12:49,060][ INFO][PID:2917725] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@copr", "project_name": "copr-pull-requests", "project_dirname": "copr-pull-requests:pr:4106", "submitter": null, "ended_on": 1767795169.0594923, "started_on": 1767794754.9439456, "submitted_on": null, "status": 1, "chroot": "fedora-42-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9981752-fedora-42-ppc64le", "build_id": 9981752, "package_name": "copr-rpmbuild", "package_version": "1.6.post1-1.git.89.822ccd0", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@copr/copr-pull-requests:pr:4106/copr-rpmbuild", "git_hash": "35233e32d7b7848287b37d434584d13dcd8032b3", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@copr/copr-pull-requests--aa0cdbc1-3580-4423-92c7-7377f22e8922", "results": { "packages": [ { "name": "copr-rpmbuild", "epoch": null, "version": "1.6.post1", "release": "1.git.89.822ccd0.fc42", "arch": "ppc64le" }, { "name": "copr-rpmbuild", "epoch": null, "version": "1.6.post1", "release": "1.git.89.822ccd0.fc42", "arch": "src" }, { "name": "copr-builder", "epoch": null, "version": "1.6.post1", "release": "1.git.89.822ccd0.fc42", "arch": "ppc64le" } ] }, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 1, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@copr/copr-pull-requests/fedora-42-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/", "id": "copr_coprdir", "name": "Coprdir repository", "priority": null }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/@copr/copr-dev/fedora-42-ppc64le/", "id": "copr_copr_copr_dev", "name": "Additional repo copr_copr_copr_dev" } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@copr/copr-pull-requests:pr:4106", "result_dir": "09981752-copr-rpmbuild", "built_packages": "copr-rpmbuild 1.6.post1\ncopr-builder 1.6.post1", "id": 9981752, "mockchain_macros": { "copr_username": "@copr", "copr_projectname": "copr-pull-requests", "vendor": "Fedora Project COPR (@copr/copr-pull-requests)" } } ] } [2026-01-07 14:12:49,137][ INFO][PID:2917725] Sending fedora-messaging bus message in build.end [2026-01-07 14:12:49,166][ INFO][PID:2917725] Compressing /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/builder-live.log by gzip [2026-01-07 14:12:49,167][ INFO][PID:2917725] Running command 'gzip /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/builder-live.log' as PID 3086756 [2026-01-07 14:12:49,213][ INFO][PID:2917725] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/builder-live.log) [2026-01-07 14:12:49,214][ INFO][PID:2917725] Compressing /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/backend.log by gzip [2026-01-07 14:12:49,218][ INFO][PID:2917725] Running command 'gzip /var/lib/copr/public_html/results/@copr/copr-pull-requests:pr:4106/fedora-42-ppc64le/09981752-copr-rpmbuild/backend.log' as PID 3086764