[2026-01-28 21:50:45,730][ INFO][PID:2243715] Marking build as starting [2026-01-28 21:50:45,801][ INFO][PID:2243715] Checking that /var/lib/copr/public_html/results/mochaa/protobuf/fedora-rawhide-ppc64le/repodata/repomd.xml exists [2026-01-28 21:50:45,802][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:50:45,803][ INFO][PID:2243715] VM allocation process starts [2026-01-28 21:50:45,887][ INFO][PID:2243715] Trying to allocate VM: ResallocHost, ticket_id=1078171, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-01-28 21:50:48,933][ INFO][PID:2243715] Allocated host ResallocHost, ticket_id=1078171, hostname=2620:52:6:1161:dead:beef:cafe:c112, name=vmhost_p09_02_prod_01723239_20260128_205620, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-01-28 21:50:48,934][ INFO][PID:2243715] Allocating ssh connection to builder [2026-01-28 21:50:48,934][ INFO][PID:2243715] Checking that builder machine is OK [2026-01-28 21:50:49,896][ INFO][PID:2243715] Running remote command: copr-builder-ready fedora-rawhide-ppc64le [2026-01-28 21:51:49,961][ ERROR][PID:2243715] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c112 port 22: Connection timed out [2026-01-28 21:51:59,962][ ERROR][PID:2243715] Re-try request for task on 'ResallocHost, ticket_id=1078171, hostname=2620:52:6:1161:dead:beef:cafe:c112, name=vmhost_p09_02_prod_01723239_20260128_205620, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts [2026-01-28 21:51:59,963][ INFO][PID:2243715] Releasing VM back to pool [2026-01-28 21:51:59,971][ INFO][PID:2243715] Retry #1 (on other host) [2026-01-28 21:51:59,973][ INFO][PID:2243715] Checking that /var/lib/copr/public_html/results/mochaa/protobuf/fedora-rawhide-ppc64le/repodata/repomd.xml exists [2026-01-28 21:51:59,973][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:51:59,974][ INFO][PID:2243715] VM allocation process starts [2026-01-28 21:52:00,016][ INFO][PID:2243715] Trying to allocate VM: ResallocHost, ticket_id=1078183, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-01-28 21:52:03,073][ INFO][PID:2243715] Allocated host ResallocHost, ticket_id=1078183, hostname=2620:52:6:1161:dead:beef:cafe:c195, name=vmhost_p09_04_prod_01723767_20260128_212150, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-01-28 21:52:03,073][ INFO][PID:2243715] Allocating ssh connection to builder [2026-01-28 21:52:03,074][ INFO][PID:2243715] Checking that builder machine is OK [2026-01-28 21:53:03,143][ ERROR][PID:2243715] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c195 port 22: Connection timed out [2026-01-28 21:53:13,144][ ERROR][PID:2243715] Re-try request for task on 'ResallocHost, ticket_id=1078183, hostname=2620:52:6:1161:dead:beef:cafe:c195, name=vmhost_p09_04_prod_01723767_20260128_212150, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts [2026-01-28 21:53:13,146][ INFO][PID:2243715] Releasing VM back to pool [2026-01-28 21:53:13,166][ INFO][PID:2243715] Retry #2 (on other host) [2026-01-28 21:53:13,167][ INFO][PID:2243715] Checking that /var/lib/copr/public_html/results/mochaa/protobuf/fedora-rawhide-ppc64le/repodata/repomd.xml exists [2026-01-28 21:53:13,167][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:53:13,169][ INFO][PID:2243715] VM allocation process starts [2026-01-28 21:53:13,216][ INFO][PID:2243715] Trying to allocate VM: ResallocHost, ticket_id=1078195, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-01-28 21:53:16,267][ INFO][PID:2243715] Allocated host ResallocHost, ticket_id=1078195, hostname=2620:52:6:1161:dead:beef:cafe:c116, name=vmhost_p09_02_prod_01723781_20260128_212329, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-01-28 21:53:16,268][ INFO][PID:2243715] Allocating ssh connection to builder [2026-01-28 21:53:16,268][ INFO][PID:2243715] Checking that builder machine is OK [2026-01-28 21:53:17,282][ INFO][PID:2243715] Running remote command: copr-builder-ready fedora-rawhide-ppc64le [2026-01-28 21:53:17,892][ INFO][PID:2243715] Red Hat subscription not needed for fedora-rawhide-ppc64le Builder is ready to be used [2026-01-28 21:53:17,893][ INFO][PID:2243715] Filling build.info file with builder info [2026-01-28 21:53:17,894][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:53:17,895][ INFO][PID:2243715] Not re-notifying FE and msg buses for the new host. [2026-01-28 21:53:17,896][ INFO][PID:2243715] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10073520-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached [2026-01-28 21:53:18,943][ INFO][PID:2243715] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c116' (ED25519) to the list of known hosts. [2026-01-28 21:53:18,944][ INFO][PID:2243715] Downloading the builder-live.log file, attempt 1 [2026-01-28 21:53:18,945][ INFO][PID:2243715] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c116 copr-rpmbuild-log [2026-01-28 21:53:23,948][ INFO][PID:2243715] Periodic builder liveness probe: alive [2026-01-28 21:53:23,949][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:53:28,950][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:53:33,951][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:53:38,952][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:53:43,953][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:53:48,954][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:53:53,955][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:53:58,956][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:54:03,957][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:54:08,958][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:54:13,959][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:54:18,960][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:54:23,961][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:54:28,962][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:54:33,963][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:54:38,964][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:54:43,966][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:54:48,967][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:54:53,968][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:54:58,969][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:55:03,970][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:55:08,971][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:55:13,972][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:55:18,973][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:55:23,975][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:55:28,977][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:55:33,977][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:55:38,978][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:55:43,979][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:55:48,980][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:55:53,981][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:55:58,982][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:56:03,984][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:56:08,985][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:56:13,986][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:56:18,987][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:56:23,988][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:56:28,989][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:56:33,990][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:56:38,990][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:56:43,992][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:56:48,992][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:56:53,993][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:56:58,994][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:57:03,995][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:57:08,997][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:57:13,998][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:57:18,998][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:57:24,000][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:57:29,001][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:57:34,001][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:57:39,002][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:57:44,003][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:57:49,004][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:57:54,005][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:57:59,006][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:58:04,007][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:58:09,008][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:58:14,009][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:58:19,010][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:58:24,011][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:58:29,050][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:58:34,051][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:58:39,052][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:58:44,053][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:58:49,053][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:58:54,055][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:58:59,056][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:59:04,057][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:59:09,058][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:59:14,059][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:59:19,060][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:59:24,061][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:59:29,061][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:59:34,062][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:59:39,063][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:59:44,064][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:59:49,065][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:59:54,066][ INFO][PID:2243715] Checking for cancel request [2026-01-28 21:59:59,067][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:00:04,068][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:00:09,069][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:00:14,070][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:00:19,071][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:00:24,072][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:00:29,073][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:00:34,078][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:00:39,081][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:00:44,093][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:00:49,095][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:00:54,212][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:00:59,213][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:01:04,215][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:01:09,238][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:01:14,276][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:01:19,279][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:01:24,347][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:01:29,349][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:01:34,390][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:01:39,392][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:01:44,426][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:01:49,428][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:01:54,453][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:01:59,455][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:02:04,492][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:02:09,494][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:02:14,496][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:02:19,555][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:02:24,562][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:02:29,565][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:02:34,604][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:02:39,606][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:02:44,607][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:02:49,608][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:02:54,687][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:02:57,369][ ERROR][PID:2243715] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-28 22:02:59,690][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:03:04,692][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:03:07,370][WARNING][PID:2243715] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-28 22:03:09,695][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:03:14,697][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:03:17,371][ INFO][PID:2243715] Downloading the builder-live.log file, attempt 2 [2026-01-28 22:03:17,373][ INFO][PID:2243715] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c116 copr-rpmbuild-log [2026-01-28 22:03:18,420][ ERROR][PID:2243715] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-28 22:03:19,698][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:03:24,706][ INFO][PID:2243715] Periodic builder liveness probe: alive [2026-01-28 22:03:24,707][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:03:28,435][WARNING][PID:2243715] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-28 22:03:29,708][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:03:34,709][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:03:38,436][ INFO][PID:2243715] Downloading the builder-live.log file, attempt 3 [2026-01-28 22:03:38,437][ INFO][PID:2243715] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c116 copr-rpmbuild-log [2026-01-28 22:03:39,709][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:03:39,905][ ERROR][PID:2243715] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-28 22:03:44,762][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:03:49,764][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:03:49,920][WARNING][PID:2243715] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-28 22:03:54,765][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:03:59,767][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:03:59,920][ INFO][PID:2243715] Downloading the builder-live.log file, attempt 4 [2026-01-28 22:03:59,983][ INFO][PID:2243715] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c116 copr-rpmbuild-log [2026-01-28 22:04:01,046][ ERROR][PID:2243715] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-28 22:04:04,986][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:04:09,990][ INFO][PID:2243715] Checking for cancel request [2026-01-28 22:04:11,052][ ERROR][PID:2243715] Giving up for unstable SSH, failures: 1769637787.3706825, 1769637808.4350154, 1769637829.9200416, 1769637851.0525687 [2026-01-28 22:04:11,053][ ERROR][PID:2243715] Re-try request for task on 'ResallocHost, ticket_id=1078195, hostname=2620:52:6:1161:dead:beef:cafe:c116, name=vmhost_p09_02_prod_01723781_20260128_212329, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-01-28 22:04:11,054][ INFO][PID:2243715] Releasing VM back to pool [2026-01-28 22:04:11,066][ ERROR][PID:2243715] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c195', '2620:52:6:1161:dead:beef:cafe:c112', '2620:52:6:1161:dead:beef:cafe:c116'} [2026-01-28 22:04:11,066][WARNING][PID:2243715] Switching not-finished job state to 'failed' [2026-01-28 22:04:11,066][ INFO][PID:2243715] Worker failed build, took 653.1707556247711 [2026-01-28 22:04:11,067][ INFO][PID:2243715] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "mochaa", "project_name": "protobuf", "project_dirname": "protobuf", "submitter": "https://src.fedoraproject.org/user/music", "ended_on": 1769637851.0665603, "started_on": 1769637197.8958046, "submitted_on": null, "status": 0, "chroot": "fedora-rawhide-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "10073520-fedora-rawhide-ppc64le", "build_id": 10073520, "package_name": "frr", "package_version": "10.5.0-8", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/mochaa/protobuf/frr", "git_hash": "56df77b138881c98d191dbdef8763c7220f5db98", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "mochaa/protobuf--https://src.fedoraproject.org/user/music", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/mochaa/protobuf/fedora-rawhide-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/mochaa/protobuf", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/mochaa/protobuf", "result_dir": "10073520-frr", "built_packages": "", "id": 10073520, "mockchain_macros": { "copr_username": "mochaa", "copr_projectname": "protobuf", "vendor": "Fedora Project COPR (mochaa/protobuf)" } } ] } [2026-01-28 22:04:11,119][ INFO][PID:2243715] Sending fedora-messaging bus message in build.end [2026-01-28 22:04:11,644][ INFO][PID:2243715] Compressing /var/lib/copr/public_html/results/mochaa/protobuf/fedora-rawhide-ppc64le/10073520-frr/builder-live.log by gzip [2026-01-28 22:04:11,645][ INFO][PID:2243715] Running command 'gzip /var/lib/copr/public_html/results/mochaa/protobuf/fedora-rawhide-ppc64le/10073520-frr/builder-live.log' as PID 2298522 [2026-01-28 22:04:11,647][ INFO][PID:2243715] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/mochaa/protobuf/fedora-rawhide-ppc64le/10073520-frr/builder-live.log) [2026-01-28 22:04:11,647][ INFO][PID:2243715] Compressing /var/lib/copr/public_html/results/mochaa/protobuf/fedora-rawhide-ppc64le/10073520-frr/backend.log by gzip [2026-01-28 22:04:11,648][ INFO][PID:2243715] Running command 'gzip /var/lib/copr/public_html/results/mochaa/protobuf/fedora-rawhide-ppc64le/10073520-frr/backend.log' as PID 2298523