[2026-04-06 19:42:06,106][ INFO][PID:3364743] Marking build as starting [2026-04-06 19:42:06,149][ INFO][PID:3364743] Checking that https://download.copr.fedorainfracloud.org/results/pvermeer/sunshine/fedora-43-x86_64//repodata/repomd.xml exists [2026-04-06 19:42:06,557][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:42:06,558][ INFO][PID:3364743] VM allocation process starts [2026-04-06 19:42:06,577][ INFO][PID:3364743] Trying to allocate VM: ResallocHost, ticket_id=2194225, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-06 19:42:09,588][ INFO][PID:3364743] Allocated host ResallocHost, ticket_id=2194225, hostname=54.209.245.238, name=aws_x86_64_spot_prod_05879533_20260406_193624, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-06 19:42:09,588][ INFO][PID:3364743] Allocating ssh connection to builder [2026-04-06 19:42:09,589][ INFO][PID:3364743] Checking that builder machine is OK [2026-04-06 19:42:09,924][ INFO][PID:3364743] Running remote command: copr-builder-ready fedora-43-x86_64 [2026-04-06 19:42:10,124][ INFO][PID:3364743] Red Hat subscription not needed for fedora-43-x86_64 Builder is ready to be used [2026-04-06 19:42:10,125][ INFO][PID:3364743] Filling build.info file with builder info [2026-04-06 19:42:10,125][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:42:10,126][ INFO][PID:3364743] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "pvermeer", "project_name": "sunshine", "project_dirname": "sunshine", "submitter": null, "ended_on": null, "started_on": 1775504530.1265516, "submitted_on": null, "status": 3, "chroot": "fedora-43-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10295699-fedora-43-x86_64", "build_id": 10295699, "package_name": "sunshine", "package_version": "2025.924.154138-2", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/pvermeer/sunshine/sunshine", "git_hash": "c2000470e90444440cd4f4877cf21e365611958c", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "pvermeer/sunshine--cfe0e8d9-c9b5-4933-a5e3-3e73e394c6f9", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 1, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/pvermeer/sunshine/fedora-43-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/pvermeer/sunshine", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/pvermeer/sunshine", "result_dir": "10295699-sunshine", "built_packages": "", "id": 10295699, "mockchain_macros": { "copr_username": "pvermeer", "copr_projectname": "sunshine", "vendor": "Fedora Project COPR (pvermeer/sunshine)" } } ] } [2026-04-06 19:42:10,176][ INFO][PID:3364743] Sending fedora-messaging bus message in build.start [2026-04-06 19:42:10,716][ INFO][PID:3364743] Sending fedora-messaging bus message in chroot.start [2026-04-06 19:42:10,746][ INFO][PID:3364743] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10295699-fedora-43-x86_64 --chroot fedora-43-x86_64 --detached [2026-04-06 19:42:11,081][ INFO][PID:3364743] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '54.209.245.238' (ED25519) to the list of known hosts. [2026-04-06 19:42:11,081][ INFO][PID:3364743] Downloading the builder-live.log file, attempt 1 [2026-04-06 19:42:11,083][ INFO][PID:3364743] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.209.245.238 copr-rpmbuild-log [2026-04-06 19:42:16,089][ INFO][PID:3364743] Periodic builder liveness probe: alive [2026-04-06 19:42:16,089][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:42:21,090][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:42:26,091][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:42:31,092][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:42:36,093][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:42:41,095][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:42:46,096][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:42:51,097][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:42:56,098][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:43:01,099][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:43:06,100][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:43:11,101][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:43:16,102][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:43:21,103][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:43:26,104][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:43:31,105][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:43:36,106][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:43:41,107][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:43:46,108][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:43:51,109][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:43:56,110][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:44:01,111][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:44:06,112][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:44:11,113][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:44:16,114][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:44:21,115][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:44:26,116][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:44:31,117][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:44:36,118][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:44:41,119][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:44:46,120][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:44:51,121][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:44:56,122][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:45:01,123][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:45:06,123][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:45:11,124][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:45:13,545][ ERROR][PID:3364743] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-06 19:45:16,125][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:45:21,126][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:45:24,576][WARNING][PID:3364743] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-06 19:45:26,127][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:45:31,128][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:45:34,576][ INFO][PID:3364743] Downloading the builder-live.log file, attempt 2 [2026-04-06 19:45:34,578][ INFO][PID:3364743] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.209.245.238 copr-rpmbuild-log [2026-04-06 19:45:36,129][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:45:41,130][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:45:46,131][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:45:51,132][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:45:56,133][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:46:01,134][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:46:06,135][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:46:11,136][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:46:16,137][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:46:21,137][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:46:26,138][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:46:31,139][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:46:34,628][ ERROR][PID:3364743] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-06 19:46:36,140][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:46:41,141][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:46:44,632][WARNING][PID:3364743] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-06 19:46:46,142][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:46:51,143][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:46:54,633][ INFO][PID:3364743] Downloading the builder-live.log file, attempt 3 [2026-04-06 19:46:54,978][ INFO][PID:3364743] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.209.245.238 copr-rpmbuild-log [2026-04-06 19:46:56,144][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:47:01,145][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:47:06,146][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:47:11,147][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:47:16,147][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:47:21,148][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:47:26,149][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:47:31,150][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:47:36,151][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:47:41,152][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:47:46,153][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:47:51,154][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:47:55,043][ ERROR][PID:3364743] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-06 19:47:56,155][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:48:01,156][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:48:05,051][WARNING][PID:3364743] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-06 19:48:06,157][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:48:11,158][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:48:15,052][ INFO][PID:3364743] Downloading the builder-live.log file, attempt 4 [2026-04-06 19:48:15,053][ INFO][PID:3364743] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.209.245.238 copr-rpmbuild-log [2026-04-06 19:48:16,159][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:48:21,160][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:48:26,161][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:48:31,163][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:48:36,164][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:48:41,165][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:48:46,166][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:48:51,167][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:48:56,168][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:49:01,169][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:49:06,170][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:49:11,171][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:49:15,120][ ERROR][PID:3364743] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-06 19:49:16,172][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:49:21,172][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:49:25,281][ ERROR][PID:3364743] Giving up for unstable SSH, failures: 1775504724.5759985, 1775504804.6325936, 1775504885.0519595, 1775504965.281587 [2026-04-06 19:49:25,282][ ERROR][PID:3364743] Re-try request for task on 'ResallocHost, ticket_id=2194225, hostname=54.209.245.238, name=aws_x86_64_spot_prod_05879533_20260406_193624, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-04-06 19:49:25,283][ INFO][PID:3364743] Releasing VM back to pool [2026-04-06 19:49:25,292][ INFO][PID:3364743] Retry #1 (on other host) [2026-04-06 19:49:25,292][ INFO][PID:3364743] Checking that https://download.copr.fedorainfracloud.org/results/pvermeer/sunshine/fedora-43-x86_64//repodata/repomd.xml exists [2026-04-06 19:49:26,169][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:49:26,170][ INFO][PID:3364743] VM allocation process starts [2026-04-06 19:49:26,181][ INFO][PID:3364743] Trying to allocate VM: ResallocHost, ticket_id=2194266, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-06 19:49:29,197][ INFO][PID:3364743] Allocated host ResallocHost, ticket_id=2194266, hostname=54.164.26.115, name=aws_x86_64_spot_prod_05879616_20260406_194213, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-06 19:49:29,198][ INFO][PID:3364743] Allocating ssh connection to builder [2026-04-06 19:49:29,198][ INFO][PID:3364743] Checking that builder machine is OK [2026-04-06 19:49:29,505][ INFO][PID:3364743] Running remote command: copr-builder-ready fedora-43-x86_64 [2026-04-06 19:49:29,710][ INFO][PID:3364743] Red Hat subscription not needed for fedora-43-x86_64 Builder is ready to be used [2026-04-06 19:49:29,710][ INFO][PID:3364743] Filling build.info file with builder info [2026-04-06 19:49:29,713][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:49:29,713][ INFO][PID:3364743] Not re-notifying FE and msg buses for the new host. [2026-04-06 19:49:29,714][ INFO][PID:3364743] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10295699-fedora-43-x86_64 --chroot fedora-43-x86_64 --detached [2026-04-06 19:49:30,039][ INFO][PID:3364743] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '54.164.26.115' (ED25519) to the list of known hosts. [2026-04-06 19:49:30,040][ INFO][PID:3364743] Downloading the builder-live.log file, attempt 1 [2026-04-06 19:49:30,041][ INFO][PID:3364743] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.164.26.115 copr-rpmbuild-log [2026-04-06 19:49:35,040][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:49:40,041][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:49:45,042][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:49:50,043][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:49:55,044][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:50:00,045][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:50:05,046][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:50:10,047][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:50:15,048][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:50:20,049][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:50:25,049][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:50:30,050][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:50:35,347][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:50:40,348][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:50:45,349][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:50:47,200][ ERROR][PID:3364743] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-06 19:50:50,350][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:50:55,351][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:50:57,201][WARNING][PID:3364743] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-06 19:51:00,352][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:51:05,354][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:51:07,202][ INFO][PID:3364743] Downloading the builder-live.log file, attempt 2 [2026-04-06 19:51:07,203][ INFO][PID:3364743] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.164.26.115 copr-rpmbuild-log [2026-04-06 19:51:10,483][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:51:15,485][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:51:20,486][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:51:25,487][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:51:30,488][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:51:35,490][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:51:40,490][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:51:45,492][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:51:50,493][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:51:55,671][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:52:00,672][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:52:05,673][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:52:07,270][ ERROR][PID:3364743] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-06 19:52:10,674][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:52:15,966][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:52:17,290][WARNING][PID:3364743] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-06 19:52:20,975][ INFO][PID:3364743] Periodic builder liveness probe: alive [2026-04-06 19:52:20,975][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:52:25,976][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:52:27,291][ INFO][PID:3364743] Downloading the builder-live.log file, attempt 3 [2026-04-06 19:52:27,293][ INFO][PID:3364743] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.164.26.115 copr-rpmbuild-log [2026-04-06 19:52:30,978][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:52:36,076][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:52:41,078][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:52:46,079][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:52:51,080][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:52:56,089][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:53:01,091][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:53:06,177][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:53:11,253][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:53:16,309][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:53:21,317][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:53:26,374][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:53:27,361][ ERROR][PID:3364743] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-06 19:53:31,629][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:53:36,637][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:53:37,362][WARNING][PID:3364743] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-06 19:53:41,640][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:53:46,642][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:53:47,364][ INFO][PID:3364743] Downloading the builder-live.log file, attempt 4 [2026-04-06 19:53:47,365][ INFO][PID:3364743] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.164.26.115 copr-rpmbuild-log [2026-04-06 19:53:51,826][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:53:56,827][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:54:01,828][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:54:06,830][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:54:12,354][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:54:17,357][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:54:22,358][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:54:27,358][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:54:32,359][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:54:37,360][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:54:42,389][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:54:47,410][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:54:47,429][ ERROR][PID:3364743] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-06 19:54:52,412][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:54:57,430][ ERROR][PID:3364743] Giving up for unstable SSH, failures: 1775505057.201473, 1775505137.290479, 1775505217.36286, 1775505297.4305446 [2026-04-06 19:54:57,432][ ERROR][PID:3364743] Re-try request for task on 'ResallocHost, ticket_id=2194266, hostname=54.164.26.115, name=aws_x86_64_spot_prod_05879616_20260406_194213, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-04-06 19:54:57,432][ INFO][PID:3364743] Releasing VM back to pool [2026-04-06 19:54:57,444][ INFO][PID:3364743] Retry #2 (on other host) [2026-04-06 19:54:57,445][ INFO][PID:3364743] Checking that https://download.copr.fedorainfracloud.org/results/pvermeer/sunshine/fedora-43-x86_64//repodata/repomd.xml exists [2026-04-06 19:54:57,935][ INFO][PID:3364743] Checking for cancel request [2026-04-06 19:54:57,937][ INFO][PID:3364743] VM allocation process starts [2026-04-06 19:54:57,950][ INFO][PID:3364743] Trying to allocate VM: ResallocHost, ticket_id=2194320, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-06 19:55:00,968][ INFO][PID:3364743] Allocated host ResallocHost, ticket_id=2194320, hostname=3.85.7.185, name=aws_x86_64_spot_prod_05879763_20260406_195102, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-06 19:55:00,969][ INFO][PID:3364743] Allocating ssh connection to builder [2026-04-06 19:55:00,971][ INFO][PID:3364743] Checking that builder machine is OK [2026-04-06 19:56:01,040][ ERROR][PID:3364743] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 3.85.7.185 port 22: Connection timed out [2026-04-06 19:56:11,062][ ERROR][PID:3364743] Re-try request for task on 'ResallocHost, ticket_id=2194320, hostname=3.85.7.185, name=aws_x86_64_spot_prod_05879763_20260406_195102, requested_tags=['copr_builder', 'arch_x86_64']': Unable to finish after 1 SSH attempts [2026-04-06 19:56:11,062][ INFO][PID:3364743] Releasing VM back to pool [2026-04-06 19:56:11,072][ ERROR][PID:3364743] Backend process error: Three host tried without success: {'3.85.7.185', '54.209.245.238', '54.164.26.115'} [2026-04-06 19:56:11,073][WARNING][PID:3364743] Switching not-finished job state to 'failed' [2026-04-06 19:56:11,073][ INFO][PID:3364743] Worker failed build, took 401.3591504096985 [2026-04-06 19:56:11,073][ INFO][PID:3364743] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "pvermeer", "project_name": "sunshine", "project_dirname": "sunshine", "submitter": null, "ended_on": 1775505371.0730245, "started_on": 1775504969.713874, "submitted_on": null, "status": 0, "chroot": "fedora-43-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10295699-fedora-43-x86_64", "build_id": 10295699, "package_name": "sunshine", "package_version": "2025.924.154138-2", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/pvermeer/sunshine/sunshine", "git_hash": "c2000470e90444440cd4f4877cf21e365611958c", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "pvermeer/sunshine--cfe0e8d9-c9b5-4933-a5e3-3e73e394c6f9", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 1, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/pvermeer/sunshine/fedora-43-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/pvermeer/sunshine", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/pvermeer/sunshine", "result_dir": "10295699-sunshine", "built_packages": "", "id": 10295699, "mockchain_macros": { "copr_username": "pvermeer", "copr_projectname": "sunshine", "vendor": "Fedora Project COPR (pvermeer/sunshine)" } } ] } [2026-04-06 19:56:11,117][ INFO][PID:3364743] Sending fedora-messaging bus message in build.end [2026-04-06 19:56:11,147][ INFO][PID:3364743] Compressing /var/lib/copr/public_html/results/pvermeer/sunshine/fedora-43-x86_64/10295699-sunshine/builder-live.log by gzip [2026-04-06 19:56:11,148][ INFO][PID:3364743] Running command 'gzip /var/lib/copr/public_html/results/pvermeer/sunshine/fedora-43-x86_64/10295699-sunshine/builder-live.log' as PID 3430058 [2026-04-06 19:56:11,162][ INFO][PID:3364743] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/pvermeer/sunshine/fedora-43-x86_64/10295699-sunshine/builder-live.log) [2026-04-06 19:56:11,163][ INFO][PID:3364743] Compressing /var/lib/copr/public_html/results/pvermeer/sunshine/fedora-43-x86_64/10295699-sunshine/backend.log by gzip [2026-04-06 19:56:11,164][ INFO][PID:3364743] Running command 'gzip /var/lib/copr/public_html/results/pvermeer/sunshine/fedora-43-x86_64/10295699-sunshine/backend.log' as PID 3430061