[2025-08-25 13:38:34,174][ INFO][PID:1451911] Marking build as starting [2025-08-25 13:38:34,268][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:38:34,270][ INFO][PID:1451911] VM allocation process starts [2025-08-25 13:38:34,305][ INFO][PID:1451911] Trying to allocate VM: ResallocHost, ticket_id=4707585, requested_tags=['copr_builder', 'arch_x86_64'] [2025-08-25 13:38:39,312][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:38:44,315][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:38:49,317][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:38:54,323][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:38:59,327][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:39:04,329][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:39:09,333][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:39:14,334][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:39:19,336][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:39:24,337][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:39:29,339][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:39:34,341][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:39:39,349][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:39:44,356][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:39:49,358][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:39:54,359][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:39:59,362][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:40:04,365][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:40:09,367][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:40:14,368][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:40:19,369][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:40:24,371][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:40:29,375][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:40:34,378][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:40:39,382][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:40:44,384][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:40:49,385][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:40:54,386][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:40:59,387][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:41:04,390][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:41:09,395][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:41:14,402][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:41:19,406][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:41:24,407][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:41:29,410][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:41:34,414][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:41:39,418][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:41:44,421][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:41:49,422][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:41:52,521][ INFO][PID:1451911] Allocated host ResallocHost, ticket_id=4707585, hostname=54.84.86.199, name=aws_x86_64_spot_prod_06410452_20250825_134017, requested_tags=['copr_builder', 'arch_x86_64'] [2025-08-25 13:41:52,522][ INFO][PID:1451911] Allocating ssh connection to builder [2025-08-25 13:41:52,526][ INFO][PID:1451911] Checking that builder machine is OK [2025-08-25 13:41:53,034][ INFO][PID:1451911] Running remote command: copr-builder-ready fedora-43-x86_64 [2025-08-25 13:41:53,246][ INFO][PID:1451911] Red Hat subscription not needed for fedora-43-x86_64 Builder is ready to be used [2025-08-25 13:41:53,247][ INFO][PID:1451911] Filling build.info file with builder info [2025-08-25 13:41:53,252][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:41:53,257][ INFO][PID:1451911] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@python", "project_name": "python-in-pulp", "project_dirname": "python-in-pulp", "submitter": "https://src.fedoraproject.org/user/nikic", "ended_on": null, "started_on": 1756129313.2540321, "submitted_on": null, "status": 3, "chroot": "fedora-43-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9472015-fedora-43-x86_64", "build_id": 9472015, "package_name": "llvm", "package_version": "20.1.8-26", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@python/python-in-pulp/llvm", "git_hash": "c23a5508edbf47b022ada8ee1cc64ed6d6677f2d", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@python/python-in-pulp--https://src.fedoraproject.org/user/nikic", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 1, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@python/python-in-pulp/fedora-43-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@python/python-in-pulp", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@python/python-in-pulp", "result_dir": "09472015-llvm", "built_packages": "", "id": 9472015, "mockchain_macros": { "copr_username": "@python", "copr_projectname": "python-in-pulp", "vendor": "Fedora Project COPR (@python/python-in-pulp)" } } ] } [2025-08-25 13:41:53,298][ INFO][PID:1451911] Sending fedora-messaging bus message in build.start [2025-08-25 13:41:54,253][ INFO][PID:1451911] Sending fedora-messaging bus message in chroot.start [2025-08-25 13:41:54,279][ INFO][PID:1451911] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9472015-fedora-43-x86_64 --chroot fedora-43-x86_64 --detached [2025-08-25 13:41:54,615][ INFO][PID:1451911] Downloading the builder-live.log file, attempt 1 [2025-08-25 13:41:54,617][ INFO][PID:1451911] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.84.86.199 copr-rpmbuild-log [2025-08-25 13:41:59,626][ INFO][PID:1451911] Periodic builder liveness probe: alive [2025-08-25 13:41:59,627][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:42:04,628][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:42:09,630][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:42:14,631][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:42:19,633][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:42:24,634][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:42:29,635][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:42:34,637][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:42:39,638][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:42:44,639][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:42:49,641][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:42:54,642][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:42:59,643][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:43:04,645][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:43:09,646][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:43:14,648][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:43:19,654][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:43:24,656][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:43:29,658][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:43:34,659][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:43:39,660][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:43:44,661][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:43:49,662][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:43:54,663][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:43:59,665][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:44:04,666][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:44:09,667][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:44:14,668][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:44:19,669][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:44:24,671][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:44:29,672][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:44:34,674][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:44:39,675][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:44:44,676][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:44:49,677][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:44:54,678][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:44:59,679][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:45:04,680][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:45:09,681][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:45:14,682][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:45:19,684][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:45:24,685][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:45:29,686][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:45:34,687][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:45:39,688][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:45:44,689][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:45:49,690][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:45:54,691][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:45:59,692][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:46:04,694][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:46:09,695][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:46:14,696][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:46:19,697][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:46:24,698][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:46:29,699][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:46:34,700][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:46:39,701][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:46:44,703][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:46:49,704][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:46:54,705][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:46:59,707][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:47:04,708][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:47:09,709][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:47:14,711][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:47:19,712][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:47:24,713][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:47:29,714][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:47:34,715][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:47:39,716][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:47:44,717][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:47:49,718][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:47:54,719][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:47:59,721][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:48:04,722][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:48:09,723][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:48:14,724][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:48:19,725][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:48:24,726][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:48:29,730][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:48:34,731][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:48:39,732][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:48:44,733][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:48:49,734][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:48:54,736][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:48:59,737][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:49:04,738][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:49:09,739][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:49:14,741][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:49:19,745][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:49:24,747][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:49:29,749][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:49:34,750][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:49:39,751][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:49:44,752][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:49:49,754][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:49:54,755][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:49:59,757][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:50:04,758][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:50:09,761][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:50:10,295][ ERROR][PID:1451911] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-25 13:50:14,761][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:50:19,763][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:50:20,295][WARNING][PID:1451911] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-08-25 13:50:24,765][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:50:29,766][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:50:30,296][ INFO][PID:1451911] Downloading the builder-live.log file, attempt 2 [2025-08-25 13:50:30,299][ INFO][PID:1451911] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.84.86.199 copr-rpmbuild-log [2025-08-25 13:50:34,767][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:50:39,768][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:50:44,769][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:50:49,770][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:50:54,772][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:50:59,773][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:51:04,775][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:51:09,776][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:51:14,777][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:51:19,784][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:51:24,785][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:51:29,786][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:51:30,367][ ERROR][PID:1451911] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-25 13:51:34,787][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:51:39,788][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:51:40,370][WARNING][PID:1451911] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-08-25 13:51:44,789][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:51:49,791][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:51:50,371][ INFO][PID:1451911] Downloading the builder-live.log file, attempt 3 [2025-08-25 13:51:50,372][ INFO][PID:1451911] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.84.86.199 copr-rpmbuild-log [2025-08-25 13:51:54,792][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:51:59,803][ INFO][PID:1451911] Periodic builder liveness probe: alive [2025-08-25 13:51:59,804][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:52:04,806][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:52:09,807][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:52:14,808][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:52:19,810][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:52:24,811][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:52:29,812][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:52:34,813][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:52:39,814][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:52:44,815][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:52:49,816][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:52:50,440][ ERROR][PID:1451911] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-25 13:52:54,817][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:52:59,818][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:53:00,445][WARNING][PID:1451911] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-08-25 13:53:04,819][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:53:09,820][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:53:10,445][ INFO][PID:1451911] Downloading the builder-live.log file, attempt 4 [2025-08-25 13:53:10,447][ INFO][PID:1451911] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.84.86.199 copr-rpmbuild-log [2025-08-25 13:53:14,822][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:53:19,823][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:53:24,824][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:53:29,825][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:53:34,826][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:53:39,827][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:53:44,828][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:53:49,829][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:53:54,831][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:53:59,832][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:54:04,833][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:54:09,834][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:54:10,516][ ERROR][PID:1451911] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-25 13:54:14,836][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:54:19,837][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:54:20,522][ ERROR][PID:1451911] Giving up for unstable SSH, failures: 1756129820.2957509, 1756129900.3706326, 1756129980.4450672, 1756130060.522734 [2025-08-25 13:54:20,523][ ERROR][PID:1451911] Re-try request for task on 'ResallocHost, ticket_id=4707585, hostname=54.84.86.199, name=aws_x86_64_spot_prod_06410452_20250825_134017, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-08-25 13:54:20,524][ INFO][PID:1451911] Releasing VM back to pool [2025-08-25 13:54:20,535][ INFO][PID:1451911] Retry #1 (on other host) [2025-08-25 13:54:20,658][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:54:20,659][ INFO][PID:1451911] VM allocation process starts [2025-08-25 13:54:20,672][ INFO][PID:1451911] Trying to allocate VM: ResallocHost, ticket_id=4707794, requested_tags=['copr_builder', 'arch_x86_64'] [2025-08-25 13:54:25,674][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:54:30,675][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:54:35,676][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:54:40,677][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:54:45,678][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:54:50,679][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:54:55,680][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:55:00,681][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:55:05,683][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:55:10,684][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:55:15,688][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:55:20,689][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:55:25,690][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:55:30,691][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:55:35,692][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:55:40,693][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:55:45,694][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:55:50,695][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:55:55,696][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:56:00,697][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:56:05,699][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:56:10,700][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:56:15,701][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:56:20,702][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:56:25,703][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:56:30,704][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:56:35,705][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:56:40,706][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:56:45,707][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:56:50,708][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:56:55,709][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:57:00,710][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:57:05,711][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:57:10,712][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:57:15,713][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:57:20,714][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:57:25,716][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:57:30,717][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:57:35,718][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:57:40,719][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:57:45,720][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:57:50,721][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:57:55,722][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:58:00,723][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:58:05,724][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:58:10,725][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:58:15,726][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:58:20,727][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:58:25,729][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:58:30,730][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:58:35,731][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:58:40,732][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:58:45,733][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:58:50,734][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:58:55,735][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:59:00,738][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:59:05,739][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:59:10,740][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:59:15,742][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:59:20,742][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:59:25,743][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:59:30,744][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:59:35,745][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:59:40,746][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:59:45,748][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:59:50,749][ INFO][PID:1451911] Checking for cancel request [2025-08-25 13:59:55,750][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:00:00,751][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:00:05,752][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:00:10,753][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:00:15,754][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:00:20,755][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:00:25,756][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:00:30,757][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:00:35,758][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:00:38,769][ INFO][PID:1451911] Allocated host ResallocHost, ticket_id=4707794, hostname=54.152.7.140, name=aws_x86_64_spot_prod_06411557_20250825_135859, requested_tags=['copr_builder', 'arch_x86_64'] [2025-08-25 14:00:38,769][ INFO][PID:1451911] Allocating ssh connection to builder [2025-08-25 14:00:38,770][ INFO][PID:1451911] Checking that builder machine is OK [2025-08-25 14:00:39,177][ INFO][PID:1451911] Running remote command: copr-builder-ready fedora-43-x86_64 [2025-08-25 14:00:39,373][ INFO][PID:1451911] Red Hat subscription not needed for fedora-43-x86_64 Builder is ready to be used [2025-08-25 14:00:39,373][ INFO][PID:1451911] Filling build.info file with builder info [2025-08-25 14:00:39,377][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:00:39,377][ INFO][PID:1451911] Not re-notifying FE and msg buses for the new host. [2025-08-25 14:00:39,378][ INFO][PID:1451911] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9472015-fedora-43-x86_64 --chroot fedora-43-x86_64 --detached [2025-08-25 14:00:39,691][ INFO][PID:1451911] Downloading the builder-live.log file, attempt 1 [2025-08-25 14:00:39,693][ INFO][PID:1451911] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.152.7.140 copr-rpmbuild-log [2025-08-25 14:00:44,692][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:00:49,693][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:00:54,694][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:00:59,695][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:01:04,697][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:01:09,698][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:01:14,699][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:01:19,700][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:01:24,701][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:01:29,702][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:01:34,703][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:01:39,704][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:01:44,705][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:01:49,707][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:01:54,708][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:01:59,709][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:02:04,722][ INFO][PID:1451911] Periodic builder liveness probe: alive [2025-08-25 14:02:04,722][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:02:09,723][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:02:14,724][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:02:19,725][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:02:24,726][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:02:29,728][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:02:34,729][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:02:39,730][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:02:44,731][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:02:49,732][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:02:54,733][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:02:59,734][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:03:04,736][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:03:09,737][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:03:14,738][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:03:19,739][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:03:24,741][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:03:29,742][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:03:34,744][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:03:39,745][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:03:44,746][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:03:49,747][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:03:54,748][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:03:59,749][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:04:04,750][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:04:09,751][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:04:14,753][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:04:19,754][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:04:24,755][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:04:29,756][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:04:34,757][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:04:39,759][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:04:44,760][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:04:49,761][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:04:54,762][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:04:59,764][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:05:04,765][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:05:09,766][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:05:14,767][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:05:19,768][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:05:24,769][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:05:29,771][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:05:34,772][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:05:39,778][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:05:44,780][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:05:49,782][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:05:54,783][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:05:59,784][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:06:04,786][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:06:09,787][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:06:14,789][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:06:19,790][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:06:24,793][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:06:29,797][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:06:34,798][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:06:39,799][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:06:44,800][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:06:49,802][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:06:54,803][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:06:59,804][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:07:04,805][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:07:09,806][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:07:14,811][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:07:19,813][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:07:24,814][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:07:29,816][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:07:34,817][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:07:39,819][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:07:44,820][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:07:49,824][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:07:54,825][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:07:59,830][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:08:04,831][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:08:09,832][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:08:14,833][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:08:19,838][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:08:24,839][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:08:29,843][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:08:34,848][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:08:39,852][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:08:44,859][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:08:49,865][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:08:54,868][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:08:59,870][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:09:04,874][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:09:09,877][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:09:14,878][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:09:19,888][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:09:24,891][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:09:29,894][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:09:34,896][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:09:39,907][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:09:44,909][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:09:49,912][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:09:54,917][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:09:59,922][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:10:04,924][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:10:09,926][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:10:14,932][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:10:19,933][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:10:24,937][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:10:29,941][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:10:34,943][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:10:39,944][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:10:44,946][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:10:49,948][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:10:54,952][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:10:59,955][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:11:04,957][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:11:09,962][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:11:14,964][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:11:19,966][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:11:24,969][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:11:29,970][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:11:34,973][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:11:39,978][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:11:44,981][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:11:49,985][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:11:54,987][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:11:59,988][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:12:05,000][ INFO][PID:1451911] Periodic builder liveness probe: alive [2025-08-25 14:12:05,001][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:12:10,003][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:12:15,006][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:12:20,009][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:12:25,010][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:12:30,015][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:12:35,019][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:12:40,020][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:12:45,021][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:12:50,023][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:12:55,025][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:13:00,026][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:13:05,028][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:13:10,029][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:13:15,032][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:13:20,034][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:13:25,035][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:13:30,038][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:13:35,039][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:13:40,042][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:13:45,044][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:13:50,045][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:13:51,033][ ERROR][PID:1451911] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-25 14:13:55,046][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:14:00,048][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:14:01,034][WARNING][PID:1451911] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-08-25 14:14:05,049][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:14:10,050][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:14:11,035][ INFO][PID:1451911] Downloading the builder-live.log file, attempt 2 [2025-08-25 14:14:11,037][ INFO][PID:1451911] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.152.7.140 copr-rpmbuild-log [2025-08-25 14:14:15,051][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:14:20,052][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:14:25,054][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:14:30,055][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:14:35,056][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:14:40,058][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:14:45,059][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:14:50,060][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:14:55,061][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:15:00,063][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:15:05,065][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:15:10,066][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:15:11,109][ ERROR][PID:1451911] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-25 14:15:15,069][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:15:20,079][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:15:21,113][WARNING][PID:1451911] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-08-25 14:15:25,080][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:15:30,082][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:15:31,115][ INFO][PID:1451911] Downloading the builder-live.log file, attempt 3 [2025-08-25 14:15:31,118][ INFO][PID:1451911] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.152.7.140 copr-rpmbuild-log [2025-08-25 14:15:35,084][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:15:40,086][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:15:45,089][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:15:50,090][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:15:55,091][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:16:00,103][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:16:05,104][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:16:10,105][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:16:15,107][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:16:20,108][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:16:25,109][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:16:30,110][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:16:31,188][ ERROR][PID:1451911] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-25 14:16:35,111][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:16:40,112][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:16:41,191][WARNING][PID:1451911] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-08-25 14:16:45,113][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:16:50,114][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:16:51,192][ INFO][PID:1451911] Downloading the builder-live.log file, attempt 4 [2025-08-25 14:16:51,196][ INFO][PID:1451911] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.152.7.140 copr-rpmbuild-log [2025-08-25 14:16:55,116][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:17:00,119][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:17:05,120][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:17:10,127][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:17:15,128][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:17:20,129][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:17:25,130][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:17:30,132][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:17:35,133][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:17:40,134][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:17:45,136][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:17:50,137][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:17:51,268][ ERROR][PID:1451911] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-25 14:17:55,139][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:18:00,140][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:18:01,273][ ERROR][PID:1451911] Giving up for unstable SSH, failures: 1756131241.034417, 1756131321.1135302, 1756131401.1912053, 1756131481.2736309 [2025-08-25 14:18:01,274][ ERROR][PID:1451911] Re-try request for task on 'ResallocHost, ticket_id=4707794, hostname=54.152.7.140, name=aws_x86_64_spot_prod_06411557_20250825_135859, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-08-25 14:18:01,275][ INFO][PID:1451911] Releasing VM back to pool [2025-08-25 14:18:01,289][ INFO][PID:1451911] Retry #2 (on other host) [2025-08-25 14:18:01,317][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:18:01,319][ INFO][PID:1451911] VM allocation process starts [2025-08-25 14:18:01,383][ INFO][PID:1451911] Trying to allocate VM: ResallocHost, ticket_id=4708310, requested_tags=['copr_builder', 'arch_x86_64'] [2025-08-25 14:18:06,385][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:18:11,389][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:18:16,391][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:18:21,393][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:18:26,395][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:18:31,397][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:18:36,398][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:18:41,403][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:18:46,404][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:18:51,406][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:18:56,410][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:19:01,411][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:19:06,422][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:19:11,426][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:19:16,427][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:19:21,428][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:19:26,430][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:19:29,494][ INFO][PID:1451911] Allocated host ResallocHost, ticket_id=4708310, hostname=13.217.56.17, name=aws_x86_64_spot_prod_06411978_20250825_141756, requested_tags=['copr_builder', 'arch_x86_64'] [2025-08-25 14:19:29,495][ INFO][PID:1451911] Allocating ssh connection to builder [2025-08-25 14:19:29,495][ INFO][PID:1451911] Checking that builder machine is OK [2025-08-25 14:19:29,954][ INFO][PID:1451911] Running remote command: copr-builder-ready fedora-43-x86_64 [2025-08-25 14:19:30,164][ INFO][PID:1451911] Red Hat subscription not needed for fedora-43-x86_64 Builder is ready to be used [2025-08-25 14:19:30,165][ INFO][PID:1451911] Filling build.info file with builder info [2025-08-25 14:19:30,170][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:19:30,172][ INFO][PID:1451911] Not re-notifying FE and msg buses for the new host. [2025-08-25 14:19:30,172][ INFO][PID:1451911] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9472015-fedora-43-x86_64 --chroot fedora-43-x86_64 --detached [2025-08-25 14:19:30,537][ INFO][PID:1451911] Downloading the builder-live.log file, attempt 1 [2025-08-25 14:19:30,538][ INFO][PID:1451911] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.217.56.17 copr-rpmbuild-log [2025-08-25 14:19:35,537][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:19:40,538][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:19:45,540][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:19:50,543][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:19:55,545][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:20:00,548][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:20:05,550][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:20:10,552][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:20:15,553][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:20:20,555][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:20:25,557][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:20:30,561][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:20:35,567][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:20:40,571][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:20:45,573][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:20:50,574][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:20:55,575][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:21:00,577][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:21:05,579][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:21:10,580][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:21:15,582][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:21:20,583][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:21:25,584][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:21:30,585][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:21:35,586][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:21:40,587][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:21:45,588][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:21:50,590][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:21:55,592][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:22:00,598][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:22:05,610][ INFO][PID:1451911] Periodic builder liveness probe: alive [2025-08-25 14:22:05,612][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:22:10,613][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:22:15,615][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:22:20,619][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:22:25,620][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:22:30,623][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:22:35,624][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:22:40,625][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:22:45,631][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:22:50,633][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:22:55,634][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:23:00,635][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:23:05,637][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:23:10,639][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:23:15,641][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:23:20,645][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:23:25,648][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:23:30,649][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:23:35,650][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:23:40,651][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:23:45,653][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:23:50,654][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:23:55,656][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:24:00,661][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:24:05,663][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:24:10,664][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:24:15,666][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:24:20,667][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:24:25,668][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:24:30,671][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:24:35,672][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:24:40,673][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:24:45,676][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:24:50,677][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:24:55,678][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:25:00,679][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:25:05,682][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:25:10,687][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:25:15,688][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:25:20,690][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:25:25,692][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:25:30,693][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:25:35,695][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:25:40,697][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:25:45,698][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:25:50,702][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:25:55,703][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:26:00,704][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:26:05,705][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:26:10,706][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:26:15,708][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:26:20,710][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:26:25,712][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:26:30,714][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:26:35,717][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:26:40,724][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:26:45,726][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:26:50,732][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:26:55,733][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:27:00,739][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:27:05,740][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:27:10,741][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:27:15,744][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:27:20,754][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:27:25,755][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:27:30,757][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:27:35,763][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:27:40,765][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:27:45,767][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:27:50,769][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:27:55,771][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:28:00,772][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:28:05,774][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:28:10,777][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:28:15,779][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:28:20,780][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:28:25,784][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:28:30,786][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:28:35,798][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:28:40,803][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:28:45,805][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:28:50,806][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:28:55,808][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:29:00,809][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:29:05,812][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:29:10,816][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:29:15,818][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:29:20,819][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:29:25,820][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:29:30,823][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:29:35,826][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:29:40,828][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:29:45,835][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:29:50,836][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:29:55,838][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:30:00,840][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:30:05,847][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:30:10,851][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:30:15,853][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:30:20,854][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:30:25,855][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:30:30,858][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:30:35,860][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:30:40,861][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:30:45,863][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:30:50,865][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:30:55,866][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:31:00,868][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:31:05,869][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:31:10,870][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:31:15,873][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:31:20,874][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:31:25,877][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:31:30,878][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:31:35,880][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:31:40,881][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:31:45,886][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:31:50,887][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:31:55,888][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:32:00,890][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:32:05,901][ INFO][PID:1451911] Periodic builder liveness probe: alive [2025-08-25 14:32:05,901][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:32:10,903][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:32:15,906][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:32:20,910][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:32:25,911][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:32:30,914][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:32:35,915][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:32:40,918][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:32:45,919][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:32:50,921][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:32:55,923][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:33:00,933][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:33:05,936][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:33:10,937][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:33:15,939][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:33:20,940][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:33:25,942][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:33:30,944][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:33:35,945][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:33:40,947][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:33:45,949][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:33:50,951][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:33:55,953][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:34:00,957][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:34:05,958][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:34:10,963][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:34:15,965][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:34:16,033][ ERROR][PID:1451911] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-25 14:34:20,967][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:34:25,970][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:34:26,038][WARNING][PID:1451911] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-08-25 14:34:30,976][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:34:35,980][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:34:36,039][ INFO][PID:1451911] Downloading the builder-live.log file, attempt 2 [2025-08-25 14:34:36,042][ INFO][PID:1451911] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.217.56.17 copr-rpmbuild-log [2025-08-25 14:34:40,981][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:34:45,987][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:34:50,990][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:34:55,991][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:35:00,993][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:35:05,995][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:35:10,997][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:35:15,998][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:35:21,001][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:35:26,004][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:35:31,006][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:35:36,009][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:35:36,111][ ERROR][PID:1451911] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-25 14:35:41,010][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:35:46,011][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:35:46,114][WARNING][PID:1451911] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-08-25 14:35:51,012][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:35:56,013][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:35:56,115][ INFO][PID:1451911] Downloading the builder-live.log file, attempt 3 [2025-08-25 14:35:56,116][ INFO][PID:1451911] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.217.56.17 copr-rpmbuild-log [2025-08-25 14:36:01,015][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:36:06,016][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:36:11,018][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:36:16,019][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:36:21,022][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:36:26,024][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:36:31,025][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:36:36,028][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:36:41,029][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:36:46,032][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:36:51,033][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:36:56,034][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:36:56,184][ ERROR][PID:1451911] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-25 14:37:01,035][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:37:06,037][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:37:06,185][WARNING][PID:1451911] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-08-25 14:37:11,042][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:37:16,044][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:37:16,186][ INFO][PID:1451911] Downloading the builder-live.log file, attempt 4 [2025-08-25 14:37:16,188][ INFO][PID:1451911] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.217.56.17 copr-rpmbuild-log [2025-08-25 14:37:21,046][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:37:26,048][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:37:31,049][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:37:36,050][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:37:41,053][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:37:46,054][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:37:51,056][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:37:56,066][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:38:01,067][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:38:06,068][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:38:11,071][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:38:16,074][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:38:16,266][ ERROR][PID:1451911] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-25 14:38:21,075][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:38:26,077][ INFO][PID:1451911] Checking for cancel request [2025-08-25 14:38:26,268][ ERROR][PID:1451911] Giving up for unstable SSH, failures: 1756132466.03814, 1756132546.1142151, 1756132626.1852007, 1756132706.2684288 [2025-08-25 14:38:26,269][ ERROR][PID:1451911] Re-try request for task on 'ResallocHost, ticket_id=4708310, hostname=13.217.56.17, name=aws_x86_64_spot_prod_06411978_20250825_141756, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-08-25 14:38:26,270][ INFO][PID:1451911] Releasing VM back to pool [2025-08-25 14:38:26,288][ ERROR][PID:1451911] Backend process error: Three host tried without success: {'54.84.86.199', '13.217.56.17', '54.152.7.140'} [2025-08-25 14:38:26,289][WARNING][PID:1451911] Switching not-finished job state to 'failed' [2025-08-25 14:38:26,289][ INFO][PID:1451911] Worker failed build, took 1136.117115497589 [2025-08-25 14:38:26,290][ INFO][PID:1451911] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@python", "project_name": "python-in-pulp", "project_dirname": "python-in-pulp", "submitter": "https://src.fedoraproject.org/user/nikic", "ended_on": 1756132706.289088, "started_on": 1756131570.1719725, "submitted_on": null, "status": 0, "chroot": "fedora-43-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9472015-fedora-43-x86_64", "build_id": 9472015, "package_name": "llvm", "package_version": "20.1.8-26", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@python/python-in-pulp/llvm", "git_hash": "c23a5508edbf47b022ada8ee1cc64ed6d6677f2d", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@python/python-in-pulp--https://src.fedoraproject.org/user/nikic", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 1, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@python/python-in-pulp/fedora-43-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@python/python-in-pulp", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@python/python-in-pulp", "result_dir": "09472015-llvm", "built_packages": "", "id": 9472015, "mockchain_macros": { "copr_username": "@python", "copr_projectname": "python-in-pulp", "vendor": "Fedora Project COPR (@python/python-in-pulp)" } } ] } [2025-08-25 14:38:26,339][ INFO][PID:1451911] Sending fedora-messaging bus message in build.end [2025-08-25 14:38:26,365][ INFO][PID:1451911] Compressing /var/lib/copr/public_html/results/@python/python-in-pulp/fedora-43-x86_64/09472015-llvm/builder-live.log by gzip [2025-08-25 14:38:26,366][ INFO][PID:1451911] Running command 'gzip /var/lib/copr/public_html/results/@python/python-in-pulp/fedora-43-x86_64/09472015-llvm/builder-live.log' as PID 1906003 [2025-08-25 14:38:26,367][ INFO][PID:1451911] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@python/python-in-pulp/fedora-43-x86_64/09472015-llvm/builder-live.log) [2025-08-25 14:38:26,368][ INFO][PID:1451911] Compressing /var/lib/copr/public_html/results/@python/python-in-pulp/fedora-43-x86_64/09472015-llvm/backend.log by gzip [2025-08-25 14:38:26,369][ INFO][PID:1451911] Running command 'gzip /var/lib/copr/public_html/results/@python/python-in-pulp/fedora-43-x86_64/09472015-llvm/backend.log' as PID 1906004