[2026-04-16 15:51:17,466][ INFO][PID:2336890] Marking build as starting [2026-04-16 15:51:17,560][ INFO][PID:2336890] Checking that /var/lib/copr/public_html/results/rhcontainerbot/bootc/rhel-10-x86_64/repodata/repomd.xml exists [2026-04-16 15:51:17,578][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:51:17,579][ INFO][PID:2336890] VM allocation process starts [2026-04-16 15:51:17,620][ INFO][PID:2336890] Trying to allocate VM: ResallocHost, ticket_id=2374052, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-16 15:51:22,623][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:51:23,702][ INFO][PID:2336890] Allocated host ResallocHost, ticket_id=2374052, hostname=174.129.128.105, name=aws_x86_64_spot_prod_06076938_20260416_154902, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-16 15:51:23,703][ INFO][PID:2336890] Allocating ssh connection to builder [2026-04-16 15:51:23,704][ INFO][PID:2336890] Checking that builder machine is OK [2026-04-16 15:51:24,095][ INFO][PID:2336890] Installed copr-rpmbuild version: 1.8 [2026-04-16 15:51:24,096][ INFO][PID:2336890] Running remote command: copr-builder-ready rhel-10-x86_64 [2026-04-16 15:51:24,320][ INFO][PID:2336890] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2026-04-16 15:51:24,322][ INFO][PID:2336890] Filling build.info file with builder info [2026-04-16 15:51:24,327][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:51:24,332][ INFO][PID:2336890] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "rhcontainerbot", "project_name": "bootc", "project_dirname": "bootc", "submitter": "packit", "ended_on": null, "started_on": 1776354684.3325312, "submitted_on": null, "status": 3, "chroot": "rhel-10-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10339742-rhel-10-x86_64", "build_id": 10339742, "package_name": "bootc", "package_version": "202604161547.g47c362089b-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rhcontainerbot/bootc/bootc", "git_hash": "4b38c193f0d328a0d06a6f37b2e8d37e6545dbfb", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "rhcontainerbot/bootc--packit", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/rhel-10-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/rhcontainerbot/bootc", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc", "result_dir": "10339742-bootc", "built_packages": "", "id": 10339742, "mockchain_macros": { "copr_username": "rhcontainerbot", "copr_projectname": "bootc", "vendor": "Fedora Project COPR (rhcontainerbot/bootc)" } } ] } [2026-04-16 15:51:24,414][ INFO][PID:2336890] Sending fedora-messaging bus message in build.start [2026-04-16 15:51:25,366][ INFO][PID:2336890] Sending fedora-messaging bus message in chroot.start [2026-04-16 15:51:25,403][ INFO][PID:2336890] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10339742-rhel-10-x86_64 --chroot rhel-10-x86_64 --detached [2026-04-16 15:51:25,894][ INFO][PID:2336890] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '174.129.128.105' (ED25519) to the list of known hosts. [2026-04-16 15:51:25,896][ INFO][PID:2336890] Downloading the builder-live.log file, attempt 1 [2026-04-16 15:51:25,899][ INFO][PID:2336890] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@174.129.128.105 copr-rpmbuild-log [2026-04-16 15:51:30,906][ INFO][PID:2336890] Periodic builder liveness probe: alive [2026-04-16 15:51:30,907][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:51:35,910][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:51:40,912][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:51:45,922][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:51:50,923][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:51:55,926][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:52:00,929][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:52:05,932][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:52:10,940][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:52:15,942][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:52:20,946][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:52:25,947][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:52:30,951][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:52:35,953][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:52:40,955][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:52:45,962][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:52:50,966][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:52:55,968][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:53:00,972][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:53:05,974][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:53:10,976][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:53:15,978][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:53:20,979][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:53:25,980][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:53:30,982][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:53:33,318][ ERROR][PID:2336890] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-16 15:53:35,983][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:53:40,985][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:53:43,319][WARNING][PID:2336890] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-16 15:53:45,986][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:53:50,987][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:53:53,320][ INFO][PID:2336890] Downloading the builder-live.log file, attempt 2 [2026-04-16 15:53:53,321][ INFO][PID:2336890] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@174.129.128.105 copr-rpmbuild-log [2026-04-16 15:53:55,988][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:54:00,989][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:54:05,989][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:54:10,990][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:54:15,991][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:54:20,992][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:54:25,993][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:54:30,995][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:54:35,996][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:54:40,997][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:54:45,998][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:54:50,999][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:54:53,389][ ERROR][PID:2336890] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-16 15:54:56,000][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:55:01,000][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:55:03,393][WARNING][PID:2336890] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-16 15:55:06,002][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:55:11,003][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:55:13,393][ INFO][PID:2336890] Downloading the builder-live.log file, attempt 3 [2026-04-16 15:55:13,395][ INFO][PID:2336890] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@174.129.128.105 copr-rpmbuild-log [2026-04-16 15:55:16,003][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:55:21,004][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:55:26,005][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:55:31,006][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:55:36,007][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:55:41,008][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:55:46,009][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:55:51,010][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:55:56,011][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:56:01,013][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:56:06,013][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:56:11,014][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:56:13,459][ ERROR][PID:2336890] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-16 15:56:16,015][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:56:21,016][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:56:23,465][WARNING][PID:2336890] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-16 15:56:26,017][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:56:31,018][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:56:33,466][ INFO][PID:2336890] Downloading the builder-live.log file, attempt 4 [2026-04-16 15:56:33,467][ INFO][PID:2336890] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@174.129.128.105 copr-rpmbuild-log [2026-04-16 15:56:36,019][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:56:41,020][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:56:46,021][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:56:51,022][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:56:56,023][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:57:01,023][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:57:06,025][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:57:11,026][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:57:16,027][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:57:21,028][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:57:26,029][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:57:31,031][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:57:33,533][ ERROR][PID:2336890] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-16 15:57:36,032][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:57:41,034][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:57:43,539][ ERROR][PID:2336890] Giving up for unstable SSH, failures: 1776354823.3194802, 1776354903.3931704, 1776354983.4655197, 1776355063.5394635 [2026-04-16 15:57:43,561][ ERROR][PID:2336890] Re-try request for task on 'ResallocHost, ticket_id=2374052, hostname=174.129.128.105, name=aws_x86_64_spot_prod_06076938_20260416_154902, requested_tags=['arch_x86_64', 'copr_builder']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-04-16 15:57:43,562][ INFO][PID:2336890] VM Release request [2026-04-16 15:57:43,572][ INFO][PID:2336890] Retry #1 (on other host) [2026-04-16 15:57:43,572][ INFO][PID:2336890] Checking that /var/lib/copr/public_html/results/rhcontainerbot/bootc/rhel-10-x86_64/repodata/repomd.xml exists [2026-04-16 15:57:43,573][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:57:43,574][ INFO][PID:2336890] VM allocation process starts [2026-04-16 15:57:43,612][ INFO][PID:2336890] Trying to allocate VM: ResallocHost, ticket_id=2374148, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-16 15:57:46,661][ INFO][PID:2336890] Allocated host ResallocHost, ticket_id=2374148, hostname=100.30.214.222, name=aws_x86_64_spot_prod_06076940_20260416_154918, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-16 15:57:46,662][ INFO][PID:2336890] Allocating ssh connection to builder [2026-04-16 15:57:46,662][ INFO][PID:2336890] Checking that builder machine is OK [2026-04-16 15:57:47,016][ INFO][PID:2336890] Installed copr-rpmbuild version: 1.8 [2026-04-16 15:57:47,016][ INFO][PID:2336890] Running remote command: copr-builder-ready rhel-10-x86_64 [2026-04-16 15:57:47,226][ INFO][PID:2336890] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2026-04-16 15:57:47,226][ INFO][PID:2336890] Filling build.info file with builder info [2026-04-16 15:57:47,230][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:57:47,231][ INFO][PID:2336890] Not re-notifying FE and msg buses for the new host. [2026-04-16 15:57:47,232][ INFO][PID:2336890] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10339742-rhel-10-x86_64 --chroot rhel-10-x86_64 --detached [2026-04-16 15:57:47,650][ INFO][PID:2336890] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '100.30.214.222' (ED25519) to the list of known hosts. [2026-04-16 15:57:47,651][ INFO][PID:2336890] Downloading the builder-live.log file, attempt 1 [2026-04-16 15:57:47,652][ INFO][PID:2336890] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@100.30.214.222 copr-rpmbuild-log [2026-04-16 15:57:52,651][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:57:57,652][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:58:02,653][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:58:07,654][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:58:12,661][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:58:17,665][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:58:22,666][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:58:27,667][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:58:32,676][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:58:37,677][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:58:42,678][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:58:47,679][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:58:52,680][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:58:57,681][ INFO][PID:2336890] Checking for cancel request [2026-04-16 15:58:57,681][ INFO][PID:2336890] Calling cancel callback, and waiting for thread.join() [2026-04-16 15:58:57,682][ INFO][PID:2336890] Canceling the build on the remote machine [2026-04-16 15:59:01,152][ INFO][PID:2336890] Cancel request succeeded out: success err: Warning: Permanently added '100.30.214.222' (ED25519) to the list of known hosts. [2026-04-16 15:59:01,153][ ERROR][PID:2336890] Build was canceled [2026-04-16 15:59:01,153][ INFO][PID:2336890] VM Release request [2026-04-16 15:59:01,163][WARNING][PID:2336890] Switching not-finished job state to 'failed' [2026-04-16 15:59:01,164][ INFO][PID:2336890] Worker failed build, took 73.93213629722595 [2026-04-16 15:59:01,165][ INFO][PID:2336890] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "rhcontainerbot", "project_name": "bootc", "project_dirname": "bootc", "submitter": "packit", "ended_on": 1776355141.163786, "started_on": 1776355067.2316496, "submitted_on": null, "status": 0, "chroot": "rhel-10-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10339742-rhel-10-x86_64", "build_id": 10339742, "package_name": "bootc", "package_version": "202604161547.g47c362089b-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rhcontainerbot/bootc/bootc", "git_hash": "4b38c193f0d328a0d06a6f37b2e8d37e6545dbfb", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "rhcontainerbot/bootc--packit", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/rhel-10-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/rhcontainerbot/bootc", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc", "result_dir": "10339742-bootc", "built_packages": "", "id": 10339742, "mockchain_macros": { "copr_username": "rhcontainerbot", "copr_projectname": "bootc", "vendor": "Fedora Project COPR (rhcontainerbot/bootc)" } } ] } [2026-04-16 15:59:01,240][ INFO][PID:2336890] Sending fedora-messaging bus message in build.end [2026-04-16 15:59:01,268][ INFO][PID:2336890] Compressing /var/lib/copr/public_html/results/rhcontainerbot/bootc/rhel-10-x86_64/10339742-bootc/builder-live.log by gzip [2026-04-16 15:59:01,270][ INFO][PID:2336890] Running command 'gzip /var/lib/copr/public_html/results/rhcontainerbot/bootc/rhel-10-x86_64/10339742-bootc/builder-live.log' as PID 2385557 [2026-04-16 15:59:01,289][ INFO][PID:2336890] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/rhcontainerbot/bootc/rhel-10-x86_64/10339742-bootc/builder-live.log) [2026-04-16 15:59:01,289][ INFO][PID:2336890] Compressing /var/lib/copr/public_html/results/rhcontainerbot/bootc/rhel-10-x86_64/10339742-bootc/backend.log by gzip [2026-04-16 15:59:01,290][ INFO][PID:2336890] Running command 'gzip /var/lib/copr/public_html/results/rhcontainerbot/bootc/rhel-10-x86_64/10339742-bootc/backend.log' as PID 2385559