[2026-03-26 09:57:06,378][ INFO][PID:1110843] Marking build as starting [2026-03-26 09:57:06,461][ INFO][PID:1110843] Checking that /var/lib/copr/public_html/results/bieszczaders/kernel-cachyos-dev/fedora-rawhide-x86_64/repodata/repomd.xml exists [2026-03-26 09:57:06,694][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:57:06,703][ INFO][PID:1110843] VM allocation process starts [2026-03-26 09:57:06,718][ INFO][PID:1110843] Trying to allocate VM: ResallocHost, ticket_id=2036644, requested_tags=['copr_builder', 'arch_x86_64'] [2026-03-26 09:57:09,744][ INFO][PID:1110843] Allocated host ResallocHost, ticket_id=2036644, hostname=54.90.209.223, name=aws_x86_64_spot_prod_04278692_20260326_095505, requested_tags=['copr_builder', 'arch_x86_64'] [2026-03-26 09:57:09,745][ INFO][PID:1110843] Allocating ssh connection to builder [2026-03-26 09:57:09,745][ INFO][PID:1110843] Checking that builder machine is OK [2026-03-26 09:57:10,077][ INFO][PID:1110843] Running remote command: copr-builder-ready fedora-rawhide-x86_64 [2026-03-26 09:57:10,298][ INFO][PID:1110843] Red Hat subscription not needed for fedora-rawhide-x86_64 Builder is ready to be used [2026-03-26 09:57:10,299][ INFO][PID:1110843] Filling build.info file with builder info [2026-03-26 09:57:10,318][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:57:10,320][ INFO][PID:1110843] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "bieszczaders", "project_name": "kernel-cachyos-dev", "project_dirname": "kernel-cachyos-dev", "submitter": "trixieua", "ended_on": null, "started_on": 1774519030.3205, "submitted_on": null, "status": 3, "chroot": "fedora-rawhide-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10263160-fedora-rawhide-x86_64", "build_id": 10263160, "package_name": "kernel-cachyos-lto", "package_version": "6.19.10-cachyos1.lto", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/bieszczaders/kernel-cachyos-dev/kernel-cachyos-lto", "git_hash": "8d97dea644af51d9a867e04de2f226a1405a51b7", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "bieszczaders/kernel-cachyos-dev--trixieua", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/bieszczaders/kernel-cachyos-dev/fedora-rawhide-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/bieszczaders/kernel-cachyos-dev", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/bieszczaders/kernel-cachyos-dev", "result_dir": "10263160-kernel-cachyos-lto", "built_packages": "", "id": 10263160, "mockchain_macros": { "copr_username": "bieszczaders", "copr_projectname": "kernel-cachyos-dev", "vendor": "Fedora Project COPR (bieszczaders/kernel-cachyos-dev)" } } ] } [2026-03-26 09:57:10,373][ INFO][PID:1110843] Sending fedora-messaging bus message in build.start [2026-03-26 09:57:11,533][ INFO][PID:1110843] Sending fedora-messaging bus message in chroot.start [2026-03-26 09:57:11,588][ INFO][PID:1110843] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10263160-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached [2026-03-26 09:57:11,933][ INFO][PID:1110843] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '54.90.209.223' (ED25519) to the list of known hosts. [2026-03-26 09:57:11,934][ INFO][PID:1110843] Downloading the builder-live.log file, attempt 1 [2026-03-26 09:57:11,935][ INFO][PID:1110843] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.90.209.223 copr-rpmbuild-log [2026-03-26 09:57:16,947][ INFO][PID:1110843] Periodic builder liveness probe: alive [2026-03-26 09:57:16,948][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:57:21,949][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:57:26,950][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:57:31,954][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:57:36,956][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:57:41,957][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:57:46,959][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:57:51,960][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:57:56,962][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:58:01,967][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:58:06,973][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:58:11,975][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:58:16,976][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:58:21,980][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:58:26,983][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:58:31,990][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:58:36,992][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:58:41,993][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:58:46,995][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:58:51,996][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:58:56,997][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:59:01,998][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:59:06,999][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:59:12,000][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:59:17,002][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:59:22,003][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:59:27,004][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:59:32,004][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:59:37,005][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:59:42,006][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:59:47,007][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:59:52,008][ INFO][PID:1110843] Checking for cancel request [2026-03-26 09:59:57,009][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:00:02,010][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:00:07,011][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:00:12,012][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:00:17,013][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:00:22,014][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:00:27,015][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:00:32,016][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:00:37,017][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:00:42,018][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:00:47,019][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:00:52,020][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:00:57,021][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:01:02,022][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:01:07,023][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:01:12,024][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:01:17,025][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:01:22,026][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:01:27,027][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:01:32,028][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:01:37,029][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:01:42,031][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:01:47,032][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:01:52,033][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:01:57,034][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:02:02,035][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:02:07,036][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:02:12,037][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:02:17,039][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:02:22,040][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:02:27,041][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:02:32,042][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:02:37,043][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:02:42,044][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:02:47,045][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:02:52,046][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:02:57,047][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:03:02,048][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:03:07,048][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:03:12,049][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:03:17,050][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:03:22,051][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:03:27,052][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:03:32,053][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:03:37,054][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:03:42,055][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:03:47,056][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:03:52,057][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:03:57,061][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:04:02,088][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:04:07,089][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:04:12,090][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:04:17,090][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:04:22,092][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:04:27,092][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:04:32,093][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:04:34,223][ ERROR][PID:1110843] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-26 10:04:37,094][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:04:42,095][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:04:44,224][WARNING][PID:1110843] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-03-26 10:04:47,096][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:04:52,097][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:04:54,225][ INFO][PID:1110843] Downloading the builder-live.log file, attempt 2 [2026-03-26 10:04:54,227][ INFO][PID:1110843] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.90.209.223 copr-rpmbuild-log [2026-03-26 10:04:57,099][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:05:02,100][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:05:07,102][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:05:12,104][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:05:17,105][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:05:22,106][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:05:27,108][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:05:32,110][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:05:37,111][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:05:42,112][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:05:47,113][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:05:52,114][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:05:54,296][ ERROR][PID:1110843] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-26 10:05:57,117][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:06:02,120][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:06:04,323][WARNING][PID:1110843] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-03-26 10:06:07,121][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:06:12,122][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:06:14,324][ INFO][PID:1110843] Downloading the builder-live.log file, attempt 3 [2026-03-26 10:06:14,327][ INFO][PID:1110843] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.90.209.223 copr-rpmbuild-log [2026-03-26 10:06:17,125][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:06:22,129][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:06:27,130][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:06:32,133][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:06:37,135][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:06:42,136][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:06:47,137][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:06:52,139][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:06:57,142][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:07:02,143][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:07:07,144][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:07:12,145][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:07:14,398][ ERROR][PID:1110843] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-26 10:07:17,183][ INFO][PID:1110843] Periodic builder liveness probe: alive [2026-03-26 10:07:17,183][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:07:22,185][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:07:24,418][WARNING][PID:1110843] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-03-26 10:07:27,186][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:07:32,187][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:07:34,419][ INFO][PID:1110843] Downloading the builder-live.log file, attempt 4 [2026-03-26 10:07:34,420][ INFO][PID:1110843] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.90.209.223 copr-rpmbuild-log [2026-03-26 10:07:37,188][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:07:42,189][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:07:47,190][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:07:52,191][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:07:57,228][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:08:02,229][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:08:07,232][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:08:12,235][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:08:17,239][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:08:22,240][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:08:27,248][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:08:32,254][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:08:34,489][ ERROR][PID:1110843] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-26 10:08:37,260][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:08:42,263][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:08:44,540][ ERROR][PID:1110843] Giving up for unstable SSH, failures: 1774519484.2246864, 1774519564.3237941, 1774519644.4181485, 1774519724.5406804 [2026-03-26 10:08:44,554][ ERROR][PID:1110843] Re-try request for task on 'ResallocHost, ticket_id=2036644, hostname=54.90.209.223, name=aws_x86_64_spot_prod_04278692_20260326_095505, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-03-26 10:08:44,554][ INFO][PID:1110843] Releasing VM back to pool [2026-03-26 10:08:44,605][ INFO][PID:1110843] Retry #1 (on other host) [2026-03-26 10:08:44,606][ INFO][PID:1110843] Checking that /var/lib/copr/public_html/results/bieszczaders/kernel-cachyos-dev/fedora-rawhide-x86_64/repodata/repomd.xml exists [2026-03-26 10:08:44,610][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:08:44,611][ INFO][PID:1110843] VM allocation process starts [2026-03-26 10:08:44,653][ INFO][PID:1110843] Trying to allocate VM: ResallocHost, ticket_id=2036885, requested_tags=['copr_builder', 'arch_x86_64'] [2026-03-26 10:08:47,723][ INFO][PID:1110843] Allocated host ResallocHost, ticket_id=2036885, hostname=52.91.61.187, name=aws_x86_64_spot_prod_04279091_20260326_100654, requested_tags=['copr_builder', 'arch_x86_64'] [2026-03-26 10:08:47,724][ INFO][PID:1110843] Allocating ssh connection to builder [2026-03-26 10:08:47,724][ INFO][PID:1110843] Checking that builder machine is OK [2026-03-26 10:08:48,126][ INFO][PID:1110843] Running remote command: copr-builder-ready fedora-rawhide-x86_64 [2026-03-26 10:08:48,403][ INFO][PID:1110843] Red Hat subscription not needed for fedora-rawhide-x86_64 Builder is ready to be used [2026-03-26 10:08:48,403][ INFO][PID:1110843] Filling build.info file with builder info [2026-03-26 10:08:48,503][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:08:48,505][ INFO][PID:1110843] Not re-notifying FE and msg buses for the new host. [2026-03-26 10:08:48,505][ INFO][PID:1110843] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10263160-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached [2026-03-26 10:08:48,945][ INFO][PID:1110843] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '52.91.61.187' (ED25519) to the list of known hosts. [2026-03-26 10:08:48,947][ INFO][PID:1110843] Downloading the builder-live.log file, attempt 1 [2026-03-26 10:08:48,952][ INFO][PID:1110843] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.91.61.187 copr-rpmbuild-log [2026-03-26 10:08:53,947][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:08:58,949][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:09:03,958][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:09:08,961][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:09:13,969][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:09:18,972][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:09:23,973][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:09:28,981][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:09:33,985][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:09:38,988][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:09:43,992][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:09:49,000][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:09:54,003][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:09:59,007][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:10:04,012][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:10:09,018][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:10:14,020][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:10:19,022][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:10:24,030][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:10:29,037][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:10:34,039][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:10:39,040][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:10:44,044][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:10:49,048][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:10:54,052][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:10:59,055][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:11:04,056][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:11:09,057][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:11:14,059][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:11:19,063][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:11:24,064][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:11:29,068][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:11:34,069][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:11:39,070][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:11:44,072][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:11:49,073][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:11:54,075][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:11:59,076][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:12:04,077][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:12:09,078][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:12:14,079][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:12:18,335][ ERROR][PID:1110843] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-26 10:12:19,080][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:12:24,081][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:12:28,340][WARNING][PID:1110843] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-03-26 10:12:29,082][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:12:34,083][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:12:38,341][ INFO][PID:1110843] Downloading the builder-live.log file, attempt 2 [2026-03-26 10:12:38,342][ INFO][PID:1110843] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.91.61.187 copr-rpmbuild-log [2026-03-26 10:12:39,084][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:12:44,085][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:12:49,086][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:12:54,087][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:12:59,088][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:13:04,089][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:13:09,090][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:13:14,091][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:13:19,092][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:13:24,093][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:13:29,093][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:13:34,106][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:13:38,433][ ERROR][PID:1110843] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-26 10:13:39,107][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:13:44,108][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:13:48,441][WARNING][PID:1110843] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-03-26 10:13:49,109][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:13:54,110][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:13:58,442][ INFO][PID:1110843] Downloading the builder-live.log file, attempt 3 [2026-03-26 10:13:58,474][ INFO][PID:1110843] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.91.61.187 copr-rpmbuild-log [2026-03-26 10:13:59,111][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:14:04,113][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:14:09,114][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:14:14,117][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:14:19,118][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:14:24,119][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:14:29,120][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:14:34,121][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:14:39,123][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:14:44,124][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:14:49,125][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:14:54,126][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:14:58,541][ ERROR][PID:1110843] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-26 10:14:59,127][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:15:04,128][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:15:08,566][WARNING][PID:1110843] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-03-26 10:15:09,129][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:15:14,129][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:15:18,567][ INFO][PID:1110843] Downloading the builder-live.log file, attempt 4 [2026-03-26 10:15:18,577][ INFO][PID:1110843] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.91.61.187 copr-rpmbuild-log [2026-03-26 10:15:19,131][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:15:24,132][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:15:29,134][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:15:34,135][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:15:39,136][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:15:44,139][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:15:49,141][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:15:54,142][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:15:59,144][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:16:04,145][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:16:09,146][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:16:14,147][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:16:18,656][ ERROR][PID:1110843] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-26 10:16:19,148][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:16:24,150][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:16:28,912][ ERROR][PID:1110843] Giving up for unstable SSH, failures: 1774519948.340784, 1774520028.4418328, 1774520108.5667543, 1774520188.9124422 [2026-03-26 10:16:28,913][ ERROR][PID:1110843] Re-try request for task on 'ResallocHost, ticket_id=2036885, hostname=52.91.61.187, name=aws_x86_64_spot_prod_04279091_20260326_100654, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-03-26 10:16:28,914][ INFO][PID:1110843] Releasing VM back to pool [2026-03-26 10:16:28,922][ INFO][PID:1110843] Retry #2 (on other host) [2026-03-26 10:16:28,922][ INFO][PID:1110843] Checking that /var/lib/copr/public_html/results/bieszczaders/kernel-cachyos-dev/fedora-rawhide-x86_64/repodata/repomd.xml exists [2026-03-26 10:16:29,170][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:16:29,171][ INFO][PID:1110843] VM allocation process starts [2026-03-26 10:16:29,181][ INFO][PID:1110843] Trying to allocate VM: ResallocHost, ticket_id=2036999, requested_tags=['copr_builder', 'arch_x86_64'] [2026-03-26 10:16:32,195][ INFO][PID:1110843] Allocated host ResallocHost, ticket_id=2036999, hostname=3.90.149.236, name=aws_x86_64_spot_prod_04279246_20260326_101316, requested_tags=['copr_builder', 'arch_x86_64'] [2026-03-26 10:16:32,195][ INFO][PID:1110843] Allocating ssh connection to builder [2026-03-26 10:16:32,196][ INFO][PID:1110843] Checking that builder machine is OK [2026-03-26 10:16:32,538][ INFO][PID:1110843] Running remote command: copr-builder-ready fedora-rawhide-x86_64 [2026-03-26 10:16:32,755][ INFO][PID:1110843] Red Hat subscription not needed for fedora-rawhide-x86_64 Builder is ready to be used [2026-03-26 10:16:32,755][ INFO][PID:1110843] Filling build.info file with builder info [2026-03-26 10:16:32,893][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:16:32,894][ INFO][PID:1110843] Not re-notifying FE and msg buses for the new host. [2026-03-26 10:16:32,895][ INFO][PID:1110843] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10263160-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached [2026-03-26 10:16:33,262][ INFO][PID:1110843] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '3.90.149.236' (ED25519) to the list of known hosts. [2026-03-26 10:16:33,263][ INFO][PID:1110843] Downloading the builder-live.log file, attempt 1 [2026-03-26 10:16:33,264][ INFO][PID:1110843] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.90.149.236 copr-rpmbuild-log [2026-03-26 10:16:38,263][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:16:43,264][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:16:48,265][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:16:53,266][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:16:58,267][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:17:03,268][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:17:08,269][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:17:13,270][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:17:18,277][ INFO][PID:1110843] Periodic builder liveness probe: alive [2026-03-26 10:17:18,278][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:17:23,278][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:17:28,279][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:17:33,280][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:17:38,282][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:17:43,283][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:17:48,284][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:17:53,287][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:17:58,288][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:18:03,289][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:18:08,290][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:18:13,293][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:18:18,294][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:18:23,296][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:18:28,297][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:18:33,298][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:18:38,300][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:18:43,301][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:18:48,302][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:18:53,304][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:18:58,305][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:19:03,306][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:19:08,308][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:19:13,309][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:19:18,310][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:19:23,311][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:19:28,312][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:19:33,313][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:19:38,315][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:19:43,316][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:19:48,321][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:19:53,326][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:19:58,327][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:20:03,328][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:20:08,329][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:20:13,331][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:20:18,333][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:20:23,334][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:20:28,335][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:20:33,336][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:20:38,338][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:20:43,379][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:20:48,380][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:20:53,381][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:20:58,382][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:21:03,383][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:21:08,385][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:21:13,385][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:21:18,387][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:21:23,388][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:21:28,390][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:21:33,392][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:21:38,396][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:21:43,398][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:21:48,401][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:21:53,437][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:21:58,442][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:22:03,443][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:22:08,560][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:22:13,562][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:22:18,568][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:22:23,570][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:22:28,572][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:22:33,576][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:22:38,578][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:22:43,584][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:22:48,587][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:22:53,590][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:22:58,693][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:23:03,695][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:23:08,697][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:23:13,698][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:23:18,700][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:23:23,702][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:23:28,705][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:23:33,710][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:23:38,713][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:23:43,728][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:23:48,730][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:23:53,731][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:23:58,733][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:24:03,736][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:24:08,738][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:24:13,740][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:24:18,742][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:24:23,744][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:24:28,747][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:24:33,748][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:24:38,751][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:24:43,754][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:24:48,755][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:24:53,756][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:24:58,757][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:25:03,760][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:25:08,761][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:25:13,762][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:25:18,763][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:25:23,764][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:25:28,765][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:25:33,765][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:25:38,766][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:25:43,767][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:25:48,768][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:25:53,769][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:25:58,770][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:26:03,771][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:26:08,772][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:26:13,774][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:26:18,775][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:26:23,776][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:26:28,777][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:26:33,778][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:26:38,779][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:26:43,780][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:26:48,781][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:26:53,782][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:26:58,783][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:27:03,784][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:27:08,785][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:27:13,786][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:27:18,793][ INFO][PID:1110843] Periodic builder liveness probe: alive [2026-03-26 10:27:18,794][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:27:23,795][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:27:28,796][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:27:33,797][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:27:38,799][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:27:43,800][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:27:48,801][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:27:53,804][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:27:58,806][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:28:03,807][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:28:08,808][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:28:13,809][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:28:18,810][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:28:23,812][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:28:28,813][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:28:33,814][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:28:38,815][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:28:43,816][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:28:48,817][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:28:53,818][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:28:58,820][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:29:03,821][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:29:08,822][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:29:13,823][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:29:18,825][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:29:23,826][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:29:28,828][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:29:33,829][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:29:38,830][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:29:43,831][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:29:48,836][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:29:53,837][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:29:58,838][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:30:03,839][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:30:08,840][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:30:13,841][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:30:18,845][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:30:23,847][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:30:28,848][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:30:33,849][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:30:38,850][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:30:43,851][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:30:48,852][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:30:53,853][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:30:58,854][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:31:03,855][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:31:08,856][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:31:13,857][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:31:18,859][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:31:23,860][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:31:28,861][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:31:33,862][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:31:38,864][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:31:43,868][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:31:48,869][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:31:53,870][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:31:58,871][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:32:03,872][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:32:08,874][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:32:13,875][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:32:18,876][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:32:23,878][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:32:28,879][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:32:33,880][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:32:38,881][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:32:43,882][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:32:48,883][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:32:53,884][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:32:58,892][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:33:03,893][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:33:08,895][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:33:13,897][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:33:18,898][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:33:23,899][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:33:28,900][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:33:33,901][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:33:38,903][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:33:43,904][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:33:48,906][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:33:53,907][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:33:58,909][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:34:03,910][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:34:08,911][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:34:13,912][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:34:18,913][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:34:23,914][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:34:28,915][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:34:33,916][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:34:38,917][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:34:43,918][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:34:48,919][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:34:53,921][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:34:58,922][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:35:03,923][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:35:08,924][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:35:13,926][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:35:18,927][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:35:23,929][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:35:28,930][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:35:33,931][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:35:38,932][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:35:43,934][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:35:48,936][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:35:53,938][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:35:58,944][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:36:03,945][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:36:08,967][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:36:13,969][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:36:18,972][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:36:23,973][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:36:28,974][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:36:33,975][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:36:38,976][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:36:43,977][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:36:48,978][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:36:53,980][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:36:58,981][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:37:03,982][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:37:08,983][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:37:13,984][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:37:19,042][ INFO][PID:1110843] Periodic builder liveness probe: alive [2026-03-26 10:37:19,043][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:37:24,044][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:37:29,045][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:37:34,046][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:37:39,047][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:37:44,047][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:37:49,048][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:37:54,049][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:37:59,050][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:38:04,125][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:38:09,126][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:38:14,169][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:38:19,170][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:38:24,171][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:38:29,172][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:38:34,173][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:38:39,174][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:38:44,175][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:38:49,481][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:38:54,527][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:38:59,528][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:39:04,529][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:39:09,531][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:39:14,532][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:39:19,533][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:39:24,534][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:39:29,538][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:39:34,538][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:39:39,539][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:39:44,540][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:39:49,541][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:39:54,542][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:39:59,543][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:40:04,544][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:40:09,545][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:40:14,547][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:40:19,549][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:40:24,550][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:40:29,552][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:40:34,553][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:40:39,554][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:40:44,555][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:40:49,556][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:40:54,557][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:40:59,558][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:41:04,569][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:41:09,571][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:41:14,572][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:41:19,573][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:41:24,574][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:41:29,575][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:41:34,576][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:41:39,577][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:41:44,578][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:41:49,579][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:41:54,580][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:41:59,604][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:42:04,605][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:42:09,606][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:42:14,607][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:42:19,608][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:42:24,609][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:42:29,610][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:42:34,611][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:42:39,611][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:42:44,612][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:42:49,613][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:42:54,614][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:42:59,615][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:43:04,616][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:43:09,617][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:43:14,618][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:43:19,619][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:43:24,620][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:43:29,621][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:43:34,622][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:43:39,623][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:43:44,624][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:43:49,625][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:43:54,626][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:43:59,627][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:44:04,631][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:44:09,632][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:44:14,633][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:44:19,634][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:44:24,634][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:44:29,635][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:44:34,636][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:44:39,637][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:44:44,638][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:44:49,639][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:44:54,640][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:44:59,641][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:45:04,643][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:45:09,644][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:45:14,645][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:45:19,646][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:45:24,647][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:45:29,648][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:45:34,649][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:45:39,650][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:45:44,651][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:45:49,652][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:45:54,653][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:45:59,654][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:46:04,655][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:46:09,656][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:46:14,657][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:46:19,658][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:46:24,659][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:46:29,660][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:46:34,661][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:46:39,662][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:46:44,663][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:46:49,664][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:46:54,665][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:46:59,665][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:47:04,666][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:47:09,667][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:47:14,668][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:47:19,681][ INFO][PID:1110843] Periodic builder liveness probe: alive [2026-03-26 10:47:19,682][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:47:24,683][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:47:29,684][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:47:34,685][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:47:39,686][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:47:44,687][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:47:49,688][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:47:54,689][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:47:59,690][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:48:04,692][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:48:09,693][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:48:14,695][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:48:19,699][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:48:24,700][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:48:29,701][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:48:34,702][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:48:39,704][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:48:44,707][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:48:49,714][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:48:54,715][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:48:59,717][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:49:04,718][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:49:09,721][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:49:14,724][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:49:19,728][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:49:24,730][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:49:29,731][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:49:34,732][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:49:39,733][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:49:44,791][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:49:49,792][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:49:54,793][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:49:59,795][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:50:04,796][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:50:09,797][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:50:14,801][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:50:19,803][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:50:24,809][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:50:29,810][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:50:34,811][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:50:39,813][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:50:44,814][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:50:49,815][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:50:54,816][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:50:59,818][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:51:04,819][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:51:09,820][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:51:14,821][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:51:19,822][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:51:24,823][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:51:29,825][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:51:34,827][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:51:39,828][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:51:44,829][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:51:49,831][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:51:54,833][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:51:59,842][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:52:04,844][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:52:09,845][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:52:14,846][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:52:19,847][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:52:24,848][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:52:29,951][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:52:34,954][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:52:39,955][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:52:44,958][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:52:49,963][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:52:54,965][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:52:59,968][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:53:05,021][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:53:10,024][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:53:15,026][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:53:20,027][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:53:25,029][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:53:30,031][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:53:35,032][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:53:40,033][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:53:45,034][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:53:50,041][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:53:55,042][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:54:00,043][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:54:05,044][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:54:10,045][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:54:15,046][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:54:20,047][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:54:25,048][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:54:30,049][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:54:35,050][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:54:40,051][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:54:45,052][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:54:50,053][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:54:55,054][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:55:00,114][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:55:05,118][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:55:10,119][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:55:15,121][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:55:20,122][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:55:25,123][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:55:30,124][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:55:35,125][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:55:40,143][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:55:45,145][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:55:50,146][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:55:55,148][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:56:00,149][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:56:05,153][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:56:10,160][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:56:15,164][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:56:20,385][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:56:25,390][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:56:30,393][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:56:35,394][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:56:40,397][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:56:45,399][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:56:50,400][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:56:55,401][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:57:00,406][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:57:05,407][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:57:10,408][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:57:15,409][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:57:20,417][ INFO][PID:1110843] Periodic builder liveness probe: alive [2026-03-26 10:57:20,417][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:57:25,418][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:57:30,420][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:57:35,676][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:57:40,685][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:57:45,688][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:57:50,694][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:57:55,695][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:58:00,696][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:58:05,696][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:58:10,698][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:58:15,699][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:58:20,701][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:58:25,702][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:58:30,710][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:58:35,717][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:58:40,728][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:58:45,750][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:58:50,767][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:58:55,768][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:59:00,769][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:59:05,770][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:59:10,771][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:59:15,772][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:59:20,773][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:59:25,774][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:59:30,775][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:59:35,776][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:59:40,777][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:59:45,778][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:59:50,779][ INFO][PID:1110843] Checking for cancel request [2026-03-26 10:59:55,780][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:00:00,781][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:00:05,782][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:00:10,783][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:00:15,784][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:00:20,785][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:00:25,786][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:00:30,788][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:00:35,788][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:00:40,803][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:00:45,813][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:00:50,814][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:00:55,815][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:01:00,816][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:01:05,817][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:01:10,818][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:01:15,819][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:01:20,934][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:01:25,939][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:01:30,941][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:01:35,947][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:01:40,948][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:01:45,949][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:01:50,950][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:01:55,952][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:02:00,953][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:02:05,955][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:02:10,956][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:02:15,958][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:02:20,959][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:02:25,960][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:02:30,961][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:02:35,962][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:02:40,963][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:02:45,964][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:02:50,965][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:02:55,966][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:03:00,978][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:03:05,987][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:03:11,020][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:03:16,021][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:03:21,022][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:03:26,023][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:03:31,024][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:03:36,025][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:03:41,026][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:03:46,027][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:03:51,028][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:03:56,029][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:04:01,030][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:04:06,031][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:04:11,032][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:04:16,033][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:04:21,034][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:04:26,035][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:04:31,036][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:04:36,037][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:04:38,282][ ERROR][PID:1110843] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-26 11:04:41,038][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:04:46,039][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:04:48,283][WARNING][PID:1110843] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-03-26 11:04:51,040][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:04:56,041][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:04:58,284][ INFO][PID:1110843] Downloading the builder-live.log file, attempt 2 [2026-03-26 11:04:58,321][ INFO][PID:1110843] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.90.149.236 copr-rpmbuild-log [2026-03-26 11:04:58,329][ ERROR][PID:1110843] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-26 11:05:01,042][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:05:06,043][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:05:08,344][WARNING][PID:1110843] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-03-26 11:05:11,044][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:05:16,046][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:05:18,345][ INFO][PID:1110843] Downloading the builder-live.log file, attempt 3 [2026-03-26 11:05:18,350][ INFO][PID:1110843] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.90.149.236 copr-rpmbuild-log [2026-03-26 11:05:18,361][ ERROR][PID:1110843] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-26 11:05:21,049][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:05:26,050][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:05:28,368][WARNING][PID:1110843] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-03-26 11:05:31,051][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:05:36,052][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:05:38,368][ INFO][PID:1110843] Downloading the builder-live.log file, attempt 4 [2026-03-26 11:05:38,370][ INFO][PID:1110843] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.90.149.236 copr-rpmbuild-log [2026-03-26 11:05:41,053][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:05:46,054][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:05:51,055][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:05:56,056][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:06:01,057][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:06:06,058][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:06:11,059][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:06:16,060][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:06:21,061][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:06:26,062][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:06:31,063][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:06:36,064][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:06:38,426][ ERROR][PID:1110843] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-26 11:06:41,065][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:06:46,066][ INFO][PID:1110843] Checking for cancel request [2026-03-26 11:06:48,473][ ERROR][PID:1110843] Giving up for unstable SSH, failures: 1774523088.2835858, 1774523108.3445675, 1774523128.36807, 1774523208.4736588 [2026-03-26 11:06:48,475][ ERROR][PID:1110843] Re-try request for task on 'ResallocHost, ticket_id=2036999, hostname=3.90.149.236, name=aws_x86_64_spot_prod_04279246_20260326_101316, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-03-26 11:06:48,475][ INFO][PID:1110843] Releasing VM back to pool [2026-03-26 11:06:48,486][ ERROR][PID:1110843] Backend process error: Three host tried without success: {'54.90.209.223', '52.91.61.187', '3.90.149.236'} [2026-03-26 11:06:48,486][WARNING][PID:1110843] Switching not-finished job state to 'failed' [2026-03-26 11:06:48,487][ INFO][PID:1110843] Worker failed build, took 3015.5921654701233 [2026-03-26 11:06:48,487][ INFO][PID:1110843] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "bieszczaders", "project_name": "kernel-cachyos-dev", "project_dirname": "kernel-cachyos-dev", "submitter": "trixieua", "ended_on": 1774523208.4866598, "started_on": 1774520192.8944943, "submitted_on": null, "status": 0, "chroot": "fedora-rawhide-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10263160-fedora-rawhide-x86_64", "build_id": 10263160, "package_name": "kernel-cachyos-lto", "package_version": "6.19.10-cachyos1.lto", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/bieszczaders/kernel-cachyos-dev/kernel-cachyos-lto", "git_hash": "8d97dea644af51d9a867e04de2f226a1405a51b7", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "bieszczaders/kernel-cachyos-dev--trixieua", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/bieszczaders/kernel-cachyos-dev/fedora-rawhide-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/bieszczaders/kernel-cachyos-dev", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/bieszczaders/kernel-cachyos-dev", "result_dir": "10263160-kernel-cachyos-lto", "built_packages": "", "id": 10263160, "mockchain_macros": { "copr_username": "bieszczaders", "copr_projectname": "kernel-cachyos-dev", "vendor": "Fedora Project COPR (bieszczaders/kernel-cachyos-dev)" } } ] } [2026-03-26 11:06:48,551][ INFO][PID:1110843] Sending fedora-messaging bus message in build.end [2026-03-26 11:06:48,602][ INFO][PID:1110843] Compressing /var/lib/copr/public_html/results/bieszczaders/kernel-cachyos-dev/fedora-rawhide-x86_64/10263160-kernel-cachyos-lto/builder-live.log by gzip [2026-03-26 11:06:48,603][ INFO][PID:1110843] Running command 'gzip /var/lib/copr/public_html/results/bieszczaders/kernel-cachyos-dev/fedora-rawhide-x86_64/10263160-kernel-cachyos-lto/builder-live.log' as PID 1627233 [2026-03-26 11:06:48,873][ INFO][PID:1110843] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/bieszczaders/kernel-cachyos-dev/fedora-rawhide-x86_64/10263160-kernel-cachyos-lto/builder-live.log) [2026-03-26 11:06:48,873][ INFO][PID:1110843] Compressing /var/lib/copr/public_html/results/bieszczaders/kernel-cachyos-dev/fedora-rawhide-x86_64/10263160-kernel-cachyos-lto/backend.log by gzip [2026-03-26 11:06:48,874][ INFO][PID:1110843] Running command 'gzip /var/lib/copr/public_html/results/bieszczaders/kernel-cachyos-dev/fedora-rawhide-x86_64/10263160-kernel-cachyos-lto/backend.log' as PID 1627275