[2025-10-17 14:28:21,021][ INFO][PID:2826756] Marking build as starting [2025-10-17 14:28:21,097][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:28:21,106][ INFO][PID:2826756] VM allocation process starts [2025-10-17 14:28:21,155][ INFO][PID:2826756] Trying to allocate VM: ResallocHost, ticket_id=5712798, requested_tags=['copr_builder', 'arch_x86_64'] [2025-10-17 14:28:24,182][ INFO][PID:2826756] Allocated host ResallocHost, ticket_id=5712798, hostname=3.236.239.116, name=aws_x86_64_spot_prod_07442980_20251017_142453, requested_tags=['copr_builder', 'arch_x86_64'] [2025-10-17 14:28:24,183][ INFO][PID:2826756] Allocating ssh connection to builder [2025-10-17 14:28:24,183][ INFO][PID:2826756] Checking that builder machine is OK [2025-10-17 14:28:24,496][ INFO][PID:2826756] Running remote command: copr-builder-ready rhel-9-x86_64 [2025-10-17 14:28:24,688][ INFO][PID:2826756] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2025-10-17 14:28:24,688][ INFO][PID:2826756] Filling build.info file with builder info [2025-10-17 14:28:24,689][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:28:24,690][ INFO][PID:2826756] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "rhcontainerbot", "project_name": "bootc", "project_dirname": "bootc", "submitter": "packit", "ended_on": null, "started_on": 1760711304.690023, "submitted_on": null, "status": 3, "chroot": "rhel-9-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9699865-rhel-9-x86_64", "build_id": 9699865, "package_name": "bootc", "package_version": "202510171419.g02ecf37181-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rhcontainerbot/bootc/bootc", "git_hash": "d2bf61c853217ca747f57f306f0028a919e34732", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "rhcontainerbot/bootc--packit", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/rhel-9-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/rhcontainerbot/bootc", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc", "result_dir": "09699865-bootc", "built_packages": "", "id": 9699865, "mockchain_macros": { "copr_username": "rhcontainerbot", "copr_projectname": "bootc", "vendor": "Fedora Project COPR (rhcontainerbot/bootc)" } } ] } [2025-10-17 14:28:24,751][ INFO][PID:2826756] Sending fedora-messaging bus message in build.start [2025-10-17 14:28:25,465][ INFO][PID:2826756] Sending fedora-messaging bus message in chroot.start [2025-10-17 14:28:25,490][ INFO][PID:2826756] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9699865-rhel-9-x86_64 --chroot rhel-9-x86_64 --detached [2025-10-17 14:28:25,808][ INFO][PID:2826756] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '3.236.239.116' (ED25519) to the list of known hosts. [2025-10-17 14:28:25,809][ INFO][PID:2826756] Downloading the builder-live.log file, attempt 1 [2025-10-17 14:28:25,810][ INFO][PID:2826756] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.239.116 copr-rpmbuild-log [2025-10-17 14:28:30,817][ INFO][PID:2826756] Periodic builder liveness probe: alive [2025-10-17 14:28:30,818][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:28:35,819][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:28:40,820][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:28:45,821][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:28:50,822][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:28:55,823][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:29:00,824][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:29:05,304][ ERROR][PID:2826756] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-10-17 14:29:05,825][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:29:10,826][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:29:15,305][WARNING][PID:2826756] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-10-17 14:29:15,827][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:29:20,828][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:29:25,306][ INFO][PID:2826756] Downloading the builder-live.log file, attempt 2 [2025-10-17 14:29:25,307][ INFO][PID:2826756] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.239.116 copr-rpmbuild-log [2025-10-17 14:29:25,830][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:29:30,831][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:29:35,832][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:29:40,833][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:29:45,834][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:29:50,835][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:29:55,836][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:30:00,838][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:30:05,840][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:30:10,845][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:30:15,847][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:30:20,848][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:30:25,376][ ERROR][PID:2826756] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-10-17 14:30:25,848][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:30:30,850][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:30:35,379][WARNING][PID:2826756] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-10-17 14:30:35,851][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:30:40,852][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:30:45,380][ INFO][PID:2826756] Downloading the builder-live.log file, attempt 3 [2025-10-17 14:30:45,381][ INFO][PID:2826756] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.239.116 copr-rpmbuild-log [2025-10-17 14:30:45,853][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:30:50,854][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:30:55,855][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:31:00,856][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:31:05,858][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:31:10,859][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:31:15,860][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:31:20,861][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:31:25,862][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:31:30,863][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:31:35,864][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:31:40,865][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:31:45,449][ ERROR][PID:2826756] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-10-17 14:31:45,866][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:31:50,867][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:31:55,452][WARNING][PID:2826756] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-10-17 14:31:55,870][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:32:00,871][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:32:05,453][ INFO][PID:2826756] Downloading the builder-live.log file, attempt 4 [2025-10-17 14:32:05,454][ INFO][PID:2826756] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.239.116 copr-rpmbuild-log [2025-10-17 14:32:05,872][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:32:10,873][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:32:15,874][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:32:20,875][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:32:25,876][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:32:30,877][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:32:35,878][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:32:40,879][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:32:45,881][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:32:50,882][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:32:55,883][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:33:00,884][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:33:05,515][ ERROR][PID:2826756] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-10-17 14:33:05,885][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:33:10,886][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:33:15,520][ ERROR][PID:2826756] Giving up for unstable SSH, failures: 1760711355.3052983, 1760711435.379511, 1760711515.4526713, 1760711595.5201826 [2025-10-17 14:33:15,521][ ERROR][PID:2826756] Re-try request for task on 'ResallocHost, ticket_id=5712798, hostname=3.236.239.116, name=aws_x86_64_spot_prod_07442980_20251017_142453, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-10-17 14:33:15,521][ INFO][PID:2826756] Releasing VM back to pool [2025-10-17 14:33:15,530][ INFO][PID:2826756] Retry #1 (on other host) [2025-10-17 14:33:15,531][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:33:15,531][ INFO][PID:2826756] VM allocation process starts [2025-10-17 14:33:15,586][ INFO][PID:2826756] Trying to allocate VM: ResallocHost, ticket_id=5712869, requested_tags=['copr_builder', 'arch_x86_64'] [2025-10-17 14:33:18,611][ INFO][PID:2826756] Allocated host ResallocHost, ticket_id=5712869, hostname=13.221.132.69, name=aws_x86_64_spot_prod_07443013_20251017_142628, requested_tags=['copr_builder', 'arch_x86_64'] [2025-10-17 14:33:18,612][ INFO][PID:2826756] Allocating ssh connection to builder [2025-10-17 14:33:18,612][ INFO][PID:2826756] Checking that builder machine is OK [2025-10-17 14:33:18,985][ INFO][PID:2826756] Running remote command: copr-builder-ready rhel-9-x86_64 [2025-10-17 14:33:19,198][ INFO][PID:2826756] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2025-10-17 14:33:19,199][ INFO][PID:2826756] Filling build.info file with builder info [2025-10-17 14:33:19,203][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:33:19,204][ INFO][PID:2826756] Not re-notifying FE and msg buses for the new host. [2025-10-17 14:33:19,204][ INFO][PID:2826756] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9699865-rhel-9-x86_64 --chroot rhel-9-x86_64 --detached [2025-10-17 14:33:19,584][ INFO][PID:2826756] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '13.221.132.69' (ED25519) to the list of known hosts. [2025-10-17 14:33:19,585][ INFO][PID:2826756] Downloading the builder-live.log file, attempt 1 [2025-10-17 14:33:19,586][ INFO][PID:2826756] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.221.132.69 copr-rpmbuild-log [2025-10-17 14:33:24,585][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:33:29,586][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:33:34,587][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:33:39,588][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:33:44,589][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:33:49,590][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:33:54,591][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:33:59,592][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:34:04,593][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:34:09,594][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:34:14,595][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:34:19,596][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:34:24,597][ INFO][PID:2826756] Checking for cancel request [2025-10-17 14:34:25,435][ INFO][PID:2826756] Downloading results from builder [2025-10-17 14:34:25,436][ INFO][PID:2826756] rsyncing of mockbuilder@13.221.132.69:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/rhcontainerbot/bootc/rhel-9-x86_64/09699865-bootc started [2025-10-17 14:34:25,436][ INFO][PID:2826756] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@13.221.132.69:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/rhcontainerbot/bootc/rhel-9-x86_64/09699865-bootc/ &> /var/lib/copr/public_html/results/rhcontainerbot/bootc/rhel-9-x86_64/09699865-bootc/build-09699865.rsync.log [2025-10-17 14:34:25,759][ INFO][PID:2826756] rsyncing finished. [2025-10-17 14:34:25,759][ INFO][PID:2826756] Releasing VM back to pool [2025-10-17 14:34:25,768][ INFO][PID:2826756] Searching for 'success' file in resultdir [2025-10-17 14:34:25,769][ ERROR][PID:2826756] Build failed: Backend process error: No success file => build failure [2025-10-17 14:34:25,769][ INFO][PID:2826756] Finished build: id=9699865 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/rhcontainerbot/bootc chroot=rhel-9-x86_64 [2025-10-17 14:34:25,771][ ERROR][PID:2826756] Backend process error: No success file => build failure [2025-10-17 14:34:25,772][ INFO][PID:2826756] Worker failed build, took 66.5678927898407 [2025-10-17 14:34:25,772][ INFO][PID:2826756] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "rhcontainerbot", "project_name": "bootc", "project_dirname": "bootc", "submitter": "packit", "ended_on": 1760711665.77197, "started_on": 1760711599.2040772, "submitted_on": null, "status": 0, "chroot": "rhel-9-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9699865-rhel-9-x86_64", "build_id": 9699865, "package_name": "bootc", "package_version": "202510171419.g02ecf37181-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rhcontainerbot/bootc/bootc", "git_hash": "d2bf61c853217ca747f57f306f0028a919e34732", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "rhcontainerbot/bootc--packit", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/rhel-9-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/rhcontainerbot/bootc", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc", "result_dir": "09699865-bootc", "built_packages": "", "id": 9699865, "mockchain_macros": { "copr_username": "rhcontainerbot", "copr_projectname": "bootc", "vendor": "Fedora Project COPR (rhcontainerbot/bootc)" } } ] } [2025-10-17 14:34:25,832][ INFO][PID:2826756] Sending fedora-messaging bus message in build.end [2025-10-17 14:34:25,858][ INFO][PID:2826756] Compressing /var/lib/copr/public_html/results/rhcontainerbot/bootc/rhel-9-x86_64/09699865-bootc/builder-live.log by gzip [2025-10-17 14:34:25,859][ INFO][PID:2826756] Running command 'gzip /var/lib/copr/public_html/results/rhcontainerbot/bootc/rhel-9-x86_64/09699865-bootc/builder-live.log' as PID 2869402 [2025-10-17 14:34:25,865][ INFO][PID:2826756] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/rhcontainerbot/bootc/rhel-9-x86_64/09699865-bootc/builder-live.log) [2025-10-17 14:34:25,866][ INFO][PID:2826756] Compressing /var/lib/copr/public_html/results/rhcontainerbot/bootc/rhel-9-x86_64/09699865-bootc/backend.log by gzip [2025-10-17 14:34:25,867][ INFO][PID:2826756] Running command 'gzip /var/lib/copr/public_html/results/rhcontainerbot/bootc/rhel-9-x86_64/09699865-bootc/backend.log' as PID 2869406