[2026-04-16 15:51:16,546][ INFO][PID:2336746] Marking build as starting [2026-04-16 15:51:16,763][ INFO][PID:2336746] Checking that /var/lib/copr/public_html/results/rhcontainerbot/bootc/fedora-42-s390x/repodata/repomd.xml exists [2026-04-16 15:51:16,785][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:51:16,787][ INFO][PID:2336746] VM allocation process starts [2026-04-16 15:51:16,853][ INFO][PID:2336746] Trying to allocate VM: ResallocHost, ticket_id=2374046, requested_tags=['arch_s390x', 'copr_builder'] [2026-04-16 15:51:21,859][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:51:22,934][ INFO][PID:2336746] Allocated host ResallocHost, ticket_id=2374046, hostname=54.198.0.61, name=aws_x86_64_spot_prod_06076897_20260416_154808, requested_tags=['arch_s390x', 'copr_builder'] [2026-04-16 15:51:22,934][ INFO][PID:2336746] Allocating ssh connection to builder [2026-04-16 15:51:22,935][ INFO][PID:2336746] Checking that builder machine is OK [2026-04-16 15:51:23,251][ INFO][PID:2336746] Installed copr-rpmbuild version: 1.8 [2026-04-16 15:51:23,251][ INFO][PID:2336746] Running remote command: copr-builder-ready fedora-42-s390x [2026-04-16 15:51:23,465][ INFO][PID:2336746] Red Hat subscription not needed for fedora-42-s390x Builder is ready to be used [2026-04-16 15:51:23,467][ INFO][PID:2336746] Filling build.info file with builder info [2026-04-16 15:51:23,469][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:51:23,471][ INFO][PID:2336746] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "rhcontainerbot", "project_name": "bootc", "project_dirname": "bootc", "submitter": "packit", "ended_on": null, "started_on": 1776354683.4714298, "submitted_on": null, "status": 3, "chroot": "fedora-42-s390x", "arch": "s390x", "buildroot_pkgs": [], "task_id": "10339742-fedora-42-s390x", "build_id": 10339742, "package_name": "bootc", "package_version": "202604161547.g47c362089b-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rhcontainerbot/bootc/bootc", "git_hash": "384161d1c0c2b1cdce2efa6d59b015fd3bca9451", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "rhcontainerbot/bootc--packit", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/fedora-42-s390x/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_s390x" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/rhcontainerbot/bootc", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc", "result_dir": "10339742-bootc", "built_packages": "", "id": 10339742, "mockchain_macros": { "copr_username": "rhcontainerbot", "copr_projectname": "bootc", "vendor": "Fedora Project COPR (rhcontainerbot/bootc)" } } ] } [2026-04-16 15:51:23,584][ INFO][PID:2336746] Sending fedora-messaging bus message in build.start [2026-04-16 15:51:24,291][ INFO][PID:2336746] Sending fedora-messaging bus message in chroot.start [2026-04-16 15:51:24,331][ INFO][PID:2336746] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10339742-fedora-42-s390x --chroot fedora-42-s390x --detached [2026-04-16 15:51:24,780][ INFO][PID:2336746] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '54.198.0.61' (ED25519) to the list of known hosts. [2026-04-16 15:51:24,782][ INFO][PID:2336746] Downloading the builder-live.log file, attempt 1 [2026-04-16 15:51:24,790][ INFO][PID:2336746] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.198.0.61 copr-rpmbuild-log [2026-04-16 15:51:29,793][ INFO][PID:2336746] Periodic builder liveness probe: alive [2026-04-16 15:51:29,794][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:51:34,805][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:51:39,812][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:51:44,813][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:51:49,815][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:51:54,822][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:51:59,824][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:52:04,826][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:52:09,829][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:52:14,836][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:52:19,838][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:52:24,840][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:52:29,842][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:52:34,844][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:52:39,847][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:52:44,849][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:52:49,851][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:52:54,852][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:52:59,855][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:53:04,857][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:53:09,859][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:53:14,860][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:53:19,861][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:53:24,862][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:53:29,863][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:53:34,865][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:53:39,871][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:53:44,872][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:53:49,873][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:53:54,874][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:53:59,875][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:54:04,876][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:54:09,877][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:54:14,878][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:54:19,879][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:54:24,880][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:54:29,881][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:54:34,882][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:54:39,883][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:54:44,884][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:54:49,885][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:54:54,886][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:54:59,887][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:55:04,888][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:55:09,889][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:55:14,890][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:55:19,891][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:55:24,892][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:55:29,893][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:55:34,894][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:55:39,895][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:55:44,896][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:55:49,897][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:55:54,898][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:55:59,898][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:56:04,899][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:56:09,900][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:56:14,901][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:56:19,902][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:56:24,903][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:56:29,904][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:56:34,905][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:56:39,906][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:56:44,907][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:56:49,908][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:56:54,909][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:56:59,910][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:57:04,911][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:57:09,916][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:57:14,918][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:57:19,387][ ERROR][PID:2336746] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-16 15:57:19,919][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:57:24,921][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:57:29,388][WARNING][PID:2336746] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-16 15:57:29,922][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:57:34,923][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:57:39,389][ INFO][PID:2336746] Downloading the builder-live.log file, attempt 2 [2026-04-16 15:57:39,391][ INFO][PID:2336746] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.198.0.61 copr-rpmbuild-log [2026-04-16 15:57:39,398][ ERROR][PID:2336746] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-16 15:57:39,924][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:57:44,925][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:57:49,417][WARNING][PID:2336746] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-16 15:57:49,926][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:57:54,927][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:57:59,418][ INFO][PID:2336746] Downloading the builder-live.log file, attempt 3 [2026-04-16 15:57:59,419][ INFO][PID:2336746] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.198.0.61 copr-rpmbuild-log [2026-04-16 15:57:59,434][ ERROR][PID:2336746] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-16 15:57:59,928][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:58:04,929][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:58:09,450][WARNING][PID:2336746] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-16 15:58:09,930][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:58:14,931][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:58:19,451][ INFO][PID:2336746] Downloading the builder-live.log file, attempt 4 [2026-04-16 15:58:19,452][ INFO][PID:2336746] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.198.0.61 copr-rpmbuild-log [2026-04-16 15:58:19,932][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:58:24,933][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:58:29,934][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:58:34,936][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:58:39,947][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:58:44,979][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:58:49,980][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:58:54,981][ INFO][PID:2336746] Checking for cancel request [2026-04-16 15:58:54,981][ INFO][PID:2336746] Calling cancel callback, and waiting for thread.join() [2026-04-16 15:58:54,982][ INFO][PID:2336746] Canceling the build on the remote machine [2026-04-16 15:59:19,520][ ERROR][PID:2336746] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-16 15:59:29,521][ ERROR][PID:2336746] Giving up for unstable SSH, failures: 1776355049.3886662, 1776355069.4177926, 1776355089.4508133, 1776355169.521398 [2026-04-16 15:59:55,051][ ERROR][PID:2336746] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 54.198.0.61 port 22: Connection timed out [2026-04-16 16:01:05,120][ ERROR][PID:2336746] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 54.198.0.61 port 22: Connection timed out [2026-04-16 16:02:15,284][ ERROR][PID:2336746] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 54.198.0.61 port 22: Connection timed out [2026-04-16 16:03:25,379][ ERROR][PID:2336746] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 54.198.0.61 port 22: Connection timed out [2026-04-16 16:03:35,392][ ERROR][PID:2336746] Can't ssh to cancel build. [2026-04-16 16:03:35,393][ ERROR][PID:2336746] Build was canceled [2026-04-16 16:03:35,394][ INFO][PID:2336746] VM Release request [2026-04-16 16:03:35,406][WARNING][PID:2336746] Switching not-finished job state to 'failed' [2026-04-16 16:03:35,406][ INFO][PID:2336746] Worker failed build, took 731.9345111846924 [2026-04-16 16:03:35,407][ INFO][PID:2336746] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "rhcontainerbot", "project_name": "bootc", "project_dirname": "bootc", "submitter": "packit", "ended_on": 1776355415.405941, "started_on": 1776354683.4714298, "submitted_on": null, "status": 0, "chroot": "fedora-42-s390x", "arch": "s390x", "buildroot_pkgs": [], "task_id": "10339742-fedora-42-s390x", "build_id": 10339742, "package_name": "bootc", "package_version": "202604161547.g47c362089b-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rhcontainerbot/bootc/bootc", "git_hash": "384161d1c0c2b1cdce2efa6d59b015fd3bca9451", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "rhcontainerbot/bootc--packit", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/fedora-42-s390x/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_s390x" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/rhcontainerbot/bootc", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc", "result_dir": "10339742-bootc", "built_packages": "", "id": 10339742, "mockchain_macros": { "copr_username": "rhcontainerbot", "copr_projectname": "bootc", "vendor": "Fedora Project COPR (rhcontainerbot/bootc)" } } ] } [2026-04-16 16:03:35,479][ INFO][PID:2336746] Sending fedora-messaging bus message in build.end [2026-04-16 16:03:35,522][ INFO][PID:2336746] Compressing /var/lib/copr/public_html/results/rhcontainerbot/bootc/fedora-42-s390x/10339742-bootc/builder-live.log by gzip [2026-04-16 16:03:35,523][ INFO][PID:2336746] Running command 'gzip /var/lib/copr/public_html/results/rhcontainerbot/bootc/fedora-42-s390x/10339742-bootc/builder-live.log' as PID 2432558 [2026-04-16 16:03:35,563][ INFO][PID:2336746] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/rhcontainerbot/bootc/fedora-42-s390x/10339742-bootc/builder-live.log) [2026-04-16 16:03:35,563][ INFO][PID:2336746] Compressing /var/lib/copr/public_html/results/rhcontainerbot/bootc/fedora-42-s390x/10339742-bootc/backend.log by gzip [2026-04-16 16:03:35,565][ INFO][PID:2336746] Running command 'gzip /var/lib/copr/public_html/results/rhcontainerbot/bootc/fedora-42-s390x/10339742-bootc/backend.log' as PID 2432564