[2026-04-17 18:28:06,654][ INFO][PID:77238] Marking build as starting [2026-04-17 18:28:06,769][ INFO][PID:77238] Checking that /var/lib/copr/public_html/results/@osbuild/osbuild-composer/fedora-42-x86_64/repodata/repomd.xml exists [2026-04-17 18:28:06,779][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:28:06,781][ INFO][PID:77238] VM allocation process starts [2026-04-17 18:28:06,825][ INFO][PID:77238] Trying to allocate VM: ResallocHost, ticket_id=2393541, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-17 18:28:11,829][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:28:16,830][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:28:18,878][ INFO][PID:77238] Allocated host ResallocHost, ticket_id=2393541, hostname=2620:52:6:1161:dead:beef:cafe:c17c, name=vmhost_x86_04_prod_06200637_20260417_182602, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-17 18:28:18,879][ INFO][PID:77238] Allocating ssh connection to builder [2026-04-17 18:28:18,880][ INFO][PID:77238] Checking that builder machine is OK [2026-04-17 18:28:19,517][ INFO][PID:77238] Installed copr-rpmbuild version: 1.8 [2026-04-17 18:28:19,518][ INFO][PID:77238] Running remote command: copr-builder-ready fedora-42-x86_64 [2026-04-17 18:28:19,895][ INFO][PID:77238] Red Hat subscription not needed for fedora-42-x86_64 Builder is ready to be used [2026-04-17 18:28:19,895][ INFO][PID:77238] Filling build.info file with builder info [2026-04-17 18:28:19,896][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:28:19,896][ INFO][PID:77238] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@osbuild", "project_name": "osbuild-composer", "project_dirname": "osbuild-composer", "submitter": "packit", "ended_on": null, "started_on": 1776450499.8966591, "submitted_on": null, "status": 3, "chroot": "fedora-42-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10343214-fedora-42-x86_64", "build_id": 10343214, "package_name": "osbuild-composer", "package_version": "169-1.20260417182706903968.main.2.g755a65ce1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@osbuild/osbuild-composer/osbuild-composer", "git_hash": "a1f1f6214d7ab74653037ea68a95d574b06aa708", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@osbuild/osbuild-composer--packit", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@osbuild/osbuild-composer/fedora-42-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@osbuild/osbuild-composer", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@osbuild/osbuild-composer", "result_dir": "10343214-osbuild-composer", "built_packages": "", "id": 10343214, "mockchain_macros": { "copr_username": "@osbuild", "copr_projectname": "osbuild-composer", "vendor": "Fedora Project COPR (@osbuild/osbuild-composer)" } } ] } [2026-04-17 18:28:19,958][ INFO][PID:77238] Sending fedora-messaging bus message in build.start [2026-04-17 18:28:20,504][ INFO][PID:77238] Sending fedora-messaging bus message in chroot.start [2026-04-17 18:28:20,531][ INFO][PID:77238] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10343214-fedora-42-x86_64 --chroot fedora-42-x86_64 --detached [2026-04-17 18:28:21,247][ INFO][PID:77238] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c17c' (ED25519) to the list of known hosts. [2026-04-17 18:28:21,248][ INFO][PID:77238] Downloading the builder-live.log file, attempt 1 [2026-04-17 18:28:21,249][ INFO][PID:77238] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c17c copr-rpmbuild-log [2026-04-17 18:28:26,256][ INFO][PID:77238] Periodic builder liveness probe: alive [2026-04-17 18:28:26,257][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:28:31,258][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:28:36,259][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:28:41,260][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:28:46,261][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:28:51,262][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:28:56,263][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:29:01,264][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:29:06,265][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:29:11,266][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:29:16,267][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:29:21,268][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:29:26,269][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:29:31,270][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:29:36,271][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:29:41,272][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:29:46,273][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:29:51,274][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:29:56,275][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:30:01,287][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:30:06,289][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:30:11,292][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:30:16,293][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:30:21,294][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:30:26,295][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:30:31,296][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:30:36,297][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:30:41,298][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:30:46,314][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:30:51,315][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:30:56,316][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:31:01,317][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:31:06,318][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:31:11,319][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:31:16,320][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:31:21,321][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:31:26,322][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:31:31,323][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:31:36,324][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:31:41,325][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:31:46,325][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:31:51,326][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:31:56,327][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:32:01,328][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:32:06,329][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:32:11,330][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:32:16,331][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:32:21,332][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:32:26,333][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:32:31,334][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:32:36,335][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:32:41,336][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:32:46,337][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:32:51,338][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:32:56,340][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:33:01,341][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:33:06,341][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:33:11,342][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:33:16,349][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:33:21,351][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:33:26,352][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:33:31,358][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:33:36,360][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:33:41,361][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:33:46,362][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:33:51,363][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:33:56,363][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:34:01,365][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:34:06,370][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:34:11,371][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:34:16,355][ ERROR][PID:77238] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-17 18:34:16,371][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:34:21,372][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:34:26,356][WARNING][PID:77238] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-17 18:34:26,373][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:34:31,374][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:34:36,357][ INFO][PID:77238] Downloading the builder-live.log file, attempt 2 [2026-04-17 18:34:36,358][ INFO][PID:77238] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c17c copr-rpmbuild-log [2026-04-17 18:34:36,375][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:34:37,394][ ERROR][PID:77238] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-17 18:34:41,376][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:34:46,377][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:34:47,395][WARNING][PID:77238] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-17 18:34:51,378][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:34:56,384][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:34:57,396][ INFO][PID:77238] Downloading the builder-live.log file, attempt 3 [2026-04-17 18:34:57,397][ INFO][PID:77238] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c17c copr-rpmbuild-log [2026-04-17 18:34:58,461][ ERROR][PID:77238] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-17 18:35:01,385][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:35:06,386][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:35:08,466][WARNING][PID:77238] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-17 18:35:11,387][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:35:16,388][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:35:18,466][ INFO][PID:77238] Downloading the builder-live.log file, attempt 4 [2026-04-17 18:35:18,468][ INFO][PID:77238] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c17c copr-rpmbuild-log [2026-04-17 18:35:19,511][ ERROR][PID:77238] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-17 18:35:21,394][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:35:26,395][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:35:29,516][ ERROR][PID:77238] Giving up for unstable SSH, failures: 1776450866.3565607, 1776450887.3953874, 1776450908.466129, 1776450929.5164642 [2026-04-17 18:35:29,517][ ERROR][PID:77238] Re-try request for task on 'ResallocHost, ticket_id=2393541, hostname=2620:52:6:1161:dead:beef:cafe:c17c, name=vmhost_x86_04_prod_06200637_20260417_182602, requested_tags=['arch_x86_64', 'copr_builder']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-04-17 18:35:29,518][ INFO][PID:77238] VM Release request [2026-04-17 18:35:29,534][ INFO][PID:77238] Retry #1 (on other host) [2026-04-17 18:35:29,535][ INFO][PID:77238] Checking that /var/lib/copr/public_html/results/@osbuild/osbuild-composer/fedora-42-x86_64/repodata/repomd.xml exists [2026-04-17 18:35:29,535][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:35:29,536][ INFO][PID:77238] VM allocation process starts [2026-04-17 18:35:29,564][ INFO][PID:77238] Trying to allocate VM: ResallocHost, ticket_id=2393597, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-17 18:35:32,616][ INFO][PID:77238] Allocated host ResallocHost, ticket_id=2393597, hostname=2620:52:6:1161:dead:beef:cafe:c189, name=vmhost_x86_04_prod_06200680_20260417_182922, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-17 18:35:32,617][ INFO][PID:77238] Allocating ssh connection to builder [2026-04-17 18:35:32,618][ INFO][PID:77238] Checking that builder machine is OK [2026-04-17 18:35:33,722][ ERROR][PID:77238] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c189 port 22: Permission denied [2026-04-17 18:35:43,752][ ERROR][PID:77238] Re-try request for task on 'ResallocHost, ticket_id=2393597, hostname=2620:52:6:1161:dead:beef:cafe:c189, name=vmhost_x86_04_prod_06200680_20260417_182922, requested_tags=['arch_x86_64', 'copr_builder']': Unable to finish after 1 SSH attempts [2026-04-17 18:35:43,864][ INFO][PID:77238] VM Release request [2026-04-17 18:35:43,877][ INFO][PID:77238] Retry #2 (on other host) [2026-04-17 18:35:43,878][ INFO][PID:77238] Checking that /var/lib/copr/public_html/results/@osbuild/osbuild-composer/fedora-42-x86_64/repodata/repomd.xml exists [2026-04-17 18:35:43,878][ INFO][PID:77238] Checking for cancel request [2026-04-17 18:35:43,879][ INFO][PID:77238] VM allocation process starts [2026-04-17 18:35:43,892][ INFO][PID:77238] Trying to allocate VM: ResallocHost, ticket_id=2393609, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-17 18:35:46,915][ INFO][PID:77238] Allocated host ResallocHost, ticket_id=2393609, hostname=18.234.119.25, name=aws_x86_64_spot_prod_06199488_20260417_180807, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-17 18:35:46,916][ INFO][PID:77238] Allocating ssh connection to builder [2026-04-17 18:35:46,916][ INFO][PID:77238] Checking that builder machine is OK [2026-04-17 18:36:46,985][ ERROR][PID:77238] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 18.234.119.25 port 22: Connection timed out [2026-04-17 18:36:56,991][ ERROR][PID:77238] Re-try request for task on 'ResallocHost, ticket_id=2393609, hostname=18.234.119.25, name=aws_x86_64_spot_prod_06199488_20260417_180807, requested_tags=['arch_x86_64', 'copr_builder']': Unable to finish after 1 SSH attempts [2026-04-17 18:36:56,992][ INFO][PID:77238] VM Release request [2026-04-17 18:36:57,003][ ERROR][PID:77238] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c17c', '2620:52:6:1161:dead:beef:cafe:c189', '18.234.119.25'} [2026-04-17 18:36:57,003][WARNING][PID:77238] Switching not-finished job state to 'failed' [2026-04-17 18:36:57,004][ INFO][PID:77238] Worker failed build, took 517.1070523262024 [2026-04-17 18:36:57,004][ INFO][PID:77238] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@osbuild", "project_name": "osbuild-composer", "project_dirname": "osbuild-composer", "submitter": "packit", "ended_on": 1776451017.0037115, "started_on": 1776450499.8966591, "submitted_on": null, "status": 0, "chroot": "fedora-42-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10343214-fedora-42-x86_64", "build_id": 10343214, "package_name": "osbuild-composer", "package_version": "169-1.20260417182706903968.main.2.g755a65ce1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@osbuild/osbuild-composer/osbuild-composer", "git_hash": "a1f1f6214d7ab74653037ea68a95d574b06aa708", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@osbuild/osbuild-composer--packit", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@osbuild/osbuild-composer/fedora-42-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@osbuild/osbuild-composer", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@osbuild/osbuild-composer", "result_dir": "10343214-osbuild-composer", "built_packages": "", "id": 10343214, "mockchain_macros": { "copr_username": "@osbuild", "copr_projectname": "osbuild-composer", "vendor": "Fedora Project COPR (@osbuild/osbuild-composer)" } } ] } [2026-04-17 18:36:57,066][ INFO][PID:77238] Sending fedora-messaging bus message in build.end [2026-04-17 18:36:57,107][ INFO][PID:77238] Compressing /var/lib/copr/public_html/results/@osbuild/osbuild-composer/fedora-42-x86_64/10343214-osbuild-composer/builder-live.log by gzip [2026-04-17 18:36:57,108][ INFO][PID:77238] Running command 'gzip /var/lib/copr/public_html/results/@osbuild/osbuild-composer/fedora-42-x86_64/10343214-osbuild-composer/builder-live.log' as PID 122683 [2026-04-17 18:36:57,231][ INFO][PID:77238] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@osbuild/osbuild-composer/fedora-42-x86_64/10343214-osbuild-composer/builder-live.log) [2026-04-17 18:36:57,231][ INFO][PID:77238] Compressing /var/lib/copr/public_html/results/@osbuild/osbuild-composer/fedora-42-x86_64/10343214-osbuild-composer/backend.log by gzip [2026-04-17 18:36:57,233][ INFO][PID:77238] Running command 'gzip /var/lib/copr/public_html/results/@osbuild/osbuild-composer/fedora-42-x86_64/10343214-osbuild-composer/backend.log' as PID 122715