[2026-05-28 21:19:11,250][ INFO][PID:3890941] Marking build as starting [2026-05-28 21:19:11,332][ INFO][PID:3890941] Checking that /var/lib/copr/public_html/results/ublue-os/staging/epel-10-x86_64/repodata/repomd.xml exists [2026-05-28 21:19:11,334][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:19:11,336][ INFO][PID:3890941] VM allocation process starts [2026-05-28 21:19:11,405][ INFO][PID:3890941] Trying to allocate VM: ResallocHost, ticket_id=3123299, requested_tags=['arch_x86_64', 'copr_builder'] [2026-05-28 21:19:14,427][ INFO][PID:3890941] Allocated host ResallocHost, ticket_id=3123299, hostname=34.224.93.98, name=aws_x86_64_spot_prod_10360970_20260528_211536, requested_tags=['arch_x86_64', 'copr_builder'] [2026-05-28 21:19:14,429][ INFO][PID:3890941] Allocating ssh connection to builder [2026-05-28 21:19:14,430][ INFO][PID:3890941] Checking that builder machine is OK [2026-05-28 21:19:14,782][ INFO][PID:3890941] Installed copr-rpmbuild version: 1.8 [2026-05-28 21:19:14,783][ INFO][PID:3890941] Running remote command: copr-builder-ready epel-10-x86_64 [2026-05-28 21:19:15,017][ INFO][PID:3890941] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2026-05-28 21:19:15,019][ INFO][PID:3890941] Filling build.info file with builder info [2026-05-28 21:19:15,020][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:19:15,022][ INFO][PID:3890941] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "ublue-os", "project_name": "staging", "project_dirname": "staging", "submitter": "https://api.github.com/users/ubot-7274%5Bbot%5D", "ended_on": null, "started_on": 1780003155.0220594, "submitted_on": null, "status": 3, "chroot": "epel-10-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10521988-epel-10-x86_64", "build_id": 10521988, "package_name": "fwupd", "package_version": "2.1.3-100.ublue", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/ublue-os/staging/fwupd", "git_hash": "32e01627dc158b491de85df41360446bfd4afc85", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "ublue-os/staging--https://api.github.com/users/ubot-7274%5Bbot%5D", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/ublue-os/staging/epel-10-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/ublue-os/staging", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/ublue-os/staging", "result_dir": "10521988-fwupd", "built_packages": "", "id": 10521988, "mockchain_macros": { "copr_username": "ublue-os", "copr_projectname": "staging", "vendor": "Fedora Project COPR (ublue-os/staging)" } } ] } [2026-05-28 21:19:15,087][ INFO][PID:3890941] Sending fedora-messaging bus message in build.start [2026-05-28 21:19:15,636][ INFO][PID:3890941] Sending fedora-messaging bus message in chroot.start [2026-05-28 21:19:15,667][ INFO][PID:3890941] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10521988-epel-10-x86_64 --chroot epel-10-x86_64 --detached [2026-05-28 21:19:16,101][ INFO][PID:3890941] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '34.224.93.98' (ED25519) to the list of known hosts. [2026-05-28 21:19:16,102][ INFO][PID:3890941] Downloading the builder-live.log file, attempt 1 [2026-05-28 21:19:16,104][ INFO][PID:3890941] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.224.93.98 copr-rpmbuild-log [2026-05-28 21:19:21,110][ INFO][PID:3890941] Periodic builder liveness probe: alive [2026-05-28 21:19:21,118][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:19:26,119][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:19:29,219][ ERROR][PID:3890941] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-05-28 21:19:31,121][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:19:36,122][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:19:39,228][WARNING][PID:3890941] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-05-28 21:19:41,124][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:19:46,125][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:19:49,229][ INFO][PID:3890941] Downloading the builder-live.log file, attempt 2 [2026-05-28 21:19:49,231][ INFO][PID:3890941] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.224.93.98 copr-rpmbuild-log [2026-05-28 21:19:51,127][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:19:56,129][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:20:01,144][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:20:06,146][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:20:11,149][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:20:16,152][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:20:21,158][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:20:26,159][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:20:31,162][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:20:36,163][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:20:41,165][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:20:46,166][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:20:49,298][ ERROR][PID:3890941] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-05-28 21:20:51,168][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:20:56,173][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:20:59,308][WARNING][PID:3890941] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-05-28 21:21:01,174][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:21:06,175][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:21:09,309][ INFO][PID:3890941] Downloading the builder-live.log file, attempt 3 [2026-05-28 21:21:09,311][ INFO][PID:3890941] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.224.93.98 copr-rpmbuild-log [2026-05-28 21:21:11,178][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:21:16,180][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:21:21,181][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:21:26,182][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:21:31,189][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:21:36,190][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:21:41,192][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:21:46,193][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:21:51,195][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:21:56,196][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:22:01,197][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:22:06,209][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:22:09,380][ ERROR][PID:3890941] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-05-28 21:22:11,210][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:22:16,212][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:22:19,387][WARNING][PID:3890941] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-05-28 21:22:21,213][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:22:26,215][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:22:29,388][ INFO][PID:3890941] Downloading the builder-live.log file, attempt 4 [2026-05-28 21:22:29,390][ INFO][PID:3890941] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.224.93.98 copr-rpmbuild-log [2026-05-28 21:22:31,216][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:22:36,217][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:22:41,219][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:22:46,220][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:22:51,221][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:22:56,253][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:23:01,255][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:23:06,256][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:23:11,257][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:23:16,259][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:23:21,260][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:23:26,261][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:23:29,458][ ERROR][PID:3890941] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-05-28 21:23:31,290][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:23:36,292][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:23:39,649][ ERROR][PID:3890941] Giving up for unstable SSH, failures: 1780003179.2288842, 1780003259.3085504, 1780003339.3878336, 1780003419.48162 [2026-05-28 21:23:39,912][ ERROR][PID:3890941] Re-try request for task on 'ResallocHost, ticket_id=3123299, hostname=34.224.93.98, name=aws_x86_64_spot_prod_10360970_20260528_211536, requested_tags=['arch_x86_64', 'copr_builder']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-05-28 21:23:40,160][ INFO][PID:3890941] VM Release request [2026-05-28 21:23:40,580][ INFO][PID:3890941] Retry #1 (on other host) [2026-05-28 21:23:40,612][ INFO][PID:3890941] Checking that /var/lib/copr/public_html/results/ublue-os/staging/epel-10-x86_64/repodata/repomd.xml exists [2026-05-28 21:23:40,862][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:23:40,888][ INFO][PID:3890941] VM allocation process starts [2026-05-28 21:23:41,146][ INFO][PID:3890941] Trying to allocate VM: ResallocHost, ticket_id=3123332, requested_tags=['arch_x86_64', 'copr_builder'] [2026-05-28 21:23:45,503][ INFO][PID:3890941] Allocated host ResallocHost, ticket_id=3123332, hostname=44.212.44.205, name=aws_x86_64_reserved_prod_10361078_20260528_212217, requested_tags=['arch_x86_64', 'copr_builder'] [2026-05-28 21:23:45,510][ INFO][PID:3890941] Allocating ssh connection to builder [2026-05-28 21:23:45,516][ INFO][PID:3890941] Checking that builder machine is OK [2026-05-28 21:23:45,883][ INFO][PID:3890941] Installed copr-rpmbuild version: 1.8 [2026-05-28 21:23:45,957][ INFO][PID:3890941] Running remote command: copr-builder-ready epel-10-x86_64 [2026-05-28 21:23:46,186][ INFO][PID:3890941] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2026-05-28 21:23:46,187][ INFO][PID:3890941] Filling build.info file with builder info [2026-05-28 21:23:46,202][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:23:46,239][ INFO][PID:3890941] Not re-notifying FE and msg buses for the new host. [2026-05-28 21:23:46,285][ INFO][PID:3890941] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10521988-epel-10-x86_64 --chroot epel-10-x86_64 --detached [2026-05-28 21:23:46,761][ INFO][PID:3890941] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '44.212.44.205' (ED25519) to the list of known hosts. [2026-05-28 21:23:46,762][ INFO][PID:3890941] Downloading the builder-live.log file, attempt 1 [2026-05-28 21:23:46,764][ INFO][PID:3890941] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.212.44.205 copr-rpmbuild-log [2026-05-28 21:23:51,762][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:23:56,764][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:24:01,765][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:24:06,767][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:24:11,770][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:24:16,771][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:24:21,773][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:24:26,776][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:24:31,778][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:24:36,779][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:24:41,780][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:24:46,782][ INFO][PID:3890941] Checking for cancel request [2026-05-28 21:24:50,301][ INFO][PID:3890941] Downloading results from builder [2026-05-28 21:24:50,315][ INFO][PID:3890941] rsyncing of mockbuilder@44.212.44.205:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/ublue-os/staging/epel-10-x86_64/10521988-fwupd started [2026-05-28 21:24:50,318][ INFO][PID:3890941] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@44.212.44.205:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/ublue-os/staging/epel-10-x86_64/10521988-fwupd/ &> /var/lib/copr/public_html/results/ublue-os/staging/epel-10-x86_64/10521988-fwupd/build-10521988.rsync.log [2026-05-28 21:24:50,654][ INFO][PID:3890941] rsyncing finished. [2026-05-28 21:24:50,660][ INFO][PID:3890941] VM Release request [2026-05-28 21:24:50,723][ INFO][PID:3890941] Searching for 'success' file in resultdir [2026-05-28 21:24:50,724][ ERROR][PID:3890941] Build failed: Backend process error: No success file => build failure [2026-05-28 21:24:50,880][ INFO][PID:3890941] Finished build: id=10521988 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/ublue-os/staging chroot=epel-10-x86_64 [2026-05-28 21:24:50,936][ ERROR][PID:3890941] Backend process error: No success file => build failure [2026-05-28 21:24:50,955][ INFO][PID:3890941] Worker failed build, took 64.75249886512756 [2026-05-28 21:24:50,960][ INFO][PID:3890941] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "ublue-os", "project_name": "staging", "project_dirname": "staging", "submitter": "https://api.github.com/users/ubot-7274%5Bbot%5D", "ended_on": 1780003490.9557607, "started_on": 1780003426.2032619, "submitted_on": null, "status": 0, "chroot": "epel-10-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10521988-epel-10-x86_64", "build_id": 10521988, "package_name": "fwupd", "package_version": "2.1.3-100.ublue", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/ublue-os/staging/fwupd", "git_hash": "32e01627dc158b491de85df41360446bfd4afc85", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "ublue-os/staging--https://api.github.com/users/ubot-7274%5Bbot%5D", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/ublue-os/staging/epel-10-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/ublue-os/staging", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/ublue-os/staging", "result_dir": "10521988-fwupd", "built_packages": "", "id": 10521988, "mockchain_macros": { "copr_username": "ublue-os", "copr_projectname": "staging", "vendor": "Fedora Project COPR (ublue-os/staging)" } } ] } [2026-05-28 21:24:52,311][ INFO][PID:3890941] Sending fedora-messaging bus message in build.end [2026-05-28 21:24:52,829][ INFO][PID:3890941] Compressing /var/lib/copr/public_html/results/ublue-os/staging/epel-10-x86_64/10521988-fwupd/builder-live.log by gzip [2026-05-28 21:24:52,847][ INFO][PID:3890941] Running command 'gzip /var/lib/copr/public_html/results/ublue-os/staging/epel-10-x86_64/10521988-fwupd/builder-live.log' as PID 3919335 [2026-05-28 21:24:52,868][ INFO][PID:3890941] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/ublue-os/staging/epel-10-x86_64/10521988-fwupd/builder-live.log) [2026-05-28 21:24:52,869][ INFO][PID:3890941] Compressing /var/lib/copr/public_html/results/ublue-os/staging/epel-10-x86_64/10521988-fwupd/backend.log by gzip [2026-05-28 21:24:52,870][ INFO][PID:3890941] Running command 'gzip /var/lib/copr/public_html/results/ublue-os/staging/epel-10-x86_64/10521988-fwupd/backend.log' as PID 3919338