[2026-04-08 09:48:52,723][ INFO][PID:3180697] Marking build as starting [2026-04-08 09:48:52,805][ INFO][PID:3180697] Checking that /var/lib/copr/public_html/results/luka222/mesa-cachyos-fedora/rhel-10-x86_64/repodata/repomd.xml exists [2026-04-08 09:48:53,021][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:48:53,022][ INFO][PID:3180697] VM allocation process starts [2026-04-08 09:48:53,048][ INFO][PID:3180697] Trying to allocate VM: ResallocHost, ticket_id=2222610, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-08 09:48:58,055][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:48:59,105][ INFO][PID:3180697] Allocated host ResallocHost, ticket_id=2222610, hostname=98.85.223.43, name=aws_x86_64_spot_prod_05923903_20260408_094630, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-08 09:48:59,106][ INFO][PID:3180697] Allocating ssh connection to builder [2026-04-08 09:48:59,106][ INFO][PID:3180697] Checking that builder machine is OK [2026-04-08 09:48:59,471][ INFO][PID:3180697] Running remote command: copr-builder-ready rhel-10-x86_64 [2026-04-08 09:48:59,687][ INFO][PID:3180697] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2026-04-08 09:48:59,689][ INFO][PID:3180697] Filling build.info file with builder info [2026-04-08 09:48:59,690][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:48:59,692][ INFO][PID:3180697] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "luka222", "project_name": "mesa-cachyos-fedora", "project_dirname": "mesa-cachyos-fedora", "submitter": "https://src.fedoraproject.org/user/churchyard", "ended_on": null, "started_on": 1775641739.6921563, "submitted_on": null, "status": 3, "chroot": "rhel-10-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10300845-rhel-10-x86_64", "build_id": 10300845, "package_name": "python-pytest-subprocess", "package_version": "1.5.4-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/luka222/mesa-cachyos-fedora/python-pytest-subprocess", "git_hash": "5f84d5620fc60386aed7adae4d8a11945cbb3247", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "luka222/mesa-cachyos-fedora--https://src.fedoraproject.org/user/churchyard", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/luka222/mesa-cachyos-fedora/rhel-10-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/luka222/DirectX-Headers-Git/rhel-10-x86_64/", "id": "https_download_copr_fedorainfracloud_org_results_luka222_DirectX_Headers_Git_chroot", "name": "Additional repo https_download_copr_fedorainfracloud_org_results_luka222_DirectX_Headers_Git_chroot" }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/luka222/rocm-rhel10/rhel-10-x86_64/", "id": "https_download_copr_fedorainfracloud_org_results_luka222_rocm_rhel10_chroot", "name": "Additional repo https_download_copr_fedorainfracloud_org_results_luka222_rocm_rhel10_chroot" } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/luka222/mesa-cachyos-fedora", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/luka222/mesa-cachyos-fedora", "result_dir": "10300845-python-pytest-subprocess", "built_packages": "", "id": 10300845, "mockchain_macros": { "copr_username": "luka222", "copr_projectname": "mesa-cachyos-fedora", "vendor": "Fedora Project COPR (luka222/mesa-cachyos-fedora)" } } ] } [2026-04-08 09:48:59,737][ INFO][PID:3180697] Sending fedora-messaging bus message in build.start [2026-04-08 09:49:00,514][ INFO][PID:3180697] Sending fedora-messaging bus message in chroot.start [2026-04-08 09:49:00,561][ INFO][PID:3180697] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10300845-rhel-10-x86_64 --chroot rhel-10-x86_64 --detached [2026-04-08 09:49:00,962][ INFO][PID:3180697] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '98.85.223.43' (ED25519) to the list of known hosts. [2026-04-08 09:49:00,964][ INFO][PID:3180697] Downloading the builder-live.log file, attempt 1 [2026-04-08 09:49:00,967][ INFO][PID:3180697] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.85.223.43 copr-rpmbuild-log [2026-04-08 09:49:05,986][ INFO][PID:3180697] Periodic builder liveness probe: alive [2026-04-08 09:49:05,987][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:49:10,988][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:49:15,990][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:49:20,991][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:49:25,992][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:49:30,994][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:49:36,044][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:49:41,047][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:49:46,051][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:49:51,057][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:49:56,058][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:49:59,684][ ERROR][PID:3180697] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-08 09:50:01,060][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:50:06,062][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:50:09,688][WARNING][PID:3180697] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-08 09:50:11,069][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:50:16,071][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:50:19,692][ INFO][PID:3180697] Downloading the builder-live.log file, attempt 2 [2026-04-08 09:50:19,694][ INFO][PID:3180697] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.85.223.43 copr-rpmbuild-log [2026-04-08 09:50:21,079][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:50:26,081][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:50:31,082][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:50:36,084][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:50:41,089][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:50:46,093][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:50:51,103][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:50:56,104][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:51:01,108][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:51:06,109][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:51:11,111][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:51:16,113][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:51:19,815][ ERROR][PID:3180697] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-08 09:51:21,118][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:51:26,123][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:51:29,823][WARNING][PID:3180697] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-08 09:51:31,128][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:51:36,132][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:51:39,828][ INFO][PID:3180697] Downloading the builder-live.log file, attempt 3 [2026-04-08 09:51:39,837][ INFO][PID:3180697] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.85.223.43 copr-rpmbuild-log [2026-04-08 09:51:41,138][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:51:46,140][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:51:51,142][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:51:56,143][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:52:01,144][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:52:06,145][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:52:11,146][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:52:16,267][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:52:21,271][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:52:26,278][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:52:31,280][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:52:36,281][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:52:39,921][ ERROR][PID:3180697] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-08 09:52:41,282][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:52:46,283][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:52:49,938][WARNING][PID:3180697] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-08 09:52:51,285][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:52:56,287][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:52:59,939][ INFO][PID:3180697] Downloading the builder-live.log file, attempt 4 [2026-04-08 09:52:59,940][ INFO][PID:3180697] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.85.223.43 copr-rpmbuild-log [2026-04-08 09:53:01,288][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:53:06,289][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:53:11,290][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:53:16,291][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:53:21,293][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:53:26,293][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:53:31,828][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:53:36,829][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:53:41,830][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:53:46,831][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:53:51,832][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:53:56,834][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:54:00,007][ ERROR][PID:3180697] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-08 09:54:01,835][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:54:06,836][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:54:10,008][ ERROR][PID:3180697] Giving up for unstable SSH, failures: 1775641809.688332, 1775641889.8234885, 1775641969.938432, 1775642050.008437 [2026-04-08 09:54:10,009][ ERROR][PID:3180697] Re-try request for task on 'ResallocHost, ticket_id=2222610, hostname=98.85.223.43, name=aws_x86_64_spot_prod_05923903_20260408_094630, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-04-08 09:54:10,010][ INFO][PID:3180697] Releasing VM back to pool [2026-04-08 09:54:10,020][ INFO][PID:3180697] Retry #1 (on other host) [2026-04-08 09:54:10,020][ INFO][PID:3180697] Checking that /var/lib/copr/public_html/results/luka222/mesa-cachyos-fedora/rhel-10-x86_64/repodata/repomd.xml exists [2026-04-08 09:54:10,021][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:54:10,022][ INFO][PID:3180697] VM allocation process starts [2026-04-08 09:54:10,034][ INFO][PID:3180697] Trying to allocate VM: ResallocHost, ticket_id=2222705, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-08 09:54:13,048][ INFO][PID:3180697] Allocated host ResallocHost, ticket_id=2222705, hostname=3.84.11.248, name=aws_x86_64_normalreserved_prod_05924070_20260408_095132, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-08 09:54:13,048][ INFO][PID:3180697] Allocating ssh connection to builder [2026-04-08 09:54:13,049][ INFO][PID:3180697] Checking that builder machine is OK [2026-04-08 09:54:13,383][ INFO][PID:3180697] Running remote command: copr-builder-ready rhel-10-x86_64 [2026-04-08 09:54:13,625][ INFO][PID:3180697] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2026-04-08 09:54:13,626][ INFO][PID:3180697] Filling build.info file with builder info [2026-04-08 09:54:13,637][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:54:13,638][ INFO][PID:3180697] Not re-notifying FE and msg buses for the new host. [2026-04-08 09:54:13,638][ INFO][PID:3180697] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10300845-rhel-10-x86_64 --chroot rhel-10-x86_64 --detached [2026-04-08 09:54:13,981][ INFO][PID:3180697] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '3.84.11.248' (ED25519) to the list of known hosts. [2026-04-08 09:54:13,982][ INFO][PID:3180697] Downloading the builder-live.log file, attempt 1 [2026-04-08 09:54:13,984][ INFO][PID:3180697] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.84.11.248 copr-rpmbuild-log [2026-04-08 09:54:18,983][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:54:23,984][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:54:28,985][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:54:33,986][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:54:38,987][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:54:43,988][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:54:48,989][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:54:53,990][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:54:58,991][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:55:03,992][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:55:08,993][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:55:13,995][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:55:18,996][ INFO][PID:3180697] Checking for cancel request [2026-04-08 09:55:19,715][ INFO][PID:3180697] Downloading results from builder [2026-04-08 09:55:19,715][ INFO][PID:3180697] rsyncing of mockbuilder@3.84.11.248:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/luka222/mesa-cachyos-fedora/rhel-10-x86_64/10300845-python-pytest-subprocess started [2026-04-08 09:55:19,716][ INFO][PID:3180697] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@3.84.11.248:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/luka222/mesa-cachyos-fedora/rhel-10-x86_64/10300845-python-pytest-subprocess/ &> /var/lib/copr/public_html/results/luka222/mesa-cachyos-fedora/rhel-10-x86_64/10300845-python-pytest-subprocess/build-10300845.rsync.log [2026-04-08 09:55:19,998][ INFO][PID:3180697] rsyncing finished. [2026-04-08 09:55:19,999][ INFO][PID:3180697] Releasing VM back to pool [2026-04-08 09:55:20,008][ INFO][PID:3180697] Searching for 'success' file in resultdir [2026-04-08 09:55:20,009][ ERROR][PID:3180697] Build failed: Backend process error: No success file => build failure [2026-04-08 09:55:20,009][ INFO][PID:3180697] Finished build: id=10300845 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/luka222/mesa-cachyos-fedora chroot=rhel-10-x86_64 [2026-04-08 09:55:20,011][ ERROR][PID:3180697] Backend process error: No success file => build failure [2026-04-08 09:55:20,012][ INFO][PID:3180697] Worker failed build, took 66.3739447593689 [2026-04-08 09:55:20,012][ INFO][PID:3180697] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "luka222", "project_name": "mesa-cachyos-fedora", "project_dirname": "mesa-cachyos-fedora", "submitter": "https://src.fedoraproject.org/user/churchyard", "ended_on": 1775642120.012101, "started_on": 1775642053.6381562, "submitted_on": null, "status": 0, "chroot": "rhel-10-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10300845-rhel-10-x86_64", "build_id": 10300845, "package_name": "python-pytest-subprocess", "package_version": "1.5.4-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/luka222/mesa-cachyos-fedora/python-pytest-subprocess", "git_hash": "5f84d5620fc60386aed7adae4d8a11945cbb3247", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "luka222/mesa-cachyos-fedora--https://src.fedoraproject.org/user/churchyard", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/luka222/mesa-cachyos-fedora/rhel-10-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/luka222/DirectX-Headers-Git/rhel-10-x86_64/", "id": "https_download_copr_fedorainfracloud_org_results_luka222_DirectX_Headers_Git_chroot", "name": "Additional repo https_download_copr_fedorainfracloud_org_results_luka222_DirectX_Headers_Git_chroot" }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/luka222/rocm-rhel10/rhel-10-x86_64/", "id": "https_download_copr_fedorainfracloud_org_results_luka222_rocm_rhel10_chroot", "name": "Additional repo https_download_copr_fedorainfracloud_org_results_luka222_rocm_rhel10_chroot" } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/luka222/mesa-cachyos-fedora", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/luka222/mesa-cachyos-fedora", "result_dir": "10300845-python-pytest-subprocess", "built_packages": "", "id": 10300845, "mockchain_macros": { "copr_username": "luka222", "copr_projectname": "mesa-cachyos-fedora", "vendor": "Fedora Project COPR (luka222/mesa-cachyos-fedora)" } } ] } [2026-04-08 09:55:20,049][ INFO][PID:3180697] Sending fedora-messaging bus message in build.end [2026-04-08 09:55:20,075][ INFO][PID:3180697] Compressing /var/lib/copr/public_html/results/luka222/mesa-cachyos-fedora/rhel-10-x86_64/10300845-python-pytest-subprocess/builder-live.log by gzip [2026-04-08 09:55:20,076][ INFO][PID:3180697] Running command 'gzip /var/lib/copr/public_html/results/luka222/mesa-cachyos-fedora/rhel-10-x86_64/10300845-python-pytest-subprocess/builder-live.log' as PID 3237798 [2026-04-08 09:55:20,081][ INFO][PID:3180697] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/luka222/mesa-cachyos-fedora/rhel-10-x86_64/10300845-python-pytest-subprocess/builder-live.log) [2026-04-08 09:55:20,081][ INFO][PID:3180697] Compressing /var/lib/copr/public_html/results/luka222/mesa-cachyos-fedora/rhel-10-x86_64/10300845-python-pytest-subprocess/backend.log by gzip [2026-04-08 09:55:20,082][ INFO][PID:3180697] Running command 'gzip /var/lib/copr/public_html/results/luka222/mesa-cachyos-fedora/rhel-10-x86_64/10300845-python-pytest-subprocess/backend.log' as PID 3237800