[2026-04-08 20:14:44,857][ INFO][PID:2915321] Marking build as starting [2026-04-08 20:14:44,904][ INFO][PID:2915321] Checking that /var/lib/copr/public_html/results/dchen/el-pkgs/rhel+epel-10-x86_64/repodata/repomd.xml exists [2026-04-08 20:14:44,956][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:14:44,958][ INFO][PID:2915321] VM allocation process starts [2026-04-08 20:14:44,973][ INFO][PID:2915321] Trying to allocate VM: ResallocHost, ticket_id=2232214, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-08 20:14:48,011][ INFO][PID:2915321] Allocated host ResallocHost, ticket_id=2232214, hostname=3.231.168.51, name=aws_x86_64_spot_prod_05947359_20260408_201107, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-08 20:14:48,012][ INFO][PID:2915321] Allocating ssh connection to builder [2026-04-08 20:14:48,013][ INFO][PID:2915321] Checking that builder machine is OK [2026-04-08 20:14:48,395][ INFO][PID:2915321] Running remote command: copr-builder-ready rhel+epel-10-x86_64 [2026-04-08 20:14:48,643][ INFO][PID:2915321] Red Hat subscription not needed for rhel+epel-10-x86_64 Builder is ready to be used [2026-04-08 20:14:48,646][ INFO][PID:2915321] Filling build.info file with builder info [2026-04-08 20:14:48,649][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:14:48,663][ INFO][PID:2915321] Sending build state back to frontend: { "builds": [ { "timeout": 36000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "dchen", "project_name": "el-pkgs", "project_dirname": "el-pkgs", "submitter": "https://src.fedoraproject.org/user/trix", "ended_on": null, "started_on": 1775679288.6627846, "submitted_on": null, "status": 3, "chroot": "rhel+epel-10-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10303000-rhel+epel-10-x86_64", "build_id": 10303000, "package_name": "rocm-compilersupport", "package_version": "22-8.rocm7.2.1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/dchen/el-pkgs/rocm-compilersupport", "git_hash": "2d38754e17524367755cdcde0fc92f72cadde227", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "dchen/el-pkgs--https://src.fedoraproject.org/user/trix", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/dchen/el-pkgs/rhel+epel-10-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "bootstrap": "off", "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/dchen/el-pkgs", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/dchen/el-pkgs", "result_dir": "10303000-rocm-compilersupport", "built_packages": "", "id": 10303000, "mockchain_macros": { "copr_username": "dchen", "copr_projectname": "el-pkgs", "vendor": "Fedora Project COPR (dchen/el-pkgs)" } } ] } [2026-04-08 20:14:48,788][ INFO][PID:2915321] Sending fedora-messaging bus message in build.start [2026-04-08 20:14:49,912][ INFO][PID:2915321] Sending fedora-messaging bus message in chroot.start [2026-04-08 20:14:49,948][ INFO][PID:2915321] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10303000-rhel+epel-10-x86_64 --chroot rhel+epel-10-x86_64 --detached [2026-04-08 20:14:50,318][ INFO][PID:2915321] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '3.231.168.51' (ED25519) to the list of known hosts. [2026-04-08 20:14:50,325][ INFO][PID:2915321] Downloading the builder-live.log file, attempt 1 [2026-04-08 20:14:50,328][ INFO][PID:2915321] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.231.168.51 copr-rpmbuild-log [2026-04-08 20:14:55,352][ INFO][PID:2915321] Periodic builder liveness probe: alive [2026-04-08 20:14:55,354][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:15:00,359][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:15:05,366][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:15:10,369][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:15:15,370][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:15:20,371][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:15:25,375][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:15:30,378][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:15:35,385][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:15:40,387][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:15:45,390][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:15:50,399][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:15:55,400][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:16:00,402][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:16:01,827][ ERROR][PID:2915321] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-08 20:16:05,403][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:16:10,407][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:16:11,829][WARNING][PID:2915321] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-08 20:16:15,408][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:16:20,410][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:16:21,830][ INFO][PID:2915321] Downloading the builder-live.log file, attempt 2 [2026-04-08 20:16:21,831][ INFO][PID:2915321] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.231.168.51 copr-rpmbuild-log [2026-04-08 20:16:25,411][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:16:30,412][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:16:35,413][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:16:40,452][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:16:45,712][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:16:50,713][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:16:55,714][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:17:01,132][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:17:06,135][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:17:11,138][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:17:16,517][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:17:21,714][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:17:21,895][ ERROR][PID:2915321] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-08 20:17:26,760][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:17:31,763][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:17:32,210][WARNING][PID:2915321] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-08 20:17:37,198][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:17:42,212][ INFO][PID:2915321] Downloading the builder-live.log file, attempt 3 [2026-04-08 20:17:42,213][ INFO][PID:2915321] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.231.168.51 copr-rpmbuild-log [2026-04-08 20:17:42,952][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:17:48,423][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:17:53,678][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:17:58,710][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:18:03,729][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:18:08,766][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:18:13,769][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:18:19,020][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:18:24,021][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:18:29,097][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:18:34,152][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:18:39,184][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:18:42,281][ ERROR][PID:2915321] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-08 20:18:44,201][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:18:49,203][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:18:52,285][WARNING][PID:2915321] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-08 20:18:54,204][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:18:59,206][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:19:02,286][ INFO][PID:2915321] Downloading the builder-live.log file, attempt 4 [2026-04-08 20:19:02,288][ INFO][PID:2915321] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.231.168.51 copr-rpmbuild-log [2026-04-08 20:19:04,236][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:19:09,243][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:19:14,251][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:19:19,268][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:19:24,271][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:19:29,277][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:19:34,438][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:19:39,446][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:19:44,447][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:19:49,451][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:19:54,457][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:19:59,459][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:20:02,354][ ERROR][PID:2915321] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-08 20:20:04,475][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:20:09,947][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:20:12,370][ ERROR][PID:2915321] Giving up for unstable SSH, failures: 1775679371.8288836, 1775679452.210194, 1775679532.2858021, 1775679612.3705215 [2026-04-08 20:20:12,371][ ERROR][PID:2915321] Re-try request for task on 'ResallocHost, ticket_id=2232214, hostname=3.231.168.51, name=aws_x86_64_spot_prod_05947359_20260408_201107, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-04-08 20:20:12,372][ INFO][PID:2915321] Releasing VM back to pool [2026-04-08 20:20:12,381][ INFO][PID:2915321] Retry #1 (on other host) [2026-04-08 20:20:12,381][ INFO][PID:2915321] Checking that /var/lib/copr/public_html/results/dchen/el-pkgs/rhel+epel-10-x86_64/repodata/repomd.xml exists [2026-04-08 20:20:12,382][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:20:12,382][ INFO][PID:2915321] VM allocation process starts [2026-04-08 20:20:12,393][ INFO][PID:2915321] Trying to allocate VM: ResallocHost, ticket_id=2232289, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-08 20:20:15,406][ INFO][PID:2915321] Allocated host ResallocHost, ticket_id=2232289, hostname=54.158.28.61, name=aws_x86_64_normalreserved_prod_05947509_20260408_201852, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-08 20:20:15,406][ INFO][PID:2915321] Allocating ssh connection to builder [2026-04-08 20:20:15,407][ INFO][PID:2915321] Checking that builder machine is OK [2026-04-08 20:20:15,739][ INFO][PID:2915321] Running remote command: copr-builder-ready rhel+epel-10-x86_64 [2026-04-08 20:20:15,949][ INFO][PID:2915321] Red Hat subscription not needed for rhel+epel-10-x86_64 Builder is ready to be used [2026-04-08 20:20:15,949][ INFO][PID:2915321] Filling build.info file with builder info [2026-04-08 20:20:15,955][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:20:15,956][ INFO][PID:2915321] Not re-notifying FE and msg buses for the new host. [2026-04-08 20:20:15,957][ INFO][PID:2915321] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10303000-rhel+epel-10-x86_64 --chroot rhel+epel-10-x86_64 --detached [2026-04-08 20:20:16,308][ INFO][PID:2915321] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '54.158.28.61' (ED25519) to the list of known hosts. [2026-04-08 20:20:16,308][ INFO][PID:2915321] Downloading the builder-live.log file, attempt 1 [2026-04-08 20:20:16,310][ INFO][PID:2915321] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.158.28.61 copr-rpmbuild-log [2026-04-08 20:20:21,309][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:20:26,310][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:20:31,311][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:20:36,315][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:20:41,320][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:20:46,321][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:20:51,322][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:20:56,322][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:21:01,324][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:21:06,330][ INFO][PID:2915321] Checking for cancel request [2026-04-08 20:21:07,801][ INFO][PID:2915321] Downloading results from builder [2026-04-08 20:21:07,801][ INFO][PID:2915321] rsyncing of mockbuilder@54.158.28.61:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/dchen/el-pkgs/rhel+epel-10-x86_64/10303000-rocm-compilersupport started [2026-04-08 20:21:07,802][ INFO][PID:2915321] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@54.158.28.61:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/dchen/el-pkgs/rhel+epel-10-x86_64/10303000-rocm-compilersupport/ &> /var/lib/copr/public_html/results/dchen/el-pkgs/rhel+epel-10-x86_64/10303000-rocm-compilersupport/build-10303000.rsync.log [2026-04-08 20:21:08,020][ INFO][PID:2915321] rsyncing finished. [2026-04-08 20:21:08,020][ INFO][PID:2915321] Releasing VM back to pool [2026-04-08 20:21:08,031][ INFO][PID:2915321] Searching for 'success' file in resultdir [2026-04-08 20:21:08,031][ ERROR][PID:2915321] Build failed: Backend process error: No success file => build failure [2026-04-08 20:21:08,032][ INFO][PID:2915321] Finished build: id=10303000 failed=True timeout=36000 destdir=/var/lib/copr/public_html/results/dchen/el-pkgs chroot=rhel+epel-10-x86_64 [2026-04-08 20:21:08,033][ ERROR][PID:2915321] Backend process error: No success file => build failure [2026-04-08 20:21:08,033][ INFO][PID:2915321] Worker failed build, took 52.07716774940491 [2026-04-08 20:21:08,034][ INFO][PID:2915321] Sending build state back to frontend: { "builds": [ { "timeout": 36000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "dchen", "project_name": "el-pkgs", "project_dirname": "el-pkgs", "submitter": "https://src.fedoraproject.org/user/trix", "ended_on": 1775679668.0338311, "started_on": 1775679615.9566634, "submitted_on": null, "status": 0, "chroot": "rhel+epel-10-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10303000-rhel+epel-10-x86_64", "build_id": 10303000, "package_name": "rocm-compilersupport", "package_version": "22-8.rocm7.2.1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/dchen/el-pkgs/rocm-compilersupport", "git_hash": "2d38754e17524367755cdcde0fc92f72cadde227", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "dchen/el-pkgs--https://src.fedoraproject.org/user/trix", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/dchen/el-pkgs/rhel+epel-10-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "bootstrap": "off", "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/dchen/el-pkgs", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/dchen/el-pkgs", "result_dir": "10303000-rocm-compilersupport", "built_packages": "", "id": 10303000, "mockchain_macros": { "copr_username": "dchen", "copr_projectname": "el-pkgs", "vendor": "Fedora Project COPR (dchen/el-pkgs)" } } ] } [2026-04-08 20:21:08,074][ INFO][PID:2915321] Sending fedora-messaging bus message in build.end [2026-04-08 20:21:08,101][ INFO][PID:2915321] Compressing /var/lib/copr/public_html/results/dchen/el-pkgs/rhel+epel-10-x86_64/10303000-rocm-compilersupport/builder-live.log by gzip [2026-04-08 20:21:08,102][ INFO][PID:2915321] Running command 'gzip /var/lib/copr/public_html/results/dchen/el-pkgs/rhel+epel-10-x86_64/10303000-rocm-compilersupport/builder-live.log' as PID 2948720 [2026-04-08 20:21:08,165][ INFO][PID:2915321] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/dchen/el-pkgs/rhel+epel-10-x86_64/10303000-rocm-compilersupport/builder-live.log) [2026-04-08 20:21:08,166][ INFO][PID:2915321] Compressing /var/lib/copr/public_html/results/dchen/el-pkgs/rhel+epel-10-x86_64/10303000-rocm-compilersupport/backend.log by gzip [2026-04-08 20:21:08,167][ INFO][PID:2915321] Running command 'gzip /var/lib/copr/public_html/results/dchen/el-pkgs/rhel+epel-10-x86_64/10303000-rocm-compilersupport/backend.log' as PID 2948739