[2025-12-10 03:15:56,324][ INFO][PID:2829885] Marking build as starting [2025-12-10 03:15:56,399][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:15:56,401][ INFO][PID:2829885] VM allocation process starts [2025-12-10 03:15:56,418][ INFO][PID:2829885] Trying to allocate VM: ResallocHost, ticket_id=252379, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2025-12-10 03:15:59,440][ INFO][PID:2829885] Allocated host ResallocHost, ticket_id=252379, hostname=2620:52:6:1161:dead:beef:cafe:c190, name=vmhost_p09_04_prod_00296160_20251210_030752, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2025-12-10 03:15:59,440][ INFO][PID:2829885] Allocating ssh connection to builder [2025-12-10 03:15:59,441][ INFO][PID:2829885] Checking that builder machine is OK [2025-12-10 03:16:59,513][ ERROR][PID:2829885] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c190 port 22: Connection timed out [2025-12-10 03:17:09,515][ ERROR][PID:2829885] Re-try request for task on 'ResallocHost, ticket_id=252379, hostname=2620:52:6:1161:dead:beef:cafe:c190, name=vmhost_p09_04_prod_00296160_20251210_030752, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 1 SSH attempts [2025-12-10 03:17:09,516][ INFO][PID:2829885] Releasing VM back to pool [2025-12-10 03:17:09,524][ INFO][PID:2829885] Retry #1 (on other host) [2025-12-10 03:17:09,525][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:17:09,526][ INFO][PID:2829885] VM allocation process starts [2025-12-10 03:17:09,560][ INFO][PID:2829885] Trying to allocate VM: ResallocHost, ticket_id=252398, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2025-12-10 03:17:12,591][ INFO][PID:2829885] Allocated host ResallocHost, ticket_id=252398, hostname=2620:52:6:1161:dead:beef:cafe:c192, name=vmhost_p09_04_prod_00296104_20251210_030550, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2025-12-10 03:17:12,592][ INFO][PID:2829885] Allocating ssh connection to builder [2025-12-10 03:17:12,592][ INFO][PID:2829885] Checking that builder machine is OK [2025-12-10 03:17:13,474][ INFO][PID:2829885] Running remote command: copr-builder-ready epel-10-ppc64le [2025-12-10 03:18:13,546][ ERROR][PID:2829885] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c192 port 22: Connection timed out [2025-12-10 03:18:23,547][ ERROR][PID:2829885] Re-try request for task on 'ResallocHost, ticket_id=252398, hostname=2620:52:6:1161:dead:beef:cafe:c192, name=vmhost_p09_04_prod_00296104_20251210_030550, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 1 SSH attempts [2025-12-10 03:18:23,547][ INFO][PID:2829885] Releasing VM back to pool [2025-12-10 03:18:23,557][ INFO][PID:2829885] Retry #2 (on other host) [2025-12-10 03:18:23,558][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:18:23,559][ INFO][PID:2829885] VM allocation process starts [2025-12-10 03:18:23,570][ INFO][PID:2829885] Trying to allocate VM: ResallocHost, ticket_id=252416, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2025-12-10 03:18:26,589][ INFO][PID:2829885] Allocated host ResallocHost, ticket_id=252416, hostname=2620:52:6:1161:dead:beef:cafe:c115, name=vmhost_p09_02_prod_00296220_20251210_031054, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2025-12-10 03:18:26,589][ INFO][PID:2829885] Allocating ssh connection to builder [2025-12-10 03:18:26,590][ INFO][PID:2829885] Checking that builder machine is OK [2025-12-10 03:18:27,544][ INFO][PID:2829885] Running remote command: copr-builder-ready epel-10-ppc64le [2025-12-10 03:18:28,050][ INFO][PID:2829885] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2025-12-10 03:18:28,051][ INFO][PID:2829885] Filling build.info file with builder info [2025-12-10 03:18:28,052][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:18:28,052][ INFO][PID:2829885] Not re-notifying FE and msg buses for the new host. [2025-12-10 03:18:28,053][ INFO][PID:2829885] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9893595-epel-10-ppc64le --chroot epel-10-ppc64le --detached [2025-12-10 03:18:29,010][ INFO][PID:2829885] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c115' (ED25519) to the list of known hosts. [2025-12-10 03:18:29,011][ INFO][PID:2829885] Downloading the builder-live.log file, attempt 1 [2025-12-10 03:18:29,012][ INFO][PID:2829885] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c115 copr-rpmbuild-log [2025-12-10 03:18:34,030][ INFO][PID:2829885] Periodic builder liveness probe: alive [2025-12-10 03:18:34,032][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:18:39,033][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:18:44,034][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:18:49,035][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:18:54,036][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:18:59,038][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:19:04,039][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:19:09,040][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:19:14,041][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:19:19,042][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:19:24,043][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:19:29,045][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:19:34,047][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:19:39,048][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:19:44,050][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:19:49,051][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:19:54,054][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:19:59,055][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:20:04,059][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:20:09,062][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:20:14,064][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:20:19,065][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:20:24,066][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:20:29,067][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:20:34,069][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:20:39,148][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:20:44,149][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:20:49,156][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:20:54,229][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:20:59,230][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:21:04,232][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:21:09,234][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:21:14,236][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:21:19,239][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:21:24,241][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:21:29,247][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:21:34,248][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:21:39,251][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:21:44,252][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:21:49,253][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:21:54,254][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:21:59,255][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:22:04,256][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:22:09,258][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:22:14,259][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:22:19,260][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:22:24,261][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:22:29,263][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:22:34,265][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:22:39,272][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:22:44,274][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:22:49,277][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:22:54,280][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:22:59,281][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:23:04,289][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:23:09,293][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:23:14,294][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:23:19,296][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:23:24,298][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:23:29,301][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:23:34,303][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:23:39,304][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:23:44,305][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:23:49,387][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:23:54,388][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:23:59,389][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:24:04,390][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:24:09,391][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:24:14,465][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:24:19,466][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:24:24,467][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:24:29,469][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:24:34,476][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:24:39,479][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:24:44,480][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:24:49,482][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:24:54,483][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:24:59,485][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:25:04,486][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:25:09,487][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:25:14,488][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:25:19,489][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:25:24,489][ INFO][PID:2829885] Checking for cancel request [2025-12-10 03:25:28,936][ INFO][PID:2829885] Downloading results from builder [2025-12-10 03:25:28,947][ INFO][PID:2829885] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c115]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/brycensranch/gpu-screen-recorder-git/epel-10-ppc64le/09893595-gpu-screen-recorder-ui started [2025-12-10 03:25:28,953][ INFO][PID:2829885] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c115]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/brycensranch/gpu-screen-recorder-git/epel-10-ppc64le/09893595-gpu-screen-recorder-ui/ &> /var/lib/copr/public_html/results/brycensranch/gpu-screen-recorder-git/epel-10-ppc64le/09893595-gpu-screen-recorder-ui/build-09893595.rsync.log [2025-12-10 03:26:29,074][ INFO][PID:2829885] rsyncing finished. [2025-12-10 03:26:29,079][ ERROR][PID:2829885] Failed to download data from builder due to rsync error, see the rsync log file for details. [2025-12-10 03:26:29,081][ ERROR][PID:2829885] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2025-12-10 03:26:39,083][ INFO][PID:2829885] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c115]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/brycensranch/gpu-screen-recorder-git/epel-10-ppc64le/09893595-gpu-screen-recorder-ui started [2025-12-10 03:26:39,085][ INFO][PID:2829885] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c115]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/brycensranch/gpu-screen-recorder-git/epel-10-ppc64le/09893595-gpu-screen-recorder-ui/ &> /var/lib/copr/public_html/results/brycensranch/gpu-screen-recorder-git/epel-10-ppc64le/09893595-gpu-screen-recorder-ui/build-09893595.rsync.log [2025-12-10 03:27:39,173][ INFO][PID:2829885] rsyncing finished. [2025-12-10 03:27:39,176][ ERROR][PID:2829885] Failed to download data from builder due to rsync error, see the rsync log file for details. [2025-12-10 03:27:39,179][ ERROR][PID:2829885] SSH connection lost on #2 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2025-12-10 03:27:49,187][ INFO][PID:2829885] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c115]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/brycensranch/gpu-screen-recorder-git/epel-10-ppc64le/09893595-gpu-screen-recorder-ui started [2025-12-10 03:27:49,192][ INFO][PID:2829885] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c115]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/brycensranch/gpu-screen-recorder-git/epel-10-ppc64le/09893595-gpu-screen-recorder-ui/ &> /var/lib/copr/public_html/results/brycensranch/gpu-screen-recorder-git/epel-10-ppc64le/09893595-gpu-screen-recorder-ui/build-09893595.rsync.log [2025-12-10 03:28:49,470][ INFO][PID:2829885] rsyncing finished. [2025-12-10 03:28:49,471][ ERROR][PID:2829885] Failed to download data from builder due to rsync error, see the rsync log file for details. [2025-12-10 03:28:49,472][ ERROR][PID:2829885] SSH connection lost on #3 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2025-12-10 03:28:59,475][ ERROR][PID:2829885] Re-try request for task on 'ResallocHost, ticket_id=252416, hostname=2620:52:6:1161:dead:beef:cafe:c115, name=vmhost_p09_02_prod_00296220_20251210_031054, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 3 SSH attempts [2025-12-10 03:28:59,481][ INFO][PID:2829885] Releasing VM back to pool [2025-12-10 03:28:59,692][ ERROR][PID:2829885] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c115', '2620:52:6:1161:dead:beef:cafe:c190', '2620:52:6:1161:dead:beef:cafe:c192'} [2025-12-10 03:28:59,705][WARNING][PID:2829885] Switching not-finished job state to 'failed' [2025-12-10 03:28:59,708][ INFO][PID:2829885] Worker failed build, took 631.6424307823181 [2025-12-10 03:28:59,736][ INFO][PID:2829885] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "brycensranch", "project_name": "gpu-screen-recorder-git", "project_dirname": "gpu-screen-recorder-git", "submitter": "https://api.github.com/users/BrycensRanch", "ended_on": 1765337339.6953168, "started_on": 1765336708.052886, "submitted_on": null, "status": 0, "chroot": "epel-10-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9893595-epel-10-ppc64le", "build_id": 9893595, "package_name": "gpu-screen-recorder-ui", "package_version": "2:1.8.3-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/brycensranch/gpu-screen-recorder-git/gpu-screen-recorder-ui", "git_hash": "c106ba25d688ccd6334459b62c75e5502cc47c3f", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "brycensranch/gpu-screen-recorder-git--https://api.github.com/users/BrycensRanch", "results": null, "appstream": true, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/brycensranch/gpu-screen-recorder-git/epel-10-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le", "arch_power9" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/brycensranch/gpu-screen-recorder-git", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/brycensranch/gpu-screen-recorder-git", "result_dir": "09893595-gpu-screen-recorder-ui", "built_packages": "", "id": 9893595, "mockchain_macros": { "copr_username": "brycensranch", "copr_projectname": "gpu-screen-recorder-git", "vendor": "Fedora Project COPR (brycensranch/gpu-screen-recorder-git)" } } ] } [2025-12-10 03:29:00,621][ INFO][PID:2829885] Sending fedora-messaging bus message in build.end [2025-12-10 03:29:04,122][ INFO][PID:2829885] Compressing /var/lib/copr/public_html/results/brycensranch/gpu-screen-recorder-git/epel-10-ppc64le/09893595-gpu-screen-recorder-ui/builder-live.log by gzip [2025-12-10 03:29:04,125][ INFO][PID:2829885] Running command 'gzip /var/lib/copr/public_html/results/brycensranch/gpu-screen-recorder-git/epel-10-ppc64le/09893595-gpu-screen-recorder-ui/builder-live.log' as PID 2920851 [2025-12-10 03:29:04,141][ INFO][PID:2829885] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/brycensranch/gpu-screen-recorder-git/epel-10-ppc64le/09893595-gpu-screen-recorder-ui/builder-live.log) [2025-12-10 03:29:04,142][ INFO][PID:2829885] Compressing /var/lib/copr/public_html/results/brycensranch/gpu-screen-recorder-git/epel-10-ppc64le/09893595-gpu-screen-recorder-ui/backend.log by gzip [2025-12-10 03:29:04,143][ INFO][PID:2829885] Running command 'gzip /var/lib/copr/public_html/results/brycensranch/gpu-screen-recorder-git/epel-10-ppc64le/09893595-gpu-screen-recorder-ui/backend.log' as PID 2920857