[2025-11-12 16:21:07,808][ INFO][PID:1962115] Marking build as starting [2025-11-12 16:21:07,858][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:21:07,860][ INFO][PID:1962115] VM allocation process starts [2025-11-12 16:21:07,873][ INFO][PID:1962115] Trying to allocate VM: ResallocHost, ticket_id=6146698, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-11-12 16:21:12,875][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:21:17,879][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:21:22,881][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:21:25,939][ INFO][PID:1962115] Allocated host ResallocHost, ticket_id=6146698, hostname=2620:52:6:1161:dead:beef:cafe:c10a, name=vmhost_p09_02_prod_07959398_20251112_155904, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-11-12 16:21:25,939][ INFO][PID:1962115] Allocating ssh connection to builder [2025-11-12 16:21:25,940][ INFO][PID:1962115] Checking that builder machine is OK [2025-11-12 16:21:27,255][ INFO][PID:1962115] Running remote command: copr-builder-ready fedora-rawhide-ppc64le [2025-11-12 16:21:28,144][ INFO][PID:1962115] Red Hat subscription not needed for fedora-rawhide-ppc64le Builder is ready to be used [2025-11-12 16:21:28,146][ INFO][PID:1962115] Filling build.info file with builder info [2025-11-12 16:21:28,150][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:21:28,155][ INFO][PID:1962115] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "lecris", "project_name": "cmake-4.0", "project_dirname": "cmake-4.0", "submitter": "lecris", "ended_on": null, "started_on": 1762964488.1538448, "submitted_on": null, "status": 3, "chroot": "fedora-rawhide-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9791571-fedora-rawhide-ppc64le", "build_id": 9791571, "package_name": "clementine", "package_version": "1.4.1-7", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/lecris/cmake-4.0/clementine", "git_hash": "7d63c7c897dbc05f8b1f5cdfab677a5a6497f954", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "lecris/cmake-4.0--lecris", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/lecris/cmake-4.0/fedora-rawhide-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/lecris/cmake-4.0", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/lecris/cmake-4.0", "result_dir": "09791571-clementine", "built_packages": "", "id": 9791571, "mockchain_macros": { "copr_username": "lecris", "copr_projectname": "cmake-4.0", "vendor": "Fedora Project COPR (lecris/cmake-4.0)" } } ] } [2025-11-12 16:21:28,206][ INFO][PID:1962115] Sending fedora-messaging bus message in build.start [2025-11-12 16:21:29,716][ INFO][PID:1962115] Sending fedora-messaging bus message in chroot.start [2025-11-12 16:21:29,791][ INFO][PID:1962115] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9791571-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached [2025-11-12 16:21:31,070][ INFO][PID:1962115] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c10a' (ED25519) to the list of known hosts. [2025-11-12 16:21:31,072][ INFO][PID:1962115] Downloading the builder-live.log file, attempt 1 [2025-11-12 16:21:31,074][ INFO][PID:1962115] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c10a copr-rpmbuild-log [2025-11-12 16:21:36,083][ INFO][PID:1962115] Periodic builder liveness probe: alive [2025-11-12 16:21:36,085][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:21:41,087][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:21:46,088][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:21:51,089][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:21:56,091][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:22:01,092][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:22:06,094][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:22:11,095][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:22:16,096][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:22:21,098][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:22:26,099][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:22:31,101][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:22:31,147][ ERROR][PID:1962115] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-11-12 16:22:36,103][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:22:41,105][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:22:41,150][WARNING][PID:1962115] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-11-12 16:22:46,106][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:22:51,107][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:22:51,151][ INFO][PID:1962115] Downloading the builder-live.log file, attempt 2 [2025-11-12 16:22:51,153][ INFO][PID:1962115] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c10a copr-rpmbuild-log [2025-11-12 16:22:56,109][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:23:01,111][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:23:06,113][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:23:11,115][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:23:16,116][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:23:21,117][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:23:26,119][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:23:31,121][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:23:36,123][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:23:41,124][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:23:46,125][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:23:51,127][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:23:51,222][ ERROR][PID:1962115] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-11-12 16:23:56,130][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:24:01,131][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:24:01,234][WARNING][PID:1962115] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-11-12 16:24:06,132][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:24:11,133][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:24:11,235][ INFO][PID:1962115] Downloading the builder-live.log file, attempt 3 [2025-11-12 16:24:11,237][ INFO][PID:1962115] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c10a copr-rpmbuild-log [2025-11-12 16:24:16,135][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:24:21,136][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:24:26,139][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:24:31,140][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:24:36,141][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:24:41,143][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:24:46,152][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:24:51,154][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:24:56,156][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:25:01,164][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:25:06,167][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:25:11,170][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:25:11,317][ ERROR][PID:1962115] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-11-12 16:25:16,174][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:25:21,177][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:25:21,334][WARNING][PID:1962115] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-11-12 16:25:26,179][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:25:31,181][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:25:31,335][ INFO][PID:1962115] Downloading the builder-live.log file, attempt 4 [2025-11-12 16:25:31,341][ INFO][PID:1962115] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c10a copr-rpmbuild-log [2025-11-12 16:25:36,184][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:25:41,191][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:25:46,192][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:25:51,193][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:25:56,197][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:26:01,198][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:26:06,202][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:26:11,203][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:26:16,205][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:26:21,210][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:26:26,211][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:26:31,214][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:26:31,430][ ERROR][PID:1962115] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-11-12 16:26:36,217][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:26:41,218][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:26:41,449][ ERROR][PID:1962115] Giving up for unstable SSH, failures: 1762964561.150549, 1762964641.2343946, 1762964721.3346572, 1762964801.4494543 [2025-11-12 16:26:41,451][ ERROR][PID:1962115] Re-try request for task on 'ResallocHost, ticket_id=6146698, hostname=2620:52:6:1161:dead:beef:cafe:c10a, name=vmhost_p09_02_prod_07959398_20251112_155904, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-11-12 16:26:41,453][ INFO][PID:1962115] Releasing VM back to pool [2025-11-12 16:26:41,467][ INFO][PID:1962115] Retry #1 (on other host) [2025-11-12 16:26:41,468][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:26:41,470][ INFO][PID:1962115] VM allocation process starts [2025-11-12 16:26:41,485][ INFO][PID:1962115] Trying to allocate VM: ResallocHost, ticket_id=6146878, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-11-12 16:26:46,487][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:26:51,489][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:26:56,497][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:27:01,501][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:27:06,504][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:27:11,510][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:27:16,513][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:27:21,517][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:27:26,519][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:27:31,520][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:27:36,528][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:27:41,531][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:27:46,537][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:27:49,704][ INFO][PID:1962115] Allocated host ResallocHost, ticket_id=6146878, hostname=2620:52:3:1:dead:beef:cafe:c298, name=vmhost_p09_01_prod_07958963_20251112_153727, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-11-12 16:27:49,707][ INFO][PID:1962115] Allocating ssh connection to builder [2025-11-12 16:27:49,709][ INFO][PID:1962115] Checking that builder machine is OK [2025-11-12 16:27:50,834][ INFO][PID:1962115] Running remote command: copr-builder-ready fedora-rawhide-ppc64le [2025-11-12 16:27:51,450][ INFO][PID:1962115] Red Hat subscription not needed for fedora-rawhide-ppc64le Builder is ready to be used [2025-11-12 16:27:51,450][ INFO][PID:1962115] Filling build.info file with builder info [2025-11-12 16:27:51,452][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:27:51,458][ INFO][PID:1962115] Not re-notifying FE and msg buses for the new host. [2025-11-12 16:27:51,459][ INFO][PID:1962115] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9791571-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached [2025-11-12 16:27:52,435][ INFO][PID:1962115] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c298' (ED25519) to the list of known hosts. [2025-11-12 16:27:52,437][ INFO][PID:1962115] Downloading the builder-live.log file, attempt 1 [2025-11-12 16:27:52,439][ INFO][PID:1962115] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c298 copr-rpmbuild-log [2025-11-12 16:27:57,437][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:28:02,438][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:28:07,441][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:28:12,445][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:28:17,452][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:28:22,464][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:28:27,470][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:28:32,476][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:28:37,481][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:28:42,483][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:28:47,485][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:28:52,491][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:28:57,502][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:29:02,505][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:29:07,508][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:29:12,512][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:29:17,515][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:29:22,516][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:29:27,521][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:29:32,527][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:29:37,549][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:29:42,572][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:29:47,574][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:29:52,575][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:29:57,578][ INFO][PID:1962115] Checking for cancel request [2025-11-12 16:29:59,603][ INFO][PID:1962115] Downloading results from builder [2025-11-12 16:29:59,605][ INFO][PID:1962115] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c298]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/lecris/cmake-4.0/fedora-rawhide-ppc64le/09791571-clementine started [2025-11-12 16:29:59,607][ INFO][PID:1962115] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c298]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/lecris/cmake-4.0/fedora-rawhide-ppc64le/09791571-clementine/ &> /var/lib/copr/public_html/results/lecris/cmake-4.0/fedora-rawhide-ppc64le/09791571-clementine/build-09791571.rsync.log [2025-11-12 16:30:00,363][ INFO][PID:1962115] rsyncing finished. [2025-11-12 16:30:00,364][ INFO][PID:1962115] Releasing VM back to pool [2025-11-12 16:30:00,386][ INFO][PID:1962115] Searching for 'success' file in resultdir [2025-11-12 16:30:00,387][ ERROR][PID:1962115] Build failed: Backend process error: No success file => build failure [2025-11-12 16:30:00,388][ INFO][PID:1962115] Finished build: id=9791571 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/lecris/cmake-4.0 chroot=fedora-rawhide-ppc64le [2025-11-12 16:30:00,394][ ERROR][PID:1962115] Backend process error: No success file => build failure [2025-11-12 16:30:00,395][ INFO][PID:1962115] Worker failed build, took 128.9369399547577 [2025-11-12 16:30:00,397][ INFO][PID:1962115] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "lecris", "project_name": "cmake-4.0", "project_dirname": "cmake-4.0", "submitter": "lecris", "ended_on": 1762965000.3956563, "started_on": 1762964871.4587164, "submitted_on": null, "status": 0, "chroot": "fedora-rawhide-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9791571-fedora-rawhide-ppc64le", "build_id": 9791571, "package_name": "clementine", "package_version": "1.4.1-7", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/lecris/cmake-4.0/clementine", "git_hash": "7d63c7c897dbc05f8b1f5cdfab677a5a6497f954", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "lecris/cmake-4.0--lecris", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/lecris/cmake-4.0/fedora-rawhide-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/lecris/cmake-4.0", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/lecris/cmake-4.0", "result_dir": "09791571-clementine", "built_packages": "", "id": 9791571, "mockchain_macros": { "copr_username": "lecris", "copr_projectname": "cmake-4.0", "vendor": "Fedora Project COPR (lecris/cmake-4.0)" } } ] } [2025-11-12 16:30:00,446][ INFO][PID:1962115] Sending fedora-messaging bus message in build.end [2025-11-12 16:30:00,497][ INFO][PID:1962115] Compressing /var/lib/copr/public_html/results/lecris/cmake-4.0/fedora-rawhide-ppc64le/09791571-clementine/builder-live.log by gzip [2025-11-12 16:30:00,500][ INFO][PID:1962115] Running command 'gzip /var/lib/copr/public_html/results/lecris/cmake-4.0/fedora-rawhide-ppc64le/09791571-clementine/builder-live.log' as PID 2029359 [2025-11-12 16:30:00,519][ INFO][PID:1962115] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/lecris/cmake-4.0/fedora-rawhide-ppc64le/09791571-clementine/builder-live.log) [2025-11-12 16:30:00,521][ INFO][PID:1962115] Compressing /var/lib/copr/public_html/results/lecris/cmake-4.0/fedora-rawhide-ppc64le/09791571-clementine/backend.log by gzip