[2025-12-15 09:12:43,930][ INFO][PID:2612413] Marking build as starting [2025-12-15 09:12:44,007][ INFO][PID:2612413] Checking that /var/lib/copr/public_html/results/psimovec/scipy-1.10.0/fedora-41-ppc64le/repodata/repomd.xml exists [2025-12-15 09:12:44,069][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:12:44,070][ INFO][PID:2612413] VM allocation process starts [2025-12-15 09:12:44,082][ INFO][PID:2612413] Trying to allocate VM: ResallocHost, ticket_id=345902, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-12-15 09:12:47,099][ INFO][PID:2612413] Allocated host ResallocHost, ticket_id=345902, hostname=2620:52:6:1161:dead:beef:cafe:c10d, name=vmhost_p09_02_prod_00556020_20251215_090639, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-12-15 09:12:47,100][ INFO][PID:2612413] Allocating ssh connection to builder [2025-12-15 09:12:47,100][ INFO][PID:2612413] Checking that builder machine is OK [2025-12-15 09:12:48,144][ INFO][PID:2612413] Running remote command: copr-builder-ready fedora-41-ppc64le [2025-12-15 09:12:48,708][ INFO][PID:2612413] Red Hat subscription not needed for fedora-41-ppc64le Builder is ready to be used [2025-12-15 09:12:48,708][ INFO][PID:2612413] Filling build.info file with builder info [2025-12-15 09:12:48,709][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:12:48,710][ INFO][PID:2612413] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "psimovec", "project_name": "scipy-1.10.0", "project_dirname": "scipy-1.10.0", "submitter": "https://src.fedoraproject.org/user/davidsch", "ended_on": null, "started_on": 1765789968.7099457, "submitted_on": null, "status": 3, "chroot": "fedora-41-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9910409-fedora-41-ppc64le", "build_id": 9910409, "package_name": "bout++", "package_version": "5.2.0-4", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/psimovec/scipy-1.10.0/bout++", "git_hash": "0a82a212344b82b2f93fb5fce0a9fa6962685e50", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "psimovec/scipy-1.10.0--https://src.fedoraproject.org/user/davidsch", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/psimovec/scipy-1.10.0/fedora-41-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/", "id": "http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch", "name": "Additional repo http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch" } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/psimovec/scipy-1.10.0", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/psimovec/scipy-1.10.0", "result_dir": "09910409-bout++", "built_packages": "", "id": 9910409, "mockchain_macros": { "copr_username": "psimovec", "copr_projectname": "scipy-1.10.0", "vendor": "Fedora Project COPR (psimovec/scipy-1.10.0)" } } ] } [2025-12-15 09:12:48,774][ INFO][PID:2612413] Sending fedora-messaging bus message in build.start [2025-12-15 09:12:49,325][ INFO][PID:2612413] Sending fedora-messaging bus message in chroot.start [2025-12-15 09:12:49,352][ INFO][PID:2612413] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9910409-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached [2025-12-15 09:12:50,532][ INFO][PID:2612413] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c10d' (ED25519) to the list of known hosts. [2025-12-15 09:12:50,533][ INFO][PID:2612413] Downloading the builder-live.log file, attempt 1 [2025-12-15 09:12:50,534][ INFO][PID:2612413] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c10d copr-rpmbuild-log [2025-12-15 09:12:55,541][ INFO][PID:2612413] Periodic builder liveness probe: alive [2025-12-15 09:12:55,542][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:13:00,543][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:13:05,544][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:13:10,545][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:13:15,546][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:13:20,547][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:13:25,548][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:13:30,549][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:13:35,550][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:13:40,551][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:13:45,553][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:13:50,554][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:13:55,555][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:14:00,556][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:14:05,558][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:14:10,559][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:14:15,560][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:14:20,561][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:14:25,562][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:14:30,563][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:14:35,564][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:14:40,566][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:14:45,567][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:14:50,568][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:14:55,569][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:15:00,570][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:15:05,571][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:15:10,572][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:15:15,573][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:15:20,574][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:15:25,575][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:15:30,576][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:15:35,577][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:15:40,578][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:15:45,582][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:15:50,582][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:15:55,583][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:16:00,584][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:16:05,586][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:16:10,587][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:16:15,588][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:16:20,589][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:16:25,590][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:16:30,591][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:16:35,592][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:16:40,593][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:16:45,594][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:16:50,596][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:16:55,598][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:17:00,599][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:17:05,370][ ERROR][PID:2612413] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-15 09:17:05,600][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:17:10,601][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:17:15,371][WARNING][PID:2612413] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-15 09:17:15,602][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:17:20,603][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:17:25,372][ INFO][PID:2612413] Downloading the builder-live.log file, attempt 2 [2025-12-15 09:17:25,417][ INFO][PID:2612413] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c10d copr-rpmbuild-log [2025-12-15 09:17:25,604][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:17:30,605][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:17:35,606][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:17:40,607][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:17:45,608][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:17:50,610][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:17:55,611][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:18:00,612][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:18:05,613][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:18:10,614][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:18:15,615][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:18:20,616][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:18:25,617][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:18:30,618][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:18:35,620][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:18:40,621][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:18:45,622][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:18:50,623][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:18:55,624][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:19:00,625][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:19:05,626][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:19:10,627][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:19:15,628][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:19:20,629][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:19:25,630][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:19:30,631][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:19:35,632][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:19:40,633][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:19:45,634][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:19:50,635][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:19:55,636][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:20:00,637][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:20:05,638][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:20:10,639][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:20:15,640][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:20:20,641][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:20:25,642][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:20:30,643][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:20:35,644][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:20:40,645][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:20:45,798][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:20:50,799][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:20:55,800][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:21:00,801][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:21:05,802][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:21:10,803][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:21:15,804][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:21:20,805][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:21:25,918][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:21:30,919][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:21:35,920][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:21:40,921][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:21:45,923][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:21:50,923][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:21:55,924][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:22:00,926][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:22:05,937][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:22:10,938][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:22:15,939][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:22:20,940][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:22:25,942][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:22:30,943][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:22:36,238][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:22:41,239][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:22:46,241][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:22:51,243][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:22:56,253][ INFO][PID:2612413] Periodic builder liveness probe: alive [2025-12-15 09:22:56,253][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:23:01,254][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:23:06,255][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:23:11,256][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:23:16,257][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:23:21,259][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:23:26,260][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:23:31,261][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:23:36,262][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:23:41,263][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:23:46,264][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:23:51,265][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:23:56,266][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:24:01,267][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:24:06,268][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:24:11,269][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:24:16,270][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:24:21,271][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:24:26,273][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:24:31,275][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:24:36,275][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:24:41,276][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:24:46,277][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:24:51,307][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:24:56,532][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:25:01,533][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:25:06,534][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:25:11,535][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:25:16,537][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:25:21,538][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:25:26,540][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:25:31,541][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:25:36,542][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:25:41,543][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:25:46,547][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:25:51,549][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:25:56,550][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:26:01,551][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:26:06,553][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:26:11,555][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:26:16,556][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:26:21,557][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:26:26,558][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:26:31,560][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:26:36,563][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:26:41,565][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:26:46,566][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:26:51,569][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:26:56,571][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:27:01,572][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:27:06,577][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:27:11,580][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:27:16,582][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:27:21,583][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:27:26,585][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:27:31,586][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:27:36,587][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:27:41,588][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:27:46,589][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:27:51,590][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:27:56,606][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:28:01,607][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:28:06,608][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:28:11,609][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:28:16,610][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:28:21,611][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:28:26,612][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:28:31,613][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:28:36,614][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:28:41,615][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:28:46,616][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:28:51,617][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:28:56,618][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:29:01,620][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:29:06,621][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:29:11,622][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:29:16,623][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:29:21,624][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:29:26,625][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:29:31,626][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:29:36,627][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:29:41,628][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:29:46,630][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:29:51,633][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:29:56,639][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:30:01,642][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:30:06,644][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:30:11,645][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:30:16,646][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:30:21,647][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:30:26,648][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:30:31,649][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:30:36,650][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:30:41,651][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:30:46,652][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:30:51,653][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:30:56,654][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:31:01,655][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:31:06,656][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:31:11,657][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:31:16,658][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:31:21,660][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:31:26,661][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:31:31,724][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:31:36,953][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:31:41,958][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:31:46,960][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:31:51,962][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:31:56,962][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:32:01,965][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:32:06,966][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:32:11,967][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:32:16,968][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:32:21,969][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:32:26,970][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:32:31,972][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:32:36,973][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:32:41,975][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:32:46,976][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:32:51,978][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:32:56,989][ INFO][PID:2612413] Periodic builder liveness probe: alive [2025-12-15 09:32:56,990][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:33:01,990][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:33:06,991][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:33:11,992][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:33:16,994][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:33:21,995][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:33:26,996][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:33:31,998][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:33:37,001][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:33:42,002][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:33:47,004][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:33:52,008][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:33:57,009][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:34:02,011][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:34:07,012][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:34:12,013][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:34:17,014][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:34:22,015][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:34:27,018][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:34:32,019][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:34:37,020][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:34:42,021][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:34:47,022][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:34:52,024][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:34:57,025][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:35:02,026][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:35:07,413][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:35:12,414][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:35:17,415][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:35:22,416][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:35:27,419][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:35:32,420][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:35:37,421][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:35:42,421][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:35:47,422][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:35:52,432][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:35:57,433][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:36:02,434][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:36:07,435][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:36:12,436][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:36:17,437][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:36:22,438][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:36:27,439][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:36:32,440][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:36:37,441][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:36:42,542][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:36:47,543][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:36:52,544][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:36:57,545][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:37:02,548][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:37:07,549][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:37:12,550][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:37:17,551][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:37:22,554][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:37:27,555][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:37:32,556][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:37:37,557][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:37:42,558][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:37:47,559][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:37:52,560][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:37:57,561][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:38:02,562][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:38:07,563][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:38:12,564][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:38:17,564][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:38:22,574][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:38:27,577][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:38:32,579][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:38:37,581][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:38:42,582][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:38:47,583][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:38:52,584][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:38:57,606][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:39:02,620][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:39:07,743][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:39:12,744][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:39:17,745][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:39:22,747][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:39:27,748][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:39:32,749][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:39:37,751][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:39:42,753][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:39:47,754][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:39:52,755][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:39:57,756][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:40:02,757][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:40:07,758][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:40:12,762][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:40:17,765][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:40:22,776][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:40:27,777][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:40:32,779][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:40:37,781][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:40:42,958][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:40:47,973][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:40:52,993][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:40:57,995][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:41:02,997][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:41:08,009][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:41:13,017][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:41:18,019][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:41:23,021][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:41:28,022][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:41:33,043][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:41:38,044][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:41:43,045][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:41:48,046][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:41:53,047][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:41:58,048][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:42:03,049][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:42:08,050][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:42:13,051][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:42:18,052][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:42:23,054][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:42:28,083][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:42:33,124][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:42:38,134][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:42:43,157][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:42:48,158][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:42:53,162][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:42:58,387][ INFO][PID:2612413] Periodic builder liveness probe: dead [2025-12-15 09:42:58,421][ INFO][PID:2612413] calling cancel callback, and waiting [2025-12-15 09:43:02,273][ INFO][PID:2612413] Cancel request succeeded out: success err: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c10d' (ED25519) to the list of known hosts. [2025-12-15 09:43:02,298][ ERROR][PID:2612413] Re-try request for task on 'ResallocHost, ticket_id=345902, hostname=2620:52:6:1161:dead:beef:cafe:c10d, name=vmhost_p09_02_prod_00556020_20251215_090639, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Resalloc ticket FAILED [2025-12-15 09:43:02,302][ INFO][PID:2612413] Releasing VM back to pool [2025-12-15 09:43:02,336][ INFO][PID:2612413] Retry #1 (on other host) [2025-12-15 09:43:02,361][ INFO][PID:2612413] Checking that /var/lib/copr/public_html/results/psimovec/scipy-1.10.0/fedora-41-ppc64le/repodata/repomd.xml exists [2025-12-15 09:43:02,368][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:43:02,371][ INFO][PID:2612413] VM allocation process starts [2025-12-15 09:43:02,452][ INFO][PID:2612413] Trying to allocate VM: ResallocHost, ticket_id=346265, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-12-15 09:43:05,616][ INFO][PID:2612413] Allocated host ResallocHost, ticket_id=346265, hostname=2620:52:6:1161:dead:beef:cafe:c18d, name=vmhost_p09_04_prod_00556543_20251215_093417, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-12-15 09:43:05,620][ INFO][PID:2612413] Allocating ssh connection to builder [2025-12-15 09:43:05,633][ INFO][PID:2612413] Checking that builder machine is OK [2025-12-15 09:44:05,706][ ERROR][PID:2612413] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c18d port 22: Connection timed out [2025-12-15 09:44:15,708][ ERROR][PID:2612413] Re-try request for task on 'ResallocHost, ticket_id=346265, hostname=2620:52:6:1161:dead:beef:cafe:c18d, name=vmhost_p09_04_prod_00556543_20251215_093417, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts [2025-12-15 09:44:15,708][ INFO][PID:2612413] Releasing VM back to pool [2025-12-15 09:44:15,726][ INFO][PID:2612413] Retry #2 (on other host) [2025-12-15 09:44:15,727][ INFO][PID:2612413] Checking that /var/lib/copr/public_html/results/psimovec/scipy-1.10.0/fedora-41-ppc64le/repodata/repomd.xml exists [2025-12-15 09:44:15,729][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:44:15,730][ INFO][PID:2612413] VM allocation process starts [2025-12-15 09:44:15,749][ INFO][PID:2612413] Trying to allocate VM: ResallocHost, ticket_id=346274, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-12-15 09:44:18,805][ INFO][PID:2612413] Allocated host ResallocHost, ticket_id=346274, hostname=140.211.168.14, name=copr_osuosl_p09_prod_00556400_20251215_092657, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-12-15 09:44:18,807][ INFO][PID:2612413] Allocating ssh connection to builder [2025-12-15 09:44:18,807][ INFO][PID:2612413] Checking that builder machine is OK [2025-12-15 09:44:20,572][ INFO][PID:2612413] Running remote command: copr-builder-ready fedora-41-ppc64le [2025-12-15 09:44:21,635][ INFO][PID:2612413] Red Hat subscription not needed for fedora-41-ppc64le Builder is ready to be used [2025-12-15 09:44:21,642][ INFO][PID:2612413] Filling build.info file with builder info [2025-12-15 09:44:21,673][ INFO][PID:2612413] Checking for cancel request [2025-12-15 09:44:21,682][ INFO][PID:2612413] Not re-notifying FE and msg buses for the new host. [2025-12-15 09:44:21,691][ INFO][PID:2612413] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9910409-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached [2025-12-15 09:44:23,321][ INFO][PID:2612413] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '140.211.168.14' (ED25519) to the list of known hosts. [2025-12-15 09:44:23,323][ INFO][PID:2612413] Downloading the builder-live.log file, attempt 1 [2025-12-15 09:44:23,344][ INFO][PID:2612413] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.14 copr-rpmbuild-log [2025-12-15 09:44:28,324][ INFO][PID:2612413] calling cancel callback, and waiting [2025-12-15 09:44:35,568][ INFO][PID:2612413] Cancel request succeeded out: success err: Warning: Permanently added '140.211.168.14' (ED25519) to the list of known hosts. [2025-12-15 09:44:35,571][ INFO][PID:2612413] Downloading results from builder [2025-12-15 09:44:35,582][ INFO][PID:2612413] rsyncing of mockbuilder@140.211.168.14:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/psimovec/scipy-1.10.0/fedora-41-ppc64le/09910409-bout++ started [2025-12-15 09:44:35,588][ INFO][PID:2612413] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@140.211.168.14:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/psimovec/scipy-1.10.0/fedora-41-ppc64le/09910409-bout++/ &> /var/lib/copr/public_html/results/psimovec/scipy-1.10.0/fedora-41-ppc64le/09910409-bout++/build-09910409.rsync.log [2025-12-15 09:44:37,212][ INFO][PID:2612413] rsyncing finished. [2025-12-15 09:44:37,217][ INFO][PID:2612413] Releasing VM back to pool [2025-12-15 09:44:37,234][ INFO][PID:2612413] Searching for 'success' file in resultdir [2025-12-15 09:44:37,240][ ERROR][PID:2612413] Build failed: Backend process error: No success file => build failure [2025-12-15 09:44:37,370][ INFO][PID:2612413] Finished build: id=9910409 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/psimovec/scipy-1.10.0 chroot=fedora-41-ppc64le [2025-12-15 09:44:37,816][ ERROR][PID:2612413] Backend process error: No success file => build failure [2025-12-15 09:44:37,818][ INFO][PID:2612413] Worker failed build, took 16.143149852752686 [2025-12-15 09:44:37,923][ INFO][PID:2612413] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "psimovec", "project_name": "scipy-1.10.0", "project_dirname": "scipy-1.10.0", "submitter": "https://src.fedoraproject.org/user/davidsch", "ended_on": 1765791877.8183765, "started_on": 1765791861.6752267, "submitted_on": null, "status": 0, "chroot": "fedora-41-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9910409-fedora-41-ppc64le", "build_id": 9910409, "package_name": "bout++", "package_version": "5.2.0-4", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/psimovec/scipy-1.10.0/bout++", "git_hash": "0a82a212344b82b2f93fb5fce0a9fa6962685e50", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "psimovec/scipy-1.10.0--https://src.fedoraproject.org/user/davidsch", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/psimovec/scipy-1.10.0/fedora-41-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/", "id": "http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch", "name": "Additional repo http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch" } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/psimovec/scipy-1.10.0", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/psimovec/scipy-1.10.0", "result_dir": "09910409-bout++", "built_packages": "", "id": 9910409, "mockchain_macros": { "copr_username": "psimovec", "copr_projectname": "scipy-1.10.0", "vendor": "Fedora Project COPR (psimovec/scipy-1.10.0)" } } ] } [2025-12-15 09:44:40,366][ INFO][PID:2612413] Sending fedora-messaging bus message in build.end [2025-12-15 09:44:41,162][ INFO][PID:2612413] Compressing /var/lib/copr/public_html/results/psimovec/scipy-1.10.0/fedora-41-ppc64le/09910409-bout++/builder-live.log by gzip [2025-12-15 09:44:41,165][ INFO][PID:2612413] Running command 'gzip /var/lib/copr/public_html/results/psimovec/scipy-1.10.0/fedora-41-ppc64le/09910409-bout++/builder-live.log' as PID 2801038 [2025-12-15 09:44:41,193][ INFO][PID:2612413] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/psimovec/scipy-1.10.0/fedora-41-ppc64le/09910409-bout++/builder-live.log) [2025-12-15 09:44:41,199][ INFO][PID:2612413] Compressing /var/lib/copr/public_html/results/psimovec/scipy-1.10.0/fedora-41-ppc64le/09910409-bout++/backend.log by gzip [2025-12-15 09:44:41,200][ INFO][PID:2612413] Running command 'gzip /var/lib/copr/public_html/results/psimovec/scipy-1.10.0/fedora-41-ppc64le/09910409-bout++/backend.log' as PID 2801045