[2026-02-10 12:33:53,648][ INFO][PID:3839208] Marking build as starting [2026-02-10 12:33:53,756][ INFO][PID:3839208] Checking that /var/lib/copr/public_html/results/@fedora-llvm-team/llvm22/fedora-44-ppc64le/repodata/repomd.xml exists [2026-02-10 12:33:54,182][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:33:54,184][ INFO][PID:3839208] VM allocation process starts [2026-02-10 12:33:54,197][ INFO][PID:3839208] Trying to allocate VM: ResallocHost, ticket_id=1255709, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-02-10 12:33:57,215][ INFO][PID:3839208] Allocated host ResallocHost, ticket_id=1255709, hostname=2620:52:6:1161:dead:beef:cafe:c14a, name=vmhost_p09_03_prod_02123346_20260210_120402, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-02-10 12:33:57,220][ INFO][PID:3839208] Allocating ssh connection to builder [2026-02-10 12:33:57,224][ INFO][PID:3839208] Checking that builder machine is OK [2026-02-10 12:33:58,308][ INFO][PID:3839208] Running remote command: copr-builder-ready fedora-44-ppc64le [2026-02-10 12:33:58,861][ INFO][PID:3839208] Red Hat subscription not needed for fedora-44-ppc64le Builder is ready to be used [2026-02-10 12:33:58,862][ INFO][PID:3839208] Filling build.info file with builder info [2026-02-10 12:33:58,863][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:33:58,868][ INFO][PID:3839208] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@fedora-llvm-team", "project_name": "llvm22", "project_dirname": "llvm22", "submitter": "tuliom", "ended_on": null, "started_on": 1770726838.8685806, "submitted_on": null, "status": 3, "chroot": "fedora-44-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "10112092-fedora-44-ppc64le", "build_id": 10112092, "package_name": "llvm", "package_version": "22.1.0~rc3-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@fedora-llvm-team/llvm22/llvm", "git_hash": "d610a9c1c2090ca6f70825c67325ac2beb548293", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@fedora-llvm-team/llvm22--tuliom", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm22/fedora-44-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@fedora-llvm-team/llvm22", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm22", "result_dir": "10112092-llvm", "built_packages": "", "id": 10112092, "mockchain_macros": { "copr_username": "@fedora-llvm-team", "copr_projectname": "llvm22", "vendor": "Fedora Project COPR (@fedora-llvm-team/llvm22)" } } ] } [2026-02-10 12:33:58,952][ INFO][PID:3839208] Sending fedora-messaging bus message in build.start [2026-02-10 12:33:59,937][ INFO][PID:3839208] Sending fedora-messaging bus message in chroot.start [2026-02-10 12:34:00,061][ INFO][PID:3839208] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10112092-fedora-44-ppc64le --chroot fedora-44-ppc64le --detached [2026-02-10 12:34:01,137][ INFO][PID:3839208] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c14a' (ED25519) to the list of known hosts. [2026-02-10 12:34:01,138][ INFO][PID:3839208] Downloading the builder-live.log file, attempt 1 [2026-02-10 12:34:01,143][ INFO][PID:3839208] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c14a copr-rpmbuild-log [2026-02-10 12:34:06,157][ INFO][PID:3839208] Periodic builder liveness probe: alive [2026-02-10 12:34:06,160][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:34:11,163][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:34:16,170][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:34:21,172][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:34:26,175][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:34:31,178][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:34:36,180][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:34:41,181][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:34:46,186][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:34:51,187][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:34:56,190][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:35:01,261][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:35:06,271][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:35:11,273][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:35:16,275][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:35:21,280][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:35:26,284][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:35:31,287][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:35:36,289][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:35:41,290][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:35:46,294][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:35:51,299][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:35:56,303][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:36:01,304][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:36:06,307][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:36:11,310][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:36:16,316][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:36:21,319][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:36:26,321][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:36:31,325][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:36:36,329][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:36:41,330][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:36:46,334][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:36:51,337][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:36:56,342][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:37:01,346][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:37:06,349][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:37:11,359][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:37:16,371][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:37:21,375][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:37:26,379][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:37:31,381][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:37:36,383][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:37:41,387][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:37:46,389][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:37:51,391][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:37:56,392][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:38:01,497][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:38:06,502][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:38:11,557][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:38:16,568][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:38:21,576][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:38:26,585][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:38:31,587][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:38:36,601][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:38:41,620][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:38:46,631][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:38:51,713][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:38:56,715][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:39:01,738][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:39:06,741][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:39:11,744][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:39:16,747][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:39:21,780][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:39:26,787][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:39:31,838][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:39:36,840][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:39:41,843][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:39:46,845][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:39:51,847][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:39:56,862][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:40:01,930][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:40:07,064][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:40:12,069][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:40:17,075][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:40:22,076][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:40:27,078][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:40:32,082][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:40:37,089][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:40:42,091][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:40:47,096][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:40:52,099][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:40:57,104][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:41:02,108][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:41:07,112][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:41:12,114][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:41:17,115][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:41:22,116][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:41:27,117][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:41:32,120][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:41:37,122][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:41:42,126][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:41:47,131][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:41:52,132][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:41:57,137][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:42:02,144][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:42:07,146][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:42:12,152][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:42:17,159][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:42:22,161][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:42:27,164][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:42:32,168][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:42:37,174][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:42:42,175][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:42:47,178][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:42:52,183][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:42:57,348][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:43:02,352][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:43:07,361][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:43:12,367][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:43:17,374][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:43:22,377][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:43:27,379][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:43:32,381][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:43:37,389][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:43:42,398][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:43:47,406][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:43:52,411][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:43:57,421][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:44:02,424][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:44:07,500][ INFO][PID:3839208] Periodic builder liveness probe: alive [2026-02-10 12:44:07,502][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:44:12,506][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:44:17,521][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:44:22,534][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:44:27,537][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:44:32,540][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:44:37,543][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:44:42,555][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:44:47,561][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:44:52,563][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:44:57,568][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:45:02,577][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:45:07,629][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:45:12,700][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:45:17,708][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:45:22,712][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:45:27,713][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:45:32,715][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:45:37,717][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:45:42,723][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:45:47,724][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:45:52,731][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:45:57,737][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:46:02,739][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:46:07,741][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:46:12,742][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:46:17,769][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:46:22,773][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:46:27,780][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:46:32,781][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:46:37,785][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:46:42,786][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:46:47,821][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:46:52,829][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:46:57,833][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:47:02,836][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:47:07,837][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:47:12,857][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:47:17,959][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:47:22,963][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:47:27,965][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:47:32,967][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:47:37,969][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:47:42,972][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:47:48,046][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:47:53,048][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:47:58,049][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:48:03,050][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:48:08,052][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:48:13,094][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:48:18,099][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:48:23,213][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:48:28,318][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:48:33,326][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:48:38,327][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:48:43,328][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:48:48,329][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:48:53,330][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:48:58,332][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:49:03,333][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:49:08,346][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:49:13,348][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:49:18,349][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:49:23,390][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:49:28,446][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:49:33,459][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:49:38,461][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:49:43,465][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:49:48,466][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:49:53,468][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:49:58,471][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:50:03,473][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:50:08,474][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:50:13,478][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:50:18,482][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:50:23,483][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:50:28,484][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:50:33,485][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:50:38,520][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:50:43,522][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:50:48,524][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:50:53,525][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:50:58,526][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:51:03,530][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:51:08,531][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:51:13,532][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:51:18,533][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:51:23,534][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:51:28,536][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:51:33,537][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:51:38,617][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:51:43,619][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:51:48,620][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:51:53,621][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:51:58,622][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:52:03,623][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:52:08,649][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:52:13,651][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:52:18,654][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:52:23,655][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:52:28,832][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:52:33,835][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:52:38,836][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:52:44,033][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:52:49,035][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:52:54,036][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:52:59,038][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:53:04,040][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:53:09,042][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:53:14,053][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:53:19,286][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:53:24,288][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:53:29,289][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:53:34,320][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:53:39,322][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:53:44,323][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:53:49,398][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:53:54,400][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:53:59,401][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:54:04,402][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:54:09,455][ INFO][PID:3839208] Periodic builder liveness probe: alive [2026-02-10 12:54:09,456][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:54:14,457][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:54:19,459][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:54:24,460][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:54:29,462][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:54:34,475][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:54:39,477][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:54:44,478][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:54:49,482][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:54:54,753][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:54:59,756][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:55:04,757][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:55:09,758][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:55:14,760][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:55:19,761][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:55:24,764][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:55:29,766][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:55:34,845][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:55:39,847][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:55:44,848][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:55:49,849][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:55:54,850][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:55:59,851][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:56:04,852][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:56:09,853][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:56:14,854][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:56:19,855][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:56:24,856][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:56:29,857][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:56:34,859][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:56:39,860][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:56:44,861][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:56:49,909][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:56:54,910][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:56:59,911][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:57:04,912][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:57:09,914][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:57:14,916][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:57:19,919][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:57:24,960][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:57:29,970][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:57:34,972][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:57:39,974][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:57:44,975][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:57:49,975][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:57:54,980][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:58:00,069][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:58:05,070][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:58:10,071][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:58:15,072][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:58:20,074][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:58:25,075][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:58:30,076][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:58:35,080][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:58:40,081][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:58:45,083][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:58:50,084][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:58:55,085][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:59:00,086][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:59:05,087][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:59:10,121][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:59:15,122][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:59:15,376][ ERROR][PID:3839208] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-10 12:59:20,123][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:59:25,124][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:59:25,377][WARNING][PID:3839208] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-10 12:59:30,125][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:59:35,126][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:59:35,378][ INFO][PID:3839208] Downloading the builder-live.log file, attempt 2 [2026-02-10 12:59:35,485][ INFO][PID:3839208] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c14a copr-rpmbuild-log [2026-02-10 12:59:40,127][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:59:45,128][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:59:50,129][ INFO][PID:3839208] Checking for cancel request [2026-02-10 12:59:55,130][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:00:00,131][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:00:05,132][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:00:10,133][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:00:15,185][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:00:20,252][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:00:25,256][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:00:30,260][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:00:35,264][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:00:40,270][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:00:45,271][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:00:50,272][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:00:55,275][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:01:00,277][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:01:05,280][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:01:10,281][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:01:15,282][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:01:20,284][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:01:25,287][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:01:30,297][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:01:35,421][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:01:40,423][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:01:45,425][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:01:50,429][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:01:55,431][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:02:00,432][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:02:05,453][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:02:10,498][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:02:15,741][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:02:19,418][ ERROR][PID:3839208] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-10 13:02:20,750][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:02:25,754][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:02:29,471][WARNING][PID:3839208] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-10 13:02:30,755][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:02:35,756][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:02:39,472][ INFO][PID:3839208] Downloading the builder-live.log file, attempt 3 [2026-02-10 13:02:39,584][ INFO][PID:3839208] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c14a copr-rpmbuild-log [2026-02-10 13:02:40,836][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:02:45,838][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:02:50,839][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:02:55,840][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:03:00,841][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:03:05,842][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:03:10,844][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:03:15,882][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:03:21,262][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:03:26,264][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:03:31,265][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:03:36,337][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:03:39,651][ ERROR][PID:3839208] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-10 13:03:41,338][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:03:46,339][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:03:49,696][WARNING][PID:3839208] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-10 13:03:51,340][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:03:56,341][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:03:59,697][ INFO][PID:3839208] Downloading the builder-live.log file, attempt 4 [2026-02-10 13:03:59,698][ INFO][PID:3839208] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c14a copr-rpmbuild-log [2026-02-10 13:04:01,342][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:04:06,343][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:04:11,349][ INFO][PID:3839208] Periodic builder liveness probe: dead [2026-02-10 13:04:11,350][ INFO][PID:3839208] calling cancel callback, and waiting [2026-02-10 13:04:59,766][ ERROR][PID:3839208] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-10 13:05:09,798][ ERROR][PID:3839208] Giving up for unstable SSH, failures: 1770728365.377241, 1770728549.4714844, 1770728629.696259, 1770728709.7985632 [2026-02-10 13:05:11,419][ ERROR][PID:3839208] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c14a port 22: Connection timed out [2026-02-10 13:05:34,210][WARNING][PID:3839208] Can't cancel build out: err: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c14a' (ED25519) to the list of known hosts. Traceback (most recent call last): File "/usr/bin/copr-rpmbuild-cancel", line 85, in _main() ~~~~~^^ File "/usr/bin/copr-rpmbuild-cancel", line 69, in _main with open(pidfile, "r") as pidfd: ~~~~^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/copr-rpmbuild/pid' [2026-02-10 13:05:34,210][ ERROR][PID:3839208] Re-try request for task on 'ResallocHost, ticket_id=1255709, hostname=2620:52:6:1161:dead:beef:cafe:c14a, name=vmhost_p09_03_prod_02123346_20260210_120402, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Resalloc ticket FAILED [2026-02-10 13:05:34,211][ INFO][PID:3839208] Releasing VM back to pool [2026-02-10 13:05:34,222][ INFO][PID:3839208] Retry #1 (on other host) [2026-02-10 13:05:34,222][ INFO][PID:3839208] Checking that /var/lib/copr/public_html/results/@fedora-llvm-team/llvm22/fedora-44-ppc64le/repodata/repomd.xml exists [2026-02-10 13:05:34,459][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:05:34,461][ INFO][PID:3839208] VM allocation process starts [2026-02-10 13:05:34,474][ INFO][PID:3839208] Trying to allocate VM: ResallocHost, ticket_id=1256070, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-02-10 13:05:37,488][ INFO][PID:3839208] Allocated host ResallocHost, ticket_id=1256070, hostname=2620:52:6:1161:dead:beef:cafe:c153, name=vmhost_p09_03_prod_02125881_20260210_124753, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-02-10 13:05:37,489][ INFO][PID:3839208] Allocating ssh connection to builder [2026-02-10 13:05:37,489][ INFO][PID:3839208] Checking that builder machine is OK [2026-02-10 13:05:39,943][ INFO][PID:3839208] Running remote command: copr-builder-ready fedora-44-ppc64le [2026-02-10 13:05:42,003][ INFO][PID:3839208] Red Hat subscription not needed for fedora-44-ppc64le Builder is ready to be used [2026-02-10 13:05:42,003][ INFO][PID:3839208] Filling build.info file with builder info [2026-02-10 13:05:42,061][ INFO][PID:3839208] Checking for cancel request [2026-02-10 13:05:42,062][ INFO][PID:3839208] Not re-notifying FE and msg buses for the new host. [2026-02-10 13:05:42,063][ INFO][PID:3839208] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10112092-fedora-44-ppc64le --chroot fedora-44-ppc64le --detached [2026-02-10 13:05:44,894][ INFO][PID:3839208] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c153' (ED25519) to the list of known hosts. [2026-02-10 13:05:44,895][ INFO][PID:3839208] Downloading the builder-live.log file, attempt 1 [2026-02-10 13:05:44,896][ INFO][PID:3839208] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c153 copr-rpmbuild-log [2026-02-10 13:05:49,895][ INFO][PID:3839208] calling cancel callback, and waiting [2026-02-10 13:05:54,924][ INFO][PID:3839208] Cancel request succeeded out: success err: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c153' (ED25519) to the list of known hosts. [2026-02-10 13:05:54,925][ INFO][PID:3839208] Downloading results from builder [2026-02-10 13:05:54,925][ INFO][PID:3839208] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c153]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@fedora-llvm-team/llvm22/fedora-44-ppc64le/10112092-llvm started [2026-02-10 13:05:54,926][ INFO][PID:3839208] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c153]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@fedora-llvm-team/llvm22/fedora-44-ppc64le/10112092-llvm/ &> /var/lib/copr/public_html/results/@fedora-llvm-team/llvm22/fedora-44-ppc64le/10112092-llvm/build-10112092.rsync.log [2026-02-10 13:05:57,254][ INFO][PID:3839208] rsyncing finished. [2026-02-10 13:05:57,255][ INFO][PID:3839208] Releasing VM back to pool [2026-02-10 13:05:57,295][ INFO][PID:3839208] Searching for 'success' file in resultdir [2026-02-10 13:05:57,295][ ERROR][PID:3839208] Build failed: Backend process error: No success file => build failure [2026-02-10 13:05:57,296][ INFO][PID:3839208] Finished build: id=10112092 failed=True timeout=108000 destdir=/var/lib/copr/public_html/results/@fedora-llvm-team/llvm22 chroot=fedora-44-ppc64le [2026-02-10 13:05:57,297][ ERROR][PID:3839208] Backend process error: No success file => build failure [2026-02-10 13:05:57,298][ INFO][PID:3839208] Worker failed build, took 15.235503435134888 [2026-02-10 13:05:57,298][ INFO][PID:3839208] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@fedora-llvm-team", "project_name": "llvm22", "project_dirname": "llvm22", "submitter": "tuliom", "ended_on": 1770728757.2979827, "started_on": 1770728742.0624793, "submitted_on": null, "status": 0, "chroot": "fedora-44-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "10112092-fedora-44-ppc64le", "build_id": 10112092, "package_name": "llvm", "package_version": "22.1.0~rc3-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@fedora-llvm-team/llvm22/llvm", "git_hash": "d610a9c1c2090ca6f70825c67325ac2beb548293", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@fedora-llvm-team/llvm22--tuliom", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm22/fedora-44-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@fedora-llvm-team/llvm22", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm22", "result_dir": "10112092-llvm", "built_packages": "", "id": 10112092, "mockchain_macros": { "copr_username": "@fedora-llvm-team", "copr_projectname": "llvm22", "vendor": "Fedora Project COPR (@fedora-llvm-team/llvm22)" } } ] } [2026-02-10 13:05:57,352][ INFO][PID:3839208] Sending fedora-messaging bus message in build.end [2026-02-10 13:05:57,382][ INFO][PID:3839208] Compressing /var/lib/copr/public_html/results/@fedora-llvm-team/llvm22/fedora-44-ppc64le/10112092-llvm/builder-live.log by gzip [2026-02-10 13:05:57,383][ INFO][PID:3839208] Running command 'gzip /var/lib/copr/public_html/results/@fedora-llvm-team/llvm22/fedora-44-ppc64le/10112092-llvm/builder-live.log' as PID 4087845 [2026-02-10 13:05:57,441][ INFO][PID:3839208] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@fedora-llvm-team/llvm22/fedora-44-ppc64le/10112092-llvm/builder-live.log) [2026-02-10 13:05:57,442][ INFO][PID:3839208] Compressing /var/lib/copr/public_html/results/@fedora-llvm-team/llvm22/fedora-44-ppc64le/10112092-llvm/backend.log by gzip [2026-02-10 13:05:57,443][ INFO][PID:3839208] Running command 'gzip /var/lib/copr/public_html/results/@fedora-llvm-team/llvm22/fedora-44-ppc64le/10112092-llvm/backend.log' as PID 4087863