[2025-12-18 16:09:37,106][ INFO][PID:1873708] Marking build as starting [2025-12-18 16:09:37,185][ INFO][PID:1873708] Checking that /var/lib/copr/public_html/results/alexsaezm/golang-1.26/fedora-rawhide-ppc64le/repodata/repomd.xml exists [2025-12-18 16:09:37,237][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:09:37,238][ INFO][PID:1873708] VM allocation process starts [2025-12-18 16:09:37,259][ INFO][PID:1873708] Trying to allocate VM: ResallocHost, ticket_id=426009, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-12-18 16:09:42,267][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:09:47,269][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:09:52,270][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:09:57,278][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:10:02,286][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:10:07,288][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:10:12,289][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:10:17,293][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:10:22,295][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:10:27,296][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:10:32,297][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:10:37,299][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:10:42,300][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:10:47,301][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:10:52,305][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:10:57,307][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:11:02,308][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:11:07,310][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:11:12,315][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:11:17,317][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:11:22,319][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:11:27,322][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:11:32,326][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:11:37,327][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:11:42,330][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:11:47,335][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:11:52,336][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:11:57,341][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:12:02,343][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:12:07,344][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:12:12,346][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:12:17,347][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:12:22,348][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:12:27,349][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:12:32,350][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:12:37,354][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:12:42,356][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:12:47,358][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:12:52,360][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:12:57,363][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:13:02,366][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:13:07,367][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:13:12,368][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:13:17,369][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:13:22,370][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:13:27,373][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:13:32,374][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:13:37,377][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:13:42,380][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:13:47,389][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:13:52,391][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:13:57,395][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:14:02,398][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:14:07,399][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:14:12,401][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:14:17,406][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:14:22,407][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:14:27,409][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:14:32,410][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:14:37,411][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:14:42,412][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:14:47,413][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:14:52,415][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:14:57,418][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:15:02,421][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:15:07,422][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:15:12,424][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:15:17,425][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:15:22,428][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:15:27,430][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:15:32,431][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:15:37,432][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:15:42,434][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:15:47,439][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:15:52,443][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:15:57,444][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:16:02,449][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:16:07,450][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:16:12,451][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:16:17,452][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:16:22,454][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:16:27,455][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:16:32,457][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:16:37,459][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:16:42,460][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:16:47,461][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:16:52,462][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:16:57,464][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:17:02,466][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:17:07,468][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:17:12,480][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:17:17,481][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:17:22,483][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:17:27,485][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:17:32,486][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:17:37,488][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:17:42,490][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:17:47,493][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:17:52,494][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:17:57,495][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:18:02,503][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:18:07,507][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:18:12,509][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:18:17,518][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:18:22,519][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:18:27,521][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:18:32,524][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:18:37,525][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:18:42,526][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:18:47,527][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:18:52,530][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:18:57,532][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:19:02,533][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:19:07,535][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:19:12,536][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:19:17,537][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:19:22,540][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:19:27,541][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:19:32,544][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:19:37,545][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:19:42,547][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:19:47,552][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:19:52,553][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:19:55,551][ INFO][PID:1873708] Allocated host ResallocHost, ticket_id=426009, hostname=2620:52:6:1161:dead:beef:cafe:c189, name=vmhost_p09_04_prod_00680667_20251218_161426, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-12-18 16:19:55,552][ INFO][PID:1873708] Allocating ssh connection to builder [2025-12-18 16:19:55,553][ INFO][PID:1873708] Checking that builder machine is OK [2025-12-18 16:19:56,394][ INFO][PID:1873708] Running remote command: copr-builder-ready fedora-rawhide-ppc64le [2025-12-18 16:19:56,983][ INFO][PID:1873708] Red Hat subscription not needed for fedora-rawhide-ppc64le Builder is ready to be used [2025-12-18 16:19:56,983][ INFO][PID:1873708] Filling build.info file with builder info [2025-12-18 16:19:57,055][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:19:57,065][ INFO][PID:1873708] Sending build state back to frontend: { "builds": [ { "timeout": 115200, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "alexsaezm", "project_name": "golang-1.26", "project_dirname": "golang-1.26", "submitter": "alexsaezm", "ended_on": null, "started_on": 1766074797.0653825, "submitted_on": null, "status": 3, "chroot": "fedora-rawhide-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9931589-fedora-rawhide-ppc64le", "build_id": 9931589, "package_name": "golang-x-tools", "package_version": "1:0.36.0-2", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/alexsaezm/golang-1.26/golang-x-tools", "git_hash": "d35611109c101bd3ec3ed8ccfc9d84467cb8bf4c", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "alexsaezm/golang-1.26--alexsaezm", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/alexsaezm/golang-1.26/fedora-rawhide-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/alexsaezm/golang-1.26", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/alexsaezm/golang-1.26", "result_dir": "09931589-golang-x-tools", "built_packages": "", "id": 9931589, "mockchain_macros": { "copr_username": "alexsaezm", "copr_projectname": "golang-1.26", "vendor": "Fedora Project COPR (alexsaezm/golang-1.26)" } } ] } [2025-12-18 16:19:57,230][ INFO][PID:1873708] Sending fedora-messaging bus message in build.start [2025-12-18 16:19:58,107][ INFO][PID:1873708] Sending fedora-messaging bus message in chroot.start [2025-12-18 16:19:58,139][ INFO][PID:1873708] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9931589-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached [2025-12-18 16:19:59,198][ INFO][PID:1873708] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c189' (ED25519) to the list of known hosts. [2025-12-18 16:19:59,200][ INFO][PID:1873708] Downloading the builder-live.log file, attempt 1 [2025-12-18 16:19:59,228][ INFO][PID:1873708] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c189 copr-rpmbuild-log [2025-12-18 16:20:04,216][ INFO][PID:1873708] Periodic builder liveness probe: alive [2025-12-18 16:20:04,217][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:20:09,218][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:20:14,219][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:20:19,221][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:20:24,223][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:20:29,225][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:20:34,226][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:20:39,229][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:20:44,233][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:20:49,234][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:20:54,235][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:20:59,236][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:21:04,238][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:21:09,239][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:21:14,242][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:21:19,244][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:21:24,245][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:21:29,246][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:21:34,248][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:21:39,249][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:21:44,250][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:21:49,252][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:21:54,254][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:21:59,257][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:22:04,263][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:22:09,264][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:22:14,265][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:22:19,266][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:22:24,268][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:22:29,269][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:22:34,271][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:22:39,272][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:22:44,274][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:22:49,275][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:22:54,276][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:22:59,277][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:23:04,279][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:23:09,281][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:23:14,282][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:23:19,283][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:23:24,285][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:23:29,289][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:23:34,291][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:23:39,292][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:23:44,297][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:23:49,298][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:23:54,305][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:23:59,307][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:24:04,308][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:24:09,310][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:24:14,311][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:24:19,313][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:24:24,315][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:24:29,318][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:24:34,319][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:24:34,370][ ERROR][PID:1873708] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-18 16:24:39,321][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:24:44,322][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:24:44,371][WARNING][PID:1873708] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-18 16:24:49,323][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:24:54,325][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:24:54,373][ INFO][PID:1873708] Downloading the builder-live.log file, attempt 2 [2025-12-18 16:24:54,374][ INFO][PID:1873708] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c189 copr-rpmbuild-log [2025-12-18 16:24:59,326][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:25:04,327][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:25:09,346][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:25:14,347][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:25:19,348][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:25:24,349][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:25:29,397][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:25:34,398][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:25:39,400][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:25:44,428][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:25:49,429][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:25:54,431][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:25:54,484][ ERROR][PID:1873708] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-18 16:25:59,432][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:26:04,433][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:26:04,501][WARNING][PID:1873708] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-18 16:26:09,435][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:26:14,436][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:26:14,502][ INFO][PID:1873708] Downloading the builder-live.log file, attempt 3 [2025-12-18 16:26:14,504][ INFO][PID:1873708] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c189 copr-rpmbuild-log [2025-12-18 16:26:19,437][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:26:24,438][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:26:29,440][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:26:34,441][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:26:39,443][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:26:44,444][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:26:49,445][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:26:54,449][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:26:59,460][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:27:04,461][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:27:09,462][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:27:14,463][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:27:14,572][ ERROR][PID:1873708] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-18 16:27:19,465][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:27:24,466][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:27:24,575][WARNING][PID:1873708] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-18 16:27:29,467][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:27:34,469][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:27:34,576][ INFO][PID:1873708] Downloading the builder-live.log file, attempt 4 [2025-12-18 16:27:34,577][ INFO][PID:1873708] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c189 copr-rpmbuild-log [2025-12-18 16:27:39,470][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:27:44,827][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:27:49,828][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:27:54,829][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:27:59,830][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:28:04,831][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:28:09,832][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:28:14,833][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:28:19,835][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:28:24,838][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:28:29,839][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:28:34,641][ ERROR][PID:1873708] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-18 16:28:34,850][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:28:39,851][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:28:44,642][ ERROR][PID:1873708] Giving up for unstable SSH, failures: 1766075084.3713615, 1766075164.5018988, 1766075244.575311, 1766075324.6424336 [2025-12-18 16:28:44,643][ ERROR][PID:1873708] Re-try request for task on 'ResallocHost, ticket_id=426009, hostname=2620:52:6:1161:dead:beef:cafe:c189, name=vmhost_p09_04_prod_00680667_20251218_161426, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-12-18 16:28:44,644][ INFO][PID:1873708] Releasing VM back to pool [2025-12-18 16:28:44,661][ INFO][PID:1873708] Retry #1 (on other host) [2025-12-18 16:28:44,662][ INFO][PID:1873708] Checking that /var/lib/copr/public_html/results/alexsaezm/golang-1.26/fedora-rawhide-ppc64le/repodata/repomd.xml exists [2025-12-18 16:28:44,724][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:28:44,726][ INFO][PID:1873708] VM allocation process starts [2025-12-18 16:28:44,737][ INFO][PID:1873708] Trying to allocate VM: ResallocHost, ticket_id=426368, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-12-18 16:28:49,739][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:28:54,740][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:28:59,742][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:29:04,743][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:29:09,744][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:29:14,745][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:29:19,746][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:29:24,748][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:29:29,842][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:29:34,963][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:29:39,964][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:29:45,835][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:29:50,836][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:29:55,837][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:30:00,839][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:30:05,840][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:30:10,852][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:30:15,853][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:30:20,862][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:30:25,875][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:30:30,878][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:30:35,880][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:30:40,881][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:30:45,884][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:30:50,885][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:30:55,886][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:31:00,899][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:31:05,927][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:31:11,053][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:31:16,106][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:31:21,357][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:31:26,377][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:31:31,602][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:31:36,610][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:31:41,715][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:31:46,810][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:31:51,812][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:31:56,857][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:32:01,873][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:32:02,892][ INFO][PID:1873708] Allocated host ResallocHost, ticket_id=426368, hostname=2620:52:6:1161:dead:beef:cafe:c18d, name=vmhost_p09_04_prod_00681395_20251218_162614, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-12-18 16:32:02,893][ INFO][PID:1873708] Allocating ssh connection to builder [2025-12-18 16:32:02,893][ INFO][PID:1873708] Checking that builder machine is OK [2025-12-18 16:32:03,979][ INFO][PID:1873708] Running remote command: copr-builder-ready fedora-rawhide-ppc64le [2025-12-18 16:32:04,584][ INFO][PID:1873708] Red Hat subscription not needed for fedora-rawhide-ppc64le Builder is ready to be used [2025-12-18 16:32:04,584][ INFO][PID:1873708] Filling build.info file with builder info [2025-12-18 16:32:04,593][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:32:04,594][ INFO][PID:1873708] Not re-notifying FE and msg buses for the new host. [2025-12-18 16:32:04,594][ INFO][PID:1873708] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9931589-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached [2025-12-18 16:32:05,883][ INFO][PID:1873708] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c18d' (ED25519) to the list of known hosts. [2025-12-18 16:32:05,884][ INFO][PID:1873708] Downloading the builder-live.log file, attempt 1 [2025-12-18 16:32:05,885][ INFO][PID:1873708] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18d copr-rpmbuild-log [2025-12-18 16:32:10,892][ INFO][PID:1873708] Periodic builder liveness probe: alive [2025-12-18 16:32:10,893][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:32:15,894][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:32:20,895][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:32:25,896][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:32:30,897][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:32:35,898][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:32:40,899][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:32:45,900][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:32:50,901][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:32:55,902][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:33:00,904][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:33:05,905][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:33:10,906][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:33:15,907][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:33:20,908][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:33:25,909][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:33:30,910][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:33:35,911][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:33:40,938][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:33:45,941][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:33:50,942][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:33:55,943][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:34:01,018][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:34:06,019][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:34:11,021][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:34:16,023][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:34:21,024][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:34:26,025][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:34:31,026][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:34:36,028][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:34:41,029][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:34:46,030][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:34:51,031][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:34:56,033][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:35:01,034][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:35:06,035][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:35:11,036][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:35:16,037][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:35:21,038][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:35:26,039][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:35:31,041][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:35:34,411][ ERROR][PID:1873708] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-18 16:35:36,042][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:35:41,043][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:35:44,450][WARNING][PID:1873708] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-18 16:35:46,044][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:35:51,045][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:35:54,452][ INFO][PID:1873708] Downloading the builder-live.log file, attempt 2 [2025-12-18 16:35:54,453][ INFO][PID:1873708] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18d copr-rpmbuild-log [2025-12-18 16:35:56,046][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:36:01,048][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:36:06,051][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:36:11,054][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:36:16,056][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:36:21,057][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:36:26,058][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:36:31,059][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:36:36,060][ INFO][PID:1873708] Checking for cancel request [2025-12-18 16:36:37,181][ INFO][PID:1873708] Downloading results from builder [2025-12-18 16:36:37,182][ INFO][PID:1873708] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18d]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/alexsaezm/golang-1.26/fedora-rawhide-ppc64le/09931589-golang-x-tools started [2025-12-18 16:36:37,183][ INFO][PID:1873708] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18d]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/alexsaezm/golang-1.26/fedora-rawhide-ppc64le/09931589-golang-x-tools/ &> /var/lib/copr/public_html/results/alexsaezm/golang-1.26/fedora-rawhide-ppc64le/09931589-golang-x-tools/build-09931589.rsync.log [2025-12-18 16:37:37,363][ INFO][PID:1873708] rsyncing finished. [2025-12-18 16:37:37,364][ ERROR][PID:1873708] Failed to download data from builder due to rsync error, see the rsync log file for details. [2025-12-18 16:37:37,366][ ERROR][PID:1873708] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2025-12-18 16:37:47,367][ INFO][PID:1873708] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18d]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/alexsaezm/golang-1.26/fedora-rawhide-ppc64le/09931589-golang-x-tools started [2025-12-18 16:37:47,368][ INFO][PID:1873708] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18d]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/alexsaezm/golang-1.26/fedora-rawhide-ppc64le/09931589-golang-x-tools/ &> /var/lib/copr/public_html/results/alexsaezm/golang-1.26/fedora-rawhide-ppc64le/09931589-golang-x-tools/build-09931589.rsync.log [2025-12-18 16:37:48,394][ INFO][PID:1873708] rsyncing finished. [2025-12-18 16:37:48,394][ INFO][PID:1873708] Releasing VM back to pool [2025-12-18 16:37:48,406][ INFO][PID:1873708] Searching for 'success' file in resultdir [2025-12-18 16:37:48,407][ ERROR][PID:1873708] Build failed: Backend process error: No success file => build failure [2025-12-18 16:37:48,408][ INFO][PID:1873708] Finished build: id=9931589 failed=True timeout=115200 destdir=/var/lib/copr/public_html/results/alexsaezm/golang-1.26 chroot=fedora-rawhide-ppc64le [2025-12-18 16:37:48,410][ ERROR][PID:1873708] Backend process error: No success file => build failure [2025-12-18 16:37:48,416][ INFO][PID:1873708] Worker failed build, took 343.822069644928 [2025-12-18 16:37:48,417][ INFO][PID:1873708] Sending build state back to frontend: { "builds": [ { "timeout": 115200, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "alexsaezm", "project_name": "golang-1.26", "project_dirname": "golang-1.26", "submitter": "alexsaezm", "ended_on": 1766075868.4161654, "started_on": 1766075524.5940957, "submitted_on": null, "status": 0, "chroot": "fedora-rawhide-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9931589-fedora-rawhide-ppc64le", "build_id": 9931589, "package_name": "golang-x-tools", "package_version": "1:0.36.0-2", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/alexsaezm/golang-1.26/golang-x-tools", "git_hash": "d35611109c101bd3ec3ed8ccfc9d84467cb8bf4c", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "alexsaezm/golang-1.26--alexsaezm", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/alexsaezm/golang-1.26/fedora-rawhide-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/alexsaezm/golang-1.26", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/alexsaezm/golang-1.26", "result_dir": "09931589-golang-x-tools", "built_packages": "", "id": 9931589, "mockchain_macros": { "copr_username": "alexsaezm", "copr_projectname": "golang-1.26", "vendor": "Fedora Project COPR (alexsaezm/golang-1.26)" } } ] } [2025-12-18 16:37:48,475][ INFO][PID:1873708] Sending fedora-messaging bus message in build.end [2025-12-18 16:37:48,504][ INFO][PID:1873708] Compressing /var/lib/copr/public_html/results/alexsaezm/golang-1.26/fedora-rawhide-ppc64le/09931589-golang-x-tools/builder-live.log by gzip [2025-12-18 16:37:48,505][ INFO][PID:1873708] Running command 'gzip /var/lib/copr/public_html/results/alexsaezm/golang-1.26/fedora-rawhide-ppc64le/09931589-golang-x-tools/builder-live.log' as PID 2077638 [2025-12-18 16:37:48,510][ INFO][PID:1873708] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/alexsaezm/golang-1.26/fedora-rawhide-ppc64le/09931589-golang-x-tools/builder-live.log) [2025-12-18 16:37:48,511][ INFO][PID:1873708] Compressing /var/lib/copr/public_html/results/alexsaezm/golang-1.26/fedora-rawhide-ppc64le/09931589-golang-x-tools/backend.log by gzip [2025-12-18 16:37:48,512][ INFO][PID:1873708] Running command 'gzip /var/lib/copr/public_html/results/alexsaezm/golang-1.26/fedora-rawhide-ppc64le/09931589-golang-x-tools/backend.log' as PID 2077639