[2025-12-15 17:39:34,629][  INFO][PID:1857252] Marking build as starting
[2025-12-15 17:39:34,753][  INFO][PID:1857252] Checking that /var/lib/copr/public_html/results/keithbowes/deponejo/fedora-43-ppc64le/repodata/repomd.xml exists
[2025-12-15 17:39:34,757][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:39:34,761][  INFO][PID:1857252] VM allocation process starts
[2025-12-15 17:39:34,808][  INFO][PID:1857252] Trying to allocate VM: ResallocHost, ticket_id=353155, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-15 17:39:39,817][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:39:40,852][  INFO][PID:1857252] Allocated host ResallocHost, ticket_id=353155, hostname=2620:52:6:1161:dead:beef:cafe:c109, name=vmhost_p09_02_prod_00572753_20251215_173123, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-15 17:39:40,853][  INFO][PID:1857252] Allocating ssh connection to builder
[2025-12-15 17:39:40,853][  INFO][PID:1857252] Checking that builder machine is OK
[2025-12-15 17:39:41,794][  INFO][PID:1857252] Running remote command: copr-builder-ready fedora-43-ppc64le
[2025-12-15 17:39:42,362][  INFO][PID:1857252] Red Hat subscription not needed for fedora-43-ppc64le
Builder is ready to be used

[2025-12-15 17:39:42,366][  INFO][PID:1857252] Filling build.info file with builder info
[2025-12-15 17:39:42,369][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:39:42,370][  INFO][PID:1857252] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "keithbowes",
            "project_name": "deponejo",
            "project_dirname": "deponejo",
            "submitter": "keithbowes",
            "ended_on": null,
            "started_on": 1765820382.3704543,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-43-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9911869-fedora-43-ppc64le",
            "build_id": 9911869,
            "package_name": "neomutt",
            "package_version": "20251211-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/keithbowes/deponejo/neomutt",
            "git_hash": "9b6f27218bcbfd3cc2f30a339ecc1cba008f5f87",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "keithbowes/deponejo--keithbowes",
            "results": null,
            "appstream": true,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/keithbowes/deponejo/fedora-43-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/keithbowes/deponejo",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/keithbowes/deponejo",
            "result_dir": "09911869-neomutt",
            "built_packages": "",
            "id": 9911869,
            "mockchain_macros": {
                "copr_username": "keithbowes",
                "copr_projectname": "deponejo",
                "vendor": "Fedora Project COPR (keithbowes/deponejo)"
            }
        }
    ]
}
[2025-12-15 17:39:42,452][  INFO][PID:1857252] Sending fedora-messaging bus message in build.start
[2025-12-15 17:39:43,228][  INFO][PID:1857252] Sending fedora-messaging bus message in chroot.start
[2025-12-15 17:39:43,267][  INFO][PID:1857252] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9911869-fedora-43-ppc64le --chroot fedora-43-ppc64le --detached
[2025-12-15 17:39:44,307][  INFO][PID:1857252] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c109' (ED25519) to the list of known hosts.


[2025-12-15 17:39:44,310][  INFO][PID:1857252] Downloading the builder-live.log file, attempt 1
[2025-12-15 17:39:44,314][  INFO][PID:1857252] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c109 copr-rpmbuild-log
[2025-12-15 17:39:49,345][  INFO][PID:1857252] Periodic builder liveness probe: alive
[2025-12-15 17:39:49,347][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:39:54,358][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:39:59,360][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:40:04,378][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:40:09,380][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:40:14,383][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:40:19,388][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:40:24,390][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:40:29,392][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:40:34,422][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:40:39,423][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:40:44,425][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:40:49,426][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:40:54,427][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:40:59,428][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:41:04,431][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:41:09,432][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:41:14,433][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:41:19,434][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:41:24,436][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:41:29,437][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:41:34,438][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:41:39,439][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:41:44,440][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:41:49,441][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:41:54,442][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:41:59,443][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:42:04,446][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:42:09,447][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:42:14,448][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:42:19,450][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:42:24,452][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:42:29,454][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:42:34,457][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:42:39,563][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:42:44,564][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:42:49,638][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:42:54,640][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:42:59,642][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:43:04,647][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:43:09,649][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:43:14,655][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:43:19,659][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:43:24,661][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:43:29,668][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:43:34,671][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:43:39,673][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:43:44,677][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:43:49,678][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:43:54,683][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:43:59,684][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:44:04,686][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:44:09,688][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:44:14,689][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:44:19,690][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:44:24,692][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:44:29,693][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:44:34,694][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:44:39,695][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:44:44,696][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:44:49,697][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:44:54,698][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:44:59,699][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:45:04,700][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:45:09,701][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:45:14,702][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:45:19,703][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:45:24,704][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:45:29,705][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:45:34,706][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:45:39,707][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:45:44,708][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:45:49,709][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:45:54,710][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:45:59,711][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:46:04,712][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:46:09,713][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:46:14,714][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:46:19,715][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:46:24,716][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:46:29,716][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:46:34,717][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:46:39,718][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:46:44,719][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:46:49,720][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:46:54,721][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:46:59,722][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:47:04,722][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:47:09,723][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:47:14,724][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:47:19,725][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:47:24,726][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:47:29,727][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:47:34,728][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:47:39,729][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:47:44,730][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:47:49,731][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:47:54,732][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:47:59,733][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:48:04,734][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:48:09,735][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:48:14,735][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:48:19,736][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:48:24,737][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:48:29,738][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:48:34,739][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:48:39,740][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:48:44,741][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:48:49,742][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:48:54,743][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:48:59,744][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:49:04,745][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:49:09,746][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:49:14,747][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:49:19,747][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:49:24,748][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:49:29,749][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:49:34,750][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:49:39,751][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:49:44,752][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:49:49,762][  INFO][PID:1857252] Periodic builder liveness probe: alive
[2025-12-15 17:49:49,763][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:49:54,763][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:49:59,764][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:50:04,765][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:50:09,766][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:50:14,767][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:50:19,768][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:50:24,769][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:50:29,770][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:50:34,771][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:50:39,772][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:50:44,773][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:50:49,774][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:50:54,775][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:50:59,776][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:51:04,777][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:51:09,778][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:51:14,779][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:51:19,780][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:51:24,780][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:51:29,015][  INFO][PID:1857252] Downloading results from builder
[2025-12-15 17:51:29,016][  INFO][PID:1857252] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c109]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/keithbowes/deponejo/fedora-43-ppc64le/09911869-neomutt started
[2025-12-15 17:51:29,017][  INFO][PID:1857252] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c109]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/keithbowes/deponejo/fedora-43-ppc64le/09911869-neomutt/ &> /var/lib/copr/public_html/results/keithbowes/deponejo/fedora-43-ppc64le/09911869-neomutt/build-09911869.rsync.log
[2025-12-15 17:52:29,091][  INFO][PID:1857252] rsyncing finished.
[2025-12-15 17:52:29,092][ ERROR][PID:1857252] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-15 17:52:29,093][ ERROR][PID:1857252] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-15 17:52:39,094][  INFO][PID:1857252] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c109]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/keithbowes/deponejo/fedora-43-ppc64le/09911869-neomutt started
[2025-12-15 17:52:39,095][  INFO][PID:1857252] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c109]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/keithbowes/deponejo/fedora-43-ppc64le/09911869-neomutt/ &> /var/lib/copr/public_html/results/keithbowes/deponejo/fedora-43-ppc64le/09911869-neomutt/build-09911869.rsync.log
[2025-12-15 17:53:39,166][  INFO][PID:1857252] rsyncing finished.
[2025-12-15 17:53:39,167][ ERROR][PID:1857252] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-15 17:53:39,167][ ERROR][PID:1857252] SSH connection lost on #2 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-15 17:53:49,167][  INFO][PID:1857252] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c109]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/keithbowes/deponejo/fedora-43-ppc64le/09911869-neomutt started
[2025-12-15 17:53:49,168][  INFO][PID:1857252] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c109]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/keithbowes/deponejo/fedora-43-ppc64le/09911869-neomutt/ &> /var/lib/copr/public_html/results/keithbowes/deponejo/fedora-43-ppc64le/09911869-neomutt/build-09911869.rsync.log
[2025-12-15 17:54:49,201][  INFO][PID:1857252] rsyncing finished.
[2025-12-15 17:54:49,202][ ERROR][PID:1857252] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-15 17:54:49,202][ ERROR][PID:1857252] SSH connection lost on #3 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-15 17:54:59,203][ ERROR][PID:1857252] Re-try request for task on 'ResallocHost, ticket_id=353155, hostname=2620:52:6:1161:dead:beef:cafe:c109, name=vmhost_p09_02_prod_00572753_20251215_173123, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 3 SSH attempts
[2025-12-15 17:54:59,203][  INFO][PID:1857252] Releasing VM back to pool
[2025-12-15 17:54:59,212][  INFO][PID:1857252] Retry #1 (on other host)
[2025-12-15 17:54:59,212][  INFO][PID:1857252] Checking that /var/lib/copr/public_html/results/keithbowes/deponejo/fedora-43-ppc64le/repodata/repomd.xml exists
[2025-12-15 17:54:59,212][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:54:59,213][  INFO][PID:1857252] VM allocation process starts
[2025-12-15 17:54:59,286][  INFO][PID:1857252] Trying to allocate VM: ResallocHost, ticket_id=353288, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-15 17:55:02,310][  INFO][PID:1857252] Allocated host ResallocHost, ticket_id=353288, hostname=2620:52:6:1161:dead:beef:cafe:c194, name=vmhost_p09_04_prod_00573124_20251215_174230, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-15 17:55:02,311][  INFO][PID:1857252] Allocating ssh connection to builder
[2025-12-15 17:55:02,311][  INFO][PID:1857252] Checking that builder machine is OK
[2025-12-15 17:56:02,369][ ERROR][PID:1857252] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c194 port 22: Connection timed out

[2025-12-15 17:56:12,369][ ERROR][PID:1857252] Re-try request for task on 'ResallocHost, ticket_id=353288, hostname=2620:52:6:1161:dead:beef:cafe:c194, name=vmhost_p09_04_prod_00573124_20251215_174230, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-15 17:56:12,370][  INFO][PID:1857252] Releasing VM back to pool
[2025-12-15 17:56:12,381][  INFO][PID:1857252] Retry #2 (on other host)
[2025-12-15 17:56:12,381][  INFO][PID:1857252] Checking that /var/lib/copr/public_html/results/keithbowes/deponejo/fedora-43-ppc64le/repodata/repomd.xml exists
[2025-12-15 17:56:12,382][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:56:12,382][  INFO][PID:1857252] VM allocation process starts
[2025-12-15 17:56:12,396][  INFO][PID:1857252] Trying to allocate VM: ResallocHost, ticket_id=353292, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-15 17:56:15,409][  INFO][PID:1857252] Allocated host ResallocHost, ticket_id=353292, hostname=2620:52:6:1161:dead:beef:cafe:c109, name=vmhost_p09_02_prod_00572753_20251215_173123, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-15 17:56:15,410][  INFO][PID:1857252] Allocating ssh connection to builder
[2025-12-15 17:56:15,410][  INFO][PID:1857252] Checking that builder machine is OK
[2025-12-15 17:57:15,479][ ERROR][PID:1857252] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c109 port 22: Connection timed out

[2025-12-15 17:57:25,479][ ERROR][PID:1857252] Re-try request for task on 'ResallocHost, ticket_id=353292, hostname=2620:52:6:1161:dead:beef:cafe:c109, name=vmhost_p09_02_prod_00572753_20251215_173123, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-15 17:57:25,480][  INFO][PID:1857252] Releasing VM back to pool
[2025-12-15 17:57:25,494][  INFO][PID:1857252] Retry #3 (on other host)
[2025-12-15 17:57:25,494][  INFO][PID:1857252] Checking that /var/lib/copr/public_html/results/keithbowes/deponejo/fedora-43-ppc64le/repodata/repomd.xml exists
[2025-12-15 17:57:25,495][  INFO][PID:1857252] Checking for cancel request
[2025-12-15 17:57:25,495][  INFO][PID:1857252] VM allocation process starts
[2025-12-15 17:57:25,506][  INFO][PID:1857252] Trying to allocate VM: ResallocHost, ticket_id=353306, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-15 17:57:28,519][  INFO][PID:1857252] Allocated host ResallocHost, ticket_id=353306, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_00571508_20251215_170333, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-15 17:57:28,519][  INFO][PID:1857252] Allocating ssh connection to builder
[2025-12-15 17:57:28,520][  INFO][PID:1857252] Checking that builder machine is OK
[2025-12-15 17:58:28,588][ ERROR][PID:1857252] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c191 port 22: Connection timed out

[2025-12-15 17:58:38,589][ ERROR][PID:1857252] Re-try request for task on 'ResallocHost, ticket_id=353306, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_00571508_20251215_170333, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-15 17:58:38,590][  INFO][PID:1857252] Releasing VM back to pool
[2025-12-15 17:58:38,599][ ERROR][PID:1857252] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c194', '2620:52:6:1161:dead:beef:cafe:c109', '2620:52:6:1161:dead:beef:cafe:c191'}
[2025-12-15 17:58:38,600][WARNING][PID:1857252] Switching not-finished job state to 'failed'
[2025-12-15 17:58:38,600][  INFO][PID:1857252] Worker failed build, took 1136.2297213077545
[2025-12-15 17:58:38,600][  INFO][PID:1857252] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "keithbowes",
            "project_name": "deponejo",
            "project_dirname": "deponejo",
            "submitter": "keithbowes",
            "ended_on": 1765821518.6001756,
            "started_on": 1765820382.3704543,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-43-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9911869-fedora-43-ppc64le",
            "build_id": 9911869,
            "package_name": "neomutt",
            "package_version": "20251211-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/keithbowes/deponejo/neomutt",
            "git_hash": "9b6f27218bcbfd3cc2f30a339ecc1cba008f5f87",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "keithbowes/deponejo--keithbowes",
            "results": null,
            "appstream": true,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/keithbowes/deponejo/fedora-43-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/keithbowes/deponejo",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/keithbowes/deponejo",
            "result_dir": "09911869-neomutt",
            "built_packages": "",
            "id": 9911869,
            "mockchain_macros": {
                "copr_username": "keithbowes",
                "copr_projectname": "deponejo",
                "vendor": "Fedora Project COPR (keithbowes/deponejo)"
            }
        }
    ]
}
[2025-12-15 17:58:38,664][  INFO][PID:1857252] Sending fedora-messaging bus message in build.end
[2025-12-15 17:58:38,689][  INFO][PID:1857252] Compressing /var/lib/copr/public_html/results/keithbowes/deponejo/fedora-43-ppc64le/09911869-neomutt/builder-live.log by gzip
[2025-12-15 17:58:38,690][  INFO][PID:1857252] Running command 'gzip /var/lib/copr/public_html/results/keithbowes/deponejo/fedora-43-ppc64le/09911869-neomutt/builder-live.log' as PID 1969521
[2025-12-15 17:58:38,712][  INFO][PID:1857252] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/keithbowes/deponejo/fedora-43-ppc64le/09911869-neomutt/builder-live.log)
[2025-12-15 17:58:38,713][  INFO][PID:1857252] Compressing /var/lib/copr/public_html/results/keithbowes/deponejo/fedora-43-ppc64le/09911869-neomutt/backend.log by gzip
[2025-12-15 17:58:38,714][  INFO][PID:1857252] Running command 'gzip /var/lib/copr/public_html/results/keithbowes/deponejo/fedora-43-ppc64le/09911869-neomutt/backend.log' as PID 1969524