[2025-12-11 11:51:23,345][  INFO][PID:1856476] Marking build as starting
[2025-12-11 11:51:23,418][  INFO][PID:1856476] Checking that /var/lib/copr/public_html/results/prgutier/asgi-lifespan/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2025-12-11 11:51:23,424][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:51:23,425][  INFO][PID:1856476] VM allocation process starts
[2025-12-11 11:51:23,519][  INFO][PID:1856476] Trying to allocate VM: ResallocHost, ticket_id=281927, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-11 11:51:28,526][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:51:33,527][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:51:35,607][  INFO][PID:1856476] Allocated host ResallocHost, ticket_id=281927, hostname=2620:52:6:1161:dead:beef:cafe:c18c, name=vmhost_p09_04_prod_00344332_20251211_112835, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-11 11:51:35,607][  INFO][PID:1856476] Allocating ssh connection to builder
[2025-12-11 11:51:35,608][  INFO][PID:1856476] Checking that builder machine is OK
[2025-12-11 11:52:35,679][ ERROR][PID:1856476] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c18c port 22: Connection timed out

[2025-12-11 11:52:45,680][ ERROR][PID:1856476] Re-try request for task on 'ResallocHost, ticket_id=281927, hostname=2620:52:6:1161:dead:beef:cafe:c18c, name=vmhost_p09_04_prod_00344332_20251211_112835, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-11 11:52:45,681][  INFO][PID:1856476] Releasing VM back to pool
[2025-12-11 11:52:45,694][  INFO][PID:1856476] Retry #1 (on other host)
[2025-12-11 11:52:45,695][  INFO][PID:1856476] Checking that /var/lib/copr/public_html/results/prgutier/asgi-lifespan/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2025-12-11 11:52:45,696][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:52:45,697][  INFO][PID:1856476] VM allocation process starts
[2025-12-11 11:52:45,723][  INFO][PID:1856476] Trying to allocate VM: ResallocHost, ticket_id=281952, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-11 11:52:50,726][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:52:51,767][  INFO][PID:1856476] Allocated host ResallocHost, ticket_id=281952, hostname=2620:52:6:1161:dead:beef:cafe:c18a, name=vmhost_p09_04_prod_00344295_20251211_112332, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-11 11:52:51,768][  INFO][PID:1856476] Allocating ssh connection to builder
[2025-12-11 11:52:51,769][  INFO][PID:1856476] Checking that builder machine is OK
[2025-12-11 11:53:51,841][ ERROR][PID:1856476] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c18a port 22: Connection timed out

[2025-12-11 11:54:01,842][ ERROR][PID:1856476] Re-try request for task on 'ResallocHost, ticket_id=281952, hostname=2620:52:6:1161:dead:beef:cafe:c18a, name=vmhost_p09_04_prod_00344295_20251211_112332, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-11 11:54:01,842][  INFO][PID:1856476] Releasing VM back to pool
[2025-12-11 11:54:01,855][  INFO][PID:1856476] Retry #2 (on other host)
[2025-12-11 11:54:01,856][  INFO][PID:1856476] Checking that /var/lib/copr/public_html/results/prgutier/asgi-lifespan/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2025-12-11 11:54:01,856][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:54:01,857][  INFO][PID:1856476] VM allocation process starts
[2025-12-11 11:54:01,869][  INFO][PID:1856476] Trying to allocate VM: ResallocHost, ticket_id=281989, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-11 11:54:06,870][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:54:07,896][  INFO][PID:1856476] Allocated host ResallocHost, ticket_id=281989, hostname=2620:52:6:1161:dead:beef:cafe:c196, name=vmhost_p09_04_prod_00344508_20251211_114739, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-11 11:54:07,897][  INFO][PID:1856476] Allocating ssh connection to builder
[2025-12-11 11:54:07,897][  INFO][PID:1856476] Checking that builder machine is OK
[2025-12-11 11:54:08,814][  INFO][PID:1856476] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2025-12-11 11:54:09,365][  INFO][PID:1856476] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2025-12-11 11:54:09,367][  INFO][PID:1856476] Filling build.info file with builder info
[2025-12-11 11:54:09,368][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:54:09,370][  INFO][PID:1856476] Not re-notifying FE and msg buses for the new host.
[2025-12-11 11:54:09,371][  INFO][PID:1856476] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9899104-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2025-12-11 11:54:10,307][  INFO][PID:1856476] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c196' (ED25519) to the list of known hosts.


[2025-12-11 11:54:10,308][  INFO][PID:1856476] Downloading the builder-live.log file, attempt 1
[2025-12-11 11:54:10,310][  INFO][PID:1856476] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c196 copr-rpmbuild-log
[2025-12-11 11:54:15,321][  INFO][PID:1856476] Periodic builder liveness probe: alive
[2025-12-11 11:54:15,322][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:54:20,323][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:54:25,325][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:54:30,326][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:54:35,327][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:54:40,329][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:54:45,330][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:54:50,332][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:54:55,334][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:55:00,335][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:55:05,336][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:55:10,337][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:55:15,340][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:55:20,341][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:55:25,342][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:55:30,344][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:55:35,345][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:55:40,346][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:55:45,347][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:55:50,348][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:55:55,350][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:56:00,351][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:56:05,352][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:56:10,353][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:56:15,354][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:56:20,355][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:56:25,356][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:56:30,357][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:56:35,358][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:56:40,359][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:56:45,360][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:56:50,362][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:56:55,363][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:57:00,364][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:57:05,365][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:57:10,366][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:57:15,367][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:57:20,368][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:57:25,369][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:57:30,370][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:57:32,370][ ERROR][PID:1856476] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-11 11:57:35,371][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:57:40,372][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:57:42,371][WARNING][PID:1856476] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-11 11:57:45,373][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:57:50,374][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:57:52,372][  INFO][PID:1856476] Downloading the builder-live.log file, attempt 2
[2025-12-11 11:57:52,374][  INFO][PID:1856476] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c196 copr-rpmbuild-log
[2025-12-11 11:57:55,375][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:58:00,376][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:58:05,377][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:58:10,378][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:58:15,379][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:58:20,380][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:58:25,382][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:58:30,382][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:58:35,384][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:58:40,385][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:58:45,386][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:58:50,387][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:58:52,441][ ERROR][PID:1856476] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-11 11:58:55,388][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:59:00,389][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:59:02,442][WARNING][PID:1856476] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-11 11:59:05,390][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:59:10,391][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:59:12,443][  INFO][PID:1856476] Downloading the builder-live.log file, attempt 3
[2025-12-11 11:59:12,444][  INFO][PID:1856476] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c196 copr-rpmbuild-log
[2025-12-11 11:59:15,391][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:59:20,393][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:59:25,394][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:59:30,395][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:59:35,396][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:59:40,397][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:59:45,398][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:59:50,399][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 11:59:55,400][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:00:00,401][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:00:05,402][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:00:10,403][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:00:12,511][ ERROR][PID:1856476] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-11 12:00:15,405][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:00:20,406][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:00:22,515][WARNING][PID:1856476] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-11 12:00:25,407][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:00:30,408][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:00:32,516][  INFO][PID:1856476] Downloading the builder-live.log file, attempt 4
[2025-12-11 12:00:32,517][  INFO][PID:1856476] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c196 copr-rpmbuild-log
[2025-12-11 12:00:35,409][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:00:40,411][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:00:45,412][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:00:50,413][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:00:55,416][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:01:00,417][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:01:05,418][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:01:10,421][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:01:15,424][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:01:20,425][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:01:25,426][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:01:30,427][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:01:32,585][ ERROR][PID:1856476] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-11 12:01:35,429][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:01:40,430][  INFO][PID:1856476] Checking for cancel request
[2025-12-11 12:01:42,587][ ERROR][PID:1856476] Giving up for unstable SSH, failures: 1765454262.3715718, 1765454342.4425318, 1765454422.5155644, 1765454502.5874531
[2025-12-11 12:01:42,588][ ERROR][PID:1856476] Re-try request for task on 'ResallocHost, ticket_id=281989, hostname=2620:52:6:1161:dead:beef:cafe:c196, name=vmhost_p09_04_prod_00344508_20251211_114739, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-12-11 12:01:42,589][  INFO][PID:1856476] Releasing VM back to pool
[2025-12-11 12:01:42,598][ ERROR][PID:1856476] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c18a', '2620:52:6:1161:dead:beef:cafe:c18c', '2620:52:6:1161:dead:beef:cafe:c196'}
[2025-12-11 12:01:42,599][WARNING][PID:1856476] Switching not-finished job state to 'failed'
[2025-12-11 12:01:42,599][  INFO][PID:1856476] Worker failed build, took 453.2287802696228
[2025-12-11 12:01:42,600][  INFO][PID:1856476] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "prgutier",
            "project_name": "asgi-lifespan",
            "project_dirname": "asgi-lifespan",
            "submitter": "prgutier",
            "ended_on": 1765454502.5990295,
            "started_on": 1765454049.3702493,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9899104-fedora-rawhide-ppc64le",
            "build_id": 9899104,
            "package_name": "python-asgi-lifespan",
            "package_version": "2.1.0-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/prgutier/asgi-lifespan/python-asgi-lifespan",
            "git_hash": "b79a429eb31443a2f5de043bd9ca3e95319d57a9",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "prgutier/asgi-lifespan--prgutier",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/prgutier/asgi-lifespan/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/prgutier/asgi-lifespan",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/prgutier/asgi-lifespan",
            "result_dir": "09899104-python-asgi-lifespan",
            "built_packages": "",
            "id": 9899104,
            "mockchain_macros": {
                "copr_username": "prgutier",
                "copr_projectname": "asgi-lifespan",
                "vendor": "Fedora Project COPR (prgutier/asgi-lifespan)"
            }
        }
    ]
}
[2025-12-11 12:01:42,635][  INFO][PID:1856476] Sending fedora-messaging bus message in build.end
[2025-12-11 12:01:43,217][  INFO][PID:1856476] Compressing /var/lib/copr/public_html/results/prgutier/asgi-lifespan/fedora-rawhide-ppc64le/09899104-python-asgi-lifespan/builder-live.log by gzip
[2025-12-11 12:01:43,219][  INFO][PID:1856476] Running command 'gzip /var/lib/copr/public_html/results/prgutier/asgi-lifespan/fedora-rawhide-ppc64le/09899104-python-asgi-lifespan/builder-live.log' as PID 1937157
[2025-12-11 12:01:43,221][  INFO][PID:1856476] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/prgutier/asgi-lifespan/fedora-rawhide-ppc64le/09899104-python-asgi-lifespan/builder-live.log)
[2025-12-11 12:01:43,221][  INFO][PID:1856476] Compressing /var/lib/copr/public_html/results/prgutier/asgi-lifespan/fedora-rawhide-ppc64le/09899104-python-asgi-lifespan/backend.log by gzip
[2025-12-11 12:01:43,222][  INFO][PID:1856476] Running command 'gzip /var/lib/copr/public_html/results/prgutier/asgi-lifespan/fedora-rawhide-ppc64le/09899104-python-asgi-lifespan/backend.log' as PID 1937159