[2026-02-10 18:11:43,764][  INFO][PID:2463851] Marking build as starting
[2026-02-10 18:11:43,839][  INFO][PID:2463851] Checking that https://download.copr.fedorainfracloud.org/results/@kicad/kicad-testing/fedora-44-ppc64le//repodata/repomd.xml exists
[2026-02-10 18:11:44,354][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:11:44,355][  INFO][PID:2463851] VM allocation process starts
[2026-02-10 18:11:44,371][  INFO][PID:2463851] Trying to allocate VM: ResallocHost, ticket_id=1261202, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-10 18:11:47,398][  INFO][PID:2463851] Allocated host ResallocHost, ticket_id=1261202, hostname=2620:52:6:1161:dead:beef:cafe:c18f, name=vmhost_p09_04_prod_02148456_20260210_180249, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-10 18:11:47,398][  INFO][PID:2463851] Allocating ssh connection to builder
[2026-02-10 18:11:47,399][  INFO][PID:2463851] Checking that builder machine is OK
[2026-02-10 18:12:47,470][ ERROR][PID:2463851] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c18f port 22: Connection timed out

[2026-02-10 18:12:57,479][ ERROR][PID:2463851] Re-try request for task on 'ResallocHost, ticket_id=1261202, hostname=2620:52:6:1161:dead:beef:cafe:c18f, name=vmhost_p09_04_prod_02148456_20260210_180249, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2026-02-10 18:12:57,482][  INFO][PID:2463851] Releasing VM back to pool
[2026-02-10 18:12:57,510][  INFO][PID:2463851] Retry #1 (on other host)
[2026-02-10 18:12:57,512][  INFO][PID:2463851] Checking that https://download.copr.fedorainfracloud.org/results/@kicad/kicad-testing/fedora-44-ppc64le//repodata/repomd.xml exists
[2026-02-10 18:12:57,890][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:12:57,892][  INFO][PID:2463851] VM allocation process starts
[2026-02-10 18:12:57,909][  INFO][PID:2463851] Trying to allocate VM: ResallocHost, ticket_id=1261225, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-10 18:13:00,926][  INFO][PID:2463851] Allocated host ResallocHost, ticket_id=1261225, hostname=2620:52:6:1161:dead:beef:cafe:c10a, name=vmhost_p09_02_prod_02148545_20260210_180705, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-10 18:13:00,927][  INFO][PID:2463851] Allocating ssh connection to builder
[2026-02-10 18:13:00,927][  INFO][PID:2463851] Checking that builder machine is OK
[2026-02-10 18:14:00,978][ ERROR][PID:2463851] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c10a port 22: Connection timed out

[2026-02-10 18:14:11,003][ ERROR][PID:2463851] Re-try request for task on 'ResallocHost, ticket_id=1261225, hostname=2620:52:6:1161:dead:beef:cafe:c10a, name=vmhost_p09_02_prod_02148545_20260210_180705, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2026-02-10 18:14:11,004][  INFO][PID:2463851] Releasing VM back to pool
[2026-02-10 18:14:11,016][  INFO][PID:2463851] Retry #2 (on other host)
[2026-02-10 18:14:11,016][  INFO][PID:2463851] Checking that https://download.copr.fedorainfracloud.org/results/@kicad/kicad-testing/fedora-44-ppc64le//repodata/repomd.xml exists
[2026-02-10 18:14:11,288][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:14:11,290][  INFO][PID:2463851] VM allocation process starts
[2026-02-10 18:14:11,307][  INFO][PID:2463851] Trying to allocate VM: ResallocHost, ticket_id=1261242, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-10 18:14:14,326][  INFO][PID:2463851] Allocated host ResallocHost, ticket_id=1261242, hostname=2620:52:6:1161:dead:beef:cafe:c190, name=vmhost_p09_04_prod_02148555_20260210_180719, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-10 18:14:14,327][  INFO][PID:2463851] Allocating ssh connection to builder
[2026-02-10 18:14:14,328][  INFO][PID:2463851] Checking that builder machine is OK
[2026-02-10 18:14:15,319][  INFO][PID:2463851] Running remote command: copr-builder-ready fedora-44-ppc64le
[2026-02-10 18:14:15,901][  INFO][PID:2463851] Red Hat subscription not needed for fedora-44-ppc64le
Builder is ready to be used

[2026-02-10 18:14:15,902][  INFO][PID:2463851] Filling build.info file with builder info
[2026-02-10 18:14:16,246][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:14:16,247][  INFO][PID:2463851] Not re-notifying FE and msg buses for the new host.
[2026-02-10 18:14:16,247][  INFO][PID:2463851] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10113684-fedora-44-ppc64le --chroot fedora-44-ppc64le --detached
[2026-02-10 18:14:17,241][  INFO][PID:2463851] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c190' (ED25519) to the list of known hosts.


[2026-02-10 18:14:17,242][  INFO][PID:2463851] Downloading the builder-live.log file, attempt 1
[2026-02-10 18:14:17,244][  INFO][PID:2463851] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c190 copr-rpmbuild-log
[2026-02-10 18:14:22,248][  INFO][PID:2463851] Periodic builder liveness probe: alive
[2026-02-10 18:14:22,248][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:14:27,249][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:14:32,250][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:14:37,251][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:14:42,252][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:14:47,253][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:14:52,254][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:14:57,255][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:15:02,256][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:15:07,257][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:15:19,066][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:15:24,173][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:15:29,201][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:15:34,318][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:15:39,320][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:15:44,360][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:15:49,361][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:15:54,383][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:15:59,405][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:16:04,447][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:16:09,451][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:16:14,452][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:16:19,454][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:16:24,455][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:16:29,457][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:16:34,636][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:16:39,638][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:16:44,641][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:16:49,642][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:16:54,643][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:16:59,644][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:17:04,646][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:17:09,647][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:17:14,649][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:17:19,657][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:17:24,658][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:17:29,659][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:17:34,661][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:17:39,664][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:17:44,666][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:17:49,667][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:17:54,695][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:17:59,710][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:18:04,712][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:18:07,412][ ERROR][PID:2463851] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-10 18:18:09,712][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:18:14,714][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:18:17,413][WARNING][PID:2463851] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-10 18:18:19,750][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:18:24,751][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:18:27,414][  INFO][PID:2463851] Downloading the builder-live.log file, attempt 2
[2026-02-10 18:18:27,415][  INFO][PID:2463851] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c190 copr-rpmbuild-log
[2026-02-10 18:18:29,754][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:18:34,755][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:18:39,756][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:18:44,758][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:18:49,759][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:18:54,763][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:18:59,818][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:19:04,819][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:19:09,821][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:19:14,822][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:19:19,823][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:19:24,826][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:19:27,483][ ERROR][PID:2463851] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-10 18:19:29,827][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:19:34,828][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:19:37,490][WARNING][PID:2463851] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-10 18:19:39,830][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:19:44,831][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:19:47,490][  INFO][PID:2463851] Downloading the builder-live.log file, attempt 3
[2026-02-10 18:19:47,492][  INFO][PID:2463851] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c190 copr-rpmbuild-log
[2026-02-10 18:19:49,840][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:19:54,841][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:19:59,842][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:20:04,850][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:20:10,541][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:20:15,544][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:20:20,991][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:20:26,010][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:20:31,012][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:20:36,013][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:20:41,014][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:20:46,019][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:20:47,563][ ERROR][PID:2463851] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-10 18:20:51,022][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:20:56,026][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:20:57,565][WARNING][PID:2463851] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-10 18:21:01,040][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:21:06,042][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:21:07,570][  INFO][PID:2463851] Downloading the builder-live.log file, attempt 4
[2026-02-10 18:21:07,574][  INFO][PID:2463851] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c190 copr-rpmbuild-log
[2026-02-10 18:21:11,045][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:21:16,049][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:21:21,071][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:21:26,074][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:21:31,076][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:21:36,077][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:21:41,082][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:21:46,083][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:21:51,086][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:21:56,088][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:22:01,089][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:22:06,090][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:22:07,663][ ERROR][PID:2463851] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-10 18:22:11,092][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:22:16,094][  INFO][PID:2463851] Checking for cancel request
[2026-02-10 18:22:17,664][ ERROR][PID:2463851] Giving up for unstable SSH, failures: 1770747497.4134471, 1770747577.4900882, 1770747657.5657191, 1770747737.664477
[2026-02-10 18:22:17,665][ ERROR][PID:2463851] Re-try request for task on 'ResallocHost, ticket_id=1261242, hostname=2620:52:6:1161:dead:beef:cafe:c190, name=vmhost_p09_04_prod_02148555_20260210_180719, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-02-10 18:22:17,666][  INFO][PID:2463851] Releasing VM back to pool
[2026-02-10 18:22:17,676][ ERROR][PID:2463851] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c190', '2620:52:6:1161:dead:beef:cafe:c10a', '2620:52:6:1161:dead:beef:cafe:c18f'}
[2026-02-10 18:22:17,676][WARNING][PID:2463851] Switching not-finished job state to 'failed'
[2026-02-10 18:22:17,677][  INFO][PID:2463851] Worker failed build, took 481.4295778274536
[2026-02-10 18:22:17,677][  INFO][PID:2463851] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 72000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@kicad",
            "project_name": "kicad-testing",
            "project_dirname": "kicad-testing",
            "submitter": "stevenfalco",
            "ended_on": 1770747737.6769145,
            "started_on": 1770747256.2473366,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-44-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "10113684-fedora-44-ppc64le",
            "build_id": 10113684,
            "package_name": "kicad",
            "package_version": "1:9.0.7-1000.20260210gitd5305ad",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@kicad/kicad-testing/kicad",
            "git_hash": "f109461726e0ec7be00d037ed7e72a6c0b2af390",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@kicad/kicad-testing--stevenfalco",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 1,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@kicad/kicad-testing/fedora-44-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "bootstrap": "on",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@kicad/kicad-testing",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@kicad/kicad-testing",
            "result_dir": "10113684-kicad",
            "built_packages": "",
            "id": 10113684,
            "mockchain_macros": {
                "copr_username": "@kicad",
                "copr_projectname": "kicad-testing",
                "vendor": "Fedora Project COPR (@kicad/kicad-testing)"
            }
        }
    ]
}
[2026-02-10 18:22:17,752][  INFO][PID:2463851] Sending fedora-messaging bus message in build.end
[2026-02-10 18:22:18,301][  INFO][PID:2463851] Compressing /var/lib/copr/public_html/results/@kicad/kicad-testing/fedora-44-ppc64le/10113684-kicad/builder-live.log by gzip
[2026-02-10 18:22:18,302][  INFO][PID:2463851] Running command 'gzip /var/lib/copr/public_html/results/@kicad/kicad-testing/fedora-44-ppc64le/10113684-kicad/builder-live.log' as PID 2526941
[2026-02-10 18:22:18,367][  INFO][PID:2463851] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@kicad/kicad-testing/fedora-44-ppc64le/10113684-kicad/builder-live.log)
[2026-02-10 18:22:18,368][  INFO][PID:2463851] Compressing /var/lib/copr/public_html/results/@kicad/kicad-testing/fedora-44-ppc64le/10113684-kicad/backend.log by gzip
[2026-02-10 18:22:18,369][  INFO][PID:2463851] Running command 'gzip /var/lib/copr/public_html/results/@kicad/kicad-testing/fedora-44-ppc64le/10113684-kicad/backend.log' as PID 2526947