[2025-12-28 04:42:09,375][  INFO][PID:642836] Marking build as starting
[2025-12-28 04:42:09,657][  INFO][PID:642836] Checking that /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2025-12-28 04:42:09,677][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:42:09,679][  INFO][PID:642836] VM allocation process starts
[2025-12-28 04:42:09,691][  INFO][PID:642836] Trying to allocate VM: ResallocHost, ticket_id=541500, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-28 04:42:14,693][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:42:15,722][  INFO][PID:642836] Allocated host ResallocHost, ticket_id=541500, hostname=2620:52:6:1161:dead:beef:cafe:c195, name=vmhost_p09_04_prod_00796137_20251228_042900, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-28 04:42:15,722][  INFO][PID:642836] Allocating ssh connection to builder
[2025-12-28 04:42:15,723][  INFO][PID:642836] Checking that builder machine is OK
[2025-12-28 04:43:15,791][ ERROR][PID:642836] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c195 port 22: Connection timed out

[2025-12-28 04:43:25,791][ ERROR][PID:642836] Re-try request for task on 'ResallocHost, ticket_id=541500, hostname=2620:52:6:1161:dead:beef:cafe:c195, name=vmhost_p09_04_prod_00796137_20251228_042900, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-28 04:43:25,792][  INFO][PID:642836] Releasing VM back to pool
[2025-12-28 04:43:25,801][  INFO][PID:642836] Retry #1 (on other host)
[2025-12-28 04:43:25,801][  INFO][PID:642836] Checking that /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2025-12-28 04:43:25,802][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:43:25,802][  INFO][PID:642836] VM allocation process starts
[2025-12-28 04:43:25,839][  INFO][PID:642836] Trying to allocate VM: ResallocHost, ticket_id=541508, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-28 04:43:28,879][  INFO][PID:642836] Allocated host ResallocHost, ticket_id=541508, hostname=2620:52:6:1161:dead:beef:cafe:c14d, name=vmhost_p09_03_prod_00796145_20251228_043039, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-28 04:43:28,880][  INFO][PID:642836] Allocating ssh connection to builder
[2025-12-28 04:43:28,880][  INFO][PID:642836] Checking that builder machine is OK
[2025-12-28 04:44:28,914][ ERROR][PID:642836] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c14d port 22: Connection timed out

[2025-12-28 04:44:38,915][ ERROR][PID:642836] Re-try request for task on 'ResallocHost, ticket_id=541508, hostname=2620:52:6:1161:dead:beef:cafe:c14d, name=vmhost_p09_03_prod_00796145_20251228_043039, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-28 04:44:38,916][  INFO][PID:642836] Releasing VM back to pool
[2025-12-28 04:44:38,925][  INFO][PID:642836] Retry #2 (on other host)
[2025-12-28 04:44:38,926][  INFO][PID:642836] Checking that /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2025-12-28 04:44:38,927][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:44:38,928][  INFO][PID:642836] VM allocation process starts
[2025-12-28 04:44:38,963][  INFO][PID:642836] Trying to allocate VM: ResallocHost, ticket_id=541512, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-28 04:44:42,019][  INFO][PID:642836] Allocated host ResallocHost, ticket_id=541512, hostname=2620:52:6:1161:dead:beef:cafe:c148, name=vmhost_p09_03_prod_00796218_20251228_043536, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-28 04:44:42,020][  INFO][PID:642836] Allocating ssh connection to builder
[2025-12-28 04:44:42,020][  INFO][PID:642836] Checking that builder machine is OK
[2025-12-28 04:44:42,891][  INFO][PID:642836] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2025-12-28 04:44:43,373][  INFO][PID:642836] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2025-12-28 04:44:43,374][  INFO][PID:642836] Filling build.info file with builder info
[2025-12-28 04:44:43,374][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:44:43,375][  INFO][PID:642836] Not re-notifying FE and msg buses for the new host.
[2025-12-28 04:44:43,375][  INFO][PID:642836] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9954231-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2025-12-28 04:44:44,348][  INFO][PID:642836] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c148' (ED25519) to the list of known hosts.


[2025-12-28 04:44:44,348][  INFO][PID:642836] Downloading the builder-live.log file, attempt 1
[2025-12-28 04:44:44,350][  INFO][PID:642836] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c148 copr-rpmbuild-log
[2025-12-28 04:44:49,357][  INFO][PID:642836] Periodic builder liveness probe: alive
[2025-12-28 04:44:49,358][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:44:54,359][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:44:59,360][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:45:04,361][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:45:09,362][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:45:14,363][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:45:19,364][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:45:24,365][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:45:29,366][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:45:34,367][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:45:39,368][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:45:44,369][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:45:49,370][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:45:54,371][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:45:59,372][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:46:04,373][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:46:09,469][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:46:14,471][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:46:19,514][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:46:24,677][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:46:29,687][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:46:34,689][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:46:39,690][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:46:44,692][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:46:49,693][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:46:54,694][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:46:59,817][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:47:04,818][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:47:09,820][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:47:14,821][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:47:19,821][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:47:25,020][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:47:30,021][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:47:35,023][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:47:40,024][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:47:45,025][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:47:50,026][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:47:55,027][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:48:00,028][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:48:05,029][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:48:10,030][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:48:15,031][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:48:20,032][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:48:25,033][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:48:30,033][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:48:35,034][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:48:40,133][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:48:45,134][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:48:50,135][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:48:55,136][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:49:00,137][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:49:05,138][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:49:10,139][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:49:15,140][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:49:20,141][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:49:25,141][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:49:30,142][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:49:35,144][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:49:40,145][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:49:45,243][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:49:50,244][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:49:55,245][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:50:00,311][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:50:05,405][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:50:10,407][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:50:15,408][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:50:20,409][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:50:25,411][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:50:30,442][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:50:35,443][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:50:40,444][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:50:45,445][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:50:50,639][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:50:55,640][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:51:00,641][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:51:05,711][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:51:10,713][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:51:15,768][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:51:20,770][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:51:25,801][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:51:30,824][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:51:35,910][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:51:40,911][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:51:45,912][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:51:50,914][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:51:55,932][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:52:00,933][  INFO][PID:642836] Checking for cancel request
[2025-12-28 04:52:01,851][  INFO][PID:642836] Downloading results from builder
[2025-12-28 04:52:01,852][  INFO][PID:642836] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c148]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/09954231-python-uncertainties started
[2025-12-28 04:52:01,853][  INFO][PID:642836] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c148]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/09954231-python-uncertainties/ &> /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/09954231-python-uncertainties/build-09954231.rsync.log
[2025-12-28 04:52:02,475][  INFO][PID:642836] rsyncing finished.
[2025-12-28 04:52:02,476][  INFO][PID:642836] Releasing VM back to pool
[2025-12-28 04:52:02,485][  INFO][PID:642836] Searching for 'success' file in resultdir
[2025-12-28 04:52:02,485][ ERROR][PID:642836] Build failed: Backend process error: No success file => build failure
[2025-12-28 04:52:02,486][  INFO][PID:642836] Finished build: id=9954231 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/topazus/test-review chroot=fedora-rawhide-ppc64le 
[2025-12-28 04:52:02,488][ ERROR][PID:642836] Backend process error: No success file => build failure
[2025-12-28 04:52:02,488][  INFO][PID:642836] Worker failed build, took 439.1131663322449
[2025-12-28 04:52:02,488][  INFO][PID:642836] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "topazus",
            "project_name": "test-review",
            "project_dirname": "test-review",
            "submitter": "topazus",
            "ended_on": 1766897522.4883418,
            "started_on": 1766897083.3751755,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9954231-fedora-rawhide-ppc64le",
            "build_id": 9954231,
            "package_name": "python-uncertainties",
            "package_version": "3.2.3-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/topazus/test-review/python-uncertainties",
            "git_hash": "e2c52bd0e495bdd5a1773f1ee31bbb87cb3657fa",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "topazus/test-review--topazus",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test-review/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test-review/fedora-rawhide-ppc64le/",
                    "id": "copr_topazus_test_review",
                    "name": "Additional repo copr_topazus_test_review"
                }
            ],
            "background": false,
            "fedora_review": true,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/topazus/test-review",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/topazus/test-review",
            "result_dir": "09954231-python-uncertainties",
            "built_packages": "",
            "id": 9954231,
            "mockchain_macros": {
                "copr_username": "topazus",
                "copr_projectname": "test-review",
                "vendor": "Fedora Project COPR (topazus/test-review)"
            }
        }
    ]
}
[2025-12-28 04:52:02,546][  INFO][PID:642836] Sending fedora-messaging bus message in build.end
[2025-12-28 04:52:03,019][  INFO][PID:642836] Compressing /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/09954231-python-uncertainties/builder-live.log by gzip
[2025-12-28 04:52:03,020][  INFO][PID:642836] Running command 'gzip /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/09954231-python-uncertainties/builder-live.log' as PID 697868
[2025-12-28 04:52:03,067][  INFO][PID:642836] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/09954231-python-uncertainties/builder-live.log)
[2025-12-28 04:52:03,068][  INFO][PID:642836] Compressing /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/09954231-python-uncertainties/backend.log by gzip
[2025-12-28 04:52:03,068][  INFO][PID:642836] Running command 'gzip /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/09954231-python-uncertainties/backend.log' as PID 697872