[2026-02-11 13:49:06,006][  INFO][PID:3859344] Marking build as starting
[2026-02-11 13:49:06,255][  INFO][PID:3859344] Checking that /var/lib/copr/public_html/results/dreua/trash-cli/epel-8-ppc64le/repodata/repomd.xml exists
[2026-02-11 13:49:06,270][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:49:06,273][  INFO][PID:3859344] VM allocation process starts
[2026-02-11 13:49:06,292][  INFO][PID:3859344] Trying to allocate VM: ResallocHost, ticket_id=1275201, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-11 13:49:09,315][  INFO][PID:3859344] Allocated host ResallocHost, ticket_id=1275201, hostname=2620:52:6:1161:dead:beef:cafe:c150, name=vmhost_p09_03_prod_02162095_20260211_134348, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-11 13:49:09,317][  INFO][PID:3859344] Allocating ssh connection to builder
[2026-02-11 13:49:09,318][  INFO][PID:3859344] Checking that builder machine is OK
[2026-02-11 13:49:13,058][  INFO][PID:3859344] Running remote command: copr-builder-ready epel-8-ppc64le
[2026-02-11 13:49:15,743][  INFO][PID:3859344] Checking Red Hat subscription (attempt #1)
Red Hat subscription active
Builder is ready to be used

[2026-02-11 13:49:15,744][  INFO][PID:3859344] Filling build.info file with builder info
[2026-02-11 13:49:15,744][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:49:15,745][  INFO][PID:3859344] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "dreua",
            "project_name": "trash-cli",
            "project_dirname": "trash-cli",
            "submitter": "https://src.fedoraproject.org/user/salimma",
            "ended_on": null,
            "started_on": 1770817755.745417,
            "submitted_on": null,
            "status": 3,
            "chroot": "epel-8-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "10116442-epel-8-ppc64le",
            "build_id": 10116442,
            "package_name": "trash-cli",
            "package_version": "0.24.5.26-18",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/dreua/trash-cli/trash-cli",
            "git_hash": "c22c9b85edf7aaebae3e6b45b63349c020b896f5",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "dreua/trash-cli--https://src.fedoraproject.org/user/salimma",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/dreua/trash-cli/epel-8-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/dreua/trash-cli",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/dreua/trash-cli",
            "result_dir": "10116442-trash-cli",
            "built_packages": "",
            "id": 10116442,
            "mockchain_macros": {
                "copr_username": "dreua",
                "copr_projectname": "trash-cli",
                "vendor": "Fedora Project COPR (dreua/trash-cli)"
            }
        }
    ]
}
[2026-02-11 13:49:15,806][  INFO][PID:3859344] Sending fedora-messaging bus message in build.start
[2026-02-11 13:49:16,548][  INFO][PID:3859344] Sending fedora-messaging bus message in chroot.start
[2026-02-11 13:49:16,579][  INFO][PID:3859344] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10116442-epel-8-ppc64le --chroot epel-8-ppc64le --detached
[2026-02-11 13:49:20,522][  INFO][PID:3859344] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c150' (ED25519) to the list of known hosts.


[2026-02-11 13:49:20,524][  INFO][PID:3859344] Downloading the builder-live.log file, attempt 1
[2026-02-11 13:49:20,525][  INFO][PID:3859344] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c150 copr-rpmbuild-log
[2026-02-11 13:49:25,541][  INFO][PID:3859344] Periodic builder liveness probe: alive
[2026-02-11 13:49:25,542][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:49:30,544][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:49:35,547][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:49:40,548][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:49:45,569][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:49:50,573][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:49:55,575][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:50:00,577][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:50:05,580][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:50:10,582][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:50:15,583][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:50:20,640][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:50:25,641][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:50:30,696][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:50:35,697][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:50:40,699][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:50:45,700][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:50:50,701][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:50:55,703][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:51:00,704][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:51:05,705][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:51:10,706][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:51:15,709][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:51:20,714][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:51:25,717][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:51:30,718][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:51:35,722][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:51:40,725][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:51:45,729][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:51:50,731][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:51:55,738][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:52:00,781][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:52:05,783][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:52:10,784][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:52:15,786][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:52:20,796][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:52:25,801][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:52:30,806][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:52:35,820][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:52:40,827][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:52:45,835][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:52:50,836][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:52:55,837][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:53:00,844][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:53:05,848][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:53:10,852][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:53:15,856][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:53:20,858][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:53:25,862][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:53:30,865][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:53:35,867][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:53:40,873][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:53:45,902][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:53:50,906][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:53:55,908][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:54:00,912][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:54:05,913][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:54:10,918][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:54:15,923][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:54:20,940][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:54:25,944][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:54:30,953][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:54:35,963][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:54:40,974][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:54:45,979][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:54:50,984][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:54:55,989][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:55:00,990][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:55:05,994][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:55:07,467][ ERROR][PID:3859344] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-11 13:55:10,996][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:55:16,039][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:55:17,468][WARNING][PID:3859344] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-11 13:55:21,043][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:55:26,048][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:55:27,469][  INFO][PID:3859344] Downloading the builder-live.log file, attempt 2
[2026-02-11 13:55:27,476][  INFO][PID:3859344] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c150 copr-rpmbuild-log
[2026-02-11 13:55:31,054][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:55:36,057][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:55:41,060][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:55:46,074][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:55:51,141][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:55:56,305][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:56:01,382][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:56:06,389][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:56:11,435][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:56:16,438][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:56:21,443][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:56:26,450][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:56:27,570][ ERROR][PID:3859344] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-11 13:56:31,451][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:56:36,463][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:56:37,770][WARNING][PID:3859344] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-11 13:56:41,465][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:56:46,469][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:56:47,773][  INFO][PID:3859344] Downloading the builder-live.log file, attempt 3
[2026-02-11 13:56:47,775][  INFO][PID:3859344] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c150 copr-rpmbuild-log
[2026-02-11 13:56:51,471][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:56:56,540][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:57:01,542][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:57:06,561][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:57:11,615][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:57:16,645][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:57:21,672][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:57:26,680][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:57:31,719][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:57:36,815][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:57:41,866][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:57:46,885][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:57:47,857][ ERROR][PID:3859344] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-11 13:57:51,887][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:57:56,892][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:57:57,996][WARNING][PID:3859344] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-11 13:58:01,893][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:58:06,916][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:58:07,997][  INFO][PID:3859344] Downloading the builder-live.log file, attempt 4
[2026-02-11 13:58:08,002][  INFO][PID:3859344] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c150 copr-rpmbuild-log
[2026-02-11 13:58:11,927][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:58:16,930][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:58:21,933][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:58:26,935][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:58:31,944][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:58:36,946][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:58:41,947][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:58:46,950][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:58:51,957][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:58:56,960][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:59:01,965][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:59:07,105][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:59:08,080][ ERROR][PID:3859344] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-11 13:59:12,111][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:59:17,116][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:59:18,088][ ERROR][PID:3859344] Giving up for unstable SSH, failures: 1770818117.4687111, 1770818197.7707267, 1770818277.9968276, 1770818358.0884023
[2026-02-11 13:59:18,090][ ERROR][PID:3859344] Re-try request for task on 'ResallocHost, ticket_id=1275201, hostname=2620:52:6:1161:dead:beef:cafe:c150, name=vmhost_p09_03_prod_02162095_20260211_134348, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-02-11 13:59:18,091][  INFO][PID:3859344] Releasing VM back to pool
[2026-02-11 13:59:18,116][  INFO][PID:3859344] Retry #1 (on other host)
[2026-02-11 13:59:18,120][  INFO][PID:3859344] Checking that /var/lib/copr/public_html/results/dreua/trash-cli/epel-8-ppc64le/repodata/repomd.xml exists
[2026-02-11 13:59:18,142][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:59:18,144][  INFO][PID:3859344] VM allocation process starts
[2026-02-11 13:59:18,179][  INFO][PID:3859344] Trying to allocate VM: ResallocHost, ticket_id=1275406, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-11 13:59:23,182][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:59:28,193][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:59:30,299][  INFO][PID:3859344] Allocated host ResallocHost, ticket_id=1275406, hostname=2620:52:6:1161:dead:beef:cafe:c154, name=vmhost_p09_03_prod_02163433_20260211_135344, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-11 13:59:30,302][  INFO][PID:3859344] Allocating ssh connection to builder
[2026-02-11 13:59:30,304][  INFO][PID:3859344] Checking that builder machine is OK
[2026-02-11 13:59:31,422][  INFO][PID:3859344] Running remote command: copr-builder-ready epel-8-ppc64le
[2026-02-11 13:59:31,972][  INFO][PID:3859344] Checking Red Hat subscription (attempt #1)
Red Hat subscription active
Builder is ready to be used

[2026-02-11 13:59:31,972][  INFO][PID:3859344] Filling build.info file with builder info
[2026-02-11 13:59:32,239][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:59:32,241][  INFO][PID:3859344] Not re-notifying FE and msg buses for the new host.
[2026-02-11 13:59:32,242][  INFO][PID:3859344] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10116442-epel-8-ppc64le --chroot epel-8-ppc64le --detached
[2026-02-11 13:59:33,210][  INFO][PID:3859344] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c154' (ED25519) to the list of known hosts.


[2026-02-11 13:59:33,210][  INFO][PID:3859344] Downloading the builder-live.log file, attempt 1
[2026-02-11 13:59:33,216][  INFO][PID:3859344] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c154 copr-rpmbuild-log
[2026-02-11 13:59:38,242][  INFO][PID:3859344] Periodic builder liveness probe: alive
[2026-02-11 13:59:38,243][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:59:43,258][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:59:48,263][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:59:53,266][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 13:59:58,268][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:00:03,311][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:00:08,317][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:00:13,320][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:00:18,321][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:00:23,325][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:00:28,360][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:00:33,375][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:00:38,376][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:00:43,381][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:00:48,385][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:00:53,389][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:00:58,401][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:01:03,402][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:01:08,405][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:01:13,409][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:01:18,417][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:01:23,419][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:01:28,420][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:01:33,422][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:01:38,460][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:01:43,476][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:01:48,539][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:01:53,540][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:01:58,542][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:02:03,544][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:02:08,549][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:02:13,553][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:02:18,558][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:02:23,560][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:02:28,566][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:02:33,568][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:02:38,569][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:02:43,573][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:02:48,579][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:02:53,585][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:02:58,588][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:03:03,593][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:03:08,602][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:03:13,609][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:03:18,613][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:03:23,617][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:03:28,662][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:03:33,668][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:03:38,669][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:03:43,670][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:03:48,672][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:03:53,677][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:03:58,679][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:04:03,684][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:04:08,690][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:04:13,694][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:04:18,695][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:04:23,698][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:04:28,702][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:04:33,709][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:04:38,742][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:04:43,744][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:04:48,747][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:04:53,749][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:04:58,753][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:05:03,755][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:05:08,433][ ERROR][PID:3859344] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-11 14:05:08,758][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:05:13,759][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:05:18,435][WARNING][PID:3859344] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-11 14:05:18,766][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:05:23,769][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:05:28,436][  INFO][PID:3859344] Downloading the builder-live.log file, attempt 2
[2026-02-11 14:05:28,438][  INFO][PID:3859344] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c154 copr-rpmbuild-log
[2026-02-11 14:05:28,777][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:05:33,781][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:05:38,785][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:05:43,790][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:05:48,795][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:05:53,801][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:05:58,803][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:06:03,808][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:06:08,813][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:06:13,816][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:06:18,817][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:06:23,821][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:06:28,514][ ERROR][PID:3859344] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-11 14:06:28,823][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:06:33,825][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:06:38,553][WARNING][PID:3859344] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-11 14:06:38,830][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:06:43,834][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:06:48,555][  INFO][PID:3859344] Downloading the builder-live.log file, attempt 3
[2026-02-11 14:06:48,560][  INFO][PID:3859344] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c154 copr-rpmbuild-log
[2026-02-11 14:06:48,852][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:06:53,856][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:06:58,859][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:07:04,040][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:07:09,047][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:07:14,052][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:07:19,117][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:07:24,248][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:07:29,306][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:07:34,332][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:07:39,342][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:07:44,347][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:07:48,655][ ERROR][PID:3859344] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-11 14:07:49,356][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:07:54,368][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:07:58,716][WARNING][PID:3859344] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-11 14:07:59,370][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:08:04,378][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:08:08,717][  INFO][PID:3859344] Downloading the builder-live.log file, attempt 4
[2026-02-11 14:08:08,721][  INFO][PID:3859344] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c154 copr-rpmbuild-log
[2026-02-11 14:08:09,482][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:08:14,486][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:08:19,488][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:08:24,492][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:08:29,498][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:08:34,500][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:08:39,504][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:08:44,523][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:08:49,524][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:08:54,525][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:08:59,529][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:09:04,625][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:09:08,793][ ERROR][PID:3859344] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-11 14:09:09,628][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:09:14,629][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:09:18,794][ ERROR][PID:3859344] Giving up for unstable SSH, failures: 1770818718.4353325, 1770818798.5537808, 1770818878.7159512, 1770818958.794803
[2026-02-11 14:09:18,796][ ERROR][PID:3859344] Re-try request for task on 'ResallocHost, ticket_id=1275406, hostname=2620:52:6:1161:dead:beef:cafe:c154, name=vmhost_p09_03_prod_02163433_20260211_135344, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-02-11 14:09:18,797][  INFO][PID:3859344] Releasing VM back to pool
[2026-02-11 14:09:18,831][  INFO][PID:3859344] Retry #2 (on other host)
[2026-02-11 14:09:18,832][  INFO][PID:3859344] Checking that /var/lib/copr/public_html/results/dreua/trash-cli/epel-8-ppc64le/repodata/repomd.xml exists
[2026-02-11 14:09:18,835][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:09:18,838][  INFO][PID:3859344] VM allocation process starts
[2026-02-11 14:09:18,875][  INFO][PID:3859344] Trying to allocate VM: ResallocHost, ticket_id=1275611, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-11 14:09:21,929][  INFO][PID:3859344] Allocated host ResallocHost, ticket_id=1275611, hostname=2620:52:6:1161:dead:beef:cafe:c188, name=vmhost_p09_04_prod_02165611_20260211_140403, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-11 14:09:21,932][  INFO][PID:3859344] Allocating ssh connection to builder
[2026-02-11 14:09:21,934][  INFO][PID:3859344] Checking that builder machine is OK
[2026-02-11 14:09:23,098][  INFO][PID:3859344] Running remote command: copr-builder-ready epel-8-ppc64le
[2026-02-11 14:09:23,898][  INFO][PID:3859344] Checking Red Hat subscription (attempt #1)
Red Hat subscription active
Builder is ready to be used

[2026-02-11 14:09:23,903][  INFO][PID:3859344] Filling build.info file with builder info
[2026-02-11 14:09:24,045][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:09:24,047][  INFO][PID:3859344] Not re-notifying FE and msg buses for the new host.
[2026-02-11 14:09:24,049][  INFO][PID:3859344] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10116442-epel-8-ppc64le --chroot epel-8-ppc64le --detached
[2026-02-11 14:09:25,219][  INFO][PID:3859344] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c188' (ED25519) to the list of known hosts.


[2026-02-11 14:09:25,223][  INFO][PID:3859344] Downloading the builder-live.log file, attempt 1
[2026-02-11 14:09:25,243][  INFO][PID:3859344] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c188 copr-rpmbuild-log
[2026-02-11 14:09:30,224][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:09:35,234][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:09:40,274][  INFO][PID:3859344] Periodic builder liveness probe: alive
[2026-02-11 14:09:40,285][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:09:45,288][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:09:50,290][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:09:55,294][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:10:00,300][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:10:05,423][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:10:10,429][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:10:15,432][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:10:20,451][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:10:25,452][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:10:30,466][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:10:35,469][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:10:40,471][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:10:45,473][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:10:50,596][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:10:55,649][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:11:00,651][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:11:05,652][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:11:10,656][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:11:15,664][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:11:20,671][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:11:25,672][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:11:30,680][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:11:35,687][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:11:40,695][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:11:45,696][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:11:50,701][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:11:55,708][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:12:00,709][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:12:05,719][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:12:10,721][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:12:15,724][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:12:20,726][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:12:25,730][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:12:30,733][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:12:35,394][ ERROR][PID:3859344] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-11 14:12:35,738][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:12:40,739][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:12:45,474][WARNING][PID:3859344] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-11 14:12:45,740][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:12:50,743][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:12:55,482][  INFO][PID:3859344] Downloading the builder-live.log file, attempt 2
[2026-02-11 14:12:55,518][  INFO][PID:3859344] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c188 copr-rpmbuild-log
[2026-02-11 14:12:55,746][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:13:00,750][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:13:05,753][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:13:10,824][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:13:15,825][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:13:20,827][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:13:25,851][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:13:30,854][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:13:35,858][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:13:40,863][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:13:45,870][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:13:50,873][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:13:55,875][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:14:01,079][  INFO][PID:3859344] Checking for cancel request
[2026-02-11 14:14:01,359][  INFO][PID:3859344] Downloading results from builder
[2026-02-11 14:14:01,360][  INFO][PID:3859344] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c188]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/dreua/trash-cli/epel-8-ppc64le/10116442-trash-cli started
[2026-02-11 14:14:01,362][  INFO][PID:3859344] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c188]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/dreua/trash-cli/epel-8-ppc64le/10116442-trash-cli/ &> /var/lib/copr/public_html/results/dreua/trash-cli/epel-8-ppc64le/10116442-trash-cli/build-10116442.rsync.log
[2026-02-11 14:14:02,170][  INFO][PID:3859344] rsyncing finished.
[2026-02-11 14:14:02,171][  INFO][PID:3859344] Releasing VM back to pool
[2026-02-11 14:14:02,187][  INFO][PID:3859344] Searching for 'success' file in resultdir
[2026-02-11 14:14:02,188][ ERROR][PID:3859344] Build failed: Backend process error: No success file => build failure
[2026-02-11 14:14:02,192][  INFO][PID:3859344] Finished build: id=10116442 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/dreua/trash-cli chroot=epel-8-ppc64le 
[2026-02-11 14:14:02,195][ ERROR][PID:3859344] Backend process error: No success file => build failure
[2026-02-11 14:14:02,198][  INFO][PID:3859344] Worker failed build, took 278.15163016319275
[2026-02-11 14:14:02,200][  INFO][PID:3859344] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "dreua",
            "project_name": "trash-cli",
            "project_dirname": "trash-cli",
            "submitter": "https://src.fedoraproject.org/user/salimma",
            "ended_on": 1770819242.1988554,
            "started_on": 1770818964.0472252,
            "submitted_on": null,
            "status": 0,
            "chroot": "epel-8-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "10116442-epel-8-ppc64le",
            "build_id": 10116442,
            "package_name": "trash-cli",
            "package_version": "0.24.5.26-18",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/dreua/trash-cli/trash-cli",
            "git_hash": "c22c9b85edf7aaebae3e6b45b63349c020b896f5",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "dreua/trash-cli--https://src.fedoraproject.org/user/salimma",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/dreua/trash-cli/epel-8-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/dreua/trash-cli",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/dreua/trash-cli",
            "result_dir": "10116442-trash-cli",
            "built_packages": "",
            "id": 10116442,
            "mockchain_macros": {
                "copr_username": "dreua",
                "copr_projectname": "trash-cli",
                "vendor": "Fedora Project COPR (dreua/trash-cli)"
            }
        }
    ]
}
[2026-02-11 14:14:02,253][  INFO][PID:3859344] Sending fedora-messaging bus message in build.end
[2026-02-11 14:14:02,288][  INFO][PID:3859344] Compressing /var/lib/copr/public_html/results/dreua/trash-cli/epel-8-ppc64le/10116442-trash-cli/builder-live.log by gzip
[2026-02-11 14:14:02,293][  INFO][PID:3859344] Running command 'gzip /var/lib/copr/public_html/results/dreua/trash-cli/epel-8-ppc64le/10116442-trash-cli/builder-live.log' as PID 4097584
[2026-02-11 14:14:02,428][  INFO][PID:3859344] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/dreua/trash-cli/epel-8-ppc64le/10116442-trash-cli/builder-live.log)
[2026-02-11 14:14:02,429][  INFO][PID:3859344] Compressing /var/lib/copr/public_html/results/dreua/trash-cli/epel-8-ppc64le/10116442-trash-cli/backend.log by gzip
[2026-02-11 14:14:02,437][  INFO][PID:3859344] Running command 'gzip /var/lib/copr/public_html/results/dreua/trash-cli/epel-8-ppc64le/10116442-trash-cli/backend.log' as PID 4097599