[2026-03-26 14:18:13,684][  INFO][PID:3172468] Marking build as starting
[2026-03-26 14:18:13,935][  INFO][PID:3172468] Checking that /var/lib/copr/public_html/results/leo/86box/fedora-42-x86_64/repodata/repomd.xml exists
[2026-03-26 14:18:13,975][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:18:13,977][  INFO][PID:3172468] VM allocation process starts
[2026-03-26 14:18:14,008][  INFO][PID:3172468] Trying to allocate VM: ResallocHost, ticket_id=2041433, requested_tags=['copr_builder', 'arch_x86_64']
[2026-03-26 14:18:17,040][  INFO][PID:3172468] Allocated host ResallocHost, ticket_id=2041433, hostname=3.84.32.120, name=aws_x86_64_spot_prod_04293156_20260326_141617, requested_tags=['copr_builder', 'arch_x86_64']
[2026-03-26 14:18:17,040][  INFO][PID:3172468] Allocating ssh connection to builder
[2026-03-26 14:18:17,041][  INFO][PID:3172468] Checking that builder machine is OK
[2026-03-26 14:18:17,376][  INFO][PID:3172468] Running remote command: copr-builder-ready fedora-42-x86_64
[2026-03-26 14:18:17,595][  INFO][PID:3172468] Red Hat subscription not needed for fedora-42-x86_64
Builder is ready to be used

[2026-03-26 14:18:17,596][  INFO][PID:3172468] Filling build.info file with builder info
[2026-03-26 14:18:17,597][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:18:17,598][  INFO][PID:3172468] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "leo",
            "project_name": "86box",
            "project_dirname": "86box",
            "submitter": "https://pagure.io/user/leo",
            "ended_on": null,
            "started_on": 1774534697.5985904,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-42-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "10264081-fedora-42-x86_64",
            "build_id": 10264081,
            "package_name": "86Box",
            "package_version": "5.3-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/leo/86box/86Box",
            "git_hash": "2ee42410f7054516ce816cce478c9d3934b5666c",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "leo/86box--https://pagure.io/user/leo",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/leo/86box/fedora-42-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/leo/86box",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/leo/86box",
            "result_dir": "10264081-86Box",
            "built_packages": "",
            "id": 10264081,
            "mockchain_macros": {
                "copr_username": "leo",
                "copr_projectname": "86box",
                "vendor": "Fedora Project COPR (leo/86box)"
            }
        }
    ]
}
[2026-03-26 14:18:17,804][  INFO][PID:3172468] Sending fedora-messaging bus message in build.start
[2026-03-26 14:18:18,508][  INFO][PID:3172468] Sending fedora-messaging bus message in chroot.start
[2026-03-26 14:18:18,554][  INFO][PID:3172468] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10264081-fedora-42-x86_64 --chroot fedora-42-x86_64 --detached
[2026-03-26 14:18:18,925][  INFO][PID:3172468] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '3.84.32.120' (ED25519) to the list of known hosts.


[2026-03-26 14:18:18,927][  INFO][PID:3172468] Downloading the builder-live.log file, attempt 1
[2026-03-26 14:18:18,929][  INFO][PID:3172468] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.84.32.120 copr-rpmbuild-log
[2026-03-26 14:18:23,937][  INFO][PID:3172468] Periodic builder liveness probe: alive
[2026-03-26 14:18:23,938][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:18:28,939][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:18:33,941][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:18:38,944][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:18:43,945][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:18:48,946][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:18:53,948][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:18:58,950][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:19:03,951][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:19:08,952][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:19:13,953][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:19:18,954][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:19:23,956][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:19:28,959][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:19:33,961][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:19:38,963][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:19:43,964][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:19:48,965][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:19:53,968][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:19:58,969][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:20:03,987][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:20:05,516][ ERROR][PID:3172468] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 14:20:08,990][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:20:13,992][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:20:15,520][WARNING][PID:3172468] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-03-26 14:20:18,994][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:20:23,996][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:20:25,522][  INFO][PID:3172468] Downloading the builder-live.log file, attempt 2
[2026-03-26 14:20:25,527][  INFO][PID:3172468] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.84.32.120 copr-rpmbuild-log
[2026-03-26 14:20:29,000][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:20:34,001][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:20:39,003][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:20:44,005][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:20:49,009][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:20:54,011][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:20:59,018][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:21:04,020][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:21:09,025][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:21:14,027][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:21:19,028][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:21:24,029][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:21:25,605][ ERROR][PID:3172468] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 14:21:29,030][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:21:34,031][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:21:35,623][WARNING][PID:3172468] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-03-26 14:21:39,034][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:21:44,036][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:21:45,625][  INFO][PID:3172468] Downloading the builder-live.log file, attempt 3
[2026-03-26 14:21:45,627][  INFO][PID:3172468] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.84.32.120 copr-rpmbuild-log
[2026-03-26 14:21:49,044][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:21:54,046][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:21:59,048][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:22:04,049][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:22:09,050][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:22:14,053][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:22:19,056][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:22:24,065][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:22:29,067][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:22:34,071][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:22:39,072][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:22:44,073][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:22:45,700][ ERROR][PID:3172468] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 14:22:49,075][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:22:54,077][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:22:55,831][WARNING][PID:3172468] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-03-26 14:22:59,089][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:23:04,176][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:23:05,833][  INFO][PID:3172468] Downloading the builder-live.log file, attempt 4
[2026-03-26 14:23:05,834][  INFO][PID:3172468] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.84.32.120 copr-rpmbuild-log
[2026-03-26 14:23:09,183][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:23:14,264][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:23:19,273][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:23:24,275][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:23:29,278][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:23:34,279][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:23:39,283][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:23:44,288][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:23:49,290][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:23:54,294][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:23:59,295][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:24:04,296][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:24:05,905][ ERROR][PID:3172468] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 14:24:09,322][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:24:14,324][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:24:15,992][ ERROR][PID:3172468] Giving up for unstable SSH, failures: 1774534815.5209117, 1774534895.623576, 1774534975.83154, 1774535055.9924917
[2026-03-26 14:24:15,994][ ERROR][PID:3172468] Re-try request for task on 'ResallocHost, ticket_id=2041433, hostname=3.84.32.120, name=aws_x86_64_spot_prod_04293156_20260326_141617, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-03-26 14:24:15,995][  INFO][PID:3172468] Releasing VM back to pool
[2026-03-26 14:24:16,015][  INFO][PID:3172468] Retry #1 (on other host)
[2026-03-26 14:24:16,016][  INFO][PID:3172468] Checking that /var/lib/copr/public_html/results/leo/86box/fedora-42-x86_64/repodata/repomd.xml exists
[2026-03-26 14:24:16,017][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:24:16,018][  INFO][PID:3172468] VM allocation process starts
[2026-03-26 14:24:16,031][  INFO][PID:3172468] Trying to allocate VM: ResallocHost, ticket_id=2041544, requested_tags=['copr_builder', 'arch_x86_64']
[2026-03-26 14:24:19,072][  INFO][PID:3172468] Allocated host ResallocHost, ticket_id=2041544, hostname=3.89.126.176, name=aws_x86_64_spot_prod_04293161_20260326_141621, requested_tags=['copr_builder', 'arch_x86_64']
[2026-03-26 14:24:19,073][  INFO][PID:3172468] Allocating ssh connection to builder
[2026-03-26 14:24:19,073][  INFO][PID:3172468] Checking that builder machine is OK
[2026-03-26 14:24:19,448][  INFO][PID:3172468] Running remote command: copr-builder-ready fedora-42-x86_64
[2026-03-26 14:24:19,700][  INFO][PID:3172468] Red Hat subscription not needed for fedora-42-x86_64
Builder is ready to be used

[2026-03-26 14:24:19,701][  INFO][PID:3172468] Filling build.info file with builder info
[2026-03-26 14:24:19,722][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:24:19,724][  INFO][PID:3172468] Not re-notifying FE and msg buses for the new host.
[2026-03-26 14:24:19,725][  INFO][PID:3172468] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10264081-fedora-42-x86_64 --chroot fedora-42-x86_64 --detached
[2026-03-26 14:24:20,069][  INFO][PID:3172468] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '3.89.126.176' (ED25519) to the list of known hosts.


[2026-03-26 14:24:20,071][  INFO][PID:3172468] Downloading the builder-live.log file, attempt 1
[2026-03-26 14:24:20,075][  INFO][PID:3172468] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.89.126.176 copr-rpmbuild-log
[2026-03-26 14:24:25,072][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:24:30,074][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:24:35,085][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:24:40,086][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:24:45,088][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:24:50,090][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:24:55,091][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:25:00,092][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:25:05,094][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:25:10,095][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:25:15,096][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:25:20,097][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:25:25,098][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:25:30,101][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:25:35,104][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:25:40,105][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:25:45,107][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:25:50,108][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:25:55,110][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:26:00,116][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:26:05,117][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:26:10,118][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:26:15,119][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:26:20,121][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:26:25,124][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:26:30,125][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:26:35,127][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:26:40,132][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:26:45,135][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:26:50,140][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:26:55,141][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:26:55,880][ ERROR][PID:3172468] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 14:27:00,143][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:27:05,148][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:27:05,893][WARNING][PID:3172468] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-03-26 14:27:10,150][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:27:15,156][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:27:15,894][  INFO][PID:3172468] Downloading the builder-live.log file, attempt 2
[2026-03-26 14:27:15,898][  INFO][PID:3172468] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.89.126.176 copr-rpmbuild-log
[2026-03-26 14:27:20,158][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:27:25,160][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:27:30,163][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:27:35,168][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:27:40,170][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:27:45,173][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:27:50,177][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:27:55,179][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:28:00,181][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:28:05,182][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:28:10,185][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:28:15,189][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:28:15,970][ ERROR][PID:3172468] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 14:28:20,191][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:28:25,200][  INFO][PID:3172468] Periodic builder liveness probe: alive
[2026-03-26 14:28:25,201][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:28:25,983][WARNING][PID:3172468] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-03-26 14:28:30,203][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:28:35,205][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:28:35,984][  INFO][PID:3172468] Downloading the builder-live.log file, attempt 3
[2026-03-26 14:28:35,986][  INFO][PID:3172468] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.89.126.176 copr-rpmbuild-log
[2026-03-26 14:28:40,207][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:28:45,208][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:28:50,217][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:28:55,226][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:29:00,254][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:29:05,255][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:29:10,257][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:29:15,259][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:29:20,261][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:29:25,263][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:29:30,266][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:29:35,268][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:29:36,054][ ERROR][PID:3172468] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 14:29:40,272][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:29:45,274][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:29:46,105][WARNING][PID:3172468] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-03-26 14:29:50,278][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:29:55,280][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:29:56,108][  INFO][PID:3172468] Downloading the builder-live.log file, attempt 4
[2026-03-26 14:29:56,111][  INFO][PID:3172468] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.89.126.176 copr-rpmbuild-log
[2026-03-26 14:30:00,281][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:30:05,283][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:30:10,285][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:30:15,286][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:30:20,292][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:30:25,293][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:30:30,295][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:30:35,296][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:30:40,297][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:30:45,299][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:30:50,301][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:30:55,302][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:30:56,181][ ERROR][PID:3172468] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 14:31:00,303][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:31:05,304][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:31:06,206][ ERROR][PID:3172468] Giving up for unstable SSH, failures: 1774535225.8937144, 1774535305.9837925, 1774535386.1056569, 1774535466.2060285
[2026-03-26 14:31:06,207][ ERROR][PID:3172468] Re-try request for task on 'ResallocHost, ticket_id=2041544, hostname=3.89.126.176, name=aws_x86_64_spot_prod_04293161_20260326_141621, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-03-26 14:31:06,207][  INFO][PID:3172468] Releasing VM back to pool
[2026-03-26 14:31:06,220][  INFO][PID:3172468] Retry #2 (on other host)
[2026-03-26 14:31:06,221][  INFO][PID:3172468] Checking that /var/lib/copr/public_html/results/leo/86box/fedora-42-x86_64/repodata/repomd.xml exists
[2026-03-26 14:31:06,226][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:31:06,227][  INFO][PID:3172468] VM allocation process starts
[2026-03-26 14:31:06,237][  INFO][PID:3172468] Trying to allocate VM: ResallocHost, ticket_id=2041636, requested_tags=['copr_builder', 'arch_x86_64']
[2026-03-26 14:31:09,248][  INFO][PID:3172468] Allocated host ResallocHost, ticket_id=2041636, hostname=3.83.133.40, name=aws_x86_64_spot_prod_04294116_20260326_142947, requested_tags=['copr_builder', 'arch_x86_64']
[2026-03-26 14:31:09,249][  INFO][PID:3172468] Allocating ssh connection to builder
[2026-03-26 14:31:09,249][  INFO][PID:3172468] Checking that builder machine is OK
[2026-03-26 14:31:09,570][  INFO][PID:3172468] Running remote command: copr-builder-ready fedora-42-x86_64
[2026-03-26 14:31:09,772][  INFO][PID:3172468] Red Hat subscription not needed for fedora-42-x86_64
Builder is ready to be used

[2026-03-26 14:31:09,773][  INFO][PID:3172468] Filling build.info file with builder info
[2026-03-26 14:31:09,788][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:31:09,789][  INFO][PID:3172468] Not re-notifying FE and msg buses for the new host.
[2026-03-26 14:31:09,790][  INFO][PID:3172468] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10264081-fedora-42-x86_64 --chroot fedora-42-x86_64 --detached
[2026-03-26 14:31:10,112][  INFO][PID:3172468] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '3.83.133.40' (ED25519) to the list of known hosts.


[2026-03-26 14:31:10,113][  INFO][PID:3172468] Downloading the builder-live.log file, attempt 1
[2026-03-26 14:31:10,114][  INFO][PID:3172468] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.83.133.40 copr-rpmbuild-log
[2026-03-26 14:31:15,113][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:31:20,114][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:31:25,115][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:31:30,117][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:31:35,118][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:31:40,119][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:31:45,120][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:31:50,121][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:31:55,124][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:32:00,125][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:32:05,128][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:32:10,131][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:32:15,132][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:32:20,134][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:32:25,136][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:32:30,137][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:32:35,138][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:32:40,139][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:32:45,141][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:32:50,142][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:32:55,143][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:33:00,144][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:33:05,145][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:33:10,146][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:33:15,147][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:33:20,148][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:33:25,149][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:33:29,164][ ERROR][PID:3172468] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 14:33:30,150][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:33:35,151][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:33:39,172][WARNING][PID:3172468] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-03-26 14:33:40,153][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:33:45,155][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:33:49,173][  INFO][PID:3172468] Downloading the builder-live.log file, attempt 2
[2026-03-26 14:33:49,176][  INFO][PID:3172468] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.83.133.40 copr-rpmbuild-log
[2026-03-26 14:33:50,158][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:33:55,159][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:34:00,166][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:34:05,170][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:34:10,173][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:34:15,175][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:34:20,177][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:34:25,178][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:34:30,245][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:34:35,253][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:34:40,255][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:34:45,257][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:34:49,250][ ERROR][PID:3172468] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 14:34:50,259][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:34:55,262][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:34:59,256][WARNING][PID:3172468] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-03-26 14:35:00,263][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:35:05,264][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:35:09,257][  INFO][PID:3172468] Downloading the builder-live.log file, attempt 3
[2026-03-26 14:35:09,258][  INFO][PID:3172468] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.83.133.40 copr-rpmbuild-log
[2026-03-26 14:35:10,266][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:35:15,268][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:35:20,269][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:35:25,271][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:35:30,272][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:35:35,273][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:35:40,274][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:35:45,275][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:35:50,277][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:35:55,278][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:36:00,282][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:36:05,284][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:36:09,330][ ERROR][PID:3172468] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 14:36:10,285][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:36:15,286][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:36:19,350][WARNING][PID:3172468] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-03-26 14:36:20,287][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:36:25,288][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:36:29,351][  INFO][PID:3172468] Downloading the builder-live.log file, attempt 4
[2026-03-26 14:36:29,352][  INFO][PID:3172468] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.83.133.40 copr-rpmbuild-log
[2026-03-26 14:36:30,290][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:36:35,291][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:36:40,292][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:36:45,293][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:36:50,294][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:36:55,296][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:37:00,297][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:37:05,298][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:37:10,299][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:37:15,300][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:37:20,301][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:37:25,303][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:37:29,420][ ERROR][PID:3172468] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 14:37:30,304][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:37:35,306][  INFO][PID:3172468] Checking for cancel request
[2026-03-26 14:37:39,425][ ERROR][PID:3172468] Giving up for unstable SSH, failures: 1774535619.1725428, 1774535699.256361, 1774535779.3506386, 1774535859.4256465
[2026-03-26 14:37:39,426][ ERROR][PID:3172468] Re-try request for task on 'ResallocHost, ticket_id=2041636, hostname=3.83.133.40, name=aws_x86_64_spot_prod_04294116_20260326_142947, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-03-26 14:37:39,427][  INFO][PID:3172468] Releasing VM back to pool
[2026-03-26 14:37:39,468][ ERROR][PID:3172468] Backend process error: Three host tried without success: {'3.89.126.176', '3.84.32.120', '3.83.133.40'}
[2026-03-26 14:37:39,469][WARNING][PID:3172468] Switching not-finished job state to 'failed'
[2026-03-26 14:37:39,469][  INFO][PID:3172468] Worker failed build, took 389.6795275211334
[2026-03-26 14:37:39,470][  INFO][PID:3172468] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "leo",
            "project_name": "86box",
            "project_dirname": "86box",
            "submitter": "https://pagure.io/user/leo",
            "ended_on": 1774535859.469073,
            "started_on": 1774535469.7895455,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-42-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "10264081-fedora-42-x86_64",
            "build_id": 10264081,
            "package_name": "86Box",
            "package_version": "5.3-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/leo/86box/86Box",
            "git_hash": "2ee42410f7054516ce816cce478c9d3934b5666c",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "leo/86box--https://pagure.io/user/leo",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/leo/86box/fedora-42-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/leo/86box",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/leo/86box",
            "result_dir": "10264081-86Box",
            "built_packages": "",
            "id": 10264081,
            "mockchain_macros": {
                "copr_username": "leo",
                "copr_projectname": "86box",
                "vendor": "Fedora Project COPR (leo/86box)"
            }
        }
    ]
}
[2026-03-26 14:37:39,709][  INFO][PID:3172468] Sending fedora-messaging bus message in build.end
[2026-03-26 14:37:39,752][  INFO][PID:3172468] Compressing /var/lib/copr/public_html/results/leo/86box/fedora-42-x86_64/10264081-86Box/builder-live.log by gzip
[2026-03-26 14:37:39,754][  INFO][PID:3172468] Running command 'gzip /var/lib/copr/public_html/results/leo/86box/fedora-42-x86_64/10264081-86Box/builder-live.log' as PID 3323829
[2026-03-26 14:37:39,756][  INFO][PID:3172468] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/leo/86box/fedora-42-x86_64/10264081-86Box/builder-live.log)
[2026-03-26 14:37:39,756][  INFO][PID:3172468] Compressing /var/lib/copr/public_html/results/leo/86box/fedora-42-x86_64/10264081-86Box/backend.log by gzip
[2026-03-26 14:37:39,757][  INFO][PID:3172468] Running command 'gzip /var/lib/copr/public_html/results/leo/86box/fedora-42-x86_64/10264081-86Box/backend.log' as PID 3323830