[2025-09-11 15:56:28,655][  INFO][PID:3764215] Marking build as starting
[2025-09-11 15:56:28,827][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:56:28,828][  INFO][PID:3764215] VM allocation process starts
[2025-09-11 15:56:28,847][  INFO][PID:3764215] Trying to allocate VM: ResallocHost, ticket_id=5009718, requested_tags=['copr_builder', 'arch_x86_64']
[2025-09-11 15:56:33,849][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:56:38,851][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:56:43,854][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:56:48,855][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:56:53,857][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:56:58,859][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:57:03,860][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:57:08,861][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:57:13,863][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:57:18,864][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:57:23,865][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:57:28,866][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:57:33,867][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:57:38,868][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:57:43,869][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:57:48,870][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:57:53,872][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:57:58,873][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:58:03,877][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:58:08,878][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:58:13,880][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:58:18,881][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:58:23,883][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:58:28,887][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:58:33,891][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:58:38,895][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:58:43,897][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:58:48,899][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:58:53,900][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:58:58,902][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:59:03,903][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:59:08,906][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:59:13,912][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:59:18,913][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:59:23,915][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:59:28,917][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:59:33,919][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:59:38,920][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:59:43,922][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:59:48,924][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:59:53,932][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 15:59:58,933][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:00:03,937][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:00:08,938][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:00:13,940][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:00:18,943][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:00:23,944][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:00:28,945][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:00:33,947][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:00:38,949][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:00:43,951][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:00:48,952][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:00:53,954][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:00:58,957][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:01:03,958][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:01:08,959][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:01:13,962][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:01:18,971][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:01:23,972][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:01:28,974][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:01:33,982][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:01:38,984][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:01:43,987][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:01:48,988][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:01:53,989][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:01:58,991][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:02:03,992][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:02:08,993][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:02:13,995][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:02:18,996][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:02:24,000][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:02:29,002][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:02:34,004][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:02:39,005][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:02:44,008][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:02:49,010][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:02:54,011][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:02:59,014][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:03:04,015][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:03:09,017][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:03:14,022][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:03:19,023][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:03:24,025][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:03:29,026][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:03:34,027][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:03:39,032][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:03:44,033][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:03:49,034][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:03:54,036][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:03:59,038][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:04:04,043][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:04:09,051][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:04:14,053][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:04:19,054][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:04:24,055][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:04:29,057][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:04:34,059][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:04:39,060][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:04:44,062][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:04:47,082][  INFO][PID:3764215] Allocated host ResallocHost, ticket_id=5009718, hostname=3.88.198.239, name=aws_x86_64_spot_prod_06717606_20250911_160254, requested_tags=['copr_builder', 'arch_x86_64']
[2025-09-11 16:04:47,083][  INFO][PID:3764215] Allocating ssh connection to builder
[2025-09-11 16:04:47,084][  INFO][PID:3764215] Checking that builder machine is OK
[2025-09-11 16:04:47,671][  INFO][PID:3764215] Running remote command: copr-builder-ready fedora-43-x86_64
[2025-09-11 16:04:47,895][  INFO][PID:3764215] Red Hat subscription not needed for fedora-43-x86_64
Builder is ready to be used

[2025-09-11 16:04:47,895][  INFO][PID:3764215] Filling build.info file with builder info
[2025-09-11 16:04:47,899][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:04:47,900][  INFO][PID:3764215] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@python",
            "project_name": "python-in-pulp",
            "project_dirname": "python-in-pulp",
            "submitter": "https://src.fedoraproject.org/user/tstellar",
            "ended_on": null,
            "started_on": 1757606687.9006414,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-43-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "9546197-fedora-43-x86_64",
            "build_id": 9546197,
            "package_name": "llvm",
            "package_version": "21.1.1-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@python/python-in-pulp/llvm",
            "git_hash": "d2986837e304a9a77b0fe0aa39c080e0374360ae",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@python/python-in-pulp--https://src.fedoraproject.org/user/tstellar",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 1,
            "repos": [
                {
                    "baseurl": "https://console.redhat.com/api/pulp-content/public-copr/@python/python-in-pulp/fedora-43-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@python/python-in-pulp",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@python/python-in-pulp",
            "result_dir": "09546197-llvm",
            "built_packages": "",
            "id": 9546197,
            "mockchain_macros": {
                "copr_username": "@python",
                "copr_projectname": "python-in-pulp",
                "vendor": "Fedora Project COPR (@python/python-in-pulp)"
            }
        }
    ]
}
[2025-09-11 16:04:47,985][  INFO][PID:3764215] Sending fedora-messaging bus message in build.start
[2025-09-11 16:04:49,078][  INFO][PID:3764215] Sending fedora-messaging bus message in chroot.start
[2025-09-11 16:04:49,112][  INFO][PID:3764215] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9546197-fedora-43-x86_64 --chroot fedora-43-x86_64 --detached
[2025-09-11 16:04:49,470][  INFO][PID:3764215] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '3.88.198.239' (ED25519) to the list of known hosts.


[2025-09-11 16:04:49,471][  INFO][PID:3764215] Downloading the builder-live.log file, attempt 1
[2025-09-11 16:04:49,476][  INFO][PID:3764215] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.88.198.239 copr-rpmbuild-log
[2025-09-11 16:04:54,501][  INFO][PID:3764215] Periodic builder liveness probe: alive
[2025-09-11 16:04:54,502][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:04:59,504][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:05:04,505][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:05:09,508][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:05:14,510][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:05:19,512][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:05:24,513][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:05:29,514][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:05:34,515][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:05:39,516][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:05:44,517][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:05:49,519][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:05:54,521][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:05:59,525][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:06:04,526][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:06:09,528][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:06:14,532][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:06:19,534][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:06:24,535][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:06:29,536][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:06:34,537][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:06:39,543][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:06:44,547][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:06:49,549][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:06:54,554][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:06:59,555][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:07:04,556][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:07:09,558][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:07:14,559][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:07:19,561][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:07:24,562][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:07:29,563][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:07:34,564][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:07:39,565][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:07:44,567][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:07:49,569][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:07:54,574][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:07:59,578][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:08:04,581][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:08:09,588][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:08:14,590][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:08:19,591][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:08:24,592][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:08:29,594][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:08:34,597][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:08:39,599][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:08:44,601][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:08:49,603][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:08:54,604][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:08:59,606][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:09:04,608][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:09:09,609][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:09:14,610][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:09:19,614][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:09:24,616][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:09:29,617][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:09:34,621][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:09:39,622][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:09:44,625][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:09:49,626][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:09:54,630][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:09:59,636][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:10:04,638][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:10:09,642][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:10:14,643][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:10:19,644][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:10:24,645][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:10:29,646][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:10:34,648][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:10:39,649][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:10:44,650][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:10:47,790][ ERROR][PID:3764215] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:10:49,651][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:10:54,652][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:10:57,791][WARNING][PID:3764215] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 16:10:59,653][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:11:04,654][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:11:07,792][  INFO][PID:3764215] Downloading the builder-live.log file, attempt 2
[2025-09-11 16:11:07,794][  INFO][PID:3764215] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.88.198.239 copr-rpmbuild-log
[2025-09-11 16:11:09,656][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:11:14,657][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:11:19,659][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:11:24,661][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:11:29,663][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:11:34,664][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:11:39,665][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:11:44,666][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:11:49,667][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:11:54,668][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:11:59,670][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:12:04,671][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:12:07,864][ ERROR][PID:3764215] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:12:09,673][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:12:14,675][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:12:17,865][WARNING][PID:3764215] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 16:12:19,676][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:12:24,678][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:12:27,867][  INFO][PID:3764215] Downloading the builder-live.log file, attempt 3
[2025-09-11 16:12:27,869][  INFO][PID:3764215] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.88.198.239 copr-rpmbuild-log
[2025-09-11 16:12:29,681][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:12:34,682][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:12:39,684][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:12:44,687][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:12:49,689][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:12:54,694][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:12:59,705][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:13:04,707][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:13:09,708][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:13:14,709][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:13:19,710][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:13:24,713][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:13:27,943][ ERROR][PID:3764215] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:13:29,715][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:13:34,720][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:13:37,947][WARNING][PID:3764215] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 16:13:39,723][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:13:44,726][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:13:47,948][  INFO][PID:3764215] Downloading the builder-live.log file, attempt 4
[2025-09-11 16:13:47,953][  INFO][PID:3764215] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.88.198.239 copr-rpmbuild-log
[2025-09-11 16:13:49,729][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:13:54,733][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:13:59,735][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:14:04,737][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:14:09,740][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:14:14,745][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:14:19,747][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:14:24,760][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:14:29,762][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:14:34,768][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:14:39,771][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:14:44,773][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:14:48,035][ ERROR][PID:3764215] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:14:49,780][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:14:54,793][  INFO][PID:3764215] Periodic builder liveness probe: alive
[2025-09-11 16:14:54,794][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:14:58,040][ ERROR][PID:3764215] Giving up for unstable SSH, failures: 1757607057.791195, 1757607137.8656182, 1757607217.9469614, 1757607298.0403633
[2025-09-11 16:14:58,041][ ERROR][PID:3764215] Re-try request for task on 'ResallocHost, ticket_id=5009718, hostname=3.88.198.239, name=aws_x86_64_spot_prod_06717606_20250911_160254, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-09-11 16:14:58,042][  INFO][PID:3764215] Releasing VM back to pool
[2025-09-11 16:14:58,065][  INFO][PID:3764215] Retry #1 (on other host)
[2025-09-11 16:14:58,161][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:14:58,162][  INFO][PID:3764215] VM allocation process starts
[2025-09-11 16:14:58,177][  INFO][PID:3764215] Trying to allocate VM: ResallocHost, ticket_id=5010199, requested_tags=['copr_builder', 'arch_x86_64']
[2025-09-11 16:15:03,178][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:15:08,180][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:15:13,182][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:15:18,185][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:15:23,186][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:15:28,191][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:15:33,195][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:15:38,196][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:15:43,201][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:15:48,208][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:15:53,214][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:15:58,219][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:16:03,224][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:16:08,237][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:16:13,241][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:16:18,242][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:16:23,247][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:16:28,254][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:16:33,257][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:16:38,273][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:16:43,280][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:16:48,286][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:16:53,296][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:16:58,297][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:17:03,301][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:17:08,302][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:17:13,305][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:17:18,307][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:17:23,310][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:17:28,313][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:17:33,317][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:17:38,321][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:17:43,323][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:17:48,326][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:17:53,331][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:17:58,348][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:18:03,353][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:18:08,358][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:18:13,360][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:18:18,367][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:18:23,368][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:18:28,371][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:18:33,378][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:18:38,380][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:18:43,387][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:18:48,394][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:18:53,398][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:18:58,406][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:19:03,410][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:19:08,415][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:19:13,423][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:19:18,424][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:19:23,426][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:19:28,430][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:19:33,432][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:19:38,433][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:19:43,436][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:19:48,438][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:19:53,446][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:19:58,447][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:20:03,449][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:20:08,450][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:20:13,455][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:20:18,463][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:20:23,473][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:20:28,475][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:20:33,480][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:20:38,484][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:20:43,491][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:20:48,501][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:20:53,505][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:20:58,510][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:21:03,511][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:21:08,513][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:21:13,515][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:21:16,720][  INFO][PID:3764215] Allocated host ResallocHost, ticket_id=5010199, hostname=98.86.149.23, name=aws_x86_64_spot_prod_06718125_20250911_161824, requested_tags=['copr_builder', 'arch_x86_64']
[2025-09-11 16:21:16,722][  INFO][PID:3764215] Allocating ssh connection to builder
[2025-09-11 16:21:16,725][  INFO][PID:3764215] Checking that builder machine is OK
[2025-09-11 16:21:17,187][  INFO][PID:3764215] Running remote command: copr-builder-ready fedora-43-x86_64
[2025-09-11 16:21:17,474][  INFO][PID:3764215] Red Hat subscription not needed for fedora-43-x86_64
Builder is ready to be used

[2025-09-11 16:21:17,474][  INFO][PID:3764215] Filling build.info file with builder info
[2025-09-11 16:21:17,479][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:21:17,481][  INFO][PID:3764215] Not re-notifying FE and msg buses for the new host.
[2025-09-11 16:21:17,482][  INFO][PID:3764215] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9546197-fedora-43-x86_64 --chroot fedora-43-x86_64 --detached
[2025-09-11 16:21:17,881][  INFO][PID:3764215] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '98.86.149.23' (ED25519) to the list of known hosts.


[2025-09-11 16:21:17,884][  INFO][PID:3764215] Downloading the builder-live.log file, attempt 1
[2025-09-11 16:21:17,887][  INFO][PID:3764215] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.86.149.23 copr-rpmbuild-log
[2025-09-11 16:21:22,884][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:21:27,892][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:21:32,898][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:21:37,906][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:21:42,153][ ERROR][PID:3764215] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:21:42,910][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:21:47,921][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:21:52,160][WARNING][PID:3764215] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 16:21:52,929][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:21:57,938][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:22:02,170][  INFO][PID:3764215] Downloading the builder-live.log file, attempt 2
[2025-09-11 16:22:02,189][  INFO][PID:3764215] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.86.149.23 copr-rpmbuild-log
[2025-09-11 16:22:02,942][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:22:07,944][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:22:12,946][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:22:17,949][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:22:22,953][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:22:27,959][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:22:32,970][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:22:37,972][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:22:42,977][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:22:47,985][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:22:52,987][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:22:57,993][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:23:02,297][ ERROR][PID:3764215] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:23:03,000][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:23:08,008][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:23:12,300][WARNING][PID:3764215] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 16:23:13,009][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:23:18,010][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:23:22,301][  INFO][PID:3764215] Downloading the builder-live.log file, attempt 3
[2025-09-11 16:23:22,306][  INFO][PID:3764215] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.86.149.23 copr-rpmbuild-log
[2025-09-11 16:23:23,020][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:23:28,023][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:23:33,026][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:23:38,027][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:23:43,034][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:23:48,037][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:23:53,038][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:23:58,040][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:24:03,041][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:24:08,043][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:24:13,044][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:24:18,046][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:24:22,391][ ERROR][PID:3764215] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:24:23,048][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:24:28,053][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:24:32,394][WARNING][PID:3764215] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 16:24:33,060][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:24:38,062][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:24:42,395][  INFO][PID:3764215] Downloading the builder-live.log file, attempt 4
[2025-09-11 16:24:42,397][  INFO][PID:3764215] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.86.149.23 copr-rpmbuild-log
[2025-09-11 16:24:43,066][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:24:48,067][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:24:53,068][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:24:58,100][  INFO][PID:3764215] Periodic builder liveness probe: alive
[2025-09-11 16:24:58,106][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:25:03,111][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:25:08,116][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:25:13,118][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:25:18,123][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:25:23,127][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:25:28,129][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:25:33,132][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:25:38,136][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:25:42,467][ ERROR][PID:3764215] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:25:43,143][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:25:48,146][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:25:52,473][ ERROR][PID:3764215] Giving up for unstable SSH, failures: 1757607712.160245, 1757607792.3008742, 1757607872.3940198, 1757607952.4738257
[2025-09-11 16:25:52,475][ ERROR][PID:3764215] Re-try request for task on 'ResallocHost, ticket_id=5010199, hostname=98.86.149.23, name=aws_x86_64_spot_prod_06718125_20250911_161824, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-09-11 16:25:52,475][  INFO][PID:3764215] Releasing VM back to pool
[2025-09-11 16:25:52,491][  INFO][PID:3764215] Retry #2 (on other host)
[2025-09-11 16:25:52,548][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:25:52,550][  INFO][PID:3764215] VM allocation process starts
[2025-09-11 16:25:52,570][  INFO][PID:3764215] Trying to allocate VM: ResallocHost, ticket_id=5010489, requested_tags=['copr_builder', 'arch_x86_64']
[2025-09-11 16:25:57,572][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:26:02,577][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:26:07,578][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:26:12,579][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:26:17,582][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:26:22,583][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:26:27,584][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:26:32,586][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:26:37,587][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:26:42,591][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:26:47,599][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:26:52,603][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:26:57,606][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:27:02,607][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:27:07,612][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:27:12,614][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:27:17,615][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:27:22,617][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:27:27,619][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:27:32,622][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:27:37,623][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:27:42,626][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:27:47,628][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:27:52,630][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:27:57,636][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:28:02,637][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:28:07,638][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:28:12,640][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:28:17,642][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:28:22,648][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:28:27,648][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:28:32,651][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:28:37,653][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:28:42,659][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:28:47,661][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:28:52,662][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:28:57,663][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:29:02,665][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:29:07,667][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:29:10,711][  INFO][PID:3764215] Allocated host ResallocHost, ticket_id=5010489, hostname=52.207.157.207, name=aws_x86_64_spot_prod_06718335_20250911_162710, requested_tags=['copr_builder', 'arch_x86_64']
[2025-09-11 16:29:10,711][  INFO][PID:3764215] Allocating ssh connection to builder
[2025-09-11 16:29:10,712][  INFO][PID:3764215] Checking that builder machine is OK
[2025-09-11 16:29:11,125][  INFO][PID:3764215] Running remote command: copr-builder-ready fedora-43-x86_64
[2025-09-11 16:29:11,321][  INFO][PID:3764215] Red Hat subscription not needed for fedora-43-x86_64
Builder is ready to be used

[2025-09-11 16:29:11,322][  INFO][PID:3764215] Filling build.info file with builder info
[2025-09-11 16:29:11,325][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:29:11,325][  INFO][PID:3764215] Not re-notifying FE and msg buses for the new host.
[2025-09-11 16:29:11,326][  INFO][PID:3764215] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9546197-fedora-43-x86_64 --chroot fedora-43-x86_64 --detached
[2025-09-11 16:29:11,641][  INFO][PID:3764215] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '52.207.157.207' (ED25519) to the list of known hosts.


[2025-09-11 16:29:11,642][  INFO][PID:3764215] Downloading the builder-live.log file, attempt 1
[2025-09-11 16:29:11,643][  INFO][PID:3764215] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.207.157.207 copr-rpmbuild-log
[2025-09-11 16:29:16,642][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:29:21,644][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:29:26,646][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:29:31,650][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:29:36,651][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:29:41,653][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:29:46,654][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:29:51,655][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:29:56,660][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:30:01,662][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:30:06,664][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:30:11,666][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:30:16,670][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:30:21,672][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:30:26,678][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:30:31,686][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:30:36,689][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:30:41,694][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:30:46,698][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:30:51,699][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:30:56,700][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:31:01,702][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:31:06,703][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:31:11,704][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:31:16,706][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:31:21,707][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:31:26,709][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:31:31,711][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:31:36,712][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:31:41,713][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:31:46,714][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:31:51,715][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:31:56,716][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:32:01,718][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:32:06,719][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:32:11,720][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:32:16,721][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:32:21,723][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:32:26,725][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:32:31,726][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:32:36,727][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:32:41,728][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:32:46,730][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:32:51,731][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:32:56,733][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:33:01,735][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:33:06,736][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:33:11,738][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:33:16,739][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:33:21,741][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:33:26,742][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:33:31,743][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:33:36,744][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:33:41,750][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:33:46,751][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:33:51,752][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:33:56,754][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:34:01,758][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:34:06,763][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:34:11,765][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:34:11,955][ ERROR][PID:3764215] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:34:16,767][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:34:21,769][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:34:21,961][WARNING][PID:3764215] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 16:34:26,771][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:34:31,772][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:34:31,962][  INFO][PID:3764215] Downloading the builder-live.log file, attempt 2
[2025-09-11 16:34:31,965][  INFO][PID:3764215] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.207.157.207 copr-rpmbuild-log
[2025-09-11 16:34:36,773][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:34:41,774][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:34:46,775][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:34:51,776][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:34:56,778][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:35:01,786][  INFO][PID:3764215] Periodic builder liveness probe: alive
[2025-09-11 16:35:01,787][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:35:06,788][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:35:11,789][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:35:16,790][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:35:21,791][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:35:26,792][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:35:31,794][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:35:32,036][ ERROR][PID:3764215] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:35:36,795][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:35:41,797][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:35:42,041][WARNING][PID:3764215] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 16:35:46,798][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:35:51,799][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:35:52,042][  INFO][PID:3764215] Downloading the builder-live.log file, attempt 3
[2025-09-11 16:35:52,043][  INFO][PID:3764215] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.207.157.207 copr-rpmbuild-log
[2025-09-11 16:35:56,800][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:36:01,801][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:36:06,802][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:36:11,803][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:36:16,804][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:36:21,805][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:36:26,807][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:36:31,809][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:36:36,810][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:36:41,814][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:36:46,816][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:36:51,817][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:36:52,112][ ERROR][PID:3764215] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:36:56,818][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:37:01,819][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:37:02,113][WARNING][PID:3764215] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 16:37:06,820][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:37:11,821][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:37:12,114][  INFO][PID:3764215] Downloading the builder-live.log file, attempt 4
[2025-09-11 16:37:12,115][  INFO][PID:3764215] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.207.157.207 copr-rpmbuild-log
[2025-09-11 16:37:16,822][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:37:21,824][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:37:26,825][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:37:31,827][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:37:36,828][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:37:41,829][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:37:46,830][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:37:51,831][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:37:56,832][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:38:01,833][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:38:06,834][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:38:11,835][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:38:12,186][ ERROR][PID:3764215] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:38:16,837][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:38:21,839][  INFO][PID:3764215] Checking for cancel request
[2025-09-11 16:38:22,187][ ERROR][PID:3764215] Giving up for unstable SSH, failures: 1757608461.9612043, 1757608542.0415616, 1757608622.1135795, 1757608702.1872745
[2025-09-11 16:38:22,188][ ERROR][PID:3764215] Re-try request for task on 'ResallocHost, ticket_id=5010489, hostname=52.207.157.207, name=aws_x86_64_spot_prod_06718335_20250911_162710, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-09-11 16:38:22,190][  INFO][PID:3764215] Releasing VM back to pool
[2025-09-11 16:38:22,201][ ERROR][PID:3764215] Backend process error: Three host tried without success: {'98.86.149.23', '52.207.157.207', '3.88.198.239'}
[2025-09-11 16:38:22,202][WARNING][PID:3764215] Switching not-finished job state to 'failed'
[2025-09-11 16:38:22,203][  INFO][PID:3764215] Worker failed build, took 550.8765659332275
[2025-09-11 16:38:22,203][  INFO][PID:3764215] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@python",
            "project_name": "python-in-pulp",
            "project_dirname": "python-in-pulp",
            "submitter": "https://src.fedoraproject.org/user/tstellar",
            "ended_on": 1757608702.2024689,
            "started_on": 1757608151.325903,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-43-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "9546197-fedora-43-x86_64",
            "build_id": 9546197,
            "package_name": "llvm",
            "package_version": "21.1.1-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@python/python-in-pulp/llvm",
            "git_hash": "d2986837e304a9a77b0fe0aa39c080e0374360ae",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@python/python-in-pulp--https://src.fedoraproject.org/user/tstellar",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 1,
            "repos": [
                {
                    "baseurl": "https://console.redhat.com/api/pulp-content/public-copr/@python/python-in-pulp/fedora-43-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@python/python-in-pulp",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@python/python-in-pulp",
            "result_dir": "09546197-llvm",
            "built_packages": "",
            "id": 9546197,
            "mockchain_macros": {
                "copr_username": "@python",
                "copr_projectname": "python-in-pulp",
                "vendor": "Fedora Project COPR (@python/python-in-pulp)"
            }
        }
    ]
}
[2025-09-11 16:38:22,231][  INFO][PID:3764215] Sending fedora-messaging bus message in build.end
[2025-09-11 16:38:22,266][  INFO][PID:3764215] Compressing /var/lib/copr/public_html/results/@python/python-in-pulp/fedora-43-x86_64/09546197-llvm/builder-live.log by gzip
[2025-09-11 16:38:22,267][  INFO][PID:3764215] Running command 'gzip /var/lib/copr/public_html/results/@python/python-in-pulp/fedora-43-x86_64/09546197-llvm/builder-live.log' as PID 4159099
[2025-09-11 16:38:22,270][  INFO][PID:3764215] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@python/python-in-pulp/fedora-43-x86_64/09546197-llvm/builder-live.log)
[2025-09-11 16:38:22,271][  INFO][PID:3764215] Compressing /var/lib/copr/public_html/results/@python/python-in-pulp/fedora-43-x86_64/09546197-llvm/backend.log by gzip
[2025-09-11 16:38:22,279][  INFO][PID:3764215] Running command 'gzip /var/lib/copr/public_html/results/@python/python-in-pulp/fedora-43-x86_64/09546197-llvm/backend.log' as PID 4159101