[2026-06-02 11:45:45,053][  INFO][PID:1624576] Marking build as starting
[2026-06-02 11:45:45,229][  INFO][PID:1624576] Checking that /var/lib/copr/public_html/results/che/mesa/fedora-44-x86_64/repodata/repomd.xml exists
[2026-06-02 11:45:45,239][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:45:45,311][  INFO][PID:1624576] rpmeta: predicted 18 min for mesa on fedora-44-x86_64 -> normal builder sufficient
[2026-06-02 11:45:45,315][  INFO][PID:1624576] VM allocation process starts
[2026-06-02 11:45:45,365][  INFO][PID:1624576] Trying to allocate VM: ResallocHost, ticket_id=3182595, requested_tags=['arch_x86_64', 'copr_builder']
[2026-06-02 11:45:50,373][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:45:55,377][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:46:00,391][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:46:05,393][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:46:10,400][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:46:15,405][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:46:20,411][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:46:25,415][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:46:30,418][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:46:35,419][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:46:40,424][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:46:45,437][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:46:50,445][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:46:55,447][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:47:00,449][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:47:05,464][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:47:10,472][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:47:15,476][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:47:20,481][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:47:25,499][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:47:30,502][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:47:35,516][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:47:40,524][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:47:45,531][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:47:50,535][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:47:55,537][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:48:00,538][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:48:05,540][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:48:10,546][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:48:15,549][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:48:20,551][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:48:25,553][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:48:30,559][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:48:35,565][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:48:40,571][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:48:45,573][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:48:50,583][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:48:55,601][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:49:00,605][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:49:03,581][  INFO][PID:1624576] Allocated host ResallocHost, ticket_id=3182595, hostname=98.92.62.77, name=aws_x86_64_spot_prod_10451694_20260602_114720, requested_tags=['arch_x86_64', 'copr_builder']
[2026-06-02 11:49:03,582][  INFO][PID:1624576] Allocating ssh connection to builder
[2026-06-02 11:49:03,583][  INFO][PID:1624576] Checking that builder machine is OK
[2026-06-02 11:49:03,979][  INFO][PID:1624576] Installed copr-rpmbuild version: 1.8
[2026-06-02 11:49:03,980][  INFO][PID:1624576] Running remote command: copr-builder-ready fedora-44-x86_64
[2026-06-02 11:49:04,211][  INFO][PID:1624576] Red Hat subscription not needed for fedora-44-x86_64
Builder is ready to be used

[2026-06-02 11:49:04,212][  INFO][PID:1624576] Filling build.info file with builder info
[2026-06-02 11:49:04,220][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:49:04,222][  INFO][PID:1624576] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "che",
            "project_name": "mesa",
            "project_dirname": "mesa",
            "submitter": "che",
            "ended_on": null,
            "started_on": 1780400944.2220197,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-44-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "10536621-fedora-44-x86_64",
            "build_id": 10536621,
            "package_name": "mesa",
            "package_version": "26.1.0-0.37.git8ec56cc",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/che/mesa/mesa",
            "git_hash": "6921539d5bb3d6091f3de654b540478271ca75f8",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "che/mesa--che",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/che/mesa/fedora-44-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/che/llvm/fedora-44-x86_64/",
                    "id": "copr_che_llvm",
                    "name": "Additional repo copr_che_llvm"
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/che/DirectX-Headers/fedora-44-x86_64/",
                    "id": "copr_che_DirectX_Headers",
                    "name": "Additional repo copr_che_DirectX_Headers"
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/che/mesa",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/che/mesa",
            "result_dir": "10536621-mesa",
            "built_packages": "",
            "rpmeta_prediction": 18,
            "id": 10536621,
            "mockchain_macros": {
                "copr_username": "che",
                "copr_projectname": "mesa",
                "vendor": "Fedora Project COPR (che/mesa)"
            }
        }
    ]
}
[2026-06-02 11:49:04,277][  INFO][PID:1624576] Sending fedora-messaging bus message in build.start
[2026-06-02 11:49:04,812][  INFO][PID:1624576] Sending fedora-messaging bus message in chroot.start
[2026-06-02 11:49:04,841][  INFO][PID:1624576] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10536621-fedora-44-x86_64 --chroot fedora-44-x86_64 --detached
[2026-06-02 11:49:05,311][  INFO][PID:1624576] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '98.92.62.77' (ED25519) to the list of known hosts.


[2026-06-02 11:49:05,314][  INFO][PID:1624576] Downloading the builder-live.log file, attempt 1
[2026-06-02 11:49:05,318][  INFO][PID:1624576] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.92.62.77 copr-rpmbuild-log
[2026-06-02 11:49:10,341][  INFO][PID:1624576] Periodic builder liveness probe: alive
[2026-06-02 11:49:10,343][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:49:15,347][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:49:20,349][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:49:25,350][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:49:30,352][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:49:35,356][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:49:40,357][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:49:45,359][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:49:50,361][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:49:55,364][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:50:00,367][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:50:05,369][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:50:07,899][ ERROR][PID:1624576] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-06-02 11:50:10,371][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:50:15,374][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:50:17,918][WARNING][PID:1624576] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-06-02 11:50:20,380][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:50:25,382][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:50:27,923][  INFO][PID:1624576] Downloading the builder-live.log file, attempt 2
[2026-06-02 11:50:27,928][  INFO][PID:1624576] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.92.62.77 copr-rpmbuild-log
[2026-06-02 11:50:30,384][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:50:35,387][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:50:40,392][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:50:45,400][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:50:50,402][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:50:55,404][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:51:00,477][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:51:05,489][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:51:10,491][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:51:15,493][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:51:20,499][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:51:25,502][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:51:28,000][ ERROR][PID:1624576] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-06-02 11:51:30,506][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:51:35,515][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:51:38,019][WARNING][PID:1624576] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-06-02 11:51:40,525][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:51:45,527][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:51:48,022][  INFO][PID:1624576] Downloading the builder-live.log file, attempt 3
[2026-06-02 11:51:48,024][  INFO][PID:1624576] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.92.62.77 copr-rpmbuild-log
[2026-06-02 11:51:50,529][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:51:55,530][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:52:00,533][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:52:05,536][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:52:10,599][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:52:15,617][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:52:20,639][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:52:25,642][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:52:30,645][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:52:35,648][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:52:40,665][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:52:45,689][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:52:48,117][ ERROR][PID:1624576] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-06-02 11:52:50,698][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:52:55,702][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:52:58,252][WARNING][PID:1624576] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-06-02 11:53:00,710][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:53:05,716][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:53:08,310][  INFO][PID:1624576] Downloading the builder-live.log file, attempt 4
[2026-06-02 11:53:08,400][  INFO][PID:1624576] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.92.62.77 copr-rpmbuild-log
[2026-06-02 11:53:10,720][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:53:15,726][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:53:20,729][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:53:25,744][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:53:30,746][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:53:35,751][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:53:40,753][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:53:45,755][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:53:50,758][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:53:55,762][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:54:00,781][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:54:05,787][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:54:08,493][ ERROR][PID:1624576] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-06-02 11:54:10,791][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:54:15,794][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:54:18,693][ ERROR][PID:1624576] Giving up for unstable SSH, failures: 1780401017.9187593, 1780401098.0195591, 1780401178.2526608, 1780401258.5787754
[2026-06-02 11:54:18,734][ ERROR][PID:1624576] Re-try request for task on 'ResallocHost, ticket_id=3182595, hostname=98.92.62.77, name=aws_x86_64_spot_prod_10451694_20260602_114720, requested_tags=['arch_x86_64', 'copr_builder']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-06-02 11:54:18,754][  INFO][PID:1624576] VM Release request
[2026-06-02 11:54:18,804][  INFO][PID:1624576] Retry #1 (on other host)
[2026-06-02 11:54:18,804][  INFO][PID:1624576] Checking that /var/lib/copr/public_html/results/che/mesa/fedora-44-x86_64/repodata/repomd.xml exists
[2026-06-02 11:54:18,813][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:54:18,901][  INFO][PID:1624576] rpmeta: predicted 18 min for mesa on fedora-44-x86_64 -> normal builder sufficient
[2026-06-02 11:54:18,904][  INFO][PID:1624576] VM allocation process starts
[2026-06-02 11:54:18,939][  INFO][PID:1624576] Trying to allocate VM: ResallocHost, ticket_id=3183169, requested_tags=['arch_x86_64', 'copr_builder']
[2026-06-02 11:54:23,944][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:54:28,946][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:54:33,948][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:54:38,951][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:54:43,954][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:54:48,960][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:54:53,995][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:54:59,004][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:55:04,012][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:55:09,015][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:55:14,017][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:55:19,051][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:55:24,057][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:55:29,069][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:55:34,073][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:55:39,074][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:55:44,078][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:55:49,080][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:55:54,087][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:55:59,107][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:56:04,110][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:56:09,114][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:56:14,115][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:56:19,117][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:56:24,122][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:56:29,124][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:56:34,125][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:56:39,134][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:56:44,137][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:56:49,140][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:56:54,147][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:56:59,165][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:57:04,185][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:57:09,196][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:57:14,201][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:57:19,204][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:57:24,208][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:57:29,211][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:57:34,212][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:57:39,214][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:57:44,217][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:57:49,220][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:57:54,224][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:57:59,229][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:58:04,230][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:58:09,232][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:58:14,240][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:58:19,242][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:58:24,257][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:58:29,261][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:58:34,263][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:58:39,267][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:58:44,273][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:58:49,277][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:58:54,280][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:58:59,282][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:59:04,289][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:59:09,297][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:59:14,309][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:59:19,311][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:59:24,315][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:59:29,318][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:59:34,322][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:59:37,305][  INFO][PID:1624576] Allocated host ResallocHost, ticket_id=3183169, hostname=3.80.114.220, name=aws_x86_64_spot_prod_10452146_20260602_115725, requested_tags=['arch_x86_64', 'copr_builder']
[2026-06-02 11:59:37,322][  INFO][PID:1624576] Allocating ssh connection to builder
[2026-06-02 11:59:37,331][  INFO][PID:1624576] Checking that builder machine is OK
[2026-06-02 11:59:37,736][  INFO][PID:1624576] Installed copr-rpmbuild version: 1.8
[2026-06-02 11:59:37,753][  INFO][PID:1624576] Running remote command: copr-builder-ready fedora-44-x86_64
[2026-06-02 11:59:38,117][  INFO][PID:1624576] Red Hat subscription not needed for fedora-44-x86_64
Builder is ready to be used

[2026-06-02 11:59:38,122][  INFO][PID:1624576] Filling build.info file with builder info
[2026-06-02 11:59:38,209][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:59:38,213][  INFO][PID:1624576] Not re-notifying FE and msg buses for the new host.
[2026-06-02 11:59:38,219][  INFO][PID:1624576] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10536621-fedora-44-x86_64 --chroot fedora-44-x86_64 --detached
[2026-06-02 11:59:38,800][  INFO][PID:1624576] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '3.80.114.220' (ED25519) to the list of known hosts.


[2026-06-02 11:59:38,803][  INFO][PID:1624576] Downloading the builder-live.log file, attempt 1
[2026-06-02 11:59:38,815][  INFO][PID:1624576] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.80.114.220 copr-rpmbuild-log
[2026-06-02 11:59:43,837][  INFO][PID:1624576] Periodic builder liveness probe: alive
[2026-06-02 11:59:43,866][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:59:48,878][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:59:53,882][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 11:59:58,906][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:00:03,908][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:00:08,914][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:00:13,924][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:00:18,945][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:00:23,948][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:00:28,953][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:00:33,972][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:00:38,976][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:00:43,982][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:00:48,988][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:00:54,001][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:00:59,008][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:01:04,010][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:01:09,015][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:01:14,028][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:01:19,030][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:01:24,035][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:01:29,039][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:01:34,054][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:01:39,059][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:01:44,062][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:01:49,073][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:01:54,078][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:01:59,100][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:02:04,105][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:02:09,107][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:02:14,125][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:02:19,139][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:02:24,150][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:02:29,164][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:02:34,169][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:02:39,176][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:02:44,178][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:02:49,204][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:02:54,206][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:02:59,212][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:03:04,215][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:03:09,225][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:03:14,234][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:03:19,256][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:03:24,266][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:03:29,273][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:03:34,279][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:03:39,298][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:03:44,319][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:03:49,329][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:03:54,335][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:03:59,340][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:04:04,341][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:04:09,347][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:04:14,351][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:04:19,355][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:04:24,359][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:04:29,365][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:04:34,367][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:04:39,372][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:04:44,380][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:04:49,388][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:04:54,391][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:04:59,399][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:05:04,405][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:05:09,418][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:05:14,425][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:05:19,429][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:05:23,341][ ERROR][PID:1624576] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-06-02 12:05:24,432][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:05:29,451][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:05:33,568][WARNING][PID:1624576] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-06-02 12:05:34,455][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:05:39,473][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:05:43,569][  INFO][PID:1624576] Downloading the builder-live.log file, attempt 2
[2026-06-02 12:05:43,594][  INFO][PID:1624576] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.80.114.220 copr-rpmbuild-log
[2026-06-02 12:05:44,478][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:05:49,484][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:05:54,489][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:05:59,493][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:06:04,506][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:06:09,508][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:06:14,511][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:06:19,514][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:06:24,522][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:06:29,524][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:06:34,527][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:06:39,534][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:06:43,668][ ERROR][PID:1624576] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-06-02 12:06:44,544][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:06:49,547][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:06:53,746][WARNING][PID:1624576] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-06-02 12:06:54,551][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:06:59,563][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:07:03,749][  INFO][PID:1624576] Downloading the builder-live.log file, attempt 3
[2026-06-02 12:07:03,755][  INFO][PID:1624576] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.80.114.220 copr-rpmbuild-log
[2026-06-02 12:07:04,568][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:07:09,573][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:07:14,587][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:07:19,589][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:07:24,608][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:07:29,612][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:07:34,620][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:07:39,627][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:07:44,632][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:07:49,636][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:07:54,639][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:07:59,643][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:08:03,873][ ERROR][PID:1624576] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-06-02 12:08:04,652][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:08:09,670][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:08:14,044][WARNING][PID:1624576] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-06-02 12:08:14,675][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:08:19,676][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:08:24,059][  INFO][PID:1624576] Downloading the builder-live.log file, attempt 4
[2026-06-02 12:08:24,069][  INFO][PID:1624576] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.80.114.220 copr-rpmbuild-log
[2026-06-02 12:08:24,695][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:08:29,698][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:08:34,703][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:08:39,706][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:08:44,708][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:08:49,714][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:08:54,726][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:08:59,739][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:09:04,740][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:09:09,745][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:09:14,749][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:09:19,756][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:09:24,189][ ERROR][PID:1624576] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-06-02 12:09:24,764][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:09:29,770][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:09:34,411][ ERROR][PID:1624576] Giving up for unstable SSH, failures: 1780401933.5669947, 1780402013.7465718, 1780402094.0445106, 1780402174.3955688
[2026-06-02 12:09:34,478][ ERROR][PID:1624576] Re-try request for task on 'ResallocHost, ticket_id=3183169, hostname=3.80.114.220, name=aws_x86_64_spot_prod_10452146_20260602_115725, requested_tags=['arch_x86_64', 'copr_builder']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-06-02 12:09:34,540][  INFO][PID:1624576] VM Release request
[2026-06-02 12:09:34,564][  INFO][PID:1624576] Retry #2 (on other host)
[2026-06-02 12:09:34,568][  INFO][PID:1624576] Checking that /var/lib/copr/public_html/results/che/mesa/fedora-44-x86_64/repodata/repomd.xml exists
[2026-06-02 12:09:34,573][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:09:34,838][  INFO][PID:1624576] rpmeta: predicted 18 min for mesa on fedora-44-x86_64 -> normal builder sufficient
[2026-06-02 12:09:34,845][  INFO][PID:1624576] VM allocation process starts
[2026-06-02 12:09:34,878][  INFO][PID:1624576] Trying to allocate VM: ResallocHost, ticket_id=3183977, requested_tags=['arch_x86_64', 'copr_builder']
[2026-06-02 12:09:39,889][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:09:44,891][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:09:49,896][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:09:54,898][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:09:59,904][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:10:04,905][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:10:09,907][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:10:14,913][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:10:19,919][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:10:24,922][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:10:29,931][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:10:34,937][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:10:39,945][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:10:44,954][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:10:49,959][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:10:54,966][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:10:59,973][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:11:03,157][  INFO][PID:1624576] Allocated host ResallocHost, ticket_id=3183977, hostname=100.54.12.252, name=aws_x86_64_spot_prod_10452480_20260602_120844, requested_tags=['arch_x86_64', 'copr_builder']
[2026-06-02 12:11:03,161][  INFO][PID:1624576] Allocating ssh connection to builder
[2026-06-02 12:11:03,162][  INFO][PID:1624576] Checking that builder machine is OK
[2026-06-02 12:11:03,688][  INFO][PID:1624576] Installed copr-rpmbuild version: 1.8
[2026-06-02 12:11:03,692][  INFO][PID:1624576] Running remote command: copr-builder-ready fedora-44-x86_64
[2026-06-02 12:11:03,997][  INFO][PID:1624576] Red Hat subscription not needed for fedora-44-x86_64
Builder is ready to be used

[2026-06-02 12:11:03,998][  INFO][PID:1624576] Filling build.info file with builder info
[2026-06-02 12:11:04,022][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:11:04,027][  INFO][PID:1624576] Not re-notifying FE and msg buses for the new host.
[2026-06-02 12:11:04,036][  INFO][PID:1624576] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10536621-fedora-44-x86_64 --chroot fedora-44-x86_64 --detached
[2026-06-02 12:11:04,790][  INFO][PID:1624576] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '100.54.12.252' (ED25519) to the list of known hosts.


[2026-06-02 12:11:04,800][  INFO][PID:1624576] Downloading the builder-live.log file, attempt 1
[2026-06-02 12:11:04,814][  INFO][PID:1624576] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@100.54.12.252 copr-rpmbuild-log
[2026-06-02 12:11:09,813][  INFO][PID:1624576] Periodic builder liveness probe: alive
[2026-06-02 12:11:09,817][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:11:14,819][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:11:19,823][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:11:24,827][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:11:29,829][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:11:34,832][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:11:39,836][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:11:44,844][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:11:49,847][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:11:54,849][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:11:59,851][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:12:04,856][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:12:09,864][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:12:14,867][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:12:19,869][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:12:24,889][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:12:29,900][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:12:34,903][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:12:39,923][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:12:44,927][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:12:49,931][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:12:54,932][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:12:59,935][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:13:04,937][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:13:09,946][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:13:14,950][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:13:19,954][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:13:24,959][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:13:29,963][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:13:34,972][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:13:39,975][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:13:44,977][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:13:49,991][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:13:54,993][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:13:59,997][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:14:05,002][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:14:10,028][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:14:15,033][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:14:20,034][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:14:25,047][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:14:30,056][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:14:35,061][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:14:40,090][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:14:45,093][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:14:50,096][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:14:55,106][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:15:00,127][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:15:05,131][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:15:10,155][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:15:15,158][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:15:20,166][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:15:25,172][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:15:30,193][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:15:35,196][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:15:40,206][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:15:45,208][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:15:50,215][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:15:55,218][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:16:00,225][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:16:05,230][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:16:10,237][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:16:15,243][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:16:20,253][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:16:25,256][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:16:30,262][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:16:35,277][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:16:40,280][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:16:45,286][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:16:50,290][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:16:55,293][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:17:00,295][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:17:05,301][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:17:10,309][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:17:15,312][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:17:20,315][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:17:25,318][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:17:30,323][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:17:35,327][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:17:40,333][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:17:45,341][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:17:50,371][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:17:55,385][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:18:00,393][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:18:05,397][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:18:10,402][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:18:15,404][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:18:20,405][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:18:25,410][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:18:30,428][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:18:35,431][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:18:40,436][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:18:45,441][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:18:50,448][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:18:55,473][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:19:00,480][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:19:05,482][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:19:10,489][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:19:15,498][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:19:20,503][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:19:25,514][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:19:30,523][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:19:35,558][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:19:40,560][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:19:45,562][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:19:50,568][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:19:55,583][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:20:00,586][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:20:05,591][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:20:10,595][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:20:15,604][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:20:20,619][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:20:25,623][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:20:30,625][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:20:35,632][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:20:40,646][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:20:45,648][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:20:50,650][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:20:55,653][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:21:00,660][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:21:05,663][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:21:10,716][  INFO][PID:1624576] Periodic builder liveness probe: alive
[2026-06-02 12:21:10,726][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:21:15,728][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:21:20,734][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:21:25,736][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:21:30,739][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:21:35,743][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:21:40,749][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:21:45,752][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:21:50,754][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:21:55,759][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:22:00,762][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:22:05,765][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:22:10,767][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:22:15,769][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:22:20,773][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:22:25,780][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:22:30,782][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:22:35,784][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:22:40,792][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:22:45,798][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:22:50,804][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:22:55,808][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:23:00,809][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:23:05,811][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:23:10,813][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:23:15,816][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:23:20,819][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:23:25,833][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:23:30,851][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:23:35,854][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:23:40,856][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:23:45,859][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:23:50,861][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:23:55,869][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:24:00,874][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:24:05,876][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:24:10,879][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:24:15,887][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:24:20,896][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:24:25,898][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:24:30,914][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:24:35,917][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:24:40,920][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:24:45,922][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:24:50,926][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:24:55,929][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:25:00,931][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:25:05,933][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:25:10,935][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:25:15,941][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:25:20,946][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:25:25,949][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:25:30,954][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:25:35,965][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:25:40,969][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:25:45,971][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:25:50,976][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:25:55,979][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:26:00,993][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:26:05,995][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:26:10,997][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:26:16,000][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:26:21,005][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:26:26,008][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:26:31,010][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:26:36,012][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:26:41,014][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:26:46,023][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:26:51,024][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:26:56,026][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:27:01,028][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:27:06,032][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:27:11,033][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:27:16,035][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:27:21,039][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:27:26,041][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:27:31,042][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:27:36,044][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:27:41,045][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:27:46,048][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:27:51,050][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:27:56,054][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:28:01,056][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:28:06,058][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:28:11,062][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:28:16,063][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:28:21,065][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:28:26,074][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:28:31,076][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:28:36,077][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:28:41,079][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:28:46,082][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:28:51,087][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:28:56,093][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:29:01,096][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:29:06,098][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:29:11,101][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:29:16,106][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:29:21,108][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:29:26,109][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:29:31,112][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:29:36,133][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:29:41,135][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:29:46,137][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:29:51,140][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:29:56,149][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:30:01,151][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:30:06,152][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:30:11,158][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:30:16,165][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:30:21,167][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:30:26,168][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:30:31,170][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:30:36,179][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:30:41,195][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:30:46,196][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:30:51,197][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:30:56,201][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:31:01,203][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:31:06,205][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:31:11,219][  INFO][PID:1624576] Periodic builder liveness probe: alive
[2026-06-02 12:31:11,220][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:31:16,221][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:31:21,224][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:31:26,225][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:31:31,227][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:31:36,228][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:31:41,231][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:31:46,232][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:31:49,658][ ERROR][PID:1624576] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-06-02 12:31:51,234][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:31:56,236][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:31:59,806][WARNING][PID:1624576] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-06-02 12:32:01,239][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:32:06,240][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:32:09,827][  INFO][PID:1624576] Downloading the builder-live.log file, attempt 2
[2026-06-02 12:32:10,060][  INFO][PID:1624576] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@100.54.12.252 copr-rpmbuild-log
[2026-06-02 12:32:11,242][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:32:16,243][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:32:21,245][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:32:26,246][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:32:31,247][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:32:36,249][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:32:41,251][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:32:46,255][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:32:51,257][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:32:56,259][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:33:01,265][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:33:06,270][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:33:10,079][ ERROR][PID:1624576] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-06-02 12:33:11,274][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:33:16,276][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:33:20,112][WARNING][PID:1624576] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-06-02 12:33:21,278][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:33:26,284][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:33:30,120][  INFO][PID:1624576] Downloading the builder-live.log file, attempt 3
[2026-06-02 12:33:30,152][  INFO][PID:1624576] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@100.54.12.252 copr-rpmbuild-log
[2026-06-02 12:33:31,292][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:33:36,294][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:33:41,296][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:33:46,301][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:33:51,303][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:33:56,304][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:34:01,306][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:34:06,309][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:34:11,316][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:34:16,318][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:34:21,319][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:34:26,321][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:34:30,238][ ERROR][PID:1624576] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-06-02 12:34:31,323][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:34:36,325][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:34:40,267][WARNING][PID:1624576] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-06-02 12:34:41,328][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:34:46,330][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:34:50,275][  INFO][PID:1624576] Downloading the builder-live.log file, attempt 4
[2026-06-02 12:34:50,278][  INFO][PID:1624576] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@100.54.12.252 copr-rpmbuild-log
[2026-06-02 12:34:51,339][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:34:56,340][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:35:01,343][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:35:06,345][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:35:11,347][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:35:16,352][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:35:21,353][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:35:26,355][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:35:31,359][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:35:36,361][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:35:41,365][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:35:46,367][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:35:50,348][ ERROR][PID:1624576] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-06-02 12:35:51,368][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:35:56,370][  INFO][PID:1624576] Checking for cancel request
[2026-06-02 12:36:00,739][ ERROR][PID:1624576] Giving up for unstable SSH, failures: 1780403519.80652, 1780403600.1121583, 1780403680.2673957, 1780403760.4077625
[2026-06-02 12:36:01,189][ ERROR][PID:1624576] Re-try request for task on 'ResallocHost, ticket_id=3183977, hostname=100.54.12.252, name=aws_x86_64_spot_prod_10452480_20260602_120844, requested_tags=['arch_x86_64', 'copr_builder']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-06-02 12:36:01,397][  INFO][PID:1624576] VM Release request
[2026-06-02 12:36:02,286][ ERROR][PID:1624576] Backend process error: Three host tried without success: {'98.92.62.77', '100.54.12.252', '3.80.114.220'}
[2026-06-02 12:36:02,393][WARNING][PID:1624576] Switching not-finished job state to 'failed'
[2026-06-02 12:36:02,401][  INFO][PID:1624576] Worker failed build, took 1498.3229775428772
[2026-06-02 12:36:02,526][  INFO][PID:1624576] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "che",
            "project_name": "mesa",
            "project_dirname": "mesa",
            "submitter": "che",
            "ended_on": 1780403762.3502789,
            "started_on": 1780402264.0273013,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-44-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "10536621-fedora-44-x86_64",
            "build_id": 10536621,
            "package_name": "mesa",
            "package_version": "26.1.0-0.37.git8ec56cc",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/che/mesa/mesa",
            "git_hash": "6921539d5bb3d6091f3de654b540478271ca75f8",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "che/mesa--che",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/che/mesa/fedora-44-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/che/llvm/fedora-44-x86_64/",
                    "id": "copr_che_llvm",
                    "name": "Additional repo copr_che_llvm"
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/che/DirectX-Headers/fedora-44-x86_64/",
                    "id": "copr_che_DirectX_Headers",
                    "name": "Additional repo copr_che_DirectX_Headers"
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/che/mesa",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/che/mesa",
            "result_dir": "10536621-mesa",
            "built_packages": "",
            "rpmeta_prediction": 18,
            "id": 10536621,
            "mockchain_macros": {
                "copr_username": "che",
                "copr_projectname": "mesa",
                "vendor": "Fedora Project COPR (che/mesa)"
            }
        }
    ]
}
[2026-06-02 12:36:05,843][  INFO][PID:1624576] Sending fedora-messaging bus message in build.end
[2026-06-02 12:36:09,507][  INFO][PID:1624576] Compressing /var/lib/copr/public_html/results/che/mesa/fedora-44-x86_64/10536621-mesa/builder-live.log by gzip
[2026-06-02 12:36:09,582][  INFO][PID:1624576] Running command 'gzip /var/lib/copr/public_html/results/che/mesa/fedora-44-x86_64/10536621-mesa/builder-live.log' as PID 2126907
[2026-06-02 12:36:10,482][  INFO][PID:1624576] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/che/mesa/fedora-44-x86_64/10536621-mesa/builder-live.log)
[2026-06-02 12:36:10,536][  INFO][PID:1624576] Compressing /var/lib/copr/public_html/results/che/mesa/fedora-44-x86_64/10536621-mesa/backend.log by gzip
[2026-06-02 12:36:10,538][  INFO][PID:1624576] Running command 'gzip /var/lib/copr/public_html/results/che/mesa/fedora-44-x86_64/10536621-mesa/backend.log' as PID 2126986