[2026-04-20 14:08:09,693][  INFO][PID:2674439] Marking build as starting
[2026-04-20 14:08:09,809][  INFO][PID:2674439] Checking that https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64//repodata/repomd.xml exists
[2026-04-20 14:08:10,592][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:08:10,596][  INFO][PID:2674439] VM allocation process starts
[2026-04-20 14:08:10,619][  INFO][PID:2674439] Trying to allocate VM: ResallocHost, ticket_id=2423635, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:08:15,623][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:08:20,628][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:08:25,656][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:08:30,666][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:08:35,693][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:08:40,699][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:08:45,707][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:08:50,729][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:08:55,737][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:09:00,755][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:09:05,776][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:09:10,793][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:09:15,795][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:09:20,801][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:09:25,804][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:09:30,807][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:09:35,812][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:09:40,820][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:09:45,827][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:09:50,838][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:09:55,854][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:10:00,875][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:10:05,898][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:10:10,915][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:10:15,922][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:10:20,941][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:10:25,968][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:10:30,977][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:10:35,983][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:10:40,989][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:10:45,998][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:10:51,007][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:10:56,027][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:11:01,043][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:11:06,058][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:11:11,060][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:11:16,074][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:11:21,087][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:11:26,093][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:11:31,099][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:11:36,101][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:11:41,113][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:11:46,137][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:11:51,143][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:11:56,153][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:12:01,162][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:12:06,210][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:12:11,255][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:12:16,287][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:12:21,319][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:12:26,354][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:12:31,381][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:12:36,419][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:12:41,424][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:12:46,432][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:12:51,443][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:12:56,449][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:13:01,452][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:13:06,458][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:13:11,494][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:13:16,511][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:13:21,531][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:13:26,533][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:13:31,538][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:13:36,540][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:13:41,547][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:13:46,556][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:13:51,574][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:13:56,577][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:14:01,605][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:14:06,618][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:14:11,622][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:14:16,630][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:14:21,639][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:14:26,657][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:14:29,615][  INFO][PID:2674439] Allocated host ResallocHost, ticket_id=2423635, hostname=100.31.59.181, name=aws_aarch64_reserved_prod_07134634_20260420_141052, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:14:29,617][  INFO][PID:2674439] Allocating ssh connection to builder
[2026-04-20 14:14:29,619][  INFO][PID:2674439] Checking that builder machine is OK
[2026-04-20 14:14:29,853][ ERROR][PID:2674439] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '100.31.59.181' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@100.31.59.181: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:14:39,992][ ERROR][PID:2674439] Re-try request for task on 'ResallocHost, ticket_id=2423635, hostname=100.31.59.181, name=aws_aarch64_reserved_prod_07134634_20260420_141052, requested_tags=['arch_aarch64', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:14:40,035][  INFO][PID:2674439] VM Release request
[2026-04-20 14:14:40,264][  INFO][PID:2674439] Retry #1 (on other host)
[2026-04-20 14:14:40,271][  INFO][PID:2674439] Checking that https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64//repodata/repomd.xml exists
[2026-04-20 14:14:40,933][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:14:40,950][  INFO][PID:2674439] VM allocation process starts
[2026-04-20 14:14:41,039][  INFO][PID:2674439] Trying to allocate VM: ResallocHost, ticket_id=2423777, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:14:46,049][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:14:51,059][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:14:56,064][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:15:01,070][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:15:06,086][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:15:11,100][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:15:16,114][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:15:21,117][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:15:26,128][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:15:31,133][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:15:36,141][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:15:41,155][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:15:46,164][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:15:51,168][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:15:56,183][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:16:01,189][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:16:06,196][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:16:11,202][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:16:16,211][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:16:21,221][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:16:26,253][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:16:31,268][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:16:36,272][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:16:41,288][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:16:46,295][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:16:51,312][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:16:56,333][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:17:01,355][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:17:06,370][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:17:11,384][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:17:16,398][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:17:21,416][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:17:26,434][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:17:31,437][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:17:36,440][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:17:41,453][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:17:46,456][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:17:51,460][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:17:56,463][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:17:59,977][  INFO][PID:2674439] Allocated host ResallocHost, ticket_id=2423777, hostname=52.201.233.133, name=aws_aarch64_reserved_prod_07134844_20260420_141412, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:17:59,979][  INFO][PID:2674439] Allocating ssh connection to builder
[2026-04-20 14:17:59,980][  INFO][PID:2674439] Checking that builder machine is OK
[2026-04-20 14:18:00,427][ ERROR][PID:2674439] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '52.201.233.133' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@52.201.233.133: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:18:10,466][ ERROR][PID:2674439] Re-try request for task on 'ResallocHost, ticket_id=2423777, hostname=52.201.233.133, name=aws_aarch64_reserved_prod_07134844_20260420_141412, requested_tags=['arch_aarch64', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:18:10,647][  INFO][PID:2674439] VM Release request
[2026-04-20 14:18:10,819][  INFO][PID:2674439] Retry #2 (on other host)
[2026-04-20 14:18:10,828][  INFO][PID:2674439] Checking that https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64//repodata/repomd.xml exists
[2026-04-20 14:18:11,463][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:18:11,469][  INFO][PID:2674439] VM allocation process starts
[2026-04-20 14:18:11,538][  INFO][PID:2674439] Trying to allocate VM: ResallocHost, ticket_id=2423858, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:18:16,548][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:18:21,558][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:18:26,588][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:18:31,595][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:18:36,603][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:18:41,609][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:18:46,666][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:18:51,672][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:18:56,674][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:19:01,681][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:19:06,697][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:19:11,704][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:19:16,707][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:19:21,719][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:19:26,726][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:19:31,743][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:19:36,761][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:19:41,786][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:19:46,792][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:19:51,802][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:19:56,809][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:20:01,833][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:20:06,855][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:20:11,860][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:20:16,864][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:20:21,872][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:20:26,875][  INFO][PID:2674439] Checking for cancel request
[2026-04-20 14:20:30,277][  INFO][PID:2674439] Allocated host ResallocHost, ticket_id=2423858, hostname=13.217.105.33, name=aws_aarch64_reserved_prod_07135034_20260420_141709, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:20:30,282][  INFO][PID:2674439] Allocating ssh connection to builder
[2026-04-20 14:20:30,284][  INFO][PID:2674439] Checking that builder machine is OK
[2026-04-20 14:20:30,445][ ERROR][PID:2674439] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '13.217.105.33' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@13.217.105.33: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:20:40,509][ ERROR][PID:2674439] Re-try request for task on 'ResallocHost, ticket_id=2423858, hostname=13.217.105.33, name=aws_aarch64_reserved_prod_07135034_20260420_141709, requested_tags=['arch_aarch64', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:20:40,587][  INFO][PID:2674439] VM Release request
[2026-04-20 14:20:40,699][ ERROR][PID:2674439] Backend process error: Three host tried without success: {'52.201.233.133', '13.217.105.33', '100.31.59.181'}
[2026-04-20 14:20:40,756][WARNING][PID:2674439] Switching not-finished job state to 'failed'
[2026-04-20 14:20:40,770][  INFO][PID:2674439] Worker failed build, took 751.1470639705658
[2026-04-20 14:20:40,772][  INFO][PID:2674439] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "xsnrg",
            "project_name": "indi-3rdparty-bleeding",
            "project_dirname": "indi-3rdparty-bleeding",
            "submitter": "https://api.github.com/users/knro",
            "ended_on": 1776694840.756808,
            "started_on": 1776694089.609744,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-aarch64",
            "arch": "aarch64",
            "buildroot_pkgs": [],
            "task_id": "10348527-fedora-rawhide-aarch64",
            "build_id": 10348527,
            "package_name": "libstarshootg",
            "package_version": "2.1.3.git-20260420131439",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/xsnrg/indi-3rdparty-bleeding/libstarshootg",
            "git_hash": "aacc15a7138f1067123a8d0f9b77c7db5b22afeb",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "xsnrg/indi-3rdparty-bleeding--https://api.github.com/users/knro",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 1,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xsnrg/libindi-bleeding/fedora-rawhide-aarch64/",
                    "id": "copr_xsnrg_libindi_bleeding",
                    "name": "Additional repo copr_xsnrg_libindi_bleeding"
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/",
                    "id": "copr_xsnrg_indi_3rdparty_bleeding",
                    "name": "Additional repo copr_xsnrg_indi_3rdparty_bleeding"
                }
            ],
            "background": true,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_aarch64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding",
            "result_dir": "10348527-libstarshootg",
            "built_packages": "",
            "id": 10348527,
            "mockchain_macros": {
                "copr_username": "xsnrg",
                "copr_projectname": "indi-3rdparty-bleeding",
                "vendor": "Fedora Project COPR (xsnrg/indi-3rdparty-bleeding)"
            }
        }
    ]
}
[2026-04-20 14:20:40,885][  INFO][PID:2674439] Sending fedora-messaging bus message in build.end
[2026-04-20 14:20:42,345][  INFO][PID:2674439] Compressing /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348527-libstarshootg/builder-live.log by gzip
[2026-04-20 14:20:42,358][  INFO][PID:2674439] Running command 'gzip /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348527-libstarshootg/builder-live.log' as PID 2848522
[2026-04-20 14:20:42,360][  INFO][PID:2674439] Finished after 0 seconds with exit code 1 (gzip /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348527-libstarshootg/builder-live.log)
stdout:

stderr:
gzip: /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348527-libstarshootg/builder-live.log: No such file or directory


[2026-04-20 14:20:42,361][ ERROR][PID:2674439] Unable to compress file /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348527-libstarshootg/builder-live.log
[2026-04-20 14:20:42,500][  INFO][PID:2674439] Compressing /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348527-libstarshootg/backend.log by gzip
[2026-04-20 14:20:42,505][  INFO][PID:2674439] Running command 'gzip /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348527-libstarshootg/backend.log' as PID 2848559