[2026-04-20 14:06:59,472][  INFO][PID:2663577] Marking build as starting
[2026-04-20 14:06:59,755][  INFO][PID:2663577] Checking that https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64//repodata/repomd.xml exists
[2026-04-20 14:07:02,141][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:07:02,148][  INFO][PID:2663577] VM allocation process starts
[2026-04-20 14:07:02,179][  INFO][PID:2663577] Trying to allocate VM: ResallocHost, ticket_id=2423609, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:07:07,186][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:07:12,191][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:07:17,197][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:07:22,219][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:07:27,254][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:07:32,279][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:07:37,289][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:07:42,301][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:07:47,309][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:07:52,317][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:07:57,319][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:08:02,359][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:08:07,362][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:08:12,377][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:08:17,380][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:08:22,409][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:08:27,442][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:08:32,445][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:08:37,449][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:08:42,487][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:08:47,505][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:08:52,532][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:08:57,551][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:09:02,572][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:09:07,577][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:09:12,583][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:09:17,590][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:09:22,597][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:09:27,619][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:09:32,646][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:09:37,690][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:09:42,703][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:09:47,714][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:09:52,726][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:09:57,735][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:10:02,747][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:10:07,753][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:10:12,780][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:10:17,788][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:10:22,809][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:10:27,812][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:10:32,831][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:10:37,836][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:10:42,858][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:10:47,870][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:10:52,875][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:10:57,880][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:11:02,887][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:11:07,994][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:11:12,995][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:11:18,002][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:11:23,009][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:11:24,140][  INFO][PID:2663577] Allocated host ResallocHost, ticket_id=2423609, hostname=44.202.106.240, name=aws_aarch64_reserved_prod_07134443_20260420_140723, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:11:24,146][  INFO][PID:2663577] Allocating ssh connection to builder
[2026-04-20 14:11:24,146][  INFO][PID:2663577] Checking that builder machine is OK
[2026-04-20 14:11:24,323][ ERROR][PID:2663577] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '44.202.106.240' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@44.202.106.240: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:11:34,404][ ERROR][PID:2663577] Re-try request for task on 'ResallocHost, ticket_id=2423609, hostname=44.202.106.240, name=aws_aarch64_reserved_prod_07134443_20260420_140723, requested_tags=['arch_aarch64', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:11:34,439][  INFO][PID:2663577] VM Release request
[2026-04-20 14:11:34,494][  INFO][PID:2663577] Retry #1 (on other host)
[2026-04-20 14:11:34,500][  INFO][PID:2663577] Checking that https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64//repodata/repomd.xml exists
[2026-04-20 14:11:35,482][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:11:35,503][  INFO][PID:2663577] VM allocation process starts
[2026-04-20 14:11:35,649][  INFO][PID:2663577] Trying to allocate VM: ResallocHost, ticket_id=2423681, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:11:40,661][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:11:45,689][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:11:50,716][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:11:55,740][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:12:00,775][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:12:05,788][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:12:10,798][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:12:15,810][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:12:20,839][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:12:25,854][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:12:30,859][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:12:35,867][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:12:40,871][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:12:45,878][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:12:50,886][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:12:55,889][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:13:00,896][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:13:05,902][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:13:10,918][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:13:15,939][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:13:20,960][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:13:25,977][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:13:30,979][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:13:35,987][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:13:40,997][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:13:46,008][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:13:51,020][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:13:56,027][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:14:01,029][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:14:06,038][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:14:11,043][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:14:16,047][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:14:21,061][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:14:26,074][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:14:31,082][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:14:36,086][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:14:41,104][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:14:46,131][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:14:51,149][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:14:54,647][  INFO][PID:2663577] Allocated host ResallocHost, ticket_id=2423681, hostname=54.173.234.106, name=aws_aarch64_reserved_prod_07134663_20260420_141111, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:14:54,652][  INFO][PID:2663577] Allocating ssh connection to builder
[2026-04-20 14:14:54,653][  INFO][PID:2663577] Checking that builder machine is OK
[2026-04-20 14:14:54,848][ ERROR][PID:2663577] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '54.173.234.106' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@54.173.234.106: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:15:04,996][ ERROR][PID:2663577] Re-try request for task on 'ResallocHost, ticket_id=2423681, hostname=54.173.234.106, name=aws_aarch64_reserved_prod_07134663_20260420_141111, requested_tags=['arch_aarch64', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:15:05,046][  INFO][PID:2663577] VM Release request
[2026-04-20 14:15:05,081][  INFO][PID:2663577] Retry #2 (on other host)
[2026-04-20 14:15:05,083][  INFO][PID:2663577] Checking that https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64//repodata/repomd.xml exists
[2026-04-20 14:15:05,852][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:15:05,867][  INFO][PID:2663577] VM allocation process starts
[2026-04-20 14:15:05,957][  INFO][PID:2663577] Trying to allocate VM: ResallocHost, ticket_id=2423801, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:15:10,969][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:15:16,006][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:15:21,008][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:15:26,010][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:15:31,016][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:15:36,030][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:15:41,048][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:15:46,059][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:15:51,061][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:15:56,066][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:16:01,079][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:16:06,090][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:16:11,092][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:16:16,124][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:16:21,132][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:16:26,159][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:16:31,180][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:16:36,203][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:16:41,208][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:16:46,213][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:16:51,222][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:16:56,233][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:17:01,274][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:17:06,288][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:17:11,307][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:17:16,310][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:17:21,332][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:17:26,335][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:17:31,343][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:17:36,349][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:17:41,365][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:17:46,377][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:17:51,394][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:17:56,399][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:18:01,407][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:18:06,427][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:18:11,434][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:18:16,447][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:18:21,460][  INFO][PID:2663577] Checking for cancel request
[2026-04-20 14:18:24,723][  INFO][PID:2663577] Allocated host ResallocHost, ticket_id=2423801, hostname=98.91.203.16, name=aws_aarch64_reserved_prod_07134859_20260420_141420, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:18:24,725][  INFO][PID:2663577] Allocating ssh connection to builder
[2026-04-20 14:18:24,731][  INFO][PID:2663577] Checking that builder machine is OK
[2026-04-20 14:18:25,024][ ERROR][PID:2663577] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '98.91.203.16' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@98.91.203.16: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:18:35,405][ ERROR][PID:2663577] Re-try request for task on 'ResallocHost, ticket_id=2423801, hostname=98.91.203.16, name=aws_aarch64_reserved_prod_07134859_20260420_141420, requested_tags=['arch_aarch64', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:18:35,682][  INFO][PID:2663577] VM Release request
[2026-04-20 14:18:35,793][ ERROR][PID:2663577] Backend process error: Three host tried without success: {'44.202.106.240', '98.91.203.16', '54.173.234.106'}
[2026-04-20 14:18:35,871][WARNING][PID:2663577] Switching not-finished job state to 'failed'
[2026-04-20 14:18:35,876][  INFO][PID:2663577] Worker failed build, took 696.4695398807526
[2026-04-20 14:18:35,877][  INFO][PID:2663577] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "xsnrg",
            "project_name": "indi-3rdparty-bleeding",
            "project_dirname": "indi-3rdparty-bleeding",
            "submitter": "https://api.github.com/users/knro",
            "ended_on": 1776694715.871183,
            "started_on": 1776694019.401643,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-43-aarch64",
            "arch": "aarch64",
            "buildroot_pkgs": [],
            "task_id": "10348527-fedora-43-aarch64",
            "build_id": 10348527,
            "package_name": "libstarshootg",
            "package_version": "2.1.3.git-20260420131439",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/xsnrg/indi-3rdparty-bleeding/libstarshootg",
            "git_hash": "ce67615a2007127d7c11da5330c76469ad2c66dd",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "xsnrg/indi-3rdparty-bleeding--https://api.github.com/users/knro",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 1,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xsnrg/libindi-bleeding/fedora-43-aarch64/",
                    "id": "copr_xsnrg_libindi_bleeding",
                    "name": "Additional repo copr_xsnrg_libindi_bleeding"
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64/",
                    "id": "copr_xsnrg_indi_3rdparty_bleeding",
                    "name": "Additional repo copr_xsnrg_indi_3rdparty_bleeding"
                }
            ],
            "background": true,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_aarch64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding",
            "result_dir": "10348527-libstarshootg",
            "built_packages": "",
            "id": 10348527,
            "mockchain_macros": {
                "copr_username": "xsnrg",
                "copr_projectname": "indi-3rdparty-bleeding",
                "vendor": "Fedora Project COPR (xsnrg/indi-3rdparty-bleeding)"
            }
        }
    ]
}
[2026-04-20 14:18:35,953][  INFO][PID:2663577] Sending fedora-messaging bus message in build.end
[2026-04-20 14:18:37,499][  INFO][PID:2663577] Compressing /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64/10348527-libstarshootg/builder-live.log by gzip
[2026-04-20 14:18:37,516][  INFO][PID:2663577] Running command 'gzip /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64/10348527-libstarshootg/builder-live.log' as PID 2817140
[2026-04-20 14:18:37,554][  INFO][PID:2663577] Finished after 0 seconds with exit code 1 (gzip /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64/10348527-libstarshootg/builder-live.log)
stdout:

stderr:
gzip: /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64/10348527-libstarshootg/builder-live.log: No such file or directory


[2026-04-20 14:18:37,556][ ERROR][PID:2663577] Unable to compress file /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64/10348527-libstarshootg/builder-live.log
[2026-04-20 14:18:37,581][  INFO][PID:2663577] Compressing /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64/10348527-libstarshootg/backend.log by gzip
[2026-04-20 14:18:37,590][  INFO][PID:2663577] Running command 'gzip /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64/10348527-libstarshootg/backend.log' as PID 2817162