[2026-04-20 14:09:37,336][  INFO][PID:2690280] Marking build as starting
[2026-04-20 14:09:37,472][  INFO][PID:2690280] Checking that https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64//repodata/repomd.xml exists
[2026-04-20 14:09:38,008][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:09:38,060][  INFO][PID:2690280] VM allocation process starts
[2026-04-20 14:09:38,221][  INFO][PID:2690280] Trying to allocate VM: ResallocHost, ticket_id=2423657, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:09:43,257][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:09:48,273][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:09:53,281][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:09:58,301][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:10:03,320][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:10:08,336][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:10:13,347][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:10:18,365][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:10:23,401][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:10:28,424][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:10:33,426][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:10:38,434][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:10:43,442][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:10:48,464][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:10:53,477][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:10:58,497][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:11:03,515][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:11:08,520][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:11:13,532][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:11:18,541][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:11:23,548][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:11:28,559][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:11:33,581][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:11:38,588][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:11:43,598][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:11:48,615][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:11:53,618][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:11:58,623][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:12:03,626][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:12:08,655][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:12:13,657][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:12:18,665][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:12:23,673][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:12:28,683][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:12:33,688][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:12:38,701][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:12:43,712][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:12:48,719][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:12:53,721][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:12:58,724][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:13:03,795][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:13:08,834][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:13:13,849][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:13:18,892][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:13:23,911][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:13:28,947][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:13:33,975][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:13:38,988][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:13:44,025][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:13:49,047][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:13:54,071][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:13:59,084][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:14:04,097][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:14:09,167][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:14:14,175][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:14:19,182][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:14:24,201][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:14:29,293][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:14:34,315][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:14:39,322][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:14:44,331][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:14:49,386][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:14:54,389][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:14:57,401][  INFO][PID:2690280] Allocated host ResallocHost, ticket_id=2423657, hostname=18.215.163.169, name=aws_aarch64_reserved_prod_07134632_20260420_141052, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:14:57,409][  INFO][PID:2690280] Allocating ssh connection to builder
[2026-04-20 14:14:57,426][  INFO][PID:2690280] Checking that builder machine is OK
[2026-04-20 14:14:57,860][ ERROR][PID:2690280] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '18.215.163.169' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@18.215.163.169: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:15:07,988][ ERROR][PID:2690280] Re-try request for task on 'ResallocHost, ticket_id=2423657, hostname=18.215.163.169, name=aws_aarch64_reserved_prod_07134632_20260420_141052, requested_tags=['arch_aarch64', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:15:08,269][  INFO][PID:2690280] VM Release request
[2026-04-20 14:15:08,333][  INFO][PID:2690280] Retry #1 (on other host)
[2026-04-20 14:15:08,339][  INFO][PID:2690280] Checking that https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64//repodata/repomd.xml exists
[2026-04-20 14:15:08,940][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:15:08,976][  INFO][PID:2690280] VM allocation process starts
[2026-04-20 14:15:09,026][  INFO][PID:2690280] Trying to allocate VM: ResallocHost, ticket_id=2423803, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:15:14,041][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:15:19,054][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:15:24,076][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:15:29,104][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:15:34,110][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:15:39,127][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:15:44,132][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:15:49,137][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:15:54,143][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:15:59,150][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:16:04,154][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:16:09,158][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:16:14,175][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:16:19,190][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:16:24,199][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:16:29,203][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:16:34,212][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:16:39,216][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:16:44,222][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:16:49,241][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:16:54,259][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:16:59,276][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:17:04,288][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:17:09,291][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:17:14,308][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:17:19,337][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:17:24,349][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:17:29,367][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:17:34,378][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:17:39,383][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:17:44,400][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:17:49,424][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:17:54,430][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:17:59,444][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:18:04,469][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:18:09,490][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:18:14,517][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:18:19,522][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:18:24,540][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:18:27,887][  INFO][PID:2690280] Allocated host ResallocHost, ticket_id=2423803, hostname=54.167.202.32, name=aws_aarch64_reserved_prod_07134861_20260420_141420, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:18:27,888][  INFO][PID:2690280] Allocating ssh connection to builder
[2026-04-20 14:18:27,891][  INFO][PID:2690280] Checking that builder machine is OK
[2026-04-20 14:18:28,286][ ERROR][PID:2690280] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '54.167.202.32' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@54.167.202.32: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:18:38,323][ ERROR][PID:2690280] Re-try request for task on 'ResallocHost, ticket_id=2423803, hostname=54.167.202.32, name=aws_aarch64_reserved_prod_07134861_20260420_141420, requested_tags=['arch_aarch64', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:18:38,361][  INFO][PID:2690280] VM Release request
[2026-04-20 14:18:38,394][  INFO][PID:2690280] Retry #2 (on other host)
[2026-04-20 14:18:38,396][  INFO][PID:2690280] Checking that https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64//repodata/repomd.xml exists
[2026-04-20 14:18:39,406][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:18:39,462][  INFO][PID:2690280] VM allocation process starts
[2026-04-20 14:18:39,540][  INFO][PID:2690280] Trying to allocate VM: ResallocHost, ticket_id=2423879, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:18:44,551][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:18:49,559][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:18:54,565][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:18:59,571][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:19:04,583][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:19:09,585][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:19:14,611][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:19:19,622][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:19:24,632][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:19:29,636][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:19:34,648][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:19:39,655][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:19:44,659][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:19:49,668][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:19:54,675][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:19:59,683][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:20:04,689][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:20:09,714][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:20:14,721][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:20:19,727][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:20:24,734][  INFO][PID:2690280] Checking for cancel request
[2026-04-20 14:20:28,352][  INFO][PID:2690280] Allocated host ResallocHost, ticket_id=2423879, hostname=54.196.125.163, name=aws_aarch64_reserved_prod_07135059_20260420_141721, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:20:28,353][  INFO][PID:2690280] Allocating ssh connection to builder
[2026-04-20 14:20:28,358][  INFO][PID:2690280] Checking that builder machine is OK
[2026-04-20 14:20:28,666][ ERROR][PID:2690280] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '54.196.125.163' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@54.196.125.163: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:20:38,801][ ERROR][PID:2690280] Re-try request for task on 'ResallocHost, ticket_id=2423879, hostname=54.196.125.163, name=aws_aarch64_reserved_prod_07135059_20260420_141721, requested_tags=['arch_aarch64', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:20:38,955][  INFO][PID:2690280] VM Release request
[2026-04-20 14:20:39,030][ ERROR][PID:2690280] Backend process error: Three host tried without success: {'18.215.163.169', '54.167.202.32', '54.196.125.163'}
[2026-04-20 14:20:39,217][WARNING][PID:2690280] Switching not-finished job state to 'failed'
[2026-04-20 14:20:39,237][  INFO][PID:2690280] Worker failed build, took 661.8910598754883
[2026-04-20 14:20:39,242][  INFO][PID:2690280] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "xsnrg",
            "project_name": "indi-3rdparty-bleeding",
            "project_dirname": "indi-3rdparty-bleeding",
            "submitter": "https://api.github.com/users/knro",
            "ended_on": 1776694839.217124,
            "started_on": 1776694177.326064,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-aarch64",
            "arch": "aarch64",
            "buildroot_pkgs": [],
            "task_id": "10348529-fedora-rawhide-aarch64",
            "build_id": 10348529,
            "package_name": "libsvbony",
            "package_version": "2.1.3.git-20260420131439",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/xsnrg/indi-3rdparty-bleeding/libsvbony",
            "git_hash": "aeacca6a079c3ee6658a96e8231699500f66d802",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "xsnrg/indi-3rdparty-bleeding--https://api.github.com/users/knro",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 1,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xsnrg/libindi-bleeding/fedora-rawhide-aarch64/",
                    "id": "copr_xsnrg_libindi_bleeding",
                    "name": "Additional repo copr_xsnrg_libindi_bleeding"
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/",
                    "id": "copr_xsnrg_indi_3rdparty_bleeding",
                    "name": "Additional repo copr_xsnrg_indi_3rdparty_bleeding"
                }
            ],
            "background": true,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_aarch64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding",
            "result_dir": "10348529-libsvbony",
            "built_packages": "",
            "id": 10348529,
            "mockchain_macros": {
                "copr_username": "xsnrg",
                "copr_projectname": "indi-3rdparty-bleeding",
                "vendor": "Fedora Project COPR (xsnrg/indi-3rdparty-bleeding)"
            }
        }
    ]
}
[2026-04-20 14:20:39,580][  INFO][PID:2690280] Sending fedora-messaging bus message in build.end
[2026-04-20 14:20:42,128][  INFO][PID:2690280] Compressing /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348529-libsvbony/builder-live.log by gzip
[2026-04-20 14:20:42,143][  INFO][PID:2690280] Running command 'gzip /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348529-libsvbony/builder-live.log' as PID 2848485
[2026-04-20 14:20:42,152][  INFO][PID:2690280] Finished after 0 seconds with exit code 1 (gzip /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348529-libsvbony/builder-live.log)
stdout:

stderr:
gzip: /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348529-libsvbony/builder-live.log: No such file or directory


[2026-04-20 14:20:42,210][ ERROR][PID:2690280] Unable to compress file /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348529-libsvbony/builder-live.log
[2026-04-20 14:20:42,546][  INFO][PID:2690280] Compressing /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348529-libsvbony/backend.log by gzip
[2026-04-20 14:20:42,572][  INFO][PID:2690280] Running command 'gzip /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348529-libsvbony/backend.log' as PID 2848579