[2026-04-20 14:03:38,750][  INFO][PID:2607315] Marking build as starting
[2026-04-20 14:03:38,925][  INFO][PID:2607315] Checking that https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64//repodata/repomd.xml exists
[2026-04-20 14:03:40,221][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:03:40,235][  INFO][PID:2607315] VM allocation process starts
[2026-04-20 14:03:40,285][  INFO][PID:2607315] Trying to allocate VM: ResallocHost, ticket_id=2423496, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:03:45,300][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:03:50,304][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:03:55,309][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:04:00,311][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:04:05,319][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:04:10,322][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:04:15,330][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:04:20,342][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:04:25,368][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:04:30,381][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:04:35,385][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:04:40,390][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:04:45,398][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:04:50,408][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:04:55,442][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:05:00,464][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:05:05,471][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:05:10,488][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:05:15,528][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:05:20,531][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:05:25,568][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:05:30,576][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:05:35,580][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:05:40,592][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:05:45,596][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:05:50,606][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:05:55,609][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:06:00,617][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:06:05,622][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:06:10,627][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:06:15,646][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:06:20,672][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:06:25,697][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:06:30,702][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:06:35,716][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:06:40,722][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:06:45,726][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:06:50,732][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:06:55,741][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:06:59,027][  INFO][PID:2607315] Allocated host ResallocHost, ticket_id=2423496, hostname=98.85.223.216, name=aws_aarch64_reserved_prod_07134145_20260420_140321, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:06:59,029][  INFO][PID:2607315] Allocating ssh connection to builder
[2026-04-20 14:06:59,031][  INFO][PID:2607315] Checking that builder machine is OK
[2026-04-20 14:06:59,461][ ERROR][PID:2607315] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '98.85.223.216' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@98.85.223.216: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:07:09,591][ ERROR][PID:2607315] Re-try request for task on 'ResallocHost, ticket_id=2423496, hostname=98.85.223.216, name=aws_aarch64_reserved_prod_07134145_20260420_140321, requested_tags=['arch_aarch64', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:07:09,618][  INFO][PID:2607315] VM Release request
[2026-04-20 14:07:09,666][  INFO][PID:2607315] Retry #1 (on other host)
[2026-04-20 14:07:09,668][  INFO][PID:2607315] Checking that https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64//repodata/repomd.xml exists
[2026-04-20 14:07:11,160][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:07:11,168][  INFO][PID:2607315] VM allocation process starts
[2026-04-20 14:07:11,245][  INFO][PID:2607315] Trying to allocate VM: ResallocHost, ticket_id=2423613, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:07:16,255][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:07:21,267][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:07:26,274][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:07:31,280][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:07:36,289][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:07:41,302][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:07:46,304][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:07:51,313][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:07:56,318][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:08:01,321][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:08:06,328][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:08:11,344][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:08:16,346][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:08:21,350][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:08:26,357][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:08:31,377][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:08:36,385][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:08:41,387][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:08:46,397][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:08:51,403][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:08:56,414][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:09:01,433][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:09:06,445][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:09:11,455][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:09:16,477][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:09:21,495][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:09:26,523][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:09:31,539][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:09:36,576][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:09:41,603][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:09:46,611][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:09:51,615][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:09:56,623][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:10:01,647][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:10:06,672][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:10:11,690][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:10:16,709][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:10:21,720][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:10:26,747][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:10:31,756][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:10:36,761][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:10:41,763][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:10:46,766][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:10:51,776][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:10:56,789][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:11:01,803][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:11:06,805][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:11:11,810][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:11:16,827][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:11:21,835][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:11:26,841][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:11:31,846][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:11:33,127][  INFO][PID:2607315] Allocated host ResallocHost, ticket_id=2423613, hostname=54.196.32.172, name=aws_aarch64_reserved_prod_07134494_20260420_140752, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:11:33,130][  INFO][PID:2607315] Allocating ssh connection to builder
[2026-04-20 14:11:33,140][  INFO][PID:2607315] Checking that builder machine is OK
[2026-04-20 14:11:33,407][ ERROR][PID:2607315] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '54.196.32.172' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@54.196.32.172: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:11:43,535][ ERROR][PID:2607315] Re-try request for task on 'ResallocHost, ticket_id=2423613, hostname=54.196.32.172, name=aws_aarch64_reserved_prod_07134494_20260420_140752, requested_tags=['arch_aarch64', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:11:43,704][  INFO][PID:2607315] VM Release request
[2026-04-20 14:11:43,849][  INFO][PID:2607315] Retry #2 (on other host)
[2026-04-20 14:11:43,851][  INFO][PID:2607315] Checking that https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64//repodata/repomd.xml exists
[2026-04-20 14:11:45,049][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:11:45,058][  INFO][PID:2607315] VM allocation process starts
[2026-04-20 14:11:45,123][  INFO][PID:2607315] Trying to allocate VM: ResallocHost, ticket_id=2423690, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:11:50,139][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:11:55,155][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:12:00,157][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:12:05,170][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:12:10,172][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:12:15,181][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:12:20,202][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:12:25,218][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:12:30,243][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:12:35,262][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:12:40,271][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:12:45,279][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:12:50,288][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:12:55,292][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:13:00,304][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:13:05,309][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:13:10,318][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:13:15,331][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:13:20,345][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:13:25,359][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:13:30,362][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:13:35,364][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:13:40,373][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:13:45,391][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:13:50,408][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:13:55,414][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:14:00,419][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:14:05,431][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:14:10,434][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:14:15,442][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:14:20,449][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:14:25,454][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:14:30,463][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:14:35,488][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:14:40,493][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:14:45,510][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:14:50,545][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:14:55,552][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:15:00,571][  INFO][PID:2607315] Checking for cancel request
[2026-04-20 14:15:04,086][  INFO][PID:2607315] Allocated host ResallocHost, ticket_id=2423690, hostname=3.80.42.36, name=aws_aarch64_reserved_prod_07134667_20260420_141111, requested_tags=['arch_aarch64', 'copr_builder']
[2026-04-20 14:15:04,087][  INFO][PID:2607315] Allocating ssh connection to builder
[2026-04-20 14:15:04,090][  INFO][PID:2607315] Checking that builder machine is OK
[2026-04-20 14:15:04,271][ ERROR][PID:2607315] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '3.80.42.36' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@3.80.42.36: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:15:14,346][ ERROR][PID:2607315] Re-try request for task on 'ResallocHost, ticket_id=2423690, hostname=3.80.42.36, name=aws_aarch64_reserved_prod_07134667_20260420_141111, requested_tags=['arch_aarch64', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:15:14,561][  INFO][PID:2607315] VM Release request
[2026-04-20 14:15:14,679][ ERROR][PID:2607315] Backend process error: Three host tried without success: {'54.196.32.172', '3.80.42.36', '98.85.223.216'}
[2026-04-20 14:15:14,889][WARNING][PID:2607315] Switching not-finished job state to 'failed'
[2026-04-20 14:15:14,900][  INFO][PID:2607315] Worker failed build, took 696.1804840564728
[2026-04-20 14:15:14,901][  INFO][PID:2607315] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "xsnrg",
            "project_name": "indi-3rdparty-bleeding",
            "project_dirname": "indi-3rdparty-bleeding",
            "submitter": "https://api.github.com/users/knro",
            "ended_on": 1776694514.8896673,
            "started_on": 1776693818.7091832,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-aarch64",
            "arch": "aarch64",
            "buildroot_pkgs": [],
            "task_id": "10348526-fedora-rawhide-aarch64",
            "build_id": 10348526,
            "package_name": "libomegonprocam",
            "package_version": "2.1.3.git-20260420131438",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/xsnrg/indi-3rdparty-bleeding/libomegonprocam",
            "git_hash": "469251a50fb44fa02803a6875b5c713a81918d85",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "xsnrg/indi-3rdparty-bleeding--https://api.github.com/users/knro",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 1,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xsnrg/libindi-bleeding/fedora-rawhide-aarch64/",
                    "id": "copr_xsnrg_libindi_bleeding",
                    "name": "Additional repo copr_xsnrg_libindi_bleeding"
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/",
                    "id": "copr_xsnrg_indi_3rdparty_bleeding",
                    "name": "Additional repo copr_xsnrg_indi_3rdparty_bleeding"
                }
            ],
            "background": true,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_aarch64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding",
            "result_dir": "10348526-libomegonprocam",
            "built_packages": "",
            "id": 10348526,
            "mockchain_macros": {
                "copr_username": "xsnrg",
                "copr_projectname": "indi-3rdparty-bleeding",
                "vendor": "Fedora Project COPR (xsnrg/indi-3rdparty-bleeding)"
            }
        }
    ]
}
[2026-04-20 14:15:15,160][  INFO][PID:2607315] Sending fedora-messaging bus message in build.end
[2026-04-20 14:15:21,287][  INFO][PID:2607315] Compressing /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348526-libomegonprocam/builder-live.log by gzip
[2026-04-20 14:15:21,338][  INFO][PID:2607315] Running command 'gzip /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348526-libomegonprocam/builder-live.log' as PID 2771623
[2026-04-20 14:15:21,373][  INFO][PID:2607315] Finished after 0 seconds with exit code 1 (gzip /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348526-libomegonprocam/builder-live.log)
stdout:

stderr:
gzip: /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348526-libomegonprocam/builder-live.log: No such file or directory


[2026-04-20 14:15:21,374][ ERROR][PID:2607315] Unable to compress file /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348526-libomegonprocam/builder-live.log
[2026-04-20 14:15:21,571][  INFO][PID:2607315] Compressing /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348526-libomegonprocam/backend.log by gzip
[2026-04-20 14:15:21,634][  INFO][PID:2607315] Running command 'gzip /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-rawhide-aarch64/10348526-libomegonprocam/backend.log' as PID 2771704