[2026-04-20 13:58:00,231][ INFO][PID:2508357] Marking build as starting [2026-04-20 13:58:00,375][ INFO][PID:2508357] Checking that https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64//repodata/repomd.xml exists [2026-04-20 13:58:01,009][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:58:01,079][ INFO][PID:2508357] VM allocation process starts [2026-04-20 13:58:01,172][ INFO][PID:2508357] Trying to allocate VM: ResallocHost, ticket_id=2423336, requested_tags=['arch_aarch64', 'copr_builder'] [2026-04-20 13:58:06,193][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:58:11,198][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:58:16,202][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:58:21,214][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:58:26,220][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:58:31,222][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:58:36,247][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:58:41,249][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:58:46,252][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:58:51,265][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:58:56,267][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:59:01,278][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:59:06,300][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:59:11,315][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:59:16,343][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:59:21,366][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:59:26,377][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:59:31,380][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:59:36,390][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:59:41,395][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:59:46,411][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:59:51,417][ INFO][PID:2508357] Checking for cancel request [2026-04-20 13:59:56,422][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:00:01,426][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:00:06,448][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:00:11,463][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:00:16,465][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:00:21,471][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:00:26,481][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:00:31,484][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:00:36,505][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:00:41,512][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:00:46,517][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:00:51,553][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:00:56,579][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:01:01,588][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:01:06,594][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:01:11,608][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:01:16,613][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:01:21,644][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:01:26,672][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:01:31,681][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:01:36,719][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:01:41,743][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:01:46,761][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:01:51,786][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:01:56,819][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:02:01,847][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:02:06,856][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:02:11,860][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:02:16,890][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:02:21,892][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:02:26,913][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:02:31,929][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:02:36,943][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:02:41,982][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:02:46,991][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:02:52,023][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:02:57,083][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:03:02,093][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:03:07,118][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:03:12,127][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:03:17,139][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:03:19,894][ INFO][PID:2508357] Allocated host ResallocHost, ticket_id=2423336, hostname=98.80.71.92, name=aws_aarch64_normal_prod_07133956_20260420_135932, requested_tags=['arch_aarch64', 'copr_builder'] [2026-04-20 14:03:19,896][ INFO][PID:2508357] Allocating ssh connection to builder [2026-04-20 14:03:19,898][ INFO][PID:2508357] Checking that builder machine is OK [2026-04-20 14:03:20,204][ ERROR][PID:2508357] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: Warning: Permanently added '98.80.71.92' (ED25519) to the list of known hosts. Load key "/home/copr/.ssh/id_rsa": Permission denied Permission denied, please try again. Permission denied, please try again. mockbuilder@98.80.71.92: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). [2026-04-20 14:03:30,443][ ERROR][PID:2508357] Re-try request for task on 'ResallocHost, ticket_id=2423336, hostname=98.80.71.92, name=aws_aarch64_normal_prod_07133956_20260420_135932, requested_tags=['arch_aarch64', 'copr_builder']': Unable to finish after 1 SSH attempts [2026-04-20 14:03:30,650][ INFO][PID:2508357] VM Release request [2026-04-20 14:03:30,719][ INFO][PID:2508357] Retry #1 (on other host) [2026-04-20 14:03:30,723][ INFO][PID:2508357] Checking that https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64//repodata/repomd.xml exists [2026-04-20 14:03:32,316][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:03:32,350][ INFO][PID:2508357] VM allocation process starts [2026-04-20 14:03:32,452][ INFO][PID:2508357] Trying to allocate VM: ResallocHost, ticket_id=2423490, requested_tags=['arch_aarch64', 'copr_builder'] [2026-04-20 14:03:37,481][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:03:42,484][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:03:47,488][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:03:52,493][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:03:57,514][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:04:02,519][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:04:07,522][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:04:12,550][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:04:17,554][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:04:22,565][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:04:27,597][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:04:32,606][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:04:37,614][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:04:42,627][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:04:47,631][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:04:52,659][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:04:57,665][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:05:02,694][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:05:07,717][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:05:12,741][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:05:17,772][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:05:22,796][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:05:27,844][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:05:32,868][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:05:37,890][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:05:42,911][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:05:47,946][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:05:52,953][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:05:57,959][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:06:02,963][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:06:07,967][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:06:12,975][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:06:17,982][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:06:22,988][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:06:28,004][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:06:33,014][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:06:38,017][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:06:43,022][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:06:48,029][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:06:51,245][ INFO][PID:2508357] Allocated host ResallocHost, ticket_id=2423490, hostname=3.84.134.246, name=aws_aarch64_reserved_prod_07134143_20260420_140321, requested_tags=['arch_aarch64', 'copr_builder'] [2026-04-20 14:06:51,246][ INFO][PID:2508357] Allocating ssh connection to builder [2026-04-20 14:06:51,248][ INFO][PID:2508357] Checking that builder machine is OK [2026-04-20 14:06:51,682][ ERROR][PID:2508357] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: Warning: Permanently added '3.84.134.246' (ED25519) to the list of known hosts. Load key "/home/copr/.ssh/id_rsa": Permission denied Permission denied, please try again. Permission denied, please try again. mockbuilder@3.84.134.246: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). [2026-04-20 14:07:01,789][ ERROR][PID:2508357] Re-try request for task on 'ResallocHost, ticket_id=2423490, hostname=3.84.134.246, name=aws_aarch64_reserved_prod_07134143_20260420_140321, requested_tags=['arch_aarch64', 'copr_builder']': Unable to finish after 1 SSH attempts [2026-04-20 14:07:01,888][ INFO][PID:2508357] VM Release request [2026-04-20 14:07:01,939][ INFO][PID:2508357] Retry #2 (on other host) [2026-04-20 14:07:01,940][ INFO][PID:2508357] Checking that https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64//repodata/repomd.xml exists [2026-04-20 14:07:02,817][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:07:02,828][ INFO][PID:2508357] VM allocation process starts [2026-04-20 14:07:02,891][ INFO][PID:2508357] Trying to allocate VM: ResallocHost, ticket_id=2423611, requested_tags=['arch_aarch64', 'copr_builder'] [2026-04-20 14:07:07,901][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:07:12,916][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:07:17,929][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:07:22,935][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:07:27,940][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:07:32,953][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:07:37,961][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:07:42,986][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:07:47,989][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:07:53,000][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:07:58,006][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:08:03,012][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:08:08,017][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:08:13,019][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:08:18,026][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:08:23,034][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:08:28,038][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:08:33,047][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:08:38,051][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:08:43,055][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:08:48,062][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:08:53,068][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:08:58,075][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:09:03,084][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:09:08,116][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:09:13,131][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:09:18,134][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:09:23,154][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:09:28,186][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:09:33,201][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:09:38,209][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:09:43,219][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:09:48,226][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:09:53,230][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:09:58,241][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:10:03,270][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:10:08,280][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:10:13,290][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:10:18,297][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:10:23,329][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:10:28,333][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:10:33,335][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:10:38,357][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:10:43,373][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:10:48,384][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:10:53,402][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:10:58,417][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:11:03,421][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:11:08,444][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:11:13,446][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:11:18,453][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:11:23,460][ INFO][PID:2508357] Checking for cancel request [2026-04-20 14:11:24,855][ INFO][PID:2508357] Allocated host ResallocHost, ticket_id=2423611, hostname=18.208.165.98, name=aws_aarch64_reserved_prod_07134482_20260420_140739, requested_tags=['arch_aarch64', 'copr_builder'] [2026-04-20 14:11:24,856][ INFO][PID:2508357] Allocating ssh connection to builder [2026-04-20 14:11:24,857][ INFO][PID:2508357] Checking that builder machine is OK [2026-04-20 14:11:25,168][ ERROR][PID:2508357] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: Warning: Permanently added '18.208.165.98' (ED25519) to the list of known hosts. Load key "/home/copr/.ssh/id_rsa": Permission denied Permission denied, please try again. Permission denied, please try again. mockbuilder@18.208.165.98: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). [2026-04-20 14:11:35,201][ ERROR][PID:2508357] Re-try request for task on 'ResallocHost, ticket_id=2423611, hostname=18.208.165.98, name=aws_aarch64_reserved_prod_07134482_20260420_140739, requested_tags=['arch_aarch64', 'copr_builder']': Unable to finish after 1 SSH attempts [2026-04-20 14:11:35,445][ INFO][PID:2508357] VM Release request [2026-04-20 14:11:35,683][ ERROR][PID:2508357] Backend process error: Three host tried without success: {'3.84.134.246', '98.80.71.92', '18.208.165.98'} [2026-04-20 14:11:35,821][WARNING][PID:2508357] Switching not-finished job state to 'failed' [2026-04-20 14:11:35,827][ INFO][PID:2508357] Worker failed build, took 815.6323013305664 [2026-04-20 14:11:35,834][ INFO][PID:2508357] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "xsnrg", "project_name": "indi-3rdparty-bleeding", "project_dirname": "indi-3rdparty-bleeding", "submitter": "https://api.github.com/users/knro", "ended_on": 1776694295.8218467, "started_on": 1776693480.1895454, "submitted_on": null, "status": 0, "chroot": "fedora-43-aarch64", "arch": "aarch64", "buildroot_pkgs": [], "task_id": "10348524-fedora-43-aarch64", "build_id": 10348524, "package_name": "indi-qsi", "package_version": "2.1.3.git-20260420131439", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/xsnrg/indi-3rdparty-bleeding/indi-qsi", "git_hash": "e1cfdb6da0644cf1e458aac788fa5cd29f0c2ea4", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "xsnrg/indi-3rdparty-bleeding--https://api.github.com/users/knro", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 1, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/xsnrg/libindi-bleeding/fedora-43-aarch64/", "id": "copr_xsnrg_libindi_bleeding", "name": "Additional repo copr_xsnrg_libindi_bleeding" }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64/", "id": "copr_xsnrg_indi_3rdparty_bleeding", "name": "Additional repo copr_xsnrg_indi_3rdparty_bleeding" } ], "background": true, "bootstrap": "off", "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_aarch64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/xsnrg/indi-3rdparty-bleeding", "result_dir": "10348524-indi-qsi", "built_packages": "", "id": 10348524, "mockchain_macros": { "copr_username": "xsnrg", "copr_projectname": "indi-3rdparty-bleeding", "vendor": "Fedora Project COPR (xsnrg/indi-3rdparty-bleeding)" } } ] } [2026-04-20 14:11:36,076][ INFO][PID:2508357] Sending fedora-messaging bus message in build.end [2026-04-20 14:11:37,415][ INFO][PID:2508357] Compressing /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64/10348524-indi-qsi/builder-live.log by gzip [2026-04-20 14:11:37,419][ INFO][PID:2508357] Running command 'gzip /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64/10348524-indi-qsi/builder-live.log' as PID 2721078 [2026-04-20 14:11:37,429][ INFO][PID:2508357] Finished after 0 seconds with exit code 1 (gzip /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64/10348524-indi-qsi/builder-live.log) stdout: stderr: gzip: /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64/10348524-indi-qsi/builder-live.log: No such file or directory [2026-04-20 14:11:37,443][ ERROR][PID:2508357] Unable to compress file /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64/10348524-indi-qsi/builder-live.log [2026-04-20 14:11:37,562][ INFO][PID:2508357] Compressing /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64/10348524-indi-qsi/backend.log by gzip [2026-04-20 14:11:37,571][ INFO][PID:2508357] Running command 'gzip /var/lib/copr/public_html/results/xsnrg/indi-3rdparty-bleeding/fedora-43-aarch64/10348524-indi-qsi/backend.log' as PID 2721097