[2026-04-20 13:44:48,734][  INFO][PID:2323948] Marking build as starting
[2026-04-20 13:44:48,927][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:44:48,930][  INFO][PID:2323948] VM allocation process starts
[2026-04-20 13:44:48,980][  INFO][PID:2323948] Trying to allocate VM: ResallocHost, ticket_id=2422870, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:44:53,988][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:44:59,003][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:45:04,007][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:45:09,011][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:45:14,024][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:45:19,038][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:45:24,042][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:45:29,049][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:45:34,063][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:45:39,077][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:45:44,087][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:45:49,094][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:45:54,108][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:45:57,245][  INFO][PID:2323948] Allocated host ResallocHost, ticket_id=2422870, hostname=2620:52:6:1161:dead:beef:cafe:c159, name=vmhost_p09_02_prod_07132520_20260420_133322, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:45:57,246][  INFO][PID:2323948] Allocating ssh connection to builder
[2026-04-20 13:45:57,248][  INFO][PID:2323948] Checking that builder machine is OK
[2026-04-20 13:45:57,636][ ERROR][PID:2323948] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c159' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c159: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 13:46:07,712][ ERROR][PID:2323948] Re-try request for task on 'ResallocHost, ticket_id=2422870, hostname=2620:52:6:1161:dead:beef:cafe:c159, name=vmhost_p09_02_prod_07132520_20260420_133322, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 13:46:07,726][  INFO][PID:2323948] VM Release request
[2026-04-20 13:46:07,751][  INFO][PID:2323948] Retry #1 (on other host)
[2026-04-20 13:46:07,752][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:46:07,754][  INFO][PID:2323948] VM allocation process starts
[2026-04-20 13:46:07,792][  INFO][PID:2323948] Trying to allocate VM: ResallocHost, ticket_id=2422906, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:46:12,798][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:46:17,807][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:46:22,810][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:46:27,822][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:46:32,834][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:46:35,964][  INFO][PID:2323948] Allocated host ResallocHost, ticket_id=2422906, hostname=2620:52:6:1161:dead:beef:cafe:c159, name=vmhost_p09_02_prod_07132520_20260420_133322, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:46:35,966][  INFO][PID:2323948] Allocating ssh connection to builder
[2026-04-20 13:46:35,969][  INFO][PID:2323948] Checking that builder machine is OK
[2026-04-20 13:46:36,258][ ERROR][PID:2323948] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c159' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c159: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 13:46:46,342][ ERROR][PID:2323948] Re-try request for task on 'ResallocHost, ticket_id=2422906, hostname=2620:52:6:1161:dead:beef:cafe:c159, name=vmhost_p09_02_prod_07132520_20260420_133322, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 13:46:46,391][  INFO][PID:2323948] VM Release request
[2026-04-20 13:46:46,440][  INFO][PID:2323948] Retry #2 (on other host)
[2026-04-20 13:46:46,444][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:46:46,450][  INFO][PID:2323948] VM allocation process starts
[2026-04-20 13:46:46,488][  INFO][PID:2323948] Trying to allocate VM: ResallocHost, ticket_id=2422925, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:46:51,496][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:46:56,506][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:47:01,508][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:47:06,520][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:47:11,522][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:47:16,526][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:47:21,535][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:47:24,750][  INFO][PID:2323948] Allocated host ResallocHost, ticket_id=2422925, hostname=2620:52:6:1161:dead:beef:cafe:c159, name=vmhost_p09_02_prod_07132520_20260420_133322, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:47:24,754][  INFO][PID:2323948] Allocating ssh connection to builder
[2026-04-20 13:47:24,758][  INFO][PID:2323948] Checking that builder machine is OK
[2026-04-20 13:47:25,145][ ERROR][PID:2323948] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c159' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c159: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 13:47:35,266][ ERROR][PID:2323948] Re-try request for task on 'ResallocHost, ticket_id=2422925, hostname=2620:52:6:1161:dead:beef:cafe:c159, name=vmhost_p09_02_prod_07132520_20260420_133322, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 13:47:35,308][  INFO][PID:2323948] VM Release request
[2026-04-20 13:47:35,357][  INFO][PID:2323948] Retry #3 (on other host)
[2026-04-20 13:47:35,365][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:47:35,377][  INFO][PID:2323948] VM allocation process starts
[2026-04-20 13:47:35,473][  INFO][PID:2323948] Trying to allocate VM: ResallocHost, ticket_id=2422967, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:47:40,489][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:47:45,494][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:47:50,500][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:47:55,513][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:48:00,516][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:48:05,528][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:48:10,539][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:48:15,546][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:48:20,556][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:48:23,743][  INFO][PID:2323948] Allocated host ResallocHost, ticket_id=2422967, hostname=2620:52:6:1161:dead:beef:cafe:c159, name=vmhost_p09_02_prod_07132520_20260420_133322, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:48:23,744][  INFO][PID:2323948] Allocating ssh connection to builder
[2026-04-20 13:48:23,744][  INFO][PID:2323948] Checking that builder machine is OK
[2026-04-20 13:48:24,023][ ERROR][PID:2323948] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c159' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c159: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 13:48:34,039][ ERROR][PID:2323948] Re-try request for task on 'ResallocHost, ticket_id=2422967, hostname=2620:52:6:1161:dead:beef:cafe:c159, name=vmhost_p09_02_prod_07132520_20260420_133322, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 13:48:34,149][  INFO][PID:2323948] VM Release request
[2026-04-20 13:48:34,207][  INFO][PID:2323948] Retry #4 (on other host)
[2026-04-20 13:48:34,209][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:48:34,221][  INFO][PID:2323948] VM allocation process starts
[2026-04-20 13:48:34,323][  INFO][PID:2323948] Trying to allocate VM: ResallocHost, ticket_id=2423001, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:48:39,342][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:48:44,346][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:48:49,350][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:48:54,355][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:48:59,359][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:49:04,361][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:49:09,366][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:49:12,567][  INFO][PID:2323948] Allocated host ResallocHost, ticket_id=2423001, hostname=2620:52:6:1161:dead:beef:cafe:c159, name=vmhost_p09_02_prod_07132520_20260420_133322, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:49:12,568][  INFO][PID:2323948] Allocating ssh connection to builder
[2026-04-20 13:49:12,572][  INFO][PID:2323948] Checking that builder machine is OK
[2026-04-20 13:49:12,925][ ERROR][PID:2323948] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c159' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c159: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 13:49:22,991][ ERROR][PID:2323948] Re-try request for task on 'ResallocHost, ticket_id=2423001, hostname=2620:52:6:1161:dead:beef:cafe:c159, name=vmhost_p09_02_prod_07132520_20260420_133322, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 13:49:23,021][  INFO][PID:2323948] VM Release request
[2026-04-20 13:49:23,108][  INFO][PID:2323948] Retry #5 (on other host)
[2026-04-20 13:49:23,112][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:49:23,117][  INFO][PID:2323948] VM allocation process starts
[2026-04-20 13:49:23,212][  INFO][PID:2323948] Trying to allocate VM: ResallocHost, ticket_id=2423035, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:49:28,227][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:49:33,234][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:49:38,243][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:49:43,252][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:49:48,264][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:49:53,285][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:49:58,329][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:50:03,347][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:50:08,364][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:50:11,699][  INFO][PID:2323948] Allocated host ResallocHost, ticket_id=2423035, hostname=3.235.228.185, name=aws_x86_64_spot_prod_07133381_20260420_134727, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:50:11,701][  INFO][PID:2323948] Allocating ssh connection to builder
[2026-04-20 13:50:11,721][  INFO][PID:2323948] Checking that builder machine is OK
[2026-04-20 13:50:12,116][ ERROR][PID:2323948] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '3.235.228.185' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@3.235.228.185: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 13:50:22,398][ ERROR][PID:2323948] Re-try request for task on 'ResallocHost, ticket_id=2423035, hostname=3.235.228.185, name=aws_x86_64_spot_prod_07133381_20260420_134727, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 13:50:22,431][  INFO][PID:2323948] VM Release request
[2026-04-20 13:50:22,514][  INFO][PID:2323948] Retry #6 (on other host)
[2026-04-20 13:50:22,516][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:50:22,522][  INFO][PID:2323948] VM allocation process starts
[2026-04-20 13:50:22,565][  INFO][PID:2323948] Trying to allocate VM: ResallocHost, ticket_id=2423072, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:50:27,589][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:50:32,596][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:50:37,607][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:50:42,609][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:50:47,613][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:50:52,618][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:50:57,628][  INFO][PID:2323948] Checking for cancel request
[2026-04-20 13:51:00,866][  INFO][PID:2323948] Allocated host ResallocHost, ticket_id=2423072, hostname=2620:52:6:1161:dead:beef:cafe:c13f, name=vmhost_p09_04_prod_07132782_20260420_133748, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:51:00,868][  INFO][PID:2323948] Allocating ssh connection to builder
[2026-04-20 13:51:00,872][  INFO][PID:2323948] Checking that builder machine is OK
[2026-04-20 13:51:01,277][ ERROR][PID:2323948] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c13f' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c13f: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 13:51:11,393][ ERROR][PID:2323948] Re-try request for task on 'ResallocHost, ticket_id=2423072, hostname=2620:52:6:1161:dead:beef:cafe:c13f, name=vmhost_p09_04_prod_07132782_20260420_133748, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 13:51:11,562][  INFO][PID:2323948] VM Release request
[2026-04-20 13:51:11,600][ ERROR][PID:2323948] Backend process error: Three host tried without success: {'3.235.228.185', '2620:52:6:1161:dead:beef:cafe:c13f', '2620:52:6:1161:dead:beef:cafe:c159'}
[2026-04-20 13:51:11,647][WARNING][PID:2323948] Switching not-finished job state to 'failed'
[2026-04-20 13:51:11,663][  INFO][PID:2323948] Worker failed build, took 383.09557247161865
[2026-04-20 13:51:11,665][  INFO][PID:2323948] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "thrnciar",
            "project_name": "python-setuptools",
            "project_dirname": "python-setuptools",
            "submitter": "https://src.fedoraproject.org/user/mcrha",
            "ended_on": 1776693071.6465933,
            "started_on": 1776692688.5510209,
            "submitted_on": null,
            "status": 0,
            "chroot": "srpm-builds",
            "arch": null,
            "buildroot_pkgs": null,
            "task_id": "10348639",
            "build_id": 10348639,
            "package_name": "gtk4",
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 10,
            "source_json": "{\"clone_url\": \"https://src.fedoraproject.org/rpms/gtk4\", \"distgit\": \"fedora\", \"committish\": \"3462f3991e259c50fb6b75c11abb3fb48193ca35\"}",
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": null,
            "sandbox": "thrnciar/python-setuptools--https://src.fedoraproject.org/user/mcrha",
            "results": null,
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [],
            "background": true,
            "distributions_in_build": [
                "fedora-rawhide"
            ],
            "distributions_in_project": [
                "fedora-rawhide"
            ],
            "destdir": "/var/lib/copr/public_html/results/thrnciar/python-setuptools",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/thrnciar/python-setuptools",
            "result_dir": "10348639",
            "built_packages": "",
            "tags": [
                "arch_noarch"
            ],
            "id": 10348639,
            "mockchain_macros": {
                "copr_username": "thrnciar",
                "copr_projectname": "python-setuptools",
                "vendor": "Fedora Project COPR (thrnciar/python-setuptools)"
            }
        }
    ]
}
[2026-04-20 13:51:11,904][  INFO][PID:2323948] Sending fedora-messaging bus message in build.end
[2026-04-20 13:51:12,763][  INFO][PID:2323948] Compressing /var/lib/copr/public_html/results/thrnciar/python-setuptools/srpm-builds/10348639/builder-live.log by gzip
[2026-04-20 13:51:12,783][  INFO][PID:2323948] Running command 'gzip /var/lib/copr/public_html/results/thrnciar/python-setuptools/srpm-builds/10348639/builder-live.log' as PID 2416632
[2026-04-20 13:51:12,797][  INFO][PID:2323948] Finished after 0 seconds with exit code 1 (gzip /var/lib/copr/public_html/results/thrnciar/python-setuptools/srpm-builds/10348639/builder-live.log)
stdout:

stderr:
gzip: /var/lib/copr/public_html/results/thrnciar/python-setuptools/srpm-builds/10348639/builder-live.log: No such file or directory


[2026-04-20 13:51:12,802][ ERROR][PID:2323948] Unable to compress file /var/lib/copr/public_html/results/thrnciar/python-setuptools/srpm-builds/10348639/builder-live.log
[2026-04-20 13:51:12,967][  INFO][PID:2323948] Compressing /var/lib/copr/public_html/results/thrnciar/python-setuptools/srpm-builds/10348639/backend.log by gzip
[2026-04-20 13:51:12,980][  INFO][PID:2323948] Running command 'gzip /var/lib/copr/public_html/results/thrnciar/python-setuptools/srpm-builds/10348639/backend.log' as PID 2416683