[2025-12-08 12:15:20,836][  INFO][PID:2035860] Marking build as starting
[2025-12-08 12:15:21,034][  INFO][PID:2035860] Checking for cancel request
[2025-12-08 12:15:21,036][  INFO][PID:2035860] VM allocation process starts
[2025-12-08 12:15:21,050][  INFO][PID:2035860] Trying to allocate VM: ResallocHost, ticket_id=215941, requested_tags=['copr_builder', 'arch_x86_64']
[2025-12-08 12:15:26,052][  INFO][PID:2035860] Checking for cancel request
[2025-12-08 12:15:27,088][  INFO][PID:2035860] Allocated host ResallocHost, ticket_id=215941, hostname=2620:52:3:1:dead:beef:cafe:c15a, name=copr_hv_x86_64_02_prod_00209741_20251208_105450, requested_tags=['copr_builder', 'arch_x86_64']
[2025-12-08 12:15:27,088][  INFO][PID:2035860] Allocating ssh connection to builder
[2025-12-08 12:15:27,089][  INFO][PID:2035860] Checking that builder machine is OK
[2025-12-08 12:16:27,158][ ERROR][PID:2035860] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:3:1:dead:beef:cafe:c15a port 22: Connection timed out

[2025-12-08 12:16:37,186][ ERROR][PID:2035860] Re-try request for task on 'ResallocHost, ticket_id=215941, hostname=2620:52:3:1:dead:beef:cafe:c15a, name=copr_hv_x86_64_02_prod_00209741_20251208_105450, requested_tags=['copr_builder', 'arch_x86_64']': Unable to finish after 1 SSH attempts
[2025-12-08 12:16:37,187][  INFO][PID:2035860] Releasing VM back to pool
[2025-12-08 12:16:37,197][  INFO][PID:2035860] Retry #1 (on other host)
[2025-12-08 12:16:37,197][  INFO][PID:2035860] Checking for cancel request
[2025-12-08 12:16:37,198][  INFO][PID:2035860] VM allocation process starts
[2025-12-08 12:16:37,211][  INFO][PID:2035860] Trying to allocate VM: ResallocHost, ticket_id=215991, requested_tags=['copr_builder', 'arch_x86_64']
[2025-12-08 12:16:42,212][  INFO][PID:2035860] Checking for cancel request
[2025-12-08 12:16:43,242][  INFO][PID:2035860] Allocated host ResallocHost, ticket_id=215991, hostname=2620:52:3:1:dead:beef:cafe:c11b, name=copr_hv_x86_64_01_prod_00210138_20251208_112118, requested_tags=['copr_builder', 'arch_x86_64']
[2025-12-08 12:16:43,242][  INFO][PID:2035860] Allocating ssh connection to builder
[2025-12-08 12:16:43,243][  INFO][PID:2035860] Checking that builder machine is OK
[2025-12-08 12:17:43,312][ ERROR][PID:2035860] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:3:1:dead:beef:cafe:c11b port 22: Connection timed out

[2025-12-08 12:17:53,312][ ERROR][PID:2035860] Re-try request for task on 'ResallocHost, ticket_id=215991, hostname=2620:52:3:1:dead:beef:cafe:c11b, name=copr_hv_x86_64_01_prod_00210138_20251208_112118, requested_tags=['copr_builder', 'arch_x86_64']': Unable to finish after 1 SSH attempts
[2025-12-08 12:17:53,313][  INFO][PID:2035860] Releasing VM back to pool
[2025-12-08 12:17:53,383][  INFO][PID:2035860] Retry #2 (on other host)
[2025-12-08 12:17:53,384][  INFO][PID:2035860] Checking for cancel request
[2025-12-08 12:17:53,385][  INFO][PID:2035860] VM allocation process starts
[2025-12-08 12:17:53,466][  INFO][PID:2035860] Trying to allocate VM: ResallocHost, ticket_id=216067, requested_tags=['copr_builder', 'arch_x86_64']
[2025-12-08 12:17:58,467][  INFO][PID:2035860] Checking for cancel request
[2025-12-08 12:18:03,591][  INFO][PID:2035860] Checking for cancel request
[2025-12-08 12:18:08,596][  INFO][PID:2035860] Checking for cancel request
[2025-12-08 12:18:11,549][  INFO][PID:2035860] Allocated host ResallocHost, ticket_id=216067, hostname=2620:52:3:1:dead:beef:cafe:c109, name=copr_hv_x86_64_01_prod_00210220_20251208_112636, requested_tags=['copr_builder', 'arch_x86_64']
[2025-12-08 12:18:11,552][  INFO][PID:2035860] Allocating ssh connection to builder
[2025-12-08 12:18:11,553][  INFO][PID:2035860] Checking that builder machine is OK
[2025-12-08 12:19:11,623][ ERROR][PID:2035860] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:3:1:dead:beef:cafe:c109 port 22: Connection timed out

[2025-12-08 12:19:21,715][ ERROR][PID:2035860] Re-try request for task on 'ResallocHost, ticket_id=216067, hostname=2620:52:3:1:dead:beef:cafe:c109, name=copr_hv_x86_64_01_prod_00210220_20251208_112636, requested_tags=['copr_builder', 'arch_x86_64']': Unable to finish after 1 SSH attempts
[2025-12-08 12:19:21,716][  INFO][PID:2035860] Releasing VM back to pool
[2025-12-08 12:19:21,727][ ERROR][PID:2035860] Backend process error: Three host tried without success: {'2620:52:3:1:dead:beef:cafe:c109', '2620:52:3:1:dead:beef:cafe:c15a', '2620:52:3:1:dead:beef:cafe:c11b'}
[2025-12-08 12:19:21,728][WARNING][PID:2035860] Switching not-finished job state to 'failed'
[2025-12-08 12:19:21,728][  INFO][PID:2035860] Worker failed build, took 241.8315305709839
[2025-12-08 12:19:21,729][  INFO][PID:2035860] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "lbalhar",
            "project_name": "python-setuptools_rawhide",
            "project_dirname": "python-setuptools_rawhide",
            "submitter": "https://src.fedoraproject.org/user/music",
            "ended_on": 1765196361.7282965,
            "started_on": 1765196119.896766,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "9886077-fedora-rawhide-x86_64",
            "build_id": 9886077,
            "package_name": "snakemake",
            "package_version": "9.14.3-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/lbalhar/python-setuptools_rawhide/snakemake",
            "git_hash": "e4326cac4d2f741d876b7bac48d5d703e6925323",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "lbalhar/python-setuptools_rawhide--https://src.fedoraproject.org/user/music",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/lbalhar/python-setuptools_rawhide/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/lbalhar/python-setuptools_rawhide",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/lbalhar/python-setuptools_rawhide",
            "result_dir": "09886077-snakemake",
            "built_packages": "",
            "id": 9886077,
            "mockchain_macros": {
                "copr_username": "lbalhar",
                "copr_projectname": "python-setuptools_rawhide",
                "vendor": "Fedora Project COPR (lbalhar/python-setuptools_rawhide)"
            }
        }
    ]
}
[2025-12-08 12:19:21,766][  INFO][PID:2035860] Sending fedora-messaging bus message in build.end
[2025-12-08 12:19:22,336][  INFO][PID:2035860] Compressing /var/lib/copr/public_html/results/lbalhar/python-setuptools_rawhide/fedora-rawhide-x86_64/09886077-snakemake/builder-live.log by gzip
[2025-12-08 12:19:22,337][  INFO][PID:2035860] Running command 'gzip /var/lib/copr/public_html/results/lbalhar/python-setuptools_rawhide/fedora-rawhide-x86_64/09886077-snakemake/builder-live.log' as PID 2065931
[2025-12-08 12:19:22,338][  INFO][PID:2035860] Finished after 0 seconds with exit code 1 (gzip /var/lib/copr/public_html/results/lbalhar/python-setuptools_rawhide/fedora-rawhide-x86_64/09886077-snakemake/builder-live.log)
stdout:

stderr:
gzip: /var/lib/copr/public_html/results/lbalhar/python-setuptools_rawhide/fedora-rawhide-x86_64/09886077-snakemake/builder-live.log: No such file or directory


[2025-12-08 12:19:22,339][ ERROR][PID:2035860] Unable to compress file /var/lib/copr/public_html/results/lbalhar/python-setuptools_rawhide/fedora-rawhide-x86_64/09886077-snakemake/builder-live.log
[2025-12-08 12:19:22,339][  INFO][PID:2035860] Compressing /var/lib/copr/public_html/results/lbalhar/python-setuptools_rawhide/fedora-rawhide-x86_64/09886077-snakemake/backend.log by gzip
[2025-12-08 12:19:22,340][  INFO][PID:2035860] Running command 'gzip /var/lib/copr/public_html/results/lbalhar/python-setuptools_rawhide/fedora-rawhide-x86_64/09886077-snakemake/backend.log' as PID 2065932