[2024-11-07 17:04:37,191][  INFO][PID:2278338] Marking build as starting
[2024-11-07 17:04:37,322][  INFO][PID:2278338] Trying to allocate VM
[2024-11-07 17:06:05,645][  INFO][PID:2278338] Allocated host ResallocHost, ticket_id=5757067, hostname=18.234.153.229
[2024-11-07 17:06:05,646][  INFO][PID:2278338] Allocating ssh connection to builder
[2024-11-07 17:06:05,647][  INFO][PID:2278338] Checking that builder machine is OK
[2024-11-07 17:06:05,983][  INFO][PID:2278338] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2024-11-07 17:06:06,162][  INFO][PID:2278338] 
[2024-11-07 17:06:06,163][  INFO][PID:2278338] Warning: Permanently added '18.234.153.229' (ED25519) to the list of known hosts.
bash: line 1: copr-builder-ready: command not found

[2024-11-07 17:06:06,163][ ERROR][PID:2278338] Re-try request for task on 'ResallocHost, ticket_id=5757067, hostname=18.234.153.229': Builder wasn't ready, trying a new one
[2024-11-07 17:06:06,164][  INFO][PID:2278338] Releasing VM back to pool
[2024-11-07 17:06:06,221][  INFO][PID:2278338] Retry #1 (on other host)
[2024-11-07 17:06:06,223][  INFO][PID:2278338] Trying to allocate VM
[2024-11-07 17:06:24,373][  INFO][PID:2278338] Allocated host ResallocHost, ticket_id=5757221, hostname=18.234.153.229
[2024-11-07 17:06:24,374][  INFO][PID:2278338] Allocating ssh connection to builder
[2024-11-07 17:06:24,375][  INFO][PID:2278338] Checking that builder machine is OK
[2024-11-07 17:06:24,703][  INFO][PID:2278338] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2024-11-07 17:06:24,890][  INFO][PID:2278338] 
[2024-11-07 17:06:24,893][  INFO][PID:2278338] Warning: Permanently added '18.234.153.229' (ED25519) to the list of known hosts.
bash: line 1: copr-builder-ready: command not found

[2024-11-07 17:06:24,896][ ERROR][PID:2278338] Re-try request for task on 'ResallocHost, ticket_id=5757221, hostname=18.234.153.229': Builder wasn't ready, trying a new one
[2024-11-07 17:06:24,896][  INFO][PID:2278338] Releasing VM back to pool
[2024-11-07 17:06:24,969][  INFO][PID:2278338] Retry #2 (on other host)
[2024-11-07 17:06:24,970][  INFO][PID:2278338] Trying to allocate VM
[2024-11-07 17:06:43,069][  INFO][PID:2278338] Allocated host ResallocHost, ticket_id=5757250, hostname=18.234.153.229
[2024-11-07 17:06:43,070][  INFO][PID:2278338] Allocating ssh connection to builder
[2024-11-07 17:06:43,070][  INFO][PID:2278338] Checking that builder machine is OK
[2024-11-07 17:06:43,398][  INFO][PID:2278338] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2024-11-07 17:06:43,585][  INFO][PID:2278338] 
[2024-11-07 17:06:43,586][  INFO][PID:2278338] Warning: Permanently added '18.234.153.229' (ED25519) to the list of known hosts.
bash: line 1: copr-builder-ready: command not found

[2024-11-07 17:06:43,586][ ERROR][PID:2278338] Re-try request for task on 'ResallocHost, ticket_id=5757250, hostname=18.234.153.229': Builder wasn't ready, trying a new one
[2024-11-07 17:06:43,587][  INFO][PID:2278338] Releasing VM back to pool
[2024-11-07 17:06:43,606][  INFO][PID:2278338] Retry #3 (on other host)
[2024-11-07 17:06:43,607][  INFO][PID:2278338] Trying to allocate VM
[2024-11-07 17:06:55,693][  INFO][PID:2278338] Allocated host ResallocHost, ticket_id=5757278, hostname=18.234.153.229
[2024-11-07 17:06:55,694][  INFO][PID:2278338] Allocating ssh connection to builder
[2024-11-07 17:06:55,694][  INFO][PID:2278338] Checking that builder machine is OK
[2024-11-07 17:06:56,030][  INFO][PID:2278338] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2024-11-07 17:06:56,215][  INFO][PID:2278338] 
[2024-11-07 17:06:56,216][  INFO][PID:2278338] Warning: Permanently added '18.234.153.229' (ED25519) to the list of known hosts.
bash: line 1: copr-builder-ready: command not found

[2024-11-07 17:06:56,216][ ERROR][PID:2278338] Re-try request for task on 'ResallocHost, ticket_id=5757278, hostname=18.234.153.229': Builder wasn't ready, trying a new one
[2024-11-07 17:06:56,217][  INFO][PID:2278338] Releasing VM back to pool
[2024-11-07 17:06:56,239][  INFO][PID:2278338] Retry #4 (on other host)
[2024-11-07 17:06:56,240][  INFO][PID:2278338] Trying to allocate VM
[2024-11-07 17:07:08,333][  INFO][PID:2278338] Allocated host ResallocHost, ticket_id=5757304, hostname=18.234.153.229
[2024-11-07 17:07:08,335][  INFO][PID:2278338] Allocating ssh connection to builder
[2024-11-07 17:07:08,336][  INFO][PID:2278338] Checking that builder machine is OK
[2024-11-07 17:08:08,408][ ERROR][PID:2278338] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 18.234.153.229 port 22: Connection timed out

[2024-11-07 17:08:18,409][ ERROR][PID:2278338] Re-try request for task on 'ResallocHost, ticket_id=5757304, hostname=18.234.153.229': Unable to finish after 1 SSH attempts
[2024-11-07 17:08:18,413][  INFO][PID:2278338] Releasing VM back to pool
[2024-11-07 17:08:18,434][  INFO][PID:2278338] Retry #5 (on other host)
[2024-11-07 17:08:18,436][  INFO][PID:2278338] Trying to allocate VM
[2024-11-07 17:09:46,686][  INFO][PID:2278338] Allocated host ResallocHost, ticket_id=5757413, hostname=2620:52:3:1:dead:beef:cafe:c1d8
[2024-11-07 17:09:46,687][  INFO][PID:2278338] Allocating ssh connection to builder
[2024-11-07 17:09:46,688][  INFO][PID:2278338] Checking that builder machine is OK
[2024-11-07 17:09:47,287][  INFO][PID:2278338] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2024-11-07 17:09:47,608][  INFO][PID:2278338] 
[2024-11-07 17:09:47,609][  INFO][PID:2278338] Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c1d8' (ED25519) to the list of known hosts.
bash: line 1: copr-builder-ready: command not found

[2024-11-07 17:09:47,609][ ERROR][PID:2278338] Re-try request for task on 'ResallocHost, ticket_id=5757413, hostname=2620:52:3:1:dead:beef:cafe:c1d8': Builder wasn't ready, trying a new one
[2024-11-07 17:09:47,610][  INFO][PID:2278338] Releasing VM back to pool
[2024-11-07 17:09:47,626][  INFO][PID:2278338] Retry #6 (on other host)
[2024-11-07 17:09:47,629][  INFO][PID:2278338] Trying to allocate VM
[2024-11-07 17:10:05,741][  INFO][PID:2278338] Allocated host ResallocHost, ticket_id=5757598, hostname=2620:52:3:1:dead:beef:cafe:c1d8
[2024-11-07 17:10:05,742][  INFO][PID:2278338] Allocating ssh connection to builder
[2024-11-07 17:10:05,742][  INFO][PID:2278338] Checking that builder machine is OK
[2024-11-07 17:10:06,320][  INFO][PID:2278338] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2024-11-07 17:10:06,633][  INFO][PID:2278338] 
[2024-11-07 17:10:06,637][  INFO][PID:2278338] Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c1d8' (ED25519) to the list of known hosts.
bash: line 1: copr-builder-ready: command not found

[2024-11-07 17:10:06,638][ ERROR][PID:2278338] Re-try request for task on 'ResallocHost, ticket_id=5757598, hostname=2620:52:3:1:dead:beef:cafe:c1d8': Builder wasn't ready, trying a new one
[2024-11-07 17:10:06,639][  INFO][PID:2278338] Releasing VM back to pool
[2024-11-07 17:10:06,706][  INFO][PID:2278338] Retry #7 (on other host)
[2024-11-07 17:10:06,708][  INFO][PID:2278338] Trying to allocate VM
[2024-11-07 17:10:24,854][  INFO][PID:2278338] Allocated host ResallocHost, ticket_id=5757654, hostname=2620:52:3:1:dead:beef:cafe:c1d8
[2024-11-07 17:10:24,855][  INFO][PID:2278338] Allocating ssh connection to builder
[2024-11-07 17:10:24,859][  INFO][PID:2278338] Checking that builder machine is OK
[2024-11-07 17:10:25,459][  INFO][PID:2278338] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2024-11-07 17:10:25,773][  INFO][PID:2278338] 
[2024-11-07 17:10:25,775][  INFO][PID:2278338] Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c1d8' (ED25519) to the list of known hosts.
bash: line 1: copr-builder-ready: command not found

[2024-11-07 17:10:25,776][ ERROR][PID:2278338] Re-try request for task on 'ResallocHost, ticket_id=5757654, hostname=2620:52:3:1:dead:beef:cafe:c1d8': Builder wasn't ready, trying a new one
[2024-11-07 17:10:25,777][  INFO][PID:2278338] Releasing VM back to pool
[2024-11-07 17:10:25,815][  INFO][PID:2278338] Retry #8 (on other host)
[2024-11-07 17:10:25,817][  INFO][PID:2278338] Trying to allocate VM
[2024-11-07 17:10:43,959][  INFO][PID:2278338] Allocated host ResallocHost, ticket_id=5757727, hostname=2620:52:3:1:dead:beef:cafe:c1d8
[2024-11-07 17:10:43,961][  INFO][PID:2278338] Allocating ssh connection to builder
[2024-11-07 17:10:43,962][  INFO][PID:2278338] Checking that builder machine is OK
[2024-11-07 17:10:44,630][  INFO][PID:2278338] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2024-11-07 17:10:44,978][  INFO][PID:2278338] 
[2024-11-07 17:10:44,978][  INFO][PID:2278338] Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c1d8' (ED25519) to the list of known hosts.
bash: line 1: copr-builder-ready: command not found

[2024-11-07 17:10:44,979][ ERROR][PID:2278338] Re-try request for task on 'ResallocHost, ticket_id=5757727, hostname=2620:52:3:1:dead:beef:cafe:c1d8': Builder wasn't ready, trying a new one
[2024-11-07 17:10:44,979][  INFO][PID:2278338] Releasing VM back to pool
[2024-11-07 17:10:45,013][  INFO][PID:2278338] Retry #9 (on other host)
[2024-11-07 17:10:45,014][  INFO][PID:2278338] Trying to allocate VM
[2024-11-07 17:10:57,161][  INFO][PID:2278338] Allocated host ResallocHost, ticket_id=5757782, hostname=2620:52:3:1:dead:beef:cafe:c1d8
[2024-11-07 17:10:57,164][  INFO][PID:2278338] Allocating ssh connection to builder
[2024-11-07 17:10:57,165][  INFO][PID:2278338] Checking that builder machine is OK
[2024-11-07 17:10:57,768][  INFO][PID:2278338] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2024-11-07 17:10:58,095][  INFO][PID:2278338] 
[2024-11-07 17:10:58,096][  INFO][PID:2278338] Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c1d8' (ED25519) to the list of known hosts.
bash: line 1: copr-builder-ready: command not found

[2024-11-07 17:10:58,096][ ERROR][PID:2278338] Re-try request for task on 'ResallocHost, ticket_id=5757782, hostname=2620:52:3:1:dead:beef:cafe:c1d8': Builder wasn't ready, trying a new one
[2024-11-07 17:10:58,097][  INFO][PID:2278338] Releasing VM back to pool
[2024-11-07 17:10:58,246][  INFO][PID:2278338] Retry #10 (on other host)
[2024-11-07 17:10:58,247][  INFO][PID:2278338] Trying to allocate VM
[2024-11-07 17:11:16,542][  INFO][PID:2278338] Allocated host ResallocHost, ticket_id=5757829, hostname=2620:52:3:1:dead:beef:cafe:c1d8
[2024-11-07 17:11:16,543][  INFO][PID:2278338] Allocating ssh connection to builder
[2024-11-07 17:11:16,544][  INFO][PID:2278338] Checking that builder machine is OK
[2024-11-07 17:11:17,124][  INFO][PID:2278338] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2024-11-07 17:11:17,442][  INFO][PID:2278338] 
[2024-11-07 17:11:17,443][  INFO][PID:2278338] Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c1d8' (ED25519) to the list of known hosts.
bash: line 1: copr-builder-ready: command not found

[2024-11-07 17:11:17,444][ ERROR][PID:2278338] Re-try request for task on 'ResallocHost, ticket_id=5757829, hostname=2620:52:3:1:dead:beef:cafe:c1d8': Builder wasn't ready, trying a new one
[2024-11-07 17:11:17,445][  INFO][PID:2278338] Releasing VM back to pool
[2024-11-07 17:11:17,485][  INFO][PID:2278338] Retry #11 (on other host)
[2024-11-07 17:11:17,487][  INFO][PID:2278338] Trying to allocate VM
[2024-11-07 17:11:35,632][  INFO][PID:2278338] Allocated host ResallocHost, ticket_id=5757886, hostname=2620:52:3:1:dead:beef:cafe:c1d8
[2024-11-07 17:11:35,633][  INFO][PID:2278338] Allocating ssh connection to builder
[2024-11-07 17:11:35,634][  INFO][PID:2278338] Checking that builder machine is OK
[2024-11-07 17:11:36,222][  INFO][PID:2278338] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2024-11-07 17:11:36,549][  INFO][PID:2278338] 
[2024-11-07 17:11:36,553][  INFO][PID:2278338] Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c1d8' (ED25519) to the list of known hosts.
bash: line 1: copr-builder-ready: command not found

[2024-11-07 17:11:36,555][ ERROR][PID:2278338] Re-try request for task on 'ResallocHost, ticket_id=5757886, hostname=2620:52:3:1:dead:beef:cafe:c1d8': Builder wasn't ready, trying a new one
[2024-11-07 17:11:36,556][  INFO][PID:2278338] Releasing VM back to pool
[2024-11-07 17:11:36,573][  INFO][PID:2278338] Retry #12 (on other host)
[2024-11-07 17:11:36,575][  INFO][PID:2278338] Trying to allocate VM
[2024-11-07 17:11:54,686][  INFO][PID:2278338] Allocated host ResallocHost, ticket_id=5757919, hostname=2620:52:3:1:dead:beef:cafe:c1d8
[2024-11-07 17:11:54,687][  INFO][PID:2278338] Allocating ssh connection to builder
[2024-11-07 17:11:54,688][  INFO][PID:2278338] Checking that builder machine is OK
[2024-11-07 17:11:55,246][  INFO][PID:2278338] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2024-11-07 17:11:55,803][  INFO][PID:2278338] 
[2024-11-07 17:11:55,804][  INFO][PID:2278338] Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c1d8' (ED25519) to the list of known hosts.
bash: line 1: copr-builder-ready: command not found

[2024-11-07 17:11:55,804][ ERROR][PID:2278338] Re-try request for task on 'ResallocHost, ticket_id=5757919, hostname=2620:52:3:1:dead:beef:cafe:c1d8': Builder wasn't ready, trying a new one
[2024-11-07 17:11:55,805][  INFO][PID:2278338] Releasing VM back to pool
[2024-11-07 17:11:55,828][  INFO][PID:2278338] Retry #13 (on other host)
[2024-11-07 17:11:55,829][  INFO][PID:2278338] Trying to allocate VM
[2024-11-07 17:12:13,947][  INFO][PID:2278338] Allocated host ResallocHost, ticket_id=5757966, hostname=2620:52:3:1:dead:beef:cafe:c1d8
[2024-11-07 17:12:13,948][  INFO][PID:2278338] Allocating ssh connection to builder
[2024-11-07 17:12:13,948][  INFO][PID:2278338] Checking that builder machine is OK
[2024-11-07 17:12:14,556][  INFO][PID:2278338] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2024-11-07 17:12:14,890][  INFO][PID:2278338] 
[2024-11-07 17:12:14,891][  INFO][PID:2278338] Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c1d8' (ED25519) to the list of known hosts.
bash: line 1: copr-builder-ready: command not found

[2024-11-07 17:12:14,891][ ERROR][PID:2278338] Re-try request for task on 'ResallocHost, ticket_id=5757966, hostname=2620:52:3:1:dead:beef:cafe:c1d8': Builder wasn't ready, trying a new one
[2024-11-07 17:12:14,892][  INFO][PID:2278338] Releasing VM back to pool
[2024-11-07 17:12:14,908][  INFO][PID:2278338] Retry #14 (on other host)
[2024-11-07 17:12:14,911][  INFO][PID:2278338] Trying to allocate VM
[2024-11-07 17:13:43,060][  INFO][PID:2278338] Allocated host ResallocHost, ticket_id=5758001, hostname=54.158.5.155
[2024-11-07 17:13:43,061][  INFO][PID:2278338] Allocating ssh connection to builder
[2024-11-07 17:13:43,062][  INFO][PID:2278338] Checking that builder machine is OK
[2024-11-07 17:13:43,424][  INFO][PID:2278338] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2024-11-07 17:13:43,624][  INFO][PID:2278338] 
[2024-11-07 17:13:43,625][  INFO][PID:2278338] Warning: Permanently added '54.158.5.155' (ED25519) to the list of known hosts.
bash: line 1: copr-builder-ready: command not found

[2024-11-07 17:13:43,626][ ERROR][PID:2278338] Re-try request for task on 'ResallocHost, ticket_id=5758001, hostname=54.158.5.155': Builder wasn't ready, trying a new one
[2024-11-07 17:13:43,626][  INFO][PID:2278338] Releasing VM back to pool
[2024-11-07 17:13:43,647][ ERROR][PID:2278338] Backend process error: Three host tried without success: {'18.234.153.229', '54.158.5.155', '2620:52:3:1:dead:beef:cafe:c1d8'}
[2024-11-07 17:13:43,648][WARNING][PID:2278338] Switching not-finished job state to 'failed'
[2024-11-07 17:13:43,649][  INFO][PID:2278338] Worker failed build, took 546.4875848293304
[2024-11-07 17:13:43,650][  INFO][PID:2278338] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "thrnciar",
            "project_name": "pytest",
            "project_dirname": "pytest:pr:120",
            "submitter": "https://src.fedoraproject.org/user/msuchy",
            "ended_on": 1730999623.6488588,
            "started_on": 1730999077.161274,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "8227074-fedora-rawhide-x86_64",
            "build_id": 8227074,
            "package_name": "python-setuptools",
            "package_version": "74.1.3-3",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/thrnciar/pytest:pr:120/python-setuptools",
            "git_hash": "f041caefba62c118f9bae2e252ec933d9804b13f",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "thrnciar/pytest--https://src.fedoraproject.org/user/msuchy",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/thrnciar/pytest/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/thrnciar/pytest:pr:120",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/thrnciar/pytest:pr:120",
            "result_dir": "08227074-python-setuptools",
            "built_packages": "",
            "id": 8227074,
            "mockchain_macros": {
                "copr_username": "thrnciar",
                "copr_projectname": "pytest",
                "vendor": "Fedora Project COPR (thrnciar/pytest)"
            }
        }
    ]
}
[2024-11-07 17:13:43,710][  INFO][PID:2278338] Sending fedora-messaging bus message in build.end
[2024-11-07 17:13:44,439][  INFO][PID:2278338] Compressing /var/lib/copr/public_html/results/thrnciar/pytest:pr:120/fedora-rawhide-x86_64/08227074-python-setuptools/builder-live.log by gzip
[2024-11-07 17:13:44,442][  INFO][PID:2278338] Running command 'gzip /var/lib/copr/public_html/results/thrnciar/pytest:pr:120/fedora-rawhide-x86_64/08227074-python-setuptools/builder-live.log' as PID 2382973
[2024-11-07 17:13:44,443][  INFO][PID:2278338] Finished after 0 seconds with exit code 1 (gzip /var/lib/copr/public_html/results/thrnciar/pytest:pr:120/fedora-rawhide-x86_64/08227074-python-setuptools/builder-live.log)
stdout:

stderr:
gzip: /var/lib/copr/public_html/results/thrnciar/pytest:pr:120/fedora-rawhide-x86_64/08227074-python-setuptools/builder-live.log: No such file or directory


[2024-11-07 17:13:44,444][ ERROR][PID:2278338] Unable to compress file /var/lib/copr/public_html/results/thrnciar/pytest:pr:120/fedora-rawhide-x86_64/08227074-python-setuptools/builder-live.log
[2024-11-07 17:13:44,445][  INFO][PID:2278338] Compressing /var/lib/copr/public_html/results/thrnciar/pytest:pr:120/fedora-rawhide-x86_64/08227074-python-setuptools/backend.log by gzip
[2024-11-07 17:13:44,446][  INFO][PID:2278338] Running command 'gzip /var/lib/copr/public_html/results/thrnciar/pytest:pr:120/fedora-rawhide-x86_64/08227074-python-setuptools/backend.log' as PID 2382976