[2024-10-24 19:13:11,658][  INFO][PID:3786892] Marking build as starting
[2024-10-24 19:13:11,766][  INFO][PID:3786892] Trying to allocate VM
[2024-10-24 19:16:30,124][  INFO][PID:3786892] Allocated host ResallocHost, ticket_id=5548701, hostname=3.238.5.208
[2024-10-24 19:16:30,125][  INFO][PID:3786892] Allocating ssh connection to builder
[2024-10-24 19:16:30,126][  INFO][PID:3786892] Checking that builder machine is OK
[2024-10-24 19:16:30,499][  INFO][PID:3786892] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2024-10-24 19:16:30,736][  INFO][PID:3786892] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2024-10-24 19:16:30,738][  INFO][PID:3786892] Filling build.info file with builder info
[2024-10-24 19:16:30,740][  INFO][PID:3786892] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 36000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "gui1ty",
            "project_name": "NumPy-2.x",
            "project_dirname": "NumPy-2.x",
            "submitter": "gui1ty",
            "ended_on": null,
            "started_on": 1729797390.7396302,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "8173331-fedora-rawhide-x86_64",
            "build_id": 8173331,
            "package_name": "veusz",
            "package_version": "3.6.2-6",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/gui1ty/NumPy-2.x/veusz",
            "git_hash": "46f998b7792403ebffccb5832a9f21e25073b3af",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "gui1ty/NumPy-2.x--gui1ty",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/gui1ty/NumPy-2.x/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/gui1ty/NumPy-2.x",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/gui1ty/NumPy-2.x",
            "result_dir": "08173331-veusz",
            "built_packages": "",
            "id": 8173331,
            "mockchain_macros": {
                "copr_username": "gui1ty",
                "copr_projectname": "NumPy-2.x",
                "vendor": "Fedora Project COPR (gui1ty/NumPy-2.x)"
            }
        }
    ]
}
[2024-10-24 19:16:30,803][  INFO][PID:3786892] Sending fedora-messaging bus message in build.start
[2024-10-24 19:16:31,565][  INFO][PID:3786892] Sending fedora-messaging bus message in chroot.start
[2024-10-24 19:16:31,591][  INFO][PID:3786892] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8173331-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2024-10-24 19:16:32,005][  INFO][PID:3786892] Downloading the builder-live.log file, attempt 1
[2024-10-24 19:16:32,011][  INFO][PID:3786892] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.5.208 copr-rpmbuild-log
[2024-10-24 19:16:37,016][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 19:26:37,131][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 19:36:37,226][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 19:46:37,381][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 19:56:37,505][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 20:03:45,195][ ERROR][PID:3786892] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-10-24 20:03:55,196][WARNING][PID:3786892] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2024-10-24 20:04:05,197][  INFO][PID:3786892] Downloading the builder-live.log file, attempt 2
[2024-10-24 20:04:05,201][  INFO][PID:3786892] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.5.208 copr-rpmbuild-log
[2024-10-24 20:05:05,269][ ERROR][PID:3786892] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-10-24 20:05:15,284][WARNING][PID:3786892] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2024-10-24 20:05:25,285][  INFO][PID:3786892] Downloading the builder-live.log file, attempt 3
[2024-10-24 20:05:25,287][  INFO][PID:3786892] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.5.208 copr-rpmbuild-log
[2024-10-24 20:06:25,356][ ERROR][PID:3786892] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-10-24 20:06:35,363][WARNING][PID:3786892] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2024-10-24 20:06:37,642][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 20:06:45,376][  INFO][PID:3786892] Downloading the builder-live.log file, attempt 4
[2024-10-24 20:06:45,378][  INFO][PID:3786892] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.5.208 copr-rpmbuild-log
[2024-10-24 20:07:45,469][ ERROR][PID:3786892] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-10-24 20:07:55,472][ ERROR][PID:3786892] Giving up for unstable SSH, failures: 1729800235.1964245, 1729800315.2841165, 1729800395.3637688, 1729800475.4720316
[2024-10-24 20:07:55,473][ ERROR][PID:3786892] Re-try request for task on 'ResallocHost, ticket_id=5548701, hostname=3.238.5.208': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2024-10-24 20:07:55,473][  INFO][PID:3786892] Releasing VM back to pool
[2024-10-24 20:07:55,483][  INFO][PID:3786892] Retry #1 (on other host)
[2024-10-24 20:07:55,484][  INFO][PID:3786892] Trying to allocate VM
[2024-10-24 20:07:58,579][  INFO][PID:3786892] Allocated host ResallocHost, ticket_id=5549623, hostname=2620:52:3:1:dead:beef:cafe:c114
[2024-10-24 20:07:58,580][  INFO][PID:3786892] Allocating ssh connection to builder
[2024-10-24 20:07:58,581][  INFO][PID:3786892] Checking that builder machine is OK
[2024-10-24 20:07:59,156][  INFO][PID:3786892] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2024-10-24 20:07:59,513][  INFO][PID:3786892] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2024-10-24 20:07:59,514][  INFO][PID:3786892] Filling build.info file with builder info
[2024-10-24 20:07:59,518][  INFO][PID:3786892] Not re-notifying FE and msg buses for the new host.
[2024-10-24 20:07:59,518][  INFO][PID:3786892] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8173331-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2024-10-24 20:08:00,101][  INFO][PID:3786892] Downloading the builder-live.log file, attempt 1
[2024-10-24 20:08:00,103][  INFO][PID:3786892] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c114 copr-rpmbuild-log
[2024-10-24 20:16:40,247][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 20:26:40,397][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 20:36:40,494][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 20:46:40,571][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 20:56:40,648][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 21:06:40,782][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 21:16:40,985][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 21:26:41,070][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 21:36:41,148][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 21:46:41,228][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 21:56:41,306][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 22:06:41,408][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 22:16:41,501][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 22:26:41,581][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 22:36:41,657][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 22:46:41,731][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 22:56:41,812][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 23:06:41,900][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 23:16:42,063][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 23:26:42,143][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 23:36:42,217][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 23:46:42,289][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-24 23:56:42,360][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 00:06:42,431][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 00:16:42,510][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 00:26:42,601][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 00:36:42,697][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 00:46:42,882][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 00:56:42,966][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 01:06:43,045][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 01:16:43,187][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 01:26:43,310][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 01:36:43,408][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 01:46:43,523][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 01:56:43,650][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 02:06:43,730][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 02:16:46,286][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 02:26:46,390][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 02:36:46,463][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 02:46:46,535][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 02:56:46,608][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 03:06:46,722][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 03:16:46,917][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 03:26:47,040][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 03:36:47,164][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 03:46:47,256][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 03:56:47,353][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 04:06:47,512][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 04:16:47,836][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 04:26:47,954][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 04:36:48,040][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 04:46:48,129][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 04:56:48,223][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 05:06:48,310][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 05:16:48,431][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 05:26:48,516][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 05:36:48,599][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 05:46:48,689][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 05:56:48,774][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 06:06:48,863][  INFO][PID:3786892] Periodic builder liveness probe: alive
[2024-10-25 06:08:03,574][  INFO][PID:3786892] Downloading results from builder
[2024-10-25 06:08:03,601][  INFO][PID:3786892] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c114]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/gui1ty/NumPy-2.x/fedora-rawhide-x86_64/08173331-veusz started
[2024-10-25 06:08:03,629][  INFO][PID:3786892] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c114]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/gui1ty/NumPy-2.x/fedora-rawhide-x86_64/08173331-veusz/ &> /var/lib/copr/public_html/results/gui1ty/NumPy-2.x/fedora-rawhide-x86_64/08173331-veusz/build-08173331.rsync.log
[2024-10-25 06:08:04,342][  INFO][PID:3786892] rsyncing finished.
[2024-10-25 06:08:04,344][  INFO][PID:3786892] Releasing VM back to pool
[2024-10-25 06:08:04,363][  INFO][PID:3786892] Searching for 'success' file in resultdir
[2024-10-25 06:08:04,371][ ERROR][PID:3786892] Build failed: Backend process error: No success file => build failure
[2024-10-25 06:08:04,485][  INFO][PID:3786892] Finished build: id=8173331 failed=True timeout=36000 destdir=/var/lib/copr/public_html/results/gui1ty/NumPy-2.x chroot=fedora-rawhide-x86_64 
[2024-10-25 06:08:04,590][ ERROR][PID:3786892] Backend process error: No success file => build failure
[2024-10-25 06:08:04,593][  INFO][PID:3786892] Worker failed build, took 36005.07381558418
[2024-10-25 06:08:04,607][  INFO][PID:3786892] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 36000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "gui1ty",
            "project_name": "NumPy-2.x",
            "project_dirname": "NumPy-2.x",
            "submitter": "gui1ty",
            "ended_on": 1729836484.5918956,
            "started_on": 1729800479.51808,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "8173331-fedora-rawhide-x86_64",
            "build_id": 8173331,
            "package_name": "veusz",
            "package_version": "3.6.2-6",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/gui1ty/NumPy-2.x/veusz",
            "git_hash": "46f998b7792403ebffccb5832a9f21e25073b3af",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "gui1ty/NumPy-2.x--gui1ty",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/gui1ty/NumPy-2.x/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/gui1ty/NumPy-2.x",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/gui1ty/NumPy-2.x",
            "result_dir": "08173331-veusz",
            "built_packages": "",
            "id": 8173331,
            "mockchain_macros": {
                "copr_username": "gui1ty",
                "copr_projectname": "NumPy-2.x",
                "vendor": "Fedora Project COPR (gui1ty/NumPy-2.x)"
            }
        }
    ]
}
[2024-10-25 06:08:05,710][  INFO][PID:3786892] Sending fedora-messaging bus message in build.end
[2024-10-25 06:08:06,067][  INFO][PID:3786892] Compressing /var/lib/copr/public_html/results/gui1ty/NumPy-2.x/fedora-rawhide-x86_64/08173331-veusz/builder-live.log by gzip
[2024-10-25 06:08:06,092][  INFO][PID:3786892] Running command 'gzip /var/lib/copr/public_html/results/gui1ty/NumPy-2.x/fedora-rawhide-x86_64/08173331-veusz/builder-live.log' as PID 1937081
[2024-10-25 06:08:06,129][  INFO][PID:3786892] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/gui1ty/NumPy-2.x/fedora-rawhide-x86_64/08173331-veusz/builder-live.log)
[2024-10-25 06:08:06,140][  INFO][PID:3786892] Compressing /var/lib/copr/public_html/results/gui1ty/NumPy-2.x/fedora-rawhide-x86_64/08173331-veusz/backend.log by gzip
[2024-10-25 06:08:06,142][  INFO][PID:3786892] Running command 'gzip /var/lib/copr/public_html/results/gui1ty/NumPy-2.x/fedora-rawhide-x86_64/08173331-veusz/backend.log' as PID 1937082