[2024-11-16 02:17:12,621][  INFO][PID:2935521] Marking build as starting
[2024-11-16 02:17:12,703][  INFO][PID:2935521] Trying to allocate VM
[2024-11-16 02:17:15,785][  INFO][PID:2935521] Allocated host ResallocHost, ticket_id=5909943, hostname=2620:52:3:1:dead:beef:cafe:c29f
[2024-11-16 02:17:15,787][  INFO][PID:2935521] Allocating ssh connection to builder
[2024-11-16 02:17:15,788][  INFO][PID:2935521] Checking that builder machine is OK
[2024-11-16 02:17:16,704][  INFO][PID:2935521] Running remote command: copr-builder-ready fedora-39-ppc64le
[2024-11-16 02:17:17,308][  INFO][PID:2935521] Red Hat subscription not needed for fedora-39-ppc64le
Builder is ready to be used

[2024-11-16 02:17:17,309][  INFO][PID:2935521] Filling build.info file with builder info
[2024-11-16 02:17:17,319][  INFO][PID:2935521] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "@fedora-llvm-team",
            "project_name": "llvm-snapshots-big-merge-20241116",
            "project_dirname": "llvm-snapshots-big-merge-20241116",
            "submitter": "tstellar",
            "ended_on": null,
            "started_on": 1731723437.3189917,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-39-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "8270412-fedora-39-ppc64le",
            "build_id": 8270412,
            "package_name": "llvm",
            "package_version": "19.1.3-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@fedora-llvm-team/llvm-snapshots-big-merge-20241116/llvm",
            "git_hash": "15f8d640dd13a31ca4181399fbcf877d4ce9ac74",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@fedora-llvm-team/llvm-snapshots-big-merge-20241116--tstellar",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20241116/fedora-39-ppc64le/",
                    "id": "copr_base",
                    "module_hotfixes": true,
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [
                "snapshot_build"
            ],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20241116",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20241116",
            "result_dir": "08270412-llvm",
            "built_packages": "",
            "id": 8270412,
            "mockchain_macros": {
                "copr_username": "@fedora-llvm-team",
                "copr_projectname": "llvm-snapshots-big-merge-20241116",
                "vendor": "Fedora Project COPR (@fedora-llvm-team/llvm-snapshots-big-merge-20241116)"
            }
        }
    ]
}
[2024-11-16 02:17:17,445][  INFO][PID:2935521] Sending fedora-messaging bus message in build.start
[2024-11-16 02:17:18,579][  INFO][PID:2935521] Sending fedora-messaging bus message in chroot.start
[2024-11-16 02:17:18,605][  INFO][PID:2935521] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8270412-fedora-39-ppc64le --chroot fedora-39-ppc64le --detached
[2024-11-16 02:17:19,409][  INFO][PID:2935521] Downloading the builder-live.log file, attempt 1
[2024-11-16 02:17:19,415][  INFO][PID:2935521] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29f copr-rpmbuild-log
[2024-11-16 02:17:24,448][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 02:27:24,554][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 02:37:24,629][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 02:47:24,703][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 02:57:24,777][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 03:07:24,870][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 03:17:24,953][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 03:27:25,042][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 03:37:25,123][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 03:47:25,259][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 03:57:25,445][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 04:07:25,568][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 04:17:25,677][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 04:27:25,767][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 04:37:25,859][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 04:47:25,949][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 04:57:26,036][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 05:07:26,136][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 05:17:26,231][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 05:21:37,949][ ERROR][PID:2935521] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-11-16 05:21:47,950][WARNING][PID:2935521] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2024-11-16 05:21:57,951][  INFO][PID:2935521] Downloading the builder-live.log file, attempt 2
[2024-11-16 05:21:57,966][  INFO][PID:2935521] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29f copr-rpmbuild-log
[2024-11-16 05:22:58,035][ ERROR][PID:2935521] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-11-16 05:23:08,036][WARNING][PID:2935521] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2024-11-16 05:23:18,037][  INFO][PID:2935521] Downloading the builder-live.log file, attempt 3
[2024-11-16 05:23:18,039][  INFO][PID:2935521] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29f copr-rpmbuild-log
[2024-11-16 05:24:18,108][ ERROR][PID:2935521] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-11-16 05:24:28,131][WARNING][PID:2935521] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2024-11-16 05:24:38,131][  INFO][PID:2935521] Downloading the builder-live.log file, attempt 4
[2024-11-16 05:24:38,133][  INFO][PID:2935521] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29f copr-rpmbuild-log
[2024-11-16 05:25:38,202][ ERROR][PID:2935521] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-11-16 05:25:48,203][ ERROR][PID:2935521] Giving up for unstable SSH, failures: 1731734507.9500492, 1731734588.0363183, 1731734668.1311336, 1731734748.2037299
[2024-11-16 05:25:48,204][ ERROR][PID:2935521] Re-try request for task on 'ResallocHost, ticket_id=5909943, hostname=2620:52:3:1:dead:beef:cafe:c29f': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2024-11-16 05:25:48,205][  INFO][PID:2935521] Releasing VM back to pool
[2024-11-16 05:25:48,215][  INFO][PID:2935521] Retry #1 (on other host)
[2024-11-16 05:25:48,216][  INFO][PID:2935521] Trying to allocate VM
[2024-11-16 05:25:51,319][  INFO][PID:2935521] Allocated host ResallocHost, ticket_id=5910711, hostname=2620:52:3:1:dead:beef:cafe:c20c
[2024-11-16 05:25:51,320][  INFO][PID:2935521] Allocating ssh connection to builder
[2024-11-16 05:25:51,320][  INFO][PID:2935521] Checking that builder machine is OK
[2024-11-16 05:25:52,159][  INFO][PID:2935521] Running remote command: copr-builder-ready fedora-39-ppc64le
[2024-11-16 05:25:52,578][  INFO][PID:2935521] Red Hat subscription not needed for fedora-39-ppc64le
Builder is ready to be used

[2024-11-16 05:25:52,578][  INFO][PID:2935521] Filling build.info file with builder info
[2024-11-16 05:25:52,603][  INFO][PID:2935521] Not re-notifying FE and msg buses for the new host.
[2024-11-16 05:25:52,605][  INFO][PID:2935521] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8270412-fedora-39-ppc64le --chroot fedora-39-ppc64le --detached
[2024-11-16 05:25:53,342][  INFO][PID:2935521] Downloading the builder-live.log file, attempt 1
[2024-11-16 05:25:53,345][  INFO][PID:2935521] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c20c copr-rpmbuild-log
[2024-11-16 05:27:28,363][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 05:37:28,480][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 05:47:28,570][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 05:57:28,662][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 06:07:28,752][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 06:17:28,889][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 06:27:29,040][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 06:37:29,129][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 06:47:29,215][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 06:57:29,318][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 07:07:29,406][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 07:17:29,495][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 07:27:29,627][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 07:37:29,779][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 07:47:29,910][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 07:57:29,995][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 08:07:30,075][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 08:17:30,190][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 08:27:30,271][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 08:37:30,349][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 08:47:30,425][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 08:57:30,530][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 09:07:30,623][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 09:17:30,719][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 09:27:30,809][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 09:37:30,895][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 09:47:30,993][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 09:57:31,083][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 10:07:31,179][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 10:17:31,259][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 10:27:31,335][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 10:37:31,414][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 10:47:31,511][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 10:57:31,589][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 11:07:31,681][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 11:17:31,908][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 11:27:31,992][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 11:37:32,123][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 11:47:32,276][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 11:57:32,358][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 12:07:32,431][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 12:17:32,518][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 12:27:32,682][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 12:37:32,772][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 12:47:32,852][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 12:57:32,932][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 13:07:33,036][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 13:17:33,135][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 13:27:33,219][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 13:37:33,295][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 13:47:33,370][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 13:57:33,449][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 14:07:33,534][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 14:17:33,648][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 14:27:33,766][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 14:37:33,844][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 14:47:33,938][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 14:57:34,042][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 15:07:34,158][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 15:17:34,353][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 15:27:34,467][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 15:37:34,569][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 15:47:34,648][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 15:57:34,721][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 16:07:34,796][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 16:17:34,875][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 16:27:34,952][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 16:37:35,027][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 16:47:35,103][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 16:57:35,187][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 17:07:35,308][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 17:17:35,405][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 17:27:35,490][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 17:37:35,579][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 17:47:35,654][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 17:57:35,731][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 18:07:35,916][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 18:17:36,019][  INFO][PID:2935521] Periodic builder liveness probe: alive
[2024-11-16 18:27:36,098][  INFO][PID:2935521] Periodic builder liveness probe: alive