[2026-02-12 04:45:31,675][  INFO][PID:451485] Marking build as starting
[2026-02-12 04:45:31,723][  INFO][PID:451485] Checking that /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2026-02-12 04:45:31,724][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:45:31,724][  INFO][PID:451485] VM allocation process starts
[2026-02-12 04:45:31,734][  INFO][PID:451485] Trying to allocate VM: ResallocHost, ticket_id=1286704, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-12 04:45:34,784][  INFO][PID:451485] Allocated host ResallocHost, ticket_id=1286704, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_02200440_20260212_034205, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-12 04:45:34,785][  INFO][PID:451485] Allocating ssh connection to builder
[2026-02-12 04:45:34,785][  INFO][PID:451485] Checking that builder machine is OK
[2026-02-12 04:45:35,783][  INFO][PID:451485] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2026-02-12 04:45:36,544][  INFO][PID:451485] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2026-02-12 04:45:36,545][  INFO][PID:451485] Filling build.info file with builder info
[2026-02-12 04:45:36,546][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:45:36,546][  INFO][PID:451485] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 90000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@fedora-llvm-team",
            "project_name": "clang-fedora-centos-testing",
            "project_dirname": "clang-fedora-centos-testing",
            "submitter": "tbaeder",
            "ended_on": null,
            "started_on": 1770871536.546651,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "10116205-fedora-rawhide-ppc64le",
            "build_id": 10116205,
            "package_name": "cosmic-osd",
            "package_version": "1.0.4-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@fedora-llvm-team/clang-fedora-centos-testing/cosmic-osd",
            "git_hash": "f9fa6b7a00a9cc54695522790175b024c23fcc74",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@fedora-llvm-team/clang-fedora-centos-testing--tbaeder",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": 1000
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-compat-packages/fedora-rawhide-ppc64le/",
                    "id": "copr_fedora_llvm_team_llvm_compat_packages",
                    "name": "Additional repo copr_fedora_llvm_team_llvm_compat_packages"
                },
                {
                    "baseurl": "https://kojihub.stream.centos.org/kojifiles/repos/c9s-build/latest/x86_64/",
                    "id": "https_kojihub_stream_centos_org_kojifiles_repos_c9s_build_latest_x86_64",
                    "name": "Additional repo https_kojihub_stream_centos_org_kojifiles_repos_c9s_build_latest_x86_64"
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20260211/fedora-rawhide-ppc64le/",
                    "id": "copr_fedora_llvm_team_llvm_snapshots_big_merge_20260211",
                    "module_hotfixes": true,
                    "name": "Additional repo copr_fedora_llvm_team_llvm_snapshots_big_merge_20260211"
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": 1000,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/clang-fedora-centos-testing",
            "result_dir": "10116205-cosmic-osd",
            "built_packages": "",
            "id": 10116205,
            "mockchain_macros": {
                "copr_username": "@fedora-llvm-team",
                "copr_projectname": "clang-fedora-centos-testing",
                "vendor": "Fedora Project COPR (@fedora-llvm-team/clang-fedora-centos-testing)"
            }
        }
    ]
}
[2026-02-12 04:45:36,588][  INFO][PID:451485] Sending fedora-messaging bus message in build.start
[2026-02-12 04:45:37,096][  INFO][PID:451485] Sending fedora-messaging bus message in chroot.start
[2026-02-12 04:45:37,123][  INFO][PID:451485] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10116205-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2026-02-12 04:45:37,957][  INFO][PID:451485] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c191' (ED25519) to the list of known hosts.


[2026-02-12 04:45:37,957][  INFO][PID:451485] Downloading the builder-live.log file, attempt 1
[2026-02-12 04:45:37,959][  INFO][PID:451485] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c191 copr-rpmbuild-log
[2026-02-12 04:45:42,964][  INFO][PID:451485] Periodic builder liveness probe: alive
[2026-02-12 04:45:42,965][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:45:47,966][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:45:52,967][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:45:57,968][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:46:02,969][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:46:07,971][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:46:12,972][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:46:17,973][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:46:22,974][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:46:27,975][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:46:32,976][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:46:37,977][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:46:42,986][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:46:47,987][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:46:52,987][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:46:57,988][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:47:02,990][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:47:07,991][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:47:12,992][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:47:17,993][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:47:22,994][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:47:27,995][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:47:32,996][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:47:37,997][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:47:42,998][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:47:47,999][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:47:53,000][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:47:58,000][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:48:03,001][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:48:08,002][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:48:13,003][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:48:18,004][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:48:23,005][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:48:28,006][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:48:33,007][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:48:38,008][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:48:43,009][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:48:48,010][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:48:53,011][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:48:58,012][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:49:03,012][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:49:08,013][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:49:13,014][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:49:18,015][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:49:23,016][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:49:28,017][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:49:33,018][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:49:38,019][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:49:43,020][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:49:48,021][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:49:53,036][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:49:58,164][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:50:03,165][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:50:08,166][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:50:13,166][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:50:18,167][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:50:23,168][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:50:28,169][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:50:33,170][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:50:38,171][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:50:43,172][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:50:48,173][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:50:53,174][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:50:58,175][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:51:03,176][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:51:08,176][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:51:13,177][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:51:18,178][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:51:23,208][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:51:28,209][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:51:33,210][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:51:38,211][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:51:43,211][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:51:48,212][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:51:53,214][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:51:58,214][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:52:03,215][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:52:08,216][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:52:13,217][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:52:18,218][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:52:23,219][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:52:28,220][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:52:33,221][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:52:38,222][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:52:43,223][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:52:48,224][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:52:53,225][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:52:58,226][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:53:03,227][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:53:08,228][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:53:13,229][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:53:18,230][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:53:23,230][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:53:28,231][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:53:33,232][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:53:38,233][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:53:43,234][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:53:48,235][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:53:53,236][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:53:58,237][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:54:03,238][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:54:08,239][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:54:13,240][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:54:18,241][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:54:23,242][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:54:28,242][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:54:33,243][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:54:38,244][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:54:43,245][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:54:48,246][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:54:53,247][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:54:58,248][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:55:03,249][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:55:08,250][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:55:13,251][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:55:18,251][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:55:23,252][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:55:28,253][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:55:33,254][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:55:38,255][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:55:43,261][  INFO][PID:451485] Periodic builder liveness probe: alive
[2026-02-12 04:55:43,261][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:55:48,262][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:55:53,264][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:55:58,267][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:56:03,268][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:56:08,269][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:56:13,270][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:56:13,381][ ERROR][PID:451485] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-12 04:56:18,270][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:56:23,271][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:56:23,381][WARNING][PID:451485] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-12 04:56:28,273][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:56:33,285][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:56:33,382][  INFO][PID:451485] Downloading the builder-live.log file, attempt 2
[2026-02-12 04:56:33,387][  INFO][PID:451485] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c191 copr-rpmbuild-log
[2026-02-12 04:56:38,318][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:56:43,320][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:56:48,322][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:56:53,324][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:56:58,325][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:57:03,327][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:57:08,328][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:57:13,330][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:57:19,027][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:57:24,028][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:57:29,029][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:57:33,535][ ERROR][PID:451485] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-12 04:57:34,031][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:57:39,032][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:57:43,558][WARNING][PID:451485] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-12 04:57:44,033][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:57:49,034][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:57:53,558][  INFO][PID:451485] Downloading the builder-live.log file, attempt 3
[2026-02-12 04:57:53,560][  INFO][PID:451485] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c191 copr-rpmbuild-log
[2026-02-12 04:57:54,036][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:57:59,037][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:58:04,039][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:58:09,560][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:58:14,561][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:58:19,561][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:58:24,562][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:58:29,563][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:58:34,564][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:58:39,565][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:58:44,566][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:58:49,567][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:58:53,617][ ERROR][PID:451485] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-12 04:58:54,568][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:58:59,569][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:59:03,632][WARNING][PID:451485] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-12 04:59:04,570][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:59:09,570][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:59:13,633][  INFO][PID:451485] Downloading the builder-live.log file, attempt 4
[2026-02-12 04:59:13,635][  INFO][PID:451485] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c191 copr-rpmbuild-log
[2026-02-12 04:59:14,571][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:59:19,572][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:59:24,573][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:59:29,574][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:59:34,575][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:59:39,576][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:59:44,577][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:59:49,578][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:59:54,751][  INFO][PID:451485] Checking for cancel request
[2026-02-12 04:59:59,752][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:00:04,754][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:00:09,755][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:00:13,694][ ERROR][PID:451485] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-12 05:00:14,985][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:00:19,987][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:00:23,712][ ERROR][PID:451485] Giving up for unstable SSH, failures: 1770872183.3817537, 1770872263.5582192, 1770872343.6327043, 1770872423.7118893
[2026-02-12 05:00:23,712][ ERROR][PID:451485] Re-try request for task on 'ResallocHost, ticket_id=1286704, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_02200440_20260212_034205, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-02-12 05:00:23,713][  INFO][PID:451485] Releasing VM back to pool
[2026-02-12 05:00:23,722][  INFO][PID:451485] Retry #1 (on other host)
[2026-02-12 05:00:23,722][  INFO][PID:451485] Checking that /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2026-02-12 05:00:23,723][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:00:23,723][  INFO][PID:451485] VM allocation process starts
[2026-02-12 05:00:23,777][  INFO][PID:451485] Trying to allocate VM: ResallocHost, ticket_id=1286737, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-12 05:00:26,813][  INFO][PID:451485] Allocated host ResallocHost, ticket_id=1286737, hostname=2620:52:6:1161:dead:beef:cafe:c110, name=vmhost_p09_02_prod_02200493_20260212_035256, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-12 05:00:26,814][  INFO][PID:451485] Allocating ssh connection to builder
[2026-02-12 05:00:26,814][  INFO][PID:451485] Checking that builder machine is OK
[2026-02-12 05:01:26,878][ ERROR][PID:451485] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c110 port 22: Connection timed out

[2026-02-12 05:01:36,879][ ERROR][PID:451485] Re-try request for task on 'ResallocHost, ticket_id=1286737, hostname=2620:52:6:1161:dead:beef:cafe:c110, name=vmhost_p09_02_prod_02200493_20260212_035256, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2026-02-12 05:01:36,889][  INFO][PID:451485] Releasing VM back to pool
[2026-02-12 05:01:36,897][  INFO][PID:451485] Retry #2 (on other host)
[2026-02-12 05:01:36,898][  INFO][PID:451485] Checking that /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2026-02-12 05:01:36,898][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:01:36,899][  INFO][PID:451485] VM allocation process starts
[2026-02-12 05:01:36,932][  INFO][PID:451485] Trying to allocate VM: ResallocHost, ticket_id=1286740, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-12 05:01:39,983][  INFO][PID:451485] Allocated host ResallocHost, ticket_id=1286740, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_02200440_20260212_034205, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-12 05:01:39,984][  INFO][PID:451485] Allocating ssh connection to builder
[2026-02-12 05:01:39,984][  INFO][PID:451485] Checking that builder machine is OK
[2026-02-12 05:01:40,536][  INFO][PID:451485] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2026-02-12 05:01:41,120][  INFO][PID:451485] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2026-02-12 05:01:41,120][  INFO][PID:451485] Filling build.info file with builder info
[2026-02-12 05:01:41,149][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:01:41,149][  INFO][PID:451485] Not re-notifying FE and msg buses for the new host.
[2026-02-12 05:01:41,150][  INFO][PID:451485] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10116205-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2026-02-12 05:01:41,976][  INFO][PID:451485] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c191' (ED25519) to the list of known hosts.


[2026-02-12 05:01:41,977][  INFO][PID:451485] Downloading the builder-live.log file, attempt 1
[2026-02-12 05:01:41,978][  INFO][PID:451485] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c191 copr-rpmbuild-log
[2026-02-12 05:01:46,977][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:01:51,978][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:01:56,979][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:02:01,980][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:02:06,981][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:02:11,982][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:02:16,983][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:02:21,984][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:02:26,985][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:02:31,986][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:02:36,987][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:02:41,988][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:02:46,989][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:02:51,990][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:02:56,991][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:03:01,992][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:03:06,993][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:03:11,994][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:03:16,995][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:03:21,996][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:03:27,004][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:03:32,005][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:03:37,006][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:03:42,007][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:03:47,008][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:03:52,009][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:03:57,010][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:04:02,010][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:04:07,012][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:04:12,013][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:04:17,014][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:04:22,015][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:04:27,016][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:04:32,017][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:04:37,018][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:04:42,019][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:04:47,020][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:04:52,021][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:04:57,022][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:05:02,023][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:05:07,024][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:05:12,024][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:05:17,026][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:05:22,027][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:05:27,027][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:05:32,028][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:05:37,029][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:05:42,030][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:05:47,036][  INFO][PID:451485] Periodic builder liveness probe: alive
[2026-02-12 05:05:47,036][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:05:52,037][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:05:57,038][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:06:02,039][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:06:07,040][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:06:12,041][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:06:17,043][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:06:22,043][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:06:27,044][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:06:32,045][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:06:37,047][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:06:42,048][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:06:47,048][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:06:52,049][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:06:57,050][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:07:02,051][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:07:07,052][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:07:12,054][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:07:17,055][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:07:22,056][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:07:27,057][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:07:32,067][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:07:37,068][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:07:42,070][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:07:47,071][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:07:52,075][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:07:57,120][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:08:02,124][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:08:07,126][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:08:12,128][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:08:17,132][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:08:22,134][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:08:27,136][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:08:32,137][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:08:37,140][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:08:42,142][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:08:47,144][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:08:52,147][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:08:57,149][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:09:02,150][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:09:07,182][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:09:12,183][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:09:17,184][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:09:22,186][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:09:27,187][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:09:32,188][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:09:37,193][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:09:42,194][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:09:47,195][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:09:52,196][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:09:57,197][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:10:02,198][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:10:07,199][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:10:12,200][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:10:17,201][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:10:22,251][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:10:27,252][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:10:32,260][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:10:37,261][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:10:42,262][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:10:47,263][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:10:52,924][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:10:57,926][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:11:02,928][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:11:07,929][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:11:12,931][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:11:17,932][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:11:22,933][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:11:27,935][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:11:32,936][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:11:37,938][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:11:42,943][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:11:47,945][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:11:49,870][  INFO][PID:451485] Downloading results from builder
[2026-02-12 05:11:49,872][  INFO][PID:451485] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/10116205-cosmic-osd started
[2026-02-12 05:11:49,873][  INFO][PID:451485] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/10116205-cosmic-osd/ &> /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/10116205-cosmic-osd/build-10116205.rsync.log
[2026-02-12 05:12:49,975][  INFO][PID:451485] rsyncing finished.
[2026-02-12 05:12:49,975][ ERROR][PID:451485] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-02-12 05:12:49,976][ ERROR][PID:451485] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-02-12 05:12:59,976][  INFO][PID:451485] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/10116205-cosmic-osd started
[2026-02-12 05:12:59,978][  INFO][PID:451485] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/10116205-cosmic-osd/ &> /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/10116205-cosmic-osd/build-10116205.rsync.log
[2026-02-12 05:14:00,081][  INFO][PID:451485] rsyncing finished.
[2026-02-12 05:14:00,083][ ERROR][PID:451485] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-02-12 05:14:00,084][ ERROR][PID:451485] SSH connection lost on #2 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-02-12 05:14:10,087][  INFO][PID:451485] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/10116205-cosmic-osd started
[2026-02-12 05:14:10,088][  INFO][PID:451485] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/10116205-cosmic-osd/ &> /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/10116205-cosmic-osd/build-10116205.rsync.log
[2026-02-12 05:15:10,184][  INFO][PID:451485] rsyncing finished.
[2026-02-12 05:15:10,203][ ERROR][PID:451485] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-02-12 05:15:10,204][ ERROR][PID:451485] SSH connection lost on #3 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-02-12 05:15:20,205][ ERROR][PID:451485] Re-try request for task on 'ResallocHost, ticket_id=1286740, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_02200440_20260212_034205, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 3 SSH attempts
[2026-02-12 05:15:20,206][  INFO][PID:451485] Releasing VM back to pool
[2026-02-12 05:15:20,216][  INFO][PID:451485] Retry #3 (on other host)
[2026-02-12 05:15:20,217][  INFO][PID:451485] Checking that /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2026-02-12 05:15:20,217][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:15:20,218][  INFO][PID:451485] VM allocation process starts
[2026-02-12 05:15:20,260][  INFO][PID:451485] Trying to allocate VM: ResallocHost, ticket_id=1286841, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-12 05:15:23,324][  INFO][PID:451485] Allocated host ResallocHost, ticket_id=1286841, hostname=2620:52:6:1161:dead:beef:cafe:c188, name=vmhost_p09_04_prod_02200673_20260212_041441, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-02-12 05:15:23,325][  INFO][PID:451485] Allocating ssh connection to builder
[2026-02-12 05:15:23,325][  INFO][PID:451485] Checking that builder machine is OK
[2026-02-12 05:15:24,309][  INFO][PID:451485] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2026-02-12 05:15:25,033][  INFO][PID:451485] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2026-02-12 05:15:25,033][  INFO][PID:451485] Filling build.info file with builder info
[2026-02-12 05:15:25,127][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:15:25,128][  INFO][PID:451485] Not re-notifying FE and msg buses for the new host.
[2026-02-12 05:15:25,129][  INFO][PID:451485] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10116205-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2026-02-12 05:15:26,174][  INFO][PID:451485] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c188' (ED25519) to the list of known hosts.


[2026-02-12 05:15:26,175][  INFO][PID:451485] Downloading the builder-live.log file, attempt 1
[2026-02-12 05:15:26,191][  INFO][PID:451485] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c188 copr-rpmbuild-log
[2026-02-12 05:15:31,176][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:15:36,177][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:15:41,178][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:15:46,180][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:15:51,187][  INFO][PID:451485] Periodic builder liveness probe: alive
[2026-02-12 05:15:51,187][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:15:56,189][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:16:01,190][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:16:06,192][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:16:11,193][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:16:16,194][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:16:21,195][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:16:26,197][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:16:31,198][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:16:36,199][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:16:41,201][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:16:46,300][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:16:51,325][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:16:56,326][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:17:01,327][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:17:06,328][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:17:11,336][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:17:16,338][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:17:21,339][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:17:26,340][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:17:31,341][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:17:36,342][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:17:41,343][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:17:46,344][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:17:51,345][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:17:56,346][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:18:01,347][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:18:06,348][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:18:11,349][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:18:16,350][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:18:21,373][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:18:26,374][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:18:31,375][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:18:36,376][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:18:41,377][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:18:46,378][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:18:51,379][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:18:56,398][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:19:01,398][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:19:06,400][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:19:11,428][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:19:16,429][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:19:21,448][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:19:26,451][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:19:31,485][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:19:36,500][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:19:41,517][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:19:46,549][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:19:51,550][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:19:56,551][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:20:01,576][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:20:06,610][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:20:11,612][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:20:16,613][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:20:21,614][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:20:26,615][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:20:31,616][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:20:36,618][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:20:41,619][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:20:46,620][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:20:51,621][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:20:56,622][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:21:01,623][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:21:06,624][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:21:11,639][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:21:16,639][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:21:21,641][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:21:27,284][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:21:32,286][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:21:37,308][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:21:42,309][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:21:47,310][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:21:52,311][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:21:57,312][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:22:02,313][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:22:07,320][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:22:12,322][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:22:17,324][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:22:22,325][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:22:27,511][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:22:32,512][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:22:37,513][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:22:42,514][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:22:47,526][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:22:52,613][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:22:57,614][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:23:02,616][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:23:07,618][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:23:12,658][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:23:17,674][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:23:22,681][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:23:27,828][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:23:32,830][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:23:37,831][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:23:42,833][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:23:47,860][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:23:52,861][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:23:57,862][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:24:02,863][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:24:07,923][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:24:12,930][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:24:17,961][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:24:22,963][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:24:28,084][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:24:33,086][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:24:38,089][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:24:43,332][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:24:48,357][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:24:53,359][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:24:58,373][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:25:03,374][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:25:08,398][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:25:13,413][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:25:18,441][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:25:23,442][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:25:28,443][  INFO][PID:451485] Checking for cancel request
[2026-02-12 05:25:33,141][  INFO][PID:451485] Downloading results from builder
[2026-02-12 05:25:33,237][  INFO][PID:451485] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c188]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/10116205-cosmic-osd started
[2026-02-12 05:25:33,238][  INFO][PID:451485] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c188]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/10116205-cosmic-osd/ &> /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/10116205-cosmic-osd/build-10116205.rsync.log
[2026-02-12 05:25:35,272][  INFO][PID:451485] rsyncing finished.
[2026-02-12 05:25:35,273][  INFO][PID:451485] Releasing VM back to pool
[2026-02-12 05:25:35,282][  INFO][PID:451485] Searching for 'success' file in resultdir
[2026-02-12 05:25:35,282][ ERROR][PID:451485] Build failed: Backend process error: No success file => build failure
[2026-02-12 05:25:35,283][  INFO][PID:451485] Finished build: id=10116205 failed=True timeout=90000 destdir=/var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing chroot=fedora-rawhide-ppc64le 
[2026-02-12 05:25:35,285][ ERROR][PID:451485] Backend process error: No success file => build failure
[2026-02-12 05:25:35,285][  INFO][PID:451485] Worker failed build, took 610.1571660041809
[2026-02-12 05:25:35,286][  INFO][PID:451485] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 90000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@fedora-llvm-team",
            "project_name": "clang-fedora-centos-testing",
            "project_dirname": "clang-fedora-centos-testing",
            "submitter": "tbaeder",
            "ended_on": 1770873935.285885,
            "started_on": 1770873325.128719,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "10116205-fedora-rawhide-ppc64le",
            "build_id": 10116205,
            "package_name": "cosmic-osd",
            "package_version": "1.0.4-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@fedora-llvm-team/clang-fedora-centos-testing/cosmic-osd",
            "git_hash": "f9fa6b7a00a9cc54695522790175b024c23fcc74",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@fedora-llvm-team/clang-fedora-centos-testing--tbaeder",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": 1000
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-compat-packages/fedora-rawhide-ppc64le/",
                    "id": "copr_fedora_llvm_team_llvm_compat_packages",
                    "name": "Additional repo copr_fedora_llvm_team_llvm_compat_packages"
                },
                {
                    "baseurl": "https://kojihub.stream.centos.org/kojifiles/repos/c9s-build/latest/x86_64/",
                    "id": "https_kojihub_stream_centos_org_kojifiles_repos_c9s_build_latest_x86_64",
                    "name": "Additional repo https_kojihub_stream_centos_org_kojifiles_repos_c9s_build_latest_x86_64"
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20260211/fedora-rawhide-ppc64le/",
                    "id": "copr_fedora_llvm_team_llvm_snapshots_big_merge_20260211",
                    "module_hotfixes": true,
                    "name": "Additional repo copr_fedora_llvm_team_llvm_snapshots_big_merge_20260211"
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": 1000,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/clang-fedora-centos-testing",
            "result_dir": "10116205-cosmic-osd",
            "built_packages": "",
            "id": 10116205,
            "mockchain_macros": {
                "copr_username": "@fedora-llvm-team",
                "copr_projectname": "clang-fedora-centos-testing",
                "vendor": "Fedora Project COPR (@fedora-llvm-team/clang-fedora-centos-testing)"
            }
        }
    ]
}
[2026-02-12 05:25:35,332][  INFO][PID:451485] Sending fedora-messaging bus message in build.end
[2026-02-12 05:25:35,360][  INFO][PID:451485] Compressing /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/10116205-cosmic-osd/builder-live.log by gzip
[2026-02-12 05:25:35,361][  INFO][PID:451485] Running command 'gzip /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/10116205-cosmic-osd/builder-live.log' as PID 638456
[2026-02-12 05:25:35,688][  INFO][PID:451485] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/10116205-cosmic-osd/builder-live.log)
[2026-02-12 05:25:35,689][  INFO][PID:451485] Compressing /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/10116205-cosmic-osd/backend.log by gzip
[2026-02-12 05:25:35,690][  INFO][PID:451485] Running command 'gzip /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/fedora-rawhide-ppc64le/10116205-cosmic-osd/backend.log' as PID 638501