[2026-07-16 14:00:57,788][  INFO][PID:3809387] Marking build as starting
[2026-07-16 14:00:57,926][  INFO][PID:3809387] Checking that /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-rawhide-x86_64/repodata/repomd.xml exists
[2026-07-16 14:00:57,946][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:00:58,002][WARNING][PID:3809387] rpmeta: cannot connect to https://rpmeta.fedoraproject.org, skipping
[2026-07-16 14:00:58,003][  INFO][PID:3809387] VM allocation process starts
[2026-07-16 14:00:58,157][  INFO][PID:3809387] Trying to allocate VM: ResallocHost, ticket_id=3934834, requested_tags=['arch_x86_64', 'copr_builder']
[2026-07-16 14:01:03,165][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:01:08,184][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:01:13,197][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:01:18,202][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:01:23,204][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:01:28,205][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:01:33,305][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:01:38,307][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:01:43,320][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:01:48,322][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:01:53,323][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:01:58,333][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:02:03,339][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:02:08,350][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:02:13,357][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:02:18,379][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:02:23,386][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:02:28,394][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:02:33,399][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:02:38,401][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:02:43,403][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:02:48,406][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:02:53,407][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:02:58,409][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:03:03,425][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:03:08,435][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:03:13,437][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:03:18,442][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:03:23,449][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:03:28,463][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:03:33,480][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:03:38,482][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:03:43,483][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:03:48,485][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:03:53,486][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:03:58,488][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:04:03,490][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:04:08,492][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:04:13,498][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:04:18,500][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:04:23,502][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:04:28,508][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:04:33,512][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:04:38,527][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:04:43,538][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:04:48,539][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:04:53,543][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:04:58,546][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:05:03,547][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:05:08,548][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:05:13,550][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:05:18,552][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:05:23,555][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:05:28,560][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:05:33,580][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:05:38,600][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:05:43,619][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:05:48,643][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:05:53,671][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:05:58,680][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:06:03,692][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:06:08,700][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:06:13,704][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:06:18,713][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:06:23,723][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:06:28,729][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:06:33,764][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:06:38,768][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:06:43,776][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:06:48,801][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:06:53,806][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:06:58,808][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:07:03,811][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:07:08,840][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:07:13,861][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:07:18,890][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:07:23,936][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:07:28,947][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:07:33,952][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:07:38,953][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:07:43,972][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:07:48,981][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:07:53,987][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:07:58,993][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:08:03,999][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:08:09,014][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:08:14,017][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:08:16,746][  INFO][PID:3809387] Allocated host ResallocHost, ticket_id=3934834, hostname=44.212.34.148, name=aws_x86_64_spot_prod_13115921_20260716_140424, requested_tags=['arch_x86_64', 'copr_builder']
[2026-07-16 14:08:16,747][  INFO][PID:3809387] Allocating ssh connection to builder
[2026-07-16 14:08:16,748][  INFO][PID:3809387] Checking that builder machine is OK
[2026-07-16 14:08:17,094][  INFO][PID:3809387] Installed copr-rpmbuild version: 1.8
[2026-07-16 14:08:17,095][  INFO][PID:3809387] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2026-07-16 14:08:17,344][  INFO][PID:3809387] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2026-07-16 14:08:17,346][  INFO][PID:3809387] Filling build.info file with builder info
[2026-07-16 14:08:17,347][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:08:17,350][  INFO][PID:3809387] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "xxmitsu",
            "project_name": "llvm",
            "project_dirname": "llvm",
            "submitter": null,
            "ended_on": null,
            "started_on": 1784210897.3503892,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [
                "git"
            ],
            "task_id": "10727091-fedora-rawhide-x86_64",
            "build_id": 10727091,
            "package_name": "compiler-rt",
            "package_version": "19.0.0-0.1.20240309.git0d6f9bf",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/xxmitsu/llvm/compiler-rt",
            "git_hash": "d2f7ae387f782237cd89f42f5a485a70bc6ce99d",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "xxmitsu/llvm--acfe2fa9-6423-4cd6-b40b-153aadd37342",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xxmitsu/llvm/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/xxmitsu/llvm",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/xxmitsu/llvm",
            "result_dir": "10727091-compiler-rt",
            "built_packages": "",
            "rpmeta_prediction": null,
            "id": 10727091,
            "mockchain_macros": {
                "copr_username": "xxmitsu",
                "copr_projectname": "llvm",
                "vendor": "Fedora Project COPR (xxmitsu/llvm)"
            }
        }
    ]
}
[2026-07-16 14:08:17,428][  INFO][PID:3809387] Sending fedora-messaging bus message in build.start
[2026-07-16 14:08:18,095][  INFO][PID:3809387] Sending fedora-messaging bus message in chroot.start
[2026-07-16 14:08:18,160][  INFO][PID:3809387] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10727091-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2026-07-16 14:08:18,866][  INFO][PID:3809387] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '44.212.34.148' (ED25519) to the list of known hosts.


[2026-07-16 14:08:18,867][  INFO][PID:3809387] Downloading the builder-live.log file, attempt 1
[2026-07-16 14:08:18,870][  INFO][PID:3809387] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.212.34.148 copr-rpmbuild-log
[2026-07-16 14:08:23,890][  INFO][PID:3809387] Periodic builder liveness probe: alive
[2026-07-16 14:08:23,895][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:08:28,898][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:08:33,911][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:08:36,026][ ERROR][PID:3809387] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-07-16 14:08:38,929][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:08:43,952][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:08:46,174][WARNING][PID:3809387] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-07-16 14:08:48,981][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:08:53,987][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:08:56,181][  INFO][PID:3809387] Downloading the builder-live.log file, attempt 2
[2026-07-16 14:08:56,190][  INFO][PID:3809387] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.212.34.148 copr-rpmbuild-log
[2026-07-16 14:08:59,071][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:09:04,080][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:09:09,085][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:09:14,091][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:09:19,094][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:09:24,096][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:09:29,101][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:09:34,104][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:09:39,122][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:09:44,130][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:09:49,148][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:09:54,154][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:09:56,269][ ERROR][PID:3809387] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-07-16 14:09:59,167][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:10:04,186][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:10:06,368][WARNING][PID:3809387] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-07-16 14:10:09,252][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:10:14,260][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:10:16,374][  INFO][PID:3809387] Downloading the builder-live.log file, attempt 3
[2026-07-16 14:10:16,380][  INFO][PID:3809387] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.212.34.148 copr-rpmbuild-log
[2026-07-16 14:10:19,272][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:10:24,281][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:10:29,286][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:10:34,300][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:10:39,306][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:10:44,310][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:10:49,312][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:10:54,324][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:10:59,334][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:11:04,340][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:11:09,349][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:11:14,353][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:11:16,488][ ERROR][PID:3809387] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-07-16 14:11:19,358][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:11:24,361][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:11:26,511][WARNING][PID:3809387] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-07-16 14:11:29,363][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:11:34,372][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:11:36,517][  INFO][PID:3809387] Downloading the builder-live.log file, attempt 4
[2026-07-16 14:11:36,529][  INFO][PID:3809387] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.212.34.148 copr-rpmbuild-log
[2026-07-16 14:11:39,387][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:11:44,390][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:11:49,391][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:11:54,393][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:11:59,395][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:12:04,406][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:12:09,408][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:12:14,410][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:12:19,422][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:12:24,425][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:12:29,429][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:12:34,442][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:12:36,640][ ERROR][PID:3809387] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-07-16 14:12:39,444][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:12:44,492][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:12:46,873][ ERROR][PID:3809387] Giving up for unstable SSH, failures: 1784210926.1746504, 1784211006.3687894, 1784211086.511816, 1784211166.8730042
[2026-07-16 14:12:46,902][ ERROR][PID:3809387] Re-try request for task on 'ResallocHost, ticket_id=3934834, hostname=44.212.34.148, name=aws_x86_64_spot_prod_13115921_20260716_140424, requested_tags=['arch_x86_64', 'copr_builder']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-07-16 14:12:46,924][  INFO][PID:3809387] VM Release request
[2026-07-16 14:12:47,315][  INFO][PID:3809387] Retry #1 (on other host)
[2026-07-16 14:12:47,316][  INFO][PID:3809387] Checking that /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-rawhide-x86_64/repodata/repomd.xml exists
[2026-07-16 14:12:47,349][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:12:47,392][WARNING][PID:3809387] rpmeta: cannot connect to https://rpmeta.fedoraproject.org, skipping
[2026-07-16 14:12:47,393][  INFO][PID:3809387] VM allocation process starts
[2026-07-16 14:12:47,605][  INFO][PID:3809387] Trying to allocate VM: ResallocHost, ticket_id=3935214, requested_tags=['arch_x86_64', 'copr_builder']
[2026-07-16 14:12:52,609][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:12:57,640][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:13:02,642][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:13:07,646][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:13:12,650][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:13:17,654][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:13:22,656][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:13:27,662][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:13:32,709][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:13:37,711][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:13:42,715][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:13:47,718][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:13:52,719][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:13:57,725][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:14:02,732][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:14:07,761][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:14:12,775][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:14:17,808][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:14:22,810][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:14:27,811][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:14:32,814][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:14:37,817][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:14:42,847][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:14:47,852][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:14:52,855][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:14:57,858][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:15:02,859][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:15:05,921][  INFO][PID:3809387] Allocated host ResallocHost, ticket_id=3935214, hostname=44.212.34.148, name=aws_x86_64_spot_prod_13115921_20260716_140424, requested_tags=['arch_x86_64', 'copr_builder']
[2026-07-16 14:15:05,921][  INFO][PID:3809387] Allocating ssh connection to builder
[2026-07-16 14:15:05,922][  INFO][PID:3809387] Checking that builder machine is OK
[2026-07-16 14:16:06,022][ ERROR][PID:3809387] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 44.212.34.148 port 22: Connection timed out

[2026-07-16 14:16:16,266][ ERROR][PID:3809387] Re-try request for task on 'ResallocHost, ticket_id=3935214, hostname=44.212.34.148, name=aws_x86_64_spot_prod_13115921_20260716_140424, requested_tags=['arch_x86_64', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-07-16 14:16:16,350][  INFO][PID:3809387] VM Release request
[2026-07-16 14:16:16,474][  INFO][PID:3809387] Retry #2 (on other host)
[2026-07-16 14:16:16,482][  INFO][PID:3809387] Checking that /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-rawhide-x86_64/repodata/repomd.xml exists
[2026-07-16 14:16:16,487][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:16:16,643][WARNING][PID:3809387] rpmeta: cannot connect to https://rpmeta.fedoraproject.org, skipping
[2026-07-16 14:16:16,644][  INFO][PID:3809387] VM allocation process starts
[2026-07-16 14:16:16,762][  INFO][PID:3809387] Trying to allocate VM: ResallocHost, ticket_id=3935332, requested_tags=['arch_x86_64', 'copr_builder']
[2026-07-16 14:16:21,789][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:16:26,803][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:16:31,814][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:16:36,840][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:16:41,878][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:16:46,884][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:16:51,889][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:16:56,891][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:17:01,950][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:17:06,955][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:17:11,973][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:17:16,997][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:17:22,035][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:17:27,042][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:17:32,044][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:17:37,046][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:17:42,047][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:17:47,049][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:17:52,091][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:17:57,126][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:18:02,229][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:18:07,242][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:18:12,247][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:18:17,254][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:18:22,256][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:18:27,258][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:18:32,261][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:18:37,265][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:18:42,268][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:18:47,270][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:18:52,272][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:18:57,278][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:19:02,290][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:19:07,319][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:19:12,324][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:19:17,334][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:19:22,337][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:19:27,360][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:19:32,373][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:19:35,752][  INFO][PID:3809387] Allocated host ResallocHost, ticket_id=3935332, hostname=44.212.34.148, name=aws_x86_64_spot_prod_13115921_20260716_140424, requested_tags=['arch_x86_64', 'copr_builder']
[2026-07-16 14:19:35,752][  INFO][PID:3809387] Allocating ssh connection to builder
[2026-07-16 14:19:35,753][  INFO][PID:3809387] Checking that builder machine is OK
[2026-07-16 14:20:35,824][ ERROR][PID:3809387] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 44.212.34.148 port 22: Connection timed out

[2026-07-16 14:20:45,844][ ERROR][PID:3809387] Re-try request for task on 'ResallocHost, ticket_id=3935332, hostname=44.212.34.148, name=aws_x86_64_spot_prod_13115921_20260716_140424, requested_tags=['arch_x86_64', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-07-16 14:20:45,877][  INFO][PID:3809387] VM Release request
[2026-07-16 14:20:45,909][  INFO][PID:3809387] Retry #3 (on other host)
[2026-07-16 14:20:45,913][  INFO][PID:3809387] Checking that /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-rawhide-x86_64/repodata/repomd.xml exists
[2026-07-16 14:20:45,915][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:20:45,949][WARNING][PID:3809387] rpmeta: cannot connect to https://rpmeta.fedoraproject.org, skipping
[2026-07-16 14:20:45,952][  INFO][PID:3809387] VM allocation process starts
[2026-07-16 14:20:45,980][  INFO][PID:3809387] Trying to allocate VM: ResallocHost, ticket_id=3935487, requested_tags=['arch_x86_64', 'copr_builder']
[2026-07-16 14:20:50,985][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:20:55,989][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:21:00,990][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:21:05,992][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:21:10,995][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:21:15,998][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:21:21,007][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:21:26,020][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:21:31,034][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:21:36,056][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:21:41,066][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:21:46,070][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:21:51,078][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:21:56,088][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:22:01,090][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:22:06,091][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:22:11,093][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:22:16,094][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:22:21,102][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:22:26,116][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:22:31,130][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:22:36,140][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:22:41,146][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:22:46,148][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:22:51,152][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:22:56,171][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:23:01,176][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:23:06,177][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:23:11,184][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:23:16,187][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:23:21,189][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:23:26,204][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:23:31,212][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:23:36,250][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:23:41,264][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:23:46,287][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:23:51,315][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:23:56,326][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:24:01,329][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:24:06,333][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:24:11,334][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:24:16,337][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:24:21,342][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:24:26,346][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:24:31,351][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:24:36,353][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:24:41,355][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:24:46,363][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:24:51,368][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:24:56,384][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:25:01,395][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:25:06,420][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:25:11,424][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:25:16,428][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:25:21,432][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:25:26,434][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:25:31,436][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:25:36,437][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:25:41,439][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:25:46,440][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:25:51,445][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:25:56,470][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:26:01,560][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:26:06,636][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:26:11,642][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:26:16,644][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:26:21,648][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:26:26,657][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:26:31,664][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:26:36,666][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:26:41,670][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:26:46,682][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:26:51,689][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:26:56,692][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:27:01,701][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:27:06,709][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:27:11,715][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:27:16,722][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:27:21,727][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:27:26,733][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:27:31,736][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:27:36,739][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:27:41,775][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:27:46,786][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:27:51,788][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:27:56,792][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:28:01,802][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:28:06,809][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:28:11,811][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:28:16,820][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:28:21,826][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:28:26,828][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:28:31,832][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:28:36,834][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:28:41,843][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:28:46,846][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:28:51,952][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:28:56,956][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:29:01,982][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:29:06,993][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:29:10,599][  INFO][PID:3809387] Allocated host ResallocHost, ticket_id=3935487, hostname=3.82.20.253, name=aws_x86_64_reserved_prod_13116356_20260716_142546, requested_tags=['arch_x86_64', 'copr_builder']
[2026-07-16 14:29:10,608][  INFO][PID:3809387] Allocating ssh connection to builder
[2026-07-16 14:29:10,629][  INFO][PID:3809387] Checking that builder machine is OK
[2026-07-16 14:29:11,055][  INFO][PID:3809387] Installed copr-rpmbuild version: 1.8
[2026-07-16 14:29:11,340][  INFO][PID:3809387] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2026-07-16 14:29:11,567][  INFO][PID:3809387] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2026-07-16 14:29:11,583][  INFO][PID:3809387] Filling build.info file with builder info
[2026-07-16 14:29:11,660][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:29:11,665][  INFO][PID:3809387] Not re-notifying FE and msg buses for the new host.
[2026-07-16 14:29:11,727][  INFO][PID:3809387] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10727091-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2026-07-16 14:29:12,345][  INFO][PID:3809387] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '3.82.20.253' (ED25519) to the list of known hosts.


[2026-07-16 14:29:12,360][  INFO][PID:3809387] Downloading the builder-live.log file, attempt 1
[2026-07-16 14:29:12,445][  INFO][PID:3809387] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.82.20.253 copr-rpmbuild-log
[2026-07-16 14:29:17,516][  INFO][PID:3809387] Periodic builder liveness probe: alive
[2026-07-16 14:29:17,633][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:29:22,645][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:29:27,647][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:29:32,649][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:29:37,650][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:29:42,655][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:29:47,662][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:29:52,663][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:29:57,665][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:30:02,673][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:30:07,678][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:30:12,682][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:30:17,688][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:30:22,694][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:30:27,709][  INFO][PID:3809387] Checking for cancel request
[2026-07-16 14:30:28,486][  INFO][PID:3809387] Downloading results from builder
[2026-07-16 14:30:28,572][  INFO][PID:3809387] rsyncing of mockbuilder@3.82.20.253:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-rawhide-x86_64/10727091-compiler-rt started
[2026-07-16 14:30:28,598][  INFO][PID:3809387] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@3.82.20.253:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-rawhide-x86_64/10727091-compiler-rt/ &> /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-rawhide-x86_64/10727091-compiler-rt/build-10727091.rsync.log
[2026-07-16 14:30:33,801][  INFO][PID:3809387] rsyncing finished.
[2026-07-16 14:30:33,812][  INFO][PID:3809387] VM Release request
[2026-07-16 14:30:33,854][  INFO][PID:3809387] Searching for 'success' file in resultdir
[2026-07-16 14:30:33,855][ ERROR][PID:3809387] Build failed: Backend process error: No success file => build failure
[2026-07-16 14:30:34,381][  INFO][PID:3809387] Finished build: id=10727091 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/xxmitsu/llvm chroot=fedora-rawhide-x86_64 
[2026-07-16 14:30:34,695][ ERROR][PID:3809387] Backend process error: No success file => build failure
[2026-07-16 14:30:34,726][  INFO][PID:3809387] Worker failed build, took 83.06057333946228
[2026-07-16 14:30:34,730][  INFO][PID:3809387] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "xxmitsu",
            "project_name": "llvm",
            "project_dirname": "llvm",
            "submitter": null,
            "ended_on": 1784212234.7262511,
            "started_on": 1784212151.6656778,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [
                "git"
            ],
            "task_id": "10727091-fedora-rawhide-x86_64",
            "build_id": 10727091,
            "package_name": "compiler-rt",
            "package_version": "19.0.0-0.1.20240309.git0d6f9bf",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/xxmitsu/llvm/compiler-rt",
            "git_hash": "d2f7ae387f782237cd89f42f5a485a70bc6ce99d",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "xxmitsu/llvm--acfe2fa9-6423-4cd6-b40b-153aadd37342",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xxmitsu/llvm/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/xxmitsu/llvm",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/xxmitsu/llvm",
            "result_dir": "10727091-compiler-rt",
            "built_packages": "",
            "rpmeta_prediction": null,
            "id": 10727091,
            "mockchain_macros": {
                "copr_username": "xxmitsu",
                "copr_projectname": "llvm",
                "vendor": "Fedora Project COPR (xxmitsu/llvm)"
            }
        }
    ]
}
[2026-07-16 14:30:34,855][  INFO][PID:3809387] Sending fedora-messaging bus message in build.end
[2026-07-16 14:30:37,614][  INFO][PID:3809387] Compressing /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-rawhide-x86_64/10727091-compiler-rt/builder-live.log by gzip
[2026-07-16 14:30:37,633][  INFO][PID:3809387] Running command 'gzip /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-rawhide-x86_64/10727091-compiler-rt/builder-live.log' as PID 4111628
[2026-07-16 14:30:37,740][  INFO][PID:3809387] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-rawhide-x86_64/10727091-compiler-rt/builder-live.log)
[2026-07-16 14:30:37,760][  INFO][PID:3809387] Compressing /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-rawhide-x86_64/10727091-compiler-rt/backend.log by gzip
[2026-07-16 14:30:37,762][  INFO][PID:3809387] Running command 'gzip /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-rawhide-x86_64/10727091-compiler-rt/backend.log' as PID 4111642