[2026-02-09 11:33:27,472][ INFO][PID:2987638] Marking build as starting [2026-02-09 11:33:27,653][ INFO][PID:2987638] Checking that /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-44-i386/repodata/repomd.xml exists [2026-02-09 11:33:27,805][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:33:27,807][ INFO][PID:2987638] VM allocation process starts [2026-02-09 11:33:27,827][ INFO][PID:2987638] Trying to allocate VM: ResallocHost, ticket_id=1233582, requested_tags=['copr_builder', 'arch_i386'] [2026-02-09 11:33:30,851][ INFO][PID:2987638] Allocated host ResallocHost, ticket_id=1233582, hostname=3.88.216.206, name=aws_x86_64_spot_prod_02075064_20260209_112451, requested_tags=['copr_builder', 'arch_i386'] [2026-02-09 11:33:30,852][ INFO][PID:2987638] Allocating ssh connection to builder [2026-02-09 11:33:30,852][ INFO][PID:2987638] Checking that builder machine is OK [2026-02-09 11:33:31,239][ INFO][PID:2987638] Running remote command: copr-builder-ready fedora-44-i386 [2026-02-09 11:33:31,667][ INFO][PID:2987638] Red Hat subscription not needed for fedora-44-i386 Builder is ready to be used [2026-02-09 11:33:31,746][ INFO][PID:2987638] Filling build.info file with builder info [2026-02-09 11:33:31,749][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:33:31,753][ INFO][PID:2987638] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "xxmitsu", "project_name": "llvm", "project_dirname": "llvm", "submitter": null, "ended_on": null, "started_on": 1770636811.7534475, "submitted_on": null, "status": 3, "chroot": "fedora-44-i386", "arch": "i386", "buildroot_pkgs": [ "git" ], "task_id": "10108183-fedora-44-i386", "build_id": 10108183, "package_name": "lld", "package_version": "19.0.0-0.1.20240309.git0d6f9bf", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/xxmitsu/llvm/lld", "git_hash": "afe3370b2003b55468416a9a77ea147dc83990b4", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "xxmitsu/llvm--fbc3ebbe-754b-4434-bb1c-9c374db2b87b", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/xxmitsu/llvm/fedora-44-i386/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "bootstrap": "off", "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_i386" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/xxmitsu/llvm", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/xxmitsu/llvm", "result_dir": "10108183-lld", "built_packages": "", "id": 10108183, "mockchain_macros": { "copr_username": "xxmitsu", "copr_projectname": "llvm", "vendor": "Fedora Project COPR (xxmitsu/llvm)" } } ] } [2026-02-09 11:33:31,852][ INFO][PID:2987638] Sending fedora-messaging bus message in build.start [2026-02-09 11:33:32,860][ INFO][PID:2987638] Sending fedora-messaging bus message in chroot.start [2026-02-09 11:33:32,940][ INFO][PID:2987638] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10108183-fedora-44-i386 --chroot fedora-44-i386 --detached [2026-02-09 11:33:33,357][ INFO][PID:2987638] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '3.88.216.206' (ED25519) to the list of known hosts. [2026-02-09 11:33:33,359][ INFO][PID:2987638] Downloading the builder-live.log file, attempt 1 [2026-02-09 11:33:33,361][ INFO][PID:2987638] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.88.216.206 copr-rpmbuild-log [2026-02-09 11:33:38,367][ INFO][PID:2987638] Periodic builder liveness probe: alive [2026-02-09 11:33:38,368][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:33:43,369][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:33:48,375][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:33:53,382][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:33:58,384][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:34:03,389][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:34:08,414][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:34:13,417][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:34:18,420][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:34:23,423][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:34:28,425][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:34:33,426][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:34:38,428][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:34:41,280][ ERROR][PID:2987638] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-09 11:34:43,429][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:34:48,430][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:34:51,281][WARNING][PID:2987638] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-09 11:34:53,434][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:34:58,437][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:35:01,284][ INFO][PID:2987638] Downloading the builder-live.log file, attempt 2 [2026-02-09 11:35:01,286][ INFO][PID:2987638] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.88.216.206 copr-rpmbuild-log [2026-02-09 11:35:03,440][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:35:08,444][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:35:13,447][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:35:18,450][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:35:23,459][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:35:28,460][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:35:33,462][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:35:38,469][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:35:43,470][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:35:48,472][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:35:53,473][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:35:58,478][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:36:01,356][ ERROR][PID:2987638] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-09 11:36:03,483][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:36:08,485][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:36:11,358][WARNING][PID:2987638] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-09 11:36:13,487][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:36:18,491][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:36:21,364][ INFO][PID:2987638] Downloading the builder-live.log file, attempt 3 [2026-02-09 11:36:21,368][ INFO][PID:2987638] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.88.216.206 copr-rpmbuild-log [2026-02-09 11:36:23,494][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:36:28,509][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:36:33,515][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:36:38,518][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:36:43,520][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:36:48,527][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:36:53,559][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:36:58,560][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:37:03,561][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:37:08,565][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:37:13,567][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:37:18,569][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:37:21,471][ ERROR][PID:2987638] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-09 11:37:23,586][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:37:28,587][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:37:31,509][WARNING][PID:2987638] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-09 11:37:33,589][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:37:38,591][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:37:41,510][ INFO][PID:2987638] Downloading the builder-live.log file, attempt 4 [2026-02-09 11:37:41,512][ INFO][PID:2987638] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.88.216.206 copr-rpmbuild-log [2026-02-09 11:37:43,592][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:37:48,593][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:37:53,594][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:37:58,596][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:38:03,597][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:38:08,598][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:38:13,599][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:38:18,613][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:38:23,615][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:38:28,616][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:38:33,624][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:38:38,625][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:38:41,570][ ERROR][PID:2987638] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-09 11:38:43,626][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:38:48,627][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:38:51,640][ ERROR][PID:2987638] Giving up for unstable SSH, failures: 1770636891.2817447, 1770636971.3580048, 1770637051.509957, 1770637131.6404085 [2026-02-09 11:38:51,642][ ERROR][PID:2987638] Re-try request for task on 'ResallocHost, ticket_id=1233582, hostname=3.88.216.206, name=aws_x86_64_spot_prod_02075064_20260209_112451, requested_tags=['copr_builder', 'arch_i386']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-02-09 11:38:51,647][ INFO][PID:2987638] Releasing VM back to pool [2026-02-09 11:38:51,689][ INFO][PID:2987638] Retry #1 (on other host) [2026-02-09 11:38:51,689][ INFO][PID:2987638] Checking that /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-44-i386/repodata/repomd.xml exists [2026-02-09 11:38:51,690][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:38:51,690][ INFO][PID:2987638] VM allocation process starts [2026-02-09 11:38:51,739][ INFO][PID:2987638] Trying to allocate VM: ResallocHost, ticket_id=1233845, requested_tags=['copr_builder', 'arch_i386'] [2026-02-09 11:38:54,773][ INFO][PID:2987638] Allocated host ResallocHost, ticket_id=1233845, hostname=54.90.190.130, name=aws_x86_64_normalreserved_prod_02075274_20260209_113517, requested_tags=['copr_builder', 'arch_i386'] [2026-02-09 11:38:54,774][ INFO][PID:2987638] Allocating ssh connection to builder [2026-02-09 11:38:54,774][ INFO][PID:2987638] Checking that builder machine is OK [2026-02-09 11:38:55,132][ INFO][PID:2987638] Running remote command: copr-builder-ready fedora-44-i386 [2026-02-09 11:38:55,393][ INFO][PID:2987638] Red Hat subscription not needed for fedora-44-i386 Builder is ready to be used [2026-02-09 11:38:55,394][ INFO][PID:2987638] Filling build.info file with builder info [2026-02-09 11:38:55,439][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:38:55,440][ INFO][PID:2987638] Not re-notifying FE and msg buses for the new host. [2026-02-09 11:38:55,441][ INFO][PID:2987638] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10108183-fedora-44-i386 --chroot fedora-44-i386 --detached [2026-02-09 11:38:55,818][ INFO][PID:2987638] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '54.90.190.130' (ED25519) to the list of known hosts. [2026-02-09 11:38:55,820][ INFO][PID:2987638] Downloading the builder-live.log file, attempt 1 [2026-02-09 11:38:55,823][ INFO][PID:2987638] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.90.190.130 copr-rpmbuild-log [2026-02-09 11:39:00,820][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:39:05,821][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:39:10,823][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:39:15,824][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:39:20,825][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:39:25,826][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:39:30,827][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:39:35,828][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:39:40,829][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:39:45,830][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:39:50,831][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:39:55,832][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:40:00,833][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:40:05,834][ INFO][PID:2987638] Checking for cancel request [2026-02-09 11:40:07,817][ INFO][PID:2987638] Downloading results from builder [2026-02-09 11:40:07,818][ INFO][PID:2987638] rsyncing of mockbuilder@54.90.190.130:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-44-i386/10108183-lld started [2026-02-09 11:40:07,818][ INFO][PID:2987638] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@54.90.190.130:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-44-i386/10108183-lld/ &> /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-44-i386/10108183-lld/build-10108183.rsync.log [2026-02-09 11:40:09,024][ INFO][PID:2987638] rsyncing finished. [2026-02-09 11:40:09,025][ INFO][PID:2987638] Releasing VM back to pool [2026-02-09 11:40:09,034][ INFO][PID:2987638] Searching for 'success' file in resultdir [2026-02-09 11:40:09,034][ ERROR][PID:2987638] Build failed: Backend process error: No success file => build failure [2026-02-09 11:40:09,035][ INFO][PID:2987638] Finished build: id=10108183 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/xxmitsu/llvm chroot=fedora-44-i386 [2026-02-09 11:40:09,036][ ERROR][PID:2987638] Backend process error: No success file => build failure [2026-02-09 11:40:09,037][ INFO][PID:2987638] Worker failed build, took 73.5965187549591 [2026-02-09 11:40:09,037][ INFO][PID:2987638] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "xxmitsu", "project_name": "llvm", "project_dirname": "llvm", "submitter": null, "ended_on": 1770637209.037332, "started_on": 1770637135.4408133, "submitted_on": null, "status": 0, "chroot": "fedora-44-i386", "arch": "i386", "buildroot_pkgs": [ "git" ], "task_id": "10108183-fedora-44-i386", "build_id": 10108183, "package_name": "lld", "package_version": "19.0.0-0.1.20240309.git0d6f9bf", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/xxmitsu/llvm/lld", "git_hash": "afe3370b2003b55468416a9a77ea147dc83990b4", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "xxmitsu/llvm--fbc3ebbe-754b-4434-bb1c-9c374db2b87b", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/xxmitsu/llvm/fedora-44-i386/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "bootstrap": "off", "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_i386" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/xxmitsu/llvm", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/xxmitsu/llvm", "result_dir": "10108183-lld", "built_packages": "", "id": 10108183, "mockchain_macros": { "copr_username": "xxmitsu", "copr_projectname": "llvm", "vendor": "Fedora Project COPR (xxmitsu/llvm)" } } ] } [2026-02-09 11:40:09,087][ INFO][PID:2987638] Sending fedora-messaging bus message in build.end [2026-02-09 11:40:09,115][ INFO][PID:2987638] Compressing /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-44-i386/10108183-lld/builder-live.log by gzip [2026-02-09 11:40:09,116][ INFO][PID:2987638] Running command 'gzip /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-44-i386/10108183-lld/builder-live.log' as PID 3057266 [2026-02-09 11:40:09,178][ INFO][PID:2987638] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-44-i386/10108183-lld/builder-live.log) [2026-02-09 11:40:09,179][ INFO][PID:2987638] Compressing /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-44-i386/10108183-lld/backend.log by gzip [2026-02-09 11:40:09,180][ INFO][PID:2987638] Running command 'gzip /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-44-i386/10108183-lld/backend.log' as PID 3057273