[2026-02-06 16:21:17,287][ INFO][PID:3718454] Marking build as starting [2026-02-06 16:21:17,354][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:21:17,359][ INFO][PID:3718454] VM allocation process starts [2026-02-06 16:21:17,388][ INFO][PID:3718454] Trying to allocate VM: ResallocHost, ticket_id=1203140, requested_tags=['copr_builder', 'arch_x86_64'] [2026-02-06 16:21:22,390][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:21:23,415][ INFO][PID:3718454] Allocated host ResallocHost, ticket_id=1203140, hostname=18.212.109.59, name=aws_x86_64_spot_prod_01982841_20260206_161819, requested_tags=['copr_builder', 'arch_x86_64'] [2026-02-06 16:21:23,416][ INFO][PID:3718454] Allocating ssh connection to builder [2026-02-06 16:21:23,416][ INFO][PID:3718454] Checking that builder machine is OK [2026-02-06 16:21:23,759][ INFO][PID:3718454] Running remote command: copr-builder-ready srpm-builds [2026-02-06 16:21:23,984][ INFO][PID:3718454] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-02-06 16:21:23,985][ INFO][PID:3718454] Filling build.info file with builder info [2026-02-06 16:21:23,986][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:21:23,987][ INFO][PID:3718454] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "@cockpit", "project_name": "main-builds", "project_dirname": "main-builds", "submitter": "packit", "ended_on": null, "started_on": 1770394883.987632, "submitted_on": null, "status": 3, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10100447", "build_id": 10100447, "package_name": null, "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"\\n#!/bin/sh\\n\\ngit config --global user.email \\\"hello@packit.dev\\\"\\ngit config --global user.name \\\"Packit\\\"\\nresultdir=$PWD\\npackit -d prepare-sources --result-dir \\\"$resultdir\\\" --ref main --job-config-index 4 --merged-ref 'c240f9387707db08b987e0d3f222b6cc2fc8c481' --no-create-symlinks https://github.com/cockpit-project/cockpit\\n\\n\", \"chroot\": \"fedora-43-x86_64\", \"builddeps\": \"packit automake asciidoctor gcc gettext glib2-devel jq make nodejs systemd-devel\", \"resultdir\": null, \"repos\": \"copr://packit/packit-stable\"}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "@cockpit/main-builds--packit", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/packit/packit-stable/fedora-43-x86_64/", "id": "copr_packit_packit_stable", "name": "Additional repo copr_packit_packit_stable" } ], "background": false, "destdir": "/var/lib/copr/public_html/results/@cockpit/main-builds", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@cockpit/main-builds", "result_dir": "10100447", "built_packages": "", "tags": [ "arch_x86_64" ], "id": 10100447, "mockchain_macros": { "copr_username": "@cockpit", "copr_projectname": "main-builds", "vendor": "Fedora Project COPR (@cockpit/main-builds)" } } ] } [2026-02-06 16:21:24,036][ INFO][PID:3718454] Sending fedora-messaging bus message in build.start [2026-02-06 16:21:24,689][ INFO][PID:3718454] Sending fedora-messaging bus message in chroot.start [2026-02-06 16:21:24,719][ INFO][PID:3718454] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10100447 --detached [2026-02-06 16:21:25,076][ INFO][PID:3718454] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '18.212.109.59' (ED25519) to the list of known hosts. [2026-02-06 16:21:25,077][ INFO][PID:3718454] Downloading the builder-live.log file, attempt 1 [2026-02-06 16:21:25,078][ INFO][PID:3718454] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.212.109.59 copr-rpmbuild-log [2026-02-06 16:21:30,085][ INFO][PID:3718454] Periodic builder liveness probe: alive [2026-02-06 16:21:30,086][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:21:35,087][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:21:40,088][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:21:45,089][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:21:50,090][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:21:55,091][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:22:00,092][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:22:05,093][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:22:10,094][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:22:15,095][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:22:20,097][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:22:25,101][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:22:30,102][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:22:30,970][ ERROR][PID:3718454] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-06 16:22:35,103][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:22:40,104][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:22:40,971][WARNING][PID:3718454] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-06 16:22:45,106][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:22:50,107][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:22:50,972][ INFO][PID:3718454] Downloading the builder-live.log file, attempt 2 [2026-02-06 16:22:50,974][ INFO][PID:3718454] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.212.109.59 copr-rpmbuild-log [2026-02-06 16:22:55,108][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:23:00,109][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:23:05,110][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:23:10,111][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:23:15,112][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:23:20,113][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:23:25,116][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:23:30,120][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:23:35,121][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:23:40,142][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:23:45,143][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:23:50,145][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:23:51,025][ ERROR][PID:3718454] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-06 16:23:55,146][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:24:00,147][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:24:01,040][WARNING][PID:3718454] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-06 16:24:05,148][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:24:10,150][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:24:11,041][ INFO][PID:3718454] Downloading the builder-live.log file, attempt 3 [2026-02-06 16:24:11,042][ INFO][PID:3718454] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.212.109.59 copr-rpmbuild-log [2026-02-06 16:24:15,151][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:24:20,153][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:24:25,154][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:24:30,155][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:24:35,156][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:24:40,157][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:24:45,157][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:24:50,159][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:24:55,160][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:25:00,161][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:25:05,162][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:25:10,163][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:25:11,110][ ERROR][PID:3718454] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-06 16:25:15,164][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:25:20,165][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:25:21,118][WARNING][PID:3718454] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-06 16:25:25,167][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:25:30,168][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:25:31,118][ INFO][PID:3718454] Downloading the builder-live.log file, attempt 4 [2026-02-06 16:25:31,122][ INFO][PID:3718454] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.212.109.59 copr-rpmbuild-log [2026-02-06 16:25:35,169][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:25:40,227][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:25:45,228][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:25:50,230][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:25:55,232][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:26:00,233][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:26:05,234][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:26:10,236][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:26:15,237][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:26:20,238][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:26:25,293][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:26:30,295][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:26:31,189][ ERROR][PID:3718454] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-06 16:26:35,304][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:26:40,309][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:26:41,202][ ERROR][PID:3718454] Giving up for unstable SSH, failures: 1770394960.9714398, 1770395041.0403159, 1770395121.1180246, 1770395201.2022357 [2026-02-06 16:26:41,203][ ERROR][PID:3718454] Re-try request for task on 'ResallocHost, ticket_id=1203140, hostname=18.212.109.59, name=aws_x86_64_spot_prod_01982841_20260206_161819, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-02-06 16:26:41,203][ INFO][PID:3718454] Releasing VM back to pool [2026-02-06 16:26:41,217][ INFO][PID:3718454] Retry #1 (on other host) [2026-02-06 16:26:41,218][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:26:41,219][ INFO][PID:3718454] VM allocation process starts [2026-02-06 16:26:41,229][ INFO][PID:3718454] Trying to allocate VM: ResallocHost, ticket_id=1203246, requested_tags=['copr_builder', 'arch_x86_64'] [2026-02-06 16:26:46,230][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:26:47,252][ INFO][PID:3718454] Allocated host ResallocHost, ticket_id=1203246, hostname=13.217.209.35, name=aws_x86_64_normal_prod_01982451_20260206_161349, requested_tags=['copr_builder', 'arch_x86_64'] [2026-02-06 16:26:47,253][ INFO][PID:3718454] Allocating ssh connection to builder [2026-02-06 16:26:47,256][ INFO][PID:3718454] Checking that builder machine is OK [2026-02-06 16:26:47,629][ INFO][PID:3718454] Running remote command: copr-builder-ready srpm-builds [2026-02-06 16:26:47,868][ INFO][PID:3718454] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-02-06 16:26:47,873][ INFO][PID:3718454] Filling build.info file with builder info [2026-02-06 16:26:47,972][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:26:47,973][ INFO][PID:3718454] Not re-notifying FE and msg buses for the new host. [2026-02-06 16:26:47,973][ INFO][PID:3718454] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10100447 --detached [2026-02-06 16:26:48,358][ INFO][PID:3718454] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '13.217.209.35' (ED25519) to the list of known hosts. [2026-02-06 16:26:48,359][ INFO][PID:3718454] Downloading the builder-live.log file, attempt 1 [2026-02-06 16:26:48,365][ INFO][PID:3718454] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.217.209.35 copr-rpmbuild-log [2026-02-06 16:26:53,360][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:26:58,361][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:27:03,362][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:27:08,365][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:27:13,366][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:27:18,368][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:27:23,375][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:27:28,379][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:27:33,382][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:27:38,383][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:27:43,384][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:27:48,386][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:27:53,387][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:27:58,388][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:28:03,389][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:28:08,390][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:28:13,391][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:28:18,392][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:28:23,393][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:28:28,394][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:28:33,396][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:28:38,397][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:28:43,398][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:28:48,400][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:28:53,402][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:28:58,403][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:29:03,404][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:29:08,406][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:29:13,410][ INFO][PID:3718454] Checking for cancel request [2026-02-06 16:29:16,921][ INFO][PID:3718454] Downloading results from builder [2026-02-06 16:29:16,922][ INFO][PID:3718454] rsyncing of mockbuilder@13.217.209.35:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@cockpit/main-builds/srpm-builds/10100447 started [2026-02-06 16:29:16,923][ INFO][PID:3718454] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@13.217.209.35:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@cockpit/main-builds/srpm-builds/10100447/ &> /var/lib/copr/public_html/results/@cockpit/main-builds/srpm-builds/10100447/build-10100447.rsync.log [2026-02-06 16:29:17,432][ INFO][PID:3718454] rsyncing finished. [2026-02-06 16:29:17,433][ INFO][PID:3718454] Releasing VM back to pool [2026-02-06 16:29:17,443][ INFO][PID:3718454] Searching for 'success' file in resultdir [2026-02-06 16:29:17,445][ INFO][PID:3718454] Getting build details [2026-02-06 16:29:17,445][ INFO][PID:3718454] Retrieving SRPM info from /var/lib/copr/public_html/results/@cockpit/main-builds/srpm-builds/10100447 [2026-02-06 16:29:17,446][ INFO][PID:3718454] SRPM URL: https://download.copr.fedorainfracloud.org/results/@cockpit/main-builds/srpm-builds/10100447/cockpit-355.dev57+gc240f9387-1.src.rpm [2026-02-06 16:29:17,446][ INFO][PID:3718454] build details: {'srpm_url': 'https://download.copr.fedorainfracloud.org/results/@cockpit/main-builds/srpm-builds/10100447/cockpit-355.dev57+gc240f9387-1.src.rpm', 'pkg_name': 'cockpit', 'pkg_version': '355.dev57+gc240f9387-1'} [2026-02-06 16:29:17,447][ INFO][PID:3718454] Finished build: id=10100447 failed=False timeout=108000 destdir=/var/lib/copr/public_html/results/@cockpit/main-builds chroot=srpm-builds [2026-02-06 16:29:17,449][ INFO][PID:3718454] Worker succeeded build, took 149.47580790519714 [2026-02-06 16:29:17,449][ INFO][PID:3718454] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "@cockpit", "project_name": "main-builds", "project_dirname": "main-builds", "submitter": "packit", "ended_on": 1770395357.4491959, "started_on": 1770395207.973388, "submitted_on": null, "status": 1, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10100447", "build_id": 10100447, "package_name": null, "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"\\n#!/bin/sh\\n\\ngit config --global user.email \\\"hello@packit.dev\\\"\\ngit config --global user.name \\\"Packit\\\"\\nresultdir=$PWD\\npackit -d prepare-sources --result-dir \\\"$resultdir\\\" --ref main --job-config-index 4 --merged-ref 'c240f9387707db08b987e0d3f222b6cc2fc8c481' --no-create-symlinks https://github.com/cockpit-project/cockpit\\n\\n\", \"chroot\": \"fedora-43-x86_64\", \"builddeps\": \"packit automake asciidoctor gcc gettext glib2-devel jq make nodejs systemd-devel\", \"resultdir\": null, \"repos\": \"copr://packit/packit-stable\"}", "pkg_name": "cockpit", "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": "https://download.copr.fedorainfracloud.org/results/@cockpit/main-builds/srpm-builds/10100447/cockpit-355.dev57+gc240f9387-1.src.rpm", "uses_devel_repo": null, "sandbox": "@cockpit/main-builds--packit", "results": { "name": "cockpit", "epoch": null, "version": "355.dev57+gc240f9387", "release": "1", "exclusivearch": [], "excludearch": [ "i386", "i486", "i586", "i686", "pentium3", "pentium4", "athlon", "geode" ] }, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/packit/packit-stable/fedora-43-x86_64/", "id": "copr_packit_packit_stable", "name": "Additional repo copr_packit_packit_stable" } ], "background": false, "destdir": "/var/lib/copr/public_html/results/@cockpit/main-builds", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@cockpit/main-builds", "result_dir": "10100447", "built_packages": "", "tags": [ "arch_x86_64" ], "pkg_version": "355.dev57+gc240f9387-1", "id": 10100447, "mockchain_macros": { "copr_username": "@cockpit", "copr_projectname": "main-builds", "vendor": "Fedora Project COPR (@cockpit/main-builds)" } } ] } [2026-02-06 16:29:17,508][ INFO][PID:3718454] Sending fedora-messaging bus message in build.end [2026-02-06 16:29:17,539][ INFO][PID:3718454] Compressing /var/lib/copr/public_html/results/@cockpit/main-builds/srpm-builds/10100447/builder-live.log by gzip [2026-02-06 16:29:17,540][ INFO][PID:3718454] Running command 'gzip /var/lib/copr/public_html/results/@cockpit/main-builds/srpm-builds/10100447/builder-live.log' as PID 3766017 [2026-02-06 16:29:17,767][ INFO][PID:3718454] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@cockpit/main-builds/srpm-builds/10100447/builder-live.log) [2026-02-06 16:29:17,768][ INFO][PID:3718454] Compressing /var/lib/copr/public_html/results/@cockpit/main-builds/srpm-builds/10100447/backend.log by gzip [2026-02-06 16:29:17,769][ INFO][PID:3718454] Running command 'gzip /var/lib/copr/public_html/results/@cockpit/main-builds/srpm-builds/10100447/backend.log' as PID 3766041