[2026-04-07 20:30:39,285][ INFO][PID:3113690] Marking build as starting [2026-04-07 20:30:39,329][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:30:39,331][ INFO][PID:3113690] VM allocation process starts [2026-04-07 20:30:39,418][ INFO][PID:3113690] Trying to allocate VM: ResallocHost, ticket_id=2211585, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-07 20:30:42,454][ INFO][PID:3113690] Allocated host ResallocHost, ticket_id=2211585, hostname=3.238.234.235, name=aws_x86_64_spot_prod_05912617_20260407_201732, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-07 20:30:42,455][ INFO][PID:3113690] Allocating ssh connection to builder [2026-04-07 20:30:42,456][ INFO][PID:3113690] Checking that builder machine is OK [2026-04-07 20:30:42,817][ INFO][PID:3113690] Running remote command: copr-builder-ready srpm-builds [2026-04-07 20:30:43,045][ INFO][PID:3113690] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-04-07 20:30:43,046][ INFO][PID:3113690] Filling build.info file with builder info [2026-04-07 20:30:43,046][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:30:43,048][ INFO][PID:3113690] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "averbyts", "project_name": "HEPrpms", "project_dirname": "HEPrpms", "submitter": "averbyts", "ended_on": null, "started_on": 1775593843.0484958, "submitted_on": null, "status": 3, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10298767", "build_id": 10298767, "package_name": "hevea", "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"#!/bin/bash\\ngit clone --depth 1 https://github.com/andriish/HEPrpms.git\\ncd HEPrpms\\nsh srpmsbuild.sh hevea 2.32\\n\", \"chroot\": \"epel-8-x86_64\", \"builddeps\": \"git spectool wget\", \"resultdir\": \"HEPrpms/hevea/2.32/rpmbuild/SOURCES/\", \"repos\": \"\"}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "averbyts/HEPrpms--averbyts", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": false, "destdir": "/var/lib/copr/public_html/results/averbyts/HEPrpms", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/averbyts/HEPrpms", "result_dir": "10298767", "built_packages": "", "tags": [ "arch_x86_64" ], "id": 10298767, "mockchain_macros": { "copr_username": "averbyts", "copr_projectname": "HEPrpms", "vendor": "Fedora Project COPR (averbyts/HEPrpms)" } } ] } [2026-04-07 20:30:43,088][ INFO][PID:3113690] Sending fedora-messaging bus message in build.start [2026-04-07 20:30:43,743][ INFO][PID:3113690] Sending fedora-messaging bus message in chroot.start [2026-04-07 20:30:43,773][ INFO][PID:3113690] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10298767 --detached [2026-04-07 20:30:44,108][ INFO][PID:3113690] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '3.238.234.235' (ED25519) to the list of known hosts. [2026-04-07 20:30:44,110][ INFO][PID:3113690] Downloading the builder-live.log file, attempt 1 [2026-04-07 20:30:44,113][ INFO][PID:3113690] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.234.235 copr-rpmbuild-log [2026-04-07 20:30:49,117][ INFO][PID:3113690] Periodic builder liveness probe: alive [2026-04-07 20:30:49,117][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:30:54,118][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:30:59,119][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:31:04,121][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:31:09,124][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:31:14,125][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:31:19,126][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:31:24,127][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:31:29,128][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:31:34,129][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:31:39,131][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:31:44,132][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:31:49,133][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:31:54,134][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:31:59,135][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:32:04,135][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:32:09,136][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:32:12,091][ ERROR][PID:3113690] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-07 20:32:14,138][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:32:19,138][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:32:22,092][WARNING][PID:3113690] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-07 20:32:24,139][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:32:29,140][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:32:32,093][ INFO][PID:3113690] Downloading the builder-live.log file, attempt 2 [2026-04-07 20:32:32,094][ INFO][PID:3113690] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.234.235 copr-rpmbuild-log [2026-04-07 20:32:34,141][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:32:39,142][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:32:44,143][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:32:49,144][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:32:54,145][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:32:59,146][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:33:04,147][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:33:09,158][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:33:14,159][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:33:19,160][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:33:24,161][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:33:29,162][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:33:32,159][ ERROR][PID:3113690] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-07 20:33:34,164][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:33:39,165][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:33:42,164][WARNING][PID:3113690] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-07 20:33:44,166][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:33:49,167][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:33:52,165][ INFO][PID:3113690] Downloading the builder-live.log file, attempt 3 [2026-04-07 20:33:52,166][ INFO][PID:3113690] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.234.235 copr-rpmbuild-log [2026-04-07 20:33:54,168][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:33:59,169][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:34:04,170][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:34:09,171][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:34:14,172][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:34:19,173][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:34:24,232][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:34:29,233][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:34:34,234][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:34:39,235][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:34:44,236][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:34:49,237][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:34:52,232][ ERROR][PID:3113690] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-07 20:34:54,238][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:34:59,239][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:35:02,241][WARNING][PID:3113690] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-07 20:35:04,240][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:35:09,241][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:35:12,241][ INFO][PID:3113690] Downloading the builder-live.log file, attempt 4 [2026-04-07 20:35:12,243][ INFO][PID:3113690] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.234.235 copr-rpmbuild-log [2026-04-07 20:35:14,242][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:35:19,243][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:35:24,244][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:35:29,245][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:35:34,246][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:35:39,247][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:35:44,248][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:35:49,249][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:35:54,250][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:35:59,251][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:36:04,252][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:36:09,253][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:36:12,309][ ERROR][PID:3113690] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-07 20:36:14,254][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:36:19,255][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:36:22,339][ ERROR][PID:3113690] Giving up for unstable SSH, failures: 1775593942.092317, 1775594022.1645663, 1775594102.241275, 1775594182.339378 [2026-04-07 20:36:22,340][ ERROR][PID:3113690] Re-try request for task on 'ResallocHost, ticket_id=2211585, hostname=3.238.234.235, name=aws_x86_64_spot_prod_05912617_20260407_201732, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-04-07 20:36:22,340][ INFO][PID:3113690] Releasing VM back to pool [2026-04-07 20:36:22,352][ INFO][PID:3113690] Retry #1 (on other host) [2026-04-07 20:36:22,353][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:36:22,353][ INFO][PID:3113690] VM allocation process starts [2026-04-07 20:36:22,367][ INFO][PID:3113690] Trying to allocate VM: ResallocHost, ticket_id=2211648, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-07 20:36:25,382][ INFO][PID:3113690] Allocated host ResallocHost, ticket_id=2211648, hostname=54.89.219.195, name=aws_x86_64_normalreserved_prod_05912860_20260407_203458, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-07 20:36:25,382][ INFO][PID:3113690] Allocating ssh connection to builder [2026-04-07 20:36:25,383][ INFO][PID:3113690] Checking that builder machine is OK [2026-04-07 20:36:25,737][ INFO][PID:3113690] Running remote command: copr-builder-ready srpm-builds [2026-04-07 20:36:25,951][ INFO][PID:3113690] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-04-07 20:36:26,001][ INFO][PID:3113690] Filling build.info file with builder info [2026-04-07 20:36:26,158][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:36:26,159][ INFO][PID:3113690] Not re-notifying FE and msg buses for the new host. [2026-04-07 20:36:26,160][ INFO][PID:3113690] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10298767 --detached [2026-04-07 20:36:26,516][ INFO][PID:3113690] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '54.89.219.195' (ED25519) to the list of known hosts. [2026-04-07 20:36:26,517][ INFO][PID:3113690] Downloading the builder-live.log file, attempt 1 [2026-04-07 20:36:26,518][ INFO][PID:3113690] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.89.219.195 copr-rpmbuild-log [2026-04-07 20:36:31,517][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:36:36,518][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:36:41,519][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:36:46,520][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:36:51,521][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:36:56,522][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:37:01,523][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:37:06,524][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:37:11,525][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:37:16,526][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:37:21,527][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:37:26,528][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:37:31,529][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:37:36,530][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:37:41,531][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:37:46,532][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:37:51,535][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:37:56,536][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:38:01,537][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:38:06,538][ INFO][PID:3113690] Checking for cancel request [2026-04-07 20:38:07,858][ INFO][PID:3113690] Downloading results from builder [2026-04-07 20:38:07,859][ INFO][PID:3113690] rsyncing of mockbuilder@54.89.219.195:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298767 started [2026-04-07 20:38:07,860][ INFO][PID:3113690] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@54.89.219.195:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298767/ &> /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298767/build-10298767.rsync.log [2026-04-07 20:38:08,140][ INFO][PID:3113690] rsyncing finished. [2026-04-07 20:38:08,141][ INFO][PID:3113690] Releasing VM back to pool [2026-04-07 20:38:08,149][ INFO][PID:3113690] Searching for 'success' file in resultdir [2026-04-07 20:38:08,150][ INFO][PID:3113690] Getting build details [2026-04-07 20:38:08,151][ INFO][PID:3113690] Retrieving SRPM info from /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298767 [2026-04-07 20:38:08,151][ INFO][PID:3113690] SRPM URL: https://download.copr.fedorainfracloud.org/results/averbyts/HEPrpms/srpm-builds/10298767/hevea-2.32-2.src.rpm [2026-04-07 20:38:08,152][ INFO][PID:3113690] build details: {'srpm_url': 'https://download.copr.fedorainfracloud.org/results/averbyts/HEPrpms/srpm-builds/10298767/hevea-2.32-2.src.rpm', 'pkg_name': 'hevea', 'pkg_version': '2.32-2'} [2026-04-07 20:38:08,152][ INFO][PID:3113690] Finished build: id=10298767 failed=False timeout=108000 destdir=/var/lib/copr/public_html/results/averbyts/HEPrpms chroot=srpm-builds [2026-04-07 20:38:08,154][ INFO][PID:3113690] Worker succeeded build, took 101.99450850486755 [2026-04-07 20:38:08,154][ INFO][PID:3113690] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "averbyts", "project_name": "HEPrpms", "project_dirname": "HEPrpms", "submitter": "averbyts", "ended_on": 1775594288.1540806, "started_on": 1775594186.1595721, "submitted_on": null, "status": 1, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10298767", "build_id": 10298767, "package_name": "hevea", "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"#!/bin/bash\\ngit clone --depth 1 https://github.com/andriish/HEPrpms.git\\ncd HEPrpms\\nsh srpmsbuild.sh hevea 2.32\\n\", \"chroot\": \"epel-8-x86_64\", \"builddeps\": \"git spectool wget\", \"resultdir\": \"HEPrpms/hevea/2.32/rpmbuild/SOURCES/\", \"repos\": \"\"}", "pkg_name": "hevea", "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": "https://download.copr.fedorainfracloud.org/results/averbyts/HEPrpms/srpm-builds/10298767/hevea-2.32-2.src.rpm", "uses_devel_repo": null, "sandbox": "averbyts/HEPrpms--averbyts", "results": { "name": "hevea", "epoch": null, "version": "2.32", "release": "2", "exclusivearch": [], "excludearch": [] }, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": false, "destdir": "/var/lib/copr/public_html/results/averbyts/HEPrpms", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/averbyts/HEPrpms", "result_dir": "10298767", "built_packages": "", "tags": [ "arch_x86_64" ], "pkg_version": "2.32-2", "id": 10298767, "mockchain_macros": { "copr_username": "averbyts", "copr_projectname": "HEPrpms", "vendor": "Fedora Project COPR (averbyts/HEPrpms)" } } ] } [2026-04-07 20:38:08,198][ INFO][PID:3113690] Sending fedora-messaging bus message in build.end [2026-04-07 20:38:08,226][ INFO][PID:3113690] Compressing /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298767/builder-live.log by gzip [2026-04-07 20:38:08,227][ INFO][PID:3113690] Running command 'gzip /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298767/builder-live.log' as PID 3141272 [2026-04-07 20:38:08,251][ INFO][PID:3113690] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298767/builder-live.log) [2026-04-07 20:38:08,252][ INFO][PID:3113690] Compressing /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298767/backend.log by gzip [2026-04-07 20:38:08,253][ INFO][PID:3113690] Running command 'gzip /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298767/backend.log' as PID 3141274