[2026-02-09 18:06:44,100][ INFO][PID:1680678] Marking build as starting [2026-02-09 18:06:44,204][ INFO][PID:1680678] Checking that /var/lib/copr/public_html/results/pemensik/unbound-centos/centos-stream-10-x86_64/repodata/repomd.xml exists [2026-02-09 18:06:44,251][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:06:44,252][ INFO][PID:1680678] VM allocation process starts [2026-02-09 18:06:44,269][ INFO][PID:1680678] Trying to allocate VM: ResallocHost, ticket_id=1241973, requested_tags=['copr_builder', 'arch_x86_64'] [2026-02-09 18:06:47,292][ INFO][PID:1680678] Allocated host ResallocHost, ticket_id=1241973, hostname=35.170.64.76, name=aws_x86_64_spot_prod_02097770_20260209_180308, requested_tags=['copr_builder', 'arch_x86_64'] [2026-02-09 18:06:47,292][ INFO][PID:1680678] Allocating ssh connection to builder [2026-02-09 18:06:47,293][ INFO][PID:1680678] Checking that builder machine is OK [2026-02-09 18:06:47,629][ INFO][PID:1680678] Running remote command: copr-builder-ready centos-stream-10-x86_64 [2026-02-09 18:06:47,890][ INFO][PID:1680678] Red Hat subscription not needed for centos-stream-10-x86_64 Builder is ready to be used [2026-02-09 18:06:47,891][ INFO][PID:1680678] Filling build.info file with builder info [2026-02-09 18:06:47,894][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:06:47,896][ INFO][PID:1680678] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "pemensik", "project_name": "unbound-centos", "project_dirname": "unbound-centos", "submitter": "https://src.fedoraproject.org/user/pemensik", "ended_on": null, "started_on": 1770660407.895913, "submitted_on": null, "status": 3, "chroot": "centos-stream-10-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10109760-centos-stream-10-x86_64", "build_id": 10109760, "package_name": "unbound", "package_version": "1.24.2-10", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/pemensik/unbound-centos/unbound", "git_hash": "734741587f638fb7d810cae059bef2da02201560", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "pemensik/unbound-centos--https://src.fedoraproject.org/user/pemensik", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/pemensik/unbound-centos/centos-stream-10-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/pemensik/unbound-centos", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/pemensik/unbound-centos", "result_dir": "10109760-unbound", "built_packages": "", "id": 10109760, "mockchain_macros": { "copr_username": "pemensik", "copr_projectname": "unbound-centos", "vendor": "Fedora Project COPR (pemensik/unbound-centos)" } } ] } [2026-02-09 18:06:47,982][ INFO][PID:1680678] Sending fedora-messaging bus message in build.start [2026-02-09 18:06:48,714][ INFO][PID:1680678] Sending fedora-messaging bus message in chroot.start [2026-02-09 18:06:48,745][ INFO][PID:1680678] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10109760-centos-stream-10-x86_64 --chroot centos-stream-10-x86_64 --detached [2026-02-09 18:06:49,103][ INFO][PID:1680678] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '35.170.64.76' (ED25519) to the list of known hosts. [2026-02-09 18:06:49,104][ INFO][PID:1680678] Downloading the builder-live.log file, attempt 1 [2026-02-09 18:06:49,106][ INFO][PID:1680678] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.170.64.76 copr-rpmbuild-log [2026-02-09 18:06:54,116][ INFO][PID:1680678] Periodic builder liveness probe: alive [2026-02-09 18:06:54,117][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:06:54,616][ ERROR][PID:1680678] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-09 18:06:59,118][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:07:04,119][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:07:04,617][WARNING][PID:1680678] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-09 18:07:09,121][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:07:14,122][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:07:14,618][ INFO][PID:1680678] Downloading the builder-live.log file, attempt 2 [2026-02-09 18:07:14,620][ INFO][PID:1680678] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.170.64.76 copr-rpmbuild-log [2026-02-09 18:07:19,124][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:07:24,125][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:07:29,127][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:07:34,128][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:07:39,131][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:07:44,132][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:07:49,134][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:07:54,137][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:07:59,138][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:08:04,139][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:08:09,141][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:08:14,142][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:08:14,689][ ERROR][PID:1680678] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-09 18:08:19,143][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:08:24,144][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:08:24,697][WARNING][PID:1680678] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-09 18:08:29,146][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:08:34,159][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:08:34,700][ INFO][PID:1680678] Downloading the builder-live.log file, attempt 3 [2026-02-09 18:08:34,703][ INFO][PID:1680678] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.170.64.76 copr-rpmbuild-log [2026-02-09 18:08:39,170][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:08:44,186][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:08:49,239][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:08:54,305][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:08:59,373][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:09:04,429][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:09:09,512][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:09:14,523][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:09:19,532][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:09:24,625][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:09:29,633][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:09:34,643][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:09:34,774][ ERROR][PID:1680678] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-09 18:09:39,651][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:09:44,783][WARNING][PID:1680678] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-09 18:09:45,055][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:09:50,060][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:09:54,786][ INFO][PID:1680678] Downloading the builder-live.log file, attempt 4 [2026-02-09 18:09:54,793][ INFO][PID:1680678] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.170.64.76 copr-rpmbuild-log [2026-02-09 18:09:55,300][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:10:00,306][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:10:05,348][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:10:10,364][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:10:15,463][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:10:20,499][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:10:25,888][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:10:30,910][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:10:35,978][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:10:40,984][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:10:45,987][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:10:50,996][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:10:54,916][ ERROR][PID:1680678] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-09 18:10:56,024][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:11:01,034][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:11:04,946][ ERROR][PID:1680678] Giving up for unstable SSH, failures: 1770660424.6173916, 1770660504.697822, 1770660584.783022, 1770660664.9467442 [2026-02-09 18:11:04,968][ ERROR][PID:1680678] Re-try request for task on 'ResallocHost, ticket_id=1241973, hostname=35.170.64.76, name=aws_x86_64_spot_prod_02097770_20260209_180308, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-02-09 18:11:04,972][ INFO][PID:1680678] Releasing VM back to pool [2026-02-09 18:11:05,056][ INFO][PID:1680678] Retry #1 (on other host) [2026-02-09 18:11:05,068][ INFO][PID:1680678] Checking that /var/lib/copr/public_html/results/pemensik/unbound-centos/centos-stream-10-x86_64/repodata/repomd.xml exists [2026-02-09 18:11:05,136][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:11:05,144][ INFO][PID:1680678] VM allocation process starts [2026-02-09 18:11:05,183][ INFO][PID:1680678] Trying to allocate VM: ResallocHost, ticket_id=1242114, requested_tags=['copr_builder', 'arch_x86_64'] [2026-02-09 18:11:10,189][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:11:15,218][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:11:20,248][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:11:25,253][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:11:30,254][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:11:35,256][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:11:40,257][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:11:45,275][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:11:50,285][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:11:53,382][ INFO][PID:1680678] Allocated host ResallocHost, ticket_id=1242114, hostname=44.220.184.165, name=aws_x86_64_normal_prod_02098120_20260209_181003, requested_tags=['copr_builder', 'arch_x86_64'] [2026-02-09 18:11:53,385][ INFO][PID:1680678] Allocating ssh connection to builder [2026-02-09 18:11:53,390][ INFO][PID:1680678] Checking that builder machine is OK [2026-02-09 18:11:53,844][ INFO][PID:1680678] Running remote command: copr-builder-ready centos-stream-10-x86_64 [2026-02-09 18:11:54,099][ INFO][PID:1680678] Red Hat subscription not needed for centos-stream-10-x86_64 Builder is ready to be used [2026-02-09 18:11:54,104][ INFO][PID:1680678] Filling build.info file with builder info [2026-02-09 18:11:54,254][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:11:54,262][ INFO][PID:1680678] Not re-notifying FE and msg buses for the new host. [2026-02-09 18:11:54,263][ INFO][PID:1680678] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10109760-centos-stream-10-x86_64 --chroot centos-stream-10-x86_64 --detached [2026-02-09 18:11:54,683][ INFO][PID:1680678] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '44.220.184.165' (ED25519) to the list of known hosts. [2026-02-09 18:11:54,691][ INFO][PID:1680678] Downloading the builder-live.log file, attempt 1 [2026-02-09 18:11:54,693][ INFO][PID:1680678] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.220.184.165 copr-rpmbuild-log [2026-02-09 18:11:59,693][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:12:04,700][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:12:09,704][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:12:14,706][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:12:19,711][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:12:24,774][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:12:29,780][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:12:34,783][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:12:39,785][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:12:44,850][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:12:49,852][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:12:54,854][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:12:59,855][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:13:04,858][ INFO][PID:1680678] Checking for cancel request [2026-02-09 18:13:04,946][ INFO][PID:1680678] Downloading results from builder [2026-02-09 18:13:04,946][ INFO][PID:1680678] rsyncing of mockbuilder@44.220.184.165:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/pemensik/unbound-centos/centos-stream-10-x86_64/10109760-unbound started [2026-02-09 18:13:04,951][ INFO][PID:1680678] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@44.220.184.165:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/pemensik/unbound-centos/centos-stream-10-x86_64/10109760-unbound/ &> /var/lib/copr/public_html/results/pemensik/unbound-centos/centos-stream-10-x86_64/10109760-unbound/build-10109760.rsync.log [2026-02-09 18:13:05,666][ INFO][PID:1680678] rsyncing finished. [2026-02-09 18:13:05,667][ INFO][PID:1680678] Releasing VM back to pool [2026-02-09 18:13:05,700][ INFO][PID:1680678] Searching for 'success' file in resultdir [2026-02-09 18:13:05,704][ ERROR][PID:1680678] Build failed: Backend process error: No success file => build failure [2026-02-09 18:13:05,707][ INFO][PID:1680678] Finished build: id=10109760 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/pemensik/unbound-centos chroot=centos-stream-10-x86_64 [2026-02-09 18:13:05,713][ ERROR][PID:1680678] Backend process error: No success file => build failure [2026-02-09 18:13:05,714][ INFO][PID:1680678] Worker failed build, took 71.45265460014343 [2026-02-09 18:13:05,716][ INFO][PID:1680678] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "pemensik", "project_name": "unbound-centos", "project_dirname": "unbound-centos", "submitter": "https://src.fedoraproject.org/user/pemensik", "ended_on": 1770660785.7148578, "started_on": 1770660714.2622032, "submitted_on": null, "status": 0, "chroot": "centos-stream-10-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10109760-centos-stream-10-x86_64", "build_id": 10109760, "package_name": "unbound", "package_version": "1.24.2-10", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/pemensik/unbound-centos/unbound", "git_hash": "734741587f638fb7d810cae059bef2da02201560", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "pemensik/unbound-centos--https://src.fedoraproject.org/user/pemensik", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/pemensik/unbound-centos/centos-stream-10-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/pemensik/unbound-centos", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/pemensik/unbound-centos", "result_dir": "10109760-unbound", "built_packages": "", "id": 10109760, "mockchain_macros": { "copr_username": "pemensik", "copr_projectname": "unbound-centos", "vendor": "Fedora Project COPR (pemensik/unbound-centos)" } } ] } [2026-02-09 18:13:05,791][ INFO][PID:1680678] Sending fedora-messaging bus message in build.end [2026-02-09 18:13:05,839][ INFO][PID:1680678] Compressing /var/lib/copr/public_html/results/pemensik/unbound-centos/centos-stream-10-x86_64/10109760-unbound/builder-live.log by gzip [2026-02-09 18:13:05,843][ INFO][PID:1680678] Running command 'gzip /var/lib/copr/public_html/results/pemensik/unbound-centos/centos-stream-10-x86_64/10109760-unbound/builder-live.log' as PID 1738153 [2026-02-09 18:13:05,953][ INFO][PID:1680678] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/pemensik/unbound-centos/centos-stream-10-x86_64/10109760-unbound/builder-live.log) [2026-02-09 18:13:05,954][ INFO][PID:1680678] Compressing /var/lib/copr/public_html/results/pemensik/unbound-centos/centos-stream-10-x86_64/10109760-unbound/backend.log by gzip [2026-02-09 18:13:05,955][ INFO][PID:1680678] Running command 'gzip /var/lib/copr/public_html/results/pemensik/unbound-centos/centos-stream-10-x86_64/10109760-unbound/backend.log' as PID 1738175