[2025-11-06 14:12:39,191][ INFO][PID:1148685] Marking build as starting [2025-11-06 14:12:39,269][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:12:39,272][ INFO][PID:1148685] VM allocation process starts [2025-11-06 14:12:39,303][ INFO][PID:1148685] Trying to allocate VM: ResallocHost, ticket_id=6056858, requested_tags=['copr_builder', 'arch_x86_64'] [2025-11-06 14:12:44,309][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:12:45,357][ INFO][PID:1148685] Allocated host ResallocHost, ticket_id=6056858, hostname=44.215.67.226, name=aws_x86_64_spot_prod_07863662_20251106_140850, requested_tags=['copr_builder', 'arch_x86_64'] [2025-11-06 14:12:45,358][ INFO][PID:1148685] Allocating ssh connection to builder [2025-11-06 14:12:45,359][ INFO][PID:1148685] Checking that builder machine is OK [2025-11-06 14:12:45,698][ INFO][PID:1148685] Running remote command: copr-builder-ready epel-10-x86_64 [2025-11-06 14:12:45,903][ INFO][PID:1148685] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2025-11-06 14:12:45,904][ INFO][PID:1148685] Filling build.info file with builder info [2025-11-06 14:12:45,905][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:12:45,911][ INFO][PID:1148685] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "ayoungdukie", "project_name": "Personal_Repo", "project_dirname": "Personal_Repo", "submitter": "ayoungdukie", "ended_on": null, "started_on": 1762438365.9105947, "submitted_on": null, "status": 3, "chroot": "epel-10-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9773373-epel-10-x86_64", "build_id": 9773373, "package_name": "topgrade", "package_version": "16.1.1-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/ayoungdukie/Personal_Repo/topgrade", "git_hash": "6cae1a31345c4d507346b02add3de9e910f98015", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "ayoungdukie/Personal_Repo--ayoungdukie", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/ayoungdukie/Personal_Repo/epel-10-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/ayoungdukie/Personal_Repo", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/ayoungdukie/Personal_Repo", "result_dir": "09773373-topgrade", "built_packages": "", "id": 9773373, "mockchain_macros": { "copr_username": "ayoungdukie", "copr_projectname": "Personal_Repo", "vendor": "Fedora Project COPR (ayoungdukie/Personal_Repo)" } } ] } [2025-11-06 14:12:45,955][ INFO][PID:1148685] Sending fedora-messaging bus message in build.start [2025-11-06 14:12:46,887][ INFO][PID:1148685] Sending fedora-messaging bus message in chroot.start [2025-11-06 14:12:46,936][ INFO][PID:1148685] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9773373-epel-10-x86_64 --chroot epel-10-x86_64 --detached [2025-11-06 14:12:47,264][ INFO][PID:1148685] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '44.215.67.226' (ED25519) to the list of known hosts. [2025-11-06 14:12:47,265][ INFO][PID:1148685] Downloading the builder-live.log file, attempt 1 [2025-11-06 14:12:47,268][ INFO][PID:1148685] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.215.67.226 copr-rpmbuild-log [2025-11-06 14:12:52,276][ INFO][PID:1148685] Periodic builder liveness probe: alive [2025-11-06 14:12:52,277][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:12:57,278][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:12:58,488][ ERROR][PID:1148685] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-11-06 14:13:02,280][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:13:07,282][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:13:08,491][WARNING][PID:1148685] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-11-06 14:13:12,286][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:13:17,287][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:13:18,492][ INFO][PID:1148685] Downloading the builder-live.log file, attempt 2 [2025-11-06 14:13:18,495][ INFO][PID:1148685] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.215.67.226 copr-rpmbuild-log [2025-11-06 14:13:22,291][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:13:27,295][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:13:32,296][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:13:37,297][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:13:42,300][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:13:47,309][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:13:52,314][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:13:57,315][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:14:02,316][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:14:07,318][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:14:12,320][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:14:17,322][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:14:18,567][ ERROR][PID:1148685] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-11-06 14:14:22,323][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:14:27,324][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:14:28,568][WARNING][PID:1148685] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-11-06 14:14:32,326][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:14:37,330][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:14:38,569][ INFO][PID:1148685] Downloading the builder-live.log file, attempt 3 [2025-11-06 14:14:38,571][ INFO][PID:1148685] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.215.67.226 copr-rpmbuild-log [2025-11-06 14:14:42,332][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:14:47,333][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:14:52,336][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:14:57,338][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:15:02,340][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:15:07,345][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:15:12,347][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:15:17,349][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:15:22,351][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:15:27,355][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:15:32,357][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:15:37,363][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:15:38,644][ ERROR][PID:1148685] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-11-06 14:15:42,365][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:15:47,370][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:15:48,651][WARNING][PID:1148685] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-11-06 14:15:52,372][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:15:57,376][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:15:58,652][ INFO][PID:1148685] Downloading the builder-live.log file, attempt 4 [2025-11-06 14:15:58,657][ INFO][PID:1148685] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.215.67.226 copr-rpmbuild-log [2025-11-06 14:16:02,380][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:16:07,381][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:16:12,383][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:16:17,390][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:16:22,393][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:16:27,395][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:16:32,397][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:16:37,398][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:16:42,400][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:16:47,401][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:16:52,402][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:16:57,404][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:16:58,728][ ERROR][PID:1148685] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-11-06 14:17:02,406][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:17:07,409][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:17:08,736][ ERROR][PID:1148685] Giving up for unstable SSH, failures: 1762438388.4912252, 1762438468.5685246, 1762438548.6514258, 1762438628.7360814 [2025-11-06 14:17:08,738][ ERROR][PID:1148685] Re-try request for task on 'ResallocHost, ticket_id=6056858, hostname=44.215.67.226, name=aws_x86_64_spot_prod_07863662_20251106_140850, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-11-06 14:17:08,740][ INFO][PID:1148685] Releasing VM back to pool [2025-11-06 14:17:08,800][ INFO][PID:1148685] Retry #1 (on other host) [2025-11-06 14:17:08,801][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:17:08,802][ INFO][PID:1148685] VM allocation process starts [2025-11-06 14:17:08,837][ INFO][PID:1148685] Trying to allocate VM: ResallocHost, ticket_id=6056939, requested_tags=['copr_builder', 'arch_x86_64'] [2025-11-06 14:17:11,859][ INFO][PID:1148685] Allocated host ResallocHost, ticket_id=6056939, hostname=98.92.113.238, name=aws_x86_64_normalreserved_prod_07863709_20251106_141112, requested_tags=['copr_builder', 'arch_x86_64'] [2025-11-06 14:17:11,860][ INFO][PID:1148685] Allocating ssh connection to builder [2025-11-06 14:17:11,860][ INFO][PID:1148685] Checking that builder machine is OK [2025-11-06 14:17:12,189][ INFO][PID:1148685] Running remote command: copr-builder-ready epel-10-x86_64 [2025-11-06 14:17:12,388][ INFO][PID:1148685] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2025-11-06 14:17:12,389][ INFO][PID:1148685] Filling build.info file with builder info [2025-11-06 14:17:12,393][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:17:12,394][ INFO][PID:1148685] Not re-notifying FE and msg buses for the new host. [2025-11-06 14:17:12,394][ INFO][PID:1148685] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9773373-epel-10-x86_64 --chroot epel-10-x86_64 --detached [2025-11-06 14:17:12,705][ INFO][PID:1148685] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '98.92.113.238' (ED25519) to the list of known hosts. [2025-11-06 14:17:12,706][ INFO][PID:1148685] Downloading the builder-live.log file, attempt 1 [2025-11-06 14:17:12,708][ INFO][PID:1148685] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.92.113.238 copr-rpmbuild-log [2025-11-06 14:17:17,707][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:17:22,708][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:17:27,709][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:17:32,711][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:17:37,716][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:17:42,718][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:17:47,720][ INFO][PID:1148685] Checking for cancel request [2025-11-06 14:17:49,648][ INFO][PID:1148685] Downloading results from builder [2025-11-06 14:17:49,649][ INFO][PID:1148685] rsyncing of mockbuilder@98.92.113.238:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/ayoungdukie/Personal_Repo/epel-10-x86_64/09773373-topgrade started [2025-11-06 14:17:49,650][ INFO][PID:1148685] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@98.92.113.238:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/ayoungdukie/Personal_Repo/epel-10-x86_64/09773373-topgrade/ &> /var/lib/copr/public_html/results/ayoungdukie/Personal_Repo/epel-10-x86_64/09773373-topgrade/build-09773373.rsync.log [2025-11-06 14:17:50,063][ INFO][PID:1148685] rsyncing finished. [2025-11-06 14:17:50,066][ INFO][PID:1148685] Releasing VM back to pool [2025-11-06 14:17:50,078][ INFO][PID:1148685] Searching for 'success' file in resultdir [2025-11-06 14:17:50,079][ INFO][PID:1148685] Going to sign pkgs from source: 9773373-epel-10-x86_64 in chroot: /var/lib/copr/public_html/results/ayoungdukie/Personal_Repo/epel-10-x86_64 [2025-11-06 14:17:50,081][ INFO][PID:1148685] Calling '/bin/sign -u ayoungdukie#Personal_Repo@copr.fedorahosted.org -p' (attempt #1) [2025-11-06 14:17:50,432][ INFO][PID:1148685] Calling '/bin/sign -4 -h sha256 -u ayoungdukie#Personal_Repo@copr.fedorahosted.org -r /var/lib/copr/public_html/results/ayoungdukie/Personal_Repo/epel-10-x86_64/09773373-topgrade/topgrade-16.1.1-1.el10.src.rpm' (attempt #1) [2025-11-06 14:17:51,152][ INFO][PID:1148685] signed rpm: /var/lib/copr/public_html/results/ayoungdukie/Personal_Repo/epel-10-x86_64/09773373-topgrade/topgrade-16.1.1-1.el10.src.rpm [2025-11-06 14:17:51,155][ INFO][PID:1148685] Calling '/bin/sign -4 -h sha256 -u ayoungdukie#Personal_Repo@copr.fedorahosted.org -r /var/lib/copr/public_html/results/ayoungdukie/Personal_Repo/epel-10-x86_64/09773373-topgrade/topgrade-16.1.1-1.el10.x86_64.rpm' (attempt #1) [2025-11-06 14:17:51,908][ INFO][PID:1148685] signed rpm: /var/lib/copr/public_html/results/ayoungdukie/Personal_Repo/epel-10-x86_64/09773373-topgrade/topgrade-16.1.1-1.el10.x86_64.rpm [2025-11-06 14:17:51,910][ INFO][PID:1148685] Sign done [2025-11-06 14:17:51,911][ INFO][PID:1148685] Incremental createrepo run, adding 09773373-topgrade into https://download.copr.fedorainfracloud.org/results/ayoungdukie/Personal_Repo/epel-10-x86_64, (auto-create-repo=True) [2025-11-06 14:17:51,914][ INFO][PID:1148685] Running command 'copr-repo --batched /var/lib/copr/public_html/results/ayoungdukie/Personal_Repo/epel-10-x86_64 --add 09773373-topgrade --no-appstream-metadata' as PID 1203146 [2025-11-06 14:17:52,923][ INFO][PID:1148685] Finished after 1 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/ayoungdukie/Personal_Repo/epel-10-x86_64 --add 09773373-topgrade --no-appstream-metadata) [2025-11-06 14:17:52,924][ INFO][PID:1148685] Getting build details [2025-11-06 14:17:52,924][ INFO][PID:1148685] Listing built binary packages in /var/lib/copr/public_html/results/ayoungdukie/Personal_Repo/epel-10-x86_64/09773373-topgrade [2025-11-06 14:17:52,924][ INFO][PID:1148685] Built packages: ['topgrade 16.1.1'] [2025-11-06 14:17:52,925][ INFO][PID:1148685] build details: {'built_packages': 'topgrade 16.1.1'} [2025-11-06 14:17:52,925][ INFO][PID:1148685] Retrieving pubkey [2025-11-06 14:17:52,926][ INFO][PID:1148685] Calling '/bin/sign -u ayoungdukie#Personal_Repo@copr.fedorahosted.org -p' (attempt #1) [2025-11-06 14:17:53,107][ INFO][PID:1148685] Added pubkey for user ayoungdukie project Personal_Repo into: /var/lib/copr/public_html/results/ayoungdukie/Personal_Repo/pubkey.gpg [2025-11-06 14:17:53,107][ INFO][PID:1148685] Finished build: id=9773373 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/ayoungdukie/Personal_Repo chroot=epel-10-x86_64 [2025-11-06 14:17:53,110][ INFO][PID:1148685] Worker succeeded build, took 40.715636253356934 [2025-11-06 14:17:53,110][ INFO][PID:1148685] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "ayoungdukie", "project_name": "Personal_Repo", "project_dirname": "Personal_Repo", "submitter": "ayoungdukie", "ended_on": 1762438673.1100376, "started_on": 1762438632.3944013, "submitted_on": null, "status": 1, "chroot": "epel-10-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9773373-epel-10-x86_64", "build_id": 9773373, "package_name": "topgrade", "package_version": "16.1.1-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/ayoungdukie/Personal_Repo/topgrade", "git_hash": "6cae1a31345c4d507346b02add3de9e910f98015", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "ayoungdukie/Personal_Repo--ayoungdukie", "results": { "packages": [ { "name": "topgrade", "epoch": null, "version": "16.1.1", "release": "1.el10", "arch": "x86_64" }, { "name": "topgrade", "epoch": null, "version": "16.1.1", "release": "1.el10", "arch": "src" } ] }, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/ayoungdukie/Personal_Repo/epel-10-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/ayoungdukie/Personal_Repo", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/ayoungdukie/Personal_Repo", "result_dir": "09773373-topgrade", "built_packages": "topgrade 16.1.1", "id": 9773373, "mockchain_macros": { "copr_username": "ayoungdukie", "copr_projectname": "Personal_Repo", "vendor": "Fedora Project COPR (ayoungdukie/Personal_Repo)" } } ] } [2025-11-06 14:17:53,145][ INFO][PID:1148685] Sending fedora-messaging bus message in build.end [2025-11-06 14:17:53,173][ INFO][PID:1148685] Compressing /var/lib/copr/public_html/results/ayoungdukie/Personal_Repo/epel-10-x86_64/09773373-topgrade/builder-live.log by gzip [2025-11-06 14:17:53,175][ INFO][PID:1148685] Running command 'gzip /var/lib/copr/public_html/results/ayoungdukie/Personal_Repo/epel-10-x86_64/09773373-topgrade/builder-live.log' as PID 1203412 [2025-11-06 14:17:53,179][ INFO][PID:1148685] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/ayoungdukie/Personal_Repo/epel-10-x86_64/09773373-topgrade/builder-live.log) [2025-11-06 14:17:53,180][ INFO][PID:1148685] Compressing /var/lib/copr/public_html/results/ayoungdukie/Personal_Repo/epel-10-x86_64/09773373-topgrade/backend.log by gzip [2025-11-06 14:17:53,182][ INFO][PID:1148685] Running command 'gzip /var/lib/copr/public_html/results/ayoungdukie/Personal_Repo/epel-10-x86_64/09773373-topgrade/backend.log' as PID 1203413