[2026-01-13 22:10:59,758][ INFO][PID:2062031] Marking build as starting [2026-01-13 22:10:59,884][ INFO][PID:2062031] Checking that https://download.copr.fedorainfracloud.org/results/@python/python-in-pulp/fedora-43-x86_64//repodata/repomd.xml exists [2026-01-13 22:11:00,310][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:11:00,312][ INFO][PID:2062031] VM allocation process starts [2026-01-13 22:11:00,325][ INFO][PID:2062031] Trying to allocate VM: ResallocHost, ticket_id=800625, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-13 22:11:03,349][ INFO][PID:2062031] Allocated host ResallocHost, ticket_id=800625, hostname=52.205.185.47, name=aws_x86_64_spot_prod_01213986_20260113_220555, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-13 22:11:03,349][ INFO][PID:2062031] Allocating ssh connection to builder [2026-01-13 22:11:03,350][ INFO][PID:2062031] Checking that builder machine is OK [2026-01-13 22:11:03,687][ INFO][PID:2062031] Running remote command: copr-builder-ready fedora-43-x86_64 [2026-01-13 22:11:03,971][ INFO][PID:2062031] Red Hat subscription not needed for fedora-43-x86_64 Builder is ready to be used [2026-01-13 22:11:03,973][ INFO][PID:2062031] Filling build.info file with builder info [2026-01-13 22:11:03,974][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:11:03,976][ INFO][PID:2062031] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@python", "project_name": "python-in-pulp", "project_dirname": "python-in-pulp", "submitter": "https://src.fedoraproject.org/user/farchord", "ended_on": null, "started_on": 1768342263.9758663, "submitted_on": null, "status": 3, "chroot": "fedora-43-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10005468-fedora-43-x86_64", "build_id": 10005468, "package_name": "kf6-kguiaddons", "package_version": "6.22.1-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@python/python-in-pulp/kf6-kguiaddons", "git_hash": "b57695c2f3923a702d90d5464b45f0975dca8426", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@python/python-in-pulp--https://src.fedoraproject.org/user/farchord", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 1, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@python/python-in-pulp/fedora-43-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@python/python-in-pulp", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@python/python-in-pulp", "result_dir": "10005468-kf6-kguiaddons", "built_packages": "", "id": 10005468, "mockchain_macros": { "copr_username": "@python", "copr_projectname": "python-in-pulp", "vendor": "Fedora Project COPR (@python/python-in-pulp)" } } ] } [2026-01-13 22:11:04,058][ INFO][PID:2062031] Sending fedora-messaging bus message in build.start [2026-01-13 22:11:04,834][ INFO][PID:2062031] Sending fedora-messaging bus message in chroot.start [2026-01-13 22:11:04,874][ INFO][PID:2062031] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10005468-fedora-43-x86_64 --chroot fedora-43-x86_64 --detached [2026-01-13 22:11:05,204][ INFO][PID:2062031] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '52.205.185.47' (ED25519) to the list of known hosts. [2026-01-13 22:11:05,207][ INFO][PID:2062031] Downloading the builder-live.log file, attempt 1 [2026-01-13 22:11:05,215][ INFO][PID:2062031] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.205.185.47 copr-rpmbuild-log [2026-01-13 22:11:10,222][ INFO][PID:2062031] Periodic builder liveness probe: alive [2026-01-13 22:11:10,223][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:11:15,224][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:11:20,228][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:11:25,230][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:11:28,731][ ERROR][PID:2062031] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-13 22:11:30,231][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:11:35,232][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:11:38,733][WARNING][PID:2062031] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-13 22:11:40,234][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:11:45,236][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:11:48,734][ INFO][PID:2062031] Downloading the builder-live.log file, attempt 2 [2026-01-13 22:11:48,735][ INFO][PID:2062031] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.205.185.47 copr-rpmbuild-log [2026-01-13 22:11:50,237][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:11:55,238][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:12:00,240][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:12:05,243][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:12:10,244][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:12:15,245][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:12:20,246][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:12:25,247][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:12:30,248][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:12:35,249][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:12:40,250][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:12:45,251][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:12:48,803][ ERROR][PID:2062031] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-13 22:12:50,252][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:12:55,253][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:12:58,833][WARNING][PID:2062031] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-13 22:13:00,254][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:13:05,255][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:13:08,834][ INFO][PID:2062031] Downloading the builder-live.log file, attempt 3 [2026-01-13 22:13:08,856][ INFO][PID:2062031] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.205.185.47 copr-rpmbuild-log [2026-01-13 22:13:10,256][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:13:15,258][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:13:20,259][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:13:25,260][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:13:30,262][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:13:35,265][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:13:40,267][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:13:45,271][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:13:50,299][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:13:55,300][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:14:00,301][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:14:05,303][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:14:08,924][ ERROR][PID:2062031] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-13 22:14:10,304][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:14:15,305][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:14:18,940][WARNING][PID:2062031] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-13 22:14:20,308][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:14:25,309][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:14:28,957][ INFO][PID:2062031] Downloading the builder-live.log file, attempt 4 [2026-01-13 22:14:28,958][ INFO][PID:2062031] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.205.185.47 copr-rpmbuild-log [2026-01-13 22:14:30,312][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:14:35,313][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:14:40,314][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:14:45,315][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:14:50,316][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:14:55,317][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:15:00,329][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:15:05,330][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:15:10,332][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:15:15,401][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:15:20,406][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:15:25,421][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:15:29,028][ ERROR][PID:2062031] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-13 22:15:30,425][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:15:35,427][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:15:39,043][ ERROR][PID:2062031] Giving up for unstable SSH, failures: 1768342298.7337093, 1768342378.833753, 1768342458.9402103, 1768342539.0413253 [2026-01-13 22:15:39,045][ ERROR][PID:2062031] Re-try request for task on 'ResallocHost, ticket_id=800625, hostname=52.205.185.47, name=aws_x86_64_spot_prod_01213986_20260113_220555, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-01-13 22:15:39,046][ INFO][PID:2062031] Releasing VM back to pool [2026-01-13 22:15:39,057][ INFO][PID:2062031] Retry #1 (on other host) [2026-01-13 22:15:39,058][ INFO][PID:2062031] Checking that https://download.copr.fedorainfracloud.org/results/@python/python-in-pulp/fedora-43-x86_64//repodata/repomd.xml exists [2026-01-13 22:15:39,294][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:15:39,295][ INFO][PID:2062031] VM allocation process starts [2026-01-13 22:15:39,306][ INFO][PID:2062031] Trying to allocate VM: ResallocHost, ticket_id=800705, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-13 22:15:42,319][ INFO][PID:2062031] Allocated host ResallocHost, ticket_id=800705, hostname=54.172.217.167, name=aws_x86_64_normalreserved_prod_01214233_20260113_221336, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-13 22:15:42,320][ INFO][PID:2062031] Allocating ssh connection to builder [2026-01-13 22:15:42,320][ INFO][PID:2062031] Checking that builder machine is OK [2026-01-13 22:15:42,653][ INFO][PID:2062031] Running remote command: copr-builder-ready fedora-43-x86_64 [2026-01-13 22:15:42,858][ INFO][PID:2062031] Red Hat subscription not needed for fedora-43-x86_64 Builder is ready to be used [2026-01-13 22:15:42,859][ INFO][PID:2062031] Filling build.info file with builder info [2026-01-13 22:15:42,868][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:15:42,869][ INFO][PID:2062031] Not re-notifying FE and msg buses for the new host. [2026-01-13 22:15:42,870][ INFO][PID:2062031] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10005468-fedora-43-x86_64 --chroot fedora-43-x86_64 --detached [2026-01-13 22:15:43,183][ INFO][PID:2062031] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '54.172.217.167' (ED25519) to the list of known hosts. [2026-01-13 22:15:43,185][ INFO][PID:2062031] Downloading the builder-live.log file, attempt 1 [2026-01-13 22:15:43,187][ INFO][PID:2062031] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.172.217.167 copr-rpmbuild-log [2026-01-13 22:15:48,185][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:15:53,186][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:15:58,188][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:16:03,189][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:16:08,190][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:16:13,192][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:16:18,194][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:16:23,195][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:16:28,196][ INFO][PID:2062031] Checking for cancel request [2026-01-13 22:16:30,324][ INFO][PID:2062031] Downloading results from builder [2026-01-13 22:16:30,325][ INFO][PID:2062031] rsyncing of mockbuilder@54.172.217.167:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@python/python-in-pulp/fedora-43-x86_64/10005468-kf6-kguiaddons started [2026-01-13 22:16:30,326][ INFO][PID:2062031] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@54.172.217.167:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@python/python-in-pulp/fedora-43-x86_64/10005468-kf6-kguiaddons/ &> /var/lib/copr/public_html/results/@python/python-in-pulp/fedora-43-x86_64/10005468-kf6-kguiaddons/build-10005468.rsync.log [2026-01-13 22:16:30,592][ INFO][PID:2062031] rsyncing finished. [2026-01-13 22:16:30,593][ INFO][PID:2062031] Releasing VM back to pool [2026-01-13 22:16:30,605][ INFO][PID:2062031] Searching for 'success' file in resultdir [2026-01-13 22:16:30,607][ ERROR][PID:2062031] Build failed: Backend process error: No success file => build failure [2026-01-13 22:16:30,607][ INFO][PID:2062031] Finished build: id=10005468 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/@python/python-in-pulp chroot=fedora-43-x86_64 [2026-01-13 22:16:30,610][ ERROR][PID:2062031] Backend process error: No success file => build failure [2026-01-13 22:16:30,611][ INFO][PID:2062031] Worker failed build, took 47.7414767742157 [2026-01-13 22:16:30,611][ INFO][PID:2062031] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@python", "project_name": "python-in-pulp", "project_dirname": "python-in-pulp", "submitter": "https://src.fedoraproject.org/user/farchord", "ended_on": 1768342590.6111922, "started_on": 1768342542.8697155, "submitted_on": null, "status": 0, "chroot": "fedora-43-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10005468-fedora-43-x86_64", "build_id": 10005468, "package_name": "kf6-kguiaddons", "package_version": "6.22.1-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@python/python-in-pulp/kf6-kguiaddons", "git_hash": "b57695c2f3923a702d90d5464b45f0975dca8426", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@python/python-in-pulp--https://src.fedoraproject.org/user/farchord", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 1, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@python/python-in-pulp/fedora-43-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@python/python-in-pulp", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@python/python-in-pulp", "result_dir": "10005468-kf6-kguiaddons", "built_packages": "", "id": 10005468, "mockchain_macros": { "copr_username": "@python", "copr_projectname": "python-in-pulp", "vendor": "Fedora Project COPR (@python/python-in-pulp)" } } ] } [2026-01-13 22:16:30,662][ INFO][PID:2062031] Sending fedora-messaging bus message in build.end [2026-01-13 22:16:30,696][ INFO][PID:2062031] Compressing /var/lib/copr/public_html/results/@python/python-in-pulp/fedora-43-x86_64/10005468-kf6-kguiaddons/builder-live.log by gzip [2026-01-13 22:16:30,698][ INFO][PID:2062031] Running command 'gzip /var/lib/copr/public_html/results/@python/python-in-pulp/fedora-43-x86_64/10005468-kf6-kguiaddons/builder-live.log' as PID 2097838 [2026-01-13 22:16:30,703][ INFO][PID:2062031] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@python/python-in-pulp/fedora-43-x86_64/10005468-kf6-kguiaddons/builder-live.log) [2026-01-13 22:16:30,703][ INFO][PID:2062031] Compressing /var/lib/copr/public_html/results/@python/python-in-pulp/fedora-43-x86_64/10005468-kf6-kguiaddons/backend.log by gzip [2026-01-13 22:16:30,705][ INFO][PID:2062031] Running command 'gzip /var/lib/copr/public_html/results/@python/python-in-pulp/fedora-43-x86_64/10005468-kf6-kguiaddons/backend.log' as PID 2097840