[2026-08-21 12:32:11,444][ INFO][PID:2115654] Marking build as starting [2026-08-21 12:32:11,523][ INFO][PID:2115654] Checking that https://download.copr.fedorainfracloud.org/results/@copr/PyPI/fedora-44-x86_64//repodata/repomd.xml exists [2026-08-21 12:32:12,276][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:32:12,341][ INFO][PID:2115654] rpmeta: predicted 4 min for python-sphinx-argparse on fedora-44-x86_64 -> normal builder sufficient [2026-08-21 12:32:12,349][ INFO][PID:2115654] VM allocation process starts [2026-08-21 12:32:12,413][ INFO][PID:2115654] Trying to allocate VM: ResallocHost, ticket_id=4558132, requested_tags=['arch_x86_64', 'copr_builder'] [2026-08-21 12:32:17,422][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:32:22,446][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:32:27,456][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:32:32,461][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:32:37,478][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:32:42,486][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:32:47,490][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:32:52,497][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:32:57,501][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:33:02,505][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:33:07,509][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:33:12,519][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:33:17,528][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:33:22,728][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:33:27,787][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:33:33,144][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:33:38,153][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:33:43,179][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:33:48,199][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:33:53,201][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:33:58,215][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:34:03,246][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:34:08,256][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:34:13,271][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:34:18,278][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:34:23,294][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:34:28,308][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:34:31,127][ INFO][PID:2115654] Allocated host ResallocHost, ticket_id=4558132, hostname=44.223.77.0, name=aws_x86_64_spot_prod_19666396_20260821_123106, requested_tags=['arch_x86_64', 'copr_builder'] [2026-08-21 12:34:31,134][ INFO][PID:2115654] Allocating ssh connection to builder [2026-08-21 12:34:31,142][ INFO][PID:2115654] Checking that builder machine is OK [2026-08-21 12:34:31,721][ INFO][PID:2115654] Installed copr-rpmbuild version: 1.9 [2026-08-21 12:34:31,724][ INFO][PID:2115654] Running remote command: copr-builder-ready fedora-44-x86_64 [2026-08-21 12:34:32,197][ INFO][PID:2115654] Red Hat subscription not needed for fedora-44-x86_64 Builder is ready to be used [2026-08-21 12:34:32,198][ INFO][PID:2115654] Filling build.info file with builder info [2026-08-21 12:34:32,264][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:34:32,271][ INFO][PID:2115654] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@copr", "project_name": "PyPI", "project_dirname": "PyPI", "submitter": "ksurma", "ended_on": null, "started_on": 1787315672.2709224, "submitted_on": null, "status": 3, "chroot": "fedora-44-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10887613-fedora-44-x86_64", "build_id": 10887613, "package_name": "python-sphinx-argparse", "package_version": "0.6.1-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@copr/PyPI/python-sphinx-argparse", "git_hash": "5186f4530b68e357886d0044cb2e63c244d73ceb", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@copr/PyPI--ksurma", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 1, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI/fedora-44-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null, "repo_gpgcheck": false } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@copr/PyPI", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI", "result_dir": "10887613-python-sphinx-argparse", "built_packages": "", "rpmeta_prediction": 4, "id": 10887613, "mockchain_macros": { "copr_username": "@copr", "copr_projectname": "PyPI", "vendor": "Fedora Project COPR (@copr/PyPI)" } } ] } [2026-08-21 12:34:32,416][ INFO][PID:2115654] Sending fedora-messaging bus message in build.start [2026-08-21 12:34:33,902][ INFO][PID:2115654] Sending fedora-messaging bus message in chroot.start [2026-08-21 12:34:34,034][ INFO][PID:2115654] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10887613-fedora-44-x86_64 --chroot fedora-44-x86_64 --detached [2026-08-21 12:34:34,815][ INFO][PID:2115654] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '44.223.77.0' (ED25519) to the list of known hosts. [2026-08-21 12:34:34,818][ INFO][PID:2115654] Downloading the builder-live.log file, attempt 1 [2026-08-21 12:34:34,836][ INFO][PID:2115654] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.223.77.0 copr-rpmbuild-log [2026-08-21 12:34:39,845][ INFO][PID:2115654] Periodic builder liveness probe: alive [2026-08-21 12:34:39,848][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:34:44,863][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:34:49,872][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:34:54,888][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:34:54,948][ ERROR][PID:2115654] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-08-21 12:34:59,892][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:35:04,898][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:35:05,093][WARNING][PID:2115654] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-08-21 12:35:09,901][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:35:14,904][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:35:15,100][ INFO][PID:2115654] Downloading the builder-live.log file, attempt 2 [2026-08-21 12:35:15,112][ INFO][PID:2115654] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.223.77.0 copr-rpmbuild-log [2026-08-21 12:35:19,907][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:35:24,911][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:35:29,931][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:35:35,241][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:35:40,254][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:35:45,290][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:35:50,296][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:35:55,367][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:36:00,383][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:36:05,392][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:36:10,395][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:36:15,308][ ERROR][PID:2115654] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-08-21 12:36:52,429][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:36:57,431][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:36:57,466][WARNING][PID:2115654] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-08-21 12:37:02,433][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:37:07,436][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:37:07,469][ INFO][PID:2115654] Downloading the builder-live.log file, attempt 3 [2026-08-21 12:37:07,474][ INFO][PID:2115654] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.223.77.0 copr-rpmbuild-log [2026-08-21 12:37:12,440][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:37:17,442][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:37:22,447][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:37:27,450][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:37:32,457][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:37:37,463][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:37:42,471][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:37:47,473][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:37:52,476][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:37:57,479][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:38:02,482][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:38:07,485][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:38:07,552][ ERROR][PID:2115654] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-08-21 12:38:35,823][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:38:40,827][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:38:40,837][WARNING][PID:2115654] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-08-21 12:38:45,833][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:38:50,840][ INFO][PID:2115654] Downloading the builder-live.log file, attempt 4 [2026-08-21 12:38:50,843][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:38:50,844][ INFO][PID:2115654] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.223.77.0 copr-rpmbuild-log [2026-08-21 12:38:55,848][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:39:00,850][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:39:05,852][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:39:10,854][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:39:15,857][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:39:20,861][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:39:25,865][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:39:30,867][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:39:46,059][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:39:50,914][ ERROR][PID:2115654] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-08-21 12:39:51,062][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:39:56,067][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:40:00,951][ ERROR][PID:2115654] Giving up for unstable SSH, failures: 1787315705.0936368, 1787315817.466638, 1787315920.8376923, 1787316000.9515207 [2026-08-21 12:40:00,984][ ERROR][PID:2115654] Re-try request for task on 'ResallocHost, ticket_id=4558132, hostname=44.223.77.0, name=aws_x86_64_spot_prod_19666396_20260821_123106, requested_tags=['arch_x86_64', 'copr_builder']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-08-21 12:40:01,016][ INFO][PID:2115654] VM Release request [2026-08-21 12:40:01,305][ INFO][PID:2115654] Retry #1 (on other host) [2026-08-21 12:40:01,308][ INFO][PID:2115654] Checking that https://download.copr.fedorainfracloud.org/results/@copr/PyPI/fedora-44-x86_64//repodata/repomd.xml exists [2026-08-21 12:40:01,660][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:40:01,715][ INFO][PID:2115654] rpmeta: predicted 4 min for python-sphinx-argparse on fedora-44-x86_64 -> normal builder sufficient [2026-08-21 12:40:01,718][ INFO][PID:2115654] VM allocation process starts [2026-08-21 12:40:01,756][ INFO][PID:2115654] Trying to allocate VM: ResallocHost, ticket_id=4558330, requested_tags=['arch_x86_64', 'copr_builder'] [2026-08-21 12:40:06,759][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:40:11,763][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:40:16,801][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:40:21,807][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:40:26,811][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:40:31,815][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:40:36,950][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:40:42,001][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:40:47,190][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:40:52,193][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:40:57,292][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:41:02,302][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:41:07,305][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:41:12,307][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:41:17,312][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:41:22,314][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:41:27,318][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:41:32,322][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:41:37,338][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:41:42,341][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:41:47,344][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:42:01,698][ INFO][PID:2115654] Allocated host ResallocHost, ticket_id=4558330, hostname=54.172.169.207, name=aws_x86_64_reserved_prod_19666577_20260821_123641, requested_tags=['arch_x86_64', 'copr_builder'] [2026-08-21 12:42:01,699][ INFO][PID:2115654] Allocating ssh connection to builder [2026-08-21 12:42:01,701][ INFO][PID:2115654] Checking that builder machine is OK [2026-08-21 12:42:02,080][ INFO][PID:2115654] Installed copr-rpmbuild version: 1.9 [2026-08-21 12:42:02,082][ INFO][PID:2115654] Running remote command: copr-builder-ready fedora-44-x86_64 [2026-08-21 12:42:02,306][ INFO][PID:2115654] Red Hat subscription not needed for fedora-44-x86_64 Builder is ready to be used [2026-08-21 12:42:02,307][ INFO][PID:2115654] Filling build.info file with builder info [2026-08-21 12:42:02,320][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:42:02,325][ INFO][PID:2115654] Not re-notifying FE and msg buses for the new host. [2026-08-21 12:42:02,327][ INFO][PID:2115654] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10887613-fedora-44-x86_64 --chroot fedora-44-x86_64 --detached [2026-08-21 12:42:02,797][ INFO][PID:2115654] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '54.172.169.207' (ED25519) to the list of known hosts. [2026-08-21 12:42:02,800][ INFO][PID:2115654] Downloading the builder-live.log file, attempt 1 [2026-08-21 12:42:02,805][ INFO][PID:2115654] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.172.169.207 copr-rpmbuild-log [2026-08-21 12:42:07,805][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:42:12,831][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:42:17,855][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:42:22,937][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:42:28,133][ INFO][PID:2115654] Checking for cancel request [2026-08-21 12:42:32,992][ INFO][PID:2115654] Downloading results from builder [2026-08-21 12:42:32,994][ INFO][PID:2115654] rsyncing of mockbuilder@54.172.169.207:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@copr/PyPI/fedora-44-x86_64/10887613-python-sphinx-argparse started [2026-08-21 12:42:32,998][ INFO][PID:2115654] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@54.172.169.207:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@copr/PyPI/fedora-44-x86_64/10887613-python-sphinx-argparse/ &> /var/lib/copr/public_html/results/@copr/PyPI/fedora-44-x86_64/10887613-python-sphinx-argparse/build-10887613.rsync.log [2026-08-21 12:42:33,312][ INFO][PID:2115654] rsyncing finished. [2026-08-21 12:42:33,314][ INFO][PID:2115654] VM Release request [2026-08-21 12:42:33,351][ INFO][PID:2115654] Searching for 'success' file in resultdir [2026-08-21 12:42:33,353][ ERROR][PID:2115654] Build failed: Backend process error: No success file => build failure [2026-08-21 12:42:33,377][ INFO][PID:2115654] Finished build: id=10887613 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/@copr/PyPI chroot=fedora-44-x86_64 [2026-08-21 12:42:33,382][ ERROR][PID:2115654] Backend process error: No success file => build failure [2026-08-21 12:42:33,407][ INFO][PID:2115654] Worker failed build, took 31.082590341567993 [2026-08-21 12:42:33,409][ INFO][PID:2115654] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@copr", "project_name": "PyPI", "project_dirname": "PyPI", "submitter": "ksurma", "ended_on": 1787316153.4075627, "started_on": 1787316122.3249724, "submitted_on": null, "status": 0, "chroot": "fedora-44-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10887613-fedora-44-x86_64", "build_id": 10887613, "package_name": "python-sphinx-argparse", "package_version": "0.6.1-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@copr/PyPI/python-sphinx-argparse", "git_hash": "5186f4530b68e357886d0044cb2e63c244d73ceb", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@copr/PyPI--ksurma", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 1, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI/fedora-44-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null, "repo_gpgcheck": false } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@copr/PyPI", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI", "result_dir": "10887613-python-sphinx-argparse", "built_packages": "", "rpmeta_prediction": 4, "id": 10887613, "mockchain_macros": { "copr_username": "@copr", "copr_projectname": "PyPI", "vendor": "Fedora Project COPR (@copr/PyPI)" } } ] } [2026-08-21 12:42:33,469][ INFO][PID:2115654] Sending fedora-messaging bus message in build.end [2026-08-21 12:42:33,563][ INFO][PID:2115654] Compressing /var/lib/copr/public_html/results/@copr/PyPI/fedora-44-x86_64/10887613-python-sphinx-argparse/builder-live.log by gzip [2026-08-21 12:42:33,568][ INFO][PID:2115654] Running command 'gzip /var/lib/copr/public_html/results/@copr/PyPI/fedora-44-x86_64/10887613-python-sphinx-argparse/builder-live.log' as PID 2234148 [2026-08-21 12:42:33,583][ INFO][PID:2115654] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@copr/PyPI/fedora-44-x86_64/10887613-python-sphinx-argparse/builder-live.log) [2026-08-21 12:42:33,585][ INFO][PID:2115654] Compressing /var/lib/copr/public_html/results/@copr/PyPI/fedora-44-x86_64/10887613-python-sphinx-argparse/backend.log by gzip [2026-08-21 12:42:33,588][ INFO][PID:2115654] Running command 'gzip /var/lib/copr/public_html/results/@copr/PyPI/fedora-44-x86_64/10887613-python-sphinx-argparse/backend.log' as PID 2234152