[2025-10-03 13:06:25,189][ INFO][PID:1739113] Marking build as starting [2025-10-03 13:06:25,252][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:06:25,253][ INFO][PID:1739113] VM allocation process starts [2025-10-03 13:06:25,283][ INFO][PID:1739113] Trying to allocate VM: ResallocHost, ticket_id=5461920, requested_tags=['copr_builder', 'arch_x86_64'] [2025-10-03 13:06:30,291][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:06:35,292][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:06:40,296][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:06:45,298][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:06:50,300][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:06:55,301][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:07:00,302][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:07:05,303][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:07:10,304][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:07:13,411][ INFO][PID:1739113] Allocated host ResallocHost, ticket_id=5461920, hostname=3.239.94.171, name=aws_x86_64_spot_prod_07206917_20251003_130415, requested_tags=['copr_builder', 'arch_x86_64'] [2025-10-03 13:07:13,413][ INFO][PID:1739113] Allocating ssh connection to builder [2025-10-03 13:07:13,414][ INFO][PID:1739113] Checking that builder machine is OK [2025-10-03 13:07:13,606][ INFO][PID:1739113] Running remote command: copr-builder-ready fedora-41-x86_64 [2025-10-03 13:07:13,822][ INFO][PID:1739113] Red Hat subscription not needed for fedora-41-x86_64 Builder is ready to be used [2025-10-03 13:07:13,823][ INFO][PID:1739113] Filling build.info file with builder info [2025-10-03 13:07:13,824][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:07:13,826][ INFO][PID:1739113] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@copr", "project_name": "PyPI", "project_dirname": "PyPI", "submitter": "ksurma", "ended_on": null, "started_on": 1759496833.8257868, "submitted_on": null, "status": 3, "chroot": "fedora-41-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9642763-fedora-41-x86_64", "build_id": 9642763, "package_name": "python-eolymp", "package_version": "0.12.49-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@copr/PyPI/python-eolymp", "git_hash": "bb014d39b6efa34c0b982cc02e45ac4d8c9301eb", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@copr/PyPI--ksurma", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI/fedora-41-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@copr/PyPI", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI", "result_dir": "09642763-python-eolymp", "built_packages": "", "id": 9642763, "mockchain_macros": { "copr_username": "@copr", "copr_projectname": "PyPI", "vendor": "Fedora Project COPR (@copr/PyPI)" } } ] } [2025-10-03 13:07:13,850][ INFO][PID:1739113] Sending fedora-messaging bus message in build.start [2025-10-03 13:07:14,626][ INFO][PID:1739113] Sending fedora-messaging bus message in chroot.start [2025-10-03 13:07:14,658][ INFO][PID:1739113] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9642763-fedora-41-x86_64 --chroot fedora-41-x86_64 --detached [2025-10-03 13:07:14,993][ INFO][PID:1739113] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '3.239.94.171' (ED25519) to the list of known hosts. [2025-10-03 13:07:14,994][ INFO][PID:1739113] Downloading the builder-live.log file, attempt 1 [2025-10-03 13:07:14,996][ INFO][PID:1739113] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.239.94.171 copr-rpmbuild-log [2025-10-03 13:07:20,003][ INFO][PID:1739113] Periodic builder liveness probe: alive [2025-10-03 13:07:20,004][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:07:25,006][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:07:30,009][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:07:35,010][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:07:40,011][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:07:45,012][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:07:50,013][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:07:53,049][ ERROR][PID:1739113] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-10-03 13:07:55,015][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:08:00,018][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:08:03,050][WARNING][PID:1739113] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-10-03 13:08:05,019][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:08:10,020][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:08:13,052][ INFO][PID:1739113] Downloading the builder-live.log file, attempt 2 [2025-10-03 13:08:13,054][ INFO][PID:1739113] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.239.94.171 copr-rpmbuild-log [2025-10-03 13:08:15,023][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:08:20,024][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:08:25,026][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:08:30,028][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:08:35,032][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:08:40,034][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:08:45,036][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:08:50,037][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:08:55,039][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:09:00,042][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:09:05,044][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:09:10,045][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:09:13,124][ ERROR][PID:1739113] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-10-03 13:09:15,050][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:09:20,052][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:09:23,129][WARNING][PID:1739113] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-10-03 13:09:25,054][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:09:30,057][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:09:33,131][ INFO][PID:1739113] Downloading the builder-live.log file, attempt 3 [2025-10-03 13:09:33,132][ INFO][PID:1739113] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.239.94.171 copr-rpmbuild-log [2025-10-03 13:09:35,062][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:09:40,063][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:09:45,069][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:09:50,070][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:09:55,072][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:10:00,076][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:10:05,078][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:10:10,081][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:10:15,083][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:10:20,087][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:10:25,091][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:10:30,094][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:10:33,213][ ERROR][PID:1739113] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-10-03 13:10:35,095][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:10:40,097][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:10:43,215][WARNING][PID:1739113] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-10-03 13:10:45,098][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:10:50,102][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:10:53,218][ INFO][PID:1739113] Downloading the builder-live.log file, attempt 4 [2025-10-03 13:10:53,221][ INFO][PID:1739113] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.239.94.171 copr-rpmbuild-log [2025-10-03 13:10:55,104][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:11:00,110][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:11:05,115][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:11:10,117][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:11:15,125][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:11:20,133][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:11:25,136][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:11:30,140][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:11:35,142][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:11:40,144][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:11:45,153][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:11:50,155][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:11:53,305][ ERROR][PID:1739113] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-10-03 13:11:55,158][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:12:00,161][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:12:03,315][ ERROR][PID:1739113] Giving up for unstable SSH, failures: 1759496883.0508347, 1759496963.1291146, 1759497043.2158637, 1759497123.3149176 [2025-10-03 13:12:03,317][ ERROR][PID:1739113] Re-try request for task on 'ResallocHost, ticket_id=5461920, hostname=3.239.94.171, name=aws_x86_64_spot_prod_07206917_20251003_130415, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-10-03 13:12:03,318][ INFO][PID:1739113] Releasing VM back to pool [2025-10-03 13:12:03,360][ INFO][PID:1739113] Retry #1 (on other host) [2025-10-03 13:12:03,361][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:12:03,363][ INFO][PID:1739113] VM allocation process starts [2025-10-03 13:12:03,402][ INFO][PID:1739113] Trying to allocate VM: ResallocHost, ticket_id=5462055, requested_tags=['copr_builder', 'arch_x86_64'] [2025-10-03 13:12:08,409][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:12:09,488][ INFO][PID:1739113] Allocated host ResallocHost, ticket_id=5462055, hostname=54.91.193.63, name=aws_x86_64_normalreserved_prod_07206919_20251003_130418, requested_tags=['copr_builder', 'arch_x86_64'] [2025-10-03 13:12:09,489][ INFO][PID:1739113] Allocating ssh connection to builder [2025-10-03 13:12:09,490][ INFO][PID:1739113] Checking that builder machine is OK [2025-10-03 13:12:09,935][ INFO][PID:1739113] Running remote command: copr-builder-ready fedora-41-x86_64 [2025-10-03 13:12:10,220][ INFO][PID:1739113] Red Hat subscription not needed for fedora-41-x86_64 Builder is ready to be used [2025-10-03 13:12:10,220][ INFO][PID:1739113] Filling build.info file with builder info [2025-10-03 13:12:10,227][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:12:10,232][ INFO][PID:1739113] Not re-notifying FE and msg buses for the new host. [2025-10-03 13:12:10,234][ INFO][PID:1739113] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9642763-fedora-41-x86_64 --chroot fedora-41-x86_64 --detached [2025-10-03 13:12:10,620][ INFO][PID:1739113] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '54.91.193.63' (ED25519) to the list of known hosts. [2025-10-03 13:12:10,621][ INFO][PID:1739113] Downloading the builder-live.log file, attempt 1 [2025-10-03 13:12:10,626][ INFO][PID:1739113] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.91.193.63 copr-rpmbuild-log [2025-10-03 13:12:15,623][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:12:20,625][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:12:25,627][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:12:30,630][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:12:35,640][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:12:40,642][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:12:45,645][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:12:50,646][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:12:55,651][ INFO][PID:1739113] Checking for cancel request [2025-10-03 13:12:57,926][ INFO][PID:1739113] Downloading results from builder [2025-10-03 13:12:57,927][ INFO][PID:1739113] rsyncing of mockbuilder@54.91.193.63:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@copr/PyPI/fedora-41-x86_64/09642763-python-eolymp started [2025-10-03 13:12:57,931][ INFO][PID:1739113] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@54.91.193.63:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@copr/PyPI/fedora-41-x86_64/09642763-python-eolymp/ &> /var/lib/copr/public_html/results/@copr/PyPI/fedora-41-x86_64/09642763-python-eolymp/build-09642763.rsync.log [2025-10-03 13:12:58,291][ INFO][PID:1739113] rsyncing finished. [2025-10-03 13:12:58,296][ INFO][PID:1739113] Releasing VM back to pool [2025-10-03 13:12:58,349][ INFO][PID:1739113] Searching for 'success' file in resultdir [2025-10-03 13:12:58,351][ ERROR][PID:1739113] Build failed: Backend process error: No success file => build failure [2025-10-03 13:12:58,359][ INFO][PID:1739113] Finished build: id=9642763 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/@copr/PyPI chroot=fedora-41-x86_64 [2025-10-03 13:12:58,367][ ERROR][PID:1739113] Backend process error: No success file => build failure [2025-10-03 13:12:58,368][ INFO][PID:1739113] Worker failed build, took 48.1361882686615 [2025-10-03 13:12:58,377][ INFO][PID:1739113] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@copr", "project_name": "PyPI", "project_dirname": "PyPI", "submitter": "ksurma", "ended_on": 1759497178.3681624, "started_on": 1759497130.2319741, "submitted_on": null, "status": 0, "chroot": "fedora-41-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9642763-fedora-41-x86_64", "build_id": 9642763, "package_name": "python-eolymp", "package_version": "0.12.49-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@copr/PyPI/python-eolymp", "git_hash": "bb014d39b6efa34c0b982cc02e45ac4d8c9301eb", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@copr/PyPI--ksurma", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI/fedora-41-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@copr/PyPI", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI", "result_dir": "09642763-python-eolymp", "built_packages": "", "id": 9642763, "mockchain_macros": { "copr_username": "@copr", "copr_projectname": "PyPI", "vendor": "Fedora Project COPR (@copr/PyPI)" } } ] } [2025-10-03 13:12:58,424][ INFO][PID:1739113] Sending fedora-messaging bus message in build.end [2025-10-03 13:12:58,497][ INFO][PID:1739113] Compressing /var/lib/copr/public_html/results/@copr/PyPI/fedora-41-x86_64/09642763-python-eolymp/builder-live.log by gzip [2025-10-03 13:12:58,505][ INFO][PID:1739113] Running command 'gzip /var/lib/copr/public_html/results/@copr/PyPI/fedora-41-x86_64/09642763-python-eolymp/builder-live.log' as PID 1803737 [2025-10-03 13:12:58,552][ INFO][PID:1739113] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@copr/PyPI/fedora-41-x86_64/09642763-python-eolymp/builder-live.log) [2025-10-03 13:12:58,554][ INFO][PID:1739113] Compressing /var/lib/copr/public_html/results/@copr/PyPI/fedora-41-x86_64/09642763-python-eolymp/backend.log by gzip [2025-10-03 13:12:58,558][ INFO][PID:1739113] Running command 'gzip /var/lib/copr/public_html/results/@copr/PyPI/fedora-41-x86_64/09642763-python-eolymp/backend.log' as PID 1803741