[2025-08-29 15:32:03,684][ INFO][PID:3246253] Marking build as starting [2025-08-29 15:32:03,731][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:32:03,732][ INFO][PID:3246253] VM allocation process starts [2025-08-29 15:32:03,745][ INFO][PID:3246253] Trying to allocate VM: ResallocHost, ticket_id=4820364, requested_tags=['copr_builder', 'arch_x86_64'] [2025-08-29 15:32:06,768][ INFO][PID:3246253] Allocated host ResallocHost, ticket_id=4820364, hostname=44.203.232.76, name=aws_x86_64_spot_prod_06496237_20250829_153015, requested_tags=['copr_builder', 'arch_x86_64'] [2025-08-29 15:32:06,768][ INFO][PID:3246253] Allocating ssh connection to builder [2025-08-29 15:32:06,769][ INFO][PID:3246253] Checking that builder machine is OK [2025-08-29 15:32:07,114][ INFO][PID:3246253] Running remote command: copr-builder-ready epel-9-x86_64 [2025-08-29 15:32:07,522][ INFO][PID:3246253] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2025-08-29 15:32:07,523][ INFO][PID:3246253] Filling build.info file with builder info [2025-08-29 15:32:07,524][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:32:07,526][ INFO][PID:3246253] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@copr", "project_name": "PyPI", "project_dirname": "PyPI", "submitter": "ksurma", "ended_on": null, "started_on": 1756481527.5257494, "submitted_on": null, "status": 3, "chroot": "epel-9-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9506112-epel-9-x86_64", "build_id": 9506112, "package_name": "python-polyapi", "package_version": "5.9.16-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@copr/PyPI/python-polyapi", "git_hash": "11e8549759797b0a44eb1c2de47eaca014038682", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@copr/PyPI--ksurma", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI/epel-9-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@copr/PyPI", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI", "result_dir": "09506112-python-polyapi", "built_packages": "", "id": 9506112, "mockchain_macros": { "copr_username": "@copr", "copr_projectname": "PyPI", "vendor": "Fedora Project COPR (@copr/PyPI)" } } ] } [2025-08-29 15:32:07,566][ INFO][PID:3246253] Sending fedora-messaging bus message in build.start [2025-08-29 15:32:08,682][ INFO][PID:3246253] Sending fedora-messaging bus message in chroot.start [2025-08-29 15:32:08,711][ INFO][PID:3246253] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9506112-epel-9-x86_64 --chroot epel-9-x86_64 --detached [2025-08-29 15:32:09,047][ INFO][PID:3246253] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '44.203.232.76' (ED25519) to the list of known hosts. [2025-08-29 15:32:09,048][ INFO][PID:3246253] Downloading the builder-live.log file, attempt 1 [2025-08-29 15:32:09,050][ INFO][PID:3246253] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.203.232.76 copr-rpmbuild-log [2025-08-29 15:32:14,055][ INFO][PID:3246253] Periodic builder liveness probe: alive [2025-08-29 15:32:14,056][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:32:19,057][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:32:24,059][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:32:29,063][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:32:34,066][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:32:39,069][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:32:44,071][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:32:49,074][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:32:54,075][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:32:59,077][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:33:04,078][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:33:09,080][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:33:14,081][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:33:19,084][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:33:24,087][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:33:29,089][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:33:34,091][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:33:39,092][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:33:44,095][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:33:49,097][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:33:54,100][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:33:59,103][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:34:04,104][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:34:05,030][ ERROR][PID:3246253] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-29 15:34:09,106][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:34:14,107][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:34:15,031][WARNING][PID:3246253] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-08-29 15:34:19,111][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:34:24,112][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:34:25,032][ INFO][PID:3246253] Downloading the builder-live.log file, attempt 2 [2025-08-29 15:34:25,038][ INFO][PID:3246253] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.203.232.76 copr-rpmbuild-log [2025-08-29 15:34:29,113][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:34:34,114][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:34:39,118][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:34:44,121][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:34:49,125][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:34:54,126][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:34:59,129][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:35:04,131][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:35:09,132][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:35:14,135][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:35:19,136][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:35:24,138][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:35:25,123][ ERROR][PID:3246253] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-29 15:35:29,139][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:35:34,141][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:35:35,124][WARNING][PID:3246253] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-08-29 15:35:39,144][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:35:44,145][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:35:45,126][ INFO][PID:3246253] Downloading the builder-live.log file, attempt 3 [2025-08-29 15:35:45,127][ INFO][PID:3246253] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.203.232.76 copr-rpmbuild-log [2025-08-29 15:35:49,146][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:35:54,148][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:35:59,149][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:36:04,152][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:36:09,155][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:36:14,158][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:36:19,159][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:36:24,161][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:36:29,163][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:36:34,166][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:36:39,167][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:36:44,168][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:36:45,196][ ERROR][PID:3246253] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-29 15:36:49,170][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:36:54,172][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:36:55,198][WARNING][PID:3246253] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-08-29 15:36:59,173][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:37:04,176][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:37:05,200][ INFO][PID:3246253] Downloading the builder-live.log file, attempt 4 [2025-08-29 15:37:05,205][ INFO][PID:3246253] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.203.232.76 copr-rpmbuild-log [2025-08-29 15:37:09,177][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:37:14,178][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:37:19,179][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:37:24,180][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:37:29,181][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:37:34,182][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:37:39,183][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:37:44,184][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:37:49,185][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:37:54,187][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:37:59,188][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:38:04,188][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:38:05,274][ ERROR][PID:3246253] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-29 15:38:09,189][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:38:14,190][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:38:15,275][ ERROR][PID:3246253] Giving up for unstable SSH, failures: 1756481655.0318289, 1756481735.1247108, 1756481815.1984878, 1756481895.2751226 [2025-08-29 15:38:15,276][ ERROR][PID:3246253] Re-try request for task on 'ResallocHost, ticket_id=4820364, hostname=44.203.232.76, name=aws_x86_64_spot_prod_06496237_20250829_153015, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-08-29 15:38:15,277][ INFO][PID:3246253] Releasing VM back to pool [2025-08-29 15:38:15,296][ INFO][PID:3246253] Retry #1 (on other host) [2025-08-29 15:38:15,296][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:38:15,297][ INFO][PID:3246253] VM allocation process starts [2025-08-29 15:38:15,320][ INFO][PID:3246253] Trying to allocate VM: ResallocHost, ticket_id=4820456, requested_tags=['copr_builder', 'arch_x86_64'] [2025-08-29 15:38:18,387][ INFO][PID:3246253] Allocated host ResallocHost, ticket_id=4820456, hostname=2620:52:3:1:dead:beef:cafe:c1c8, name=copr_hv_x86_64_04_prod_06496043_20250829_151936, requested_tags=['copr_builder', 'arch_x86_64'] [2025-08-29 15:38:18,387][ INFO][PID:3246253] Allocating ssh connection to builder [2025-08-29 15:38:18,388][ INFO][PID:3246253] Checking that builder machine is OK [2025-08-29 15:38:18,943][ INFO][PID:3246253] Running remote command: copr-builder-ready epel-9-x86_64 [2025-08-29 15:38:19,260][ INFO][PID:3246253] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2025-08-29 15:38:19,261][ INFO][PID:3246253] Filling build.info file with builder info [2025-08-29 15:38:19,261][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:38:19,262][ INFO][PID:3246253] Not re-notifying FE and msg buses for the new host. [2025-08-29 15:38:19,262][ INFO][PID:3246253] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9506112-epel-9-x86_64 --chroot epel-9-x86_64 --detached [2025-08-29 15:38:19,751][ INFO][PID:3246253] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c1c8' (ED25519) to the list of known hosts. [2025-08-29 15:38:19,752][ INFO][PID:3246253] Downloading the builder-live.log file, attempt 1 [2025-08-29 15:38:19,753][ INFO][PID:3246253] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c1c8 copr-rpmbuild-log [2025-08-29 15:38:24,753][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:38:29,754][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:38:34,756][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:38:39,757][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:38:44,758][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:38:49,758][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:38:54,760][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:38:59,761][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:39:04,763][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:39:09,765][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:39:14,767][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:39:19,768][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:39:24,770][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:39:29,771][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:39:34,773][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:39:39,774][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:39:44,775][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:39:49,776][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:39:54,777][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:39:59,778][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:40:04,779][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:40:09,781][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:40:14,782][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:40:19,782][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:40:24,784][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:40:29,785][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:40:34,786][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:40:39,787][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:40:44,788][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:40:49,790][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:40:54,792][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:40:59,793][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:41:04,794][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:41:09,796][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:41:14,799][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:41:19,800][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:41:24,801][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:41:29,802][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:41:34,803][ INFO][PID:3246253] Checking for cancel request [2025-08-29 15:41:36,808][ INFO][PID:3246253] Downloading results from builder [2025-08-29 15:41:36,808][ INFO][PID:3246253] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c1c8]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@copr/PyPI/epel-9-x86_64/09506112-python-polyapi started [2025-08-29 15:41:36,809][ INFO][PID:3246253] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c1c8]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@copr/PyPI/epel-9-x86_64/09506112-python-polyapi/ &> /var/lib/copr/public_html/results/@copr/PyPI/epel-9-x86_64/09506112-python-polyapi/build-09506112.rsync.log [2025-08-29 15:41:37,199][ INFO][PID:3246253] rsyncing finished. [2025-08-29 15:41:37,199][ INFO][PID:3246253] Releasing VM back to pool [2025-08-29 15:41:37,209][ INFO][PID:3246253] Searching for 'success' file in resultdir [2025-08-29 15:41:37,209][ ERROR][PID:3246253] Build failed: Backend process error: No success file => build failure [2025-08-29 15:41:37,210][ INFO][PID:3246253] Finished build: id=9506112 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/@copr/PyPI chroot=epel-9-x86_64 [2025-08-29 15:41:37,211][ ERROR][PID:3246253] Backend process error: No success file => build failure [2025-08-29 15:41:37,212][ INFO][PID:3246253] Worker failed build, took 197.9496874809265 [2025-08-29 15:41:37,212][ INFO][PID:3246253] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@copr", "project_name": "PyPI", "project_dirname": "PyPI", "submitter": "ksurma", "ended_on": 1756482097.2122169, "started_on": 1756481899.2625294, "submitted_on": null, "status": 0, "chroot": "epel-9-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9506112-epel-9-x86_64", "build_id": 9506112, "package_name": "python-polyapi", "package_version": "5.9.16-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@copr/PyPI/python-polyapi", "git_hash": "11e8549759797b0a44eb1c2de47eaca014038682", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@copr/PyPI--ksurma", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI/epel-9-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@copr/PyPI", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI", "result_dir": "09506112-python-polyapi", "built_packages": "", "id": 9506112, "mockchain_macros": { "copr_username": "@copr", "copr_projectname": "PyPI", "vendor": "Fedora Project COPR (@copr/PyPI)" } } ] } [2025-08-29 15:41:37,244][ INFO][PID:3246253] Sending fedora-messaging bus message in build.end [2025-08-29 15:41:37,268][ INFO][PID:3246253] Compressing /var/lib/copr/public_html/results/@copr/PyPI/epel-9-x86_64/09506112-python-polyapi/builder-live.log by gzip [2025-08-29 15:41:37,269][ INFO][PID:3246253] Running command 'gzip /var/lib/copr/public_html/results/@copr/PyPI/epel-9-x86_64/09506112-python-polyapi/builder-live.log' as PID 3296094 [2025-08-29 15:41:37,274][ INFO][PID:3246253] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@copr/PyPI/epel-9-x86_64/09506112-python-polyapi/builder-live.log) [2025-08-29 15:41:37,275][ INFO][PID:3246253] Compressing /var/lib/copr/public_html/results/@copr/PyPI/epel-9-x86_64/09506112-python-polyapi/backend.log by gzip [2025-08-29 15:41:37,276][ INFO][PID:3246253] Running command 'gzip /var/lib/copr/public_html/results/@copr/PyPI/epel-9-x86_64/09506112-python-polyapi/backend.log' as PID 3296095