[2024-09-20 15:07:44,598][ INFO][PID:1475012] Marking build as starting [2024-09-20 15:07:44,744][ INFO][PID:1475012] Trying to allocate VM [2024-09-20 15:15:03,006][ INFO][PID:1475012] Allocated host ResallocHost, ticket_id=4978731, hostname=44.202.212.4 [2024-09-20 15:15:03,007][ INFO][PID:1475012] Allocating ssh connection to builder [2024-09-20 15:15:03,008][ INFO][PID:1475012] Checking that builder machine is OK [2024-09-20 15:15:03,385][ INFO][PID:1475012] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.202.212.4 '/usr/bin/test -f /etc/mock/fedora-39-x86_64.cfg' [2024-09-20 15:15:03,600][ INFO][PID:1475012] Filling build.info file with builder info [2024-09-20 15:15:03,602][ INFO][PID:1475012] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@python", "project_name": "python3.12", "project_dirname": "python3.12", "submitter": "https://src.fedoraproject.org/user/kkeithle", "ended_on": null, "started_on": 1726845303.6020298, "submitted_on": null, "status": 3, "chroot": "fedora-39-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "8043666-fedora-39-x86_64", "build_id": 8043666, "package_name": "ceph", "package_version": "2:19.2.0-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@python/python3.12/ceph", "git_hash": "476eaff500412c7ca7b1d76b31259d3bd64b8b2e", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@python/python3.12--https://src.fedoraproject.org/user/kkeithle", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "background": true, "bootstrap": "off", "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@python/python3.12/fedora-39-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/", "id": "http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch", "name": "Additional repo http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch" } ], "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@python/python3.12", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@python/python3.12", "result_dir": "08043666-ceph", "built_packages": "", "id": 8043666, "mockchain_macros": { "copr_username": "@python", "copr_projectname": "python3.12", "vendor": "Fedora Project COPR (@python/python3.12)" } } ] } [2024-09-20 15:15:03,676][ INFO][PID:1475012] Sending fedora-messaging bus message in build.start [2024-09-20 15:15:04,473][ INFO][PID:1475012] Sending fedora-messaging bus message in chroot.start [2024-09-20 15:15:04,494][ INFO][PID:1475012] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8043666-fedora-39-x86_64 --chroot fedora-39-x86_64 --detached [2024-09-20 15:15:04,892][ INFO][PID:1475012] Downloading the builder-live.log file, attempt 1 [2024-09-20 15:15:04,894][ INFO][PID:1475012] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.202.212.4 copr-rpmbuild-log [2024-09-20 15:15:09,900][ INFO][PID:1475012] Status: False [2024-09-20 15:25:10,090][ INFO][PID:1475012] Status: False [2024-09-20 15:34:54,989][ ERROR][PID:1475012] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-09-20 15:35:04,992][WARNING][PID:1475012] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-09-20 15:35:10,295][ INFO][PID:1475012] Status: False [2024-09-20 15:35:14,993][ INFO][PID:1475012] Downloading the builder-live.log file, attempt 2 [2024-09-20 15:35:15,017][ INFO][PID:1475012] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.202.212.4 copr-rpmbuild-log [2024-09-20 15:36:15,087][ ERROR][PID:1475012] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-09-20 15:36:25,105][WARNING][PID:1475012] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-09-20 15:36:35,109][ INFO][PID:1475012] Downloading the builder-live.log file, attempt 3 [2024-09-20 15:36:35,118][ INFO][PID:1475012] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.202.212.4 copr-rpmbuild-log [2024-09-20 15:37:35,189][ ERROR][PID:1475012] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-09-20 15:37:45,207][WARNING][PID:1475012] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-09-20 15:37:55,212][ INFO][PID:1475012] Downloading the builder-live.log file, attempt 4 [2024-09-20 15:37:55,215][ INFO][PID:1475012] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.202.212.4 copr-rpmbuild-log [2024-09-20 15:38:55,288][ ERROR][PID:1475012] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-09-20 15:39:05,304][ ERROR][PID:1475012] Giving up for unstable SSH, failures: 1726846504.9921029, 1726846585.1056018, 1726846665.2071464, 1726846745.3038342 [2024-09-20 15:39:05,306][ ERROR][PID:1475012] Re-try request for task on 'ResallocHost, ticket_id=4978731, hostname=44.202.212.4': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2024-09-20 15:39:05,307][ INFO][PID:1475012] Releasing VM back to pool [2024-09-20 15:39:05,353][ INFO][PID:1475012] Retry #1 (on other host) [2024-09-20 15:39:05,363][ INFO][PID:1475012] Trying to allocate VM [2024-09-20 15:46:23,817][ INFO][PID:1475012] Allocated host ResallocHost, ticket_id=4979618, hostname=18.212.87.55 [2024-09-20 15:46:23,829][ INFO][PID:1475012] Allocating ssh connection to builder [2024-09-20 15:46:23,830][ INFO][PID:1475012] Checking that builder machine is OK [2024-09-20 15:46:24,194][ INFO][PID:1475012] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.212.87.55 '/usr/bin/test -f /etc/mock/fedora-39-x86_64.cfg' [2024-09-20 15:46:24,412][ INFO][PID:1475012] Filling build.info file with builder info [2024-09-20 15:46:24,462][ INFO][PID:1475012] Not re-notifying FE and msg buses for the new host. [2024-09-20 15:46:24,463][ INFO][PID:1475012] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8043666-fedora-39-x86_64 --chroot fedora-39-x86_64 --detached [2024-09-20 15:46:24,865][ INFO][PID:1475012] Downloading the builder-live.log file, attempt 1 [2024-09-20 15:46:24,867][ INFO][PID:1475012] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.212.87.55 copr-rpmbuild-log [2024-09-20 15:46:29,894][ INFO][PID:1475012] Status: False [2024-09-20 15:50:14,849][ ERROR][PID:1475012] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-09-20 15:50:24,852][WARNING][PID:1475012] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-09-20 15:50:34,855][ INFO][PID:1475012] Downloading the builder-live.log file, attempt 2 [2024-09-20 15:50:34,860][ INFO][PID:1475012] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.212.87.55 copr-rpmbuild-log [2024-09-20 15:51:34,927][ ERROR][PID:1475012] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-09-20 15:51:44,935][WARNING][PID:1475012] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-09-20 15:51:54,936][ INFO][PID:1475012] Downloading the builder-live.log file, attempt 3 [2024-09-20 15:51:54,938][ INFO][PID:1475012] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.212.87.55 copr-rpmbuild-log [2024-09-20 15:52:55,007][ ERROR][PID:1475012] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-09-20 15:53:05,020][WARNING][PID:1475012] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-09-20 15:53:15,028][ INFO][PID:1475012] Downloading the builder-live.log file, attempt 4 [2024-09-20 15:53:15,039][ INFO][PID:1475012] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.212.87.55 copr-rpmbuild-log [2024-09-20 15:54:15,111][ ERROR][PID:1475012] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-09-20 15:54:25,120][ ERROR][PID:1475012] Giving up for unstable SSH, failures: 1726847424.8526828, 1726847504.935465, 1726847585.0199947, 1726847665.1202242 [2024-09-20 15:54:25,122][ ERROR][PID:1475012] Re-try request for task on 'ResallocHost, ticket_id=4979618, hostname=18.212.87.55': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2024-09-20 15:54:25,124][ INFO][PID:1475012] Releasing VM back to pool [2024-09-20 15:54:25,150][ INFO][PID:1475012] Retry #2 (on other host) [2024-09-20 15:54:25,153][ INFO][PID:1475012] Trying to allocate VM [2024-09-20 16:00:43,527][ INFO][PID:1475012] Allocated host ResallocHost, ticket_id=4980103, hostname=44.200.227.202 [2024-09-20 16:00:43,530][ INFO][PID:1475012] Allocating ssh connection to builder [2024-09-20 16:00:43,530][ INFO][PID:1475012] Checking that builder machine is OK [2024-09-20 16:00:43,955][ INFO][PID:1475012] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.200.227.202 '/usr/bin/test -f /etc/mock/fedora-39-x86_64.cfg' [2024-09-20 16:00:44,223][ INFO][PID:1475012] Filling build.info file with builder info [2024-09-20 16:00:44,242][ INFO][PID:1475012] Not re-notifying FE and msg buses for the new host. [2024-09-20 16:00:44,244][ INFO][PID:1475012] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8043666-fedora-39-x86_64 --chroot fedora-39-x86_64 --detached [2024-09-20 16:00:44,701][ INFO][PID:1475012] Downloading the builder-live.log file, attempt 1 [2024-09-20 16:00:44,704][ INFO][PID:1475012] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.200.227.202 copr-rpmbuild-log [2024-09-20 16:00:49,720][ INFO][PID:1475012] Status: False [2024-09-20 16:02:20,679][ ERROR][PID:1475012] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-09-20 16:02:30,681][WARNING][PID:1475012] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-09-20 16:02:40,682][ INFO][PID:1475012] Downloading the builder-live.log file, attempt 2 [2024-09-20 16:02:40,687][ INFO][PID:1475012] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.200.227.202 copr-rpmbuild-log [2024-09-20 16:03:40,757][ ERROR][PID:1475012] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-09-20 16:03:50,774][WARNING][PID:1475012] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-09-20 16:04:00,776][ INFO][PID:1475012] Downloading the builder-live.log file, attempt 3 [2024-09-20 16:04:00,784][ INFO][PID:1475012] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.200.227.202 copr-rpmbuild-log [2024-09-20 16:05:00,851][ ERROR][PID:1475012] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-09-20 16:05:10,872][WARNING][PID:1475012] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-09-20 16:05:20,875][ INFO][PID:1475012] Downloading the builder-live.log file, attempt 4 [2024-09-20 16:05:20,883][ INFO][PID:1475012] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.200.227.202 copr-rpmbuild-log [2024-09-20 16:06:20,958][ ERROR][PID:1475012] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-09-20 16:06:31,016][ ERROR][PID:1475012] Giving up for unstable SSH, failures: 1726848150.6810367, 1726848230.7741385, 1726848310.872233, 1726848391.0160117 [2024-09-20 16:06:31,017][ ERROR][PID:1475012] Re-try request for task on 'ResallocHost, ticket_id=4980103, hostname=44.200.227.202': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2024-09-20 16:06:31,018][ INFO][PID:1475012] Releasing VM back to pool [2024-09-20 16:06:31,031][ ERROR][PID:1475012] Backend process error: Three host tried without success: {'44.202.212.4', '44.200.227.202', '18.212.87.55'} [2024-09-20 16:06:31,032][WARNING][PID:1475012] Switching not-finished job state to 'failed' [2024-09-20 16:06:31,033][ INFO][PID:1475012] Worker failed build, took 346.79023003578186 [2024-09-20 16:06:31,034][ INFO][PID:1475012] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@python", "project_name": "python3.12", "project_dirname": "python3.12", "submitter": "https://src.fedoraproject.org/user/kkeithle", "ended_on": 1726848391.0324311, "started_on": 1726848044.242201, "submitted_on": null, "status": 0, "chroot": "fedora-39-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "8043666-fedora-39-x86_64", "build_id": 8043666, "package_name": "ceph", "package_version": "2:19.2.0-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@python/python3.12/ceph", "git_hash": "476eaff500412c7ca7b1d76b31259d3bd64b8b2e", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@python/python3.12--https://src.fedoraproject.org/user/kkeithle", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "background": true, "bootstrap": "off", "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@python/python3.12/fedora-39-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/", "id": "http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch", "name": "Additional repo http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch" } ], "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@python/python3.12", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@python/python3.12", "result_dir": "08043666-ceph", "built_packages": "", "id": 8043666, "mockchain_macros": { "copr_username": "@python", "copr_projectname": "python3.12", "vendor": "Fedora Project COPR (@python/python3.12)" } } ] } [2024-09-20 16:06:31,102][ INFO][PID:1475012] Sending fedora-messaging bus message in build.end [2024-09-20 16:06:31,123][ INFO][PID:1475012] Compressing /var/lib/copr/public_html/results/@python/python3.12/fedora-39-x86_64/08043666-ceph/builder-live.log by gzip [2024-09-20 16:06:31,125][ INFO][PID:1475012] Running command 'gzip /var/lib/copr/public_html/results/@python/python3.12/fedora-39-x86_64/08043666-ceph/builder-live.log' as PID 1987988 [2024-09-20 16:06:31,173][ INFO][PID:1475012] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@python/python3.12/fedora-39-x86_64/08043666-ceph/builder-live.log) [2024-09-20 16:06:31,174][ INFO][PID:1475012] Compressing /var/lib/copr/public_html/results/@python/python3.12/fedora-39-x86_64/08043666-ceph/backend.log by gzip [2024-09-20 16:06:31,175][ INFO][PID:1475012] Running command 'gzip /var/lib/copr/public_html/results/@python/python3.12/fedora-39-x86_64/08043666-ceph/backend.log' as PID 1988012