[2024-05-08 17:31:41,294][ INFO][PID:2147049] Marking build as starting [2024-05-08 17:31:41,424][ INFO][PID:2147049] Trying to allocate VM [2024-05-08 17:38:59,618][ INFO][PID:2147049] Allocated host ResallocHost, ticket_id=2705671, hostname=44.222.130.232 [2024-05-08 17:38:59,619][ INFO][PID:2147049] Allocating ssh connection to builder [2024-05-08 17:38:59,619][ INFO][PID:2147049] Checking that builder machine is OK [2024-05-08 17:38:59,947][ INFO][PID:2147049] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.222.130.232 '/usr/bin/test -f /etc/mock/fedora-39-x86_64.cfg' [2024-05-08 17:39:00,161][ INFO][PID:2147049] Filling build.info file with builder info [2024-05-08 17:39:00,163][ INFO][PID:2147049] Sending build state back to frontend: { "builds": [ { "timeout": 36000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@asahi", "project_name": "kernel", "project_dirname": "kernel", "submitter": "ngompa", "ended_on": null, "started_on": 1715189940.1627123, "submitted_on": null, "status": 3, "chroot": "fedora-39-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "7427087-fedora-39-x86_64", "build_id": 7427087, "package_name": "kernel", "package_version": "6.8.9-402.asahi", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@asahi/kernel/kernel", "git_hash": "82a221a3043efc2022d39f532e13535f183661ca", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": true, "sandbox": "@asahi/kernel--ngompa", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@asahi/kernel/fedora-39-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/@asahi/kernel/fedora-39-x86_64/devel/", "id": "copr_base_devel", "name": "Copr buildroot", "priority": null } ], "tags": [ "arch_x86_64", "on_demand_powerful" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@asahi/kernel", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@asahi/kernel", "result_dir": "07427087-kernel", "built_packages": "", "id": 7427087, "mockchain_macros": { "copr_username": "@asahi", "copr_projectname": "kernel", "vendor": "Fedora Project COPR (@asahi/kernel)" } } ] } [2024-05-08 17:39:00,261][ INFO][PID:2147049] Sending fedora-messaging bus message in build.start [2024-05-08 17:39:01,039][ INFO][PID:2147049] Sending fedora-messaging bus message in chroot.start [2024-05-08 17:39:01,069][ INFO][PID:2147049] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/7427087-fedora-39-x86_64 --chroot fedora-39-x86_64 --detached [2024-05-08 17:39:01,439][ INFO][PID:2147049] Downloading the builder-live.log file, attempt 1 [2024-05-08 17:39:01,441][ INFO][PID:2147049] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.222.130.232 copr-rpmbuild-log [2024-05-08 17:39:06,444][ INFO][PID:2147049] Status: False [2024-05-08 17:49:06,564][ INFO][PID:2147049] Status: False [2024-05-08 17:50:04,737][ ERROR][PID:2147049] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-05-08 17:50:14,738][WARNING][PID:2147049] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-05-08 17:50:24,739][ INFO][PID:2147049] Downloading the builder-live.log file, attempt 2 [2024-05-08 17:50:24,756][ INFO][PID:2147049] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.222.130.232 copr-rpmbuild-log [2024-05-08 17:51:24,826][ ERROR][PID:2147049] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-05-08 17:51:34,830][WARNING][PID:2147049] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-05-08 17:51:44,830][ INFO][PID:2147049] Downloading the builder-live.log file, attempt 3 [2024-05-08 17:51:44,833][ INFO][PID:2147049] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.222.130.232 copr-rpmbuild-log [2024-05-08 17:52:44,903][ ERROR][PID:2147049] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-05-08 17:52:54,906][WARNING][PID:2147049] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-05-08 17:53:04,907][ INFO][PID:2147049] Downloading the builder-live.log file, attempt 4 [2024-05-08 17:53:04,909][ INFO][PID:2147049] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.222.130.232 copr-rpmbuild-log [2024-05-08 17:54:04,977][ ERROR][PID:2147049] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-05-08 17:54:14,979][ ERROR][PID:2147049] Giving up for unstable SSH, failures: 1715190614.7386608, 1715190694.8300104, 1715190774.9064033, 1715190854.9791238 [2024-05-08 17:54:14,980][ ERROR][PID:2147049] Re-try request for task on 'ResallocHost, ticket_id=2705671, hostname=44.222.130.232': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2024-05-08 17:54:14,981][ INFO][PID:2147049] Releasing VM back to pool [2024-05-08 17:54:14,992][ INFO][PID:2147049] Retry #1 (on other host) [2024-05-08 17:54:14,992][ INFO][PID:2147049] Trying to allocate VM [2024-05-08 18:43:33,582][ INFO][PID:2147049] Allocated host ResallocHost, ticket_id=2706424, hostname=3.238.50.223 [2024-05-08 18:43:33,586][ INFO][PID:2147049] Allocating ssh connection to builder [2024-05-08 18:43:33,587][ INFO][PID:2147049] Checking that builder machine is OK [2024-05-08 18:43:33,937][ INFO][PID:2147049] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.50.223 '/usr/bin/test -f /etc/mock/fedora-39-x86_64.cfg' [2024-05-08 18:43:34,152][ INFO][PID:2147049] Filling build.info file with builder info [2024-05-08 18:43:34,157][ INFO][PID:2147049] Not re-notifying FE and msg buses for the new host. [2024-05-08 18:43:34,158][ INFO][PID:2147049] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/7427087-fedora-39-x86_64 --chroot fedora-39-x86_64 --detached [2024-05-08 18:43:34,536][ INFO][PID:2147049] Downloading the builder-live.log file, attempt 1 [2024-05-08 18:43:34,540][ INFO][PID:2147049] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.50.223 copr-rpmbuild-log [2024-05-08 18:43:39,544][ INFO][PID:2147049] Status: False [2024-05-08 18:53:39,655][ INFO][PID:2147049] Status: False [2024-05-08 19:03:39,728][ INFO][PID:2147049] Status: False [2024-05-08 19:06:08,491][ ERROR][PID:2147049] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-05-08 19:06:18,493][WARNING][PID:2147049] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-05-08 19:06:28,494][ INFO][PID:2147049] Downloading the builder-live.log file, attempt 2 [2024-05-08 19:06:28,529][ INFO][PID:2147049] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.50.223 copr-rpmbuild-log [2024-05-08 19:07:28,596][ ERROR][PID:2147049] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-05-08 19:07:38,600][WARNING][PID:2147049] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-05-08 19:07:48,601][ INFO][PID:2147049] Downloading the builder-live.log file, attempt 3 [2024-05-08 19:07:48,603][ INFO][PID:2147049] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.50.223 copr-rpmbuild-log [2024-05-08 19:08:48,618][ ERROR][PID:2147049] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-05-08 19:08:58,622][WARNING][PID:2147049] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-05-08 19:09:08,623][ INFO][PID:2147049] Downloading the builder-live.log file, attempt 4 [2024-05-08 19:09:08,624][ INFO][PID:2147049] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.50.223 copr-rpmbuild-log [2024-05-08 19:10:08,692][ ERROR][PID:2147049] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-05-08 19:10:18,699][ ERROR][PID:2147049] Giving up for unstable SSH, failures: 1715195178.4935608, 1715195258.6001873, 1715195338.6223278, 1715195418.6995883 [2024-05-08 19:10:18,700][ ERROR][PID:2147049] Re-try request for task on 'ResallocHost, ticket_id=2706424, hostname=3.238.50.223': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2024-05-08 19:10:18,701][ INFO][PID:2147049] Releasing VM back to pool [2024-05-08 19:10:18,711][ INFO][PID:2147049] Retry #2 (on other host) [2024-05-08 19:10:18,712][ INFO][PID:2147049] Trying to allocate VM [2024-05-08 19:13:36,823][ INFO][PID:2147049] Allocated host ResallocHost, ticket_id=2707496, hostname=3.227.239.54 [2024-05-08 19:13:36,824][ INFO][PID:2147049] Allocating ssh connection to builder [2024-05-08 19:13:36,824][ INFO][PID:2147049] Checking that builder machine is OK [2024-05-08 19:13:37,184][ INFO][PID:2147049] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.227.239.54 '/usr/bin/test -f /etc/mock/fedora-39-x86_64.cfg' [2024-05-08 19:13:37,400][ INFO][PID:2147049] Filling build.info file with builder info [2024-05-08 19:13:37,404][ INFO][PID:2147049] Not re-notifying FE and msg buses for the new host. [2024-05-08 19:13:37,405][ INFO][PID:2147049] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/7427087-fedora-39-x86_64 --chroot fedora-39-x86_64 --detached [2024-05-08 19:13:37,792][ INFO][PID:2147049] Downloading the builder-live.log file, attempt 1 [2024-05-08 19:13:37,794][ INFO][PID:2147049] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.227.239.54 copr-rpmbuild-log [2024-05-08 19:13:42,798][ INFO][PID:2147049] Status: False [2024-05-08 19:23:42,875][ INFO][PID:2147049] Status: False [2024-05-08 19:33:42,944][ INFO][PID:2147049] Status: False [2024-05-08 19:34:16,407][ ERROR][PID:2147049] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-05-08 19:34:26,408][WARNING][PID:2147049] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-05-08 19:34:36,409][ INFO][PID:2147049] Downloading the builder-live.log file, attempt 2 [2024-05-08 19:34:36,436][ INFO][PID:2147049] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.227.239.54 copr-rpmbuild-log [2024-05-08 19:35:36,503][ ERROR][PID:2147049] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-05-08 19:35:46,504][WARNING][PID:2147049] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-05-08 19:35:56,504][ INFO][PID:2147049] Downloading the builder-live.log file, attempt 3 [2024-05-08 19:35:56,506][ INFO][PID:2147049] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.227.239.54 copr-rpmbuild-log [2024-05-08 19:36:56,573][ ERROR][PID:2147049] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-05-08 19:37:06,575][WARNING][PID:2147049] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-05-08 19:37:16,576][ INFO][PID:2147049] Downloading the builder-live.log file, attempt 4 [2024-05-08 19:37:16,577][ INFO][PID:2147049] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.227.239.54 copr-rpmbuild-log [2024-05-08 19:38:16,644][ ERROR][PID:2147049] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-05-08 19:38:26,645][ ERROR][PID:2147049] Giving up for unstable SSH, failures: 1715196866.4085555, 1715196946.504057, 1715197026.5753477, 1715197106.6450367 [2024-05-08 19:38:26,645][ ERROR][PID:2147049] Re-try request for task on 'ResallocHost, ticket_id=2707496, hostname=3.227.239.54': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2024-05-08 19:38:26,646][ INFO][PID:2147049] Releasing VM back to pool [2024-05-08 19:38:26,653][ ERROR][PID:2147049] Backend process error: Three host tried without success: {'3.227.239.54', '3.238.50.223', '44.222.130.232'} [2024-05-08 19:38:26,654][WARNING][PID:2147049] Switching not-finished job state to 'failed' [2024-05-08 19:38:26,654][ INFO][PID:2147049] Worker failed build, took 1489.249696969986 [2024-05-08 19:38:26,655][ INFO][PID:2147049] Sending build state back to frontend: { "builds": [ { "timeout": 36000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@asahi", "project_name": "kernel", "project_dirname": "kernel", "submitter": "ngompa", "ended_on": 1715197106.6543279, "started_on": 1715195617.404631, "submitted_on": null, "status": 0, "chroot": "fedora-39-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "7427087-fedora-39-x86_64", "build_id": 7427087, "package_name": "kernel", "package_version": "6.8.9-402.asahi", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@asahi/kernel/kernel", "git_hash": "82a221a3043efc2022d39f532e13535f183661ca", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": true, "sandbox": "@asahi/kernel--ngompa", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@asahi/kernel/fedora-39-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/@asahi/kernel/fedora-39-x86_64/devel/", "id": "copr_base_devel", "name": "Copr buildroot", "priority": null } ], "tags": [ "arch_x86_64", "on_demand_powerful" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@asahi/kernel", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@asahi/kernel", "result_dir": "07427087-kernel", "built_packages": "", "id": 7427087, "mockchain_macros": { "copr_username": "@asahi", "copr_projectname": "kernel", "vendor": "Fedora Project COPR (@asahi/kernel)" } } ] } [2024-05-08 19:38:26,736][ INFO][PID:2147049] Sending fedora-messaging bus message in build.end [2024-05-08 19:38:26,765][ INFO][PID:2147049] Compressing /var/lib/copr/public_html/results/@asahi/kernel/fedora-39-x86_64/07427087-kernel/builder-live.log by gzip [2024-05-08 19:38:26,766][ INFO][PID:2147049] Running command 'gzip /var/lib/copr/public_html/results/@asahi/kernel/fedora-39-x86_64/07427087-kernel/builder-live.log' as PID 2686847 [2024-05-08 19:38:26,767][ INFO][PID:2147049] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@asahi/kernel/fedora-39-x86_64/07427087-kernel/builder-live.log) [2024-05-08 19:38:26,768][ INFO][PID:2147049] Compressing /var/lib/copr/public_html/results/@asahi/kernel/fedora-39-x86_64/07427087-kernel/backend.log by gzip [2024-05-08 19:38:26,769][ INFO][PID:2147049] Running command 'gzip /var/lib/copr/public_html/results/@asahi/kernel/fedora-39-x86_64/07427087-kernel/backend.log' as PID 2686848