[2024-12-05 18:28:42,958][ INFO][PID:3961602] Marking build as starting [2024-12-05 18:28:43,009][ INFO][PID:3961602] Trying to allocate VM [2024-12-05 18:34:01,165][ INFO][PID:3961602] Allocated host ResallocHost, ticket_id=21394, hostname=34.205.4.48 [2024-12-05 18:34:01,166][ INFO][PID:3961602] Allocating ssh connection to builder [2024-12-05 18:34:01,167][ INFO][PID:3961602] Checking that builder machine is OK [2024-12-05 18:34:01,521][ INFO][PID:3961602] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.205.4.48 '/usr/bin/test -f /etc/mock/fedora-39-x86_64.cfg' [2024-12-05 18:34:01,736][ INFO][PID:3961602] Filling build.info file with builder info [2024-12-05 18:34:01,738][ INFO][PID:3961602] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "medzik", "project_name": "jetbrains", "project_dirname": "jetbrains", "submitter": null, "ended_on": null, "started_on": 1733423641.738171, "submitted_on": null, "status": 3, "chroot": "fedora-39-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "8345326-fedora-39-x86_64", "build_id": 8345326, "package_name": "datagrip", "package_version": "2024.3.2-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/medzik/jetbrains/datagrip", "git_hash": "fa54882a060c528d37bf8a3569bdd04b819be4ce", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "medzik/jetbrains--ead5de13-64f2-4d3f-90d7-a598bd18ff0a", "results": null, "appstream": true, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/medzik/jetbrains/fedora-39-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/medzik/jetbrains", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/medzik/jetbrains", "result_dir": "08345326-datagrip", "built_packages": "", "id": 8345326, "mockchain_macros": { "copr_username": "medzik", "copr_projectname": "jetbrains", "vendor": "Fedora Project COPR (medzik/jetbrains)" } } ] } [2024-12-05 18:34:01,773][ INFO][PID:3961602] Sending fedora-messaging bus message in build.start [2024-12-05 18:34:02,463][ INFO][PID:3961602] Sending fedora-messaging bus message in chroot.start [2024-12-05 18:34:02,482][ INFO][PID:3961602] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8345326-fedora-39-x86_64 --chroot fedora-39-x86_64 --detached [2024-12-05 18:34:02,865][ INFO][PID:3961602] Downloading the builder-live.log file, attempt 1 [2024-12-05 18:34:02,866][ INFO][PID:3961602] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.205.4.48 copr-rpmbuild-log [2024-12-05 18:34:07,878][ INFO][PID:3961602] Periodic builder liveness probe: alive [2024-12-05 18:35:44,773][ ERROR][PID:3961602] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-12-05 18:35:54,774][WARNING][PID:3961602] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-12-05 18:36:04,775][ INFO][PID:3961602] Downloading the builder-live.log file, attempt 2 [2024-12-05 18:36:04,776][ INFO][PID:3961602] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.205.4.48 copr-rpmbuild-log [2024-12-05 18:37:04,844][ ERROR][PID:3961602] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-12-05 18:37:14,847][WARNING][PID:3961602] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-12-05 18:37:24,848][ INFO][PID:3961602] Downloading the builder-live.log file, attempt 3 [2024-12-05 18:37:24,849][ INFO][PID:3961602] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.205.4.48 copr-rpmbuild-log [2024-12-05 18:38:24,917][ ERROR][PID:3961602] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-12-05 18:38:34,923][WARNING][PID:3961602] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-12-05 18:38:44,924][ INFO][PID:3961602] Downloading the builder-live.log file, attempt 4 [2024-12-05 18:38:44,926][ INFO][PID:3961602] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.205.4.48 copr-rpmbuild-log [2024-12-05 18:39:45,010][ ERROR][PID:3961602] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-12-05 18:39:55,011][ ERROR][PID:3961602] Giving up for unstable SSH, failures: 1733423754.7743504, 1733423834.8472886, 1733423914.9239497, 1733423995.0111945 [2024-12-05 18:39:55,012][ ERROR][PID:3961602] Re-try request for task on 'ResallocHost, ticket_id=21394, hostname=34.205.4.48': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2024-12-05 18:39:55,013][ INFO][PID:3961602] Releasing VM back to pool [2024-12-05 18:39:55,033][ INFO][PID:3961602] Retry #1 (on other host) [2024-12-05 18:39:55,034][ INFO][PID:3961602] Trying to allocate VM [2024-12-05 18:48:13,228][ INFO][PID:3961602] Allocated host ResallocHost, ticket_id=21528, hostname=3.236.105.227 [2024-12-05 18:48:13,234][ INFO][PID:3961602] Allocating ssh connection to builder [2024-12-05 18:48:13,234][ INFO][PID:3961602] Checking that builder machine is OK [2024-12-05 18:48:13,587][ INFO][PID:3961602] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.105.227 '/usr/bin/test -f /etc/mock/fedora-39-x86_64.cfg' [2024-12-05 18:48:13,803][ INFO][PID:3961602] Filling build.info file with builder info [2024-12-05 18:48:13,807][ INFO][PID:3961602] Not re-notifying FE and msg buses for the new host. [2024-12-05 18:48:13,808][ INFO][PID:3961602] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8345326-fedora-39-x86_64 --chroot fedora-39-x86_64 --detached [2024-12-05 18:48:14,171][ INFO][PID:3961602] Downloading the builder-live.log file, attempt 1 [2024-12-05 18:48:14,172][ INFO][PID:3961602] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.105.227 copr-rpmbuild-log [2024-12-05 18:48:19,182][ INFO][PID:3961602] Periodic builder liveness probe: alive [2024-12-05 18:50:15,722][ ERROR][PID:3961602] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-12-05 18:50:25,723][WARNING][PID:3961602] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-12-05 18:50:35,724][ INFO][PID:3961602] Downloading the builder-live.log file, attempt 2 [2024-12-05 18:50:35,726][ INFO][PID:3961602] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.105.227 copr-rpmbuild-log [2024-12-05 18:51:35,794][ ERROR][PID:3961602] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-12-05 18:51:45,795][WARNING][PID:3961602] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-12-05 18:51:55,796][ INFO][PID:3961602] Downloading the builder-live.log file, attempt 3 [2024-12-05 18:51:55,797][ INFO][PID:3961602] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.105.227 copr-rpmbuild-log [2024-12-05 18:52:55,866][ ERROR][PID:3961602] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-12-05 18:53:05,878][WARNING][PID:3961602] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-12-05 18:53:15,879][ INFO][PID:3961602] Downloading the builder-live.log file, attempt 4 [2024-12-05 18:53:15,880][ INFO][PID:3961602] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.105.227 copr-rpmbuild-log [2024-12-05 18:54:15,951][ ERROR][PID:3961602] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-12-05 18:54:25,956][ ERROR][PID:3961602] Giving up for unstable SSH, failures: 1733424625.72389, 1733424705.7951703, 1733424785.878268, 1733424865.9563262 [2024-12-05 18:54:25,957][ ERROR][PID:3961602] Re-try request for task on 'ResallocHost, ticket_id=21528, hostname=3.236.105.227': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2024-12-05 18:54:25,957][ INFO][PID:3961602] Releasing VM back to pool [2024-12-05 18:54:25,974][ INFO][PID:3961602] Retry #2 (on other host) [2024-12-05 18:54:25,975][ INFO][PID:3961602] Trying to allocate VM [2024-12-05 19:02:44,177][ INFO][PID:3961602] Allocated host ResallocHost, ticket_id=21735, hostname=34.205.134.11 [2024-12-05 19:02:44,178][ INFO][PID:3961602] Allocating ssh connection to builder [2024-12-05 19:02:44,178][ INFO][PID:3961602] Checking that builder machine is OK [2024-12-05 19:02:44,550][ INFO][PID:3961602] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.205.134.11 '/usr/bin/test -f /etc/mock/fedora-39-x86_64.cfg' [2024-12-05 19:02:44,765][ INFO][PID:3961602] Filling build.info file with builder info [2024-12-05 19:02:44,769][ INFO][PID:3961602] Not re-notifying FE and msg buses for the new host. [2024-12-05 19:02:44,769][ INFO][PID:3961602] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8345326-fedora-39-x86_64 --chroot fedora-39-x86_64 --detached [2024-12-05 19:02:45,142][ INFO][PID:3961602] Downloading the builder-live.log file, attempt 1 [2024-12-05 19:02:45,144][ INFO][PID:3961602] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.205.134.11 copr-rpmbuild-log [2024-12-05 19:02:50,197][ INFO][PID:3961602] Periodic builder liveness probe: alive [2024-12-05 19:05:28,728][ ERROR][PID:3961602] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-12-05 19:05:38,729][WARNING][PID:3961602] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-12-05 19:05:48,730][ INFO][PID:3961602] Downloading the builder-live.log file, attempt 2 [2024-12-05 19:05:48,732][ INFO][PID:3961602] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.205.134.11 copr-rpmbuild-log [2024-12-05 19:06:48,810][ ERROR][PID:3961602] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-12-05 19:06:58,823][WARNING][PID:3961602] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-12-05 19:07:08,824][ INFO][PID:3961602] Downloading the builder-live.log file, attempt 3 [2024-12-05 19:07:08,826][ INFO][PID:3961602] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.205.134.11 copr-rpmbuild-log [2024-12-05 19:08:08,893][ ERROR][PID:3961602] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-12-05 19:08:18,898][WARNING][PID:3961602] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-12-05 19:08:28,898][ INFO][PID:3961602] Downloading the builder-live.log file, attempt 4 [2024-12-05 19:08:28,900][ INFO][PID:3961602] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.205.134.11 copr-rpmbuild-log [2024-12-05 19:09:28,969][ ERROR][PID:3961602] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-12-05 19:09:38,976][ ERROR][PID:3961602] Giving up for unstable SSH, failures: 1733425538.7292457, 1733425618.823185, 1733425698.898043, 1733425778.9764793 [2024-12-05 19:09:38,978][ ERROR][PID:3961602] Re-try request for task on 'ResallocHost, ticket_id=21735, hostname=34.205.134.11': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2024-12-05 19:09:38,979][ INFO][PID:3961602] Releasing VM back to pool [2024-12-05 19:09:39,003][ ERROR][PID:3961602] Backend process error: Three host tried without success: {'3.236.105.227', '34.205.4.48', '34.205.134.11'} [2024-12-05 19:09:39,006][WARNING][PID:3961602] Switching not-finished job state to 'failed' [2024-12-05 19:09:39,007][ INFO][PID:3961602] Worker failed build, took 414.2370603084564 [2024-12-05 19:09:39,007][ INFO][PID:3961602] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "medzik", "project_name": "jetbrains", "project_dirname": "jetbrains", "submitter": null, "ended_on": 1733425779.006173, "started_on": 1733425364.7691126, "submitted_on": null, "status": 0, "chroot": "fedora-39-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "8345326-fedora-39-x86_64", "build_id": 8345326, "package_name": "datagrip", "package_version": "2024.3.2-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/medzik/jetbrains/datagrip", "git_hash": "fa54882a060c528d37bf8a3569bdd04b819be4ce", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "medzik/jetbrains--ead5de13-64f2-4d3f-90d7-a598bd18ff0a", "results": null, "appstream": true, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/medzik/jetbrains/fedora-39-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/medzik/jetbrains", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/medzik/jetbrains", "result_dir": "08345326-datagrip", "built_packages": "", "id": 8345326, "mockchain_macros": { "copr_username": "medzik", "copr_projectname": "jetbrains", "vendor": "Fedora Project COPR (medzik/jetbrains)" } } ] } [2024-12-05 19:09:39,058][ INFO][PID:3961602] Sending fedora-messaging bus message in build.end [2024-12-05 19:09:39,091][ INFO][PID:3961602] Compressing /var/lib/copr/public_html/results/medzik/jetbrains/fedora-39-x86_64/08345326-datagrip/builder-live.log by gzip [2024-12-05 19:09:39,094][ INFO][PID:3961602] Running command 'gzip /var/lib/copr/public_html/results/medzik/jetbrains/fedora-39-x86_64/08345326-datagrip/builder-live.log' as PID 16465 [2024-12-05 19:09:39,096][ INFO][PID:3961602] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/medzik/jetbrains/fedora-39-x86_64/08345326-datagrip/builder-live.log) [2024-12-05 19:09:39,099][ INFO][PID:3961602] Compressing /var/lib/copr/public_html/results/medzik/jetbrains/fedora-39-x86_64/08345326-datagrip/backend.log by gzip [2024-12-05 19:09:39,104][ INFO][PID:3961602] Running command 'gzip /var/lib/copr/public_html/results/medzik/jetbrains/fedora-39-x86_64/08345326-datagrip/backend.log' as PID 16466