[2025-02-27 15:25:54,303][ INFO][PID:2919732] Marking build as starting
[2025-02-27 15:25:54,355][ INFO][PID:2919732] Trying to allocate VM
[2025-02-27 15:32:12,709][ INFO][PID:2919732] Allocated host ResallocHost, ticket_id=1541604, hostname=3.231.33.53, name=aws_x86_64_spot_prod_01532444_20250227_153029
[2025-02-27 15:32:12,711][ INFO][PID:2919732] Allocating ssh connection to builder
[2025-02-27 15:32:12,711][ INFO][PID:2919732] Checking that builder machine is OK
[2025-02-27 15:32:13,074][ INFO][PID:2919732] Running remote command: copr-builder-ready fedora-rawhide-i386
[2025-02-27 15:32:13,294][ INFO][PID:2919732] Red Hat subscription not needed for fedora-rawhide-i386
Builder is ready to be used
[2025-02-27 15:32:13,295][ INFO][PID:2919732] Filling build.info file with builder info
[2025-02-27 15:32:13,296][ INFO][PID:2919732] Sending build state back to frontend: {
"builds": [
{
"timeout": 18000,
"frontend_base_url": "https://copr.fedorainfracloud.org",
"memory_reqs": 2048,
"enable_net": false,
"project_owner": "decathorpe",
"project_name": "sequoia-openpgp-v2",
"project_dirname": "sequoia-openpgp-v2",
"submitter": "decathorpe",
"ended_on": null,
"started_on": 1740670333.2963998,
"submitted_on": null,
"status": 3,
"chroot": "fedora-rawhide-i386",
"arch": "i386",
"buildroot_pkgs": [],
"task_id": "8706746-fedora-rawhide-i386",
"build_id": 8706746,
"package_name": "rust-sequoia-openpgp1",
"package_version": "1.22.0-1",
"git_repo": "https://copr-dist-git.fedorainfracloud.org/git/decathorpe/sequoia-openpgp-v2/rust-sequoia-openpgp1",
"git_hash": "b4ec7c714767919f4970989fbe289b5488d89359",
"git_branch": null,
"source_type": null,
"source_json": null,
"pkg_name": null,
"pkg_main_version": null,
"pkg_epoch": null,
"pkg_release": null,
"srpm_url": null,
"uses_devel_repo": false,
"sandbox": "decathorpe/sequoia-openpgp-v2--decathorpe",
"results": null,
"appstream": false,
"allow_user_ssh": false,
"ssh_public_keys": null,
"storage": 0,
"background": false,
"fedora_review": false,
"isolation": "default",
"repo_priority": null,
"repos": [
{
"baseurl": "https://download.copr.fedorainfracloud.org/results/decathorpe/sequoia-openpgp-v2/fedora-rawhide-i386/",
"id": "copr_base",
"name": "Copr repository",
"priority": null
}
],
"tags": [
"arch_i386"
],
"with_opts": [],
"without_opts": [],
"destdir": "/var/lib/copr/public_html/results/decathorpe/sequoia-openpgp-v2",
"results_repo_url": "https://download.copr.fedorainfracloud.org/results/decathorpe/sequoia-openpgp-v2",
"result_dir": "08706746-rust-sequoia-openpgp1",
"built_packages": "",
"id": 8706746,
"mockchain_macros": {
"copr_username": "decathorpe",
"copr_projectname": "sequoia-openpgp-v2",
"vendor": "Fedora Project COPR (decathorpe/sequoia-openpgp-v2)"
}
}
]
}
[2025-02-27 15:32:13,337][ INFO][PID:2919732] Sending fedora-messaging bus message in build.start
[2025-02-27 15:32:14,013][ INFO][PID:2919732] Sending fedora-messaging bus message in chroot.start
[2025-02-27 15:32:14,035][ INFO][PID:2919732] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8706746-fedora-rawhide-i386 --chroot fedora-rawhide-i386 --detached
[2025-02-27 15:32:14,393][ INFO][PID:2919732] Downloading the builder-live.log file, attempt 1
[2025-02-27 15:32:14,397][ INFO][PID:2919732] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.231.33.53 copr-rpmbuild-log
[2025-02-27 15:32:19,409][ INFO][PID:2919732] Periodic builder liveness probe: alive
[2025-02-27 15:34:37,512][ ERROR][PID:2919732] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-02-27 15:34:47,513][WARNING][PID:2919732] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-02-27 15:34:57,514][ INFO][PID:2919732] Downloading the builder-live.log file, attempt 2
[2025-02-27 15:34:57,516][ INFO][PID:2919732] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.231.33.53 copr-rpmbuild-log
[2025-02-27 15:35:57,576][ ERROR][PID:2919732] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-02-27 15:36:07,578][WARNING][PID:2919732] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-02-27 15:36:17,579][ INFO][PID:2919732] Downloading the builder-live.log file, attempt 3
[2025-02-27 15:36:17,580][ INFO][PID:2919732] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.231.33.53 copr-rpmbuild-log
[2025-02-27 15:37:17,648][ ERROR][PID:2919732] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-02-27 15:37:27,654][WARNING][PID:2919732] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-02-27 15:37:37,654][ INFO][PID:2919732] Downloading the builder-live.log file, attempt 4
[2025-02-27 15:37:37,656][ INFO][PID:2919732] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.231.33.53 copr-rpmbuild-log
[2025-02-27 15:38:37,723][ ERROR][PID:2919732] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-02-27 15:38:47,734][ ERROR][PID:2919732] Giving up for unstable SSH, failures: 1740670487.5134692, 1740670567.5783966, 1740670647.6540139, 1740670727.7342987
[2025-02-27 15:38:47,735][ ERROR][PID:2919732] Re-try request for task on 'ResallocHost, ticket_id=1541604, hostname=3.231.33.53, name=aws_x86_64_spot_prod_01532444_20250227_153029': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-02-27 15:38:47,735][ INFO][PID:2919732] Releasing VM back to pool
[2025-02-27 15:38:47,745][ INFO][PID:2919732] Retry #1 (on other host)
[2025-02-27 15:38:47,746][ INFO][PID:2919732] Trying to allocate VM
[2025-02-27 15:46:05,964][ INFO][PID:2919732] Allocated host ResallocHost, ticket_id=1541811, hostname=54.210.78.218, name=aws_x86_64_spot_prod_01533141_20250227_154341
[2025-02-27 15:46:05,964][ INFO][PID:2919732] Allocating ssh connection to builder
[2025-02-27 15:46:05,966][ INFO][PID:2919732] Checking that builder machine is OK
[2025-02-27 15:46:06,324][ INFO][PID:2919732] Running remote command: copr-builder-ready fedora-rawhide-i386
[2025-02-27 15:46:06,548][ INFO][PID:2919732] Red Hat subscription not needed for fedora-rawhide-i386
Builder is ready to be used
[2025-02-27 15:46:06,549][ INFO][PID:2919732] Filling build.info file with builder info
[2025-02-27 15:46:06,552][ INFO][PID:2919732] Not re-notifying FE and msg buses for the new host.
[2025-02-27 15:46:06,552][ INFO][PID:2919732] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8706746-fedora-rawhide-i386 --chroot fedora-rawhide-i386 --detached
[2025-02-27 15:46:06,951][ INFO][PID:2919732] Downloading the builder-live.log file, attempt 1
[2025-02-27 15:46:06,953][ INFO][PID:2919732] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.210.78.218 copr-rpmbuild-log
[2025-02-27 15:46:11,963][ INFO][PID:2919732] Periodic builder liveness probe: alive
[2025-02-27 15:56:12,095][ INFO][PID:2919732] Periodic builder liveness probe: alive
[2025-02-27 16:04:28,736][ ERROR][PID:2919732] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-02-27 16:04:38,740][WARNING][PID:2919732] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-02-27 16:04:48,741][ INFO][PID:2919732] Downloading the builder-live.log file, attempt 2
[2025-02-27 16:04:48,743][ INFO][PID:2919732] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.210.78.218 copr-rpmbuild-log
[2025-02-27 16:04:48,753][ ERROR][PID:2919732] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-02-27 16:04:58,754][WARNING][PID:2919732] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-02-27 16:05:08,755][ INFO][PID:2919732] Downloading the builder-live.log file, attempt 3
[2025-02-27 16:05:08,763][ INFO][PID:2919732] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.210.78.218 copr-rpmbuild-log
[2025-02-27 16:05:08,777][ ERROR][PID:2919732] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-02-27 16:05:18,786][WARNING][PID:2919732] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-02-27 16:05:28,788][ INFO][PID:2919732] Downloading the builder-live.log file, attempt 4
[2025-02-27 16:05:28,790][ INFO][PID:2919732] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.210.78.218 copr-rpmbuild-log
[2025-02-27 16:06:12,251][ INFO][PID:2919732] Periodic builder liveness probe: alive
[2025-02-27 16:06:28,861][ ERROR][PID:2919732] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-02-27 16:06:38,862][ ERROR][PID:2919732] Giving up for unstable SSH, failures: 1740672278.740864, 1740672298.7544374, 1740672318.7860527, 1740672398.8625407
[2025-02-27 16:06:38,864][ ERROR][PID:2919732] Re-try request for task on 'ResallocHost, ticket_id=1541811, hostname=54.210.78.218, name=aws_x86_64_spot_prod_01533141_20250227_154341': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-02-27 16:06:38,865][ INFO][PID:2919732] Releasing VM back to pool
[2025-02-27 16:06:38,879][ INFO][PID:2919732] Retry #2 (on other host)
[2025-02-27 16:06:38,881][ INFO][PID:2919732] Trying to allocate VM
[2025-02-27 16:07:06,982][ INFO][PID:2919732] Allocated host ResallocHost, ticket_id=1542403, hostname=54.210.78.218, name=aws_x86_64_spot_prod_01533141_20250227_154341
[2025-02-27 16:07:06,982][ INFO][PID:2919732] Allocating ssh connection to builder
[2025-02-27 16:07:06,983][ INFO][PID:2919732] Checking that builder machine is OK
[2025-02-27 16:08:07,059][ ERROR][PID:2919732] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:
ERR:
ssh: connect to host 54.210.78.218 port 22: Connection timed out
[2025-02-27 16:08:17,061][ ERROR][PID:2919732] Re-try request for task on 'ResallocHost, ticket_id=1542403, hostname=54.210.78.218, name=aws_x86_64_spot_prod_01533141_20250227_154341': Unable to finish after 1 SSH attempts
[2025-02-27 16:08:17,061][ INFO][PID:2919732] Releasing VM back to pool
[2025-02-27 16:08:17,103][ INFO][PID:2919732] Retry #3 (on other host)
[2025-02-27 16:08:17,105][ INFO][PID:2919732] Trying to allocate VM
[2025-02-27 16:14:35,373][ INFO][PID:2919732] Allocated host ResallocHost, ticket_id=1542473, hostname=107.22.157.92, name=aws_x86_64_spot_prod_01534146_20250227_161256
[2025-02-27 16:14:35,374][ INFO][PID:2919732] Allocating ssh connection to builder
[2025-02-27 16:14:35,375][ INFO][PID:2919732] Checking that builder machine is OK
[2025-02-27 16:14:35,702][ INFO][PID:2919732] Running remote command: copr-builder-ready fedora-rawhide-i386
[2025-02-27 16:14:35,913][ INFO][PID:2919732] Red Hat subscription not needed for fedora-rawhide-i386
Builder is ready to be used
[2025-02-27 16:14:35,913][ INFO][PID:2919732] Filling build.info file with builder info
[2025-02-27 16:14:35,918][ INFO][PID:2919732] Not re-notifying FE and msg buses for the new host.
[2025-02-27 16:14:35,918][ INFO][PID:2919732] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8706746-fedora-rawhide-i386 --chroot fedora-rawhide-i386 --detached
[2025-02-27 16:14:36,269][ INFO][PID:2919732] Downloading the builder-live.log file, attempt 1
[2025-02-27 16:14:36,271][ INFO][PID:2919732] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@107.22.157.92 copr-rpmbuild-log
[2025-02-27 16:16:16,305][ INFO][PID:2919732] Periodic builder liveness probe: alive
[2025-02-27 16:26:16,501][ INFO][PID:2919732] Periodic builder liveness probe: alive
[2025-02-27 16:33:32,881][ ERROR][PID:2919732] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-02-27 16:33:42,882][WARNING][PID:2919732] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-02-27 16:33:52,883][ INFO][PID:2919732] Downloading the builder-live.log file, attempt 2
[2025-02-27 16:33:52,885][ INFO][PID:2919732] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@107.22.157.92 copr-rpmbuild-log
[2025-02-27 16:34:52,954][ ERROR][PID:2919732] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-02-27 16:35:02,958][WARNING][PID:2919732] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-02-27 16:35:12,959][ INFO][PID:2919732] Downloading the builder-live.log file, attempt 3
[2025-02-27 16:35:12,960][ INFO][PID:2919732] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@107.22.157.92 copr-rpmbuild-log
[2025-02-27 16:36:13,029][ ERROR][PID:2919732] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-02-27 16:36:16,678][ INFO][PID:2919732] Periodic builder liveness probe: alive
[2025-02-27 16:36:23,030][WARNING][PID:2919732] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-02-27 16:36:33,030][ INFO][PID:2919732] Downloading the builder-live.log file, attempt 4
[2025-02-27 16:36:33,032][ INFO][PID:2919732] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@107.22.157.92 copr-rpmbuild-log
[2025-02-27 16:37:33,099][ ERROR][PID:2919732] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-02-27 16:37:43,102][ ERROR][PID:2919732] Giving up for unstable SSH, failures: 1740674022.8824158, 1740674102.9585907, 1740674183.0301898, 1740674263.102159
[2025-02-27 16:37:43,103][ ERROR][PID:2919732] Re-try request for task on 'ResallocHost, ticket_id=1542473, hostname=107.22.157.92, name=aws_x86_64_spot_prod_01534146_20250227_161256': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-02-27 16:37:43,103][ INFO][PID:2919732] Releasing VM back to pool
[2025-02-27 16:37:43,112][ ERROR][PID:2919732] Backend process error: Three host tried without success: {'107.22.157.92', '54.210.78.218', '3.231.33.53'}
[2025-02-27 16:37:43,113][WARNING][PID:2919732] Switching not-finished job state to 'failed'
[2025-02-27 16:37:43,113][ INFO][PID:2919732] Worker failed build, took 1387.1946952342987
[2025-02-27 16:37:43,114][ INFO][PID:2919732] Sending build state back to frontend: {
"builds": [
{
"timeout": 18000,
"frontend_base_url": "https://copr.fedorainfracloud.org",
"memory_reqs": 2048,
"enable_net": false,
"project_owner": "decathorpe",
"project_name": "sequoia-openpgp-v2",
"project_dirname": "sequoia-openpgp-v2",
"submitter": "decathorpe",
"ended_on": 1740674263.1130085,
"started_on": 1740672875.9183133,
"submitted_on": null,
"status": 0,
"chroot": "fedora-rawhide-i386",
"arch": "i386",
"buildroot_pkgs": [],
"task_id": "8706746-fedora-rawhide-i386",
"build_id": 8706746,
"package_name": "rust-sequoia-openpgp1",
"package_version": "1.22.0-1",
"git_repo": "https://copr-dist-git.fedorainfracloud.org/git/decathorpe/sequoia-openpgp-v2/rust-sequoia-openpgp1",
"git_hash": "b4ec7c714767919f4970989fbe289b5488d89359",
"git_branch": null,
"source_type": null,
"source_json": null,
"pkg_name": null,
"pkg_main_version": null,
"pkg_epoch": null,
"pkg_release": null,
"srpm_url": null,
"uses_devel_repo": false,
"sandbox": "decathorpe/sequoia-openpgp-v2--decathorpe",
"results": null,
"appstream": false,
"allow_user_ssh": false,
"ssh_public_keys": null,
"storage": 0,
"background": false,
"fedora_review": false,
"isolation": "default",
"repo_priority": null,
"repos": [
{
"baseurl": "https://download.copr.fedorainfracloud.org/results/decathorpe/sequoia-openpgp-v2/fedora-rawhide-i386/",
"id": "copr_base",
"name": "Copr repository",
"priority": null
}
],
"tags": [
"arch_i386"
],
"with_opts": [],
"without_opts": [],
"destdir": "/var/lib/copr/public_html/results/decathorpe/sequoia-openpgp-v2",
"results_repo_url": "https://download.copr.fedorainfracloud.org/results/decathorpe/sequoia-openpgp-v2",
"result_dir": "08706746-rust-sequoia-openpgp1",
"built_packages": "",
"id": 8706746,
"mockchain_macros": {
"copr_username": "decathorpe",
"copr_projectname": "sequoia-openpgp-v2",
"vendor": "Fedora Project COPR (decathorpe/sequoia-openpgp-v2)"
}
}
]
}
[2025-02-27 16:37:43,152][ INFO][PID:2919732] Sending fedora-messaging bus message in build.end
[2025-02-27 16:37:43,169][ INFO][PID:2919732] Compressing /var/lib/copr/public_html/results/decathorpe/sequoia-openpgp-v2/fedora-rawhide-i386/08706746-rust-sequoia-openpgp1/builder-live.log by gzip
[2025-02-27 16:37:43,170][ INFO][PID:2919732] Running command 'gzip /var/lib/copr/public_html/results/decathorpe/sequoia-openpgp-v2/fedora-rawhide-i386/08706746-rust-sequoia-openpgp1/builder-live.log' as PID 3576025
[2025-02-27 16:37:43,171][ INFO][PID:2919732] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/decathorpe/sequoia-openpgp-v2/fedora-rawhide-i386/08706746-rust-sequoia-openpgp1/builder-live.log)
[2025-02-27 16:37:43,172][ INFO][PID:2919732] Compressing /var/lib/copr/public_html/results/decathorpe/sequoia-openpgp-v2/fedora-rawhide-i386/08706746-rust-sequoia-openpgp1/backend.log by gzip
[2025-02-27 16:37:43,173][ INFO][PID:2919732] Running command 'gzip /var/lib/copr/public_html/results/decathorpe/sequoia-openpgp-v2/fedora-rawhide-i386/08706746-rust-sequoia-openpgp1/backend.log' as PID 3576026