[2025-12-12 13:49:10,568][  INFO][PID:424214] Marking build as starting
[2025-12-12 13:49:10,664][  INFO][PID:424214] Checking that /var/lib/copr/public_html/results/jwakely/boost-190-rebuild.checker/fedora-rawhide-x86_64/repodata/repomd.xml exists
[2025-12-12 13:49:10,666][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:49:10,668][  INFO][PID:424214] VM allocation process starts
[2025-12-12 13:49:10,697][  INFO][PID:424214] Trying to allocate VM: ResallocHost, ticket_id=304165, requested_tags=['copr_builder', 'arch_x86_64']
[2025-12-12 13:49:15,704][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:49:16,757][  INFO][PID:424214] Allocated host ResallocHost, ticket_id=304165, hostname=107.22.149.166, name=aws_x86_64_spot_prod_00496851_20251212_134733, requested_tags=['copr_builder', 'arch_x86_64']
[2025-12-12 13:49:16,758][  INFO][PID:424214] Allocating ssh connection to builder
[2025-12-12 13:49:16,759][  INFO][PID:424214] Checking that builder machine is OK
[2025-12-12 13:49:17,096][  INFO][PID:424214] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2025-12-12 13:49:17,332][  INFO][PID:424214] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2025-12-12 13:49:17,333][  INFO][PID:424214] Filling build.info file with builder info
[2025-12-12 13:49:17,334][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:49:17,336][  INFO][PID:424214] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 115200,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "jwakely",
            "project_name": "boost-190-rebuild.checker",
            "project_dirname": "boost-190-rebuild.checker",
            "submitter": "jwakely",
            "ended_on": null,
            "started_on": 1765547357.336538,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "9902737-fedora-rawhide-x86_64",
            "build_id": 9902737,
            "package_name": "SimGear",
            "package_version": "2024.1.3-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/jwakely/boost-190-rebuild.checker/SimGear",
            "git_hash": "39a3f5962b1c6cc1096ee4d09b92543d5e60c300",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "jwakely/boost-190-rebuild.checker--jwakely",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/jwakely/boost-190-rebuild.checker/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/jwakely/boost-190-rebuild.checker",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/jwakely/boost-190-rebuild.checker",
            "result_dir": "09902737-SimGear",
            "built_packages": "",
            "id": 9902737,
            "mockchain_macros": {
                "copr_username": "jwakely",
                "copr_projectname": "boost-190-rebuild.checker",
                "vendor": "Fedora Project COPR (jwakely/boost-190-rebuild.checker)"
            }
        }
    ]
}
[2025-12-12 13:49:17,439][  INFO][PID:424214] Sending fedora-messaging bus message in build.start
[2025-12-12 13:49:19,243][  INFO][PID:424214] Sending fedora-messaging bus message in chroot.start
[2025-12-12 13:49:19,298][  INFO][PID:424214] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9902737-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2025-12-12 13:49:19,678][  INFO][PID:424214] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '107.22.149.166' (ED25519) to the list of known hosts.


[2025-12-12 13:49:19,681][  INFO][PID:424214] Downloading the builder-live.log file, attempt 1
[2025-12-12 13:49:19,685][  INFO][PID:424214] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@107.22.149.166 copr-rpmbuild-log
[2025-12-12 13:49:24,693][  INFO][PID:424214] Periodic builder liveness probe: alive
[2025-12-12 13:49:24,694][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:49:29,696][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:49:34,704][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:49:39,713][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:49:44,718][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:49:49,721][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:49:54,722][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:49:59,724][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:50:04,727][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:50:09,729][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:50:14,730][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:50:19,731][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:50:24,732][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:50:29,736][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:50:34,741][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:50:39,742][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:50:44,744][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:50:49,745][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:50:54,748][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:50:59,752][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:51:04,758][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:51:09,761][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:51:14,762][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:51:19,765][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:51:24,766][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:51:29,770][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:51:34,773][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:51:39,776][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:51:44,777][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:51:49,782][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:51:54,784][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:51:59,786][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:52:04,788][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:52:09,791][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:52:14,792][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:52:19,794][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:52:24,796][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:52:29,798][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:52:30,054][ ERROR][PID:424214] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-12 13:52:34,800][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:52:39,809][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:52:40,065][WARNING][PID:424214] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-12 13:52:44,813][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:52:49,815][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:52:50,067][  INFO][PID:424214] Downloading the builder-live.log file, attempt 2
[2025-12-12 13:52:50,072][  INFO][PID:424214] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@107.22.149.166 copr-rpmbuild-log
[2025-12-12 13:52:54,817][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:52:59,821][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:53:04,823][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:53:09,824][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:53:14,825][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:53:19,826][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:53:24,828][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:53:29,830][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:53:34,833][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:53:39,834][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:53:44,836][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:53:49,837][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:53:50,158][ ERROR][PID:424214] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-12 13:53:54,838][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:53:59,840][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:54:00,163][WARNING][PID:424214] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-12 13:54:04,842][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:54:09,844][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:54:10,165][  INFO][PID:424214] Downloading the builder-live.log file, attempt 3
[2025-12-12 13:54:10,168][  INFO][PID:424214] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@107.22.149.166 copr-rpmbuild-log
[2025-12-12 13:54:14,845][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:54:19,847][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:54:24,848][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:54:29,850][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:54:34,851][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:54:39,852][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:54:44,853][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:54:49,854][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:54:54,856][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:54:59,857][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:55:04,858][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:55:09,859][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:55:10,237][ ERROR][PID:424214] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-12 13:55:14,861][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:55:19,863][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:55:20,241][WARNING][PID:424214] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-12 13:55:24,864][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:55:29,865][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:55:30,241][  INFO][PID:424214] Downloading the builder-live.log file, attempt 4
[2025-12-12 13:55:30,243][  INFO][PID:424214] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@107.22.149.166 copr-rpmbuild-log
[2025-12-12 13:55:34,866][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:55:39,868][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:55:44,869][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:55:49,870][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:55:54,871][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:55:59,872][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:56:04,873][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:56:09,874][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:56:14,876][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:56:19,877][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:56:24,878][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:56:29,879][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:56:30,311][ ERROR][PID:424214] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-12 13:56:34,880][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:56:39,881][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:56:40,319][ ERROR][PID:424214] Giving up for unstable SSH, failures: 1765547560.065724, 1765547640.163848, 1765547720.241243, 1765547800.3197029
[2025-12-12 13:56:40,320][ ERROR][PID:424214] Re-try request for task on 'ResallocHost, ticket_id=304165, hostname=107.22.149.166, name=aws_x86_64_spot_prod_00496851_20251212_134733, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-12-12 13:56:40,321][  INFO][PID:424214] Releasing VM back to pool
[2025-12-12 13:56:40,331][  INFO][PID:424214] Retry #1 (on other host)
[2025-12-12 13:56:40,331][  INFO][PID:424214] Checking that /var/lib/copr/public_html/results/jwakely/boost-190-rebuild.checker/fedora-rawhide-x86_64/repodata/repomd.xml exists
[2025-12-12 13:56:40,332][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:56:40,332][  INFO][PID:424214] VM allocation process starts
[2025-12-12 13:56:40,343][  INFO][PID:424214] Trying to allocate VM: ResallocHost, ticket_id=304341, requested_tags=['copr_builder', 'arch_x86_64']
[2025-12-12 13:56:43,358][  INFO][PID:424214] Allocated host ResallocHost, ticket_id=304341, hostname=44.223.87.13, name=aws_x86_64_spot_prod_00497212_20251212_135255, requested_tags=['copr_builder', 'arch_x86_64']
[2025-12-12 13:56:43,359][  INFO][PID:424214] Allocating ssh connection to builder
[2025-12-12 13:56:43,360][  INFO][PID:424214] Checking that builder machine is OK
[2025-12-12 13:56:43,661][  INFO][PID:424214] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2025-12-12 13:56:43,848][  INFO][PID:424214] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2025-12-12 13:56:43,848][  INFO][PID:424214] Filling build.info file with builder info
[2025-12-12 13:56:43,851][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:56:43,852][  INFO][PID:424214] Not re-notifying FE and msg buses for the new host.
[2025-12-12 13:56:43,853][  INFO][PID:424214] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9902737-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2025-12-12 13:56:44,156][  INFO][PID:424214] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '44.223.87.13' (ED25519) to the list of known hosts.


[2025-12-12 13:56:44,157][  INFO][PID:424214] Downloading the builder-live.log file, attempt 1
[2025-12-12 13:56:44,158][  INFO][PID:424214] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.223.87.13 copr-rpmbuild-log
[2025-12-12 13:56:44,841][ ERROR][PID:424214] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-12 13:56:49,157][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:56:54,160][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:56:54,843][WARNING][PID:424214] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-12 13:56:59,162][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:57:04,163][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:57:04,844][  INFO][PID:424214] Downloading the builder-live.log file, attempt 2
[2025-12-12 13:57:04,845][  INFO][PID:424214] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.223.87.13 copr-rpmbuild-log
[2025-12-12 13:57:09,164][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:57:14,165][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:57:19,166][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:57:24,167][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:57:29,168][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:57:34,169][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:57:39,170][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:57:44,171][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:57:49,172][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:57:54,173][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:57:59,174][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:58:04,175][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:58:04,895][ ERROR][PID:424214] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-12 13:58:09,176][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:58:14,177][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:58:14,900][WARNING][PID:424214] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-12 13:58:19,178][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:58:24,179][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:58:24,900][  INFO][PID:424214] Downloading the builder-live.log file, attempt 3
[2025-12-12 13:58:24,902][  INFO][PID:424214] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.223.87.13 copr-rpmbuild-log
[2025-12-12 13:58:29,180][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:58:34,181][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:58:39,182][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:58:44,183][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:58:49,184][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:58:54,185][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:58:59,186][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:59:04,187][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:59:09,189][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:59:14,190][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:59:19,191][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:59:24,192][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:59:24,969][ ERROR][PID:424214] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-12 13:59:29,200][  INFO][PID:424214] Periodic builder liveness probe: alive
[2025-12-12 13:59:29,201][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:59:34,201][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:59:34,972][WARNING][PID:424214] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-12 13:59:39,203][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:59:44,204][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:59:44,973][  INFO][PID:424214] Downloading the builder-live.log file, attempt 4
[2025-12-12 13:59:44,974][  INFO][PID:424214] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.223.87.13 copr-rpmbuild-log
[2025-12-12 13:59:49,205][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:59:54,207][  INFO][PID:424214] Checking for cancel request
[2025-12-12 13:59:59,208][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:00:04,209][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:00:09,210][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:00:14,211][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:00:19,212][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:00:24,213][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:00:29,214][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:00:34,215][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:00:39,216][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:00:44,217][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:00:45,042][ ERROR][PID:424214] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-12 14:00:49,218][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:00:54,219][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:00:55,047][ ERROR][PID:424214] Giving up for unstable SSH, failures: 1765547814.8431592, 1765547894.9000921, 1765547974.9728553, 1765548055.046916
[2025-12-12 14:00:55,047][ ERROR][PID:424214] Re-try request for task on 'ResallocHost, ticket_id=304341, hostname=44.223.87.13, name=aws_x86_64_spot_prod_00497212_20251212_135255, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-12-12 14:00:55,048][  INFO][PID:424214] Releasing VM back to pool
[2025-12-12 14:00:55,102][  INFO][PID:424214] Retry #2 (on other host)
[2025-12-12 14:00:55,103][  INFO][PID:424214] Checking that /var/lib/copr/public_html/results/jwakely/boost-190-rebuild.checker/fedora-rawhide-x86_64/repodata/repomd.xml exists
[2025-12-12 14:00:55,103][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:00:55,104][  INFO][PID:424214] VM allocation process starts
[2025-12-12 14:00:55,125][  INFO][PID:424214] Trying to allocate VM: ResallocHost, ticket_id=304417, requested_tags=['copr_builder', 'arch_x86_64']
[2025-12-12 14:00:58,146][  INFO][PID:424214] Allocated host ResallocHost, ticket_id=304417, hostname=100.31.56.98, name=aws_x86_64_spot_prod_00497229_20251212_135329, requested_tags=['copr_builder', 'arch_x86_64']
[2025-12-12 14:00:58,146][  INFO][PID:424214] Allocating ssh connection to builder
[2025-12-12 14:00:58,147][  INFO][PID:424214] Checking that builder machine is OK
[2025-12-12 14:00:58,328][  INFO][PID:424214] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2025-12-12 14:00:58,519][  INFO][PID:424214] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2025-12-12 14:00:58,519][  INFO][PID:424214] Filling build.info file with builder info
[2025-12-12 14:00:58,521][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:00:58,521][  INFO][PID:424214] Not re-notifying FE and msg buses for the new host.
[2025-12-12 14:00:58,522][  INFO][PID:424214] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9902737-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2025-12-12 14:00:58,834][  INFO][PID:424214] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '100.31.56.98' (ED25519) to the list of known hosts.


[2025-12-12 14:00:58,835][  INFO][PID:424214] Downloading the builder-live.log file, attempt 1
[2025-12-12 14:00:58,837][  INFO][PID:424214] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@100.31.56.98 copr-rpmbuild-log
[2025-12-12 14:01:03,836][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:01:08,838][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:01:13,840][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:01:18,841][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:01:23,844][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:01:28,846][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:01:33,851][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:01:38,853][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:01:43,856][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:01:48,857][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:01:53,858][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:01:58,859][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:02:03,863][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:02:08,865][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:02:13,866][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:02:18,869][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:02:23,871][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:02:28,872][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:02:33,874][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:02:38,875][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:02:43,876][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:02:48,877][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:02:53,878][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:02:58,885][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:03:03,886][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:03:08,887][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:03:13,889][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:03:18,895][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:03:23,901][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:03:28,903][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:03:33,904][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:03:38,905][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:03:43,907][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:03:48,914][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:03:53,915][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:03:58,916][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:04:03,917][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:04:08,918][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:04:13,919][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:04:18,926][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:04:23,932][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:04:28,942][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:04:33,945][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:04:38,951][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:04:43,953][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:04:48,954][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:04:53,955][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:04:58,956][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:05:03,960][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:05:08,962][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:05:13,963][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:05:14,840][ ERROR][PID:424214] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-12 14:05:18,964][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:05:23,965][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:05:24,858][WARNING][PID:424214] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-12 14:05:28,966][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:05:33,967][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:05:34,867][  INFO][PID:424214] Downloading the builder-live.log file, attempt 2
[2025-12-12 14:05:34,975][  INFO][PID:424214] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@100.31.56.98 copr-rpmbuild-log
[2025-12-12 14:05:38,968][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:05:43,975][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:05:48,975][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:05:53,977][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:05:58,978][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:06:03,979][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:06:08,980][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:06:13,981][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:06:18,982][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:06:23,983][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:06:28,986][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:06:33,988][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:06:35,024][ ERROR][PID:424214] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-12 14:06:38,989][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:06:43,992][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:06:45,027][WARNING][PID:424214] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-12 14:06:48,993][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:06:53,998][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:06:55,028][  INFO][PID:424214] Downloading the builder-live.log file, attempt 3
[2025-12-12 14:06:55,029][  INFO][PID:424214] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@100.31.56.98 copr-rpmbuild-log
[2025-12-12 14:06:58,999][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:07:04,000][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:07:09,001][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:07:14,003][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:07:19,006][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:07:24,019][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:07:29,025][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:07:34,030][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:07:39,031][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:07:44,032][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:07:49,033][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:07:54,034][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:07:55,551][ ERROR][PID:424214] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-12 14:07:59,053][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:08:04,054][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:08:05,692][WARNING][PID:424214] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-12 14:08:09,056][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:08:14,057][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:08:15,741][  INFO][PID:424214] Downloading the builder-live.log file, attempt 4
[2025-12-12 14:08:16,346][  INFO][PID:424214] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@100.31.56.98 copr-rpmbuild-log
[2025-12-12 14:08:19,058][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:08:24,059][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:08:29,061][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:08:34,063][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:08:39,064][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:08:44,066][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:08:49,067][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:08:54,082][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:08:59,083][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:09:04,088][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:09:09,090][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:09:14,098][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:09:16,377][ ERROR][PID:424214] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-12 14:09:19,185][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:09:24,309][  INFO][PID:424214] Checking for cancel request
[2025-12-12 14:09:26,926][ ERROR][PID:424214] Giving up for unstable SSH, failures: 1765548324.8555453, 1765548405.027321, 1765548485.660218, 1765548566.4671562
[2025-12-12 14:09:27,532][ ERROR][PID:424214] Re-try request for task on 'ResallocHost, ticket_id=304417, hostname=100.31.56.98, name=aws_x86_64_spot_prod_00497229_20251212_135329, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-12-12 14:09:27,576][  INFO][PID:424214] Releasing VM back to pool
[2025-12-12 14:09:30,765][ ERROR][PID:424214] Backend process error: Three host tried without success: {'107.22.149.166', '44.223.87.13', '100.31.56.98'}
[2025-12-12 14:09:30,951][WARNING][PID:424214] Switching not-finished job state to 'failed'
[2025-12-12 14:09:30,964][  INFO][PID:424214] Worker failed build, took 512.312112569809
[2025-12-12 14:09:31,216][  INFO][PID:424214] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 115200,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "jwakely",
            "project_name": "boost-190-rebuild.checker",
            "project_dirname": "boost-190-rebuild.checker",
            "submitter": "jwakely",
            "ended_on": 1765548570.8338444,
            "started_on": 1765548058.5217319,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "9902737-fedora-rawhide-x86_64",
            "build_id": 9902737,
            "package_name": "SimGear",
            "package_version": "2024.1.3-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/jwakely/boost-190-rebuild.checker/SimGear",
            "git_hash": "39a3f5962b1c6cc1096ee4d09b92543d5e60c300",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "jwakely/boost-190-rebuild.checker--jwakely",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/jwakely/boost-190-rebuild.checker/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/jwakely/boost-190-rebuild.checker",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/jwakely/boost-190-rebuild.checker",
            "result_dir": "09902737-SimGear",
            "built_packages": "",
            "id": 9902737,
            "mockchain_macros": {
                "copr_username": "jwakely",
                "copr_projectname": "boost-190-rebuild.checker",
                "vendor": "Fedora Project COPR (jwakely/boost-190-rebuild.checker)"
            }
        }
    ]
}
[2025-12-12 14:09:45,449][  INFO][PID:424214] Sending fedora-messaging bus message in build.end
[2025-12-12 14:09:52,785][  INFO][PID:424214] Compressing /var/lib/copr/public_html/results/jwakely/boost-190-rebuild.checker/fedora-rawhide-x86_64/09902737-SimGear/builder-live.log by gzip
[2025-12-12 14:09:52,882][  INFO][PID:424214] Running command 'gzip /var/lib/copr/public_html/results/jwakely/boost-190-rebuild.checker/fedora-rawhide-x86_64/09902737-SimGear/builder-live.log' as PID 588697
[2025-12-12 14:09:53,156][  INFO][PID:424214] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/jwakely/boost-190-rebuild.checker/fedora-rawhide-x86_64/09902737-SimGear/builder-live.log)
[2025-12-12 14:09:53,266][  INFO][PID:424214] Compressing /var/lib/copr/public_html/results/jwakely/boost-190-rebuild.checker/fedora-rawhide-x86_64/09902737-SimGear/backend.log by gzip
[2025-12-12 14:09:53,267][  INFO][PID:424214] Running command 'gzip /var/lib/copr/public_html/results/jwakely/boost-190-rebuild.checker/fedora-rawhide-x86_64/09902737-SimGear/backend.log' as PID 588739