[2025-09-11 16:11:10,395][  INFO][PID:3915877] Marking build as starting
[2025-09-11 16:11:10,424][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:11:10,425][  INFO][PID:3915877] VM allocation process starts
[2025-09-11 16:11:10,446][  INFO][PID:3915877] Trying to allocate VM: ResallocHost, ticket_id=5010089, requested_tags=['copr_builder', 'arch_x86_64']
[2025-09-11 16:11:15,448][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:11:20,449][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:11:25,450][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:11:30,451][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:11:35,452][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:11:40,453][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:11:45,454][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:11:50,456][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:11:55,457][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:12:00,458][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:12:05,459][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:12:10,460][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:12:15,461][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:12:20,463][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:12:25,465][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:12:30,467][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:12:35,468][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:12:40,471][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:12:45,473][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:12:50,474][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:12:55,475][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:13:00,478][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:13:05,484][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:13:10,485][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:13:15,487][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:13:20,493][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:13:25,494][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:13:30,495][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:13:35,500][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:13:40,502][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:13:45,507][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:13:50,508][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:13:55,516][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:14:00,524][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:14:05,525][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:14:10,531][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:14:15,533][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:14:20,538][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:14:25,544][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:14:30,547][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:14:35,553][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:14:40,555][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:14:45,556][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:14:50,559][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:14:55,561][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:15:00,562][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:15:05,563][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:15:10,570][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:15:15,578][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:15:20,581][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:15:25,585][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:15:30,587][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:15:35,589][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:15:40,591][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:15:45,597][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:15:50,599][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:15:55,602][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:16:00,608][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:16:05,613][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:16:10,620][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:16:15,624][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:16:20,631][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:16:25,638][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:16:30,640][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:16:35,643][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:16:40,653][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:16:45,663][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:16:50,674][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:16:55,675][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:17:00,677][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:17:05,682][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:17:10,684][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:17:15,687][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:17:20,690][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:17:25,692][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:17:28,721][  INFO][PID:3915877] Allocated host ResallocHost, ticket_id=5010089, hostname=44.201.14.59, name=aws_x86_64_spot_prod_06718023_20250911_161442, requested_tags=['copr_builder', 'arch_x86_64']
[2025-09-11 16:17:28,729][  INFO][PID:3915877] Allocating ssh connection to builder
[2025-09-11 16:17:28,731][  INFO][PID:3915877] Checking that builder machine is OK
[2025-09-11 16:17:29,169][  INFO][PID:3915877] Running remote command: copr-builder-ready epel-10-x86_64
[2025-09-11 16:17:29,407][  INFO][PID:3915877] Checking Red Hat subscription (attempt #1)
Red Hat subscription active
Builder is ready to be used

[2025-09-11 16:17:29,408][  INFO][PID:3915877] Filling build.info file with builder info
[2025-09-11 16:17:29,409][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:17:29,411][  INFO][PID:3915877] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "networkmanager",
            "project_name": "NetworkManager-main-debug",
            "project_dirname": "NetworkManager-main-debug",
            "submitter": null,
            "ended_on": null,
            "started_on": 1757607449.4116325,
            "submitted_on": null,
            "status": 3,
            "chroot": "epel-10-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "9546505-epel-10-x86_64",
            "build_id": 9546505,
            "package_name": "NetworkManager",
            "package_version": "1:1.55.3-34173.copr.40aa27690c",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/networkmanager/NetworkManager-main-debug/NetworkManager",
            "git_hash": "d5f008a9fc2b5e90432359a68a43e74141fabdaa",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "networkmanager/NetworkManager-main-debug--230a87d4-33f2-4b74-bd7b-c1eb9c707f27",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/networkmanager/NetworkManager-main-debug/epel-10-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/nmstate/nm-build-deps/epel-10-$basearch/",
                    "id": "https_download_copr_fedorainfracloud_org_results_nmstate_nm_build_deps_epel_10_basearch",
                    "name": "Additional repo https_download_copr_fedorainfracloud_org_results_nmstate_nm_build_deps_epel_10_basearch"
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/networkmanager/NetworkManager-main-debug",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/networkmanager/NetworkManager-main-debug",
            "result_dir": "09546505-NetworkManager",
            "built_packages": "",
            "id": 9546505,
            "mockchain_macros": {
                "copr_username": "networkmanager",
                "copr_projectname": "NetworkManager-main-debug",
                "vendor": "Fedora Project COPR (networkmanager/NetworkManager-main-debug)"
            }
        }
    ]
}
[2025-09-11 16:17:29,444][  INFO][PID:3915877] Sending fedora-messaging bus message in build.start
[2025-09-11 16:17:30,662][  INFO][PID:3915877] Sending fedora-messaging bus message in chroot.start
[2025-09-11 16:17:30,715][  INFO][PID:3915877] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9546505-epel-10-x86_64 --chroot epel-10-x86_64 --detached
[2025-09-11 16:17:31,105][  INFO][PID:3915877] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '44.201.14.59' (ED25519) to the list of known hosts.


[2025-09-11 16:17:31,110][  INFO][PID:3915877] Downloading the builder-live.log file, attempt 1
[2025-09-11 16:17:31,119][  INFO][PID:3915877] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.201.14.59 copr-rpmbuild-log
[2025-09-11 16:17:36,140][  INFO][PID:3915877] Periodic builder liveness probe: alive
[2025-09-11 16:17:36,143][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:17:41,150][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:17:46,158][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:17:51,160][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:17:56,164][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:18:01,165][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:18:06,170][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:18:11,172][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:18:16,174][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:18:21,178][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:18:26,186][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:18:31,188][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:18:36,196][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:18:41,199][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:18:46,212][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:18:51,215][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:18:56,218][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:19:01,224][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:19:06,229][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:19:11,239][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:19:16,248][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:19:21,249][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:19:26,250][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:19:31,261][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:19:36,264][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:19:41,268][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:19:46,270][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:19:51,274][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:19:56,276][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:20:01,278][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:20:06,280][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:20:11,281][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:20:16,284][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:20:21,287][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:20:26,290][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:20:31,293][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:20:36,298][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:20:41,312][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:20:46,314][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:20:51,326][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:20:56,332][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:21:01,337][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:21:06,340][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:21:11,342][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:21:16,344][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:21:21,347][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:21:26,353][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:21:31,354][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:21:36,359][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:21:41,364][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:21:42,521][ ERROR][PID:3915877] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:21:46,372][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:21:51,376][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:21:52,529][WARNING][PID:3915877] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 16:21:56,384][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:22:01,387][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:22:02,530][  INFO][PID:3915877] Downloading the builder-live.log file, attempt 2
[2025-09-11 16:22:02,544][  INFO][PID:3915877] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.201.14.59 copr-rpmbuild-log
[2025-09-11 16:22:06,388][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:22:11,394][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:22:16,397][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:22:21,411][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:22:26,413][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:22:31,414][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:22:36,435][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:22:41,436][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:22:46,439][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:22:51,444][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:22:56,447][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:23:01,450][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:23:02,654][ ERROR][PID:3915877] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:23:06,453][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:23:11,455][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:23:12,658][WARNING][PID:3915877] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 16:23:16,460][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:23:21,464][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:23:22,661][  INFO][PID:3915877] Downloading the builder-live.log file, attempt 3
[2025-09-11 16:23:22,665][  INFO][PID:3915877] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.201.14.59 copr-rpmbuild-log
[2025-09-11 16:23:26,466][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:23:31,472][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:23:36,473][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:23:41,477][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:23:46,482][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:23:51,484][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:23:56,486][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:24:01,488][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:24:06,490][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:24:11,491][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:24:16,493][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:24:21,494][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:24:22,744][ ERROR][PID:3915877] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:24:26,495][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:24:31,497][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:24:32,748][WARNING][PID:3915877] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 16:24:36,498][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:24:41,503][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:24:42,749][  INFO][PID:3915877] Downloading the builder-live.log file, attempt 4
[2025-09-11 16:24:42,751][  INFO][PID:3915877] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.201.14.59 copr-rpmbuild-log
[2025-09-11 16:24:46,504][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:24:51,506][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:24:56,507][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:25:01,508][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:25:06,510][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:25:11,511][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:25:16,513][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:25:21,518][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:25:26,521][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:25:31,522][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:25:36,527][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:25:41,531][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:25:42,821][ ERROR][PID:3915877] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:25:46,532][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:25:51,536][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:25:52,824][ ERROR][PID:3915877] Giving up for unstable SSH, failures: 1757607712.5290582, 1757607792.65857, 1757607872.7482605, 1757607952.8234432
[2025-09-11 16:25:52,825][ ERROR][PID:3915877] Re-try request for task on 'ResallocHost, ticket_id=5010089, hostname=44.201.14.59, name=aws_x86_64_spot_prod_06718023_20250911_161442, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-09-11 16:25:52,826][  INFO][PID:3915877] Releasing VM back to pool
[2025-09-11 16:25:52,838][  INFO][PID:3915877] Retry #1 (on other host)
[2025-09-11 16:25:52,839][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:25:52,840][  INFO][PID:3915877] VM allocation process starts
[2025-09-11 16:25:52,855][  INFO][PID:3915877] Trying to allocate VM: ResallocHost, ticket_id=5010495, requested_tags=['copr_builder', 'arch_x86_64']
[2025-09-11 16:25:57,856][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:26:02,858][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:26:07,861][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:26:12,863][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:26:17,864][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:26:22,865][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:26:27,867][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:26:32,868][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:26:37,868][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:26:42,874][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:26:47,875][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:26:52,880][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:26:57,882][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:27:02,883][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:27:07,890][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:27:12,892][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:27:17,893][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:27:22,895][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:27:27,896][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:27:32,897][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:27:37,901][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:27:42,903][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:27:47,906][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:27:52,907][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:27:57,912][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:28:02,919][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:28:07,920][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:28:12,921][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:28:17,923][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:28:22,925][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:28:27,926][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:28:32,929][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:28:37,932][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:28:42,934][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:28:47,935][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:28:52,937][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:28:57,938][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:29:02,939][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:29:07,941][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:29:12,942][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:29:17,943][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:29:22,944][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:29:27,945][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:29:32,949][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:29:37,950][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:29:42,951][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:29:47,952][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:29:52,955][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:29:57,956][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:30:02,957][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:30:07,958][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:30:10,984][  INFO][PID:3915877] Allocated host ResallocHost, ticket_id=5010495, hostname=13.222.202.174, name=aws_x86_64_spot_prod_06718342_20250911_162747, requested_tags=['copr_builder', 'arch_x86_64']
[2025-09-11 16:30:10,985][  INFO][PID:3915877] Allocating ssh connection to builder
[2025-09-11 16:30:10,986][  INFO][PID:3915877] Checking that builder machine is OK
[2025-09-11 16:30:11,416][  INFO][PID:3915877] Running remote command: copr-builder-ready epel-10-x86_64
[2025-09-11 16:30:11,634][  INFO][PID:3915877] Checking Red Hat subscription (attempt #1)
Red Hat subscription active
Builder is ready to be used

[2025-09-11 16:30:11,635][  INFO][PID:3915877] Filling build.info file with builder info
[2025-09-11 16:30:11,636][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:30:11,637][  INFO][PID:3915877] Not re-notifying FE and msg buses for the new host.
[2025-09-11 16:30:11,637][  INFO][PID:3915877] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9546505-epel-10-x86_64 --chroot epel-10-x86_64 --detached
[2025-09-11 16:30:11,966][  INFO][PID:3915877] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '13.222.202.174' (ED25519) to the list of known hosts.


[2025-09-11 16:30:11,967][  INFO][PID:3915877] Downloading the builder-live.log file, attempt 1
[2025-09-11 16:30:11,968][  INFO][PID:3915877] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.222.202.174 copr-rpmbuild-log
[2025-09-11 16:30:16,976][  INFO][PID:3915877] Periodic builder liveness probe: alive
[2025-09-11 16:30:16,977][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:30:21,982][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:30:26,983][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:30:31,988][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:30:36,994][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:30:41,995][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:30:46,997][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:30:51,998][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:30:56,999][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:31:02,001][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:31:07,003][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:31:12,004][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:31:17,007][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:31:22,009][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:31:27,010][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:31:32,011][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:31:37,013][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:31:42,014][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:31:47,017][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:31:52,018][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:31:57,019][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:32:02,020][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:32:07,021][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:32:12,022][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:32:17,023][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:32:22,024][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:32:27,027][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:32:32,029][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:32:37,030][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:32:42,031][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:32:47,032][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:32:52,033][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:32:57,036][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:33:02,037][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:33:07,039][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:33:12,040][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:33:17,041][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:33:22,042][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:33:27,044][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:33:32,047][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:33:37,051][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:33:42,054][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:33:47,055][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:33:52,056][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:33:57,057][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:34:02,063][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:34:07,066][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:34:11,798][ ERROR][PID:3915877] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:34:12,067][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:34:17,068][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:34:21,799][WARNING][PID:3915877] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 16:34:22,070][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:34:27,071][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:34:31,800][  INFO][PID:3915877] Downloading the builder-live.log file, attempt 2
[2025-09-11 16:34:31,803][  INFO][PID:3915877] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.222.202.174 copr-rpmbuild-log
[2025-09-11 16:34:32,072][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:34:37,073][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:34:42,075][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:34:47,077][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:34:52,078][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:34:57,080][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:35:02,082][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:35:07,082][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:35:12,084][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:35:17,084][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:35:22,086][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:35:27,087][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:35:31,875][ ERROR][PID:3915877] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:35:32,088][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:35:37,091][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:35:41,880][WARNING][PID:3915877] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 16:35:42,092][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:35:47,093][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:35:51,880][  INFO][PID:3915877] Downloading the builder-live.log file, attempt 3
[2025-09-11 16:35:51,882][  INFO][PID:3915877] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.222.202.174 copr-rpmbuild-log
[2025-09-11 16:35:52,095][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:35:57,096][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:36:02,097][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:36:07,098][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:36:12,099][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:36:17,100][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:36:22,101][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:36:27,103][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:36:32,103][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:36:37,105][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:36:42,106][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:36:47,107][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:36:51,950][ ERROR][PID:3915877] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:36:52,109][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:36:57,110][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:37:01,951][WARNING][PID:3915877] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 16:37:02,111][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:37:07,112][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:37:11,952][  INFO][PID:3915877] Downloading the builder-live.log file, attempt 4
[2025-09-11 16:37:11,953][  INFO][PID:3915877] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.222.202.174 copr-rpmbuild-log
[2025-09-11 16:37:12,113][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:37:17,114][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:37:22,115][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:37:27,116][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:37:32,117][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:37:37,118][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:37:42,119][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:37:47,120][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:37:52,121][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:37:57,123][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:38:02,124][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:38:07,124][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:38:12,021][ ERROR][PID:3915877] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:38:12,126][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:38:17,127][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:38:22,023][ ERROR][PID:3915877] Giving up for unstable SSH, failures: 1757608461.7997935, 1757608541.8799586, 1757608621.9513354, 1757608702.0232399
[2025-09-11 16:38:22,024][ ERROR][PID:3915877] Re-try request for task on 'ResallocHost, ticket_id=5010495, hostname=13.222.202.174, name=aws_x86_64_spot_prod_06718342_20250911_162747, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-09-11 16:38:22,024][  INFO][PID:3915877] Releasing VM back to pool
[2025-09-11 16:38:22,052][  INFO][PID:3915877] Retry #2 (on other host)
[2025-09-11 16:38:22,053][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:38:22,053][  INFO][PID:3915877] VM allocation process starts
[2025-09-11 16:38:22,071][  INFO][PID:3915877] Trying to allocate VM: ResallocHost, ticket_id=5010777, requested_tags=['copr_builder', 'arch_x86_64']
[2025-09-11 16:38:27,072][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:38:32,074][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:38:37,075][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:38:42,078][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:38:47,079][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:38:52,081][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:38:57,082][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:39:02,083][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:39:07,084][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:39:12,086][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:39:17,100][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:39:22,105][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:39:27,115][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:39:32,116][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:39:37,117][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:39:42,118][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:39:47,122][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:39:52,123][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:39:57,124][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:40:02,126][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:40:07,129][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:40:12,134][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:40:17,137][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:40:22,137][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:40:27,143][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:40:32,144][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:40:37,145][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:40:42,148][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:40:47,153][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:40:52,155][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:40:57,162][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:41:02,163][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:41:07,165][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:41:12,166][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:41:17,167][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:41:22,169][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:41:27,170][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:41:32,172][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:41:37,173][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:41:40,245][  INFO][PID:3915877] Allocated host ResallocHost, ticket_id=5010777, hostname=3.95.0.155, name=aws_x86_64_spot_prod_06718675_20250911_164006, requested_tags=['copr_builder', 'arch_x86_64']
[2025-09-11 16:41:40,246][  INFO][PID:3915877] Allocating ssh connection to builder
[2025-09-11 16:41:40,247][  INFO][PID:3915877] Checking that builder machine is OK
[2025-09-11 16:41:40,678][  INFO][PID:3915877] Running remote command: copr-builder-ready epel-10-x86_64
[2025-09-11 16:41:40,889][  INFO][PID:3915877] Checking Red Hat subscription (attempt #1)
Red Hat subscription active
Builder is ready to be used

[2025-09-11 16:41:40,889][  INFO][PID:3915877] Filling build.info file with builder info
[2025-09-11 16:41:40,891][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:41:40,892][  INFO][PID:3915877] Not re-notifying FE and msg buses for the new host.
[2025-09-11 16:41:40,893][  INFO][PID:3915877] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9546505-epel-10-x86_64 --chroot epel-10-x86_64 --detached
[2025-09-11 16:41:41,214][  INFO][PID:3915877] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '3.95.0.155' (ED25519) to the list of known hosts.


[2025-09-11 16:41:41,215][  INFO][PID:3915877] Downloading the builder-live.log file, attempt 1
[2025-09-11 16:41:41,217][  INFO][PID:3915877] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.95.0.155 copr-rpmbuild-log
[2025-09-11 16:41:46,226][  INFO][PID:3915877] Periodic builder liveness probe: alive
[2025-09-11 16:41:46,227][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:41:51,228][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:41:56,229][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:42:01,230][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:42:06,231][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:42:11,232][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:42:16,234][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:42:21,235][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:42:26,236][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:42:31,238][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:42:36,239][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:42:41,240][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:42:46,242][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:42:51,244][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:42:56,245][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:43:01,246][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:43:06,248][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:43:11,250][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:43:16,252][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:43:21,254][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:43:26,258][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:43:31,259][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:43:36,262][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:43:41,263][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:43:46,265][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:43:51,266][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:43:56,267][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:44:01,268][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:44:06,273][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:44:11,274][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:44:16,276][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:44:21,278][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:44:26,279][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:44:31,280][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:44:36,282][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:44:41,283][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:44:46,285][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:44:51,286][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:44:56,288][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:45:01,289][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:45:06,292][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:45:11,293][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:45:16,294][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:45:21,295][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:45:26,296][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:45:31,297][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:45:36,298][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:45:41,298][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:45:46,300][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:45:51,302][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:45:56,303][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:46:01,305][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:46:06,308][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:46:11,310][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:46:16,311][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:46:21,312][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:46:26,313][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:46:31,316][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:46:36,318][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:46:41,319][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:46:46,320][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:46:51,322][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:46:56,323][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:47:01,324][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:47:06,325][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:47:11,326][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:47:16,328][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:47:21,329][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:47:26,330][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:47:31,331][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:47:36,332][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:47:41,333][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:47:46,335][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:47:51,337][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:47:56,347][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:48:01,349][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:48:06,352][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:48:11,354][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:48:16,357][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:48:21,358][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:48:26,359][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:48:31,360][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:48:36,361][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:48:41,364][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:48:46,375][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:48:51,376][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:48:56,377][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:49:01,378][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:49:06,381][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:49:11,382][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:49:16,387][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:49:21,389][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:49:26,390][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:49:31,391][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:49:36,392][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:49:41,393][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:49:46,395][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:49:51,397][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:49:56,399][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:50:01,401][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:50:06,404][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:50:11,405][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:50:16,406][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:50:21,408][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:50:26,409][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:50:31,410][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:50:36,412][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:50:41,414][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:50:46,416][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:50:51,457][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:50:56,490][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:51:01,523][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:51:06,602][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:51:11,632][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:51:16,642][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:51:21,657][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:51:26,665][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:51:31,841][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:51:36,870][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:51:41,892][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:51:46,910][  INFO][PID:3915877] Periodic builder liveness probe: alive
[2025-09-11 16:51:46,992][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:51:51,994][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:51:56,998][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:52:02,062][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:52:07,117][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:52:12,161][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:52:17,185][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:52:22,215][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:52:27,261][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:52:32,293][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:52:37,296][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:52:42,297][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:52:47,302][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:52:52,318][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:52:57,414][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:53:02,431][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:53:07,445][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:53:12,474][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:53:17,483][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:53:22,484][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:53:27,486][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:53:32,495][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:53:37,502][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:53:42,517][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:53:46,119][ ERROR][PID:3915877] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:53:47,523][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:53:52,525][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:53:56,120][WARNING][PID:3915877] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 16:53:57,527][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:54:02,539][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:54:06,122][  INFO][PID:3915877] Downloading the builder-live.log file, attempt 2
[2025-09-11 16:54:06,126][  INFO][PID:3915877] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.95.0.155 copr-rpmbuild-log
[2025-09-11 16:54:07,559][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:54:12,575][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:54:17,578][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:54:22,582][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:54:27,587][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:54:32,590][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:54:37,596][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:54:42,597][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:54:47,599][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:54:52,600][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:54:57,602][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:55:02,603][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:55:06,195][ ERROR][PID:3915877] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:55:07,609][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:55:12,611][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:55:16,197][WARNING][PID:3915877] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 16:55:17,618][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:55:22,619][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:55:26,197][  INFO][PID:3915877] Downloading the builder-live.log file, attempt 3
[2025-09-11 16:55:26,199][  INFO][PID:3915877] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.95.0.155 copr-rpmbuild-log
[2025-09-11 16:55:27,623][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:55:32,626][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:55:37,628][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:55:42,630][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:55:47,634][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:55:52,635][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:55:57,636][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:56:02,637][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:56:07,639][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:56:12,640][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:56:17,641][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:56:22,643][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:56:26,269][ ERROR][PID:3915877] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:56:27,650][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:56:32,659][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:56:36,270][WARNING][PID:3915877] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 16:56:37,667][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:56:42,669][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:56:46,273][  INFO][PID:3915877] Downloading the builder-live.log file, attempt 4
[2025-09-11 16:56:46,281][  INFO][PID:3915877] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.95.0.155 copr-rpmbuild-log
[2025-09-11 16:56:47,670][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:56:52,671][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:56:57,673][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:57:02,674][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:57:07,676][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:57:12,684][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:57:17,686][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:57:22,687][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:57:27,689][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:57:32,690][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:57:37,691][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:57:42,692][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:57:46,370][ ERROR][PID:3915877] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 16:57:47,693][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:57:52,698][  INFO][PID:3915877] Checking for cancel request
[2025-09-11 16:57:56,371][ ERROR][PID:3915877] Giving up for unstable SSH, failures: 1757609636.1208458, 1757609716.196976, 1757609796.270493, 1757609876.3713987
[2025-09-11 16:57:56,372][ ERROR][PID:3915877] Re-try request for task on 'ResallocHost, ticket_id=5010777, hostname=3.95.0.155, name=aws_x86_64_spot_prod_06718675_20250911_164006, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-09-11 16:57:56,373][  INFO][PID:3915877] Releasing VM back to pool
[2025-09-11 16:57:56,398][ ERROR][PID:3915877] Backend process error: Three host tried without success: {'44.201.14.59', '3.95.0.155', '13.222.202.174'}
[2025-09-11 16:57:56,399][WARNING][PID:3915877] Switching not-finished job state to 'failed'
[2025-09-11 16:57:56,399][  INFO][PID:3915877] Worker failed build, took 975.5069279670715
[2025-09-11 16:57:56,400][  INFO][PID:3915877] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "networkmanager",
            "project_name": "NetworkManager-main-debug",
            "project_dirname": "NetworkManager-main-debug",
            "submitter": null,
            "ended_on": 1757609876.3991017,
            "started_on": 1757608900.8921738,
            "submitted_on": null,
            "status": 0,
            "chroot": "epel-10-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "9546505-epel-10-x86_64",
            "build_id": 9546505,
            "package_name": "NetworkManager",
            "package_version": "1:1.55.3-34173.copr.40aa27690c",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/networkmanager/NetworkManager-main-debug/NetworkManager",
            "git_hash": "d5f008a9fc2b5e90432359a68a43e74141fabdaa",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "networkmanager/NetworkManager-main-debug--230a87d4-33f2-4b74-bd7b-c1eb9c707f27",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/networkmanager/NetworkManager-main-debug/epel-10-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/nmstate/nm-build-deps/epel-10-$basearch/",
                    "id": "https_download_copr_fedorainfracloud_org_results_nmstate_nm_build_deps_epel_10_basearch",
                    "name": "Additional repo https_download_copr_fedorainfracloud_org_results_nmstate_nm_build_deps_epel_10_basearch"
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/networkmanager/NetworkManager-main-debug",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/networkmanager/NetworkManager-main-debug",
            "result_dir": "09546505-NetworkManager",
            "built_packages": "",
            "id": 9546505,
            "mockchain_macros": {
                "copr_username": "networkmanager",
                "copr_projectname": "NetworkManager-main-debug",
                "vendor": "Fedora Project COPR (networkmanager/NetworkManager-main-debug)"
            }
        }
    ]
}
[2025-09-11 16:57:56,427][  INFO][PID:3915877] Sending fedora-messaging bus message in build.end
[2025-09-11 16:57:56,457][  INFO][PID:3915877] Compressing /var/lib/copr/public_html/results/networkmanager/NetworkManager-main-debug/epel-10-x86_64/09546505-NetworkManager/builder-live.log by gzip
[2025-09-11 16:57:56,459][  INFO][PID:3915877] Running command 'gzip /var/lib/copr/public_html/results/networkmanager/NetworkManager-main-debug/epel-10-x86_64/09546505-NetworkManager/builder-live.log' as PID 100254
[2025-09-11 16:57:56,460][  INFO][PID:3915877] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/networkmanager/NetworkManager-main-debug/epel-10-x86_64/09546505-NetworkManager/builder-live.log)
[2025-09-11 16:57:56,462][  INFO][PID:3915877] Compressing /var/lib/copr/public_html/results/networkmanager/NetworkManager-main-debug/epel-10-x86_64/09546505-NetworkManager/backend.log by gzip
[2025-09-11 16:57:56,463][  INFO][PID:3915877] Running command 'gzip /var/lib/copr/public_html/results/networkmanager/NetworkManager-main-debug/epel-10-x86_64/09546505-NetworkManager/backend.log' as PID 100256