[2026-03-26 13:07:21,651][  INFO][PID:2592623] Marking build as starting
[2026-03-26 13:07:21,764][  INFO][PID:2592623] Checking that https://download.copr.fedorainfracloud.org/results/andersrh/SonicDE-devel/fedora-44-x86_64//repodata/repomd.xml exists
[2026-03-26 13:07:22,052][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:07:22,055][  INFO][PID:2592623] VM allocation process starts
[2026-03-26 13:07:22,077][  INFO][PID:2592623] Trying to allocate VM: ResallocHost, ticket_id=2040058, requested_tags=['copr_builder', 'arch_x86_64']
[2026-03-26 13:07:25,104][  INFO][PID:2592623] Allocated host ResallocHost, ticket_id=2040058, hostname=34.203.204.34, name=aws_x86_64_spot_prod_04287351_20260326_125932, requested_tags=['copr_builder', 'arch_x86_64']
[2026-03-26 13:07:25,104][  INFO][PID:2592623] Allocating ssh connection to builder
[2026-03-26 13:07:25,105][  INFO][PID:2592623] Checking that builder machine is OK
[2026-03-26 13:07:25,439][  INFO][PID:2592623] Running remote command: copr-builder-ready fedora-44-x86_64
[2026-03-26 13:07:25,639][  INFO][PID:2592623] Red Hat subscription not needed for fedora-44-x86_64
Builder is ready to be used

[2026-03-26 13:07:25,639][  INFO][PID:2592623] Filling build.info file with builder info
[2026-03-26 13:07:25,640][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:07:25,641][  INFO][PID:2592623] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "andersrh",
            "project_name": "SonicDE-devel",
            "project_dirname": "SonicDE-devel",
            "submitter": "andersrh",
            "ended_on": null,
            "started_on": 1774530445.6410682,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-44-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "10263869-fedora-44-x86_64",
            "build_id": 10263869,
            "package_name": "sonic-frameworks-io",
            "package_version": "6.24.0-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/andersrh/SonicDE-devel/sonic-frameworks-io",
            "git_hash": "35f0fdaaee02f7f5b95e0ad7bb258fa2be80d390",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "andersrh/SonicDE-devel--andersrh",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 1,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/andersrh/SonicDE-devel/fedora-44-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@xlibre/xlibre-xserver/fedora-44-x86_64/",
                    "id": "copr_xlibre_xlibre_xserver",
                    "name": "Additional repo copr_xlibre_xlibre_xserver"
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/andersrh/SonicDE-devel",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/andersrh/SonicDE-devel",
            "result_dir": "10263869-sonic-frameworks-io",
            "built_packages": "",
            "id": 10263869,
            "mockchain_macros": {
                "copr_username": "andersrh",
                "copr_projectname": "SonicDE-devel",
                "vendor": "Fedora Project COPR (andersrh/SonicDE-devel)"
            }
        }
    ]
}
[2026-03-26 13:07:25,716][  INFO][PID:2592623] Sending fedora-messaging bus message in build.start
[2026-03-26 13:07:26,338][  INFO][PID:2592623] Sending fedora-messaging bus message in chroot.start
[2026-03-26 13:07:26,384][  INFO][PID:2592623] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10263869-fedora-44-x86_64 --chroot fedora-44-x86_64 --detached
[2026-03-26 13:07:26,704][  INFO][PID:2592623] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '34.203.204.34' (ED25519) to the list of known hosts.


[2026-03-26 13:07:26,705][  INFO][PID:2592623] Downloading the builder-live.log file, attempt 1
[2026-03-26 13:07:26,759][  INFO][PID:2592623] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.203.204.34 copr-rpmbuild-log
[2026-03-26 13:07:31,711][  INFO][PID:2592623] Periodic builder liveness probe: alive
[2026-03-26 13:07:31,712][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:07:36,713][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:07:41,714][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:07:46,716][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:07:51,717][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:07:56,718][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:08:01,719][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:08:06,720][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:08:11,722][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:08:16,724][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:08:21,725][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:08:26,727][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:08:31,728][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:08:36,730][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:08:41,731][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:08:46,732][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:08:51,733][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:08:56,734][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:09:01,735][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:09:06,736][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:09:11,740][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:09:16,741][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:09:19,679][ ERROR][PID:2592623] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 13:09:21,742][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:09:26,743][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:09:29,680][WARNING][PID:2592623] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-03-26 13:09:31,744][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:09:36,746][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:09:39,680][  INFO][PID:2592623] Downloading the builder-live.log file, attempt 2
[2026-03-26 13:09:39,682][  INFO][PID:2592623] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.203.204.34 copr-rpmbuild-log
[2026-03-26 13:09:41,750][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:09:46,751][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:09:51,752][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:09:56,753][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:10:01,755][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:10:06,757][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:10:11,758][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:10:16,760][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:10:21,761][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:10:26,762][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:10:31,764][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:10:36,765][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:10:39,752][ ERROR][PID:2592623] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 13:10:41,766][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:10:46,767][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:10:49,804][WARNING][PID:2592623] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-03-26 13:10:51,768][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:10:56,770][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:10:59,805][  INFO][PID:2592623] Downloading the builder-live.log file, attempt 3
[2026-03-26 13:10:59,806][  INFO][PID:2592623] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.203.204.34 copr-rpmbuild-log
[2026-03-26 13:11:01,771][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:11:06,772][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:11:11,774][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:11:16,790][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:11:21,792][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:11:26,793][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:11:31,794][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:11:36,795][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:11:41,796][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:11:46,799][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:11:51,821][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:11:56,823][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:11:59,890][ ERROR][PID:2592623] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 13:12:01,824][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:12:06,825][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:12:09,911][WARNING][PID:2592623] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-03-26 13:12:11,826][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:12:16,827][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:12:19,912][  INFO][PID:2592623] Downloading the builder-live.log file, attempt 4
[2026-03-26 13:12:19,913][  INFO][PID:2592623] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.203.204.34 copr-rpmbuild-log
[2026-03-26 13:12:21,828][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:12:26,829][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:12:31,830][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:12:36,831][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:12:41,863][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:12:46,867][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:12:51,867][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:12:56,870][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:13:01,872][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:13:06,873][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:13:11,874][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:13:16,938][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:13:19,976][ ERROR][PID:2592623] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 13:13:21,948][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:13:26,949][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:13:29,978][ ERROR][PID:2592623] Giving up for unstable SSH, failures: 1774530569.679983, 1774530649.8044028, 1774530729.9114048, 1774530809.978555
[2026-03-26 13:13:29,979][ ERROR][PID:2592623] Re-try request for task on 'ResallocHost, ticket_id=2040058, hostname=34.203.204.34, name=aws_x86_64_spot_prod_04287351_20260326_125932, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-03-26 13:13:29,979][  INFO][PID:2592623] Releasing VM back to pool
[2026-03-26 13:13:29,989][  INFO][PID:2592623] Retry #1 (on other host)
[2026-03-26 13:13:29,989][  INFO][PID:2592623] Checking that https://download.copr.fedorainfracloud.org/results/andersrh/SonicDE-devel/fedora-44-x86_64//repodata/repomd.xml exists
[2026-03-26 13:13:30,337][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:13:30,339][  INFO][PID:2592623] VM allocation process starts
[2026-03-26 13:13:30,349][  INFO][PID:2592623] Trying to allocate VM: ResallocHost, ticket_id=2040211, requested_tags=['copr_builder', 'arch_x86_64']
[2026-03-26 13:13:35,350][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:13:40,351][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:13:45,352][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:13:50,353][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:13:55,354][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:14:00,355][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:14:05,356][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:14:10,357][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:14:15,359][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:14:20,363][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:14:25,364][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:14:30,365][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:14:35,367][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:14:40,368][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:14:45,370][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:14:50,374][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:14:55,375][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:15:00,376][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:15:05,378][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:15:10,379][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:15:15,380][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:15:18,415][  INFO][PID:2592623] Allocated host ResallocHost, ticket_id=2040211, hostname=3.229.130.110, name=aws_x86_64_spot_prod_04288181_20260326_131342, requested_tags=['copr_builder', 'arch_x86_64']
[2026-03-26 13:15:18,416][  INFO][PID:2592623] Allocating ssh connection to builder
[2026-03-26 13:15:18,417][  INFO][PID:2592623] Checking that builder machine is OK
[2026-03-26 13:15:18,764][  INFO][PID:2592623] Running remote command: copr-builder-ready fedora-44-x86_64
[2026-03-26 13:15:18,981][  INFO][PID:2592623] Red Hat subscription not needed for fedora-44-x86_64
Builder is ready to be used

[2026-03-26 13:15:18,981][  INFO][PID:2592623] Filling build.info file with builder info
[2026-03-26 13:15:18,995][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:15:18,996][  INFO][PID:2592623] Not re-notifying FE and msg buses for the new host.
[2026-03-26 13:15:18,997][  INFO][PID:2592623] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10263869-fedora-44-x86_64 --chroot fedora-44-x86_64 --detached
[2026-03-26 13:15:19,343][  INFO][PID:2592623] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '3.229.130.110' (ED25519) to the list of known hosts.


[2026-03-26 13:15:19,344][  INFO][PID:2592623] Downloading the builder-live.log file, attempt 1
[2026-03-26 13:15:19,346][  INFO][PID:2592623] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.229.130.110 copr-rpmbuild-log
[2026-03-26 13:15:24,345][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:15:29,350][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:15:34,353][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:15:39,356][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:15:44,360][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:15:49,361][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:15:54,363][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:15:59,365][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:16:04,368][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:16:09,370][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:16:14,374][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:16:19,375][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:16:24,376][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:16:29,379][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:16:34,380][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:16:39,382][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:16:44,383][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:16:49,384][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:16:54,385][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:16:59,387][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:17:04,388][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:17:09,389][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:17:14,390][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:17:19,391][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:17:24,394][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:17:29,396][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:17:34,429][  INFO][PID:2592623] Periodic builder liveness probe: alive
[2026-03-26 13:17:34,431][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:17:39,432][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:17:44,434][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:17:49,435][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:17:54,436][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:17:59,439][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:18:04,440][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:18:09,444][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:18:14,445][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:18:19,446][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:18:24,447][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:18:29,449][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:18:34,451][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:18:39,453][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:18:44,455][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:18:49,458][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:18:54,461][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:18:59,464][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:19:04,497][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:19:09,499][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:19:14,503][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:19:19,517][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:19:24,532][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:19:29,534][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:19:34,538][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:19:39,540][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:19:44,543][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:19:49,544][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:19:54,548][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:19:59,553][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:20:04,559][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:20:09,563][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:20:14,565][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:20:19,569][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:20:24,574][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:20:29,575][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:20:34,588][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:20:39,593][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:20:44,606][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:20:49,611][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:20:54,613][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:20:59,616][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:21:04,619][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:21:09,623][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:21:14,625][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:21:19,630][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:21:24,633][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:21:29,636][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:21:34,638][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:21:39,639][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:21:44,642][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:21:49,645][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:21:50,019][ ERROR][PID:2592623] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 13:21:54,649][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:21:59,650][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:22:00,021][WARNING][PID:2592623] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-03-26 13:22:04,654][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:22:09,656][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:22:10,022][  INFO][PID:2592623] Downloading the builder-live.log file, attempt 2
[2026-03-26 13:22:10,033][  INFO][PID:2592623] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.229.130.110 copr-rpmbuild-log
[2026-03-26 13:22:14,664][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:22:19,665][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:22:24,666][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:22:29,668][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:22:34,669][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:22:39,670][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:22:44,671][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:22:49,674][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:22:54,676][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:22:59,677][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:23:04,679][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:23:09,680][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:23:10,125][ ERROR][PID:2592623] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 13:23:14,681][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:23:19,682][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:23:20,135][WARNING][PID:2592623] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-03-26 13:23:24,683][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:23:29,711][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:23:30,136][  INFO][PID:2592623] Downloading the builder-live.log file, attempt 3
[2026-03-26 13:23:30,139][  INFO][PID:2592623] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.229.130.110 copr-rpmbuild-log
[2026-03-26 13:23:34,715][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:23:39,718][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:23:44,754][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:23:49,795][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:23:54,805][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:23:59,825][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:24:04,833][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:24:09,842][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:24:14,845][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:24:19,846][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:24:24,850][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:24:29,856][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:24:30,223][ ERROR][PID:2592623] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 13:24:34,858][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:24:39,860][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:24:40,225][WARNING][PID:2592623] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-03-26 13:24:44,880][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:24:49,882][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:24:50,227][  INFO][PID:2592623] Downloading the builder-live.log file, attempt 4
[2026-03-26 13:24:50,243][  INFO][PID:2592623] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.229.130.110 copr-rpmbuild-log
[2026-03-26 13:24:54,919][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:24:59,920][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:25:04,923][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:25:09,925][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:25:14,926][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:25:19,927][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:25:24,930][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:25:29,931][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:25:34,960][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:25:39,961][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:25:44,993][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:25:50,006][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:25:50,297][ ERROR][PID:2592623] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-26 13:25:55,007][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:26:00,008][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:26:00,309][ ERROR][PID:2592623] Giving up for unstable SSH, failures: 1774531320.021261, 1774531400.13536, 1774531480.2251904, 1774531560.309695
[2026-03-26 13:26:00,311][ ERROR][PID:2592623] Re-try request for task on 'ResallocHost, ticket_id=2040211, hostname=3.229.130.110, name=aws_x86_64_spot_prod_04288181_20260326_131342, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-03-26 13:26:00,312][  INFO][PID:2592623] Releasing VM back to pool
[2026-03-26 13:26:00,332][  INFO][PID:2592623] Retry #2 (on other host)
[2026-03-26 13:26:00,336][  INFO][PID:2592623] Checking that https://download.copr.fedorainfracloud.org/results/andersrh/SonicDE-devel/fedora-44-x86_64//repodata/repomd.xml exists
[2026-03-26 13:26:01,298][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:26:01,301][  INFO][PID:2592623] VM allocation process starts
[2026-03-26 13:26:01,329][  INFO][PID:2592623] Trying to allocate VM: ResallocHost, ticket_id=2040460, requested_tags=['copr_builder', 'arch_x86_64']
[2026-03-26 13:26:04,356][  INFO][PID:2592623] Allocated host ResallocHost, ticket_id=2040460, hostname=44.202.41.192, name=aws_x86_64_normalreserved_prod_04288238_20260326_131450, requested_tags=['copr_builder', 'arch_x86_64']
[2026-03-26 13:26:04,357][  INFO][PID:2592623] Allocating ssh connection to builder
[2026-03-26 13:26:04,358][  INFO][PID:2592623] Checking that builder machine is OK
[2026-03-26 13:26:04,709][  INFO][PID:2592623] Running remote command: copr-builder-ready fedora-44-x86_64
[2026-03-26 13:26:04,927][  INFO][PID:2592623] Red Hat subscription not needed for fedora-44-x86_64
Builder is ready to be used

[2026-03-26 13:26:04,927][  INFO][PID:2592623] Filling build.info file with builder info
[2026-03-26 13:26:04,940][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:26:04,941][  INFO][PID:2592623] Not re-notifying FE and msg buses for the new host.
[2026-03-26 13:26:04,942][  INFO][PID:2592623] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10263869-fedora-44-x86_64 --chroot fedora-44-x86_64 --detached
[2026-03-26 13:26:05,271][  INFO][PID:2592623] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '44.202.41.192' (ED25519) to the list of known hosts.


[2026-03-26 13:26:05,272][  INFO][PID:2592623] Downloading the builder-live.log file, attempt 1
[2026-03-26 13:26:05,273][  INFO][PID:2592623] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.202.41.192 copr-rpmbuild-log
[2026-03-26 13:26:10,273][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:26:15,337][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:26:20,338][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:26:25,716][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:26:30,721][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:26:35,726][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:26:40,727][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:26:45,729][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:26:50,731][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:26:55,732][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:27:00,733][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:27:05,736][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:27:10,738][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:27:15,740][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:27:20,741][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:27:25,748][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:27:30,750][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:27:35,764][  INFO][PID:2592623] Periodic builder liveness probe: alive
[2026-03-26 13:27:35,765][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:27:40,766][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:27:45,767][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:27:50,770][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:27:55,771][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:28:00,772][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:28:05,774][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:28:10,776][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:28:15,779][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:28:20,781][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:28:25,782][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:28:30,866][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:28:35,869][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:28:40,870][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:28:46,136][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:28:51,143][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:28:56,145][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:29:01,147][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:29:06,152][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:29:11,154][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:29:16,158][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:29:21,164][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:29:26,166][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:29:31,171][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:29:36,176][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:29:41,178][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:29:46,181][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:29:51,185][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:29:56,203][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:30:01,229][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:30:06,240][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:30:11,241][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:30:16,242][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:30:21,244][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:30:26,245][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:30:31,245][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:30:36,262][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:30:41,263][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:30:46,265][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:30:51,266][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:30:56,270][  INFO][PID:2592623] Checking for cancel request
[2026-03-26 13:30:56,271][  INFO][PID:2592623] calling cancel callback, and waiting
[2026-03-26 13:30:59,872][  INFO][PID:2592623] Cancel request succeeded
out:
success
err:
Warning: Permanently added '44.202.41.192' (ED25519) to the list of known hosts.

[2026-03-26 13:30:59,873][ ERROR][PID:2592623] Build was canceled
[2026-03-26 13:30:59,874][  INFO][PID:2592623] Releasing VM back to pool
[2026-03-26 13:30:59,891][WARNING][PID:2592623] Switching not-finished job state to 'failed'
[2026-03-26 13:30:59,891][  INFO][PID:2592623] Worker failed build, took 294.94941544532776
[2026-03-26 13:30:59,892][  INFO][PID:2592623] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "andersrh",
            "project_name": "SonicDE-devel",
            "project_dirname": "SonicDE-devel",
            "submitter": "andersrh",
            "ended_on": 1774531859.8911734,
            "started_on": 1774531564.941758,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-44-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "10263869-fedora-44-x86_64",
            "build_id": 10263869,
            "package_name": "sonic-frameworks-io",
            "package_version": "6.24.0-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/andersrh/SonicDE-devel/sonic-frameworks-io",
            "git_hash": "35f0fdaaee02f7f5b95e0ad7bb258fa2be80d390",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "andersrh/SonicDE-devel--andersrh",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 1,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/andersrh/SonicDE-devel/fedora-44-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@xlibre/xlibre-xserver/fedora-44-x86_64/",
                    "id": "copr_xlibre_xlibre_xserver",
                    "name": "Additional repo copr_xlibre_xlibre_xserver"
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/andersrh/SonicDE-devel",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/andersrh/SonicDE-devel",
            "result_dir": "10263869-sonic-frameworks-io",
            "built_packages": "",
            "id": 10263869,
            "mockchain_macros": {
                "copr_username": "andersrh",
                "copr_projectname": "SonicDE-devel",
                "vendor": "Fedora Project COPR (andersrh/SonicDE-devel)"
            }
        }
    ]
}
[2026-03-26 13:30:59,948][  INFO][PID:2592623] Sending fedora-messaging bus message in build.end
[2026-03-26 13:30:59,992][  INFO][PID:2592623] Compressing /var/lib/copr/public_html/results/andersrh/SonicDE-devel/fedora-44-x86_64/10263869-sonic-frameworks-io/builder-live.log by gzip
[2026-03-26 13:30:59,996][  INFO][PID:2592623] Running command 'gzip /var/lib/copr/public_html/results/andersrh/SonicDE-devel/fedora-44-x86_64/10263869-sonic-frameworks-io/builder-live.log' as PID 2777987
[2026-03-26 13:31:00,167][  INFO][PID:2592623] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/andersrh/SonicDE-devel/fedora-44-x86_64/10263869-sonic-frameworks-io/builder-live.log)
[2026-03-26 13:31:00,168][  INFO][PID:2592623] Compressing /var/lib/copr/public_html/results/andersrh/SonicDE-devel/fedora-44-x86_64/10263869-sonic-frameworks-io/backend.log by gzip
[2026-03-26 13:31:00,169][  INFO][PID:2592623] Running command 'gzip /var/lib/copr/public_html/results/andersrh/SonicDE-devel/fedora-44-x86_64/10263869-sonic-frameworks-io/backend.log' as PID 2778006