[2025-09-19 11:24:44,399][  INFO][PID:722361] Marking build as starting
[2025-09-19 11:24:44,568][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:24:44,571][  INFO][PID:722361] VM allocation process starts
[2025-09-19 11:24:44,608][  INFO][PID:722361] Trying to allocate VM: ResallocHost, ticket_id=5142573, requested_tags=['copr_builder', 'arch_noarch']
[2025-09-19 11:24:49,610][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:24:54,611][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:24:59,613][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:25:04,614][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:25:09,615][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:25:12,820][  INFO][PID:722361] Allocated host ResallocHost, ticket_id=5142573, hostname=2620:52:3:1:dead:beef:cafe:c29d, name=copr_p09_01_prod_06830667_20250919_111731, requested_tags=['copr_builder', 'arch_noarch']
[2025-09-19 11:25:12,821][  INFO][PID:722361] Allocating ssh connection to builder
[2025-09-19 11:25:12,824][  INFO][PID:722361] Checking that builder machine is OK
[2025-09-19 11:25:13,548][  INFO][PID:722361] Running remote command: copr-builder-ready srpm-builds
[2025-09-19 11:25:14,042][  INFO][PID:722361] Red Hat subscription not needed for srpm-builds
Builder is ready to be used

[2025-09-19 11:25:14,042][  INFO][PID:722361] Filling build.info file with builder info
[2025-09-19 11:25:14,043][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:25:14,044][  INFO][PID:722361] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "@python",
            "project_name": "python3.14",
            "project_dirname": "python3.14",
            "submitter": "https://src.fedoraproject.org/user/ksurma",
            "ended_on": null,
            "started_on": 1758281114.0444884,
            "submitted_on": null,
            "status": 3,
            "chroot": "srpm-builds",
            "arch": null,
            "buildroot_pkgs": null,
            "task_id": "9577078",
            "build_id": 9577078,
            "package_name": "python-dict-sorted",
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 10,
            "source_json": "{\"clone_url\": \"https://src.fedoraproject.org/rpms/python-dict-sorted\", \"distgit\": \"fedora\", \"committish\": \"1bdcc38ca487ecedf37acf28981281f3c4352b6c\"}",
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": null,
            "sandbox": "@python/python3.14--https://src.fedoraproject.org/user/ksurma",
            "results": null,
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [],
            "background": true,
            "destdir": "/var/lib/copr/public_html/results/@python/python3.14",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@python/python3.14",
            "result_dir": "09577078",
            "built_packages": "",
            "tags": [
                "arch_noarch"
            ],
            "id": 9577078,
            "mockchain_macros": {
                "copr_username": "@python",
                "copr_projectname": "python3.14",
                "vendor": "Fedora Project COPR (@python/python3.14)"
            }
        }
    ]
}
[2025-09-19 11:25:14,071][  INFO][PID:722361] Sending fedora-messaging bus message in build.start
[2025-09-19 11:25:14,872][  INFO][PID:722361] Sending fedora-messaging bus message in chroot.start
[2025-09-19 11:25:14,900][  INFO][PID:722361] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9577078 --detached
[2025-09-19 11:25:15,712][  INFO][PID:722361] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c29d' (ED25519) to the list of known hosts.


[2025-09-19 11:25:15,716][  INFO][PID:722361] Downloading the builder-live.log file, attempt 1
[2025-09-19 11:25:15,719][  INFO][PID:722361] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29d copr-rpmbuild-log
[2025-09-19 11:25:20,731][  INFO][PID:722361] Periodic builder liveness probe: alive
[2025-09-19 11:25:20,732][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:25:25,734][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:25:30,737][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:25:35,739][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:25:40,740][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:25:45,745][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:25:50,746][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:25:55,747][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:26:00,749][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:26:05,754][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:26:10,755][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:26:15,756][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:26:15,813][ ERROR][PID:722361] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-19 11:26:20,761][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:26:25,762][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:26:25,814][WARNING][PID:722361] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-19 11:26:30,763][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:26:35,765][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:26:35,815][  INFO][PID:722361] Downloading the builder-live.log file, attempt 2
[2025-09-19 11:26:35,821][  INFO][PID:722361] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29d copr-rpmbuild-log
[2025-09-19 11:26:40,772][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:26:45,776][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:26:50,778][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:26:55,783][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:27:00,785][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:27:05,789][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:27:10,790][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:27:15,793][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:27:20,797][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:27:25,799][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:27:30,802][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:27:35,804][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:27:35,911][ ERROR][PID:722361] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-19 11:27:40,806][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:27:45,807][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:27:45,913][WARNING][PID:722361] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-19 11:27:50,811][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:27:55,820][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:27:55,915][  INFO][PID:722361] Downloading the builder-live.log file, attempt 3
[2025-09-19 11:27:55,922][  INFO][PID:722361] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29d copr-rpmbuild-log
[2025-09-19 11:28:00,833][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:28:05,835][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:28:10,847][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:28:15,849][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:28:20,853][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:28:25,855][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:28:30,864][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:28:35,873][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:28:40,878][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:28:45,881][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:28:50,889][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:28:55,891][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:28:56,003][ ERROR][PID:722361] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-19 11:29:00,892][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:29:05,895][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:29:06,009][WARNING][PID:722361] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-19 11:29:10,909][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:29:15,913][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:29:16,012][  INFO][PID:722361] Downloading the builder-live.log file, attempt 4
[2025-09-19 11:29:16,015][  INFO][PID:722361] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29d copr-rpmbuild-log
[2025-09-19 11:29:20,914][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:29:25,915][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:29:30,917][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:29:35,919][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:29:40,920][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:29:45,921][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:29:50,922][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:29:55,924][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:30:00,926][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:30:05,930][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:30:10,932][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:30:15,934][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:30:16,085][ ERROR][PID:722361] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-19 11:30:20,935][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:30:25,937][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:30:26,088][ ERROR][PID:722361] Giving up for unstable SSH, failures: 1758281185.814544, 1758281265.913652, 1758281346.0095222, 1758281426.0883477
[2025-09-19 11:30:26,091][ ERROR][PID:722361] Re-try request for task on 'ResallocHost, ticket_id=5142573, hostname=2620:52:3:1:dead:beef:cafe:c29d, name=copr_p09_01_prod_06830667_20250919_111731, requested_tags=['copr_builder', 'arch_noarch']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-09-19 11:30:26,092][  INFO][PID:722361] Releasing VM back to pool
[2025-09-19 11:30:26,133][  INFO][PID:722361] Retry #1 (on other host)
[2025-09-19 11:30:26,137][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:30:26,140][  INFO][PID:722361] VM allocation process starts
[2025-09-19 11:30:26,176][  INFO][PID:722361] Trying to allocate VM: ResallocHost, ticket_id=5142934, requested_tags=['copr_builder', 'arch_noarch']
[2025-09-19 11:30:31,181][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:30:36,183][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:30:41,185][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:30:46,188][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:30:51,196][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:30:56,199][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:31:01,204][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:31:06,207][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:31:11,213][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:31:16,214][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:31:21,217][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:31:26,219][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:31:31,222][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:31:36,229][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:31:41,231][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:31:46,235][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:31:51,240][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:31:56,241][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:32:01,242][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:32:06,244][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:32:11,246][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:32:14,435][  INFO][PID:722361] Allocated host ResallocHost, ticket_id=5142934, hostname=100.28.217.61, name=aws_x86_64_normal_prod_06830911_20250919_112926, requested_tags=['copr_builder', 'arch_noarch']
[2025-09-19 11:32:14,437][  INFO][PID:722361] Allocating ssh connection to builder
[2025-09-19 11:32:14,440][  INFO][PID:722361] Checking that builder machine is OK
[2025-09-19 11:32:14,800][  INFO][PID:722361] Running remote command: copr-builder-ready srpm-builds
[2025-09-19 11:32:15,011][  INFO][PID:722361] Red Hat subscription not needed for srpm-builds
Builder is ready to be used

[2025-09-19 11:32:15,013][  INFO][PID:722361] Filling build.info file with builder info
[2025-09-19 11:32:15,014][  INFO][PID:722361] Checking for cancel request
[2025-09-19 11:32:15,016][  INFO][PID:722361] Not re-notifying FE and msg buses for the new host.
[2025-09-19 11:32:15,016][  INFO][PID:722361] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9577078 --detached
[2025-09-19 11:32:15,380][  INFO][PID:722361] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '100.28.217.61' (ED25519) to the list of known hosts.


[2025-09-19 11:32:15,381][  INFO][PID:722361] Downloading the builder-live.log file, attempt 1
[2025-09-19 11:32:15,383][  INFO][PID:722361] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@100.28.217.61 copr-rpmbuild-log
[2025-09-19 11:32:17,502][  INFO][PID:722361] Downloading results from builder
[2025-09-19 11:32:17,503][  INFO][PID:722361] rsyncing of mockbuilder@100.28.217.61:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@python/python3.14/srpm-builds/09577078 started
[2025-09-19 11:32:17,504][  INFO][PID:722361] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@100.28.217.61:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@python/python3.14/srpm-builds/09577078/ &> /var/lib/copr/public_html/results/@python/python3.14/srpm-builds/09577078/build-09577078.rsync.log
[2025-09-19 11:32:17,837][  INFO][PID:722361] rsyncing finished.
[2025-09-19 11:32:17,844][  INFO][PID:722361] Releasing VM back to pool
[2025-09-19 11:32:17,880][  INFO][PID:722361] Searching for 'success' file in resultdir
[2025-09-19 11:32:17,882][  INFO][PID:722361] Getting build details
[2025-09-19 11:32:17,882][  INFO][PID:722361] Retrieving SRPM info from /var/lib/copr/public_html/results/@python/python3.14/srpm-builds/09577078
[2025-09-19 11:32:17,883][  INFO][PID:722361] SRPM URL: https://download.copr.fedorainfracloud.org/results/@python/python3.14/srpm-builds/09577078/python-dict-sorted-1.0.0-36.src.rpm
[2025-09-19 11:32:17,884][  INFO][PID:722361] build details: {'srpm_url': 'https://download.copr.fedorainfracloud.org/results/@python/python3.14/srpm-builds/09577078/python-dict-sorted-1.0.0-36.src.rpm', 'pkg_name': 'python-dict-sorted', 'pkg_version': '1.0.0-36'}
[2025-09-19 11:32:17,888][  INFO][PID:722361] Finished build: id=9577078 failed=False timeout=108000 destdir=/var/lib/copr/public_html/results/@python/python3.14 chroot=srpm-builds 
[2025-09-19 11:32:17,901][  INFO][PID:722361] Worker succeeded build, took 2.8848235607147217
[2025-09-19 11:32:17,901][  INFO][PID:722361] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "@python",
            "project_name": "python3.14",
            "project_dirname": "python3.14",
            "submitter": "https://src.fedoraproject.org/user/ksurma",
            "ended_on": 1758281537.9009671,
            "started_on": 1758281535.0161436,
            "submitted_on": null,
            "status": 1,
            "chroot": "srpm-builds",
            "arch": null,
            "buildroot_pkgs": null,
            "task_id": "9577078",
            "build_id": 9577078,
            "package_name": "python-dict-sorted",
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 10,
            "source_json": "{\"clone_url\": \"https://src.fedoraproject.org/rpms/python-dict-sorted\", \"distgit\": \"fedora\", \"committish\": \"1bdcc38ca487ecedf37acf28981281f3c4352b6c\"}",
            "pkg_name": "python-dict-sorted",
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": "https://download.copr.fedorainfracloud.org/results/@python/python3.14/srpm-builds/09577078/python-dict-sorted-1.0.0-36.src.rpm",
            "uses_devel_repo": null,
            "sandbox": "@python/python3.14--https://src.fedoraproject.org/user/ksurma",
            "results": {
                "name": "python-dict-sorted",
                "epoch": null,
                "version": "1.0.0",
                "release": "36",
                "exclusivearch": [],
                "excludearch": []
            },
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [],
            "background": true,
            "destdir": "/var/lib/copr/public_html/results/@python/python3.14",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@python/python3.14",
            "result_dir": "09577078",
            "built_packages": "",
            "tags": [
                "arch_noarch"
            ],
            "pkg_version": "1.0.0-36",
            "id": 9577078,
            "mockchain_macros": {
                "copr_username": "@python",
                "copr_projectname": "python3.14",
                "vendor": "Fedora Project COPR (@python/python3.14)"
            }
        }
    ]
}
[2025-09-19 11:32:17,956][  INFO][PID:722361] Sending fedora-messaging bus message in build.end
[2025-09-19 11:32:18,034][  INFO][PID:722361] Compressing /var/lib/copr/public_html/results/@python/python3.14/srpm-builds/09577078/builder-live.log by gzip
[2025-09-19 11:32:18,036][  INFO][PID:722361] Running command 'gzip /var/lib/copr/public_html/results/@python/python3.14/srpm-builds/09577078/builder-live.log' as PID 820264
[2025-09-19 11:32:18,041][  INFO][PID:722361] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@python/python3.14/srpm-builds/09577078/builder-live.log)
[2025-09-19 11:32:18,044][  INFO][PID:722361] Compressing /var/lib/copr/public_html/results/@python/python3.14/srpm-builds/09577078/backend.log by gzip
[2025-09-19 11:32:18,048][  INFO][PID:722361] Running command 'gzip /var/lib/copr/public_html/results/@python/python3.14/srpm-builds/09577078/backend.log' as PID 820266