[2025-08-25 17:49:49,535][  INFO][PID:3326992] Marking build as starting
[2025-08-25 17:49:49,618][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:49:49,619][  INFO][PID:3326992] VM allocation process starts
[2025-08-25 17:49:49,637][  INFO][PID:3326992] Trying to allocate VM: ResallocHost, ticket_id=4712788, requested_tags=['copr_builder', 'arch_x86_64']
[2025-08-25 17:49:52,661][  INFO][PID:3326992] Allocated host ResallocHost, ticket_id=4712788, hostname=3.85.56.237, name=aws_x86_64_spot_prod_06418380_20250825_174528, requested_tags=['copr_builder', 'arch_x86_64']
[2025-08-25 17:49:52,662][  INFO][PID:3326992] Allocating ssh connection to builder
[2025-08-25 17:49:52,662][  INFO][PID:3326992] Checking that builder machine is OK
[2025-08-25 17:49:52,998][  INFO][PID:3326992] Running remote command: copr-builder-ready fedora-41-x86_64
[2025-08-25 17:49:53,248][  INFO][PID:3326992] Red Hat subscription not needed for fedora-41-x86_64
Builder is ready to be used

[2025-08-25 17:49:53,249][  INFO][PID:3326992] Filling build.info file with builder info
[2025-08-25 17:49:53,249][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:49:53,251][  INFO][PID:3326992] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@copr",
            "project_name": "PyPI",
            "project_dirname": "PyPI",
            "submitter": "ksurma",
            "ended_on": null,
            "started_on": 1756144193.2507656,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-41-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "9487106-fedora-41-x86_64",
            "build_id": 9487106,
            "package_name": "python-zou",
            "package_version": "0.20.67-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@copr/PyPI/python-zou",
            "git_hash": "75a206ae0c4033a60e4429c84b122c7f82e57e60",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@copr/PyPI--ksurma",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI/fedora-41-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@copr/PyPI",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI",
            "result_dir": "09487106-python-zou",
            "built_packages": "",
            "id": 9487106,
            "mockchain_macros": {
                "copr_username": "@copr",
                "copr_projectname": "PyPI",
                "vendor": "Fedora Project COPR (@copr/PyPI)"
            }
        }
    ]
}
[2025-08-25 17:49:53,280][  INFO][PID:3326992] Sending fedora-messaging bus message in build.start
[2025-08-25 17:49:54,339][  INFO][PID:3326992] Sending fedora-messaging bus message in chroot.start
[2025-08-25 17:49:54,368][  INFO][PID:3326992] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9487106-fedora-41-x86_64 --chroot fedora-41-x86_64 --detached
[2025-08-25 17:49:54,677][  INFO][PID:3326992] Downloading the builder-live.log file, attempt 1
[2025-08-25 17:49:54,679][  INFO][PID:3326992] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.85.56.237 copr-rpmbuild-log
[2025-08-25 17:49:56,883][ ERROR][PID:3326992] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-08-25 17:49:59,690][  INFO][PID:3326992] Periodic builder liveness probe: alive
[2025-08-25 17:49:59,691][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:50:04,693][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:50:06,884][WARNING][PID:3326992] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-08-25 17:50:09,695][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:50:14,699][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:50:16,885][  INFO][PID:3326992] Downloading the builder-live.log file, attempt 2
[2025-08-25 17:50:16,886][  INFO][PID:3326992] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.85.56.237 copr-rpmbuild-log
[2025-08-25 17:50:19,705][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:50:24,707][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:50:29,708][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:50:34,710][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:50:39,710][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:50:44,711][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:50:49,712][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:50:54,713][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:50:59,715][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:51:04,716][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:51:09,717][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:51:14,718][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:51:16,954][ ERROR][PID:3326992] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-08-25 17:51:19,719][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:51:24,720][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:51:26,957][WARNING][PID:3326992] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-08-25 17:51:29,722][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:51:34,723][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:51:36,958][  INFO][PID:3326992] Downloading the builder-live.log file, attempt 3
[2025-08-25 17:51:36,959][  INFO][PID:3326992] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.85.56.237 copr-rpmbuild-log
[2025-08-25 17:51:39,724][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:51:44,726][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:51:49,727][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:51:54,728][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:51:59,729][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:52:04,730][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:52:09,731][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:52:14,733][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:52:19,734][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:52:24,735][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:52:29,736][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:52:34,737][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:52:37,028][ ERROR][PID:3326992] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-08-25 17:52:39,738][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:52:44,739][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:52:47,034][WARNING][PID:3326992] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-08-25 17:52:49,741][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:52:54,742][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:52:57,034][  INFO][PID:3326992] Downloading the builder-live.log file, attempt 4
[2025-08-25 17:52:57,036][  INFO][PID:3326992] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.85.56.237 copr-rpmbuild-log
[2025-08-25 17:52:59,743][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:53:04,744][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:53:09,745][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:53:14,746][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:53:19,747][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:53:24,748][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:53:29,749][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:53:34,750][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:53:39,752][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:53:44,753][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:53:49,754][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:53:54,755][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:53:57,104][ ERROR][PID:3326992] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-08-25 17:53:59,756][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:54:04,757][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:54:07,109][ ERROR][PID:3326992] Giving up for unstable SSH, failures: 1756144206.884375, 1756144286.956985, 1756144367.0340347, 1756144447.109188
[2025-08-25 17:54:07,110][ ERROR][PID:3326992] Re-try request for task on 'ResallocHost, ticket_id=4712788, hostname=3.85.56.237, name=aws_x86_64_spot_prod_06418380_20250825_174528, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-08-25 17:54:07,110][  INFO][PID:3326992] Releasing VM back to pool
[2025-08-25 17:54:07,120][  INFO][PID:3326992] Retry #1 (on other host)
[2025-08-25 17:54:07,121][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:54:07,122][  INFO][PID:3326992] VM allocation process starts
[2025-08-25 17:54:07,131][  INFO][PID:3326992] Trying to allocate VM: ResallocHost, ticket_id=4712887, requested_tags=['copr_builder', 'arch_x86_64']
[2025-08-25 17:54:10,143][  INFO][PID:3326992] Allocated host ResallocHost, ticket_id=4712887, hostname=2620:52:3:1:dead:beef:cafe:c199, name=copr_hv_x86_64_03_prod_06418309_20250825_174218, requested_tags=['copr_builder', 'arch_x86_64']
[2025-08-25 17:54:10,143][  INFO][PID:3326992] Allocating ssh connection to builder
[2025-08-25 17:54:10,143][  INFO][PID:3326992] Checking that builder machine is OK
[2025-08-25 17:54:10,687][  INFO][PID:3326992] Running remote command: copr-builder-ready fedora-41-x86_64
[2025-08-25 17:54:10,998][  INFO][PID:3326992] Red Hat subscription not needed for fedora-41-x86_64
Builder is ready to be used

[2025-08-25 17:54:10,999][  INFO][PID:3326992] Filling build.info file with builder info
[2025-08-25 17:54:10,999][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:54:11,000][  INFO][PID:3326992] Not re-notifying FE and msg buses for the new host.
[2025-08-25 17:54:11,001][  INFO][PID:3326992] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9487106-fedora-41-x86_64 --chroot fedora-41-x86_64 --detached
[2025-08-25 17:54:11,486][  INFO][PID:3326992] Downloading the builder-live.log file, attempt 1
[2025-08-25 17:54:11,487][  INFO][PID:3326992] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c199 copr-rpmbuild-log
[2025-08-25 17:54:16,486][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:54:21,487][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:54:26,488][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:54:31,489][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:54:36,490][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:54:41,492][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:54:46,494][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:54:51,496][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:54:56,497][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:55:01,498][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:55:06,500][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:55:11,501][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:55:16,502][  INFO][PID:3326992] Checking for cancel request
[2025-08-25 17:55:19,980][  INFO][PID:3326992] Downloading results from builder
[2025-08-25 17:55:19,981][  INFO][PID:3326992] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c199]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@copr/PyPI/fedora-41-x86_64/09487106-python-zou started
[2025-08-25 17:55:19,982][  INFO][PID:3326992] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c199]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@copr/PyPI/fedora-41-x86_64/09487106-python-zou/ &> /var/lib/copr/public_html/results/@copr/PyPI/fedora-41-x86_64/09487106-python-zou/build-09487106.rsync.log
[2025-08-25 17:55:20,403][  INFO][PID:3326992] rsyncing finished.
[2025-08-25 17:55:20,404][  INFO][PID:3326992] Releasing VM back to pool
[2025-08-25 17:55:20,439][  INFO][PID:3326992] Searching for 'success' file in resultdir
[2025-08-25 17:55:20,440][ ERROR][PID:3326992] Build failed: Backend process error: No success file => build failure
[2025-08-25 17:55:20,440][  INFO][PID:3326992] Finished build: id=9487106 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/@copr/PyPI chroot=fedora-41-x86_64 
[2025-08-25 17:55:20,442][ ERROR][PID:3326992] Backend process error: No success file => build failure
[2025-08-25 17:55:20,443][  INFO][PID:3326992] Worker failed build, took 69.44257760047913
[2025-08-25 17:55:20,443][  INFO][PID:3326992] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@copr",
            "project_name": "PyPI",
            "project_dirname": "PyPI",
            "submitter": "ksurma",
            "ended_on": 1756144520.4431317,
            "started_on": 1756144451.000554,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-41-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "9487106-fedora-41-x86_64",
            "build_id": 9487106,
            "package_name": "python-zou",
            "package_version": "0.20.67-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@copr/PyPI/python-zou",
            "git_hash": "75a206ae0c4033a60e4429c84b122c7f82e57e60",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@copr/PyPI--ksurma",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI/fedora-41-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@copr/PyPI",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI",
            "result_dir": "09487106-python-zou",
            "built_packages": "",
            "id": 9487106,
            "mockchain_macros": {
                "copr_username": "@copr",
                "copr_projectname": "PyPI",
                "vendor": "Fedora Project COPR (@copr/PyPI)"
            }
        }
    ]
}
[2025-08-25 17:55:20,491][  INFO][PID:3326992] Sending fedora-messaging bus message in build.end
[2025-08-25 17:55:20,515][  INFO][PID:3326992] Compressing /var/lib/copr/public_html/results/@copr/PyPI/fedora-41-x86_64/09487106-python-zou/builder-live.log by gzip
[2025-08-25 17:55:20,517][  INFO][PID:3326992] Running command 'gzip /var/lib/copr/public_html/results/@copr/PyPI/fedora-41-x86_64/09487106-python-zou/builder-live.log' as PID 3360948
[2025-08-25 17:55:20,522][  INFO][PID:3326992] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@copr/PyPI/fedora-41-x86_64/09487106-python-zou/builder-live.log)
[2025-08-25 17:55:20,523][  INFO][PID:3326992] Compressing /var/lib/copr/public_html/results/@copr/PyPI/fedora-41-x86_64/09487106-python-zou/backend.log by gzip
[2025-08-25 17:55:20,524][  INFO][PID:3326992] Running command 'gzip /var/lib/copr/public_html/results/@copr/PyPI/fedora-41-x86_64/09487106-python-zou/backend.log' as PID 3360949