[2025-12-08 12:10:48,968][  INFO][PID:2004215] Marking build as starting
[2025-12-08 12:10:49,003][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:10:49,005][  INFO][PID:2004215] VM allocation process starts
[2025-12-08 12:10:49,019][  INFO][PID:2004215] Trying to allocate VM: ResallocHost, ticket_id=215835, requested_tags=['copr_builder', 'arch_noarch']
[2025-12-08 12:10:52,070][  INFO][PID:2004215] Allocated host ResallocHost, ticket_id=215835, hostname=2620:52:3:1:dead:beef:cafe:c192, name=copr_hv_x86_64_03_prod_00209452_20251208_103123, requested_tags=['copr_builder', 'arch_noarch']
[2025-12-08 12:10:52,071][  INFO][PID:2004215] Allocating ssh connection to builder
[2025-12-08 12:10:52,071][  INFO][PID:2004215] Checking that builder machine is OK
[2025-12-08 12:10:52,656][  INFO][PID:2004215] Running remote command: copr-builder-ready srpm-builds
[2025-12-08 12:10:52,989][  INFO][PID:2004215] Red Hat subscription not needed for srpm-builds
Builder is ready to be used

[2025-12-08 12:10:52,990][  INFO][PID:2004215] Filling build.info file with builder info
[2025-12-08 12:10:53,039][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:10:53,040][  INFO][PID:2004215] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "samoht0",
            "project_name": "testing",
            "project_dirname": "testing",
            "submitter": "samoht0",
            "ended_on": null,
            "started_on": 1765195853.0407217,
            "submitted_on": null,
            "status": 3,
            "chroot": "srpm-builds",
            "arch": null,
            "buildroot_pkgs": null,
            "task_id": "9886061",
            "build_id": 9886061,
            "package_name": null,
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 2,
            "source_json": "{\"url\": \"https://copr.fedorainfracloud.org/tmp/tmpn4mt0lvx/yt-dlp-ejs.spec\", \"pkg\": \"yt-dlp-ejs.spec\", \"tmp\": \"tmpn4mt0lvx\"}",
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": null,
            "sandbox": "samoht0/testing--samoht0",
            "results": null,
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [],
            "background": false,
            "destdir": "/var/lib/copr/public_html/results/samoht0/testing",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/samoht0/testing",
            "result_dir": "09886061",
            "built_packages": "",
            "tags": [
                "arch_noarch"
            ],
            "id": 9886061,
            "mockchain_macros": {
                "copr_username": "samoht0",
                "copr_projectname": "testing",
                "vendor": "Fedora Project COPR (samoht0/testing)"
            }
        }
    ]
}
[2025-12-08 12:10:53,078][  INFO][PID:2004215] Sending fedora-messaging bus message in build.start
[2025-12-08 12:10:53,672][  INFO][PID:2004215] Sending fedora-messaging bus message in chroot.start
[2025-12-08 12:10:53,705][  INFO][PID:2004215] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9886061 --detached
[2025-12-08 12:10:54,200][  INFO][PID:2004215] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c192' (ED25519) to the list of known hosts.


[2025-12-08 12:10:54,201][  INFO][PID:2004215] Downloading the builder-live.log file, attempt 1
[2025-12-08 12:10:54,202][  INFO][PID:2004215] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c192 copr-rpmbuild-log
[2025-12-08 12:10:59,209][  INFO][PID:2004215] Periodic builder liveness probe: alive
[2025-12-08 12:10:59,210][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:11:04,211][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:11:09,212][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:11:14,213][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:11:19,214][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:11:24,215][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:11:29,216][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:11:34,217][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:11:39,219][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:11:44,220][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:11:49,221][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:11:54,222][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:11:59,223][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:12:04,224][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:12:09,225][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:12:14,226][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:12:19,227][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:12:24,229][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:12:29,230][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:12:34,231][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:12:39,232][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:12:44,234][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:12:49,235][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:12:53,880][ ERROR][PID:2004215] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-08 12:12:54,235][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:12:59,236][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:13:03,881][WARNING][PID:2004215] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-08 12:13:04,238][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:13:09,239][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:13:13,882][  INFO][PID:2004215] Downloading the builder-live.log file, attempt 2
[2025-12-08 12:13:13,884][  INFO][PID:2004215] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c192 copr-rpmbuild-log
[2025-12-08 12:13:14,240][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:13:19,241][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:13:24,242][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:13:29,243][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:13:34,244][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:13:39,245][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:13:44,245][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:13:49,608][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:13:54,614][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:13:59,626][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:14:04,627][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:14:09,628][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:14:13,892][ ERROR][PID:2004215] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-08 12:14:14,638][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:14:19,644][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:14:23,893][WARNING][PID:2004215] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-08 12:14:24,645][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:14:29,646][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:14:33,894][  INFO][PID:2004215] Downloading the builder-live.log file, attempt 3
[2025-12-08 12:14:33,895][  INFO][PID:2004215] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c192 copr-rpmbuild-log
[2025-12-08 12:14:34,647][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:14:39,648][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:14:44,649][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:14:49,652][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:14:54,653][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:14:59,654][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:15:04,655][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:15:10,010][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:15:15,032][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:15:20,039][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:15:25,847][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:15:30,849][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:15:33,963][ ERROR][PID:2004215] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-08 12:15:36,150][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:15:41,155][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:15:43,965][WARNING][PID:2004215] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-08 12:15:46,160][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:15:51,161][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:15:53,966][  INFO][PID:2004215] Downloading the builder-live.log file, attempt 4
[2025-12-08 12:15:53,969][  INFO][PID:2004215] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c192 copr-rpmbuild-log
[2025-12-08 12:15:56,162][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:16:01,409][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:16:06,490][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:16:11,491][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:16:16,492][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:16:21,493][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:16:28,344][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:16:33,353][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:16:38,353][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:16:43,354][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:16:48,355][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:16:53,356][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:16:54,037][ ERROR][PID:2004215] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-08 12:16:58,357][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:17:03,358][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:17:04,060][ ERROR][PID:2004215] Giving up for unstable SSH, failures: 1765195983.8816757, 1765196063.8933902, 1765196143.9655147, 1765196224.0603235
[2025-12-08 12:17:04,061][ ERROR][PID:2004215] Re-try request for task on 'ResallocHost, ticket_id=215835, hostname=2620:52:3:1:dead:beef:cafe:c192, name=copr_hv_x86_64_03_prod_00209452_20251208_103123, requested_tags=['copr_builder', 'arch_noarch']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-12-08 12:17:04,061][  INFO][PID:2004215] Releasing VM back to pool
[2025-12-08 12:17:04,074][  INFO][PID:2004215] Retry #1 (on other host)
[2025-12-08 12:17:04,074][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:17:04,075][  INFO][PID:2004215] VM allocation process starts
[2025-12-08 12:17:04,089][  INFO][PID:2004215] Trying to allocate VM: ResallocHost, ticket_id=216045, requested_tags=['copr_builder', 'arch_noarch']
[2025-12-08 12:17:09,090][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:17:10,116][  INFO][PID:2004215] Allocated host ResallocHost, ticket_id=216045, hostname=44.204.161.226, name=aws_aarch64_normalreserved_prod_00210847_20251208_121423, requested_tags=['copr_builder', 'arch_noarch']
[2025-12-08 12:17:10,117][  INFO][PID:2004215] Allocating ssh connection to builder
[2025-12-08 12:17:10,118][  INFO][PID:2004215] Checking that builder machine is OK
[2025-12-08 12:17:10,491][  INFO][PID:2004215] Running remote command: copr-builder-ready srpm-builds
[2025-12-08 12:17:10,708][  INFO][PID:2004215] Red Hat subscription not needed for srpm-builds
Builder is ready to be used

[2025-12-08 12:17:10,708][  INFO][PID:2004215] Filling build.info file with builder info
[2025-12-08 12:17:10,747][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:17:10,748][  INFO][PID:2004215] Not re-notifying FE and msg buses for the new host.
[2025-12-08 12:17:10,749][  INFO][PID:2004215] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9886061 --detached
[2025-12-08 12:17:11,137][  INFO][PID:2004215] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '44.204.161.226' (ED25519) to the list of known hosts.


[2025-12-08 12:17:11,138][  INFO][PID:2004215] Downloading the builder-live.log file, attempt 1
[2025-12-08 12:17:11,140][  INFO][PID:2004215] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.204.161.226 copr-rpmbuild-log
[2025-12-08 12:17:16,138][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:17:21,139][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:17:26,140][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:17:31,141][  INFO][PID:2004215] Checking for cancel request
[2025-12-08 12:17:35,237][  INFO][PID:2004215] Downloading results from builder
[2025-12-08 12:17:35,237][  INFO][PID:2004215] rsyncing of mockbuilder@44.204.161.226:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/samoht0/testing/srpm-builds/09886061 started
[2025-12-08 12:17:35,238][  INFO][PID:2004215] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@44.204.161.226:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/samoht0/testing/srpm-builds/09886061/ &> /var/lib/copr/public_html/results/samoht0/testing/srpm-builds/09886061/build-09886061.rsync.log
[2025-12-08 12:17:35,523][  INFO][PID:2004215] rsyncing finished.
[2025-12-08 12:17:35,523][  INFO][PID:2004215] Releasing VM back to pool
[2025-12-08 12:17:35,534][  INFO][PID:2004215] Searching for 'success' file in resultdir
[2025-12-08 12:17:35,535][  INFO][PID:2004215] Getting build details
[2025-12-08 12:17:35,536][  INFO][PID:2004215] Retrieving SRPM info from /var/lib/copr/public_html/results/samoht0/testing/srpm-builds/09886061
[2025-12-08 12:17:35,536][  INFO][PID:2004215] SRPM URL: https://download.copr.fedorainfracloud.org/results/samoht0/testing/srpm-builds/09886061/yt-dlp-ejs-0.3.2-1.src.rpm
[2025-12-08 12:17:35,537][  INFO][PID:2004215] build details: {'srpm_url': 'https://download.copr.fedorainfracloud.org/results/samoht0/testing/srpm-builds/09886061/yt-dlp-ejs-0.3.2-1.src.rpm', 'pkg_name': 'yt-dlp-ejs', 'pkg_version': '0.3.2-1'}
[2025-12-08 12:17:35,537][  INFO][PID:2004215] Finished build: id=9886061 failed=False timeout=108000 destdir=/var/lib/copr/public_html/results/samoht0/testing chroot=srpm-builds 
[2025-12-08 12:17:35,539][  INFO][PID:2004215] Worker succeeded build, took 24.790656566619873
[2025-12-08 12:17:35,540][  INFO][PID:2004215] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "samoht0",
            "project_name": "testing",
            "project_dirname": "testing",
            "submitter": "samoht0",
            "ended_on": 1765196255.5395455,
            "started_on": 1765196230.748889,
            "submitted_on": null,
            "status": 1,
            "chroot": "srpm-builds",
            "arch": null,
            "buildroot_pkgs": null,
            "task_id": "9886061",
            "build_id": 9886061,
            "package_name": null,
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 2,
            "source_json": "{\"url\": \"https://copr.fedorainfracloud.org/tmp/tmpn4mt0lvx/yt-dlp-ejs.spec\", \"pkg\": \"yt-dlp-ejs.spec\", \"tmp\": \"tmpn4mt0lvx\"}",
            "pkg_name": "yt-dlp-ejs",
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": "https://download.copr.fedorainfracloud.org/results/samoht0/testing/srpm-builds/09886061/yt-dlp-ejs-0.3.2-1.src.rpm",
            "uses_devel_repo": null,
            "sandbox": "samoht0/testing--samoht0",
            "results": {
                "name": "yt-dlp-ejs",
                "epoch": null,
                "version": "0.3.2",
                "release": "1",
                "exclusivearch": [],
                "excludearch": []
            },
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [],
            "background": false,
            "destdir": "/var/lib/copr/public_html/results/samoht0/testing",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/samoht0/testing",
            "result_dir": "09886061",
            "built_packages": "",
            "tags": [
                "arch_noarch"
            ],
            "pkg_version": "0.3.2-1",
            "id": 9886061,
            "mockchain_macros": {
                "copr_username": "samoht0",
                "copr_projectname": "testing",
                "vendor": "Fedora Project COPR (samoht0/testing)"
            }
        }
    ]
}
[2025-12-08 12:17:35,588][  INFO][PID:2004215] Sending fedora-messaging bus message in build.end
[2025-12-08 12:17:35,616][  INFO][PID:2004215] Compressing /var/lib/copr/public_html/results/samoht0/testing/srpm-builds/09886061/builder-live.log by gzip
[2025-12-08 12:17:35,617][  INFO][PID:2004215] Running command 'gzip /var/lib/copr/public_html/results/samoht0/testing/srpm-builds/09886061/builder-live.log' as PID 2051574
[2025-12-08 12:17:35,619][  INFO][PID:2004215] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/samoht0/testing/srpm-builds/09886061/builder-live.log)
[2025-12-08 12:17:35,619][  INFO][PID:2004215] Compressing /var/lib/copr/public_html/results/samoht0/testing/srpm-builds/09886061/backend.log by gzip
[2025-12-08 12:17:35,620][  INFO][PID:2004215] Running command 'gzip /var/lib/copr/public_html/results/samoht0/testing/srpm-builds/09886061/backend.log' as PID 2051575