[2026-07-14 12:09:27,695][ INFO][PID:2759768] Marking build as starting [2026-07-14 12:09:27,779][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:09:27,789][ INFO][PID:2759768] VM allocation process starts [2026-07-14 12:09:27,831][ INFO][PID:2759768] Trying to allocate VM: ResallocHost, ticket_id=3895665, requested_tags=['arch_x86_64', 'copr_builder'] [2026-07-14 12:09:32,842][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:09:37,844][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:09:42,846][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:09:47,851][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:09:52,859][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:09:55,940][ INFO][PID:2759768] Allocated host ResallocHost, ticket_id=3895665, hostname=98.92.238.164, name=aws_x86_64_spot_prod_13069924_20260714_120356, requested_tags=['arch_x86_64', 'copr_builder'] [2026-07-14 12:09:55,942][ INFO][PID:2759768] Allocating ssh connection to builder [2026-07-14 12:09:55,944][ INFO][PID:2759768] Checking that builder machine is OK [2026-07-14 12:09:56,327][ INFO][PID:2759768] Installed copr-rpmbuild version: 1.8 [2026-07-14 12:09:56,331][ INFO][PID:2759768] Running remote command: copr-builder-ready srpm-builds [2026-07-14 12:09:56,531][ INFO][PID:2759768] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-07-14 12:09:56,532][ INFO][PID:2759768] Filling build.info file with builder info [2026-07-14 12:09:56,533][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:09:56,537][ INFO][PID:2759768] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "@theforeman", "project_name": "develop", "project_dirname": "develop", "submitter": "packit", "ended_on": null, "started_on": 1784030996.536939, "submitted_on": null, "status": 3, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10717293", "build_id": 10717293, "package_name": null, "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"\\n#!/bin/sh\\n\\ngit config --global user.email \\\"hello@packit.dev\\\"\\ngit config --global user.name \\\"Packit\\\"\\nresultdir=$PWD\\npackit -d prepare-sources --result-dir \\\"$resultdir\\\" --ref master --job-config-index 1 --merged-ref '709bb12862a3915a1ed2efde78dafc5ca6efaa25' --no-create-symlinks https://github.com/theforeman/foreman_remote_execution\\n\\n\", \"chroot\": \"fedora-44-x86_64\", \"builddeps\": \"packit wget rubygems\", \"resultdir\": null, \"repos\": \"copr://packit/packit-stable\"}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "@theforeman/develop--packit", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/packit/packit-stable/fedora-44-x86_64/", "id": "copr_packit_packit_stable", "name": "Additional repo copr_packit_packit_stable" } ], "background": false, "distributions_in_build": [ "rhel-9" ], "distributions_in_project": [ "rhel-9" ], "destdir": "/var/lib/copr/public_html/results/@theforeman/develop", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@theforeman/develop", "result_dir": "10717293", "built_packages": "", "tags": [ "arch_x86_64" ], "rpmeta_prediction": null, "id": 10717293, "mockchain_macros": { "copr_username": "@theforeman", "copr_projectname": "develop", "vendor": "Fedora Project COPR (@theforeman/develop)" } } ] } [2026-07-14 12:09:56,594][ INFO][PID:2759768] Sending fedora-messaging bus message in build.start [2026-07-14 12:09:57,337][ INFO][PID:2759768] Sending fedora-messaging bus message in chroot.start [2026-07-14 12:09:57,385][ INFO][PID:2759768] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10717293 --detached [2026-07-14 12:09:58,348][ INFO][PID:2759768] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '98.92.238.164' (ED25519) to the list of known hosts. [2026-07-14 12:09:58,350][ INFO][PID:2759768] Downloading the builder-live.log file, attempt 1 [2026-07-14 12:09:58,352][ INFO][PID:2759768] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.92.238.164 copr-rpmbuild-log [2026-07-14 12:09:58,362][ ERROR][PID:2759768] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-07-14 12:10:03,368][ INFO][PID:2759768] Periodic builder liveness probe: alive [2026-07-14 12:10:03,369][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:10:08,371][WARNING][PID:2759768] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-07-14 12:10:08,371][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:10:13,373][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:10:18,372][ INFO][PID:2759768] Downloading the builder-live.log file, attempt 2 [2026-07-14 12:10:18,375][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:10:18,376][ INFO][PID:2759768] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.92.238.164 copr-rpmbuild-log [2026-07-14 12:10:23,380][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:10:28,382][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:10:33,386][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:10:38,388][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:10:43,389][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:10:48,391][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:10:53,394][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:10:58,395][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:11:03,397][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:11:08,398][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:11:13,400][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:11:18,402][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:11:18,460][ ERROR][PID:2759768] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-07-14 12:11:23,405][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:11:28,413][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:11:28,477][WARNING][PID:2759768] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-07-14 12:11:33,415][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:11:38,416][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:11:38,478][ INFO][PID:2759768] Downloading the builder-live.log file, attempt 3 [2026-07-14 12:11:38,480][ INFO][PID:2759768] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.92.238.164 copr-rpmbuild-log [2026-07-14 12:11:43,418][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:11:48,419][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:11:53,422][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:11:58,423][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:12:03,425][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:12:08,427][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:12:13,430][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:12:18,432][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:12:23,436][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:12:28,438][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:12:33,440][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:12:38,443][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:12:38,552][ ERROR][PID:2759768] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-07-14 12:12:43,447][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:12:48,449][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:12:48,606][WARNING][PID:2759768] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-07-14 12:12:53,451][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:12:58,459][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:12:58,615][ INFO][PID:2759768] Downloading the builder-live.log file, attempt 4 [2026-07-14 12:12:58,626][ INFO][PID:2759768] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.92.238.164 copr-rpmbuild-log [2026-07-14 12:13:03,468][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:13:08,471][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:13:13,473][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:13:18,474][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:13:23,476][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:13:28,477][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:13:33,480][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:13:38,483][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:13:43,484][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:13:48,486][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:13:53,487][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:13:58,488][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:13:58,704][ ERROR][PID:2759768] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-07-14 12:14:03,490][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:14:08,491][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:14:08,723][ ERROR][PID:2759768] Giving up for unstable SSH, failures: 1784031008.3711147, 1784031088.4776294, 1784031168.606632, 1784031248.723538 [2026-07-14 12:14:08,741][ ERROR][PID:2759768] Re-try request for task on 'ResallocHost, ticket_id=3895665, hostname=98.92.238.164, name=aws_x86_64_spot_prod_13069924_20260714_120356, requested_tags=['arch_x86_64', 'copr_builder']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-07-14 12:14:08,763][ INFO][PID:2759768] VM Release request [2026-07-14 12:14:08,790][ INFO][PID:2759768] Retry #1 (on other host) [2026-07-14 12:14:08,791][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:14:08,793][ INFO][PID:2759768] VM allocation process starts [2026-07-14 12:14:08,813][ INFO][PID:2759768] Trying to allocate VM: ResallocHost, ticket_id=3895732, requested_tags=['arch_x86_64', 'copr_builder'] [2026-07-14 12:14:13,815][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:14:18,818][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:14:23,821][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:14:28,824][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:14:33,825][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:14:38,831][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:14:43,834][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:14:48,846][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:14:53,854][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:14:58,855][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:15:03,864][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:15:08,869][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:15:13,876][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:15:18,878][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:15:23,881][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:15:28,883][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:15:33,885][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:15:38,887][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:15:43,889][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:15:48,891][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:15:53,897][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:15:58,898][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:16:03,900][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:16:08,905][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:16:13,906][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:16:18,909][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:16:23,912][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:16:26,961][ INFO][PID:2759768] Allocated host ResallocHost, ticket_id=3895732, hostname=54.166.11.107, name=aws_x86_64_reserved_prod_13070140_20260714_121450, requested_tags=['arch_x86_64', 'copr_builder'] [2026-07-14 12:16:26,967][ INFO][PID:2759768] Allocating ssh connection to builder [2026-07-14 12:16:26,968][ INFO][PID:2759768] Checking that builder machine is OK [2026-07-14 12:16:27,291][ INFO][PID:2759768] Installed copr-rpmbuild version: 1.8 [2026-07-14 12:16:27,292][ INFO][PID:2759768] Running remote command: copr-builder-ready srpm-builds [2026-07-14 12:16:27,519][ INFO][PID:2759768] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-07-14 12:16:27,521][ INFO][PID:2759768] Filling build.info file with builder info [2026-07-14 12:16:27,535][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:16:27,537][ INFO][PID:2759768] Not re-notifying FE and msg buses for the new host. [2026-07-14 12:16:27,538][ INFO][PID:2759768] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10717293 --detached [2026-07-14 12:16:28,286][ INFO][PID:2759768] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '54.166.11.107' (ED25519) to the list of known hosts. [2026-07-14 12:16:28,287][ INFO][PID:2759768] Downloading the builder-live.log file, attempt 1 [2026-07-14 12:16:28,289][ INFO][PID:2759768] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.166.11.107 copr-rpmbuild-log [2026-07-14 12:16:33,288][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:16:38,307][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:16:43,309][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:16:48,310][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:16:53,312][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:16:58,313][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:17:03,315][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:17:08,316][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:17:13,317][ INFO][PID:2759768] Checking for cancel request [2026-07-14 12:17:16,605][ INFO][PID:2759768] Downloading results from builder [2026-07-14 12:17:16,605][ INFO][PID:2759768] rsyncing of mockbuilder@54.166.11.107:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@theforeman/develop/srpm-builds/10717293 started [2026-07-14 12:17:16,607][ INFO][PID:2759768] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@54.166.11.107:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@theforeman/develop/srpm-builds/10717293/ &> /var/lib/copr/public_html/results/@theforeman/develop/srpm-builds/10717293/build-10717293.rsync.log [2026-07-14 12:17:17,016][ INFO][PID:2759768] rsyncing finished. [2026-07-14 12:17:17,017][ INFO][PID:2759768] VM Release request [2026-07-14 12:17:17,027][ INFO][PID:2759768] Searching for 'success' file in resultdir [2026-07-14 12:17:17,028][ INFO][PID:2759768] Getting build details [2026-07-14 12:17:17,028][ INFO][PID:2759768] Retrieving SRPM info from /var/lib/copr/public_html/results/@theforeman/develop/srpm-builds/10717293 [2026-07-14 12:17:17,029][ INFO][PID:2759768] SRPM URL: https://download.copr.fedorainfracloud.org/results/@theforeman/develop/srpm-builds/10717293/rubygem-foreman_remote_execution-16.7.0-1.20260714121712414565.master.2.g709bb12.src.rpm [2026-07-14 12:17:17,030][ INFO][PID:2759768] build details: {'srpm_url': 'https://download.copr.fedorainfracloud.org/results/@theforeman/develop/srpm-builds/10717293/rubygem-foreman_remote_execution-16.7.0-1.20260714121712414565.master.2.g709bb12.src.rpm', 'pkg_name': 'rubygem-foreman_remote_execution', 'pkg_version': '16.7.0-1.20260714121712414565.master.2.g709bb12'} [2026-07-14 12:17:17,031][ INFO][PID:2759768] Finished build: id=10717293 failed=False timeout=108000 destdir=/var/lib/copr/public_html/results/@theforeman/develop chroot=srpm-builds [2026-07-14 12:17:17,033][ INFO][PID:2759768] Worker succeeded build, took 49.49629211425781 [2026-07-14 12:17:17,034][ INFO][PID:2759768] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "@theforeman", "project_name": "develop", "project_dirname": "develop", "submitter": "packit", "ended_on": 1784031437.0337033, "started_on": 1784031387.5374112, "submitted_on": null, "status": 1, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10717293", "build_id": 10717293, "package_name": null, "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"\\n#!/bin/sh\\n\\ngit config --global user.email \\\"hello@packit.dev\\\"\\ngit config --global user.name \\\"Packit\\\"\\nresultdir=$PWD\\npackit -d prepare-sources --result-dir \\\"$resultdir\\\" --ref master --job-config-index 1 --merged-ref '709bb12862a3915a1ed2efde78dafc5ca6efaa25' --no-create-symlinks https://github.com/theforeman/foreman_remote_execution\\n\\n\", \"chroot\": \"fedora-44-x86_64\", \"builddeps\": \"packit wget rubygems\", \"resultdir\": null, \"repos\": \"copr://packit/packit-stable\"}", "pkg_name": "rubygem-foreman_remote_execution", "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": "https://download.copr.fedorainfracloud.org/results/@theforeman/develop/srpm-builds/10717293/rubygem-foreman_remote_execution-16.7.0-1.20260714121712414565.master.2.g709bb12.src.rpm", "uses_devel_repo": null, "sandbox": "@theforeman/develop--packit", "results": { "architecture_specific_tags": { "rhel-9": { "buildarch": [ "noarch" ] } }, "name": "rubygem-foreman_remote_execution", "epoch": null, "version": "16.7.0", "release": "1.20260714121712414565.master.2.g709bb12" }, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/packit/packit-stable/fedora-44-x86_64/", "id": "copr_packit_packit_stable", "name": "Additional repo copr_packit_packit_stable" } ], "background": false, "distributions_in_build": [ "rhel-9" ], "distributions_in_project": [ "rhel-9" ], "destdir": "/var/lib/copr/public_html/results/@theforeman/develop", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@theforeman/develop", "result_dir": "10717293", "built_packages": "", "tags": [ "arch_x86_64" ], "rpmeta_prediction": null, "pkg_version": "16.7.0-1.20260714121712414565.master.2.g709bb12", "id": 10717293, "mockchain_macros": { "copr_username": "@theforeman", "copr_projectname": "develop", "vendor": "Fedora Project COPR (@theforeman/develop)" } } ] } [2026-07-14 12:17:17,083][ INFO][PID:2759768] Sending fedora-messaging bus message in build.end [2026-07-14 12:17:17,111][ INFO][PID:2759768] Compressing /var/lib/copr/public_html/results/@theforeman/develop/srpm-builds/10717293/builder-live.log by gzip [2026-07-14 12:17:17,112][ INFO][PID:2759768] Running command 'gzip /var/lib/copr/public_html/results/@theforeman/develop/srpm-builds/10717293/builder-live.log' as PID 2836308 [2026-07-14 12:17:17,159][ INFO][PID:2759768] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@theforeman/develop/srpm-builds/10717293/builder-live.log) [2026-07-14 12:17:17,160][ INFO][PID:2759768] Compressing /var/lib/copr/public_html/results/@theforeman/develop/srpm-builds/10717293/backend.log by gzip [2026-07-14 12:17:17,161][ INFO][PID:2759768] Running command 'gzip /var/lib/copr/public_html/results/@theforeman/develop/srpm-builds/10717293/backend.log' as PID 2836311