[2025-10-07 18:09:37,875][  INFO][PID:3085565] Marking build as starting
[2025-10-07 18:09:37,907][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:09:37,910][  INFO][PID:3085565] VM allocation process starts
[2025-10-07 18:09:37,937][  INFO][PID:3085565] Trying to allocate VM: ResallocHost, ticket_id=5529654, requested_tags=['copr_builder', 'arch_x86_64']
[2025-10-07 18:09:42,941][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:09:47,943][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:09:52,945][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:09:57,949][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:10:02,951][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:10:07,952][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:10:12,953][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:10:17,954][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:10:22,955][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:10:27,957][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:10:32,958][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:10:37,959][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:10:42,961][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:10:47,963][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:10:52,965][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:10:57,966][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:11:02,969][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:11:07,971][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:11:12,972][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:11:17,973][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:11:22,974][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:11:27,976][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:11:32,977][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:11:37,978][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:11:42,980][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:11:47,982][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:11:52,986][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:11:57,987][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:12:02,988][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:12:07,989][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:12:12,990][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:12:17,991][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:12:22,992][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:12:27,993][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:12:32,994][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:12:37,995][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:12:42,996][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:12:47,997][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:12:52,998][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:12:57,999][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:13:03,001][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:13:08,002][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:13:13,003][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:13:18,006][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:13:23,012][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:13:28,014][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:13:33,016][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:13:38,017][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:13:43,021][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:13:48,022][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:13:53,026][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:13:56,067][  INFO][PID:3085565] Allocated host ResallocHost, ticket_id=5529654, hostname=2620:52:3:1:dead:beef:cafe:c1d9, name=copr_hv_x86_64_04_prod_07266134_20251007_181120, requested_tags=['copr_builder', 'arch_x86_64']
[2025-10-07 18:13:56,069][  INFO][PID:3085565] Allocating ssh connection to builder
[2025-10-07 18:13:56,070][  INFO][PID:3085565] Checking that builder machine is OK
[2025-10-07 18:13:56,802][  INFO][PID:3085565] Running remote command: copr-builder-ready srpm-builds
[2025-10-07 18:13:57,145][  INFO][PID:3085565] Red Hat subscription not needed for srpm-builds
Builder is ready to be used

[2025-10-07 18:13:57,146][  INFO][PID:3085565] Filling build.info file with builder info
[2025-10-07 18:13:57,147][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:13:57,148][  INFO][PID:3085565] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "bbedward",
            "project_name": "dank",
            "project_dirname": "dank",
            "submitter": "bbedward",
            "ended_on": null,
            "started_on": 1759860837.1485384,
            "submitted_on": null,
            "status": 3,
            "chroot": "srpm-builds",
            "arch": "x86_64",
            "buildroot_pkgs": null,
            "task_id": "9666107",
            "build_id": 9666107,
            "package_name": null,
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 9,
            "source_json": "{\"script\": \"#!/bin/bash\\nset -e\\n\\n# Create spec file for dms-git\\ncat > dms-git.spec << 'EOF'\\n%global debug_package %{nil}\\n\\nName:           dms-git\\nVersion:        0\\nRelease:        0%{?dist}\\nSummary:        A Quickshell-based desktop shell with Material 3 design principles\\nLicense:        GPL-3.0-only\\nURL:            https://github.com/AvengeMedia/DankMaterialShell\\n\\nSource0:        https://github.com/AvengeMedia/DankMaterialShell/archive/master.tar.gz#/DankMaterialShell-master.tar.gz\\nSource1:        https://github.com/AvengeMedia/danklinux/archive/master.tar.gz#/danklinux-master.tar.gz\\n\\nBuildRequires:  git\\nBuildRequires:  golang >= 1.19\\nBuildRequires:  make\\n\\nProvides:       dms-shell = %{version}-%{release}\\nProvides:       dms = %{version}-%{release}\\n\\n%description\\nDankMaterialShell (DMS) is a modern desktop shell built on top of Quickshell,\\nimplementing Material 3 design principles.\\n\\n%prep\\n%setup -q -n DankMaterialShell-master\\n%setup -q -T -D -a 1 -n DankMaterialShell-master\\n\\n%build\\ncd danklinux-master\\nexport CGO_CPPFLAGS=\\\"${CPPFLAGS}\\\"\\nexport CGO_CFLAGS=\\\"${CFLAGS}\\\"\\nexport CGO_CXXFLAGS=\\\"${CXXFLAGS}\\\"\\nexport CGO_LDFLAGS=\\\"${LDFLAGS}\\\"\\nexport GOFLAGS=\\\"-buildmode=pie -trimpath -ldflags=-linkmode=external -mod=readonly -modcacherw\\\"\\ngo build -o dms ./cmd/dms\\n\\n%install\\ninstall -Dm755 danklinux-master/dms %{buildroot}%{_bindir}/dms\\ninstall -dm755 %{buildroot}%{_sysconfdir}/xdg/quickshell/dms\\ncp -r . %{buildroot}%{_sysconfdir}/xdg/quickshell/dms/\\nrm -rf %{buildroot}%{_sysconfdir}/xdg/quickshell/dms/danklinux-master\\nif [ -f README.md ]; then\\n    install -Dm644 README.md %{buildroot}%{_docdir}/dms/README.md\\nfi\\nrm -rf %{buildroot}%{_sysconfdir}/xdg/quickshell/dms/.git*\\nrm -rf %{buildroot}%{_sysconfdir}/xdg/quickshell/dms/*.tar.gz\\n\\n%files\\n%{_bindir}/dms\\n%{_sysconfdir}/xdg/quickshell/dms/\\n%doc %{_docdir}/dms/\\n\\n%changelog\\n* Thu Jan 01 2025 COPR Builder <copr@fedoraproject.org> - 0-0\\n- Automatic build from git sources\\nEOF\\n\\nspectool -g -R dms-git.spec\\nrpmbuild -bs dms-git.spec\\necho \\\"SRPM files:\\\"\\nls *.src.rpm\\n\", \"chroot\": \"fedora-43-x86_64\", \"builddeps\": \"git make golang\", \"resultdir\": \"\", \"repos\": \"\"}",
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": null,
            "sandbox": "bbedward/dank--bbedward",
            "results": null,
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [],
            "background": false,
            "destdir": "/var/lib/copr/public_html/results/bbedward/dank",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/bbedward/dank",
            "result_dir": "09666107",
            "built_packages": "",
            "tags": [
                "arch_x86_64"
            ],
            "id": 9666107,
            "mockchain_macros": {
                "copr_username": "bbedward",
                "copr_projectname": "dank",
                "vendor": "Fedora Project COPR (bbedward/dank)"
            }
        }
    ]
}
[2025-10-07 18:13:57,170][  INFO][PID:3085565] Sending fedora-messaging bus message in build.start
[2025-10-07 18:13:58,028][  INFO][PID:3085565] Sending fedora-messaging bus message in chroot.start
[2025-10-07 18:13:58,058][  INFO][PID:3085565] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9666107 --detached
[2025-10-07 18:13:58,691][  INFO][PID:3085565] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c1d9' (ED25519) to the list of known hosts.


[2025-10-07 18:13:58,693][  INFO][PID:3085565] Downloading the builder-live.log file, attempt 1
[2025-10-07 18:13:58,694][  INFO][PID:3085565] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c1d9 copr-rpmbuild-log
[2025-10-07 18:14:03,710][  INFO][PID:3085565] Periodic builder liveness probe: alive
[2025-10-07 18:14:03,710][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:14:08,712][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:14:13,714][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:14:18,716][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:14:23,718][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:14:28,721][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:14:33,723][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:14:38,724][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:14:43,725][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:14:48,726][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:14:53,729][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:14:58,731][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:15:03,735][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:15:08,740][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:15:13,741][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:15:18,743][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:15:23,747][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:15:28,752][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:15:33,753][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:15:38,755][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:15:43,756][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:15:48,761][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:15:53,765][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:15:58,769][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:15:59,162][ ERROR][PID:3085565] Re-try request for task on 'ResallocHost, ticket_id=5529654, hostname=2620:52:3:1:dead:beef:cafe:c1d9, name=copr_hv_x86_64_04_prod_07266134_20251007_181120, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: copr-rpmbuild-log shouldn't exit != 0
[2025-10-07 18:15:59,164][  INFO][PID:3085565] Releasing VM back to pool
[2025-10-07 18:15:59,213][  INFO][PID:3085565] Retry #1 (on other host)
[2025-10-07 18:15:59,214][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:15:59,220][  INFO][PID:3085565] VM allocation process starts
[2025-10-07 18:15:59,245][  INFO][PID:3085565] Trying to allocate VM: ResallocHost, ticket_id=5529883, requested_tags=['copr_builder', 'arch_x86_64']
[2025-10-07 18:16:04,248][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:16:09,250][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:16:11,299][  INFO][PID:3085565] Allocated host ResallocHost, ticket_id=5529883, hostname=2620:52:3:1:dead:beef:cafe:c1d9, name=copr_hv_x86_64_04_prod_07266134_20251007_181120, requested_tags=['copr_builder', 'arch_x86_64']
[2025-10-07 18:16:11,300][  INFO][PID:3085565] Allocating ssh connection to builder
[2025-10-07 18:16:11,301][  INFO][PID:3085565] Checking that builder machine is OK
[2025-10-07 18:16:11,872][  INFO][PID:3085565] Running remote command: copr-builder-ready srpm-builds
[2025-10-07 18:16:12,208][  INFO][PID:3085565] Red Hat subscription not needed for srpm-builds
Builder is ready to be used

[2025-10-07 18:16:12,209][  INFO][PID:3085565] Filling build.info file with builder info
[2025-10-07 18:16:12,213][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:16:12,214][  INFO][PID:3085565] Not re-notifying FE and msg buses for the new host.
[2025-10-07 18:16:12,214][  INFO][PID:3085565] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9666107 --detached
[2025-10-07 18:16:12,803][  INFO][PID:3085565] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c1d9' (ED25519) to the list of known hosts.


[2025-10-07 18:16:12,804][  INFO][PID:3085565] Downloading the builder-live.log file, attempt 1
[2025-10-07 18:16:12,805][  INFO][PID:3085565] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c1d9 copr-rpmbuild-log
[2025-10-07 18:16:17,804][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:16:22,810][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:16:27,814][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:16:32,816][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:16:37,821][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:16:42,824][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:16:47,825][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:16:52,827][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:16:57,829][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:17:02,835][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:17:07,842][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:17:12,853][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:17:17,856][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:17:22,861][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:17:27,862][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:17:32,867][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:17:37,872][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:17:42,874][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:17:47,877][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:17:52,879][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:17:57,882][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:18:02,887][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:18:07,896][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:18:12,903][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:18:13,499][ ERROR][PID:3085565] Re-try request for task on 'ResallocHost, ticket_id=5529883, hostname=2620:52:3:1:dead:beef:cafe:c1d9, name=copr_hv_x86_64_04_prod_07266134_20251007_181120, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: copr-rpmbuild-log shouldn't exit != 0
[2025-10-07 18:18:13,500][  INFO][PID:3085565] Releasing VM back to pool
[2025-10-07 18:18:13,513][  INFO][PID:3085565] Retry #2 (on other host)
[2025-10-07 18:18:13,513][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:18:13,514][  INFO][PID:3085565] VM allocation process starts
[2025-10-07 18:18:13,530][  INFO][PID:3085565] Trying to allocate VM: ResallocHost, ticket_id=5529953, requested_tags=['copr_builder', 'arch_x86_64']
[2025-10-07 18:18:16,551][  INFO][PID:3085565] Allocated host ResallocHost, ticket_id=5529953, hostname=2620:52:3:1:dead:beef:cafe:c1d9, name=copr_hv_x86_64_04_prod_07266134_20251007_181120, requested_tags=['copr_builder', 'arch_x86_64']
[2025-10-07 18:18:16,552][  INFO][PID:3085565] Allocating ssh connection to builder
[2025-10-07 18:18:16,552][  INFO][PID:3085565] Checking that builder machine is OK
[2025-10-07 18:18:17,953][  INFO][PID:3085565] Running remote command: copr-builder-ready srpm-builds
[2025-10-07 18:18:18,295][  INFO][PID:3085565] Red Hat subscription not needed for srpm-builds
Builder is ready to be used

[2025-10-07 18:18:18,296][  INFO][PID:3085565] Filling build.info file with builder info
[2025-10-07 18:18:18,297][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:18:18,297][  INFO][PID:3085565] Not re-notifying FE and msg buses for the new host.
[2025-10-07 18:18:18,298][  INFO][PID:3085565] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9666107 --detached
[2025-10-07 18:18:18,896][  INFO][PID:3085565] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c1d9' (ED25519) to the list of known hosts.


[2025-10-07 18:18:18,897][  INFO][PID:3085565] Downloading the builder-live.log file, attempt 1
[2025-10-07 18:18:18,900][  INFO][PID:3085565] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c1d9 copr-rpmbuild-log
[2025-10-07 18:18:23,899][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:18:28,900][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:18:33,902][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:18:38,903][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:18:43,905][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:18:48,906][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:18:53,910][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:18:58,917][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:19:03,918][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:19:08,920][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:19:13,921][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:19:18,922][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:19:23,923][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:19:28,928][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:19:33,936][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:19:38,937][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:19:43,939][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:19:48,940][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:19:53,941][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:19:58,943][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:20:03,944][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:20:08,946][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:20:13,948][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:20:18,950][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:20:19,397][ ERROR][PID:3085565] Re-try request for task on 'ResallocHost, ticket_id=5529953, hostname=2620:52:3:1:dead:beef:cafe:c1d9, name=copr_hv_x86_64_04_prod_07266134_20251007_181120, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: copr-rpmbuild-log shouldn't exit != 0
[2025-10-07 18:20:19,398][  INFO][PID:3085565] Releasing VM back to pool
[2025-10-07 18:20:19,412][  INFO][PID:3085565] Retry #3 (on other host)
[2025-10-07 18:20:19,413][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:20:19,414][  INFO][PID:3085565] VM allocation process starts
[2025-10-07 18:20:19,429][  INFO][PID:3085565] Trying to allocate VM: ResallocHost, ticket_id=5530037, requested_tags=['copr_builder', 'arch_x86_64']
[2025-10-07 18:20:24,431][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:20:25,502][  INFO][PID:3085565] Allocated host ResallocHost, ticket_id=5530037, hostname=2620:52:3:1:dead:beef:cafe:c1d9, name=copr_hv_x86_64_04_prod_07266134_20251007_181120, requested_tags=['copr_builder', 'arch_x86_64']
[2025-10-07 18:20:25,503][  INFO][PID:3085565] Allocating ssh connection to builder
[2025-10-07 18:20:25,503][  INFO][PID:3085565] Checking that builder machine is OK
[2025-10-07 18:20:25,867][  INFO][PID:3085565] Running remote command: copr-builder-ready srpm-builds
[2025-10-07 18:20:26,234][  INFO][PID:3085565] Red Hat subscription not needed for srpm-builds
Builder is ready to be used

[2025-10-07 18:20:26,235][  INFO][PID:3085565] Filling build.info file with builder info
[2025-10-07 18:20:26,238][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:20:26,240][  INFO][PID:3085565] Not re-notifying FE and msg buses for the new host.
[2025-10-07 18:20:26,241][  INFO][PID:3085565] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9666107 --detached
[2025-10-07 18:20:26,839][  INFO][PID:3085565] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c1d9' (ED25519) to the list of known hosts.


[2025-10-07 18:20:26,840][  INFO][PID:3085565] Downloading the builder-live.log file, attempt 1
[2025-10-07 18:20:26,841][  INFO][PID:3085565] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c1d9 copr-rpmbuild-log
[2025-10-07 18:20:31,840][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:20:36,842][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:20:41,868][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:20:46,869][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:20:51,870][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:20:56,871][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:21:01,873][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:21:06,875][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:21:11,878][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:21:16,879][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:21:21,882][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:21:26,883][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:21:31,884][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:21:36,886][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:21:41,888][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:21:46,889][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:21:51,891][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:21:56,892][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:22:01,893][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:22:06,894][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:22:11,895][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:22:16,897][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:22:21,899][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:22:26,900][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:22:27,556][ ERROR][PID:3085565] Re-try request for task on 'ResallocHost, ticket_id=5530037, hostname=2620:52:3:1:dead:beef:cafe:c1d9, name=copr_hv_x86_64_04_prod_07266134_20251007_181120, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: copr-rpmbuild-log shouldn't exit != 0
[2025-10-07 18:22:27,557][  INFO][PID:3085565] Releasing VM back to pool
[2025-10-07 18:22:27,592][  INFO][PID:3085565] Retry #4 (on other host)
[2025-10-07 18:22:27,592][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:22:27,594][  INFO][PID:3085565] VM allocation process starts
[2025-10-07 18:22:27,624][  INFO][PID:3085565] Trying to allocate VM: ResallocHost, ticket_id=5530098, requested_tags=['copr_builder', 'arch_x86_64']
[2025-10-07 18:22:32,631][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:22:37,632][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:22:42,633][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:22:47,634][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:22:52,635][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:22:57,636][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:23:02,638][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:23:07,639][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:23:12,641][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:23:17,642][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:23:22,643][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:23:27,644][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:23:32,648][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:23:37,658][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:23:42,660][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:23:47,662][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:23:52,664][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:23:57,668][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:24:02,669][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:24:07,671][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:24:12,672][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:24:15,774][  INFO][PID:3085565] Allocated host ResallocHost, ticket_id=5530098, hostname=13.218.96.80, name=aws_x86_64_spot_prod_07266325_20251007_182239, requested_tags=['copr_builder', 'arch_x86_64']
[2025-10-07 18:24:15,775][  INFO][PID:3085565] Allocating ssh connection to builder
[2025-10-07 18:24:15,776][  INFO][PID:3085565] Checking that builder machine is OK
[2025-10-07 18:24:16,111][  INFO][PID:3085565] Running remote command: copr-builder-ready srpm-builds
[2025-10-07 18:24:16,378][  INFO][PID:3085565] Red Hat subscription not needed for srpm-builds
Builder is ready to be used

[2025-10-07 18:24:16,379][  INFO][PID:3085565] Filling build.info file with builder info
[2025-10-07 18:24:16,379][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:24:16,382][  INFO][PID:3085565] Not re-notifying FE and msg buses for the new host.
[2025-10-07 18:24:16,383][  INFO][PID:3085565] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9666107 --detached
[2025-10-07 18:24:16,754][  INFO][PID:3085565] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '13.218.96.80' (ED25519) to the list of known hosts.


[2025-10-07 18:24:16,755][  INFO][PID:3085565] Downloading the builder-live.log file, attempt 1
[2025-10-07 18:24:16,761][  INFO][PID:3085565] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.218.96.80 copr-rpmbuild-log
[2025-10-07 18:24:21,767][  INFO][PID:3085565] Periodic builder liveness probe: alive
[2025-10-07 18:24:21,768][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:24:26,769][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:24:31,770][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:24:36,771][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:24:41,772][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:24:46,773][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:24:51,774][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:24:56,775][  INFO][PID:3085565] Checking for cancel request
[2025-10-07 18:25:01,047][  INFO][PID:3085565] Downloading results from builder
[2025-10-07 18:25:01,049][  INFO][PID:3085565] rsyncing of mockbuilder@13.218.96.80:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/bbedward/dank/srpm-builds/09666107 started
[2025-10-07 18:25:01,053][  INFO][PID:3085565] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@13.218.96.80:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/bbedward/dank/srpm-builds/09666107/ &> /var/lib/copr/public_html/results/bbedward/dank/srpm-builds/09666107/build-09666107.rsync.log
[2025-10-07 18:25:01,291][  INFO][PID:3085565] rsyncing finished.
[2025-10-07 18:25:01,292][  INFO][PID:3085565] Releasing VM back to pool
[2025-10-07 18:25:01,306][  INFO][PID:3085565] Searching for 'success' file in resultdir
[2025-10-07 18:25:01,307][ ERROR][PID:3085565] Build failed: Backend process error: No success file => build failure
[2025-10-07 18:25:01,309][  INFO][PID:3085565] Finished build: id=9666107 failed=True timeout=108000 destdir=/var/lib/copr/public_html/results/bbedward/dank chroot=srpm-builds 
[2025-10-07 18:25:01,311][ ERROR][PID:3085565] Backend process error: No success file => build failure
[2025-10-07 18:25:01,312][  INFO][PID:3085565] Worker failed build, took 44.930177211761475
[2025-10-07 18:25:01,313][  INFO][PID:3085565] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "bbedward",
            "project_name": "dank",
            "project_dirname": "dank",
            "submitter": "bbedward",
            "ended_on": 1759861501.3123024,
            "started_on": 1759861456.3821251,
            "submitted_on": null,
            "status": 0,
            "chroot": "srpm-builds",
            "arch": "x86_64",
            "buildroot_pkgs": null,
            "task_id": "9666107",
            "build_id": 9666107,
            "package_name": null,
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 9,
            "source_json": "{\"script\": \"#!/bin/bash\\nset -e\\n\\n# Create spec file for dms-git\\ncat > dms-git.spec << 'EOF'\\n%global debug_package %{nil}\\n\\nName:           dms-git\\nVersion:        0\\nRelease:        0%{?dist}\\nSummary:        A Quickshell-based desktop shell with Material 3 design principles\\nLicense:        GPL-3.0-only\\nURL:            https://github.com/AvengeMedia/DankMaterialShell\\n\\nSource0:        https://github.com/AvengeMedia/DankMaterialShell/archive/master.tar.gz#/DankMaterialShell-master.tar.gz\\nSource1:        https://github.com/AvengeMedia/danklinux/archive/master.tar.gz#/danklinux-master.tar.gz\\n\\nBuildRequires:  git\\nBuildRequires:  golang >= 1.19\\nBuildRequires:  make\\n\\nProvides:       dms-shell = %{version}-%{release}\\nProvides:       dms = %{version}-%{release}\\n\\n%description\\nDankMaterialShell (DMS) is a modern desktop shell built on top of Quickshell,\\nimplementing Material 3 design principles.\\n\\n%prep\\n%setup -q -n DankMaterialShell-master\\n%setup -q -T -D -a 1 -n DankMaterialShell-master\\n\\n%build\\ncd danklinux-master\\nexport CGO_CPPFLAGS=\\\"${CPPFLAGS}\\\"\\nexport CGO_CFLAGS=\\\"${CFLAGS}\\\"\\nexport CGO_CXXFLAGS=\\\"${CXXFLAGS}\\\"\\nexport CGO_LDFLAGS=\\\"${LDFLAGS}\\\"\\nexport GOFLAGS=\\\"-buildmode=pie -trimpath -ldflags=-linkmode=external -mod=readonly -modcacherw\\\"\\ngo build -o dms ./cmd/dms\\n\\n%install\\ninstall -Dm755 danklinux-master/dms %{buildroot}%{_bindir}/dms\\ninstall -dm755 %{buildroot}%{_sysconfdir}/xdg/quickshell/dms\\ncp -r . %{buildroot}%{_sysconfdir}/xdg/quickshell/dms/\\nrm -rf %{buildroot}%{_sysconfdir}/xdg/quickshell/dms/danklinux-master\\nif [ -f README.md ]; then\\n    install -Dm644 README.md %{buildroot}%{_docdir}/dms/README.md\\nfi\\nrm -rf %{buildroot}%{_sysconfdir}/xdg/quickshell/dms/.git*\\nrm -rf %{buildroot}%{_sysconfdir}/xdg/quickshell/dms/*.tar.gz\\n\\n%files\\n%{_bindir}/dms\\n%{_sysconfdir}/xdg/quickshell/dms/\\n%doc %{_docdir}/dms/\\n\\n%changelog\\n* Thu Jan 01 2025 COPR Builder <copr@fedoraproject.org> - 0-0\\n- Automatic build from git sources\\nEOF\\n\\nspectool -g -R dms-git.spec\\nrpmbuild -bs dms-git.spec\\necho \\\"SRPM files:\\\"\\nls *.src.rpm\\n\", \"chroot\": \"fedora-43-x86_64\", \"builddeps\": \"git make golang\", \"resultdir\": \"\", \"repos\": \"\"}",
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": null,
            "sandbox": "bbedward/dank--bbedward",
            "results": null,
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [],
            "background": false,
            "destdir": "/var/lib/copr/public_html/results/bbedward/dank",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/bbedward/dank",
            "result_dir": "09666107",
            "built_packages": "",
            "tags": [
                "arch_x86_64"
            ],
            "id": 9666107,
            "mockchain_macros": {
                "copr_username": "bbedward",
                "copr_projectname": "dank",
                "vendor": "Fedora Project COPR (bbedward/dank)"
            }
        }
    ]
}
[2025-10-07 18:25:01,373][  INFO][PID:3085565] Sending fedora-messaging bus message in build.end
[2025-10-07 18:25:01,401][  INFO][PID:3085565] Compressing /var/lib/copr/public_html/results/bbedward/dank/srpm-builds/09666107/builder-live.log by gzip
[2025-10-07 18:25:01,402][  INFO][PID:3085565] Running command 'gzip /var/lib/copr/public_html/results/bbedward/dank/srpm-builds/09666107/builder-live.log' as PID 3208043
[2025-10-07 18:25:01,408][  INFO][PID:3085565] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/bbedward/dank/srpm-builds/09666107/builder-live.log)
[2025-10-07 18:25:01,409][  INFO][PID:3085565] Compressing /var/lib/copr/public_html/results/bbedward/dank/srpm-builds/09666107/backend.log by gzip
[2025-10-07 18:25:01,410][  INFO][PID:3085565] Running command 'gzip /var/lib/copr/public_html/results/bbedward/dank/srpm-builds/09666107/backend.log' as PID 3208047