[2025-07-26 06:06:11,430][  INFO][PID:4040342] Marking build as starting
[2025-07-26 06:06:11,472][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:06:11,474][  INFO][PID:4040342] VM allocation process starts
[2025-07-26 06:06:11,506][  INFO][PID:4040342] Trying to allocate VM: ResallocHost, ticket_id=4171854, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-07-26 06:06:14,521][  INFO][PID:4040342] Allocated host ResallocHost, ticket_id=4171854, hostname=140.211.168.87, name=copr_osuosl_p09_prod_05891004_20250726_055902, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-07-26 06:06:14,521][  INFO][PID:4040342] Allocating ssh connection to builder
[2025-07-26 06:06:14,522][  INFO][PID:4040342] Checking that builder machine is OK
[2025-07-26 06:06:15,998][  INFO][PID:4040342] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2025-07-26 06:06:16,992][  INFO][PID:4040342] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2025-07-26 06:06:16,993][  INFO][PID:4040342] Filling build.info file with builder info
[2025-07-26 06:06:16,993][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:06:16,994][  INFO][PID:4040342] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "psimovec",
            "project_name": "openblas-0.3.23",
            "project_dirname": "openblas-0.3.23",
            "submitter": "https://src.fedoraproject.org/user/releng",
            "ended_on": null,
            "started_on": 1753509976.9944313,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9315269-fedora-rawhide-ppc64le",
            "build_id": 9315269,
            "package_name": "csdiff",
            "package_version": "3.5.5-2",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/psimovec/openblas-0.3.23/csdiff",
            "git_hash": "11592be02b3886e563873bb28e268f970b6a6536",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "psimovec/openblas-0.3.23--https://src.fedoraproject.org/user/releng",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/",
                    "id": "http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch",
                    "name": "Additional repo http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch"
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/psimovec/openblas-0.3.23",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/psimovec/openblas-0.3.23",
            "result_dir": "09315269-csdiff",
            "built_packages": "",
            "id": 9315269,
            "mockchain_macros": {
                "copr_username": "psimovec",
                "copr_projectname": "openblas-0.3.23",
                "vendor": "Fedora Project COPR (psimovec/openblas-0.3.23)"
            }
        }
    ]
}
[2025-07-26 06:06:17,031][  INFO][PID:4040342] Sending fedora-messaging bus message in build.start
[2025-07-26 06:06:17,659][  INFO][PID:4040342] Sending fedora-messaging bus message in chroot.start
[2025-07-26 06:06:17,684][  INFO][PID:4040342] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9315269-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2025-07-26 06:06:18,983][  INFO][PID:4040342] Downloading the builder-live.log file, attempt 1
[2025-07-26 06:06:18,984][  INFO][PID:4040342] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.87 copr-rpmbuild-log
[2025-07-26 06:06:23,988][  INFO][PID:4040342] Periodic builder liveness probe: alive
[2025-07-26 06:06:23,988][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:06:28,989][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:06:33,990][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:06:38,991][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:06:43,992][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:06:48,993][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:06:53,994][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:06:58,996][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:07:03,997][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:07:08,998][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:07:13,999][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:07:19,001][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:07:24,002][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:07:29,003][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:07:34,005][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:07:39,006][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:07:44,007][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:07:49,008][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:07:54,163][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:07:59,165][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:08:04,167][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:08:09,169][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:08:14,171][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:08:19,174][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:08:24,175][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:08:29,177][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:08:34,178][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:08:39,180][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:08:44,182][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:08:49,184][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:08:54,186][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:08:59,192][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:09:04,194][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:09:09,195][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:09:14,198][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:09:19,205][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:09:24,208][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:09:29,210][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:09:34,227][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:09:39,244][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:09:44,258][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:09:49,272][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:09:54,275][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:09:59,281][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:10:04,409][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:10:08,139][ ERROR][PID:4040342] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-26 06:10:09,444][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:10:14,707][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:10:18,140][WARNING][PID:4040342] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-26 06:10:19,909][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:10:24,912][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:10:28,140][  INFO][PID:4040342] Downloading the builder-live.log file, attempt 2
[2025-07-26 06:10:28,142][  INFO][PID:4040342] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.87 copr-rpmbuild-log
[2025-07-26 06:10:29,913][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:10:34,914][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:10:39,995][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:10:44,997][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:10:50,109][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:10:55,111][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:11:00,141][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:11:05,148][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:11:10,192][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:11:15,411][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:11:20,428][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:11:25,437][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:11:28,279][ ERROR][PID:4040342] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-26 06:11:30,439][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:11:35,540][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:11:38,387][WARNING][PID:4040342] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-26 06:11:40,541][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:11:45,543][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:11:48,388][  INFO][PID:4040342] Downloading the builder-live.log file, attempt 3
[2025-07-26 06:11:48,389][  INFO][PID:4040342] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.87 copr-rpmbuild-log
[2025-07-26 06:11:50,973][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:11:56,041][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:12:01,083][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:12:06,559][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:12:11,664][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:12:16,892][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:12:22,033][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:12:27,368][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:12:32,558][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:12:37,971][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:12:42,973][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:12:48,305][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:12:48,768][ ERROR][PID:4040342] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-26 06:12:53,767][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:12:58,772][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:12:58,773][WARNING][PID:4040342] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-26 06:13:03,774][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:13:08,775][  INFO][PID:4040342] Downloading the builder-live.log file, attempt 4
[2025-07-26 06:13:08,777][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:13:08,784][  INFO][PID:4040342] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.87 copr-rpmbuild-log
[2025-07-26 06:13:13,786][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:13:18,787][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:13:23,789][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:13:28,879][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:13:34,099][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:13:39,205][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:13:44,352][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:13:49,461][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:13:54,865][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:13:59,998][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:14:05,024][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:14:08,922][ ERROR][PID:4040342] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-26 06:14:10,029][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:14:15,036][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:14:18,925][ ERROR][PID:4040342] Giving up for unstable SSH, failures: 1753510218.140114, 1753510298.3870988, 1753510378.7730608, 1753510458.925123
[2025-07-26 06:14:18,926][ ERROR][PID:4040342] Re-try request for task on 'ResallocHost, ticket_id=4171854, hostname=140.211.168.87, name=copr_osuosl_p09_prod_05891004_20250726_055902, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-07-26 06:14:18,926][  INFO][PID:4040342] Releasing VM back to pool
[2025-07-26 06:14:18,936][  INFO][PID:4040342] Retry #1 (on other host)
[2025-07-26 06:14:18,937][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:14:18,938][  INFO][PID:4040342] VM allocation process starts
[2025-07-26 06:14:18,986][  INFO][PID:4040342] Trying to allocate VM: ResallocHost, ticket_id=4171896, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-07-26 06:14:22,023][  INFO][PID:4040342] Allocated host ResallocHost, ticket_id=4171896, hostname=2620:52:3:1:dead:beef:cafe:c24d, name=copr_hv_ppc64le_02_prod_05891060_20250726_060752, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-07-26 06:14:22,024][  INFO][PID:4040342] Allocating ssh connection to builder
[2025-07-26 06:14:22,025][  INFO][PID:4040342] Checking that builder machine is OK
[2025-07-26 06:14:24,854][  INFO][PID:4040342] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2025-07-26 06:14:26,552][  INFO][PID:4040342] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2025-07-26 06:14:26,553][  INFO][PID:4040342] Filling build.info file with builder info
[2025-07-26 06:14:26,563][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:14:26,566][  INFO][PID:4040342] Not re-notifying FE and msg buses for the new host.
[2025-07-26 06:14:26,566][  INFO][PID:4040342] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9315269-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2025-07-26 06:14:29,505][  INFO][PID:4040342] Downloading the builder-live.log file, attempt 1
[2025-07-26 06:14:29,507][  INFO][PID:4040342] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c24d copr-rpmbuild-log
[2025-07-26 06:14:34,506][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:14:39,507][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:14:44,508][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:14:49,509][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:14:54,510][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:14:59,512][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:15:04,513][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:15:09,514][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:15:14,515][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:15:19,517][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:15:24,518][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:15:29,519][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:15:34,520][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:15:39,522][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:15:44,524][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:15:49,525][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:15:54,526][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:15:59,527][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:16:04,528][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:16:09,529][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:16:14,530][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:16:19,531][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:16:24,542][  INFO][PID:4040342] Periodic builder liveness probe: alive
[2025-07-26 06:16:24,542][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:16:29,543][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:16:34,544][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:16:39,545][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:16:44,547][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:16:49,548][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:16:54,549][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:16:59,551][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:17:04,552][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:17:09,553][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:17:14,555][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:17:19,557][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:17:24,558][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:17:29,559][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:17:34,560][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:17:39,562][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:17:44,563][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:17:49,564][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:17:54,566][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:17:59,567][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:18:04,568][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:18:09,569][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:18:14,570][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:18:19,572][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:18:24,573][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:18:29,575][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:18:34,596][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:18:39,598][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:18:44,664][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:18:49,750][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:18:54,774][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:18:59,775][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:19:04,776][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:19:09,892][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:19:15,086][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:19:20,371][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:19:25,657][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:19:30,658][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:19:35,920][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:19:40,922][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:19:45,923][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:19:50,924][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:19:55,925][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:20:00,926][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:20:06,014][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:20:11,479][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:20:16,481][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:20:21,818][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:20:27,103][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:20:32,662][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:20:38,168][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:20:43,572][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:20:48,624][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:20:53,932][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:20:59,353][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:21:04,530][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:21:09,538][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:21:14,756][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:21:19,860][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:21:24,862][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:21:29,863][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:21:34,864][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:21:40,007][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:21:45,009][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:21:50,011][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:21:55,012][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:22:00,013][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:22:05,015][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:22:10,017][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:22:15,018][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:22:20,019][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:22:25,021][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:22:30,023][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:22:35,025][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:22:40,026][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:22:45,027][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:22:50,134][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:22:55,136][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:23:00,139][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:23:05,141][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:23:10,208][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:23:15,211][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:23:20,213][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:23:25,214][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:23:30,216][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:23:35,217][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:23:40,219][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:23:45,220][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:23:50,222][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:23:55,223][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:24:00,224][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:24:05,284][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:24:10,287][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:24:15,288][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:24:20,291][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:24:25,292][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:24:30,293][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:24:35,294][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:24:40,295][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:24:45,296][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:24:50,314][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:24:55,316][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:25:00,587][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:25:06,013][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:25:11,014][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:25:16,017][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:25:21,019][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:25:26,020][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:25:31,022][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:25:36,146][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:25:41,292][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:25:46,327][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:25:51,440][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:25:56,464][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:26:01,467][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:26:06,751][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:26:11,795][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:26:16,831][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:26:21,833][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:26:27,314][  INFO][PID:4040342] Periodic builder liveness probe: alive
[2025-07-26 06:26:27,315][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:26:32,316][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:26:37,317][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:26:42,318][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:26:47,318][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:26:52,319][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:26:57,321][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:27:02,322][  INFO][PID:4040342] Checking for cancel request
[2025-07-26 06:27:02,624][  INFO][PID:4040342] Downloading results from builder
[2025-07-26 06:27:02,624][  INFO][PID:4040342] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c24d]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff started
[2025-07-26 06:27:02,625][  INFO][PID:4040342] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c24d]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/ &> /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/build-09315269.rsync.log
[2025-07-26 06:27:08,142][  INFO][PID:4040342] rsyncing finished.
[2025-07-26 06:27:08,142][  INFO][PID:4040342] Releasing VM back to pool
[2025-07-26 06:27:08,151][  INFO][PID:4040342] Searching for 'success' file in resultdir
[2025-07-26 06:27:08,152][  INFO][PID:4040342] Going to sign pkgs from source: 9315269-fedora-rawhide-ppc64le in chroot: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le
[2025-07-26 06:27:08,153][  INFO][PID:4040342] Calling '/bin/sign -u psimovec#openblas-0.3.23@copr.fedorahosted.org -p' (attempt #1)
[2025-07-26 06:27:08,479][  INFO][PID:4040342] Calling '/bin/sign -4 -h sha256 -u psimovec#openblas-0.3.23@copr.fedorahosted.org -r /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/csdiff-debugsource-3.5.5-2.fc43.ppc64le.rpm' (attempt #1)
[2025-07-26 06:27:09,437][  INFO][PID:4040342] signed rpm: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/csdiff-debugsource-3.5.5-2.fc43.ppc64le.rpm
[2025-07-26 06:27:09,437][  INFO][PID:4040342] Calling '/bin/sign -4 -h sha256 -u psimovec#openblas-0.3.23@copr.fedorahosted.org -r /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/python3-csdiff-debuginfo-3.5.5-2.fc43.ppc64le.rpm' (attempt #1)
[2025-07-26 06:27:10,539][  INFO][PID:4040342] signed rpm: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/python3-csdiff-debuginfo-3.5.5-2.fc43.ppc64le.rpm
[2025-07-26 06:27:10,540][  INFO][PID:4040342] Calling '/bin/sign -4 -h sha256 -u psimovec#openblas-0.3.23@copr.fedorahosted.org -r /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/csdiff-debuginfo-3.5.5-2.fc43.ppc64le.rpm' (attempt #1)
[2025-07-26 06:27:11,920][  INFO][PID:4040342] signed rpm: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/csdiff-debuginfo-3.5.5-2.fc43.ppc64le.rpm
[2025-07-26 06:27:11,920][  INFO][PID:4040342] Calling '/bin/sign -4 -h sha256 -u psimovec#openblas-0.3.23@copr.fedorahosted.org -r /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/csdiff-static-debuginfo-3.5.5-2.fc43.ppc64le.rpm' (attempt #1)
[2025-07-26 06:27:13,090][  INFO][PID:4040342] signed rpm: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/csdiff-static-debuginfo-3.5.5-2.fc43.ppc64le.rpm
[2025-07-26 06:27:13,090][  INFO][PID:4040342] Calling '/bin/sign -4 -h sha256 -u psimovec#openblas-0.3.23@copr.fedorahosted.org -r /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/csdiff-3.5.5-2.fc43.src.rpm' (attempt #1)
[2025-07-26 06:27:14,095][  INFO][PID:4040342] signed rpm: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/csdiff-3.5.5-2.fc43.src.rpm
[2025-07-26 06:27:14,096][  INFO][PID:4040342] Calling '/bin/sign -4 -h sha256 -u psimovec#openblas-0.3.23@copr.fedorahosted.org -r /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/csdiff-3.5.5-2.fc43.ppc64le.rpm' (attempt #1)
[2025-07-26 06:27:15,126][  INFO][PID:4040342] signed rpm: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/csdiff-3.5.5-2.fc43.ppc64le.rpm
[2025-07-26 06:27:15,126][  INFO][PID:4040342] Calling '/bin/sign -4 -h sha256 -u psimovec#openblas-0.3.23@copr.fedorahosted.org -r /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/csdiff-static-3.5.5-2.fc43.ppc64le.rpm' (attempt #1)
[2025-07-26 06:27:16,304][  INFO][PID:4040342] signed rpm: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/csdiff-static-3.5.5-2.fc43.ppc64le.rpm
[2025-07-26 06:27:16,304][  INFO][PID:4040342] Calling '/bin/sign -4 -h sha256 -u psimovec#openblas-0.3.23@copr.fedorahosted.org -r /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/python3-csdiff-3.5.5-2.fc43.ppc64le.rpm' (attempt #1)
[2025-07-26 06:27:17,097][  INFO][PID:4040342] signed rpm: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/python3-csdiff-3.5.5-2.fc43.ppc64le.rpm
[2025-07-26 06:27:17,098][  INFO][PID:4040342] Sign done
[2025-07-26 06:27:17,098][  INFO][PID:4040342] Incremental createrepo run, adding 09315269-csdiff into https://download.copr.fedorainfracloud.org/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le, (auto-create-repo=True)
[2025-07-26 06:27:17,099][  INFO][PID:4040342] Running command 'copr-repo --batched /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le --add 09315269-csdiff --no-appstream-metadata' as PID 4118472
[2025-07-26 06:27:27,678][  INFO][PID:4040342] Finished after 10 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le --add 09315269-csdiff --no-appstream-metadata)
[2025-07-26 06:27:27,679][  INFO][PID:4040342] Getting build details
[2025-07-26 06:27:27,679][  INFO][PID:4040342] Listing built binary packages in /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff
[2025-07-26 06:27:27,680][  INFO][PID:4040342] Built packages:
['csdiff-debugsource 3.5.5', 'python3-csdiff 3.5.5', 'csdiff-static 3.5.5', 'python3-csdiff-debuginfo 3.5.5', 'csdiff 3.5.5', 'csdiff-debuginfo 3.5.5', 'csdiff-static-debuginfo 3.5.5']
[2025-07-26 06:27:27,680][  INFO][PID:4040342] build details: {'built_packages': 'csdiff-debugsource 3.5.5\npython3-csdiff 3.5.5\ncsdiff-static 3.5.5\npython3-csdiff-debuginfo 3.5.5\ncsdiff 3.5.5\ncsdiff-debuginfo 3.5.5\ncsdiff-static-debuginfo 3.5.5'}
[2025-07-26 06:27:27,681][  INFO][PID:4040342] Retrieving pubkey
[2025-07-26 06:27:27,681][  INFO][PID:4040342] Calling '/bin/sign -u psimovec#openblas-0.3.23@copr.fedorahosted.org -p' (attempt #1)
[2025-07-26 06:27:28,008][  INFO][PID:4040342] Added pubkey for user psimovec project openblas-0.3.23 into: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/pubkey.gpg
[2025-07-26 06:27:28,009][  INFO][PID:4040342] Finished build: id=9315269 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/psimovec/openblas-0.3.23 chroot=fedora-rawhide-ppc64le 
[2025-07-26 06:27:28,011][  INFO][PID:4040342] Worker succeeded build, took 781.4450976848602
[2025-07-26 06:27:28,012][  INFO][PID:4040342] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "psimovec",
            "project_name": "openblas-0.3.23",
            "project_dirname": "openblas-0.3.23",
            "submitter": "https://src.fedoraproject.org/user/releng",
            "ended_on": 1753511248.0112543,
            "started_on": 1753510466.5661566,
            "submitted_on": null,
            "status": 1,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9315269-fedora-rawhide-ppc64le",
            "build_id": 9315269,
            "package_name": "csdiff",
            "package_version": "3.5.5-2",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/psimovec/openblas-0.3.23/csdiff",
            "git_hash": "11592be02b3886e563873bb28e268f970b6a6536",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "psimovec/openblas-0.3.23--https://src.fedoraproject.org/user/releng",
            "results": {
                "packages": [
                    {
                        "name": "csdiff-debugsource",
                        "epoch": null,
                        "version": "3.5.5",
                        "release": "2.fc43",
                        "arch": "ppc64le"
                    },
                    {
                        "name": "python3-csdiff",
                        "epoch": null,
                        "version": "3.5.5",
                        "release": "2.fc43",
                        "arch": "ppc64le"
                    },
                    {
                        "name": "csdiff-static",
                        "epoch": null,
                        "version": "3.5.5",
                        "release": "2.fc43",
                        "arch": "ppc64le"
                    },
                    {
                        "name": "csdiff",
                        "epoch": null,
                        "version": "3.5.5",
                        "release": "2.fc43",
                        "arch": "src"
                    },
                    {
                        "name": "python3-csdiff-debuginfo",
                        "epoch": null,
                        "version": "3.5.5",
                        "release": "2.fc43",
                        "arch": "ppc64le"
                    },
                    {
                        "name": "csdiff",
                        "epoch": null,
                        "version": "3.5.5",
                        "release": "2.fc43",
                        "arch": "ppc64le"
                    },
                    {
                        "name": "csdiff-debuginfo",
                        "epoch": null,
                        "version": "3.5.5",
                        "release": "2.fc43",
                        "arch": "ppc64le"
                    },
                    {
                        "name": "csdiff-static-debuginfo",
                        "epoch": null,
                        "version": "3.5.5",
                        "release": "2.fc43",
                        "arch": "ppc64le"
                    }
                ]
            },
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/",
                    "id": "http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch",
                    "name": "Additional repo http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch"
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/psimovec/openblas-0.3.23",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/psimovec/openblas-0.3.23",
            "result_dir": "09315269-csdiff",
            "built_packages": "csdiff-debugsource 3.5.5\npython3-csdiff 3.5.5\ncsdiff-static 3.5.5\npython3-csdiff-debuginfo 3.5.5\ncsdiff 3.5.5\ncsdiff-debuginfo 3.5.5\ncsdiff-static-debuginfo 3.5.5",
            "id": 9315269,
            "mockchain_macros": {
                "copr_username": "psimovec",
                "copr_projectname": "openblas-0.3.23",
                "vendor": "Fedora Project COPR (psimovec/openblas-0.3.23)"
            }
        }
    ]
}
[2025-07-26 06:27:28,058][  INFO][PID:4040342] Sending fedora-messaging bus message in build.end
[2025-07-26 06:27:28,082][  INFO][PID:4040342] Compressing /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/builder-live.log by gzip
[2025-07-26 06:27:28,083][  INFO][PID:4040342] Running command 'gzip /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/builder-live.log' as PID 4118985
[2025-07-26 06:27:28,093][  INFO][PID:4040342] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/builder-live.log)
[2025-07-26 06:27:28,094][  INFO][PID:4040342] Compressing /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/backend.log by gzip
[2025-07-26 06:27:28,096][  INFO][PID:4040342] Running command 'gzip /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-rawhide-ppc64le/09315269-csdiff/backend.log' as PID 4118987