[2025-12-10 08:49:26,433][  INFO][PID:522240] Marking build as starting
[2025-12-10 08:49:26,531][  INFO][PID:522240] Checking that /var/lib/copr/public_html/results/dreibh/ppa/fedora-42-ppc64le/repodata/repomd.xml exists
[2025-12-10 08:49:26,533][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:49:26,535][  INFO][PID:522240] VM allocation process starts
[2025-12-10 08:49:26,550][  INFO][PID:522240] Trying to allocate VM: ResallocHost, ticket_id=257451, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-10 08:49:31,555][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:49:36,557][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:49:41,560][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:49:46,563][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:49:51,564][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:49:56,568][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:50:01,572][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:50:06,574][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:50:11,580][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:50:16,585][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:50:21,587][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:50:26,979][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:50:31,981][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:50:36,986][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:50:41,991][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:50:46,996][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:50:52,033][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:50:54,661][  INFO][PID:522240] Allocated host ResallocHost, ticket_id=257451, hostname=2620:52:6:1161:dead:beef:cafe:c114, name=vmhost_p09_02_prod_00302485_20251210_084555, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-10 08:50:54,662][  INFO][PID:522240] Allocating ssh connection to builder
[2025-12-10 08:50:54,663][  INFO][PID:522240] Checking that builder machine is OK
[2025-12-10 08:50:55,620][  INFO][PID:522240] Running remote command: copr-builder-ready fedora-42-ppc64le
[2025-12-10 08:50:56,203][  INFO][PID:522240] Red Hat subscription not needed for fedora-42-ppc64le
Builder is ready to be used

[2025-12-10 08:50:56,204][  INFO][PID:522240] Filling build.info file with builder info
[2025-12-10 08:50:56,232][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:50:56,233][  INFO][PID:522240] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "dreibh",
            "project_name": "ppa",
            "project_dirname": "ppa",
            "submitter": "dreibh",
            "ended_on": null,
            "started_on": 1765356656.233764,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-42-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9894238-fedora-42-ppc64le",
            "build_id": 9894238,
            "package_name": "neat-stuff",
            "package_version": "1.2.5~rc1.3-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/dreibh/ppa/neat-stuff",
            "git_hash": "cec42f22cf01480e348061fd8b2fcb776f1335d7",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "dreibh/ppa--dreibh",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/dreibh/ppa/fedora-42-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/dreibh/ppa",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/dreibh/ppa",
            "result_dir": "09894238-neat-stuff",
            "built_packages": "",
            "id": 9894238,
            "mockchain_macros": {
                "copr_username": "dreibh",
                "copr_projectname": "ppa",
                "vendor": "Fedora Project COPR (dreibh/ppa)"
            }
        }
    ]
}
[2025-12-10 08:50:57,354][  INFO][PID:522240] Sending fedora-messaging bus message in build.start
[2025-12-10 08:50:58,698][  INFO][PID:522240] Sending fedora-messaging bus message in chroot.start
[2025-12-10 08:50:58,729][  INFO][PID:522240] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9894238-fedora-42-ppc64le --chroot fedora-42-ppc64le --detached
[2025-12-10 08:50:59,728][  INFO][PID:522240] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c114' (ED25519) to the list of known hosts.


[2025-12-10 08:50:59,731][  INFO][PID:522240] Downloading the builder-live.log file, attempt 1
[2025-12-10 08:50:59,739][  INFO][PID:522240] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c114 copr-rpmbuild-log
[2025-12-10 08:51:04,760][  INFO][PID:522240] Periodic builder liveness probe: alive
[2025-12-10 08:51:04,762][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:51:09,764][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:51:14,767][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:51:19,776][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:51:24,778][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:51:29,779][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:51:34,780][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:51:39,782][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:51:44,784][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:51:49,787][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:51:54,792][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:51:59,794][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:52:04,796][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:52:09,797][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:52:14,798][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:52:19,800][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:52:24,801][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:52:29,806][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:52:34,827][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:52:39,833][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:52:44,835][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:52:49,836][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:52:54,837][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:52:59,838][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:53:01,370][ ERROR][PID:522240] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-10 08:53:04,840][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:53:09,844][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:53:11,371][WARNING][PID:522240] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-10 08:53:14,846][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:53:19,848][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:53:21,373][  INFO][PID:522240] Downloading the builder-live.log file, attempt 2
[2025-12-10 08:53:21,383][  INFO][PID:522240] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c114 copr-rpmbuild-log
[2025-12-10 08:53:24,852][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:53:29,861][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:53:34,864][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:53:39,866][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:53:44,869][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:53:49,876][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:53:54,878][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:53:59,881][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:54:04,883][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:54:09,885][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:54:14,887][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:54:19,890][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:54:21,465][ ERROR][PID:522240] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-10 08:54:24,892][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:54:29,895][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:54:31,469][WARNING][PID:522240] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-10 08:54:34,901][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:54:39,906][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:54:41,475][  INFO][PID:522240] Downloading the builder-live.log file, attempt 3
[2025-12-10 08:54:41,481][  INFO][PID:522240] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c114 copr-rpmbuild-log
[2025-12-10 08:54:44,910][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:54:49,915][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:54:54,916][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:54:59,919][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:55:04,923][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:55:09,925][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:55:14,928][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:55:19,931][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:55:24,937][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:55:29,940][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:55:34,943][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:55:39,944][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:55:41,559][ ERROR][PID:522240] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-10 08:55:44,946][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:55:49,952][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:55:51,565][WARNING][PID:522240] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-10 08:55:54,956][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:55:59,961][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:56:01,566][  INFO][PID:522240] Downloading the builder-live.log file, attempt 4
[2025-12-10 08:56:01,570][  INFO][PID:522240] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c114 copr-rpmbuild-log
[2025-12-10 08:56:04,964][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:56:09,967][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:56:14,970][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:56:19,972][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:56:24,974][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:56:29,978][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:56:34,982][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:56:39,986][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:56:44,989][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:56:49,993][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:56:54,997][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:56:59,999][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:57:01,655][ ERROR][PID:522240] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-10 08:57:05,005][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:57:10,006][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:57:11,661][ ERROR][PID:522240] Giving up for unstable SSH, failures: 1765356791.3718956, 1765356871.4693308, 1765356951.5649805, 1765357031.66146
[2025-12-10 08:57:11,666][ ERROR][PID:522240] Re-try request for task on 'ResallocHost, ticket_id=257451, hostname=2620:52:6:1161:dead:beef:cafe:c114, name=vmhost_p09_02_prod_00302485_20251210_084555, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-12-10 08:57:11,668][  INFO][PID:522240] Releasing VM back to pool
[2025-12-10 08:57:11,693][  INFO][PID:522240] Retry #1 (on other host)
[2025-12-10 08:57:11,694][  INFO][PID:522240] Checking that /var/lib/copr/public_html/results/dreibh/ppa/fedora-42-ppc64le/repodata/repomd.xml exists
[2025-12-10 08:57:11,696][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:57:11,697][  INFO][PID:522240] VM allocation process starts
[2025-12-10 08:57:11,721][  INFO][PID:522240] Trying to allocate VM: ResallocHost, ticket_id=257630, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-10 08:57:16,724][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:57:17,771][  INFO][PID:522240] Allocated host ResallocHost, ticket_id=257630, hostname=2620:52:6:1161:dead:beef:cafe:c112, name=vmhost_p09_02_prod_00302611_20251210_085046, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-10 08:57:17,772][  INFO][PID:522240] Allocating ssh connection to builder
[2025-12-10 08:57:17,773][  INFO][PID:522240] Checking that builder machine is OK
[2025-12-10 08:58:17,844][ ERROR][PID:522240] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c112 port 22: Connection timed out

[2025-12-10 08:58:27,845][ ERROR][PID:522240] Re-try request for task on 'ResallocHost, ticket_id=257630, hostname=2620:52:6:1161:dead:beef:cafe:c112, name=vmhost_p09_02_prod_00302611_20251210_085046, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-10 08:58:27,846][  INFO][PID:522240] Releasing VM back to pool
[2025-12-10 08:58:27,857][  INFO][PID:522240] Retry #2 (on other host)
[2025-12-10 08:58:27,857][  INFO][PID:522240] Checking that /var/lib/copr/public_html/results/dreibh/ppa/fedora-42-ppc64le/repodata/repomd.xml exists
[2025-12-10 08:58:27,858][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:58:27,859][  INFO][PID:522240] VM allocation process starts
[2025-12-10 08:58:27,870][  INFO][PID:522240] Trying to allocate VM: ResallocHost, ticket_id=257654, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-10 08:58:32,871][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:58:37,872][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:58:42,873][  INFO][PID:522240] Checking for cancel request
[2025-12-10 08:58:45,907][  INFO][PID:522240] Allocated host ResallocHost, ticket_id=257654, hostname=2620:52:6:1161:dead:beef:cafe:c193, name=vmhost_p09_04_prod_00302082_20251210_082956, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-10 08:58:45,908][  INFO][PID:522240] Allocating ssh connection to builder
[2025-12-10 08:58:45,909][  INFO][PID:522240] Checking that builder machine is OK
[2025-12-10 08:59:45,979][ ERROR][PID:522240] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c193 port 22: Connection timed out

[2025-12-10 08:59:55,980][ ERROR][PID:522240] Re-try request for task on 'ResallocHost, ticket_id=257654, hostname=2620:52:6:1161:dead:beef:cafe:c193, name=vmhost_p09_04_prod_00302082_20251210_082956, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-10 08:59:55,981][  INFO][PID:522240] Releasing VM back to pool
[2025-12-10 08:59:55,992][ ERROR][PID:522240] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c193', '2620:52:6:1161:dead:beef:cafe:c114', '2620:52:6:1161:dead:beef:cafe:c112'}
[2025-12-10 08:59:55,992][WARNING][PID:522240] Switching not-finished job state to 'failed'
[2025-12-10 08:59:55,993][  INFO][PID:522240] Worker failed build, took 539.7590818405151
[2025-12-10 08:59:55,994][  INFO][PID:522240] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "dreibh",
            "project_name": "ppa",
            "project_dirname": "ppa",
            "submitter": "dreibh",
            "ended_on": 1765357195.9928458,
            "started_on": 1765356656.233764,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-42-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9894238-fedora-42-ppc64le",
            "build_id": 9894238,
            "package_name": "neat-stuff",
            "package_version": "1.2.5~rc1.3-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/dreibh/ppa/neat-stuff",
            "git_hash": "cec42f22cf01480e348061fd8b2fcb776f1335d7",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "dreibh/ppa--dreibh",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/dreibh/ppa/fedora-42-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/dreibh/ppa",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/dreibh/ppa",
            "result_dir": "09894238-neat-stuff",
            "built_packages": "",
            "id": 9894238,
            "mockchain_macros": {
                "copr_username": "dreibh",
                "copr_projectname": "ppa",
                "vendor": "Fedora Project COPR (dreibh/ppa)"
            }
        }
    ]
}
[2025-12-10 08:59:56,082][  INFO][PID:522240] Sending fedora-messaging bus message in build.end
[2025-12-10 08:59:56,114][  INFO][PID:522240] Compressing /var/lib/copr/public_html/results/dreibh/ppa/fedora-42-ppc64le/09894238-neat-stuff/builder-live.log by gzip
[2025-12-10 08:59:56,115][  INFO][PID:522240] Running command 'gzip /var/lib/copr/public_html/results/dreibh/ppa/fedora-42-ppc64le/09894238-neat-stuff/builder-live.log' as PID 630986
[2025-12-10 08:59:56,116][  INFO][PID:522240] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/dreibh/ppa/fedora-42-ppc64le/09894238-neat-stuff/builder-live.log)
[2025-12-10 08:59:56,117][  INFO][PID:522240] Compressing /var/lib/copr/public_html/results/dreibh/ppa/fedora-42-ppc64le/09894238-neat-stuff/backend.log by gzip
[2025-12-10 08:59:56,118][  INFO][PID:522240] Running command 'gzip /var/lib/copr/public_html/results/dreibh/ppa/fedora-42-ppc64le/09894238-neat-stuff/backend.log' as PID 630987