[2024-10-05 02:10:21,767][  INFO][PID:3255715] Marking build as starting
[2024-10-05 02:10:21,987][  INFO][PID:3255715] Trying to allocate VM
[2024-10-05 02:10:40,253][  INFO][PID:3255715] Allocated host ResallocHost, ticket_id=5258085, hostname=2620:52:3:1:dead:beef:cafe:c29c
[2024-10-05 02:10:40,260][  INFO][PID:3255715] Allocating ssh connection to builder
[2024-10-05 02:10:40,262][  INFO][PID:3255715] Checking that builder machine is OK
[2024-10-05 02:10:41,333][  INFO][PID:3255715] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29c '/usr/bin/test -f /etc/mock/fedora-rawhide-ppc64le.cfg'
[2024-10-05 02:10:41,965][  INFO][PID:3255715] Filling build.info file with builder info
[2024-10-05 02:10:41,967][  INFO][PID:3255715] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "@fedora-llvm-team",
            "project_name": "llvm-snapshots-big-merge-20241005",
            "project_dirname": "llvm-snapshots-big-merge-20241005",
            "submitter": "kkleine",
            "ended_on": null,
            "started_on": 1728094241.9668903,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "8109680-fedora-rawhide-ppc64le",
            "build_id": 8109680,
            "package_name": "llvm",
            "package_version": "19.1.0-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@fedora-llvm-team/llvm-snapshots-big-merge-20241005/llvm",
            "git_hash": "3d9f4f57302d5357a2880ffb3c55589d65148985",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@fedora-llvm-team/llvm-snapshots-big-merge-20241005--kkleine",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20241005/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "module_hotfixes": true,
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [
                "snapshot_build"
            ],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20241005",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20241005",
            "result_dir": "08109680-llvm",
            "built_packages": "",
            "id": 8109680,
            "mockchain_macros": {
                "copr_username": "@fedora-llvm-team",
                "copr_projectname": "llvm-snapshots-big-merge-20241005",
                "vendor": "Fedora Project COPR (@fedora-llvm-team/llvm-snapshots-big-merge-20241005)"
            }
        }
    ]
}
[2024-10-05 02:10:42,258][  INFO][PID:3255715] Sending fedora-messaging bus message in build.start
[2024-10-05 02:10:46,430][  INFO][PID:3255715] Sending fedora-messaging bus message in chroot.start
[2024-10-05 02:10:46,505][  INFO][PID:3255715] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8109680-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2024-10-05 02:10:47,392][  INFO][PID:3255715] Downloading the builder-live.log file, attempt 1
[2024-10-05 02:10:47,397][  INFO][PID:3255715] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29c copr-rpmbuild-log
[2024-10-05 02:10:52,407][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 02:20:52,588][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 02:30:52,676][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 02:40:52,757][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 02:50:52,826][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 02:54:03,948][ ERROR][PID:3255715] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-10-05 02:54:13,949][WARNING][PID:3255715] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2024-10-05 02:54:23,950][  INFO][PID:3255715] Downloading the builder-live.log file, attempt 2
[2024-10-05 02:54:23,953][  INFO][PID:3255715] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29c copr-rpmbuild-log
[2024-10-05 02:54:27,734][ ERROR][PID:3255715] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-10-05 02:54:37,734][WARNING][PID:3255715] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2024-10-05 02:54:47,735][  INFO][PID:3255715] Downloading the builder-live.log file, attempt 3
[2024-10-05 02:54:47,736][  INFO][PID:3255715] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29c copr-rpmbuild-log
[2024-10-05 02:54:54,783][ ERROR][PID:3255715] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-10-05 02:55:04,784][WARNING][PID:3255715] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2024-10-05 02:55:14,785][  INFO][PID:3255715] Downloading the builder-live.log file, attempt 4
[2024-10-05 02:55:14,786][  INFO][PID:3255715] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29c copr-rpmbuild-log
[2024-10-05 02:55:21,827][ ERROR][PID:3255715] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-10-05 02:55:31,828][ ERROR][PID:3255715] Giving up for unstable SSH, failures: 1728096853.9494832, 1728096877.7346303, 1728096904.7844925, 1728096931.8280072
[2024-10-05 02:55:31,828][ ERROR][PID:3255715] Re-try request for task on 'ResallocHost, ticket_id=5258085, hostname=2620:52:3:1:dead:beef:cafe:c29c': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2024-10-05 02:55:31,829][  INFO][PID:3255715] Releasing VM back to pool
[2024-10-05 02:55:31,837][  INFO][PID:3255715] Retry #1 (on other host)
[2024-10-05 02:55:31,838][  INFO][PID:3255715] Trying to allocate VM
[2024-10-05 02:55:34,920][  INFO][PID:3255715] Allocated host ResallocHost, ticket_id=5258323, hostname=2620:52:3:1:dead:beef:cafe:c24e
[2024-10-05 02:55:34,920][  INFO][PID:3255715] Allocating ssh connection to builder
[2024-10-05 02:55:34,921][  INFO][PID:3255715] Checking that builder machine is OK
[2024-10-05 02:55:35,721][  INFO][PID:3255715] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c24e '/usr/bin/test -f /etc/mock/fedora-rawhide-ppc64le.cfg'
[2024-10-05 02:55:36,086][  INFO][PID:3255715] Filling build.info file with builder info
[2024-10-05 02:55:36,090][  INFO][PID:3255715] Not re-notifying FE and msg buses for the new host.
[2024-10-05 02:55:36,090][  INFO][PID:3255715] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8109680-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2024-10-05 02:55:36,853][  INFO][PID:3255715] Downloading the builder-live.log file, attempt 1
[2024-10-05 02:55:36,854][  INFO][PID:3255715] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c24e copr-rpmbuild-log
[2024-10-05 03:00:56,895][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 03:10:56,973][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 03:20:57,047][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 03:30:57,163][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 03:40:57,249][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 03:50:57,341][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 04:00:57,430][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 04:10:57,735][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 04:20:57,854][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 04:30:57,941][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 04:40:58,030][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 04:50:58,130][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 05:00:58,223][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 05:10:58,317][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 05:20:58,411][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 05:30:58,495][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 05:40:58,579][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 05:50:58,664][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 06:00:58,747][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 06:10:58,849][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 06:20:58,957][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 06:30:59,047][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 06:40:59,129][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 06:50:59,222][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 07:00:59,311][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 07:10:59,430][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 07:20:59,581][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 07:30:59,699][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 07:40:59,777][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 07:50:59,931][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 08:01:00,056][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 08:11:00,163][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 08:21:00,260][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 08:31:00,510][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 08:41:00,738][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 08:51:00,832][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 09:01:00,918][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 09:11:00,994][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 09:21:01,070][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 09:31:01,154][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 09:41:01,242][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 09:51:01,328][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 10:01:01,422][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 10:11:01,533][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 10:21:01,622][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 10:31:01,693][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 10:41:01,765][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 10:51:01,837][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 11:01:01,915][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 11:11:01,998][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 11:21:02,075][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 11:31:02,153][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 11:41:02,224][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 11:51:02,297][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 12:01:02,369][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 12:11:02,456][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 12:21:02,530][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 12:31:02,601][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 12:41:02,672][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 12:51:02,744][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 13:01:02,825][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 13:11:02,907][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 13:21:02,986][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 13:31:03,058][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 13:41:03,129][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 13:51:03,201][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 14:01:03,280][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 14:11:03,359][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 14:21:03,440][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 14:31:03,513][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 14:41:03,586][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 14:51:03,657][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 15:01:03,730][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 15:11:03,822][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 15:21:03,901][  INFO][PID:3255715] Periodic builder liveness probe: alive
[2024-10-05 15:22:50,567][  INFO][PID:3255715] Downloading results from builder
[2024-10-05 15:22:50,594][  INFO][PID:3255715] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c24e]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20241005/fedora-rawhide-ppc64le/08109680-llvm started
[2024-10-05 15:22:50,626][  INFO][PID:3255715] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c24e]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20241005/fedora-rawhide-ppc64le/08109680-llvm/ &> /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20241005/fedora-rawhide-ppc64le/08109680-llvm/build-08109680.rsync.log
[2024-10-05 15:22:56,480][  INFO][PID:3255715] rsyncing finished.
[2024-10-05 15:22:56,482][  INFO][PID:3255715] Releasing VM back to pool
[2024-10-05 15:22:56,499][  INFO][PID:3255715] Searching for 'success' file in resultdir
[2024-10-05 15:22:56,506][ ERROR][PID:3255715] Build failed: Backend process error: No success file => build failure
[2024-10-05 15:22:56,607][  INFO][PID:3255715] Finished build: id=8109680 failed=True timeout=108000 destdir=/var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20241005 chroot=fedora-rawhide-ppc64le 
[2024-10-05 15:22:56,677][ ERROR][PID:3255715] Backend process error: No success file => build failure
[2024-10-05 15:22:56,680][  INFO][PID:3255715] Worker failed build, took 44840.5899181366
[2024-10-05 15:22:56,693][  INFO][PID:3255715] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "@fedora-llvm-team",
            "project_name": "llvm-snapshots-big-merge-20241005",
            "project_dirname": "llvm-snapshots-big-merge-20241005",
            "submitter": "kkleine",
            "ended_on": 1728141776.6800196,
            "started_on": 1728096936.0901015,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "8109680-fedora-rawhide-ppc64le",
            "build_id": 8109680,
            "package_name": "llvm",
            "package_version": "19.1.0-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@fedora-llvm-team/llvm-snapshots-big-merge-20241005/llvm",
            "git_hash": "3d9f4f57302d5357a2880ffb3c55589d65148985",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@fedora-llvm-team/llvm-snapshots-big-merge-20241005--kkleine",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20241005/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "module_hotfixes": true,
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [
                "snapshot_build"
            ],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20241005",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20241005",
            "result_dir": "08109680-llvm",
            "built_packages": "",
            "id": 8109680,
            "mockchain_macros": {
                "copr_username": "@fedora-llvm-team",
                "copr_projectname": "llvm-snapshots-big-merge-20241005",
                "vendor": "Fedora Project COPR (@fedora-llvm-team/llvm-snapshots-big-merge-20241005)"
            }
        }
    ]
}
[2024-10-05 15:22:57,604][  INFO][PID:3255715] Sending fedora-messaging bus message in build.end
[2024-10-05 15:22:57,848][  INFO][PID:3255715] Compressing /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20241005/fedora-rawhide-ppc64le/08109680-llvm/builder-live.log by gzip
[2024-10-05 15:22:57,865][  INFO][PID:3255715] Running command 'gzip /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20241005/fedora-rawhide-ppc64le/08109680-llvm/builder-live.log' as PID 1089075
[2024-10-05 15:22:58,367][  INFO][PID:3255715] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20241005/fedora-rawhide-ppc64le/08109680-llvm/builder-live.log)
[2024-10-05 15:22:58,376][  INFO][PID:3255715] Compressing /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20241005/fedora-rawhide-ppc64le/08109680-llvm/backend.log by gzip
[2024-10-05 15:22:58,377][  INFO][PID:3255715] Running command 'gzip /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20241005/fedora-rawhide-ppc64le/08109680-llvm/backend.log' as PID 1089086