[2024-06-18 14:56:07,843][  INFO][PID:1275962] Marking build as starting
[2024-06-18 14:56:08,111][  INFO][PID:1275962] Trying to allocate VM
[2024-06-18 14:56:26,178][  INFO][PID:1275962] Allocated host ResallocHost, ticket_id=3381280, hostname=2620:52:3:1:dead:beef:cafe:c2a6
[2024-06-18 14:56:26,178][  INFO][PID:1275962] Allocating ssh connection to builder
[2024-06-18 14:56:26,179][  INFO][PID:1275962] Checking that builder machine is OK
[2024-06-18 14:56:27,011][  INFO][PID:1275962] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c2a6 '/usr/bin/test -f /etc/mock/fedora-40-ppc64le.cfg'
[2024-06-18 14:56:27,426][  INFO][PID:1275962] Filling build.info file with builder info
[2024-06-18 14:56:27,428][  INFO][PID:1275962] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 36000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@dotnet-sig",
            "project_name": "dotnet-preview",
            "project_dirname": "dotnet-preview",
            "submitter": "omajid",
            "ended_on": null,
            "started_on": 1718722587.4276512,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-40-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "7624836-fedora-40-ppc64le",
            "build_id": 7624836,
            "package_name": "dotnet9.0",
            "package_version": "9.0.100~preview.5.24307.1-0.9",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@dotnet-sig/dotnet-preview/dotnet9.0",
            "git_hash": "04e6f02407a91ef49dbcff1aaa474328dbe0834d",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@dotnet-sig/dotnet-preview--omajid",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "background": false,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@dotnet-sig/dotnet-preview",
            "result_dir": "07624836-dotnet9.0",
            "built_packages": "",
            "id": 7624836,
            "mockchain_macros": {
                "copr_username": "@dotnet-sig",
                "copr_projectname": "dotnet-preview",
                "vendor": "Fedora Project COPR (@dotnet-sig/dotnet-preview)"
            }
        }
    ]
}
[2024-06-18 14:56:27,508][  INFO][PID:1275962] Sending fedora-messaging bus message in build.start
[2024-06-18 14:56:28,276][  INFO][PID:1275962] Sending fedora-messaging bus message in chroot.start
[2024-06-18 14:56:28,304][  INFO][PID:1275962] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/7624836-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached
[2024-06-18 14:56:29,026][  INFO][PID:1275962] Downloading the builder-live.log file, attempt 1
[2024-06-18 14:56:29,028][  INFO][PID:1275962] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c2a6 copr-rpmbuild-log
[2024-06-18 14:56:34,038][  INFO][PID:1275962] Status: False
[2024-06-18 15:06:34,139][  INFO][PID:1275962] Status: False
[2024-06-18 15:16:34,261][  INFO][PID:1275962] Status: False
[2024-06-18 15:26:34,470][  INFO][PID:1275962] Status: False
[2024-06-18 15:36:34,573][  INFO][PID:1275962] Status: False
[2024-06-18 15:46:34,668][  INFO][PID:1275962] Status: False
[2024-06-18 15:56:34,758][  INFO][PID:1275962] Status: False
[2024-06-18 16:06:34,847][  INFO][PID:1275962] Status: False
[2024-06-18 16:12:23,066][  INFO][PID:1275962] Downloading results from builder
[2024-06-18 16:12:23,067][  INFO][PID:1275962] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c2a6]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/07624836-dotnet9.0 started
[2024-06-18 16:12:23,068][  INFO][PID:1275962] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c2a6]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/07624836-dotnet9.0/ &> /var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/07624836-dotnet9.0/build-07624836.rsync.log
[2024-06-18 16:54:39,869][  INFO][PID:1275962] rsyncing finished.
[2024-06-18 16:54:39,870][ ERROR][PID:1275962] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2024-06-18 16:54:39,871][ ERROR][PID:1275962] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2024-06-18 16:54:49,871][  INFO][PID:1275962] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c2a6]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/07624836-dotnet9.0 started
[2024-06-18 16:54:49,873][  INFO][PID:1275962] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c2a6]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/07624836-dotnet9.0/ &> /var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/07624836-dotnet9.0/build-07624836.rsync.log
[2024-06-18 16:55:49,983][  INFO][PID:1275962] rsyncing finished.
[2024-06-18 16:55:49,984][ ERROR][PID:1275962] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2024-06-18 16:55:49,984][ ERROR][PID:1275962] SSH connection lost on #2 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2024-06-18 16:55:59,985][  INFO][PID:1275962] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c2a6]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/07624836-dotnet9.0 started
[2024-06-18 16:55:59,986][  INFO][PID:1275962] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c2a6]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/07624836-dotnet9.0/ &> /var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/07624836-dotnet9.0/build-07624836.rsync.log
[2024-06-18 16:57:00,128][  INFO][PID:1275962] rsyncing finished.
[2024-06-18 16:57:00,129][ ERROR][PID:1275962] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2024-06-18 16:57:00,129][ ERROR][PID:1275962] SSH connection lost on #3 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2024-06-18 16:57:10,130][ ERROR][PID:1275962] Re-try request for task on 'ResallocHost, ticket_id=3381280, hostname=2620:52:3:1:dead:beef:cafe:c2a6': Unable to finish after 3 SSH attempts
[2024-06-18 16:57:10,131][  INFO][PID:1275962] Releasing VM back to pool
[2024-06-18 16:57:10,146][  INFO][PID:1275962] Retry #1 (on other host)
[2024-06-18 16:57:10,213][  INFO][PID:1275962] Trying to allocate VM
[2024-06-18 16:57:22,286][  INFO][PID:1275962] Allocated host ResallocHost, ticket_id=3382883, hostname=2620:52:3:1:dead:beef:cafe:c20f
[2024-06-18 16:57:22,286][  INFO][PID:1275962] Allocating ssh connection to builder
[2024-06-18 16:57:22,287][  INFO][PID:1275962] Checking that builder machine is OK
[2024-06-18 16:57:23,628][  INFO][PID:1275962] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c20f '/usr/bin/test -f /etc/mock/fedora-40-ppc64le.cfg'
[2024-06-18 16:57:24,244][  INFO][PID:1275962] Filling build.info file with builder info
[2024-06-18 16:57:24,375][  INFO][PID:1275962] Not re-notifying FE and msg buses for the new host.
[2024-06-18 16:57:24,376][  INFO][PID:1275962] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/7624836-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached
[2024-06-18 16:57:25,398][  INFO][PID:1275962] Downloading the builder-live.log file, attempt 1
[2024-06-18 16:57:25,519][  INFO][PID:1275962] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c20f copr-rpmbuild-log
[2024-06-18 16:57:30,407][  INFO][PID:1275962] Status: False
[2024-06-18 17:07:30,506][  INFO][PID:1275962] Status: False
[2024-06-18 17:17:30,657][  INFO][PID:1275962] Status: False
[2024-06-18 17:27:30,831][  INFO][PID:1275962] Status: False
[2024-06-18 17:37:31,014][  INFO][PID:1275962] Status: False
[2024-06-18 17:47:31,236][  INFO][PID:1275962] Status: False
[2024-06-18 17:57:31,355][  INFO][PID:1275962] Status: False
[2024-06-18 18:07:31,487][  INFO][PID:1275962] Status: False
[2024-06-18 18:17:31,601][  INFO][PID:1275962] Status: False
[2024-06-18 18:27:31,700][  INFO][PID:1275962] Status: False
[2024-06-18 18:37:31,774][  INFO][PID:1275962] Status: False
[2024-06-18 18:47:31,843][  INFO][PID:1275962] Status: False
[2024-06-18 18:57:31,916][  INFO][PID:1275962] Status: False
[2024-06-18 19:07:32,029][  INFO][PID:1275962] Status: False
[2024-06-18 19:17:32,115][  INFO][PID:1275962] Status: False
[2024-06-18 19:27:32,201][  INFO][PID:1275962] Status: False
[2024-06-18 19:37:32,276][  INFO][PID:1275962] Status: False
[2024-06-18 19:47:32,366][  INFO][PID:1275962] Status: False
[2024-06-18 19:57:32,445][  INFO][PID:1275962] Status: False
[2024-06-18 20:07:32,579][  INFO][PID:1275962] Status: False
[2024-06-18 20:17:32,661][  INFO][PID:1275962] Status: False
[2024-06-18 20:27:32,735][  INFO][PID:1275962] Status: False
[2024-06-18 20:32:35,062][  INFO][PID:1275962] Downloading results from builder
[2024-06-18 20:32:35,064][  INFO][PID:1275962] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c20f]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/07624836-dotnet9.0 started
[2024-06-18 20:32:35,065][  INFO][PID:1275962] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c20f]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/07624836-dotnet9.0/ &> /var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/07624836-dotnet9.0/build-07624836.rsync.log
[2024-06-18 20:33:02,354][  INFO][PID:1275962] rsyncing finished.
[2024-06-18 20:33:02,355][  INFO][PID:1275962] Releasing VM back to pool
[2024-06-18 20:33:02,363][  INFO][PID:1275962] Searching for 'success' file in resultdir
[2024-06-18 20:33:02,366][ ERROR][PID:1275962] Build failed: Backend process error: No success file => build failure
[2024-06-18 20:33:02,427][  INFO][PID:1275962] Finished build: id=7624836 failed=True timeout=36000 destdir=/var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview chroot=fedora-40-ppc64le 
[2024-06-18 20:33:02,457][ ERROR][PID:1275962] Backend process error: No success file => build failure
[2024-06-18 20:33:02,458][  INFO][PID:1275962] Worker failed build, took 12938.082358121872
[2024-06-18 20:33:02,469][  INFO][PID:1275962] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 36000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@dotnet-sig",
            "project_name": "dotnet-preview",
            "project_dirname": "dotnet-preview",
            "submitter": "omajid",
            "ended_on": 1718742782.4582152,
            "started_on": 1718729844.375857,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-40-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "7624836-fedora-40-ppc64le",
            "build_id": 7624836,
            "package_name": "dotnet9.0",
            "package_version": "9.0.100~preview.5.24307.1-0.9",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@dotnet-sig/dotnet-preview/dotnet9.0",
            "git_hash": "04e6f02407a91ef49dbcff1aaa474328dbe0834d",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@dotnet-sig/dotnet-preview--omajid",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "background": false,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@dotnet-sig/dotnet-preview",
            "result_dir": "07624836-dotnet9.0",
            "built_packages": "",
            "id": 7624836,
            "mockchain_macros": {
                "copr_username": "@dotnet-sig",
                "copr_projectname": "dotnet-preview",
                "vendor": "Fedora Project COPR (@dotnet-sig/dotnet-preview)"
            }
        }
    ]
}
[2024-06-18 20:33:02,967][  INFO][PID:1275962] Sending fedora-messaging bus message in build.end
[2024-06-18 20:33:03,193][  INFO][PID:1275962] Compressing /var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/07624836-dotnet9.0/builder-live.log by gzip
[2024-06-18 20:33:03,198][  INFO][PID:1275962] Running command 'gzip /var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/07624836-dotnet9.0/builder-live.log' as PID 2917450
[2024-06-18 20:33:04,421][  INFO][PID:1275962] Finished after 1 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/07624836-dotnet9.0/builder-live.log)
[2024-06-18 20:33:04,426][  INFO][PID:1275962] Compressing /var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/07624836-dotnet9.0/backend.log by gzip
[2024-06-18 20:33:04,428][  INFO][PID:1275962] Running command 'gzip /var/lib/copr/public_html/results/@dotnet-sig/dotnet-preview/fedora-40-ppc64le/07624836-dotnet9.0/backend.log' as PID 2917456