[2025-03-10 15:29:00,521][ INFO][PID:1953261] Marking build as starting
[2025-03-10 15:29:00,647][ INFO][PID:1953261] Trying to allocate VM
[2025-03-10 15:29:03,777][ INFO][PID:1953261] Allocated host ResallocHost, ticket_id=1727490, hostname=140.211.168.184, name=copr_osuosl_p09_prod_01820078_20250310_101859
[2025-03-10 15:29:03,777][ INFO][PID:1953261] Allocating ssh connection to builder
[2025-03-10 15:29:03,778][ INFO][PID:1953261] Checking that builder machine is OK
[2025-03-10 15:29:05,100][ INFO][PID:1953261] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2025-03-10 15:29:06,093][ INFO][PID:1953261] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used
[2025-03-10 15:29:06,094][ INFO][PID:1953261] Filling build.info file with builder info
[2025-03-10 15:29:06,095][ INFO][PID:1953261] Sending build state back to frontend: {
"builds": [
{
"timeout": 18000,
"frontend_base_url": "https://copr.fedorainfracloud.org",
"memory_reqs": 2048,
"enable_net": false,
"project_owner": "packit",
"project_name": "systemd-systemd-36656",
"project_dirname": "systemd-systemd-36656",
"submitter": "packit",
"ended_on": null,
"started_on": 1741620546.0954323,
"submitted_on": null,
"status": 3,
"chroot": "fedora-rawhide-ppc64le",
"arch": "ppc64le",
"buildroot_pkgs": [],
"task_id": "8746850-fedora-rawhide-ppc64le",
"build_id": 8746850,
"package_name": "systemd",
"package_version": "258~devel-1.20250310152826258608.pr36656.2038.g6104488f2c",
"git_repo": "https://copr-dist-git.fedorainfracloud.org/git/packit/systemd-systemd-36656/systemd",
"git_hash": "f222f23e43b5cc718d834d42e07a322b52a0dbd2",
"git_branch": null,
"source_type": null,
"source_json": null,
"pkg_name": null,
"pkg_main_version": null,
"pkg_epoch": null,
"pkg_release": null,
"srpm_url": null,
"uses_devel_repo": false,
"sandbox": "packit/systemd-systemd-36656--packit",
"results": null,
"appstream": false,
"allow_user_ssh": false,
"ssh_public_keys": null,
"storage": 0,
"background": false,
"fedora_review": false,
"isolation": "default",
"repo_priority": null,
"repos": [
{
"baseurl": "https://download.copr.fedorainfracloud.org/results/packit/systemd-systemd-36656/fedora-rawhide-ppc64le/",
"id": "copr_base",
"name": "Copr repository",
"priority": null
}
],
"tags": [
"arch_ppc64le"
],
"with_opts": [],
"without_opts": [],
"destdir": "/var/lib/copr/public_html/results/packit/systemd-systemd-36656",
"results_repo_url": "https://download.copr.fedorainfracloud.org/results/packit/systemd-systemd-36656",
"result_dir": "08746850-systemd",
"built_packages": "",
"id": 8746850,
"mockchain_macros": {
"copr_username": "packit",
"copr_projectname": "systemd-systemd-36656",
"vendor": "Fedora Project COPR (packit/systemd-systemd-36656)"
}
}
]
}
[2025-03-10 15:29:06,126][ INFO][PID:1953261] Sending fedora-messaging bus message in build.start
[2025-03-10 15:29:06,846][ INFO][PID:1953261] Sending fedora-messaging bus message in chroot.start
[2025-03-10 15:29:06,865][ INFO][PID:1953261] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8746850-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2025-03-10 15:29:08,209][ INFO][PID:1953261] Downloading the builder-live.log file, attempt 1
[2025-03-10 15:29:08,211][ INFO][PID:1953261] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.184 copr-rpmbuild-log
[2025-03-10 15:29:13,221][ INFO][PID:1953261] Periodic builder liveness probe: alive
[2025-03-10 15:39:13,330][ INFO][PID:1953261] Periodic builder liveness probe: alive
[2025-03-10 15:49:13,448][ INFO][PID:1953261] Periodic builder liveness probe: alive
[2025-03-10 15:59:13,594][ INFO][PID:1953261] Periodic builder liveness probe: alive
[2025-03-10 16:09:13,893][ INFO][PID:1953261] Periodic builder liveness probe: alive
[2025-03-10 16:19:14,000][ INFO][PID:1953261] Periodic builder liveness probe: alive
[2025-03-10 16:29:14,119][ INFO][PID:1953261] Periodic builder liveness probe: alive
[2025-03-10 16:37:24,253][ ERROR][PID:1953261] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-10 16:37:34,286][WARNING][PID:1953261] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-10 16:37:44,287][ INFO][PID:1953261] Downloading the builder-live.log file, attempt 2
[2025-03-10 16:37:44,294][ INFO][PID:1953261] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.184 copr-rpmbuild-log
[2025-03-10 16:38:44,364][ ERROR][PID:1953261] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-10 16:38:54,369][WARNING][PID:1953261] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-10 16:39:04,371][ INFO][PID:1953261] Downloading the builder-live.log file, attempt 3
[2025-03-10 16:39:04,373][ INFO][PID:1953261] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.184 copr-rpmbuild-log
[2025-03-10 16:39:14,210][ INFO][PID:1953261] Periodic builder liveness probe: dead
[2025-03-10 16:40:04,443][ ERROR][PID:1953261] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-10 16:40:14,281][ ERROR][PID:1953261] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:
ERR:
ssh: connect to host 140.211.168.184 port 22: Connection timed out
[2025-03-10 16:40:14,446][WARNING][PID:1953261] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-10 16:40:24,447][ INFO][PID:1953261] Downloading the builder-live.log file, attempt 4
[2025-03-10 16:40:24,450][ INFO][PID:1953261] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.184 copr-rpmbuild-log
[2025-03-10 16:41:24,352][ ERROR][PID:1953261] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke:
OUT:
ERR:
ssh: connect to host 140.211.168.184 port 22: Connection timed out
[2025-03-10 16:41:24,523][ ERROR][PID:1953261] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-10 16:41:34,524][ ERROR][PID:1953261] Giving up for unstable SSH, failures: 1741624654.2867787, 1741624734.3695934, 1741624814.4459949, 1741624894.5244372
[2025-03-10 16:42:34,423][ ERROR][PID:1953261] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke:
OUT:
ERR:
ssh: connect to host 140.211.168.184 port 22: Connection timed out
[2025-03-10 16:43:44,495][ ERROR][PID:1953261] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke:
OUT:
ERR:
ssh: connect to host 140.211.168.184 port 22: Connection timed out
[2025-03-10 16:43:54,497][ ERROR][PID:1953261] Can't ssh to cancel build.
[2025-03-10 16:43:54,499][ ERROR][PID:1953261] Re-try request for task on 'ResallocHost, ticket_id=1727490, hostname=140.211.168.184, name=copr_osuosl_p09_prod_01820078_20250310_101859': SSH problems when downloading live log: Resalloc ticket FAILED
[2025-03-10 16:43:54,499][ INFO][PID:1953261] Releasing VM back to pool
[2025-03-10 16:43:54,515][ INFO][PID:1953261] Retry #1 (on other host)
[2025-03-10 16:43:54,542][ INFO][PID:1953261] Trying to allocate VM
[2025-03-10 16:43:57,582][ INFO][PID:1953261] Allocated host ResallocHost, ticket_id=1728879, hostname=2620:52:3:1:dead:beef:cafe:c2a3, name=copr_p09_01_prod_01830262_20250310_161818
[2025-03-10 16:43:57,583][ INFO][PID:1953261] Allocating ssh connection to builder
[2025-03-10 16:43:57,583][ INFO][PID:1953261] Checking that builder machine is OK
[2025-03-10 16:43:58,567][ INFO][PID:1953261] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2025-03-10 16:43:59,111][ INFO][PID:1953261] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used
[2025-03-10 16:43:59,113][ INFO][PID:1953261] Filling build.info file with builder info
[2025-03-10 16:43:59,124][ INFO][PID:1953261] Not re-notifying FE and msg buses for the new host.
[2025-03-10 16:43:59,127][ INFO][PID:1953261] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8746850-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2025-03-10 16:44:00,109][ INFO][PID:1953261] Downloading the builder-live.log file, attempt 1
[2025-03-10 16:44:00,111][ INFO][PID:1953261] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c2a3 copr-rpmbuild-log
[2025-03-10 16:44:08,913][ INFO][PID:1953261] Cancel request succeeded
out:
success
err:
Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c2a3' (ED25519) to the list of known hosts.
[2025-03-10 16:44:08,915][ INFO][PID:1953261] Downloading results from builder
[2025-03-10 16:44:08,916][ INFO][PID:1953261] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c2a3]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/packit/systemd-systemd-36656/fedora-rawhide-ppc64le/08746850-systemd started
[2025-03-10 16:44:08,918][ INFO][PID:1953261] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c2a3]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/packit/systemd-systemd-36656/fedora-rawhide-ppc64le/08746850-systemd/ &> /var/lib/copr/public_html/results/packit/systemd-systemd-36656/fedora-rawhide-ppc64le/08746850-systemd/build-08746850.rsync.log
[2025-03-10 16:44:09,448][ INFO][PID:1953261] rsyncing finished.
[2025-03-10 16:44:09,449][ INFO][PID:1953261] Releasing VM back to pool
[2025-03-10 16:44:09,461][ INFO][PID:1953261] Searching for 'success' file in resultdir
[2025-03-10 16:44:09,461][ ERROR][PID:1953261] Build failed: Backend process error: No success file => build failure
[2025-03-10 16:44:09,462][ INFO][PID:1953261] Finished build: id=8746850 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/packit/systemd-systemd-36656 chroot=fedora-rawhide-ppc64le
[2025-03-10 16:44:09,464][ ERROR][PID:1953261] Backend process error: No success file => build failure
[2025-03-10 16:44:09,465][ INFO][PID:1953261] Worker failed build, took 10.341099262237549
[2025-03-10 16:44:09,465][ INFO][PID:1953261] Sending build state back to frontend: {
"builds": [
{
"timeout": 18000,
"frontend_base_url": "https://copr.fedorainfracloud.org",
"memory_reqs": 2048,
"enable_net": false,
"project_owner": "packit",
"project_name": "systemd-systemd-36656",
"project_dirname": "systemd-systemd-36656",
"submitter": "packit",
"ended_on": 1741625049.4651325,
"started_on": 1741625039.1240332,
"submitted_on": null,
"status": 0,
"chroot": "fedora-rawhide-ppc64le",
"arch": "ppc64le",
"buildroot_pkgs": [],
"task_id": "8746850-fedora-rawhide-ppc64le",
"build_id": 8746850,
"package_name": "systemd",
"package_version": "258~devel-1.20250310152826258608.pr36656.2038.g6104488f2c",
"git_repo": "https://copr-dist-git.fedorainfracloud.org/git/packit/systemd-systemd-36656/systemd",
"git_hash": "f222f23e43b5cc718d834d42e07a322b52a0dbd2",
"git_branch": null,
"source_type": null,
"source_json": null,
"pkg_name": null,
"pkg_main_version": null,
"pkg_epoch": null,
"pkg_release": null,
"srpm_url": null,
"uses_devel_repo": false,
"sandbox": "packit/systemd-systemd-36656--packit",
"results": null,
"appstream": false,
"allow_user_ssh": false,
"ssh_public_keys": null,
"storage": 0,
"background": false,
"fedora_review": false,
"isolation": "default",
"repo_priority": null,
"repos": [
{
"baseurl": "https://download.copr.fedorainfracloud.org/results/packit/systemd-systemd-36656/fedora-rawhide-ppc64le/",
"id": "copr_base",
"name": "Copr repository",
"priority": null
}
],
"tags": [
"arch_ppc64le"
],
"with_opts": [],
"without_opts": [],
"destdir": "/var/lib/copr/public_html/results/packit/systemd-systemd-36656",
"results_repo_url": "https://download.copr.fedorainfracloud.org/results/packit/systemd-systemd-36656",
"result_dir": "08746850-systemd",
"built_packages": "",
"id": 8746850,
"mockchain_macros": {
"copr_username": "packit",
"copr_projectname": "systemd-systemd-36656",
"vendor": "Fedora Project COPR (packit/systemd-systemd-36656)"
}
}
]
}
[2025-03-10 16:44:09,497][ INFO][PID:1953261] Sending fedora-messaging bus message in build.end
[2025-03-10 16:44:09,580][ INFO][PID:1953261] Compressing /var/lib/copr/public_html/results/packit/systemd-systemd-36656/fedora-rawhide-ppc64le/08746850-systemd/builder-live.log by gzip
[2025-03-10 16:44:09,581][ INFO][PID:1953261] Running command 'gzip /var/lib/copr/public_html/results/packit/systemd-systemd-36656/fedora-rawhide-ppc64le/08746850-systemd/builder-live.log' as PID 2570468
[2025-03-10 16:44:09,583][ INFO][PID:1953261] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/packit/systemd-systemd-36656/fedora-rawhide-ppc64le/08746850-systemd/builder-live.log)
[2025-03-10 16:44:09,584][ INFO][PID:1953261] Compressing /var/lib/copr/public_html/results/packit/systemd-systemd-36656/fedora-rawhide-ppc64le/08746850-systemd/backend.log by gzip
[2025-03-10 16:44:09,585][ INFO][PID:1953261] Running command 'gzip /var/lib/copr/public_html/results/packit/systemd-systemd-36656/fedora-rawhide-ppc64le/08746850-systemd/backend.log' as PID 2570469