[2025-03-13 18:07:43,827][ INFO][PID:1996324] Marking build as starting
[2025-03-13 18:07:43,948][ INFO][PID:1996324] Trying to allocate VM
[2025-03-13 18:07:56,043][ INFO][PID:1996324] Allocated host ResallocHost, ticket_id=1802093, hostname=140.211.168.171, name=copr_osuosl_p09_prod_01930443_20250313_144754
[2025-03-13 18:07:56,045][ INFO][PID:1996324] Allocating ssh connection to builder
[2025-03-13 18:07:56,045][ INFO][PID:1996324] Checking that builder machine is OK
[2025-03-13 18:07:57,392][ INFO][PID:1996324] Running remote command: copr-builder-ready fedora-41-ppc64le
[2025-03-13 18:07:58,530][ INFO][PID:1996324] Red Hat subscription not needed for fedora-41-ppc64le
Builder is ready to be used
[2025-03-13 18:07:58,531][ INFO][PID:1996324] Filling build.info file with builder info
[2025-03-13 18:07:58,535][ INFO][PID:1996324] Sending build state back to frontend: {
"builds": [
{
"timeout": 18000,
"frontend_base_url": "https://copr.fedorainfracloud.org",
"memory_reqs": 2048,
"enable_net": true,
"project_owner": "brycensranch",
"project_name": "rokon",
"project_dirname": "rokon",
"submitter": "https://api.github.com/users/renovate%5Bbot%5D",
"ended_on": null,
"started_on": 1741889278.5355527,
"submitted_on": null,
"status": 3,
"chroot": "fedora-41-ppc64le",
"arch": "ppc64le",
"buildroot_pkgs": [],
"task_id": "8766273-fedora-41-ppc64le",
"build_id": 8766273,
"package_name": "rokon-gtk",
"package_version": "1.0.0-1",
"git_repo": "https://copr-dist-git.fedorainfracloud.org/git/brycensranch/rokon/rokon-gtk",
"git_hash": "25ffa8b2a8b395657419bc8a5155391faaa3fea7",
"git_branch": null,
"source_type": null,
"source_json": null,
"pkg_name": null,
"pkg_main_version": null,
"pkg_epoch": null,
"pkg_release": null,
"srpm_url": null,
"uses_devel_repo": false,
"sandbox": "brycensranch/rokon--https://api.github.com/users/renovate%5Bbot%5D",
"results": null,
"appstream": false,
"allow_user_ssh": false,
"ssh_public_keys": null,
"storage": null,
"background": true,
"fedora_review": true,
"isolation": "default",
"repo_priority": null,
"repos": [
{
"baseurl": "https://download.copr.fedorainfracloud.org/results/brycensranch/rokon/fedora-41-ppc64le/",
"id": "copr_base",
"name": "Copr repository",
"priority": null
}
],
"tags": [
"arch_ppc64le"
],
"with_opts": [],
"without_opts": [],
"destdir": "/var/lib/copr/public_html/results/brycensranch/rokon",
"results_repo_url": "https://download.copr.fedorainfracloud.org/results/brycensranch/rokon",
"result_dir": "08766273-rokon-gtk",
"built_packages": "",
"id": 8766273,
"mockchain_macros": {
"copr_username": "brycensranch",
"copr_projectname": "rokon",
"vendor": "Fedora Project COPR (brycensranch/rokon)"
}
}
]
}
[2025-03-13 18:07:58,622][ INFO][PID:1996324] Sending fedora-messaging bus message in build.start
[2025-03-13 18:08:00,235][ INFO][PID:1996324] Sending fedora-messaging bus message in chroot.start
[2025-03-13 18:08:00,260][ INFO][PID:1996324] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8766273-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached
[2025-03-13 18:08:01,582][ INFO][PID:1996324] Downloading the builder-live.log file, attempt 1
[2025-03-13 18:08:01,587][ INFO][PID:1996324] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.171 copr-rpmbuild-log
[2025-03-13 18:08:06,592][ INFO][PID:1996324] Periodic builder liveness probe: alive
[2025-03-13 18:18:06,871][ INFO][PID:1996324] Periodic builder liveness probe: alive
[2025-03-13 18:28:07,068][ INFO][PID:1996324] Periodic builder liveness probe: alive
[2025-03-13 18:38:07,242][ INFO][PID:1996324] Periodic builder liveness probe: dead
[2025-03-13 18:39:07,315][ ERROR][PID:1996324] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:
ERR:
ssh: connect to host 140.211.168.171 port 22: Connection timed out
[2025-03-13 18:39:27,235][ ERROR][PID:1996324] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-13 18:39:37,237][WARNING][PID:1996324] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-13 18:39:47,238][ INFO][PID:1996324] Downloading the builder-live.log file, attempt 2
[2025-03-13 18:39:47,240][ INFO][PID:1996324] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.171 copr-rpmbuild-log
[2025-03-13 18:40:17,395][ ERROR][PID:1996324] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke:
OUT:
ERR:
ssh: connect to host 140.211.168.171 port 22: Connection timed out
[2025-03-13 18:40:47,310][ ERROR][PID:1996324] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-13 18:40:57,311][WARNING][PID:1996324] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-13 18:41:07,651][ INFO][PID:1996324] Downloading the builder-live.log file, attempt 3
[2025-03-13 18:41:07,654][ INFO][PID:1996324] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.171 copr-rpmbuild-log
[2025-03-13 18:41:27,468][ ERROR][PID:1996324] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke:
OUT:
ERR:
ssh: connect to host 140.211.168.171 port 22: Connection timed out
[2025-03-13 18:42:07,727][ ERROR][PID:1996324] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-13 18:42:17,940][WARNING][PID:1996324] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-13 18:42:27,941][ INFO][PID:1996324] Downloading the builder-live.log file, attempt 4
[2025-03-13 18:42:27,944][ INFO][PID:1996324] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.171 copr-rpmbuild-log
[2025-03-13 18:42:37,546][ ERROR][PID:1996324] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke:
OUT:
ERR:
ssh: connect to host 140.211.168.171 port 22: Connection timed out
[2025-03-13 18:42:47,547][ ERROR][PID:1996324] Can't ssh to cancel build.
[2025-03-13 18:43:28,014][ ERROR][PID:1996324] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-13 18:43:38,018][ ERROR][PID:1996324] Giving up for unstable SSH, failures: 1741891177.2378628, 1741891257.3111503, 1741891337.9409046, 1741891418.0186868
[2025-03-13 18:43:38,019][ ERROR][PID:1996324] Re-try request for task on 'ResallocHost, ticket_id=1802093, hostname=140.211.168.171, name=copr_osuosl_p09_prod_01930443_20250313_144754': SSH problems when downloading live log: Resalloc ticket FAILED
[2025-03-13 18:43:38,020][ INFO][PID:1996324] Releasing VM back to pool
[2025-03-13 18:43:38,032][ INFO][PID:1996324] Retry #1 (on other host)
[2025-03-13 18:43:38,033][ INFO][PID:1996324] Trying to allocate VM
[2025-03-13 18:43:41,059][ INFO][PID:1996324] Allocated host ResallocHost, ticket_id=1803588, hostname=2620:52:3:1:dead:beef:cafe:c288, name=copr_p09_01_prod_01938015_20250313_181602
[2025-03-13 18:43:41,060][ INFO][PID:1996324] Allocating ssh connection to builder
[2025-03-13 18:43:41,061][ INFO][PID:1996324] Checking that builder machine is OK
[2025-03-13 18:43:41,892][ INFO][PID:1996324] Running remote command: copr-builder-ready fedora-41-ppc64le
[2025-03-13 18:43:42,385][ INFO][PID:1996324] Red Hat subscription not needed for fedora-41-ppc64le
Builder is ready to be used
[2025-03-13 18:43:42,386][ INFO][PID:1996324] Filling build.info file with builder info
[2025-03-13 18:43:42,405][ INFO][PID:1996324] Not re-notifying FE and msg buses for the new host.
[2025-03-13 18:43:42,406][ INFO][PID:1996324] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8766273-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached
[2025-03-13 18:43:43,228][ INFO][PID:1996324] Downloading the builder-live.log file, attempt 1
[2025-03-13 18:43:43,230][ INFO][PID:1996324] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c288 copr-rpmbuild-log
[2025-03-13 18:44:48,300][ ERROR][PID:1996324] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:
ERR:
ssh: connect to host 2620:52:3:1:dead:beef:cafe:c288 port 22: Connection timed out
[2025-03-13 18:45:45,229][ ERROR][PID:1996324] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-13 18:45:55,230][WARNING][PID:1996324] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-13 18:45:58,374][ ERROR][PID:1996324] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke:
OUT:
ERR:
ssh: connect to host 2620:52:3:1:dead:beef:cafe:c288 port 22: Connection timed out
[2025-03-13 18:46:05,232][ INFO][PID:1996324] Downloading the builder-live.log file, attempt 2
[2025-03-13 18:46:05,233][ INFO][PID:1996324] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c288 copr-rpmbuild-log
[2025-03-13 18:47:05,302][ ERROR][PID:1996324] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-13 18:47:08,447][ ERROR][PID:1996324] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke:
OUT:
ERR:
ssh: connect to host 2620:52:3:1:dead:beef:cafe:c288 port 22: Connection timed out
[2025-03-13 18:47:15,303][WARNING][PID:1996324] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-13 18:47:25,304][ INFO][PID:1996324] Downloading the builder-live.log file, attempt 3
[2025-03-13 18:47:25,307][ INFO][PID:1996324] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c288 copr-rpmbuild-log
[2025-03-13 18:48:18,526][ ERROR][PID:1996324] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke:
OUT:
ERR:
ssh: connect to host 2620:52:3:1:dead:beef:cafe:c288 port 22: Connection timed out
[2025-03-13 18:48:25,380][ ERROR][PID:1996324] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-13 18:48:28,529][ ERROR][PID:1996324] Can't ssh to cancel build.
[2025-03-13 18:48:35,398][WARNING][PID:1996324] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-13 18:48:45,400][ INFO][PID:1996324] Downloading the builder-live.log file, attempt 4
[2025-03-13 18:48:45,404][ INFO][PID:1996324] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c288 copr-rpmbuild-log
[2025-03-13 18:49:45,478][ ERROR][PID:1996324] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-13 18:49:55,496][ ERROR][PID:1996324] Giving up for unstable SSH, failures: 1741891555.2307506, 1741891635.3033845, 1741891715.398786, 1741891795.4961398
[2025-03-13 18:49:55,497][ ERROR][PID:1996324] Re-try request for task on 'ResallocHost, ticket_id=1803588, hostname=2620:52:3:1:dead:beef:cafe:c288, name=copr_p09_01_prod_01938015_20250313_181602': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-03-13 18:49:55,497][ INFO][PID:1996324] Releasing VM back to pool
[2025-03-13 18:49:55,509][ INFO][PID:1996324] Retry #2 (on other host)
[2025-03-13 18:49:55,510][ INFO][PID:1996324] Trying to allocate VM
[2025-03-13 18:49:58,539][ INFO][PID:1996324] Allocated host ResallocHost, ticket_id=1803751, hostname=2620:52:3:1:dead:beef:cafe:c20c, name=copr_hv_ppc64le_01_prod_01938040_20250313_181643
[2025-03-13 18:49:58,540][ INFO][PID:1996324] Allocating ssh connection to builder
[2025-03-13 18:49:58,540][ INFO][PID:1996324] Checking that builder machine is OK
[2025-03-13 18:49:59,353][ INFO][PID:1996324] Running remote command: copr-builder-ready fedora-41-ppc64le
[2025-03-13 18:49:59,742][ INFO][PID:1996324] Red Hat subscription not needed for fedora-41-ppc64le
Builder is ready to be used
[2025-03-13 18:49:59,743][ INFO][PID:1996324] Filling build.info file with builder info
[2025-03-13 18:49:59,764][ INFO][PID:1996324] Not re-notifying FE and msg buses for the new host.
[2025-03-13 18:49:59,765][ INFO][PID:1996324] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8766273-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached
[2025-03-13 18:50:00,466][ INFO][PID:1996324] Downloading the builder-live.log file, attempt 1
[2025-03-13 18:50:00,467][ INFO][PID:1996324] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c20c copr-rpmbuild-log
[2025-03-13 18:50:05,473][ INFO][PID:1996324] Periodic builder liveness probe: alive
[2025-03-13 18:55:15,436][ INFO][PID:1996324] Downloading results from builder
[2025-03-13 18:55:15,437][ INFO][PID:1996324] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c20c]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/brycensranch/rokon/fedora-41-ppc64le/08766273-rokon-gtk started
[2025-03-13 18:55:15,438][ INFO][PID:1996324] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c20c]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/brycensranch/rokon/fedora-41-ppc64le/08766273-rokon-gtk/ &> /var/lib/copr/public_html/results/brycensranch/rokon/fedora-41-ppc64le/08766273-rokon-gtk/build-08766273.rsync.log
[2025-03-13 18:55:16,102][ INFO][PID:1996324] rsyncing finished.
[2025-03-13 18:55:16,103][ INFO][PID:1996324] Releasing VM back to pool
[2025-03-13 18:55:16,110][ INFO][PID:1996324] Searching for 'success' file in resultdir
[2025-03-13 18:55:16,111][ ERROR][PID:1996324] Build failed: Backend process error: No success file => build failure
[2025-03-13 18:55:16,111][ INFO][PID:1996324] Finished build: id=8766273 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/brycensranch/rokon chroot=fedora-41-ppc64le
[2025-03-13 18:55:16,113][ ERROR][PID:1996324] Backend process error: No success file => build failure
[2025-03-13 18:55:16,114][ INFO][PID:1996324] Worker failed build, took 316.3495104312897
[2025-03-13 18:55:16,114][ INFO][PID:1996324] Sending build state back to frontend: {
"builds": [
{
"timeout": 18000,
"frontend_base_url": "https://copr.fedorainfracloud.org",
"memory_reqs": 2048,
"enable_net": true,
"project_owner": "brycensranch",
"project_name": "rokon",
"project_dirname": "rokon",
"submitter": "https://api.github.com/users/renovate%5Bbot%5D",
"ended_on": 1741892116.1142669,
"started_on": 1741891799.7647564,
"submitted_on": null,
"status": 0,
"chroot": "fedora-41-ppc64le",
"arch": "ppc64le",
"buildroot_pkgs": [],
"task_id": "8766273-fedora-41-ppc64le",
"build_id": 8766273,
"package_name": "rokon-gtk",
"package_version": "1.0.0-1",
"git_repo": "https://copr-dist-git.fedorainfracloud.org/git/brycensranch/rokon/rokon-gtk",
"git_hash": "25ffa8b2a8b395657419bc8a5155391faaa3fea7",
"git_branch": null,
"source_type": null,
"source_json": null,
"pkg_name": null,
"pkg_main_version": null,
"pkg_epoch": null,
"pkg_release": null,
"srpm_url": null,
"uses_devel_repo": false,
"sandbox": "brycensranch/rokon--https://api.github.com/users/renovate%5Bbot%5D",
"results": null,
"appstream": false,
"allow_user_ssh": false,
"ssh_public_keys": null,
"storage": null,
"background": true,
"fedora_review": true,
"isolation": "default",
"repo_priority": null,
"repos": [
{
"baseurl": "https://download.copr.fedorainfracloud.org/results/brycensranch/rokon/fedora-41-ppc64le/",
"id": "copr_base",
"name": "Copr repository",
"priority": null
}
],
"tags": [
"arch_ppc64le"
],
"with_opts": [],
"without_opts": [],
"destdir": "/var/lib/copr/public_html/results/brycensranch/rokon",
"results_repo_url": "https://download.copr.fedorainfracloud.org/results/brycensranch/rokon",
"result_dir": "08766273-rokon-gtk",
"built_packages": "",
"id": 8766273,
"mockchain_macros": {
"copr_username": "brycensranch",
"copr_projectname": "rokon",
"vendor": "Fedora Project COPR (brycensranch/rokon)"
}
}
]
}
[2025-03-13 18:55:16,224][ INFO][PID:1996324] Sending fedora-messaging bus message in build.end
[2025-03-13 18:55:16,243][ INFO][PID:1996324] Compressing /var/lib/copr/public_html/results/brycensranch/rokon/fedora-41-ppc64le/08766273-rokon-gtk/builder-live.log by gzip
[2025-03-13 18:55:16,245][ INFO][PID:1996324] Running command 'gzip /var/lib/copr/public_html/results/brycensranch/rokon/fedora-41-ppc64le/08766273-rokon-gtk/builder-live.log' as PID 2413697
[2025-03-13 18:55:16,267][ INFO][PID:1996324] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/brycensranch/rokon/fedora-41-ppc64le/08766273-rokon-gtk/builder-live.log)
[2025-03-13 18:55:16,268][ INFO][PID:1996324] Compressing /var/lib/copr/public_html/results/brycensranch/rokon/fedora-41-ppc64le/08766273-rokon-gtk/backend.log by gzip
[2025-03-13 18:55:16,269][ INFO][PID:1996324] Running command 'gzip /var/lib/copr/public_html/results/brycensranch/rokon/fedora-41-ppc64le/08766273-rokon-gtk/backend.log' as PID 2413702