[2025-03-03 11:52:27,856][ INFO][PID:2369213] Marking build as starting
[2025-03-03 11:52:28,020][ INFO][PID:2369213] Trying to allocate VM
[2025-03-03 11:52:31,077][ INFO][PID:2369213] Allocated host ResallocHost, ticket_id=1604411, hostname=2620:52:3:1:dead:beef:cafe:c290, name=copr_p09_01_prod_01636005_20250303_114104
[2025-03-03 11:52:31,078][ INFO][PID:2369213] Allocating ssh connection to builder
[2025-03-03 11:52:31,079][ INFO][PID:2369213] Checking that builder machine is OK
[2025-03-03 11:52:31,789][ INFO][PID:2369213] Running remote command: copr-builder-ready fedora-41-ppc64le
[2025-03-03 11:52:32,222][ INFO][PID:2369213] Red Hat subscription not needed for fedora-41-ppc64le
Builder is ready to be used
[2025-03-03 11:52:32,224][ INFO][PID:2369213] Filling build.info file with builder info
[2025-03-03 11:52:32,226][ INFO][PID:2369213] Sending build state back to frontend: {
"builds": [
{
"timeout": 18000,
"frontend_base_url": "https://copr.fedorainfracloud.org",
"memory_reqs": 2048,
"enable_net": true,
"project_owner": "rhcontainerbot",
"project_name": "podman-next",
"project_dirname": "podman-next",
"submitter": "packit",
"ended_on": null,
"started_on": 1741002752.2258673,
"submitted_on": null,
"status": 3,
"chroot": "fedora-41-ppc64le",
"arch": "ppc64le",
"buildroot_pkgs": [],
"task_id": "8719265-fedora-41-ppc64le",
"build_id": 8719265,
"package_name": "netavark",
"package_version": "102:1.14.0-1.20250303115200200580.main.24.g1e21ecf",
"git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rhcontainerbot/podman-next/netavark",
"git_hash": "ad35ab2338c675e33180890a3259a0080cb2547d",
"git_branch": null,
"source_type": null,
"source_json": null,
"pkg_name": null,
"pkg_main_version": null,
"pkg_epoch": null,
"pkg_release": null,
"srpm_url": null,
"uses_devel_repo": false,
"sandbox": "rhcontainerbot/podman-next--packit",
"results": null,
"appstream": false,
"allow_user_ssh": false,
"ssh_public_keys": null,
"storage": null,
"background": false,
"fedora_review": false,
"isolation": "default",
"repo_priority": null,
"repos": [
{
"baseurl": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/podman-next/fedora-41-ppc64le/",
"id": "copr_base",
"name": "Copr repository",
"priority": null
}
],
"tags": [
"arch_ppc64le"
],
"with_opts": [],
"without_opts": [],
"destdir": "/var/lib/copr/public_html/results/rhcontainerbot/podman-next",
"results_repo_url": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/podman-next",
"result_dir": "08719265-netavark",
"built_packages": "",
"id": 8719265,
"mockchain_macros": {
"copr_username": "rhcontainerbot",
"copr_projectname": "podman-next",
"vendor": "Fedora Project COPR (rhcontainerbot/podman-next)"
}
}
]
}
[2025-03-03 11:52:32,308][ INFO][PID:2369213] Sending fedora-messaging bus message in build.start
[2025-03-03 11:52:33,128][ INFO][PID:2369213] Sending fedora-messaging bus message in chroot.start
[2025-03-03 11:52:33,146][ INFO][PID:2369213] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8719265-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached
[2025-03-03 11:52:33,861][ INFO][PID:2369213] Downloading the builder-live.log file, attempt 1
[2025-03-03 11:52:33,862][ INFO][PID:2369213] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c290 copr-rpmbuild-log
[2025-03-03 11:52:38,869][ INFO][PID:2369213] Periodic builder liveness probe: alive
[2025-03-03 11:58:31,229][ ERROR][PID:2369213] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-03 11:58:41,230][WARNING][PID:2369213] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-03 11:58:51,231][ INFO][PID:2369213] Downloading the builder-live.log file, attempt 2
[2025-03-03 11:58:51,233][ INFO][PID:2369213] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c290 copr-rpmbuild-log
[2025-03-03 11:59:51,301][ ERROR][PID:2369213] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-03 12:00:01,303][WARNING][PID:2369213] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-03 12:00:11,305][ INFO][PID:2369213] Downloading the builder-live.log file, attempt 3
[2025-03-03 12:00:11,306][ INFO][PID:2369213] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c290 copr-rpmbuild-log
[2025-03-03 12:01:11,375][ ERROR][PID:2369213] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-03 12:01:21,383][WARNING][PID:2369213] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-03 12:01:31,384][ INFO][PID:2369213] Downloading the builder-live.log file, attempt 4
[2025-03-03 12:01:31,385][ INFO][PID:2369213] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c290 copr-rpmbuild-log
[2025-03-03 12:02:31,456][ ERROR][PID:2369213] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-03 12:02:38,979][ INFO][PID:2369213] Periodic builder liveness probe: alive
[2025-03-03 12:02:41,458][ ERROR][PID:2369213] Giving up for unstable SSH, failures: 1741003121.2302458, 1741003201.303022, 1741003281.3831697, 1741003361.458308
[2025-03-03 12:02:41,459][ ERROR][PID:2369213] Re-try request for task on 'ResallocHost, ticket_id=1604411, hostname=2620:52:3:1:dead:beef:cafe:c290, name=copr_p09_01_prod_01636005_20250303_114104': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-03-03 12:02:41,461][ INFO][PID:2369213] Releasing VM back to pool
[2025-03-03 12:02:41,481][ INFO][PID:2369213] Retry #1 (on other host)
[2025-03-03 12:02:41,483][ INFO][PID:2369213] Trying to allocate VM
[2025-03-03 12:02:47,585][ INFO][PID:2369213] Allocated host ResallocHost, ticket_id=1604590, hostname=2620:52:3:1:dead:beef:cafe:c215, name=copr_hv_ppc64le_01_prod_01636118_20250303_114434
[2025-03-03 12:02:47,585][ INFO][PID:2369213] Allocating ssh connection to builder
[2025-03-03 12:02:47,586][ INFO][PID:2369213] Checking that builder machine is OK
[2025-03-03 12:02:48,359][ INFO][PID:2369213] Running remote command: copr-builder-ready fedora-41-ppc64le
[2025-03-03 12:02:48,778][ INFO][PID:2369213] Red Hat subscription not needed for fedora-41-ppc64le
Builder is ready to be used
[2025-03-03 12:02:48,779][ INFO][PID:2369213] Filling build.info file with builder info
[2025-03-03 12:02:48,781][ INFO][PID:2369213] Not re-notifying FE and msg buses for the new host.
[2025-03-03 12:02:48,782][ INFO][PID:2369213] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8719265-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached
[2025-03-03 12:02:49,555][ INFO][PID:2369213] Downloading the builder-live.log file, attempt 1
[2025-03-03 12:02:49,557][ INFO][PID:2369213] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c215 copr-rpmbuild-log
[2025-03-03 12:12:39,740][ INFO][PID:2369213] Periodic builder liveness probe: alive
[2025-03-03 12:18:36,607][ INFO][PID:2369213] Downloading results from builder
[2025-03-03 12:18:36,607][ INFO][PID:2369213] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c215]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le/08719265-netavark started
[2025-03-03 12:18:36,608][ INFO][PID:2369213] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c215]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le/08719265-netavark/ &> /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le/08719265-netavark/build-08719265.rsync.log
[2025-03-03 12:18:37,910][ INFO][PID:2369213] rsyncing finished.
[2025-03-03 12:18:37,911][ INFO][PID:2369213] Releasing VM back to pool
[2025-03-03 12:18:37,920][ INFO][PID:2369213] Searching for 'success' file in resultdir
[2025-03-03 12:18:37,921][ INFO][PID:2369213] Going to sign pkgs from source: 8719265-fedora-41-ppc64le in chroot: /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le
[2025-03-03 12:18:37,922][ INFO][PID:2369213] Calling '/bin/sign -u rhcontainerbot#podman-next@copr.fedorahosted.org -p' (attempt #1)
[2025-03-03 12:18:38,091][ INFO][PID:2369213] Calling '/bin/sign -4 -h sha256 -u rhcontainerbot#podman-next@copr.fedorahosted.org -r /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le/08719265-netavark/netavark-1.14.0-1.20250303115200200580.main.24.g1e21ecf.fc41.ppc64le.rpm' (attempt #1)
[2025-03-03 12:18:39,037][ INFO][PID:2369213] signed rpm: /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le/08719265-netavark/netavark-1.14.0-1.20250303115200200580.main.24.g1e21ecf.fc41.ppc64le.rpm
[2025-03-03 12:18:39,038][ INFO][PID:2369213] Calling '/bin/sign -4 -h sha256 -u rhcontainerbot#podman-next@copr.fedorahosted.org -r /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le/08719265-netavark/netavark-1.14.0-1.20250303115200200580.main.24.g1e21ecf.fc41.src.rpm' (attempt #1)
[2025-03-03 12:18:40,211][ INFO][PID:2369213] signed rpm: /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le/08719265-netavark/netavark-1.14.0-1.20250303115200200580.main.24.g1e21ecf.fc41.src.rpm
[2025-03-03 12:18:40,212][ INFO][PID:2369213] Calling '/bin/sign -4 -h sha256 -u rhcontainerbot#podman-next@copr.fedorahosted.org -r /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le/08719265-netavark/netavark-debugsource-1.14.0-1.20250303115200200580.main.24.g1e21ecf.fc41.ppc64le.rpm' (attempt #1)
[2025-03-03 12:18:41,154][ INFO][PID:2369213] signed rpm: /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le/08719265-netavark/netavark-debugsource-1.14.0-1.20250303115200200580.main.24.g1e21ecf.fc41.ppc64le.rpm
[2025-03-03 12:18:41,154][ INFO][PID:2369213] Calling '/bin/sign -4 -h sha256 -u rhcontainerbot#podman-next@copr.fedorahosted.org -r /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le/08719265-netavark/netavark-debuginfo-1.14.0-1.20250303115200200580.main.24.g1e21ecf.fc41.ppc64le.rpm' (attempt #1)
[2025-03-03 12:18:42,516][ INFO][PID:2369213] signed rpm: /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le/08719265-netavark/netavark-debuginfo-1.14.0-1.20250303115200200580.main.24.g1e21ecf.fc41.ppc64le.rpm
[2025-03-03 12:18:42,517][ INFO][PID:2369213] Sign done
[2025-03-03 12:18:42,518][ INFO][PID:2369213] Incremental createrepo run, adding 08719265-netavark into https://download.copr.fedorainfracloud.org/results/rhcontainerbot/podman-next/fedora-41-ppc64le, (auto-create-repo=True)
[2025-03-03 12:18:42,519][ INFO][PID:2369213] Running command 'copr-repo --batched /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le --add 08719265-netavark --no-appstream-metadata' as PID 2680075
[2025-03-03 12:18:44,356][ INFO][PID:2369213] Finished after 1 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le --add 08719265-netavark --no-appstream-metadata)
[2025-03-03 12:18:44,357][ INFO][PID:2369213] Getting build details
[2025-03-03 12:18:44,357][ INFO][PID:2369213] Listing built binary packages in /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le/08719265-netavark
[2025-03-03 12:18:44,357][ INFO][PID:2369213] Built packages:
['netavark 1.14.0', 'netavark-debuginfo 1.14.0', 'netavark-debugsource 1.14.0']
[2025-03-03 12:18:44,358][ INFO][PID:2369213] build details: {'built_packages': 'netavark 1.14.0\nnetavark-debuginfo 1.14.0\nnetavark-debugsource 1.14.0'}
[2025-03-03 12:18:44,358][ INFO][PID:2369213] Retrieving pubkey
[2025-03-03 12:18:44,359][ INFO][PID:2369213] Calling '/bin/sign -u rhcontainerbot#podman-next@copr.fedorahosted.org -p' (attempt #1)
[2025-03-03 12:18:44,531][ INFO][PID:2369213] Added pubkey for user rhcontainerbot project podman-next into: /var/lib/copr/public_html/results/rhcontainerbot/podman-next/pubkey.gpg
[2025-03-03 12:18:44,532][ INFO][PID:2369213] Finished build: id=8719265 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/rhcontainerbot/podman-next chroot=fedora-41-ppc64le
[2025-03-03 12:18:44,534][ INFO][PID:2369213] Worker succeeded build, took 955.752409696579
[2025-03-03 12:18:44,535][ INFO][PID:2369213] Sending build state back to frontend: {
"builds": [
{
"timeout": 18000,
"frontend_base_url": "https://copr.fedorainfracloud.org",
"memory_reqs": 2048,
"enable_net": true,
"project_owner": "rhcontainerbot",
"project_name": "podman-next",
"project_dirname": "podman-next",
"submitter": "packit",
"ended_on": 1741004324.5343473,
"started_on": 1741003368.7819376,
"submitted_on": null,
"status": 1,
"chroot": "fedora-41-ppc64le",
"arch": "ppc64le",
"buildroot_pkgs": [],
"task_id": "8719265-fedora-41-ppc64le",
"build_id": 8719265,
"package_name": "netavark",
"package_version": "102:1.14.0-1.20250303115200200580.main.24.g1e21ecf",
"git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rhcontainerbot/podman-next/netavark",
"git_hash": "ad35ab2338c675e33180890a3259a0080cb2547d",
"git_branch": null,
"source_type": null,
"source_json": null,
"pkg_name": null,
"pkg_main_version": null,
"pkg_epoch": null,
"pkg_release": null,
"srpm_url": null,
"uses_devel_repo": false,
"sandbox": "rhcontainerbot/podman-next--packit",
"results": {
"packages": [
{
"name": "netavark",
"epoch": 102,
"version": "1.14.0",
"release": "1.20250303115200200580.main.24.g1e21ecf.fc41",
"arch": "ppc64le"
},
{
"name": "netavark-debuginfo",
"epoch": 102,
"version": "1.14.0",
"release": "1.20250303115200200580.main.24.g1e21ecf.fc41",
"arch": "ppc64le"
},
{
"name": "netavark",
"epoch": 102,
"version": "1.14.0",
"release": "1.20250303115200200580.main.24.g1e21ecf.fc41",
"arch": "src"
},
{
"name": "netavark-debugsource",
"epoch": 102,
"version": "1.14.0",
"release": "1.20250303115200200580.main.24.g1e21ecf.fc41",
"arch": "ppc64le"
}
]
},
"appstream": false,
"allow_user_ssh": false,
"ssh_public_keys": null,
"storage": null,
"background": false,
"fedora_review": false,
"isolation": "default",
"repo_priority": null,
"repos": [
{
"baseurl": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/podman-next/fedora-41-ppc64le/",
"id": "copr_base",
"name": "Copr repository",
"priority": null
}
],
"tags": [
"arch_ppc64le"
],
"with_opts": [],
"without_opts": [],
"destdir": "/var/lib/copr/public_html/results/rhcontainerbot/podman-next",
"results_repo_url": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/podman-next",
"result_dir": "08719265-netavark",
"built_packages": "netavark 1.14.0\nnetavark-debuginfo 1.14.0\nnetavark-debugsource 1.14.0",
"id": 8719265,
"mockchain_macros": {
"copr_username": "rhcontainerbot",
"copr_projectname": "podman-next",
"vendor": "Fedora Project COPR (rhcontainerbot/podman-next)"
}
}
]
}
[2025-03-03 12:18:44,594][ INFO][PID:2369213] Sending fedora-messaging bus message in build.end
[2025-03-03 12:18:44,613][ INFO][PID:2369213] Compressing /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le/08719265-netavark/builder-live.log by gzip
[2025-03-03 12:18:44,614][ INFO][PID:2369213] Running command 'gzip /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le/08719265-netavark/builder-live.log' as PID 2680246
[2025-03-03 12:18:44,620][ INFO][PID:2369213] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le/08719265-netavark/builder-live.log)
[2025-03-03 12:18:44,620][ INFO][PID:2369213] Compressing /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le/08719265-netavark/backend.log by gzip
[2025-03-03 12:18:44,621][ INFO][PID:2369213] Running command 'gzip /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-41-ppc64le/08719265-netavark/backend.log' as PID 2680247