[2025-01-28 12:38:36,308][ INFO][PID:1168398] Marking build as starting
[2025-01-28 12:38:36,336][ INFO][PID:1168398] Trying to allocate VM
[2025-01-28 12:38:39,374][ INFO][PID:1168398] Allocated host ResallocHost, ticket_id=964955, hostname=54.162.217.157, name=aws_x86_64_spot_prod_00869390_20250128_122835
[2025-01-28 12:38:39,375][ INFO][PID:1168398] Allocating ssh connection to builder
[2025-01-28 12:38:39,375][ INFO][PID:1168398] Checking that builder machine is OK
[2025-01-28 12:38:39,699][ INFO][PID:1168398] Running remote command: copr-builder-ready fedora-40-x86_64
[2025-01-28 12:38:39,915][ INFO][PID:1168398] Red Hat subscription not needed for fedora-40-x86_64
Builder is ready to be used
[2025-01-28 12:38:39,917][ INFO][PID:1168398] Filling build.info file with builder info
[2025-01-28 12:38:39,919][ INFO][PID:1168398] Sending build state back to frontend: {
"builds": [
{
"timeout": 18000,
"frontend_base_url": "https://copr.fedorainfracloud.org",
"memory_reqs": 2048,
"enable_net": true,
"project_owner": "medzik",
"project_name": "jetbrains",
"project_dirname": "jetbrains",
"submitter": null,
"ended_on": null,
"started_on": 1738067919.9193807,
"submitted_on": null,
"status": 3,
"chroot": "fedora-40-x86_64",
"arch": "x86_64",
"buildroot_pkgs": [],
"task_id": "8581361-fedora-40-x86_64",
"build_id": 8581361,
"package_name": "pycharm-community",
"package_version": "2024.3.2-1",
"git_repo": "https://copr-dist-git.fedorainfracloud.org/git/medzik/jetbrains/pycharm-community",
"git_hash": "c9e133dc058fc92286ca70446e20cf58614c205d",
"git_branch": null,
"source_type": null,
"source_json": null,
"pkg_name": null,
"pkg_main_version": null,
"pkg_epoch": null,
"pkg_release": null,
"srpm_url": null,
"uses_devel_repo": false,
"sandbox": "medzik/jetbrains--93dff19b-7d05-4b84-87c0-f95f4b5dd029",
"results": null,
"appstream": true,
"allow_user_ssh": false,
"ssh_public_keys": null,
"storage": null,
"background": true,
"fedora_review": false,
"isolation": "default",
"repo_priority": null,
"repos": [
{
"baseurl": "https://download.copr.fedorainfracloud.org/results/medzik/jetbrains/fedora-40-x86_64/",
"id": "copr_base",
"name": "Copr repository",
"priority": null
}
],
"tags": [
"arch_x86_64"
],
"with_opts": [],
"without_opts": [],
"destdir": "/var/lib/copr/public_html/results/medzik/jetbrains",
"results_repo_url": "https://download.copr.fedorainfracloud.org/results/medzik/jetbrains",
"result_dir": "08581361-pycharm-community",
"built_packages": "",
"id": 8581361,
"mockchain_macros": {
"copr_username": "medzik",
"copr_projectname": "jetbrains",
"vendor": "Fedora Project COPR (medzik/jetbrains)"
}
}
]
}
[2025-01-28 12:38:39,951][ INFO][PID:1168398] Sending fedora-messaging bus message in build.start
[2025-01-28 12:38:40,791][ INFO][PID:1168398] Sending fedora-messaging bus message in chroot.start
[2025-01-28 12:38:40,815][ INFO][PID:1168398] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8581361-fedora-40-x86_64 --chroot fedora-40-x86_64 --detached
[2025-01-28 12:38:41,171][ INFO][PID:1168398] Downloading the builder-live.log file, attempt 1
[2025-01-28 12:38:41,173][ INFO][PID:1168398] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.162.217.157 copr-rpmbuild-log
[2025-01-28 12:38:46,192][ INFO][PID:1168398] Periodic builder liveness probe: alive
[2025-01-28 12:48:46,302][ INFO][PID:1168398] Periodic builder liveness probe: alive
[2025-01-28 12:58:46,425][ INFO][PID:1168398] Periodic builder liveness probe: alive
[2025-01-28 13:04:18,450][ ERROR][PID:1168398] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-01-28 13:04:28,451][WARNING][PID:1168398] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-01-28 13:04:38,451][ INFO][PID:1168398] Downloading the builder-live.log file, attempt 2
[2025-01-28 13:04:38,453][ INFO][PID:1168398] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.162.217.157 copr-rpmbuild-log
[2025-01-28 13:05:38,521][ ERROR][PID:1168398] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-01-28 13:05:48,524][WARNING][PID:1168398] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-01-28 13:05:58,525][ INFO][PID:1168398] Downloading the builder-live.log file, attempt 3
[2025-01-28 13:05:58,526][ INFO][PID:1168398] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.162.217.157 copr-rpmbuild-log
[2025-01-28 13:06:58,594][ ERROR][PID:1168398] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-01-28 13:07:08,598][WARNING][PID:1168398] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-01-28 13:07:18,599][ INFO][PID:1168398] Downloading the builder-live.log file, attempt 4
[2025-01-28 13:07:18,601][ INFO][PID:1168398] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.162.217.157 copr-rpmbuild-log
[2025-01-28 13:08:18,670][ ERROR][PID:1168398] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-01-28 13:08:28,672][ ERROR][PID:1168398] Giving up for unstable SSH, failures: 1738069468.4511733, 1738069548.5249224, 1738069628.5983994, 1738069708.6721795
[2025-01-28 13:08:28,673][ ERROR][PID:1168398] Re-try request for task on 'ResallocHost, ticket_id=964955, hostname=54.162.217.157, name=aws_x86_64_spot_prod_00869390_20250128_122835': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-01-28 13:08:28,673][ INFO][PID:1168398] Releasing VM back to pool
[2025-01-28 13:08:28,683][ INFO][PID:1168398] Retry #1 (on other host)
[2025-01-28 13:08:28,684][ INFO][PID:1168398] Trying to allocate VM
[2025-01-28 13:11:46,786][ INFO][PID:1168398] Allocated host ResallocHost, ticket_id=965396, hostname=44.205.19.118, name=aws_x86_64_spot_prod_00870748_20250128_130940
[2025-01-28 13:11:46,787][ INFO][PID:1168398] Allocating ssh connection to builder
[2025-01-28 13:11:46,787][ INFO][PID:1168398] Checking that builder machine is OK
[2025-01-28 13:11:47,128][ INFO][PID:1168398] Running remote command: copr-builder-ready fedora-40-x86_64
[2025-01-28 13:11:47,337][ INFO][PID:1168398] Red Hat subscription not needed for fedora-40-x86_64
Builder is ready to be used
[2025-01-28 13:11:47,338][ INFO][PID:1168398] Filling build.info file with builder info
[2025-01-28 13:11:47,343][ INFO][PID:1168398] Not re-notifying FE and msg buses for the new host.
[2025-01-28 13:11:47,344][ INFO][PID:1168398] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8581361-fedora-40-x86_64 --chroot fedora-40-x86_64 --detached
[2025-01-28 13:11:47,706][ INFO][PID:1168398] Downloading the builder-live.log file, attempt 1
[2025-01-28 13:11:47,709][ INFO][PID:1168398] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.205.19.118 copr-rpmbuild-log
[2025-01-28 13:11:52,715][ INFO][PID:1168398] Periodic builder liveness probe: alive
[2025-01-28 13:21:52,796][ INFO][PID:1168398] Periodic builder liveness probe: alive
[2025-01-28 13:31:52,943][ INFO][PID:1168398] Periodic builder liveness probe: alive
[2025-01-28 13:34:22,297][ ERROR][PID:1168398] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-01-28 13:34:32,298][WARNING][PID:1168398] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-01-28 13:34:42,299][ INFO][PID:1168398] Downloading the builder-live.log file, attempt 2
[2025-01-28 13:34:42,301][ INFO][PID:1168398] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.205.19.118 copr-rpmbuild-log
[2025-01-28 13:34:42,311][ ERROR][PID:1168398] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-01-28 13:34:52,318][WARNING][PID:1168398] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-01-28 13:35:02,319][ INFO][PID:1168398] Downloading the builder-live.log file, attempt 3
[2025-01-28 13:35:02,321][ INFO][PID:1168398] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.205.19.118 copr-rpmbuild-log
[2025-01-28 13:35:02,330][ ERROR][PID:1168398] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-01-28 13:35:12,336][WARNING][PID:1168398] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-01-28 13:35:22,337][ INFO][PID:1168398] Downloading the builder-live.log file, attempt 4
[2025-01-28 13:35:22,338][ INFO][PID:1168398] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.205.19.118 copr-rpmbuild-log
[2025-01-28 13:36:22,407][ ERROR][PID:1168398] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-01-28 13:36:32,411][ ERROR][PID:1168398] Giving up for unstable SSH, failures: 1738071272.298103, 1738071292.3188865, 1738071312.3365023, 1738071392.411836
[2025-01-28 13:36:32,412][ ERROR][PID:1168398] Re-try request for task on 'ResallocHost, ticket_id=965396, hostname=44.205.19.118, name=aws_x86_64_spot_prod_00870748_20250128_130940': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-01-28 13:36:32,413][ INFO][PID:1168398] Releasing VM back to pool
[2025-01-28 13:36:32,426][ INFO][PID:1168398] Retry #2 (on other host)
[2025-01-28 13:36:32,427][ INFO][PID:1168398] Trying to allocate VM
[2025-01-28 13:36:35,482][ INFO][PID:1168398] Allocated host ResallocHost, ticket_id=965846, hostname=2620:52:3:1:dead:beef:cafe:c157, name=copr_hv_x86_64_02_prod_00871204_20250128_133246
[2025-01-28 13:36:35,483][ INFO][PID:1168398] Allocating ssh connection to builder
[2025-01-28 13:36:35,483][ INFO][PID:1168398] Checking that builder machine is OK
[2025-01-28 13:36:36,131][ INFO][PID:1168398] Running remote command: copr-builder-ready fedora-40-x86_64
[2025-01-28 13:36:36,566][ INFO][PID:1168398] Red Hat subscription not needed for fedora-40-x86_64
Builder is ready to be used
[2025-01-28 13:36:36,567][ INFO][PID:1168398] Filling build.info file with builder info
[2025-01-28 13:36:36,571][ INFO][PID:1168398] Not re-notifying FE and msg buses for the new host.
[2025-01-28 13:36:36,573][ INFO][PID:1168398] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8581361-fedora-40-x86_64 --chroot fedora-40-x86_64 --detached
[2025-01-28 13:36:37,447][ INFO][PID:1168398] Downloading the builder-live.log file, attempt 1
[2025-01-28 13:36:37,448][ INFO][PID:1168398] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c157 copr-rpmbuild-log
[2025-01-28 13:41:57,509][ INFO][PID:1168398] Periodic builder liveness probe: alive
[2025-01-28 13:51:57,584][ INFO][PID:1168398] Periodic builder liveness probe: alive
[2025-01-28 14:01:57,662][ INFO][PID:1168398] Periodic builder liveness probe: alive
[2025-01-28 14:11:57,779][ INFO][PID:1168398] Periodic builder liveness probe: alive
[2025-01-28 14:21:57,889][ INFO][PID:1168398] Periodic builder liveness probe: alive
[2025-01-28 14:31:58,005][ INFO][PID:1168398] Periodic builder liveness probe: alive
[2025-01-28 14:35:11,146][ INFO][PID:1168398] Downloading results from builder
[2025-01-28 14:35:11,147][ INFO][PID:1168398] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c157]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08581361-pycharm-community started
[2025-01-28 14:35:11,148][ INFO][PID:1168398] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c157]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08581361-pycharm-community/ &> /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08581361-pycharm-community/build-08581361.rsync.log
[2025-01-28 14:35:18,366][ INFO][PID:1168398] rsyncing finished.
[2025-01-28 14:35:18,366][ INFO][PID:1168398] Releasing VM back to pool
[2025-01-28 14:35:18,380][ INFO][PID:1168398] Searching for 'success' file in resultdir
[2025-01-28 14:35:18,380][ INFO][PID:1168398] Going to sign pkgs from source: 8581361-fedora-40-x86_64 in chroot: /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64
[2025-01-28 14:35:18,381][ INFO][PID:1168398] Calling '/bin/sign -u medzik#jetbrains@copr.fedorahosted.org -p' (attempt #1)
[2025-01-28 14:35:18,677][ INFO][PID:1168398] Calling '/bin/sign -4 -h sha256 -u medzik#jetbrains@copr.fedorahosted.org -r /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08581361-pycharm-community/pycharm-community-jbr-2024.3.2-1.fc40.x86_64.rpm' (attempt #1)
[2025-01-28 14:35:22,581][ INFO][PID:1168398] signed rpm: /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08581361-pycharm-community/pycharm-community-jbr-2024.3.2-1.fc40.x86_64.rpm
[2025-01-28 14:35:22,582][ INFO][PID:1168398] Calling '/bin/sign -4 -h sha256 -u medzik#jetbrains@copr.fedorahosted.org -r /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08581361-pycharm-community/pycharm-community-2024.3.2-1.fc40.src.rpm' (attempt #1)
[2025-01-28 14:35:23,199][ INFO][PID:1168398] signed rpm: /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08581361-pycharm-community/pycharm-community-2024.3.2-1.fc40.src.rpm
[2025-01-28 14:35:23,200][ INFO][PID:1168398] Calling '/bin/sign -4 -h sha256 -u medzik#jetbrains@copr.fedorahosted.org -r /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08581361-pycharm-community/pycharm-community-2024.3.2-1.fc40.x86_64.rpm' (attempt #1)
[2025-01-28 14:35:28,176][ INFO][PID:1168398] signed rpm: /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08581361-pycharm-community/pycharm-community-2024.3.2-1.fc40.x86_64.rpm
[2025-01-28 14:35:28,177][ INFO][PID:1168398] Sign done
[2025-01-28 14:35:28,177][ INFO][PID:1168398] Incremental createrepo run, adding 08581361-pycharm-community into https://download.copr.fedorainfracloud.org/results/medzik/jetbrains/fedora-40-x86_64, (auto-create-repo=True)
[2025-01-28 14:35:28,179][ INFO][PID:1168398] Running command 'copr-repo --batched /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64 --add 08581361-pycharm-community' as PID 1968102
[2025-01-28 14:36:18,153][ INFO][PID:1168398] Finished after 49 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64 --add 08581361-pycharm-community)
[2025-01-28 14:36:18,154][ INFO][PID:1168398] Getting build details
[2025-01-28 14:36:18,155][ INFO][PID:1168398] Listing built binary packages in /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08581361-pycharm-community
[2025-01-28 14:36:18,155][ INFO][PID:1168398] Built packages:
['pycharm-community 2024.3.2', 'pycharm-community-jbr 2024.3.2']
[2025-01-28 14:36:18,156][ INFO][PID:1168398] build details: {'built_packages': 'pycharm-community 2024.3.2\npycharm-community-jbr 2024.3.2'}
[2025-01-28 14:36:18,157][ INFO][PID:1168398] Retrieving pubkey
[2025-01-28 14:36:18,157][ INFO][PID:1168398] Calling '/bin/sign -u medzik#jetbrains@copr.fedorahosted.org -p' (attempt #1)
[2025-01-28 14:36:18,462][ INFO][PID:1168398] Added pubkey for user medzik project jetbrains into: /var/lib/copr/public_html/results/medzik/jetbrains/pubkey.gpg
[2025-01-28 14:36:18,463][ INFO][PID:1168398] Finished build: id=8581361 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/medzik/jetbrains chroot=fedora-40-x86_64
[2025-01-28 14:36:18,465][ INFO][PID:1168398] Worker succeeded build, took 3581.8946664333344
[2025-01-28 14:36:18,466][ INFO][PID:1168398] Sending build state back to frontend: {
"builds": [
{
"timeout": 18000,
"frontend_base_url": "https://copr.fedorainfracloud.org",
"memory_reqs": 2048,
"enable_net": true,
"project_owner": "medzik",
"project_name": "jetbrains",
"project_dirname": "jetbrains",
"submitter": null,
"ended_on": 1738074978.4657981,
"started_on": 1738071396.5711317,
"submitted_on": null,
"status": 1,
"chroot": "fedora-40-x86_64",
"arch": "x86_64",
"buildroot_pkgs": [],
"task_id": "8581361-fedora-40-x86_64",
"build_id": 8581361,
"package_name": "pycharm-community",
"package_version": "2024.3.2-1",
"git_repo": "https://copr-dist-git.fedorainfracloud.org/git/medzik/jetbrains/pycharm-community",
"git_hash": "c9e133dc058fc92286ca70446e20cf58614c205d",
"git_branch": null,
"source_type": null,
"source_json": null,
"pkg_name": null,
"pkg_main_version": null,
"pkg_epoch": null,
"pkg_release": null,
"srpm_url": null,
"uses_devel_repo": false,
"sandbox": "medzik/jetbrains--93dff19b-7d05-4b84-87c0-f95f4b5dd029",
"results": {
"packages": [
{
"name": "pycharm-community",
"epoch": null,
"version": "2024.3.2",
"release": "1.fc40",
"arch": "src"
},
{
"name": "pycharm-community",
"epoch": null,
"version": "2024.3.2",
"release": "1.fc40",
"arch": "x86_64"
},
{
"name": "pycharm-community-jbr",
"epoch": null,
"version": "2024.3.2",
"release": "1.fc40",
"arch": "x86_64"
}
]
},
"appstream": true,
"allow_user_ssh": false,
"ssh_public_keys": null,
"storage": null,
"background": true,
"fedora_review": false,
"isolation": "default",
"repo_priority": null,
"repos": [
{
"baseurl": "https://download.copr.fedorainfracloud.org/results/medzik/jetbrains/fedora-40-x86_64/",
"id": "copr_base",
"name": "Copr repository",
"priority": null
}
],
"tags": [
"arch_x86_64"
],
"with_opts": [],
"without_opts": [],
"destdir": "/var/lib/copr/public_html/results/medzik/jetbrains",
"results_repo_url": "https://download.copr.fedorainfracloud.org/results/medzik/jetbrains",
"result_dir": "08581361-pycharm-community",
"built_packages": "pycharm-community 2024.3.2\npycharm-community-jbr 2024.3.2",
"id": 8581361,
"mockchain_macros": {
"copr_username": "medzik",
"copr_projectname": "jetbrains",
"vendor": "Fedora Project COPR (medzik/jetbrains)"
}
}
]
}
[2025-01-28 14:36:18,498][ INFO][PID:1168398] Sending fedora-messaging bus message in build.end
[2025-01-28 14:36:18,519][ INFO][PID:1168398] Compressing /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08581361-pycharm-community/builder-live.log by gzip
[2025-01-28 14:36:18,520][ INFO][PID:1168398] Running command 'gzip /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08581361-pycharm-community/builder-live.log' as PID 1973270
[2025-01-28 14:36:18,611][ INFO][PID:1168398] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08581361-pycharm-community/builder-live.log)
[2025-01-28 14:36:18,612][ INFO][PID:1168398] Compressing /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08581361-pycharm-community/backend.log by gzip
[2025-01-28 14:36:18,613][ INFO][PID:1168398] Running command 'gzip /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08581361-pycharm-community/backend.log' as PID 1973277