[2024-12-20 12:39:55,309][ INFO][PID:2720098] Marking build as starting [2024-12-20 12:39:55,373][ INFO][PID:2720098] Trying to allocate VM [2024-12-20 12:43:13,843][ INFO][PID:2720098] Allocated host ResallocHost, ticket_id=305117, hostname=98.84.167.39 [2024-12-20 12:43:13,849][ INFO][PID:2720098] Allocating ssh connection to builder [2024-12-20 12:43:13,853][ INFO][PID:2720098] Checking that builder machine is OK [2024-12-20 12:43:14,296][ INFO][PID:2720098] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.84.167.39 '/usr/bin/test -f /etc/mock/fedora-40-x86_64.cfg' [2024-12-20 12:43:14,514][ INFO][PID:2720098] Filling build.info file with builder info [2024-12-20 12:43:14,517][ INFO][PID:2720098] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "medzik", "project_name": "jetbrains", "project_dirname": "jetbrains", "submitter": null, "ended_on": null, "started_on": 1734698594.5167468, "submitted_on": null, "status": 3, "chroot": "fedora-40-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "8427226-fedora-40-x86_64", "build_id": 8427226, "package_name": "dataspell", "package_version": "2024.3.1.1-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/medzik/jetbrains/dataspell", "git_hash": "54c2c4ca29bea607dddb5741217ec2ef415f3eb7", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "medzik/jetbrains--6de331cc-bac4-495e-99d7-b8fb4c5e1257", "results": null, "appstream": true, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/medzik/jetbrains/fedora-40-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/medzik/jetbrains", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/medzik/jetbrains", "result_dir": "08427226-dataspell", "built_packages": "", "id": 8427226, "mockchain_macros": { "copr_username": "medzik", "copr_projectname": "jetbrains", "vendor": "Fedora Project COPR (medzik/jetbrains)" } } ] } [2024-12-20 12:43:14,618][ INFO][PID:2720098] Sending fedora-messaging bus message in build.start [2024-12-20 12:43:18,407][ INFO][PID:2720098] Sending fedora-messaging bus message in chroot.start [2024-12-20 12:43:18,511][ INFO][PID:2720098] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8427226-fedora-40-x86_64 --chroot fedora-40-x86_64 --detached [2024-12-20 12:43:18,988][ INFO][PID:2720098] Downloading the builder-live.log file, attempt 1 [2024-12-20 12:43:19,017][ INFO][PID:2720098] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.84.167.39 copr-rpmbuild-log [2024-12-20 12:43:24,042][ INFO][PID:2720098] Periodic builder liveness probe: alive [2024-12-20 12:49:33,496][ ERROR][PID:2720098] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-12-20 12:49:43,499][WARNING][PID:2720098] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-12-20 12:49:53,500][ INFO][PID:2720098] Downloading the builder-live.log file, attempt 2 [2024-12-20 12:49:53,501][ INFO][PID:2720098] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.84.167.39 copr-rpmbuild-log [2024-12-20 12:49:53,511][ ERROR][PID:2720098] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-12-20 12:50:03,513][WARNING][PID:2720098] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-12-20 12:50:13,514][ INFO][PID:2720098] Downloading the builder-live.log file, attempt 3 [2024-12-20 12:50:13,517][ INFO][PID:2720098] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.84.167.39 copr-rpmbuild-log [2024-12-20 12:50:13,529][ ERROR][PID:2720098] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-12-20 12:50:23,534][WARNING][PID:2720098] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-12-20 12:50:33,536][ INFO][PID:2720098] Downloading the builder-live.log file, attempt 4 [2024-12-20 12:50:33,540][ INFO][PID:2720098] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.84.167.39 copr-rpmbuild-log [2024-12-20 12:51:33,611][ ERROR][PID:2720098] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-12-20 12:51:43,617][ ERROR][PID:2720098] Giving up for unstable SSH, failures: 1734698983.4990308, 1734699003.51355, 1734699023.5349169, 1734699103.6167433 [2024-12-20 12:51:43,618][ ERROR][PID:2720098] Re-try request for task on 'ResallocHost, ticket_id=305117, hostname=98.84.167.39': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2024-12-20 12:51:43,620][ INFO][PID:2720098] Releasing VM back to pool [2024-12-20 12:51:43,669][ INFO][PID:2720098] Retry #1 (on other host) [2024-12-20 12:51:43,670][ INFO][PID:2720098] Trying to allocate VM [2024-12-20 12:51:46,753][ INFO][PID:2720098] Allocated host ResallocHost, ticket_id=305278, hostname=2620:52:3:1:dead:beef:cafe:c1d7 [2024-12-20 12:51:46,754][ INFO][PID:2720098] Allocating ssh connection to builder [2024-12-20 12:51:46,755][ INFO][PID:2720098] Checking that builder machine is OK [2024-12-20 12:51:47,633][ INFO][PID:2720098] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c1d7 '/usr/bin/test -f /etc/mock/fedora-40-x86_64.cfg' [2024-12-20 12:51:47,954][ INFO][PID:2720098] Filling build.info file with builder info [2024-12-20 12:51:47,961][ INFO][PID:2720098] Not re-notifying FE and msg buses for the new host. [2024-12-20 12:51:47,962][ INFO][PID:2720098] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8427226-fedora-40-x86_64 --chroot fedora-40-x86_64 --detached [2024-12-20 12:51:48,583][ INFO][PID:2720098] Downloading the builder-live.log file, attempt 1 [2024-12-20 12:51:48,586][ INFO][PID:2720098] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c1d7 copr-rpmbuild-log [2024-12-20 12:53:28,653][ INFO][PID:2720098] Periodic builder liveness probe: alive [2024-12-20 13:03:29,089][ INFO][PID:2720098] Periodic builder liveness probe: alive [2024-12-20 13:09:09,516][ INFO][PID:2720098] Downloading results from builder [2024-12-20 13:09:09,516][ INFO][PID:2720098] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c1d7]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08427226-dataspell started [2024-12-20 13:09:09,521][ INFO][PID:2720098] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c1d7]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08427226-dataspell/ &> /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08427226-dataspell/build-08427226.rsync.log [2024-12-20 13:09:45,069][ INFO][PID:2720098] rsyncing finished. [2024-12-20 13:09:45,071][ INFO][PID:2720098] Releasing VM back to pool [2024-12-20 13:09:45,122][ INFO][PID:2720098] Searching for 'success' file in resultdir [2024-12-20 13:09:45,162][ INFO][PID:2720098] Going to sign pkgs from source: 8427226-fedora-40-x86_64 in chroot: /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64 [2024-12-20 13:09:45,165][ INFO][PID:2720098] Calling '/bin/sign -u medzik#jetbrains@copr.fedorahosted.org -p' (attempt #1) [2024-12-20 13:09:45,367][ INFO][PID:2720098] Calling '/bin/sign -4 -h sha256 -u medzik#jetbrains@copr.fedorahosted.org -r /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08427226-dataspell/dataspell-jbr-2024.3.1.1-1.fc40.x86_64.rpm' (attempt #1) [2024-12-20 13:10:05,314][ INFO][PID:2720098] signed rpm: /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08427226-dataspell/dataspell-jbr-2024.3.1.1-1.fc40.x86_64.rpm [2024-12-20 13:10:05,318][ INFO][PID:2720098] Calling '/bin/sign -4 -h sha256 -u medzik#jetbrains@copr.fedorahosted.org -r /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08427226-dataspell/dataspell-2024.3.1.1-1.fc40.src.rpm' (attempt #1) [2024-12-20 13:10:05,943][ INFO][PID:2720098] signed rpm: /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08427226-dataspell/dataspell-2024.3.1.1-1.fc40.src.rpm [2024-12-20 13:10:05,945][ INFO][PID:2720098] Calling '/bin/sign -4 -h sha256 -u medzik#jetbrains@copr.fedorahosted.org -r /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08427226-dataspell/dataspell-2024.3.1.1-1.fc40.x86_64.rpm' (attempt #1) [2024-12-20 13:11:19,738][ INFO][PID:2720098] signed rpm: /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08427226-dataspell/dataspell-2024.3.1.1-1.fc40.x86_64.rpm [2024-12-20 13:11:19,753][ INFO][PID:2720098] Sign done [2024-12-20 13:11:19,755][ INFO][PID:2720098] Incremental createrepo run, adding 08427226-dataspell into https://download.copr.fedorainfracloud.org/results/medzik/jetbrains/fedora-40-x86_64, (auto-create-repo=True) [2024-12-20 13:11:19,805][ INFO][PID:2720098] Running command 'copr-repo --batched /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64 --add 08427226-dataspell' as PID 2956667 [2024-12-20 13:15:19,369][ INFO][PID:2720098] Finished after 239 seconds with exit code 1 (copr-repo --batched /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64 --add 08427226-dataspell) stdout: stderr: Sub-command failed Traceback (most recent call last): File "/usr/bin/copr-repo", line 445, in main main_try_lock(opts, batch) ~~~~~~~~~~~~~^^^^^^^^^^^^^ File "/usr/bin/copr-repo", line 397, in main_try_lock main_locked(opts, batch, opts.log) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/bin/copr-repo", line 359, in main_locked add_appdata(opts) ~~~~~~~~~~~^^^^^^ File "/usr/bin/copr-repo", line 242, in add_appdata run_cmd([ ~~~~~~~^^ "/usr/bin/timeout", "--kill-after=240", "180", ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<10 lines>... "--origin=" + origin], ^^^^^^^^^^^^^^^^^^^^^^ check=True, logger=opts.log) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.13/site-packages/copr_backend/helpers.py", line 148, in run_cmd raise CommandException(exc_msg) copr_backend.helpers.CommandException: Command '/usr/bin/timeout --kill-after=240 180 /usr/bin/appstream-builder --temp-dir=/var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/tmp --cache-dir=/var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/cache --packages-dir=/var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64 --output-dir=/var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/appdata --basename=appstream --include-failed --min-icon-size=48 --veto-ignore=missing-parents --enable-hidpi --origin=medzik/jetbrains' failed, exit code '124 [2024-12-20 13:15:19,378][ ERROR][PID:2720098] Createrepo failed [2024-12-20 13:15:19,387][ ERROR][PID:2720098] Build failed: Backend process error: createrepo failed [2024-12-20 13:15:19,392][ INFO][PID:2720098] Finished build: id=8427226 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/medzik/jetbrains chroot=fedora-40-x86_64 [2024-12-20 13:15:19,401][ ERROR][PID:2720098] Backend process error: createrepo failed [2024-12-20 13:15:19,405][ INFO][PID:2720098] Worker failed build, took 1411.443963766098 [2024-12-20 13:15:19,408][ INFO][PID:2720098] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "medzik", "project_name": "jetbrains", "project_dirname": "jetbrains", "submitter": null, "ended_on": 1734700519.4057236, "started_on": 1734699107.9617598, "submitted_on": null, "status": 0, "chroot": "fedora-40-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "8427226-fedora-40-x86_64", "build_id": 8427226, "package_name": "dataspell", "package_version": "2024.3.1.1-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/medzik/jetbrains/dataspell", "git_hash": "54c2c4ca29bea607dddb5741217ec2ef415f3eb7", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "medzik/jetbrains--6de331cc-bac4-495e-99d7-b8fb4c5e1257", "results": null, "appstream": true, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/medzik/jetbrains/fedora-40-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/medzik/jetbrains", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/medzik/jetbrains", "result_dir": "08427226-dataspell", "built_packages": "", "id": 8427226, "mockchain_macros": { "copr_username": "medzik", "copr_projectname": "jetbrains", "vendor": "Fedora Project COPR (medzik/jetbrains)" } } ] } [2024-12-20 13:15:19,476][ INFO][PID:2720098] Sending fedora-messaging bus message in build.end [2024-12-20 13:15:19,606][ INFO][PID:2720098] Compressing /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08427226-dataspell/builder-live.log by gzip [2024-12-20 13:15:19,613][ INFO][PID:2720098] Running command 'gzip /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08427226-dataspell/builder-live.log' as PID 2989931 [2024-12-20 13:15:19,679][ INFO][PID:2720098] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08427226-dataspell/builder-live.log) [2024-12-20 13:15:19,680][ INFO][PID:2720098] Compressing /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08427226-dataspell/backend.log by gzip [2024-12-20 13:15:19,686][ INFO][PID:2720098] Running command 'gzip /var/lib/copr/public_html/results/medzik/jetbrains/fedora-40-x86_64/08427226-dataspell/backend.log' as PID 2989945