[2024-11-25 17:08:44,997][  INFO][PID:3199275] Marking build as starting
[2024-11-25 17:08:45,253][  INFO][PID:3199275] Trying to allocate VM
[2024-11-25 17:09:13,497][  INFO][PID:3199275] Allocated host ResallocHost, ticket_id=6064425, hostname=52.203.150.6
[2024-11-25 17:09:13,499][  INFO][PID:3199275] Allocating ssh connection to builder
[2024-11-25 17:09:13,500][  INFO][PID:3199275] Checking that builder machine is OK
[2024-11-25 17:09:13,860][  INFO][PID:3199275] Running remote command: copr-builder-ready fedora-40-x86_64
[2024-11-25 17:09:14,074][  INFO][PID:3199275] Red Hat subscription not needed for fedora-40-x86_64
Builder is ready to be used

[2024-11-25 17:09:14,078][  INFO][PID:3199275] Filling build.info file with builder info
[2024-11-25 17:09:14,079][  INFO][PID:3199275] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "alternateved",
            "project_name": "bleeding-emacs",
            "project_dirname": "bleeding-emacs",
            "submitter": "gitlab.com:alternateved",
            "ended_on": null,
            "started_on": 1732554554.0791135,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-40-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "8312258-fedora-40-x86_64",
            "build_id": 8312258,
            "package_name": "bleeding-emacs",
            "package_version": "1:30.0.92-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/alternateved/bleeding-emacs/bleeding-emacs",
            "git_hash": "ec1410148dcd9ec14fa801ea6c1f2b1eab3bb0e6",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "alternateved/bleeding-emacs--gitlab.com:alternateved",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/alternateved/bleeding-emacs/fedora-40-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/alternateved/bleeding-emacs",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/alternateved/bleeding-emacs",
            "result_dir": "08312258-bleeding-emacs",
            "built_packages": "",
            "id": 8312258,
            "mockchain_macros": {
                "copr_username": "alternateved",
                "copr_projectname": "bleeding-emacs",
                "vendor": "Fedora Project COPR (alternateved/bleeding-emacs)"
            }
        }
    ]
}
[2024-11-25 17:09:14,178][  INFO][PID:3199275] Sending fedora-messaging bus message in build.start
[2024-11-25 17:09:15,034][  INFO][PID:3199275] Sending fedora-messaging bus message in chroot.start
[2024-11-25 17:09:15,094][  INFO][PID:3199275] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8312258-fedora-40-x86_64 --chroot fedora-40-x86_64 --detached
[2024-11-25 17:09:15,459][  INFO][PID:3199275] Downloading the builder-live.log file, attempt 1
[2024-11-25 17:09:15,461][  INFO][PID:3199275] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.203.150.6 copr-rpmbuild-log
[2024-11-25 17:09:20,470][  INFO][PID:3199275] Periodic builder liveness probe: alive
[2024-11-25 17:19:20,757][  INFO][PID:3199275] Periodic builder liveness probe: alive
[2024-11-25 17:20:23,656][ ERROR][PID:3199275] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-11-25 17:20:33,662][WARNING][PID:3199275] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2024-11-25 17:20:43,663][  INFO][PID:3199275] Downloading the builder-live.log file, attempt 2
[2024-11-25 17:20:43,667][  INFO][PID:3199275] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.203.150.6 copr-rpmbuild-log
[2024-11-25 17:21:43,736][ ERROR][PID:3199275] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-11-25 17:21:53,816][WARNING][PID:3199275] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2024-11-25 17:22:03,817][  INFO][PID:3199275] Downloading the builder-live.log file, attempt 3
[2024-11-25 17:22:03,819][  INFO][PID:3199275] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.203.150.6 copr-rpmbuild-log
[2024-11-25 17:23:03,887][ ERROR][PID:3199275] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-11-25 17:23:13,927][WARNING][PID:3199275] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2024-11-25 17:23:23,928][  INFO][PID:3199275] Downloading the builder-live.log file, attempt 4
[2024-11-25 17:23:23,935][  INFO][PID:3199275] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.203.150.6 copr-rpmbuild-log
[2024-11-25 17:24:24,028][ ERROR][PID:3199275] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-11-25 17:24:34,049][ ERROR][PID:3199275] Giving up for unstable SSH, failures: 1732555233.662396, 1732555313.8163917, 1732555393.9276178, 1732555474.0493515
[2024-11-25 17:24:34,050][ ERROR][PID:3199275] Re-try request for task on 'ResallocHost, ticket_id=6064425, hostname=52.203.150.6': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2024-11-25 17:24:34,051][  INFO][PID:3199275] Releasing VM back to pool
[2024-11-25 17:24:34,064][  INFO][PID:3199275] Retry #1 (on other host)
[2024-11-25 17:24:34,078][  INFO][PID:3199275] Trying to allocate VM
[2024-11-25 17:24:52,145][  INFO][PID:3199275] Allocated host ResallocHost, ticket_id=6064922, hostname=3.235.235.218
[2024-11-25 17:24:52,146][  INFO][PID:3199275] Allocating ssh connection to builder
[2024-11-25 17:24:52,147][  INFO][PID:3199275] Checking that builder machine is OK
[2024-11-25 17:24:52,498][  INFO][PID:3199275] Running remote command: copr-builder-ready fedora-40-x86_64
[2024-11-25 17:24:52,724][  INFO][PID:3199275] Red Hat subscription not needed for fedora-40-x86_64
Builder is ready to be used

[2024-11-25 17:24:52,725][  INFO][PID:3199275] Filling build.info file with builder info
[2024-11-25 17:24:52,751][  INFO][PID:3199275] Not re-notifying FE and msg buses for the new host.
[2024-11-25 17:24:52,752][  INFO][PID:3199275] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8312258-fedora-40-x86_64 --chroot fedora-40-x86_64 --detached
[2024-11-25 17:24:53,127][  INFO][PID:3199275] Downloading the builder-live.log file, attempt 1
[2024-11-25 17:24:53,129][  INFO][PID:3199275] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.235.235.218 copr-rpmbuild-log
[2024-11-25 17:29:23,193][  INFO][PID:3199275] Periodic builder liveness probe: alive
[2024-11-25 17:33:55,592][ ERROR][PID:3199275] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-11-25 17:34:05,610][WARNING][PID:3199275] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2024-11-25 17:34:15,611][  INFO][PID:3199275] Downloading the builder-live.log file, attempt 2
[2024-11-25 17:34:15,613][  INFO][PID:3199275] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.235.235.218 copr-rpmbuild-log
[2024-11-25 17:35:15,680][ ERROR][PID:3199275] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-11-25 17:35:25,685][WARNING][PID:3199275] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2024-11-25 17:35:35,685][  INFO][PID:3199275] Downloading the builder-live.log file, attempt 3
[2024-11-25 17:35:35,687][  INFO][PID:3199275] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.235.235.218 copr-rpmbuild-log
[2024-11-25 17:36:35,758][ ERROR][PID:3199275] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-11-25 17:36:45,778][WARNING][PID:3199275] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2024-11-25 17:36:55,779][  INFO][PID:3199275] Downloading the builder-live.log file, attempt 4
[2024-11-25 17:36:55,780][  INFO][PID:3199275] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.235.235.218 copr-rpmbuild-log
[2024-11-25 17:37:55,847][ ERROR][PID:3199275] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-11-25 17:38:05,848][ ERROR][PID:3199275] Giving up for unstable SSH, failures: 1732556045.6108885, 1732556125.6849935, 1732556205.778706, 1732556285.8487418
[2024-11-25 17:38:05,849][ ERROR][PID:3199275] Re-try request for task on 'ResallocHost, ticket_id=6064922, hostname=3.235.235.218': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2024-11-25 17:38:05,850][  INFO][PID:3199275] Releasing VM back to pool
[2024-11-25 17:38:05,887][  INFO][PID:3199275] Retry #2 (on other host)
[2024-11-25 17:38:05,888][  INFO][PID:3199275] Trying to allocate VM
[2024-11-25 17:38:08,916][  INFO][PID:3199275] Allocated host ResallocHost, ticket_id=6065110, hostname=2620:52:3:1:dead:beef:cafe:c117
[2024-11-25 17:38:08,917][  INFO][PID:3199275] Allocating ssh connection to builder
[2024-11-25 17:38:08,917][  INFO][PID:3199275] Checking that builder machine is OK
[2024-11-25 17:38:09,560][  INFO][PID:3199275] Running remote command: copr-builder-ready fedora-40-x86_64
[2024-11-25 17:38:09,987][  INFO][PID:3199275] Red Hat subscription not needed for fedora-40-x86_64
Builder is ready to be used

[2024-11-25 17:38:09,988][  INFO][PID:3199275] Filling build.info file with builder info
[2024-11-25 17:38:10,004][  INFO][PID:3199275] Not re-notifying FE and msg buses for the new host.
[2024-11-25 17:38:10,005][  INFO][PID:3199275] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8312258-fedora-40-x86_64 --chroot fedora-40-x86_64 --detached
[2024-11-25 17:38:10,689][  INFO][PID:3199275] Downloading the builder-live.log file, attempt 1
[2024-11-25 17:38:10,690][  INFO][PID:3199275] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c117 copr-rpmbuild-log
[2024-11-25 17:39:25,704][  INFO][PID:3199275] Periodic builder liveness probe: alive
[2024-11-25 17:49:25,822][  INFO][PID:3199275] Periodic builder liveness probe: alive
[2024-11-25 17:59:25,906][  INFO][PID:3199275] Periodic builder liveness probe: alive
[2024-11-25 18:09:26,121][  INFO][PID:3199275] Periodic builder liveness probe: alive
[2024-11-25 18:19:26,303][  INFO][PID:3199275] Periodic builder liveness probe: alive
[2024-11-25 18:29:26,405][  INFO][PID:3199275] Periodic builder liveness probe: alive
[2024-11-25 18:39:26,484][  INFO][PID:3199275] Periodic builder liveness probe: alive
[2024-11-25 18:49:26,587][  INFO][PID:3199275] Periodic builder liveness probe: alive
[2024-11-25 18:59:26,720][  INFO][PID:3199275] Periodic builder liveness probe: alive
[2024-11-25 19:09:26,895][  INFO][PID:3199275] Periodic builder liveness probe: alive
[2024-11-25 19:19:27,115][  INFO][PID:3199275] Periodic builder liveness probe: alive
[2024-11-25 19:29:27,230][  INFO][PID:3199275] Periodic builder liveness probe: alive
[2024-11-25 19:39:27,409][  INFO][PID:3199275] Periodic builder liveness probe: alive
[2024-11-25 19:49:27,518][  INFO][PID:3199275] Periodic builder liveness probe: alive
[2024-11-25 19:59:27,594][  INFO][PID:3199275] Periodic builder liveness probe: alive
[2024-11-25 20:06:29,949][  INFO][PID:3199275] Downloading results from builder
[2024-11-25 20:06:29,950][  INFO][PID:3199275] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c117]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/alternateved/bleeding-emacs/fedora-40-x86_64/08312258-bleeding-emacs started
[2024-11-25 20:06:29,951][  INFO][PID:3199275] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c117]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/alternateved/bleeding-emacs/fedora-40-x86_64/08312258-bleeding-emacs/ &> /var/lib/copr/public_html/results/alternateved/bleeding-emacs/fedora-40-x86_64/08312258-bleeding-emacs/build-08312258.rsync.log
[2024-11-25 20:06:30,978][  INFO][PID:3199275] rsyncing finished.
[2024-11-25 20:06:30,978][  INFO][PID:3199275] Releasing VM back to pool
[2024-11-25 20:06:30,988][  INFO][PID:3199275] Searching for 'success' file in resultdir
[2024-11-25 20:06:30,989][ ERROR][PID:3199275] Build failed: Backend process error: No success file => build failure
[2024-11-25 20:06:30,990][  INFO][PID:3199275] Finished build: id=8312258 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/alternateved/bleeding-emacs chroot=fedora-40-x86_64 
[2024-11-25 20:06:30,991][ ERROR][PID:3199275] Backend process error: No success file => build failure
[2024-11-25 20:06:30,992][  INFO][PID:3199275] Worker failed build, took 8900.987446308136
[2024-11-25 20:06:30,992][  INFO][PID:3199275] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "alternateved",
            "project_name": "bleeding-emacs",
            "project_dirname": "bleeding-emacs",
            "submitter": "gitlab.com:alternateved",
            "ended_on": 1732565190.9920316,
            "started_on": 1732556290.0045853,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-40-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "8312258-fedora-40-x86_64",
            "build_id": 8312258,
            "package_name": "bleeding-emacs",
            "package_version": "1:30.0.92-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/alternateved/bleeding-emacs/bleeding-emacs",
            "git_hash": "ec1410148dcd9ec14fa801ea6c1f2b1eab3bb0e6",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "alternateved/bleeding-emacs--gitlab.com:alternateved",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/alternateved/bleeding-emacs/fedora-40-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/alternateved/bleeding-emacs",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/alternateved/bleeding-emacs",
            "result_dir": "08312258-bleeding-emacs",
            "built_packages": "",
            "id": 8312258,
            "mockchain_macros": {
                "copr_username": "alternateved",
                "copr_projectname": "bleeding-emacs",
                "vendor": "Fedora Project COPR (alternateved/bleeding-emacs)"
            }
        }
    ]
}
[2024-11-25 20:06:31,056][  INFO][PID:3199275] Sending fedora-messaging bus message in build.end
[2024-11-25 20:06:31,086][  INFO][PID:3199275] Compressing /var/lib/copr/public_html/results/alternateved/bleeding-emacs/fedora-40-x86_64/08312258-bleeding-emacs/builder-live.log by gzip
[2024-11-25 20:06:31,087][  INFO][PID:3199275] Running command 'gzip /var/lib/copr/public_html/results/alternateved/bleeding-emacs/fedora-40-x86_64/08312258-bleeding-emacs/builder-live.log' as PID 11609
[2024-11-25 20:06:31,356][  INFO][PID:3199275] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/alternateved/bleeding-emacs/fedora-40-x86_64/08312258-bleeding-emacs/builder-live.log)
[2024-11-25 20:06:31,356][  INFO][PID:3199275] Compressing /var/lib/copr/public_html/results/alternateved/bleeding-emacs/fedora-40-x86_64/08312258-bleeding-emacs/backend.log by gzip
[2024-11-25 20:06:31,357][  INFO][PID:3199275] Running command 'gzip /var/lib/copr/public_html/results/alternateved/bleeding-emacs/fedora-40-x86_64/08312258-bleeding-emacs/backend.log' as PID 11620