[2024-12-18 14:43:10,780][  INFO][PID:347352] Marking build as starting
[2024-12-18 14:43:10,806][  INFO][PID:347352] Trying to allocate VM
[2024-12-18 14:43:16,878][  INFO][PID:347352] Allocated host ResallocHost, ticket_id=252395, hostname=54.211.55.211
[2024-12-18 14:43:16,879][  INFO][PID:347352] Allocating ssh connection to builder
[2024-12-18 14:43:16,880][  INFO][PID:347352] Checking that builder machine is OK
[2024-12-18 14:43:17,255][  INFO][PID:347352] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.211.55.211 '/usr/bin/test -f /etc/mock/fedora-rawhide-x86_64.cfg'
[2024-12-18 14:43:17,471][  INFO][PID:347352] Filling build.info file with builder info
[2024-12-18 14:43:17,473][  INFO][PID:347352] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@python",
            "project_name": "python3.14",
            "project_dirname": "python3.14",
            "submitter": "ksurma",
            "ended_on": null,
            "started_on": 1734532997.4735487,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "8410103-fedora-rawhide-x86_64",
            "build_id": 8410103,
            "package_name": "xpra",
            "package_version": "5.0.10-2",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@python/python3.14/xpra",
            "git_hash": "b5ab7beb545f0530c038d4e1bbc43c6e7ce33a60",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@python/python3.14--ksurma",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": true,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": 9,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@python/python3.14/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": 9
                },
                {
                    "baseurl": "http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/",
                    "id": "http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch",
                    "name": "Additional repo http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch"
                }
            ],
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@python/python3.14",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@python/python3.14",
            "result_dir": "08410103-xpra",
            "built_packages": "",
            "id": 8410103,
            "mockchain_macros": {
                "copr_username": "@python",
                "copr_projectname": "python3.14",
                "vendor": "Fedora Project COPR (@python/python3.14)"
            }
        }
    ]
}
[2024-12-18 14:43:17,500][  INFO][PID:347352] Sending fedora-messaging bus message in build.start
[2024-12-18 14:43:18,211][  INFO][PID:347352] Sending fedora-messaging bus message in chroot.start
[2024-12-18 14:43:18,228][  INFO][PID:347352] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8410103-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2024-12-18 14:43:18,616][  INFO][PID:347352] Downloading the builder-live.log file, attempt 1
[2024-12-18 14:43:18,618][  INFO][PID:347352] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.211.55.211 copr-rpmbuild-log
[2024-12-18 14:43:23,626][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 14:53:23,770][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 15:03:23,889][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 15:05:46,236][ ERROR][PID:347352] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-12-18 15:05:56,242][WARNING][PID:347352] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2024-12-18 15:06:06,244][  INFO][PID:347352] Downloading the builder-live.log file, attempt 2
[2024-12-18 15:06:06,258][  INFO][PID:347352] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.211.55.211 copr-rpmbuild-log
[2024-12-18 15:06:06,268][ ERROR][PID:347352] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-12-18 15:06:16,269][WARNING][PID:347352] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2024-12-18 15:06:26,270][  INFO][PID:347352] Downloading the builder-live.log file, attempt 3
[2024-12-18 15:06:26,271][  INFO][PID:347352] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.211.55.211 copr-rpmbuild-log
[2024-12-18 15:06:26,282][ ERROR][PID:347352] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-12-18 15:06:36,287][WARNING][PID:347352] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2024-12-18 15:06:46,288][  INFO][PID:347352] Downloading the builder-live.log file, attempt 4
[2024-12-18 15:06:46,290][  INFO][PID:347352] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.211.55.211 copr-rpmbuild-log
[2024-12-18 15:07:46,358][ ERROR][PID:347352] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2024-12-18 15:07:56,359][ ERROR][PID:347352] Giving up for unstable SSH, failures: 1734534356.2428641, 1734534376.2690074, 1734534396.287029, 1734534476.3595543
[2024-12-18 15:07:56,360][ ERROR][PID:347352] Re-try request for task on 'ResallocHost, ticket_id=252395, hostname=54.211.55.211': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2024-12-18 15:07:56,361][  INFO][PID:347352] Releasing VM back to pool
[2024-12-18 15:07:56,371][  INFO][PID:347352] Retry #1 (on other host)
[2024-12-18 15:07:56,372][  INFO][PID:347352] Trying to allocate VM
[2024-12-18 15:07:59,462][  INFO][PID:347352] Allocated host ResallocHost, ticket_id=252848, hostname=44.197.198.161
[2024-12-18 15:07:59,463][  INFO][PID:347352] Allocating ssh connection to builder
[2024-12-18 15:07:59,468][  INFO][PID:347352] Checking that builder machine is OK
[2024-12-18 15:07:59,882][  INFO][PID:347352] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.197.198.161 '/usr/bin/test -f /etc/mock/fedora-rawhide-x86_64.cfg'
[2024-12-18 15:08:00,097][  INFO][PID:347352] Filling build.info file with builder info
[2024-12-18 15:08:00,103][  INFO][PID:347352] Not re-notifying FE and msg buses for the new host.
[2024-12-18 15:08:00,104][  INFO][PID:347352] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8410103-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2024-12-18 15:08:00,501][  INFO][PID:347352] Downloading the builder-live.log file, attempt 1
[2024-12-18 15:08:00,503][  INFO][PID:347352] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.197.198.161 copr-rpmbuild-log
[2024-12-18 15:13:25,587][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 15:23:25,680][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 15:33:25,772][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 15:43:25,854][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 15:53:25,933][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 16:03:26,035][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 16:13:26,223][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 16:23:26,322][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 16:33:26,412][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 16:43:26,517][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 16:53:26,617][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 17:03:26,722][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 17:13:26,853][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 17:23:27,022][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 17:33:27,105][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 17:43:27,196][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 17:53:27,299][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 18:03:27,391][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 18:13:27,507][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 18:23:27,614][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 18:33:27,714][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 18:43:27,807][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 18:53:27,893][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 19:03:28,080][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 19:13:28,252][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 19:23:28,392][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 19:33:28,497][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 19:43:28,589][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 19:53:28,668][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 20:03:28,750][  INFO][PID:347352] Periodic builder liveness probe: alive
[2024-12-18 20:08:03,008][  INFO][PID:347352] Downloading results from builder
[2024-12-18 20:08:03,040][  INFO][PID:347352] rsyncing of mockbuilder@44.197.198.161:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@python/python3.14/fedora-rawhide-x86_64/08410103-xpra started
[2024-12-18 20:08:03,075][  INFO][PID:347352] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@44.197.198.161:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@python/python3.14/fedora-rawhide-x86_64/08410103-xpra/ &> /var/lib/copr/public_html/results/@python/python3.14/fedora-rawhide-x86_64/08410103-xpra/build-08410103.rsync.log
[2024-12-18 20:08:03,416][  INFO][PID:347352] rsyncing finished.
[2024-12-18 20:08:03,419][  INFO][PID:347352] Releasing VM back to pool
[2024-12-18 20:08:03,434][  INFO][PID:347352] Searching for 'success' file in resultdir
[2024-12-18 20:08:03,438][ ERROR][PID:347352] Build failed: Backend process error: No success file => build failure
[2024-12-18 20:08:03,553][  INFO][PID:347352] Finished build: id=8410103 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/@python/python3.14 chroot=fedora-rawhide-x86_64 
[2024-12-18 20:08:03,638][ ERROR][PID:347352] Backend process error: No success file => build failure
[2024-12-18 20:08:03,641][  INFO][PID:347352] Worker failed build, took 18003.53755044937
[2024-12-18 20:08:03,657][  INFO][PID:347352] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@python",
            "project_name": "python3.14",
            "project_dirname": "python3.14",
            "submitter": "ksurma",
            "ended_on": 1734552483.6414804,
            "started_on": 1734534480.10393,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "8410103-fedora-rawhide-x86_64",
            "build_id": 8410103,
            "package_name": "xpra",
            "package_version": "5.0.10-2",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@python/python3.14/xpra",
            "git_hash": "b5ab7beb545f0530c038d4e1bbc43c6e7ce33a60",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@python/python3.14--ksurma",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": true,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": 9,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@python/python3.14/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": 9
                },
                {
                    "baseurl": "http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/",
                    "id": "http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch",
                    "name": "Additional repo http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch"
                }
            ],
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@python/python3.14",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@python/python3.14",
            "result_dir": "08410103-xpra",
            "built_packages": "",
            "id": 8410103,
            "mockchain_macros": {
                "copr_username": "@python",
                "copr_projectname": "python3.14",
                "vendor": "Fedora Project COPR (@python/python3.14)"
            }
        }
    ]
}
[2024-12-18 20:08:04,309][  INFO][PID:347352] Sending fedora-messaging bus message in build.end
[2024-12-18 20:08:04,690][  INFO][PID:347352] Compressing /var/lib/copr/public_html/results/@python/python3.14/fedora-rawhide-x86_64/08410103-xpra/builder-live.log by gzip
[2024-12-18 20:08:04,729][  INFO][PID:347352] Running command 'gzip /var/lib/copr/public_html/results/@python/python3.14/fedora-rawhide-x86_64/08410103-xpra/builder-live.log' as PID 2308165
[2024-12-18 20:08:10,556][  INFO][PID:347352] Finished after 5 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@python/python3.14/fedora-rawhide-x86_64/08410103-xpra/builder-live.log)
[2024-12-18 20:08:10,566][  INFO][PID:347352] Compressing /var/lib/copr/public_html/results/@python/python3.14/fedora-rawhide-x86_64/08410103-xpra/backend.log by gzip
[2024-12-18 20:08:10,568][  INFO][PID:347352] Running command 'gzip /var/lib/copr/public_html/results/@python/python3.14/fedora-rawhide-x86_64/08410103-xpra/backend.log' as PID 2308783