[2024-11-28 02:18:08,856][  INFO][PID:273769] Marking build as starting
[2024-11-28 02:18:09,021][  INFO][PID:273769] Trying to allocate VM
[2024-11-28 02:18:37,253][  INFO][PID:273769] Allocated host ResallocHost, ticket_id=6109268, hostname=2620:52:3:1:dead:beef:cafe:c299
[2024-11-28 02:18:37,257][  INFO][PID:273769] Allocating ssh connection to builder
[2024-11-28 02:18:37,258][  INFO][PID:273769] Checking that builder machine is OK
[2024-11-28 02:18:38,220][  INFO][PID:273769] Running remote command: copr-builder-ready srpm-builds
[2024-11-28 02:18:38,797][  INFO][PID:273769] Red Hat subscription not needed for srpm-builds
Builder is ready to be used

[2024-11-28 02:18:38,798][  INFO][PID:273769] Filling build.info file with builder info
[2024-11-28 02:18:38,801][  INFO][PID:273769] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "@fedora-llvm-team",
            "project_name": "llvm-snapshots-pgo-20241128",
            "project_dirname": "llvm-snapshots-pgo-20241128",
            "submitter": "tstellar",
            "ended_on": null,
            "started_on": 1732760318.8006287,
            "submitted_on": null,
            "status": 3,
            "chroot": "srpm-builds",
            "arch": null,
            "buildroot_pkgs": null,
            "task_id": "8321193",
            "build_id": 8321193,
            "package_name": "llvm",
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 8,
            "source_json": "{\"type\": \"git\", \"clone_url\": \"https://src.fedoraproject.org/forks/kkleine/rpms/llvm.git\", \"subdirectory\": \"\", \"committish\": \"pgo\", \"spec\": \"llvm.spec\", \"srpm_build_method\": \"make_srpm\"}",
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": null,
            "sandbox": "@fedora-llvm-team/llvm-snapshots-pgo-20241128--tstellar",
            "results": null,
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "background": false,
            "repos": [],
            "destdir": "/var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128",
            "result_dir": "08321193",
            "built_packages": "",
            "tags": [
                "arch_noarch"
            ],
            "id": 8321193,
            "mockchain_macros": {
                "copr_username": "@fedora-llvm-team",
                "copr_projectname": "llvm-snapshots-pgo-20241128",
                "vendor": "Fedora Project COPR (@fedora-llvm-team/llvm-snapshots-pgo-20241128)"
            }
        }
    ]
}
[2024-11-28 02:18:38,992][  INFO][PID:273769] Sending fedora-messaging bus message in build.start
[2024-11-28 02:18:41,462][  INFO][PID:273769] Sending fedora-messaging bus message in chroot.start
[2024-11-28 02:18:41,536][  INFO][PID:273769] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/8321193 --detached
[2024-11-28 02:18:42,496][  INFO][PID:273769] Downloading the builder-live.log file, attempt 1
[2024-11-28 02:18:42,509][  INFO][PID:273769] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c299 copr-rpmbuild-log
[2024-11-28 02:18:47,534][  INFO][PID:273769] Periodic builder liveness probe: alive
[2024-11-28 02:20:50,411][  INFO][PID:273769] Downloading results from builder
[2024-11-28 02:20:50,425][  INFO][PID:273769] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c299]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193 started
[2024-11-28 02:20:50,436][  INFO][PID:273769] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c299]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193/ &> /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193/build-08321193.rsync.log
[2024-11-28 02:23:46,126][  INFO][PID:273769] rsyncing finished.
[2024-11-28 02:23:46,127][ ERROR][PID:273769] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2024-11-28 02:23:46,128][ ERROR][PID:273769] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2024-11-28 02:23:56,129][  INFO][PID:273769] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c299]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193 started
[2024-11-28 02:23:56,130][  INFO][PID:273769] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c299]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193/ &> /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193/build-08321193.rsync.log
[2024-11-28 02:24:56,207][  INFO][PID:273769] rsyncing finished.
[2024-11-28 02:24:56,208][ ERROR][PID:273769] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2024-11-28 02:24:56,209][ ERROR][PID:273769] SSH connection lost on #2 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2024-11-28 02:25:06,210][  INFO][PID:273769] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c299]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193 started
[2024-11-28 02:25:06,211][  INFO][PID:273769] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c299]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193/ &> /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193/build-08321193.rsync.log
[2024-11-28 02:26:06,361][  INFO][PID:273769] rsyncing finished.
[2024-11-28 02:26:06,367][ ERROR][PID:273769] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2024-11-28 02:26:06,368][ ERROR][PID:273769] SSH connection lost on #3 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2024-11-28 02:26:16,368][ ERROR][PID:273769] Re-try request for task on 'ResallocHost, ticket_id=6109268, hostname=2620:52:3:1:dead:beef:cafe:c299': Unable to finish after 3 SSH attempts
[2024-11-28 02:26:16,369][  INFO][PID:273769] Releasing VM back to pool
[2024-11-28 02:26:16,383][  INFO][PID:273769] Retry #1 (on other host)
[2024-11-28 02:26:16,384][  INFO][PID:273769] Trying to allocate VM
[2024-11-28 02:26:19,445][  INFO][PID:273769] Allocated host ResallocHost, ticket_id=6109376, hostname=44.200.170.206
[2024-11-28 02:26:19,445][  INFO][PID:273769] Allocating ssh connection to builder
[2024-11-28 02:26:19,446][  INFO][PID:273769] Checking that builder machine is OK
[2024-11-28 02:26:19,793][  INFO][PID:273769] Running remote command: copr-builder-ready srpm-builds
[2024-11-28 02:26:20,023][  INFO][PID:273769] Red Hat subscription not needed for srpm-builds
Builder is ready to be used

[2024-11-28 02:26:20,024][  INFO][PID:273769] Filling build.info file with builder info
[2024-11-28 02:26:20,026][  INFO][PID:273769] Not re-notifying FE and msg buses for the new host.
[2024-11-28 02:26:20,027][  INFO][PID:273769] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/8321193 --detached
[2024-11-28 02:26:20,396][  INFO][PID:273769] Downloading the builder-live.log file, attempt 1
[2024-11-28 02:26:20,398][  INFO][PID:273769] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.200.170.206 copr-rpmbuild-log
[2024-11-28 02:27:00,425][  INFO][PID:273769] Downloading results from builder
[2024-11-28 02:27:00,426][  INFO][PID:273769] rsyncing of mockbuilder@44.200.170.206:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193 started
[2024-11-28 02:27:00,427][  INFO][PID:273769] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@44.200.170.206:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193/ &> /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193/build-08321193.rsync.log
[2024-11-28 02:27:02,989][  INFO][PID:273769] rsyncing finished.
[2024-11-28 02:27:02,989][  INFO][PID:273769] Releasing VM back to pool
[2024-11-28 02:27:02,998][  INFO][PID:273769] Searching for 'success' file in resultdir
[2024-11-28 02:27:02,999][  INFO][PID:273769] Getting build details
[2024-11-28 02:27:02,999][  INFO][PID:273769] Retrieving SRPM info from /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193
[2024-11-28 02:27:03,000][  INFO][PID:273769] SRPM URL: https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193/llvm-20.0.0~pre20241128.g9bdf683ba6cd9a-1.src.rpm
[2024-11-28 02:27:03,001][  INFO][PID:273769] build details: {'srpm_url': 'https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193/llvm-20.0.0~pre20241128.g9bdf683ba6cd9a-1.src.rpm', 'pkg_name': 'llvm', 'pkg_version': '19.1.3-1'}
[2024-11-28 02:27:03,001][  INFO][PID:273769] Finished build: id=8321193 failed=False timeout=108000 destdir=/var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128 chroot=srpm-builds 
[2024-11-28 02:27:03,003][  INFO][PID:273769] Worker succeeded build, took 42.977227449417114
[2024-11-28 02:27:03,004][  INFO][PID:273769] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "@fedora-llvm-team",
            "project_name": "llvm-snapshots-pgo-20241128",
            "project_dirname": "llvm-snapshots-pgo-20241128",
            "submitter": "tstellar",
            "ended_on": 1732760823.0038521,
            "started_on": 1732760780.0266247,
            "submitted_on": null,
            "status": 1,
            "chroot": "srpm-builds",
            "arch": null,
            "buildroot_pkgs": null,
            "task_id": "8321193",
            "build_id": 8321193,
            "package_name": "llvm",
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 8,
            "source_json": "{\"type\": \"git\", \"clone_url\": \"https://src.fedoraproject.org/forks/kkleine/rpms/llvm.git\", \"subdirectory\": \"\", \"committish\": \"pgo\", \"spec\": \"llvm.spec\", \"srpm_build_method\": \"make_srpm\"}",
            "pkg_name": "llvm",
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193/llvm-20.0.0~pre20241128.g9bdf683ba6cd9a-1.src.rpm",
            "uses_devel_repo": null,
            "sandbox": "@fedora-llvm-team/llvm-snapshots-pgo-20241128--tstellar",
            "results": {
                "name": "llvm",
                "epoch": null,
                "version": "19.1.3",
                "release": "1",
                "exclusivearch": [],
                "excludearch": []
            },
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "background": false,
            "repos": [],
            "destdir": "/var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128",
            "result_dir": "08321193",
            "built_packages": "",
            "tags": [
                "arch_noarch"
            ],
            "pkg_version": "19.1.3-1",
            "id": 8321193,
            "mockchain_macros": {
                "copr_username": "@fedora-llvm-team",
                "copr_projectname": "llvm-snapshots-pgo-20241128",
                "vendor": "Fedora Project COPR (@fedora-llvm-team/llvm-snapshots-pgo-20241128)"
            }
        }
    ]
}
[2024-11-28 02:27:03,069][  INFO][PID:273769] Sending fedora-messaging bus message in build.end
[2024-11-28 02:27:03,092][  INFO][PID:273769] Compressing /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193/builder-live.log by gzip
[2024-11-28 02:27:03,093][  INFO][PID:273769] Running command 'gzip /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193/builder-live.log' as PID 360321
[2024-11-28 02:27:03,098][  INFO][PID:273769] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193/builder-live.log)
[2024-11-28 02:27:03,099][  INFO][PID:273769] Compressing /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193/backend.log by gzip
[2024-11-28 02:27:03,100][  INFO][PID:273769] Running command 'gzip /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-pgo-20241128/srpm-builds/08321193/backend.log' as PID 360322