[2024-11-18 17:42:32,957][ INFO][PID:3544514] Marking build as starting [2024-11-18 17:42:33,117][ INFO][PID:3544514] Trying to allocate VM [2024-11-18 17:44:01,300][ INFO][PID:3544514] Allocated host ResallocHost, ticket_id=5936006, hostname=34.239.147.0 [2024-11-18 17:44:01,301][ INFO][PID:3544514] Allocating ssh connection to builder [2024-11-18 17:44:01,301][ INFO][PID:3544514] Checking that builder machine is OK [2024-11-18 17:44:01,634][ INFO][PID:3544514] Running remote command: copr-builder-ready fedora-39-x86_64 [2024-11-18 17:44:01,869][ INFO][PID:3544514] Red Hat subscription not needed for fedora-39-x86_64 Builder is ready to be used [2024-11-18 17:44:01,871][ INFO][PID:3544514] Filling build.info file with builder info [2024-11-18 17:44:01,873][ INFO][PID:3544514] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "thrnciar", "project_name": "setuptools-impact-check-control", "project_dirname": "setuptools-impact-check-control:pr:337", "submitter": "https://src.fedoraproject.org/user/tuliom", "ended_on": null, "started_on": 1731951841.8731344, "submitted_on": null, "status": 3, "chroot": "fedora-39-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "8276800-fedora-39-x86_64", "build_id": 8276800, "package_name": "llvm", "package_version": "19.1.3-2", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/thrnciar/setuptools-impact-check-control:pr:337/llvm", "git_hash": "56599096981715c97e67cef3f28f04ce1489a4dc", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "thrnciar/setuptools-impact-check-control--https://src.fedoraproject.org/user/tuliom", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/thrnciar/setuptools-impact-check-control/fedora-39-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/thrnciar/setuptools-impact-check-control:pr:337", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/thrnciar/setuptools-impact-check-control:pr:337", "result_dir": "08276800-llvm", "built_packages": "", "id": 8276800, "mockchain_macros": { "copr_username": "thrnciar", "copr_projectname": "setuptools-impact-check-control", "vendor": "Fedora Project COPR (thrnciar/setuptools-impact-check-control)" } } ] } [2024-11-18 17:44:01,956][ INFO][PID:3544514] Sending fedora-messaging bus message in build.start [2024-11-18 17:44:02,799][ INFO][PID:3544514] Sending fedora-messaging bus message in chroot.start [2024-11-18 17:44:02,833][ INFO][PID:3544514] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8276800-fedora-39-x86_64 --chroot fedora-39-x86_64 --detached [2024-11-18 17:44:03,219][ INFO][PID:3544514] Downloading the builder-live.log file, attempt 1 [2024-11-18 17:44:03,223][ INFO][PID:3544514] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.239.147.0 copr-rpmbuild-log [2024-11-18 17:44:08,233][ INFO][PID:3544514] Periodic builder liveness probe: alive [2024-11-18 17:54:08,345][ INFO][PID:3544514] Periodic builder liveness probe: alive [2024-11-18 18:04:08,532][ INFO][PID:3544514] Periodic builder liveness probe: alive [2024-11-18 18:13:20,493][ ERROR][PID:3544514] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-11-18 18:13:30,494][WARNING][PID:3544514] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-11-18 18:13:40,495][ INFO][PID:3544514] Downloading the builder-live.log file, attempt 2 [2024-11-18 18:13:40,521][ INFO][PID:3544514] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.239.147.0 copr-rpmbuild-log [2024-11-18 18:14:12,312][ INFO][PID:3544514] Periodic builder liveness probe: alive [2024-11-18 18:14:40,591][ ERROR][PID:3544514] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-11-18 18:14:50,600][WARNING][PID:3544514] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-11-18 18:15:00,601][ INFO][PID:3544514] Downloading the builder-live.log file, attempt 3 [2024-11-18 18:15:00,603][ INFO][PID:3544514] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.239.147.0 copr-rpmbuild-log [2024-11-18 18:16:00,671][ ERROR][PID:3544514] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-11-18 18:16:10,673][WARNING][PID:3544514] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-11-18 18:16:20,674][ INFO][PID:3544514] Downloading the builder-live.log file, attempt 4 [2024-11-18 18:16:20,676][ INFO][PID:3544514] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.239.147.0 copr-rpmbuild-log [2024-11-18 18:17:20,744][ ERROR][PID:3544514] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-11-18 18:17:30,747][ ERROR][PID:3544514] Giving up for unstable SSH, failures: 1731953610.494244, 1731953690.6007535, 1731953770.6737792, 1731953850.7471502 [2024-11-18 18:17:30,748][ ERROR][PID:3544514] Re-try request for task on 'ResallocHost, ticket_id=5936006, hostname=34.239.147.0': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2024-11-18 18:17:30,748][ INFO][PID:3544514] Releasing VM back to pool [2024-11-18 18:17:30,761][ INFO][PID:3544514] Retry #1 (on other host) [2024-11-18 18:17:30,763][ INFO][PID:3544514] Trying to allocate VM [2024-11-18 18:25:48,955][ INFO][PID:3544514] Allocated host ResallocHost, ticket_id=5936822, hostname=3.91.228.216 [2024-11-18 18:25:48,960][ INFO][PID:3544514] Allocating ssh connection to builder [2024-11-18 18:25:48,960][ INFO][PID:3544514] Checking that builder machine is OK [2024-11-18 18:25:49,319][ INFO][PID:3544514] Running remote command: copr-builder-ready fedora-39-x86_64 [2024-11-18 18:25:49,544][ INFO][PID:3544514] Red Hat subscription not needed for fedora-39-x86_64 Builder is ready to be used [2024-11-18 18:25:49,545][ INFO][PID:3544514] Filling build.info file with builder info [2024-11-18 18:25:49,554][ INFO][PID:3544514] Not re-notifying FE and msg buses for the new host. [2024-11-18 18:25:49,555][ INFO][PID:3544514] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8276800-fedora-39-x86_64 --chroot fedora-39-x86_64 --detached [2024-11-18 18:25:49,949][ INFO][PID:3544514] Downloading the builder-live.log file, attempt 1 [2024-11-18 18:25:49,951][ INFO][PID:3544514] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.91.228.216 copr-rpmbuild-log [2024-11-18 18:25:54,965][ INFO][PID:3544514] Periodic builder liveness probe: alive [2024-11-18 18:29:41,862][ ERROR][PID:3544514] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-11-18 18:29:51,863][WARNING][PID:3544514] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-11-18 18:30:01,864][ INFO][PID:3544514] Downloading the builder-live.log file, attempt 2 [2024-11-18 18:30:01,866][ INFO][PID:3544514] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.91.228.216 copr-rpmbuild-log [2024-11-18 18:31:01,934][ ERROR][PID:3544514] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-11-18 18:31:11,959][WARNING][PID:3544514] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-11-18 18:31:21,961][ INFO][PID:3544514] Downloading the builder-live.log file, attempt 3 [2024-11-18 18:31:21,962][ INFO][PID:3544514] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.91.228.216 copr-rpmbuild-log [2024-11-18 18:32:22,030][ ERROR][PID:3544514] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-11-18 18:32:32,040][WARNING][PID:3544514] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-11-18 18:32:42,041][ INFO][PID:3544514] Downloading the builder-live.log file, attempt 4 [2024-11-18 18:32:42,043][ INFO][PID:3544514] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.91.228.216 copr-rpmbuild-log [2024-11-18 18:33:42,111][ ERROR][PID:3544514] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-11-18 18:33:52,112][ ERROR][PID:3544514] Giving up for unstable SSH, failures: 1731954591.863079, 1731954671.959618, 1731954752.040607, 1731954832.1122544 [2024-11-18 18:33:52,113][ ERROR][PID:3544514] Re-try request for task on 'ResallocHost, ticket_id=5936822, hostname=3.91.228.216': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2024-11-18 18:33:52,114][ INFO][PID:3544514] Releasing VM back to pool [2024-11-18 18:33:52,125][ INFO][PID:3544514] Retry #2 (on other host) [2024-11-18 18:33:52,126][ INFO][PID:3544514] Trying to allocate VM [2024-11-18 18:35:20,219][ INFO][PID:3544514] Allocated host ResallocHost, ticket_id=5936991, hostname=98.82.113.19 [2024-11-18 18:35:20,220][ INFO][PID:3544514] Allocating ssh connection to builder [2024-11-18 18:35:20,220][ INFO][PID:3544514] Checking that builder machine is OK [2024-11-18 18:36:20,290][ ERROR][PID:3544514] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 98.82.113.19 port 22: Connection timed out [2024-11-18 18:36:30,292][ ERROR][PID:3544514] Re-try request for task on 'ResallocHost, ticket_id=5936991, hostname=98.82.113.19': Unable to finish after 1 SSH attempts [2024-11-18 18:36:30,293][ INFO][PID:3544514] Releasing VM back to pool [2024-11-18 18:36:30,307][ ERROR][PID:3544514] Backend process error: Three host tried without success: {'98.82.113.19', '3.91.228.216', '34.239.147.0'} [2024-11-18 18:36:30,309][WARNING][PID:3544514] Switching not-finished job state to 'failed' [2024-11-18 18:36:30,309][ INFO][PID:3544514] Worker failed build, took 640.7544982433319 [2024-11-18 18:36:30,310][ INFO][PID:3544514] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "thrnciar", "project_name": "setuptools-impact-check-control", "project_dirname": "setuptools-impact-check-control:pr:337", "submitter": "https://src.fedoraproject.org/user/tuliom", "ended_on": 1731954990.3091245, "started_on": 1731954349.5546262, "submitted_on": null, "status": 0, "chroot": "fedora-39-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "8276800-fedora-39-x86_64", "build_id": 8276800, "package_name": "llvm", "package_version": "19.1.3-2", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/thrnciar/setuptools-impact-check-control:pr:337/llvm", "git_hash": "56599096981715c97e67cef3f28f04ce1489a4dc", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "thrnciar/setuptools-impact-check-control--https://src.fedoraproject.org/user/tuliom", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/thrnciar/setuptools-impact-check-control/fedora-39-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/thrnciar/setuptools-impact-check-control:pr:337", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/thrnciar/setuptools-impact-check-control:pr:337", "result_dir": "08276800-llvm", "built_packages": "", "id": 8276800, "mockchain_macros": { "copr_username": "thrnciar", "copr_projectname": "setuptools-impact-check-control", "vendor": "Fedora Project COPR (thrnciar/setuptools-impact-check-control)" } } ] } [2024-11-18 18:36:30,385][ INFO][PID:3544514] Sending fedora-messaging bus message in build.end [2024-11-18 18:36:30,418][ INFO][PID:3544514] Compressing /var/lib/copr/public_html/results/thrnciar/setuptools-impact-check-control:pr:337/fedora-39-x86_64/08276800-llvm/builder-live.log by gzip [2024-11-18 18:36:30,419][ INFO][PID:3544514] Running command 'gzip /var/lib/copr/public_html/results/thrnciar/setuptools-impact-check-control:pr:337/fedora-39-x86_64/08276800-llvm/builder-live.log' as PID 3919811 [2024-11-18 18:36:30,539][ INFO][PID:3544514] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/thrnciar/setuptools-impact-check-control:pr:337/fedora-39-x86_64/08276800-llvm/builder-live.log) [2024-11-18 18:36:30,540][ INFO][PID:3544514] Compressing /var/lib/copr/public_html/results/thrnciar/setuptools-impact-check-control:pr:337/fedora-39-x86_64/08276800-llvm/backend.log by gzip [2024-11-18 18:36:30,542][ INFO][PID:3544514] Running command 'gzip /var/lib/copr/public_html/results/thrnciar/setuptools-impact-check-control:pr:337/fedora-39-x86_64/08276800-llvm/backend.log' as PID 3919823