# Disable debug package (Go binaries are statically linked) %global debug_package %{nil} # Go import path %global goipath github.com/IONIS-AI/ionis-apps Name: ionis-apps Version: 4.0.4 Release: 1%{?dist} Summary: High-performance WSPR/Solar data ingestion tools for ClickHouse License: GPL-3.0-or-later URL: https://github.com/IONIS-AI/ionis-apps Source0: https://github.com/IONIS-AI/%{name}/archive/refs/tags/v%{version}.tar.gz#/%{name}-%{version}.tar.gz # Architecture-specific (Go compiles to native binaries) ExclusiveArch: x86_64 aarch64 Obsoletes: ki7mt-ai-lab-apps < 3.0.0 Provides: ki7mt-ai-lab-apps # Build requirements BuildRequires: golang >= 1.22 BuildRequires: git BuildRequires: make # Runtime requirements Requires: ionis-core >= 3.0.0 Requires: %{name}-wspr = %{version}-%{release} Requires: %{name}-solar = %{version}-%{release} Requires: %{name}-contest = %{version}-%{release} Requires: %{name}-pskr = %{version}-%{release} %description High-performance Go applications for IONIS WSPR (Weak Signal Propagation Reporter) and Solar flux data processing. Optimized for 10+ billion row datasets with ClickHouse backend. Ingestion tools use ch-go native protocol with LZ4 compression for maximum throughput. Benchmarks on Threadripper 9975WX: - wspr-shredder: 14.4 Mrps (uncompressed CSV) - wspr-turbo: 8.8 Mrps (streaming from .gz archives) - wspr-parquet-native: 8.4 Mrps (Parquet files) %package wspr Summary: WSPR data processing tools Requires: %{name} = %{version}-%{release} BuildRequires: systemd-rpm-macros %{?systemd_requires} Obsoletes: ki7mt-ai-lab-apps-wspr < 3.0.0 Provides: ki7mt-ai-lab-apps-wspr = %{version}-%{release} %description wspr WSPR (Weak Signal Propagation Reporter) data processing applications: - wspr-shredder: High-performance uncompressed CSV ingester (14.4 Mrps) - wspr-turbo: Zero-copy streaming ingester for .gz archives (8.8 Mrps) Supports --full, --prime, --dry-run with watermark tracking. - wspr-parquet-native: Native Parquet file ingester (8.4 Mrps) - wspr-download: Parallel archive downloader from wsprnet.org (LEGACY) - wspr-live-download: Fetch spots from wspr.live/wsprdaemon to dated .jsonl.gz files - wspr-live-ingest: Load those files into wspr.bronze with ingest_log watermark - wspr-backfill: One-shot recovery tool (fused fetch+insert; superseded by the pair) wspr-backfill is the PRIMARY WSPR ingest path as of 4.0.4. wsprnet's monthly CSV export has dropped most spots in the first ~10 minutes of every hour since 2023-10-16, and stopped publishing entirely on 2026-05-25. wspr-download is retained for the day the archives return, but the export defect is a separate, unfixed fault, so the CSV path is no longer authoritative. All ingestion tools use ch-go native protocol with LZ4 compression. %package solar Summary: Solar flux data processing tools Requires: %{name} = %{version}-%{release} Obsoletes: ki7mt-ai-lab-apps-solar < 3.0.0 Provides: ki7mt-ai-lab-apps-solar = %{version}-%{release} %description solar Solar and geomagnetic data processing applications: - solar-download: Multi-source solar data downloader (SIDC, NOAA, GOES) - solar-ingest: Solar/geomagnetic data ingestion (SFI, SSN, Kp, Ap, X-ray) - solar-refresh: Download + truncate + ingest pipeline script - solar-live-update: Now-Casting live conditions updater (15-min cron) - solar-history-load: Historical solar data loader for training (6-hour cron) - dscovr-ingest: DSCOVR L1 solar wind ingester (Bz, Bt, speed, density, temp) %package contest Summary: Contest and RBN data processing tools Requires: %{name} = %{version}-%{release} Obsoletes: ki7mt-ai-lab-apps-contest < 3.0.0 Provides: ki7mt-ai-lab-apps-contest = %{version}-%{release} %description contest Contest log and Reverse Beacon Network data processing applications: - contest-download: CQ/ARRL Cabrillo log downloader (rate-limited, hash-based) - contest-ingest: Cabrillo V2/V3 parser → ClickHouse (225.7M QSOs from 475K files) - rbn-download: RBN daily ZIP archive downloader (2009-present) - rbn-ingest: RBN ZIP→CSV→ClickHouse ingester (2.18B rows in 3m32s) All ingesters support --full, --prime, and --dry-run modes with watermark tracking via ingest_log tables for safe incremental cron-based loading. All ingestion tools use ch-go native protocol with LZ4 compression. %package pskr Summary: PSK Reporter real-time data collector Requires: %{name} = %{version}-%{release} %description pskr PSK Reporter data collection and ingestion tools: - pskr-collector: MQTT subscriber for live FT8/FT4/CW/WSPR spots (~300 spots/sec) Writes gzip JSONL to disk with hourly rotation. Forward-only collection from mqtt.pskreporter.info. - pskr-ingest: Incremental JSONL→ClickHouse loader with watermark tracking. Uses pskr.ingest_log for safe cron-based loading. %prep %autosetup -n %{name}-%{version} %build make all VERSION=%{version} %install make install DESTDIR=%{buildroot} PREFIX=%{_prefix} # systemd units. Historically every unit on the fleet was hand-copied into # /etc/systemd/system and owned by no package — `rpm -qf` on wspr-download.service # returns "not owned by any package", so upgrades never touched it and removal would # have orphaned it. Packaging them here starts closing that: the units travel with the # binary they invoke, and a version bump can no longer leave a unit pointing at a flag # the new binary does not accept. install -d -m 0755 %{buildroot}%{_unitdir} install -p -m 0644 systemd/wspr-live-download.service %{buildroot}%{_unitdir}/ install -p -m 0644 systemd/wspr-live-download.timer %{buildroot}%{_unitdir}/ install -p -m 0644 systemd/wspr-live-ingest.service %{buildroot}%{_unitdir}/ install -p -m 0644 systemd/wspr-live-ingest.timer %{buildroot}%{_unitdir}/ %files %license COPYING %doc README.md %dir %{_datadir}/%{name} %dir %{_sysconfdir}/%{name} %{_bindir}/db-validate %files wspr %{_bindir}/wspr-shredder %{_bindir}/wspr-turbo %{_bindir}/wspr-parquet-native %{_bindir}/wspr-download %{_bindir}/wspr-backfill %{_bindir}/wspr-live-download %{_bindir}/wspr-live-ingest %{_unitdir}/wspr-live-download.service %{_unitdir}/wspr-live-download.timer %{_unitdir}/wspr-live-ingest.service %{_unitdir}/wspr-live-ingest.timer %post wspr %systemd_post wspr-live-download.timer wspr-live-ingest.timer %preun wspr %systemd_preun wspr-live-download.timer wspr-live-ingest.timer %postun wspr %systemd_postun_with_restart wspr-live-download.timer wspr-live-ingest.timer %files solar %{_bindir}/solar-ingest %{_bindir}/solar-download %{_bindir}/solar-backfill %{_bindir}/dscovr-ingest %{_bindir}/solar-refresh %{_bindir}/solar-live-update %{_bindir}/solar-history-load %files contest %{_bindir}/contest-download %{_bindir}/contest-ingest %{_bindir}/rbn-download %{_bindir}/rbn-ingest %files pskr %{_bindir}/pskr-collector %{_bindir}/pskr-ingest %changelog * Sat Sep 05 2026 Greg Beam - 4.0.4-1 - dscovr-ingest: migrate to NOAA RTSW endpoints (json/rtsw/rtsw_{mag,wind}_1m.json) - NOAA retired /products/solar-wind/ entirely; ingest had failed since 2026-06-30 - Filter on the `active` spacecraft flag (feed now carries SOLAR1/ACE/IMAP concurrently) - Record originating spacecraft per-row in source_file - Fetch window is now ~24h, not 7d — outages beyond 24h need OMNIWeb backfill * Sun May 17 2026 Greg Beam - 4.0.3-1 - solar-history-load: extend the v4.0.2 NOAA Kp schema fix to the SFI 30-day endpoint, which underwent the same array-of-objects migration but was missed in v4.0.2. Replaces .[1:][]/.[0]/.[1] positional indexing with .time_tag/.flux field access. Also drops the leftover "length - 1" header-row counts for both Kp and SFI display lines. - Discovered by the morning-health-check PCS plugin pilot on its first run — see KI7MT/fleet-ops/plugins/morning-health-check. * Tue Apr 28 2026 Greg Beam - 4.0.2-1 - rbn-download: separate upstream archive gaps (HTTP 404) from real failures; the 11 known historical RBN gap days no longer trip the systemd timer with a nonzero exit code. Gap dates are summarized at the end of the run. - solar-history-load: adapt to new NOAA SWPC Kp JSON schema (array-of-objects with time_tag/Kp/a_running fields) — replaces the old positional indexing that broke the 7-day Kp refresh and produced partial solar.bronze rows. * Wed Feb 25 2026 Greg Beam - 4.0.1-1 - Update README: dataset volumes to 14.34B rows, PSKR to 26M/day * Wed Feb 25 2026 Greg Beam - 4.0.0-2 - Remove Debian packaging (Launchpad cannot build the full stack) * Wed Feb 25 2026 Greg Beam - 4.0.0-1 - Align version across all IONIS packages at 4.0.0 (Phase 4.0 release) * Sat Feb 21 2026 Greg Beam - 3.0.7-1 - Add dscovr-ingest: DSCOVR L1 solar wind ingester (Bz, Bt, speed, density, temp) - Downloads 7-day rolling magnetometer + plasma JSON from NOAA SWPC - Inserts into solar.dscovr via ch-go native protocol with LZ4 compression * Thu Feb 19 2026 Greg Beam - 3.0.6-1 - Add watermark tracking to rbn-ingest, wspr-turbo, and contest-ingest - Add internal/watermark shared package (LoadWatermark, InsertLogEntry, PrimeFiles) - All 3 ingesters now support --full, --prime, and --dry-run flags - wspr-turbo detects cumulative file growth via file_size comparison - Paired download+ingest pattern: every downloader has a partner ingester * Tue Feb 17 2026 Greg Beam - 3.0.4-1 - Add Debian packaging for Launchpad PPA (debian/ directory) - Fix Launchpad build: set GOPATH/GOCACHE to writable dirs - Fix Launchpad build: set GOTOOLCHAIN=local - Fix day-of-week in debian/changelog * Tue Feb 17 2026 Greg Beam - 3.0.3-1 - Add ionis-validate to related repositories in README * Fri Feb 13 2026 Greg Beam - 3.0.2-1 - Fix solar-history-load: temp-table merge prevents ReplacingMergeTree stream loss - Three NOAA streams (X-ray, Kp, SFI) now merged via GROUP BY before bronze insert * Fri Feb 13 2026 Greg Beam - 3.0.1-1 - Add Obsoletes/Provides to wspr, solar, contest subpackages for seamless upgrade - Add pskr subpackage to main package Requires (was missing from install) * Fri Feb 13 2026 Greg Beam - 3.0.0-1 - Rename package: ki7mt-ai-lab-apps → ionis-apps - Move to IONIS-AI GitHub org - Update Go module path: github.com/IONIS-AI/ionis-apps - Require ionis-core >= 3.0.0 - Add Obsoletes/Provides for seamless RPM upgrade * Wed Feb 11 2026 Greg Beam - 2.4.0-1 - V20 production release - Add pskr-ingest: incremental JSONL→ClickHouse loader with watermark tracking - Add pskr-ingest to pskr subpackage - Fix hardware ref: Ryzen 9 9950X3D → Threadripper 9975WX * Tue Feb 10 2026 Greg Beam - 2.3.2-1 - Version sweep: sync all hardcoded versions across scripts and sources - Fix solar-live-update: auto-create Memory table after ClickHouse restart - Add pskr subpackage for pskr-collector binary * Sun Feb 08 2026 Greg Beam - 2.3.1-1 - Medallion architecture: all defaults target 'bronze' table - Update benchmarks: wspr-turbo 22.55 Mrps with 16 workers (production config) - Update README: convert tables to code blocks for COPR, add solar-backfill * Sat Feb 07 2026 Greg Beam - 2.3.0-1 - Add contest subpackage: contest-download, contest-ingest, rbn-download, rbn-ingest - contest-ingest: Cabrillo V2/V3 parser, 225.7M QSOs from 475K files across 15 contests - rbn-ingest: RBN ZIP→CSV→ClickHouse, 2.18B rows in 3m32s (10.32 Mrps) - Align version across all lab packages at 2.3.0 * Wed Feb 04 2026 Greg Beam - 2.2.0-1 - Add solar-backfill: GFZ Potsdam historical SSN/SFI/Kp backfill (1932-present) - Add solar-backfill to RPM solar subpackage - Phase 4.1 SSN Signal Recovery: backfill 25+ years of solar indices * Tue Feb 03 2026 Greg Beam - 2.1.0-1 - Normalize band from frequency in all 5 CSV ingesters via bands.GetBand() - Add internal/bands package: ADIF-MCP v3.1.6 full-spectrum band lookup - Remove legacy Python ingester (source of band column contamination) - Update benchmarks: wspr-turbo 24.67 Mrps, wspr-shredder 21.81 Mrps (9975WX) - Align version across all lab packages at 2.1.0 * Tue Feb 03 2026 Greg Beam - 2.0.8-1 - Fix Silent Sun: solar-download now fetches NOAA SFI summary endpoint - Add 3 new NOAA data sources (SFI live, SFI 30-day, X-ray 7-day) - Add solar-live-update.sh for Now-Casting (15-min cron) - Add solar-history-load.sh for training data (6-hour cron) - Fix solar-live-update to parse correct SFI file with Cycle 25 fallback - solar-download expanded from 7 to 10 data sources * Mon Jan 20 2025 Greg Beam - 2.0.3-1 - Sync version across all lab packages - Fix maintainer email in changelog * Sat Jan 18 2025 Greg Beam - 2.0.0-1 - Major release with high-performance ingestion tools - Add wspr-shredder: 14.4 Mrps uncompressed CSV ingester - Add wspr-turbo: 8.8 Mrps streaming .gz ingester with klauspost/gzip - Add wspr-parquet-native: 8.4 Mrps Parquet ingester - All tools use ch-go native protocol with LZ4 compression - Remove CUDA/GPU dependencies (CPU-only, static binaries) - Update to Go 1.24 * Fri Jan 17 2025 Greg Beam - 1.0.0-1 - Initial package release - Add wspr-ingest: CSV ingestion with dual-path (GPU/CPU) - Add wspr-download: Parallel WSPR archive downloader - Add solar-ingest: NOAA solar flux data ingestion