<!--
SPDX-FileCopyrightText: 2025 Peter Lemenkov <lemenkov@gmail.com>
SPDX-License-Identifier: MIT
-->

# erigon_proxy

A health-checking HTTP/WebSocket reverse proxy and load balancer for a fleet of
[Erigon](https://github.com/erigontech/erigon) Ethereum RPC nodes. It routes
JSON-RPC traffic **only to nodes that are currently at the chain tip**, evicting
nodes that fall behind and re-admitting them once they catch up — so clients see
a single, always-fresh endpoint instead of a set of independently-lagging nodes.

Single file, pure Python + [aiohttp](https://docs.aiohttp.org/), no database, no
external state.

## The problem it solves

Individual archive nodes lag the chain tip unpredictably — a background
compaction stall, a slow import, a brief network blip — and a client that talks
to one node directly will intermittently get stale answers. The usual fix, a
plain round-robin load balancer, is worse: it happily routes to a node that's
50 blocks behind because it only checks that the TCP port is open.

`erigon_proxy` instead tracks every backend's block height on a fast internal
sweep and routes only to nodes at (or acceptably near) the fleet tip. The
interesting part is *what "near the tip" means* — see the health band below.

## How health works

### The health band, not exact-tip

Independent nodes import each new block with a small **stagger**: for a second or
two around every ~12-second slot, the fleet is naturally spread across a few
block numbers. Treating anything below the single fastest node as "unhealthy"
therefore evicts nodes on normal jitter — and produces a constant eviction storm
that gets *worse* on cheaper hardware (wider stagger), even though those nodes
answer RPC perfectly well one or two blocks off the tip.

So health is a **band** around the fleet tip (the max block height observed
across backends), with hysteresis:

- `lag ≤ SOFT_LAG_MAX` → **healthy** (within the propagation band).
- `SOFT_LAG_MAX < lag < HARD_LAG_IMMEDIATE` → **soft lag**: tolerated for up to
  `LAG_GRACE_SECONDS`, then evicted if it persists.
- `lag ≥ HARD_LAG_IMMEDIATE` → **frozen/broken**: evicted immediately.
- A recovering node is re-admitted once it stays **within the band** for
  `READMIT_SECONDS` (not at the exact tip — a moving tip made exact-tip re-admit
  nearly unreachable).
- The healthy pool is never reduced below `MIN_HEALTHY_FLOOR` by soft eviction —
  a slightly-stale node beats an empty pool returning 503s.

Routing always prefers the **freshest** healthy node, so widening the band never
serves stale data when a fresher backend is available; it just keeps
slightly-behind nodes in rotation as warm failover.

### External anchor

Once every `REFERENCE_CHECK_INTERVAL` seconds the proxy makes a single call to an
external reference provider, purely to detect the case where the **whole fleet**
trails the real chain tip (an upstream/reorg situation the internal sweep can't
see, since it measures the fleet against itself). This rate-limited call is kept
off the hot path so the internal sweep can run as fast as desired.

## WebSocket support

The proxy transparently relays WebSocket connections (`eth_subscribe`,
long-lived duplex RPC) to a healthy backend:

- The **freshest healthy node** is chosen at connect time and pinned for the life
  of the connection (a subscription is stateful and can't be re-routed
  mid-stream).
- A periodic heartbeat (WS ping) keeps the connection alive across load-balancer
  idle timeouts and detects dead peers.
- If the pinned backend leaves rotation (lags, is evicted, or the proxy is
  draining), the client is closed so it **reconnects onto a fresh node**. Clients
  are therefore expected to implement reconnect-on-close.

WebSocket upgrades are handled automatically — there is no flag to enable them.

## Installation

### RPM (Fedora / EPEL)

The project ships as an RPM with a systemd unit and a sysconfig file. Build it
from the spec, or install from a [Copr](https://copr.fedorainfracloud.org/)
repository if you publish one:

```sh
rpmbuild -ba erigon_proxy.spec
sudo dnf install ./erigon_proxy-*.noarch.rpm
```

The package installs:

- `/usr/bin/erigon_proxy` — the proxy
- `/usr/lib/systemd/system/erigon_proxy.service` — the service unit
- `/etc/sysconfig/erigon_proxy` — configuration (marked `config(noreplace)`)

Only runtime dependency: `python3-aiohttp`.

### Running directly

```sh
INSTANCES=10.0.0.1,10.0.0.2 REFERENCE_ADDR=example.com \
  SERVER_KEY=... BIND_ADDR=0.0.0.0 ./erigon_proxy
```

## Configuration

All configuration is via environment variables, normally set in
`/etc/sysconfig/erigon_proxy` and read by the systemd unit. `SERVER_KEY` is a
secret, so the file is installed `0600`.

### Required (no defaults — the process will not start without them)

| Variable | Meaning |
| --- | --- |
| `INSTANCES` | Comma-separated backend node IPs; each becomes `http://<ip>:8545`. |
| `REFERENCE_ADDR` | External reference provider host (used only for whole-fleet-lag detection). |
| `SERVER_KEY` | API key for the reference provider. **Secret.** |
| `BIND_ADDR` | Public interface to listen on (`0.0.0.0`, or a specific address). |

### Health band

| Variable | Default | Meaning |
| --- | --- | --- |
| `SOFT_LAG_MAX` | `3` | Blocks behind tip still considered healthy (propagation band). |
| `HARD_LAG_IMMEDIATE` | `8` | Blocks behind → evict immediately (frozen/broken). |
| `LAG_GRACE_SECONDS` | `20` | How long soft lag is tolerated before eviction (keep > one slot). |
| `READMIT_SECONDS` | `5` | How long a node must sit within the band before rejoining. |
| `MAX_FLEET_LAG` | `5` | Fleet tip this far behind the external reference → whole-fleet-lag warning. |
| `MIN_HEALTHY_FLOOR` | `1` | Never soft-evict below this many healthy nodes. |

### Operational

| Variable | Default | Meaning |
| --- | --- | --- |
| `HEALTH_CHECK_INTERVAL` | `1` | Internal sweep cadence (seconds). |
| `HEALTH_CHECK_TIMEOUT` | `2` | Per-node / per-reference call timeout (seconds). |
| `REFERENCE_CHECK_INTERVAL` | `45` | External anchor cadence (seconds). |
| `ETH_GETLOGS_LIMIT` | `75` | Max concurrent `eth_getLogs` in flight (caps backend disk-I/O amplification). |
| `SHUTDOWN_DRAIN_SECONDS` | `40` | On `SIGTERM`, keep serving this long while the LB stops routing (see draining). |

### WebSocket (optional; sane defaults apply if unset)

| Variable | Default | Meaning |
| --- | --- | --- |
| `WS_HEARTBEAT_SECONDS` | `30` | WS ping interval; keeps connections under the LB idle timeout. |
| `WS_HEALTH_POLL_SECONDS` | `5` | How often a WS connection re-checks its pinned backend's health. |

> **Note:** the WebSocket heartbeat is validated at startup against a hard-coded
> `AWS_NLB_IDLE_TIMEOUT_SECONDS` (350s). This ceiling is deliberately **not**
> configurable via environment, so the safety check can't be disabled by
> "raising the limit". If you front the proxy with something other than an AWS
> NLB (a different idle timeout), that constant in the source is the one line to
> change.

## Management API

A separate, **localhost-only** management server runs on `127.0.0.1:8081`:

| Endpoint | Method | Purpose |
| --- | --- | --- |
| `/health` | GET | Full per-backend health, fleet tip, external tip, and counts (JSON). |
| `/loglevel` | POST | Change log level at runtime, e.g. `{"level": "DEBUG"}`. |
| `/servers/add` | POST | Add a backend at runtime, e.g. `{"url": "http://10.0.0.3:8545"}`. |
| `/servers/remove` | POST | Remove a backend by URL. |

The public proxy port additionally serves `GET /_health` as a load-balancer
probe (returns `503` while draining or when no backend is healthy, `200`
otherwise).

```sh
curl -s http://127.0.0.1:8081/health | jq
```

## Logging

The proxy logs on **state change** — evictions, re-admissions, whole-fleet-lag,
and health-count changes — so a quiet log means a stable fleet. Set the level to
`DEBUG` (via `/loglevel`) to see a full per-sweep state dump.

## Draining and restarts

On `SIGTERM` (a systemd stop or package restart) the proxy flips its `/_health`
probe to `503` and keeps serving for `SHUTDOWN_DRAIN_SECONDS` so the load
balancer stops routing new traffic before the process exits. `SIGINT` (Ctrl-C)
skips the drain and exits promptly.

`SHUTDOWN_DRAIN_SECONDS` should be set **longer than the front load balancer's
deregistration delay**, so the balancer has fully stopped routing before the
proxy stops accepting connections.

## Deployment notes

- The proxy is designed to sit behind a TCP load balancer (e.g. an AWS NLB) with
  two or more instances for high availability. Because it's a plain Python
  aiohttp service it is architecture-agnostic (runs on x86-64 and ARM64 alike).
- It expects each backend to serve both HTTP-RPC and WebSocket on the same port
  (Erigon's `--ws` shares the HTTP-RPC port).
- The client's `Host` header is stripped before forwarding, so backends that
  enforce an `http.vhosts` allowlist accept the proxied request.

## License

Licensed under the [MIT License](LICENSE).
This project is [REUSE](https://reuse.software/) compliant.
