# PR #20272 — Review Fixes

## Overview

Address review feedback on erigontech/erigon#20272 (`commitment: per-goroutine
ETL collectors for concurrent rebuild`). Four fixes, easiest → most
substantive. The three CI-cost / opt-in Copilot comments are skipped per
author decision. The `prevData`-through-merge "correctness/perf" comment
from Copilot is also skipped: in the rebuild use case (the only enabled
consumer today) the domain is wiped before rebuild, so `GetLatest` on
the merge path hits empty state and is cheap — the fix would add encode/
decode complexity for no real win, and risk subtle history semantics bugs.

Fixes to apply:

1. In `TestGenerateCommitmentRebuildData`: drop the `TEST_LARGE` env var
   entirely (introduced by this PR, not a repo convention) and lean on
   `testing.Short()` as the sole gate — short runs skip, long runs
   always run at full scale. Replace the remaining `os.Getenv("TEST_DATADIR")`
   with `dbg.EnvString` (maintainer nit).
2. Remove the `ParallelProcessCount` global counter entirely, including the
   assertion that consumes it.
3. Drop the unconditional `LvlTrace` override on the per-goroutine ETL
   collector.
4. Bound the unbounded brute-force address search loops in the
   concurrent trie tests with a hard iteration cap + panic.

## Context

Files involved:

- `db/state/squeeze_test.go` — `TestGenerateCommitmentRebuildData` (lines ~540–580).
- `execution/commitment/hex_concurrent_patricia_hashed.go` — `ParallelProcessCount` (lines 18–21) and its `.Add(1)` at line 325.
- `db/state/squeeze_concurrent_rebuild_test.go` — references `ParallelProcessCount` at lines 442, 446, 474–479. The surrounding comment at 437–442 explains the counter's role and must be rewritten.
- `execution/commitment/commitmentdb/commitment_context.go` — per-goroutine collector setup in `concurrentTrieContextFactory` (line 467+), the `LvlTrace` override at line 479.
- `execution/commitment/hex_concurrent_patricia_hashed_test.go` — `findAddressForNibble` (line ~27+, unbounded `for` at line 46) and `findAddressesWithSharedPrefix` (line ~774+, unbounded `for` at 787).

Helpers already present in the repo:

- `common/dbg/dbg_env.go` — `dbg.EnvString(name, default)` and
  `dbg.EnvBool(name, default)`.

## Development Approach

- Regular (code first, then targeted tests).
- One task per commit. Each task must build and its directly affected
  tests must pass before commit.
- Per-task validation uses targeted `go build` and `go test`, not the
  full `make erigon integration` (too slow for per-task iteration).
- A final verification task runs `make lint` (repeat until clean — the
  linter is non-deterministic per the repo's CLAUDE.md) and a wider
  build check.
- Commit-message prefix style from the repo: `package(s): description`.

## Implementation Steps

### Task 1: Drop TEST_LARGE, use -short as sole gate, switch TEST_DATADIR to dbg helper

**Files:**
- Modify: `db/state/squeeze_test.go`

Scope: only `TestGenerateCommitmentRebuildData` and its doc comment.
Do not touch other tests in the file.

Current structure (lines ~538–580):
- `testing.Short()` → skip.
- Default (non-short, no `TEST_LARGE`) → small "smoke" params (1K accounts, 3 steps).
- `TEST_LARGE=true` or `TEST_DATADIR` set → large params (3M accounts, 59 steps).

Target structure:
- `testing.Short()` → skip (unchanged).
- Otherwise → always use the large params. Dead-code the smoke-test path.
- `TEST_DATADIR` still supported as an optional persistent output dir, but
  read via `dbg.EnvString`.

Rationale: `TEST_LARGE` was introduced by this PR and isn't a repo
convention. Erigon already splits work between `-short` and full runs
(`make test-short` vs `make test-all` per the repo CLAUDE.md); the
smoke/large split duplicates that. Collapse onto `testing.Short()`.

- [x] Rewrite the doc comment block (lines ~538–546): drop all mentions of `TEST_LARGE` and the smoke-vs-large distinction. State plainly that the test generates a large-scale dataset, is skipped under `-short`, and that `TEST_DATADIR` may be set to a persistent output directory (default: `t.TempDir()`).
- [x] Keep the `if testing.Short() { t.Skip(...) }` guard unchanged.
- [x] Replace `persistentDir := os.Getenv("TEST_DATADIR")` at line ~552 with `persistentDir := dbg.EnvString("TEST_DATADIR", "")`.
- [x] Delete the small-default `var (...)` block at lines ~566–572 and the `if os.Getenv("TEST_LARGE") == "true" || persistentDir != "" { ... }` wrapper at lines ~574–580. Replace the two blocks with a single unconditional `var (...)` block holding the large parameters: `stepSize=100, totalSteps=59, numAccounts=3_000_000, slotsPerAcct=2, numCodeAccounts=1_000_000`.
- [x] Add the `github.com/erigontech/erigon/common/dbg` import. If `os` becomes unused in the file after this change, remove it; otherwise leave it.
- [x] Build: `go build ./db/state/...`
- [x] Test (short): `go test -run TestGenerateCommitmentRebuildData -short -count=1 ./db/state/...` — must skip cleanly.
- [x] Do NOT run the non-short version during task validation — it's the heavy 3M-account path and will take too long for per-task iteration. Task 5 (final verification) covers the broader sanity run and still uses `-short`.
- [x] Commit: `db/state: simplify TestGenerateCommitmentRebuildData gating`

### Task 2: Remove ParallelProcessCount counter entirely

**Files:**
- Modify: `execution/commitment/hex_concurrent_patricia_hashed.go`
- Modify: `db/state/squeeze_concurrent_rebuild_test.go`

Rationale: the counter is an exported process-global used only by tests,
causing cross-test interference risk and public API bloat. The rebuild
test already verifies correctness via root equality
(`sequentialResult.root == concurrentResult.root`), which is the ground
truth that matters. Observing whether `ParallelHashSort` fired is not
worth the counter.

- [x] In `hex_concurrent_patricia_hashed.go`: remove the doc comment (lines 18–20), the `var ParallelProcessCount atomic.Int64` declaration (line 21), and the `ParallelProcessCount.Add(1)` call in `ConcurrentPatriciaHashed.Process` (around line 325). If `sync/atomic` becomes unused after removal, drop the import.
- [x] In `db/state/squeeze_concurrent_rebuild_test.go`: remove the `commitment.ParallelProcessCount.Store(0)` reset (line 446), and the assertion block + log at lines 474–479 that read `ParallelProcessCount.Load()`.
- [x] Rewrite the explanatory comment at lines 437–442: drop the "We assert this below via ParallelProcessCount" sentence. Keep the surrounding context about `ERIGON_REBUILD_CONCURRENT_COMMITMENT` and what enables the parallel path; just remove the verification claim.
- [x] If the `commitment` import in `squeeze_concurrent_rebuild_test.go` becomes unused after this change, remove it.
- [x] Build: `go build ./execution/commitment/... ./db/state/...`
- [x] Test: `go test -run TestCompareRoots -count=1 ./execution/commitment/...` (quick concurrent-trie tests still pass).
- [x] Commit: `commitment, db/state: drop ParallelProcessCount debug counter`

### Task 3: Drop LvlTrace override on per-goroutine ETL collector

**Files:**
- Modify: `execution/commitment/commitmentdb/commitment_context.go`

- [x] In `concurrentTrieContextFactory`, delete the `collector.LogLvl(log.LvlTrace)` line (around line 479). Keep the `etl.NewCollector(...)` call as-is so the collector uses its default log level.
- [x] If the `log` import becomes unused in this file after removal, drop it; otherwise leave it.
- [x] Build: `go build ./execution/commitment/...`
- [x] Test: `go test -run TestCompareRoots -count=1 ./execution/commitment/...`
- [x] Commit: `commitment: don't force LvlTrace on per-goroutine branch collector`

### Task 4: Bound the brute-force address search loops

**Files:**
- Modify: `execution/commitment/hex_concurrent_patricia_hashed_test.go`

Rationale: both helpers currently use `for { ... counter++ }` which can
hang indefinitely if the search space ever stops producing hits (hash
function change, bad inputs, etc.). Add a deterministic cap; on exceed
`panic` with a descriptive message. Both helpers are free functions
with ~30 call sites; adding a `*testing.T` parameter would be a lot of
churn for a safety cap, and a panic inside a test fails the test with
a useful stack trace.

- [x] Add a package-level `const maxAddrSearchIters = 1 << 20` (1M iterations — plenty of headroom: a nibble hit averages ~16 iters; a 16-bit shared prefix hit averages ~65k, well under the cap).
- [x] In `findAddressForNibble`: bound the `for` loop at line ~46 with a per-call iteration counter; on exceed, `panic(fmt.Sprintf("findAddressForNibble(nibble=%d, seed=%d): exceeded %d iterations", targetNibble, seed, maxAddrSearchIters))`.
- [x] In `findAddressesWithSharedPrefix`: same treatment. Count iterations across the whole `len(results) < count` outer loop, not reset per match. Panic message: `fmt.Sprintf("findAddressesWithSharedPrefix(sharedNibbles=%d, count=%d): exceeded %d iterations after collecting %d", sharedNibbles, count, maxAddrSearchIters, len(results))`.
- [x] Add the `fmt` import if not already present in this file.
- [x] Build: `go build ./execution/commitment/...`
- [x] Test: `go test -run TestCompareRoots -count=1 -timeout 2m ./execution/commitment/...` — ensures tests using both helpers still pass and the cap is not accidentally hit in the green path.
- [x] Commit: `commitment: bound brute-force address search loops in concurrent trie tests`

### Task 5: Final verification

- [x] `go build ./...`
- [x] `make lint` — run up to 3 times, since per the repo CLAUDE.md the linter is non-deterministic. Fix any errors reported; re-run until clean.
- [x] `go test -short -count=1 ./execution/commitment/... ./db/state/...` — broad sanity run on affected packages, short mode.
- [x] Move this plan to `docs/plans/completed/` and commit: `docs: mark PR 20272 review-fixes plan as completed`.
