feat: freenet-release-agent (Phase 1, dry-run only)

[sanity]

Problem

The current Freenet release pipeline (scripts/release.sh) ssh's into nova and vega as ian to invoke gateway-auto-update.sh --force. That ties the release process to a single workstation's SSH key and gives runners shell on the gateway boxes if we move it to CI. We want a narrow, signed HTTP path instead — see #4073 for the full design.

The peer-side context for why prompt gateway updates matter: peers don't poll GitHub for new releases on a timer. They detect a higher-versioned peer (in practice, the gateway), set a version-mismatch flag, and only then verify against GitHub before exiting with code 42 to be auto-updated. That makes the gateway the version oracle for the whole network — a release that's on crates.io but not on the gateways is invisible to peers (crates/core/src/bin/commands/auto_update.rs:1-14).

Solution

A small Rust HTTP service (freenet-release-agent):

• Runs on each gateway as the dedicated freenet-update system user
• Listens on 127.0.0.1:9876; Caddy fronts it at update.<host>.locut.us with TLS
• GET /version reads freenet --version
• POST /update requires:
  1. Valid HMAC-SHA256 signature over the raw body (X-Signature header)
  2. issued_at within ±5 minutes of agent clock (replay protection)
  3. Requested version exactly matches GitHub's latest release (defense in depth: a leaked token cannot install an arbitrary version)
  4. Requested version > current (no downgrades)
  5. Rate limit: one accepted request per 10 minutes per gateway
• On success in dry-run mode (Phase 1 default): logs the decision via journald, returns 200, never spawns. With dry_run = false the agent shells out to sudo --non-interactive /usr/local/bin/gateway-auto-update.sh --force. sudoers grants NOPASSWD only for that exact command line.

The agent intentionally does not contain any of the update logic itself — it's a thin shim around the existing gateway-auto-update.sh. The systemd timer that polls GitHub every 10 minutes also remains in place as a fallback, so an agent failure degrades to a ~10-minute delay rather than a release-breaking event.

Testing

• 14 unit tests cover HMAC round-trip, tamper rejection, malformed hex, length-mismatch hex, clock-skew bounds, version-string parsing (plain / v-prefixed / pre-release / garbage), TOML config parsing, and dry-run-does-not-execute.
• cargo fmt --check clean, cargo clippy --all-targets -- -D warnings clean.
• The agent has not been deployed anywhere yet — Phase 1 is build-only.

Out of scope (deferred to later phases of #4073)

• Phase 2: deploy on nova in dry-run mode (validates end-to-end auth path via shadow runs against real releases), then flip to live
• Phase 3: deploy on vega + new release.yml workflow that calls both agents + Matrix/River announcements moved to CI
• Phase 4: retire release.sh's SSH stage; the freenet:release skill becomes "trigger workflow_dispatch and watch"

Refs

#4073

[AI-assisted - Claude]

[github-actions]

Now I have enough context to write the review.

Rule Review: Clean with minor style notes

Rules checked: code-style.md, git-workflow.md
Files reviewed: 14 (all Rust sources in crates/release-agent/, scripts/gateway-auto-update.sh, scripts/release-agent/install.sh, .github/workflows/gateway-update.yml, AGENTS.md, docs/release-agent-deployment.md)

Warnings

None.

The diff is clean against the blocking criteria:

• No .unwrap() in production handler paths (update_handler, version_handler, serve, Updater::run, config/secret loading all use ? or explicit match).
• The tokio::spawn in updater.rs (the "still running after 1s" branch) is short-lived and non-critical — it only waits for child.wait(). Fire-and-forget is explicitly permitted for short-lived work.
• No biased; selects.
• No hand-rolled retry loops; no backoff logic at all (rate-limit is a single gate, not a retry).
• No unbounded per-key collections.
• Instant::now() and SystemTime::now() are in crates/release-agent/, not crates/core/, so the TimeSource prohibition does not apply.
• GC/cleanup: the rate-limit check uses a plain Instant with no exemptions — no TTL concern.
• AGENTS.md is updated to document the new crate, keeping docs current.

Info

• crates/release-agent/src/auth.rs:386 — pub fn sign() uses .expect("HMAC accepts any key length") outside #[cfg(test)]. The invariant is correct (HMAC accepts any key length per RFC 2104), and the function is only called from test code; it lives outside #[cfg(test)] by the same design as StaticLatest in github.rs (integration tests compile as a separate crate). Adding #[doc(hidden)] would make the test-helper intent explicit, matching the StaticLatest treatment. (rule: code-style.md — error handling)

• crates/release-agent/src/server.rs — AppState, VersionResponse, UpdateRequest, build_router, and serve are all pub but have no /// doc comments. UpdateResponse does have one. Minor inconsistency since the crate is publish = false, but the rule covers all public APIs. (rule: code-style.md — documentation)

---

Rule review against .claude/rules/. WARNING findings block merge.

[sanity]

Multi-model review responses

Five reviewers (4 internal + Codex CLI) returned 30-ish findings across security, testing, and big-picture alignment. Consolidated punch list categorized into Tier 1 (must-fix in this PR), Tier 2 (cheap quality wins), Tier 3 (follow-ups tracked in #4103).

Tier 1 — fixed in commit a41ce688

#	Finding	Source	Fix
1	Config::dry_run defaulted to false in code while example TOML said true. Partial config silently went live.	big-picture	Defaults to true now
2	Config lacked deny_unknown_fields. dry-run = false (hyphen typo) silently used the default.	code-first	#[serde(deny_unknown_fields)]; test pins the rejection
3	Rate-limit cooldown started before the GitHub check, so a transient 5xx burned the legitimate 10-min retry window.	Codex P1, skeptical CRIT #2, code-first	Moved bookkeeping to after spawn-success. Mutex held across spawn so the failure paths don't consume the window.
4	Updater returned Ok the instant sudo spawned; misconfigured sudoers looked like success while nothing happened.	Codex P1, testing	1s synchronous probe for immediate non-zero exit → surfaced as 500.
5	Signed version not pinned through to the script — race window between agent's GitHub check and script's own fetch.	Codex P2	New --target-version flag on gateway-auto-update.sh; agent passes v{version}. Sudoers expanded to allow the new arg.
6	issued_at: i64::MIN overflowed .abs().	skeptical HIGH #3	Reject non-positive issued_at; checked arithmetic. Tests pin both.
7	Body size unbounded.	skeptical MED #9, code-first	DefaultBodyLimit::max(4096).

Tier 2 — also fixed in a41ce688

#	Finding	Source	Fix
8	HMAC secret hex-vs-raw heuristic could silently halve a raw key whose bytes were all hex digits.	code-first	Always-hex; rejects non-hex with a clear error.
9	HEADER_SIGNATURE as &str not HeaderName.	code-first	HeaderName::from_static.
10	current_version forked freenet --version on every request — self-DoS surface.	skeptical MED #10	VersionCache keyed on binary mtime; test asserts second call doesn't respawn.
11	Updater stdout/stderr → /dev/null, debugging blind spot.	code-first	Stdio::inherit() so output lands in journald via the systemd unit.
12	Dead ReadWritePaths=/var/log/freenet-release-agent in systemd unit.	code-first	Removed. install.sh also no longer creates the unused dir.
13	fetch_latest_version silently excludes pre-releases.	code-first	Now documented on GitHubLatest.
14	no_op: true contract not documented for callers.	code-first	Rustdoc on UpdateResponse + Response-semantics table in README.
15	No integration test of update_handler; LatestSource not mockable.	testing	New LatestSource trait + StaticLatest test double. 11-test integration suite in tests/update_handler.rs covers HMAC / replay / version-pin / no-op / downgrade / rate-limit / version / healthz end-to-end.
Bonus	install.sh doesn't verify gateway-auto-update.sh privilege boundary.	skeptical (script writable by freenet-update would void sudoers)	install.sh now refuses to install if the script isn't owned by root or has perms > 0755.

Tier 3 — tracked in #4103

These need design discussion or are larger than this PR:

• Persistent replay protection (skeptical CRIT Overall architecture RFC #1) — current ±5min skew + in-memory rate limit is replayable across agent restarts.
• Artifact signature verification (skeptical HIGH Peer resource usage balancing #4) — cosign/minisign on release tarballs; current trust model assumes GitHub latest is integrity-authoritative.
• GitHub latest caching (skeptical HIGH Intelligent routing #5) — anonymous API quota DoS vector.
• Status-sentinel file (skeptical HIGH Implement join ring op #8) — last-update outcome readable post-restart.
• HMAC secret via LoadCredentialEncrypted (skeptical HIGH Social credit #6) — group-readable file is a footgun.
• cross-compile.yml entry for the agent (big-picture) — needed before Phase 3.
• AGENTS.md + skill forward-pointer (big-picture) — docs.

Test counts

• Before: 14 unit tests in 4 modules.
• After: 22 unit tests + 11 integration tests (auth/HMAC overflow/non-positive issued_at, config deny_unknown_fields, version cache, full /update pipeline).

Verified after the fix round

cargo fmt --check, cargo clippy --all-targets -- -D warnings, and cargo test -p freenet-release-agent all clean.

[AI-assisted - Claude]

[sanity]

Pushed 098cd06c — pure additive, no behavioral change:

• docs/release-agent-deployment.md — captures the per-gateway install steps install.sh can't automate: DNS A record, AWS SG sizing for vega, secret rotation procedure, dry-run acceptance checklist, troubleshooting table.
• .github/workflows/gateway-update.yml — Phase 2 rollout workflow, workflow_dispatch only. Cannot fire automatically (no release: trigger), fails fast if RELEASE_AGENT_HMAC_NOVA/_VEGA secrets are unset, and supports dry_run_workflow: true to compute+log the signed payload without POSTing. Lands as a draft so it gets reviewed in PR before any production wiring exists.

Both reversible. Phase 1 dry-run on nova is still the next real-machine step — this commit is just text.

[AI-assisted - Claude]

[sanity]

Pushed 31f5a3f6 addressing the multi-model review (4 internal + Codex CLI).

Tier 1 — production-safety (blocking)

Finding	Source	Fix
ProtectSystem=strict blocks the sudo'd child from writing /usr/local/bin/freenet — Phase 2 would have failed	Codex P1	ProtectSystem=off + RestrictNamespaces=false, with comments explaining the privilege boundary is sudoers, not the systemd sandbox
max-parallel at job level instead of strategy.max-parallel — workflow is invalid YAML	Codex P2	Moved into strategy block
Sudoers comment claimed * matches "a single non-whitespace argument" — false per man 5 sudoers	Code-first + Skeptical H2	Fixed comment; the actual safety comes from two independent regex checks (agent's semver::parse + the script's regex)
gateway-auto-update.sh had no defense-in-depth for --target-version	Code-first	Added strict semver regex check after the ${2#v} strip
VersionCache mtime-only key could lose race with cp -p (mtime preserved → stale forever → /version lies post-update)	Skeptical M5 + rule-review WARNING	Added size to cache key + regression test that preserves mtime
Deployment doc claimed "no in-memory cache" for HMAC secret, but main.rs reads at startup	Skeptical L4	Fixed rotation procedure to require systemctl restart
Mutex DoS: held across spawn().await with no timeout — stuck spawn deadlocks all /update	Skeptical H1	tokio::time::timeout(5s) on mutex acquire → 503 on timeout
clock_skew_tolerance_seconds: i64 could be misconfigured negative	Skeptical L3	Changed to u32

Tier 2 — test coverage (high-risk gaps)

Finding	Source	Fix
Spawn-failure-within-1s path uncovered — rate-limit window contract not pinned	Testing high-risk	New spawn_failure_does_not_consume_rate_limit_window test with dry_run=false + exit 1 stub
Sudoers argv shape untested — flag reorders would silently break prod	Testing high-risk	New sudoers_argv_matches_allowlist test with fake-sudo via the new Updater.sudo_command field
4 KiB body limit untested	Code-first	New oversized_body_rejected test

Tier 2 — doc drift

• AGENTS.md: added crates/release-agent/ to Repository Structure
• scripts/release.sh: future-migration comment in trigger_gateway_updates
• gateway-auto-update.sh: usage block documents --target-version
• gateway-update.yml: comment documenting split-version risk on partial-failure rollout

Test totals

37 (23 unit + 14 integration). cargo fmt + cargo clippy --all-targets -D warnings clean.

Not addressed in this PR

Tier 3 follow-ups in #4103: replay nonce store, artifact-signature verification, cross-compile entry, GitHub-latest caching, status sentinel, LoadCredentialEncrypted. Plus from this review round:

• Probe-duration tuning (Skeptical M3)
• Workflow rollback path on partial failure (Skeptical M8)
• StaticLatest cfg-gating (Skeptical L1)
• HMAC secret zeroize (Skeptical L2)

Will append these to #4103.

[AI-assisted - Claude]