fix(cdf-rebuild): protect interim NPZ + honour CUDA_VISIBLE_DEVICES; streamline README top

[lucapinello]

fix(cdf-rebuild): protect interim NPZ + honour CUDA_VISIBLE_DEVICES; streamline README top

Three changes from the 0.4.0 follow-up triage. Single PR because the HANDOFF.md edits depend on both script fixes.

Closes

• scripts/build_backgrounds_chrombpnet.py: --part variants/baselines --assay X overwrites the interim NPZ #71 / build_backgrounds_chrombpnet: --part variants/--assay X overwrites the existing interim NPZ instead of appending #73 — build_backgrounds_chrombpnet.py: --part variants/baselines --assay X overwrites the existing interim NPZ instead of appending (silent ~10 h GPU loss caught only by post-merge spot-check).
• scripts/build_backgrounds_chrombpnet.py: --gpu N silently overrides outer CUDA_VISIBLE_DEVICES #72 / build_backgrounds_chrombpnet: --gpu N overrides outer CUDA_VISIBLE_DEVICES, breaking parallel-launch pattern #74 — build_backgrounds_chrombpnet.py: --gpu N silently overrides outer CUDA_VISIBLE_DEVICES, breaking the parallel-launch pattern.

#73/#74 are exact duplicates of #71/#72 — opened ~5 min apart by concurrent agent triage. Closing both pairs.

Bug A (P1) — refuse-to-overwrite the interim NPZ

User's pick was the conservative fix: refuse to overwrite without --force, naming the conflicting track-id sets.

• New --force argparse flag.
• New _check_interim_compatibility(interim_path, new_track_ids, force, label) helper, called before each np.savez_compressed(interim_path, ...). Returns silently when the file doesn't exist (first run) or when the existing track set equals the new set (idempotent re-run); otherwise raises SystemExit with a diff (len(only_existing), len(only_new), three example track-ids per side) and a pointer at --part merge / --part merge-incremental / --force.
• Matches existing raise SystemExit("...") validation-error convention in the file (lines 245, 247).

Append-on-write (#71's option (a)) was deferred — better ergonomics but a v0.5+ improvement.

Bug B (P2) — honour pre-set CUDA_VISIBLE_DEVICES

User's pick was the trivial pattern. 3 LOC:

- os.environ["CUDA_VISIBLE_DEVICES"] = str(args.gpu)
+ if "CUDA_VISIBLE_DEVICES" not in os.environ:
+     os.environ["CUDA_VISIBLE_DEVICES"] = str(args.gpu)

Plus a one-line --help clarification on --gpu: "No-op if CUDA_VISIBLE_DEVICES is already set in the calling shell."

When the env var is set, the outer wins — matches every cluster scheduler's precedence convention.

README streamlining

Reader feedback was that the first two sections of "🚀 Get running in one lunch break" were too dense for a first-timer:

• A ~150-word Prerequisite paragraph under Step 1 (Miniforge, ~28 GB disk, both AlphaGenome backends, ChromBPNet slim mirror, lazy downloads, --all-chrombpnet, ENCODE fallback)
• A 5-line Step 2 blockquote duplicating the backend-comparison detail that's already in the deeper #### Two AlphaGenome backends section

Restructure:

Before	After
Step 1 → install commands → 150-word Prerequisite paragraph	Three-bullet "Before you start" mini-section between intro and Step 1 (Miniforge link, ~28 GB, platforms); Step 1 = just install commands
Step 2 blockquote duplicating backend detail	Step 2 = just chorus setup + the existing token / 2-min options
(no canonical home for disk-usage detail)	New #### Disk usage breakdown subsection at the top of Installation — detailed with a per-bucket table

Anchor links (#disk-usage-breakdown, #two-alphagenome-backends, #where-the-oracle-weights-come-from) all resolve. The detail isn't lost — just moved to the deeper section that's already documented as the home for "edge cases: upgrading, per-oracle setup, token plumbing, manual genome management, and backgrounds."

The first screen now reads as: title → tagline → four-step intro → three-bullet "Before you start" → install commands. No prose-walls, no premature backend complexity.

HANDOFF.md (audits/2026-04-29_chrombpnet_cdf_rebuild/)

Two short notes so future maintainers don't re-trip over the same bugs:

• A callout between Phase 1 and Phase 2 reminding to run --part merge (or merge-incremental, or pass --force) before re-running with a different --assay.
• A one-line note in the parallel-launch section explaining the CUDA_VISIBLE_DEVICES/--gpu precedence.

Test plan

• --help parses cleanly on the modified script. Both new flags visible (--force, updated --gpu help-text).
• Fast suite green: pytest tests/ -q -m "not integration and not slow" → 368 passed, 1 skipped, 5 deselected (same count as main — no regression).
• README anchor links resolve: #disk-usage-breakdown → new line 199; #two-alphagenome-backends → existing line 173; #where-the-oracle-weights-come-from → existing line 310.
• No content lost: Other ChromBPNet cell types download lazily and 5–8× faster than JAX CPU distinctive strings appear exactly once each (in their new homes).
• End-to-end --force smoke: would need conda-env infra in CI to actually invoke the script. Helper is a pure function over the existing track-id reservoir code — verified by inspection. Follow-up tracked at the file-level next time someone runs a partial CDF rebuild on an A100 box.

🤖 Generated with Claude Code