audit: rebuild ChromBPNet CDFs against chrombpnet_nobias (0.3.0+ default)

[lucapinello]

Summary

Built a fresh `chrombpnet_pertrack.npz` (786 tracks, 78.6 MB, sha256 `be61e9e8...`) on Linux/CUDA (2× A100 80 GB) following the handoff at `audits/2026-04-29_chrombpnet_cdf_rebuild/HANDOFF.md`. NPZ validated and ready for upload to `lucapinello/chorus-backgrounds`; upload itself is blocked on a write-scope HF token (the cached read-only token returned 403).

Headline numbers

Check	Result
Track count	786 (22 ATAC + 20 DNASE + 744 CHIP) — matches handoff's expected count
Effect / summary / perbin reservoirs	full at 9609 / 29004 / 928128 — zero failed model loads
All CDFs monotone (786 rows × 3 CDFs)	✅
NaN / Inf in any CDF	✅ none
Wall total	~10 h with parallel-GPU phases (vs handoff estimate 13-25 h)

Magnitude shift vs. 0.2.x CDFs (matches handoff prediction of 5-30 %)

Track	old p95	new p95	shift %
ATAC:K562	0.1621	0.1976	+21.9%
DNASE:HepG2	0.1884	0.2436	+29.3%
ATAC:HepG2	0.1332	0.1517	+13.8%
ATAC:GM12878	0.1692	0.1934	+14.3%
DNASE:K562	0.1462	0.1659	+13.5%
CHIP:K562:REST	0.0529	0.0529	0.0%
CHIP:HepG2:CTCF	0.0829	0.0828	−0.1%

ATAC/DNase shifts reflect the bias correction (`chrombpnet_nobias` strips the enzymatic motif preferences the bias-aware variant carried). CHIP/BPNet tracks unchanged because BPNet's catalogue is already nobias-equivalent.

What's blocking the upload

The cached HF token in this env (`HfApi().whoami() → lucapinello, auths: []`) is read-only:

```
huggingface_hub.errors.HfHubHTTPError: 403 Forbidden: you must use a write token to upload to a repository.
```

To finish the rebuild: maintainer re-logs in with a write-scope token (or sets `HF_TOKEN` to one) and runs the upload command from `report.md` "Upload status". The NPZ at `~/.chorus/backgrounds/chrombpnet_pertrack.npz` is unmodified since the merge step — same SHA256 as recorded in the report.

Two script bugs surfaced (worth filing as follow-ups; recoverable)

1. `--part variants/baselines --assay X` overwrites the interim NPZ. Following the handoff's documented sequence (`--assay ATAC_DNASE` then `--assay CHIP`) silently lost the 42 ATAC/DNase tracks during the merge — the second invocation replaced the interim file rather than appending. Caught by the post-merge spot-check (744 ≠ 786). Recovered via a Phase 1 redo (~50 min) + `--part merge-incremental`. The script should either append-on-write or refuse to overwrite without `--force`.
2. `--gpu N` silently overrides outer `CUDA_VISIBLE_DEVICES`. `scripts/build_backgrounds_chrombpnet.py:200` does `os.environ["CUDA_VISIBLE_DEVICES"] = str(args.gpu)` unconditionally, so the parallel-launch pattern `CUDA_VISIBLE_DEVICES=1 ... --gpu 0` puts the second job on physical GPU 0 instead of 1. Hit during the Phase 1 redo and OOM'd. The script should respect a pre-set `CUDA_VISIBLE_DEVICES` if present and only set it from `--gpu N` when no env var was provided.

Files

• `audits/2026-04-29_chrombpnet_cdf_rebuild/report.md` — full audit narrative
• 9 build logs (`bg_chrombpnet_*.log`) — per-phase, including the failed first-redo for evidence
• The NPZ itself is not in this PR (78 MB) — sha256 captured in the report so the maintainer can verify post-upload

Test plan

• Maintainer authenticates with a write-scope HF token and runs the upload command in `report.md`
• After upload, verify the NPZ on HF has sha256 `be61e9e8f9b919b43c599b7fbc9deb74f8f1e6dc1da5e2cdb92036a85bf13205`
• Spot-check that `chorus.analysis.normalization.get_pertrack_normalizer("chrombpnet")` picks up the new NPZ (auto-download from HF)
• Ranking sanity: pick one large-effect SNP at a known regulatory element (e.g. SORT1 rs12740374) and confirm relative cell-type rankings preserved between old and new CDFs

🤖 Generated with Claude Code