fix(durability): govern snapshot blobs by the pin-set GC model (root-cause fix, Part A)

[nikhilunni]

Root cause, not a fourth patch

#82 (commit-before-destroy + self-heal) and #84 (cross-host blob keys) patched symptoms of one signature: a snapshots row marked recoverable=true whose snapshots/<id>/{state.bin,sidecar} blobs were deleted, bricking resume. The prod test of #84 showed the eviction snapshot a02afad3 re-brick the identical way (a periodic checkpoint aborting the eviction snapshot ~350 ms before its commit). These aren't separate bugs — they're one missing abstraction:

Portable snapshot blobs were the only durable resource class not governed by the pin-set GC model. Chunks (chunk_gc) and bundles (bundle_gc) are "pinned by a live PG row → swept when unreferenced after a grace period, behind the chunk_generation barrier." Snapshot blobs instead used a host-local, single-slot-per-sandbox inflight_snapshots map + abort_prior_inflight_snapshot, which blind-deleted the blobs with no PG check. Every producer (idle eviction, periodic checkpoint, manual snapshot, SIGTERM) shares that one slot, so a concurrent producer's snapshot() aborts another's already-recorded snapshot. #82's commit ordering narrowed but couldn't close it — the race is producer-vs-producer on the shared slot, not ordering against destroy.

The fix (Part A of the plan)

Bring snapshots/<id>/{state.bin,sidecar,working_set} under the same pin-set model — a near-mirror of bundle_gc:

• snapshot_blob_gc sweep + snapshot_blob_gc_candidates (migration 0055), riding the existing gc_sweep_loop tick / chunk_generation barrier / ChunkGcConfig grace; admin dry-run/sweep endpoints. No new env var.
• Pin set = SELECT id FROM snapshots — every row, NO recoverable filter (the resume self-heal transiently demotes rows) and NO session_id filter (template/base captures are session_id NULL, FK'd from enabled_images.base_snapshot_id, with no resume self-heal backstop). Complete because base rows are never deleted (prune_session_snapshots is session_id IS NOT NULL only; the FK has no ON DELETE).
• abort_prior_inflight_snapshot now removes only the LOCAL dir (host-disk hygiene, rebuildable from GCS); the retention sweeper's inline blob.delete (the fix: cross-host resume materialization + orphaned snapshot-blob GC (follow-up to #82) #84 stop-gap) is removed.

Net invariant: nothing but the sweep deletes a durable snapshot blob, and only when no snapshots row references it. recoverable=true is true by construction. This retires the inline abort-delete + the #84 stop-gap rather than adding a fourth patch (net −88/+64 in the refactor commit).

One snapshot-specific adaptation vs. bundle_gc: every capture has an upload-before-record window, so the promote pass re-verifies the pin set at delete time — a candidate whose row has since landed is dropped, never deleted (snapshot ids are fresh UUIDs, so a re-pin can only mean "the row was recorded after we marked it"). Strictly safer than the across-sweep barrier alone.

Scope

This is Part A (durability root cause). Part B — generalizing the cross-host-resume self-heal into the evac/dead-host path so auto-recovery doesn't give up on a bricked latest snapshot — lands separately.

Tests

• snapshot_blob_gc unit (key parse / dedup / malformed).
• snapshot_blob_gc_live_pg (wired into the CI live-PG lane): pin-set includes session and session_id NULL template rows; orphan swept after grace while pinned (incl. template) survive; re-pinned candidate skipped (the upload-before-record safety). All pass against a local Postgres.
• Updated the two pooled_backend snapshot-lifecycle tests to the new contract (abort/retry keep the blobs).
• just check green: fmt + clippy + 968 tests.

See ADR 0028 addendum (2026-06-04 #2).

🤖 Generated with Claude Code