R-CoSim rung 1: engram-dst-cosim — the coordinator↔host boundary simulator (+ the #570 window)

[nikhilunni]

What this is

ADR 0098 Phase 3 R-CoSim, rung 1: the residual-risk item #2 the ADR named but left unbuilt — the coordinator↔host boundary is not co-simulated. engram-dst drives the real coordinator over engram-sim; engram-dst-host drives the real host-agent flows over its own world; the two are disjoint by construction (separate cargo closures, separate clocks), so the boundary where #570, #739, #602, #216, and 85e0298a all lived is explored by neither. This adds crates/engram-dst-cosim: both real sides wired against one shared SimClock + SimMetadataStore, only the transport faked.

Real vs faked, each side

	Real	Faked
Coordinator → host	full AppState/Services replica over the shared SimMetadataStore + SimClock/SimEntropy; real op verbs (create_boot, evict, resume); real idle_detector/queue_scanner	CosimHostClient — the HostClient verbs drive CosimHost's real ops
Host → coordinator	CosimHost running the REAL EvictionFinalizer capture + run_eviction_finalize_attempt finalize over a real ChunkedDiskBackend/ChunkStore, and the REAL reconcile_once	CosimCoordControlPlane — the 3 CoordControlPlane calls invoke the REAL coordinator handler cores (extracted to pub fns in host_http)

The "wire" is a direct function call into the real handler logic — that is the fidelity rung 1 buys over the toy SimCoordClient (a BTreeMap).

Standing oracle (assert_idle_snapshot_durable): the coordinator reaching Idle via eviction implies the snapshot its resume path selects is durable and at-or-above the eviction cursor — exactly what #570 violates.

The #570 verdict: REPRODUCED, and fixed

Reproduced. tests/unbind_vs_teardown_reconcile.rs drives the real D5 fast path: snapshot_begin returns (capture lock held, finalize in flight) → coordinator marks Idle + clears sessions.sandbox_id. The real reconcile_once then lists the still-resident paused VM, asks the real sandbox_ownership core, gets false (D5 unbind), strikes twice, and destroys the VM mid-upload — cancelling the finalize. No snapshot row lands; the oracle fires (resume would rewind to the stale cursor-3 periodic checkpoint, below the cursor-5 eviction). Red-then-green is a single adversarial knob, honor_capture_signal, replaying the pre-fix reconcile that never consulted the capture-in-flight signal.

Fixed (separate commit, host-side, true to "production drives the design"). The host already has the signal — the periodic checkpointer skips a sandbox whose capture_in_flight (capture lock held) is true; the reconcile ownership check just never consulted it. The fix adds capture_in_flight to the ReconcileBackend trait and to gather_input's exemption set, implemented on PooledReconcileBackend via the existing PooledBackend::capture_in_flight. GREEN: the mid-capture VM is exempt, the finalize completes, the durable snapshot lands at the eviction cursor, the oracle holds. The exemption is scoped — a genuinely-unowned sandbox with no capture in flight is still reaped (unit tests near the fix pin both).

Not changed-since: the D5 fast path + teardown-reconcile still exhibit the exact window described in the issue.

Scenarios

• smoke — create → serve → idle-evict → resume, all real code.
• unbind_vs_teardown_reconcile — the Idle-eviction snapshots silently destroyed mid-upload (D5 unbind vs teardown-reconcile race) — resume rewinds transcript to stale checkpoint, repaints successful runs as failed #570 red/green (above).
• publish_committed_but_ack_lost — a lost publish ACK is safe: the retry is idempotent w.r.t. the durable manifest pointer, and a publish from a departed binding is dropped Stale, not applied.
• postroll_rehydrate_vs_sweep_vs_unpause — the ADR 0090 survivor-reattach ownership leg of the 731df805/fix: parked survivors were invisible to the register-time NBD rehydrate (session 731df805 RCA) #739/P7 dead-plane gate fired in prod 2026-07-18: survivor left rootfs-less after roll, evict_local->resume remediation loops, session stuck host_lost #769 family: a HostLost survivor with a dropped local binding is repaired (not reaped); a terminal session's leftover VM is still reaped. Pins the HostLost stage-2 recoverability predicate divergence: dead_host settles on ANY snapshot row, reconcile filters recoverable #777 tension's current behavior (repair) with the open design call noted.

CI

New test-cosim nextest lane (directed scenarios, not a swarm binary), gated on the crate's own spanning release closure via the detector's new test_cosim flag, added to the CI Gate's needs: — never individually required (AGENTS.md). The PR's own coordinator + host-agent changes trip the flag, so the lane runs here. YAML validated with yaml.safe_load; just hakari verify clean.

Honest gaps / what rung 2 needs

• The NBD/generation/park/un-pause data-plane model stays engram-dst-host's SimHost domain — rung 1's boundary host reuses the finalize/reconcile flows keyed by the coordinator's id-space, not the slot machinery. The full postroll family (register-rehydrate ↔ stale-sweep ↔ un-pause gate ↔ the coordinator straggler sweep, together) is rung 2.
• Divergence from the ADR plan (documented in code): SimHost::new owns a fixed slot id-space and assumes it owns the session/sandbox ids; at this boundary the coordinator mints them, so we reuse the real extracted flows SimHost is built on rather than SimHost itself.
• A constraint the code forced: the coordinator's session_ops::enqueue detaches op drives onto tokio::spawn (benign for engram-dst's zero-I/O SimHostClient, which settles in one poll, but the real chunk-store/finalize I/O starves in a background task under a directed paused-clock harness). The harness drives ops inline via enqueue_claim + drive_claimed — same verb body, drained deterministically on-task. This is where the "directed, not swarm" shape of rung 1 shows.
• Rung 1 is directed, not a swarm — no random interleaving generator yet. A seeded boundary swarm (with the state-space product deliberately bounded) is the natural rung-2/3 increment.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PBLK8qJSNJQK722E1n2omR

[github-actions]

The latest Buf updates on your PR. Results from workflow CI / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	⏩ skipped	⏩ skipped	✅ passed	Jul 18, 2026, 7:39 PM

[nikhilunni]

Orchestrator review: the #570 fix is the narrowest true shape — the reconcile ownership check now consults the same capture-lock signal the periodic checkpointer always used, as a gather_input exemption alongside migration-role/live-capture; the host-internal sim's honest 'always false' impl correctly documents why this race is boundary-only territory. Architecture divergence (real extracted flows keyed by coordinator ids, not SimHost wholesale) is well-reasoned. One follow-up assertion wanted (not blocking): a unit test pinning that a QUARANTINED finalize releases the capture lock — the exemption must never become a permanent reap-shield; today that's implied by the finalize convergence oracle, not pinned directly. Merging on green.

🤖 Generated with Claude Code