R2: the #777 design calls — honest-Dead stage-2 predicate + ask-the-host bound-row policy

[nikhilunni]

Implements the two design calls recorded in #777 (final comment, user decision 2026-07-18), building on the #770 straggler sweep. One PR, two logical commits.

Commit 1 — honest-Dead: unify the HostLost stage-2 recoverability predicate

The two dead-host stage-2 sites (dead_host::evict_host and the #770 host_lost_straggler_sweep) keyed on mere snapshot presence (snapshot.is_some()), while reconcile::flip_missing already keyed on the recoverable flag. A session whose latest snapshot was un-recoverable (its BlobStorage HEAD failed at capture time) settled to a lying Idle a /resume could never honor. Per #777 the recoverable-filtered predicate is now THE stage-2 decision everywhere.

• Shape: reused what flip_missing already calls — latest_snapshot_for_session(...), reading its .recoverable field. No MetadataStore method added or changed (least-churn honest shape). dead_host::recovery_target becomes the ONE shared pub(crate) predicate (keyed on has_recoverable_snapshot OR a live disk manifest), now called by all three stage-2 sites. That also closes flip_missing's reverse lie — it ignored the live disk manifest, so a disk-recoverable session could be routed to Dead.
• Bad-capture signal: the "snapshot rows exist but none recoverable ⇒ Dead" arm gets a distinct warn! + new counter HOST_LOST_UNRECOVERABLE_SNAPSHOT_TOTAL (mirrors R0.3a: fix HostLost straggler (nightly seed 33043259, #762) #770's HOST_LOST_STRAGGLERS_SETTLED_TOTAL) at every stage-2 site.
• D4 conformance: new latest_snapshot_reports_recoverable_flag scenario pins, across BOTH stores, that latest_snapshot_for_session returns the newest row and its recoverable flag faithfully — the store foundation the predicate rests on.
• Red-then-green: engram-dst::host_lost_policy::unrecoverable_only_straggler_settles_dead_not_idle — a bound HostLost straggler whose only snapshot is recoverable=false settles Dead. Verified red first: with the pre-HostLost stage-2 recoverability predicate divergence: dead_host settles on ANY snapshot row, reconcile filters recoverable #777 snapshot.is_some() predicate it settled Idle (left: Idle, right: Dead), green after.

Commit 2 — bound rows: ask-the-host before destroying

The #770 sweep best-effort destroyed a still-bound sandbox after the 60s min-age. But a HostLost row can carry a live VM — a >60s heartbeat partition / binding desync parks the session at HostLost while the VMM is still serving (#776-review concern). Destroying it kills a live VM and rewinds resume to the last checkpoint.

• Host truth: the running-sandbox set is not persisted in PG (only a count + last_heartbeat_at — investigated host_http.rs heartbeat handler + the hosts/sandbox tables). So, per HostLost stage-2 recoverability predicate divergence: dead_host settles on ANY snapshot row, reconcile filters recoverable #777's fallback, the sweep uses the existing HostClient probe reconcile's ADR 0068 belt already uses — probe_sandbox → process_alive. No new RPC.
• Policy: process_alive ⇒ DEFER destroy+settle for the reattach machinery and bank a per-session serving-strike (HOST_LOST_STRAGGLER_DEFERRED_SERVING_TOTAL). Strikes ≥ straggler_serving_strike_cap (default 3), or a failed probe / not-alive process / host already gone from the registry ⇒ destroy+settle as before.
• Convergence (oracle chore(deps)(deps): bump toml from 0.8.23 to 1.1.2+spec-1.1.0 #8's shape): every arm terminates in a settle within bounded cycles — the serving-defer removes the destroy-a-live-VM window without reintroducing the nightly sim: quiescence-no-stragglers violated #762/P7 dead-plane gate fired in prod 2026-07-18: survivor left rootfs-less after roll, evict_local->resume remediation loops, session stuck host_lost #769 eternal wedge. Documented in the sweep doc comment.
• Strikes storage: a per-replica in-memory StragglerStrikeMap threaded through run_once, mirroring the existing ProbeMemory precedent (no natural PG column; per-pod backstop; a settle by ANY replica ends the deferral for all via the Postgres/coordinator: assign_session_host/assign_session_sandbox/rebind_session are unguarded blind UPDATEs — racing actors bind live sandboxes onto terminal rows and defeat the orphan reap #211 CAS + Conflict-idempotent transition; pruned to the live HostLost set each cycle). engram-dst threads + clears it exactly like probe_memory.
• Red-then-green: engram-dst::host_lost_policy::serving_straggler_is_deferred_then_settles_at_the_strike_cap — a bound HostLost straggler with a live world-side sandbox survives the first two sweeps (row HostLost, VM alive) and settles Idle on the third (cap reached, VM destroyed). Verified red first: with the pre-HostLost stage-2 recoverability predicate divergence: dead_host settles on ANY snapshot row, reconcile filters recoverable #777 best-effort destroy the first cycle killed the VM and settled (left: Idle, right: HostLost), green after.

Verification

• cargo fmt --check + cargo clippy --all-targets -- -D warnings on engram-coordinator / engram-sim / engram-dst: clean.
• cargo nextest run for those three crates: 421 passed, incl. both new red-then-green proofs and the pinned nightly sim: quiescence-no-stragglers violated #762 straggler seed (chaos 5000) still green.
• Conformance across BOTH stores against the reachable dev Postgres (localhost:5435): t_latest_snapshot_reports_recoverable_flag::{sim,pg} + the neighbouring host-lost / snapshot scenarios all pass.
• Swarm sanity 0..100 chaos × 1500: all 100 seeds converged, 0 regressions.

Refs #777, #770.

[nikhilunni]

Orchestrator review: rebased onto current main (clean), verified 424/424 + fmt + clippy post-rebase, and swapped StragglerStrikeMap to BTreeMap (its only iteration is the prune, so no live leak — but BTreeMap is the discipline in decision-adjacent paths). The commit-1 reverse-lie fix (flip_missing ignoring live disk manifests) is a genuine catch beyond the #777 scope. Merging on green.

🤖 Generated with Claude Code