ADR 0098 G1: the RPC hang fault — a wedged op reaches terminal via op_deadline

[nikhilunni]

What this is

Closes ADR 0098 §Coverage-gaps G1 (from #743 / session 03e6535e): a resume-class op wedged forever inside its host RPC while the within-step heartbeat kept the op row fresh — stale-op reclaim never fired, and the op pinned for 40 minutes until a pod roll. #743's op_deadline was deliberately put on the tokio timer so the simulator's paused-clock advance could fire it deterministically; the seam was ready, the fault wasn't. This lands the fault.

• SimHostState.rpc_hang: Option<Duration> replaces rpc_partitioned — which turned out to be toggled by Step::RpcPartition and healed in the quiesce block but never read anywhere (the "RPC partition" fault was inert; the real fail-fast path is up=false). Honest repurposing rather than a parallel knob.
• Every SimHostClient verb now calls maybe_hang() at entry: Some(d) sleeps d of virtual time (lock released before the sleep). Above the verb's op_deadline this is the Resume integrity: bound the stall, refuse the NBD corruption, show progress #743 wedge — only the tokio-timer deadline drops the dispatch; below it, a plain delay (the fault menu's "RPC delay" for free). Never future::pending() — an un-timed caller would deadlock the run-step-to-completion scheduler.
• Step::RpcHang(host, bool) arms 3600s (far above every deadline), keeps the old partition weight in the Chaos menu, and is cleared in the quiescence-heal block.
• Sim::execute is now pub so pinned scenarios can hand-drive exact interleavings (the engram-dst-host pattern); Step/DriverKind exported.
• Pinned scenario wedged_boot_op_reaches_terminal_via_op_deadline: heartbeat-register → Placed create (only a Placed create enqueues create_boot — the first draft of this seed passed vacuously without the registration step, caught by strengthening the assertion) → hang both hosts → the SessionOps driver's attempt returns at all only because the deadline drops the hung verb (pre-Resume integrity: bound the stall, refuse the NBD corruption, show progress #743 the step hangs the test forever) → asserts the op was genuinely attempted-and-requeued (attempts ≥ 1, not finished, session not Active) → the healed quiescence pass converges it. no_op_dropped + quiescence-no-stragglers are the standing catch from here on.
• ADR: G1 flipped to CLOSED in §Coverage gaps.

Verification

• cargo nextest run -p engram-dst — 8/8 incl. the new seed and the existing pinned chaos seeds + replay determinism.
• cargo clippy -p engram-dst --all-targets -- -D warnings clean; cargo fmt --all --check clean.
• No host-agent/coordinator production code touched (engram-dst + the ADR only).

Independent of #753 (P5) — disjoint crate closures; no merge-order constraint.

Per the standing P-series policy: leaving this open for your review — no admin-merge.

🤖 Generated with Claude Code