R2 Track A: effect queue + Router-driven workload + the expected-state model oracle

[nikhilunni]

Wave 2 Track A of the resiliency program — the three R2 rows of ADR 0098 §"Phase 3: the honest-gap program". Three logical commits over crates/engram-dst (+ engram-sim, + one contained coordinator determinism fix).

Per-commit summary

1. The effect queue — in-flight interruption + message faults. Pre-R2 every SimHostClient verb mutated world truth inline, so RPC loss/reorder/duplication and a replica crash BETWEEN the store commit (the ack the coordinator already holds) and the host-side world effect were structurally impossible (audit finding #2/#4). A mutating verb now records its world mutation as a typed Effect. Inline delivery is the default (byte-for-byte the pre-R2 world → Calm seeds unchanged, since Calm never opens a deferred window). When a host is in the deferred set (a Chaos fault window) the effect is queued under a monotonic serial (BTreeMap → deterministic) for a later scheduler step: DeferHost / DeliverEffects (serial order) / DropEffect (loss) / DuplicateEffect / ReorderEffects (seeded shuffle). Crash/restart of a host severs its in-flight effects; quiescence closes every window and flushes in order before the fleet heals. tests/effect_queue.rs pins the two now-reachable states (a deferred create withheld-then-delivered; a replica crash inside the commit→effect window followed by loss) both converging.

2. API-driven workload over the real Router/gRPC surface. workload.rs drives each replica's ACTUAL surface — the tonic AppSessionService/AppFleetService impls and the actual axum api::router via tower::oneshot — and tests/api_surface.rs exercises it deterministically (create acked + persisted + replays byte-identically; prompt+delete round-trip; the axum router runs the real HTTP handlers). SimMeta gains four PG-faithful methods the real handlers reach (were panic-stubs): broker-token insert/get/delete, teleport-target get/set, rebind_session_guarded, list_active_assignments_with_budgets_on_host, with broker-token + teleport conformance scenarios (ADR 0098 D4).

3. The expected-state model oracle (the auditor). TigerBeetle's auditor shape: a model fed ONLY by acked outcomes, diffed against world/SimMeta truth in the standing invariant pass (every step + quiescence). Asserts acked-create/resume-never-silently-lost (the #570 class) and read-your-acked-writes (image never repainted). Non-vacuity proven in tests/model_oracle.rs: drive a session to Active → drop its row directly → auditor FIRES → restore → clears.

real-wire vs handler-direct

verb	path	why
admin pause / resume	real-wire HTTP	a route exists; tower::oneshot on the real api::router runs the bearer middleware + extractors
harness-event ingest	real-wire HTTP	host->coord ingestion route; the harness_idle emitter (#775 trigger's wire)
create_session / send_prompt / resume / delete_session	handler-direct gRPC	ADR 0051 retired the REST create — gRPC-only, no axum route to oneshot. Driven through the real AppSessionService tonic impl (auth + convert.rs + the same *_core) with a constructed tonic::Request carrying the bearer; only the socket/HTTP2 hop is skipped (a live tonic server would break the paused-clock, current-thread determinism the sim rests on)
admin_drain_host	handler-direct gRPC	AppFleetService — the operator drain

Auth reuses the test shape: the app-gRPC BearerAuth is configured with a token every request carries (real auth check runs); the axum internal surface uses the empty-list dev bypass exactly like tests/api.rs::build_app. Neither path bypasses validation inside a handler.

Model-boundary honesty note

The auditor is fed ONLY by acked outcomes (a session observed to reach Active with a durable row). Because of that, a lost-response op is legitimately absent from the model and is never asserted on — a create whose boot never durably established, or an op whose reply dropped, simply isn't a fact the model holds. This is the deliberate TigerBeetle boundary: the model records what the client was TOLD, not what the system attempted.

Seed-shift notes

• Commit 1 re-weights the Chaos pick table (carves 9 points for the effect-queue arms). This shifts every Chaos seed's exploration (seeds pin to a commit, per the D6 precedent). The three pinned chaos seeds (issue_722 @600, seed_96 @1500, issue_762 @5000) were re-verified to still converge and are kept as-is; the wedged_boot pin is hand-driven (not seed-random) and is unaffected. Calm's pick table is untouched by commits 1 & 3, so Calm seeds are unchanged.
• Commits 2 & 3 do NOT touch the swarm pick tables (see the scope note in Findings), so no further seed shift.

Swarm results

Both CI windows, release build, zero failures:

• chaos --seeds 0..200 --steps 1500 -> 200/200 OK
• calm --seeds 0..100 --steps 1500 -> 100/100 OK

The effect-queue faults and the model auditor both hold across the full windows. fmt / clippy -D warnings / nextest on engram-dst + engram-sim + engram-coordinator (383/383) are green; hakari verifies.

Findings (real bugs)

1. Determinism leak — FIXED here (contained). create_session's prepare path (api/sessions.rs) minted the session id via a raw SessionId::new() (Uuid::new_v4) instead of the injected services.entropy — an ADR 0098 D1 violation; the API-create session id diverged every replay. In prod services.entropy is OsEntropy (identical randomness), so behavior-identical there; the fix purely enables deterministic replay.

2. Wire-skew / staging suppression + a driver-double-boot class — FOLLOW-UP (out of scope). Folding the API workload into the swarm requires the sim hosts to be genuinely schedulable (report the current WIRE_VERSION, advertise the seeded image in ready_images) and to produce recoverable snapshots. The pre-R2 sim left hosts wire-skewed and snapshots non-recoverable, which SILENTLY SUPPRESSED the digest-gated placement path the drivers use. Making the hosts faithful unmasks a real class the swarm fires on within a few hundred steps:

   • single-ownership — a session ends up owning 2 live sandboxes across the fleet (ADR 0090 split-brain). Reproduces from the op-path workload ALONE once hosts are schedulable (a driver re-boots a session a create/resume already booted); confirmed independent of the API verbs.
   • placement-accounting — Sigma reserved > allocatable on a host.
   • quiescence-no-stragglers — an operator-drained session stuck at Evacuating.
   • eviction snapshot-safety — a session reaches Idle without a recoverable durable copy under evict->resume cycling.
     These are genuine coordinator (or deep sim-fidelity) findings needing their own investigation; shipping them would red the swarm lane, so this PR drives the real surface deterministically in dedicated tests instead. Recommended follow-up: land the host-fidelity fixes, root-cause the double-boot (suspected: the queue-scanner / OpReclaim orphan-backstop re-booting an already-booted session once candidates are non-empty), then fold the lifecycle-mutating API verbs + #775 harness-idle eviction + operator-drain into the swarm.

Followups

• The double-boot / wire-skew-suppression follow-up above (highest value — a latent ADR 0090 violation the fidelity fixes reveal).
• Extend the auditor with acked-op-result-consistency + completed-run-repaint checks once the API workload is swarm-safe (they need acked op replies the op-path feeding doesn't surface).

🤖 Generated with Claude Code

https://claude.ai/code/session_01PBLK8qJSNJQK722E1n2omR

[nikhilunni]

Orchestrator review: the sessions.rs entropy fix is behavior-identical in prod (OsEntropy) and closes a live D1 leak in a driven path — the exact hole the R1.7b ids.rs proposal predicted. The scope call (dedicated deterministic API tests now; swarm integration held until #787's double-boot is fixed) is the right honesty tradeoff — a red lane teaches nothing. The double-boot finding is filed as #787 with your reproduction notes and marked as the next wave's opener. Merging on green.

🤖 Generated with Claude Code