Repository navigation
ADR 0098 + ADR 0099: deterministic simulation testing + correctness hardening (Proposed) - #694
Merged
Merged
Conversation
…ardening (Proposed) ADR 0098: a TigerBeetle/FoundationDB-style deterministic simulation harness for the coordinator control plane — Clock/Entropy seam on Services (clippy disallowed-methods enforced), bind-param now() in PostgresStore, a faithful in-memory SimMetadataStore kept honest by a conformance suite against real Postgres, a seeded single-threaded step scheduler driving the real run_once drivers, a fault menu (replica/host crash, partitions, PG outages, clock skew), nine invariants, and a CI swarm lane with seed replay. Phased D1-D7. ADR 0099: the weeks-scale hardening arc — per-test template-cloned Postgres databases (kills the shared-DB cross-talk class and --test-threads=1), proptest with persisted regression seeds, decode-never-panics suites for the guest-facing protos, scripted storage fault injection, ~6 targeted runtime invariants, dispositions for the known flaky tests, and the explicitly-rejected TigerBeetle practices (static allocation, single-threaded loop, zero deps). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Docs-only: two new ADRs, both
Proposed, recording the design for the "TigerBeetle lessons" program before any code lands (ADR-bookend convention).ADR 0098 — Deterministic simulation testing (months-scale arc, PRs D1–D7): a VOPR-style harness for the coordinator control plane.
Clock/Entropytraits onServicesenforced by clippydisallowed-methods; bind-paramnow()in PostgresStore (kills the SQL-now-vs-Rust-now skew class, cf. f97a1c2); a faithful in-memorySimMetadataStorekept honest by a conformance suite run against both it and real Postgres; a hand-rolled seeded step scheduler on paused current-thread tokio driving the realrun_oncedrivers (madsim/turmoil evaluated and rejected); a fault menu (replica/host crash, asymmetric partitions, PG outage windows, clock skew, preemption); nine invariants checked per-step and at quiescence; atest-simCI swarm lane with seed replay and a replay-twice-and-diff determinism self-check.ADR 0099 — Correctness hardening & test isolation (weeks-scale arc, H1–H8): per-test template-cloned Postgres databases via a new dev-only
engram-testkitcrate (makes the shared-DB cross-talk class structurally impossible and retires--test-threads=1on the PG lane); proptest with persisted regression seeds (chunk manifest, wire codecs, mkext4 round-trip); decode-never-panics suites prioritized at the guest→host vsock boundary; scripted storage fault injection (FaultyBlobStorage, durable_record torn-state sweeps); ~6 targetedinvariant!/soft_invariant!sites; dispositions for the known flaky tests (Flaky FC CI test: survivor_reconfigure_resumes_parked_io times out ('parked read must complete after RECONFIGURE: Elapsed') #582, e2e:e2e_claude_with_bogus_key_surfaces_anthropic_auth_errorconsistently failing (180s timeout) #403, two_host_live_teleport); and the explicitly-rejected TigerBeetle practices (static allocation, single-threaded loop, zero deps) with rationale.Why
We fix timing/ordering races reactively — one deflake commit per incident (#582/#598/#629, c8f8ef4, f97a1c2, the torn base-capture). The exploration behind these ADRs found the codebase unusually DST-ready: the entire external world is already behind the
Servicestrait seams, every driver is a purerun_oncebehind a timer loop, and the state machines are pure transition tables. These ADRs turn that into a program.Next
ha_listener) and H2 (remaining ~19 live_pg conversions + drop--test-threads=1) follow as separate PRs.🤖 Generated with Claude Code