Skip to content

Substrate single-writer chunk cache + engram-substrated ADR (ADR 0069/0070, stacked on #581) - #583

Merged
nikhilunni merged 1 commit into
mainfrom
epic/547-single-writer
Jul 7, 2026
Merged

Substrate single-writer chunk cache + engram-substrated ADR (ADR 0069/0070, stacked on #581)#583
nikhilunni merged 1 commit into
mainfrom
epic/547-single-writer

Conversation

@nikhilunni

@nikhilunni nikhilunni commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Interim single-writer chunk-cache discipline + the engram-substrated design ADR (#547). Stacked on #581 (which stacks on #580) — merge #580 → #581 → this. This PR's own delta is its last three commits (the two ADRs, the single-writer implementation).

What this does

The NVMe chunk-cache directory had 1 + N writers per host (host-agent + one uffd handler per Uffd sandbox), each with its own in-memory singleflight / pin set / evictor over shared bytes — the structural root of the #437 .partial race, the pin-invisible eviction hazard (a handler sweep evicting host-agent-pinned base chunks), duplicate GCS fetches, and 583+55/7d ENOENTs. This makes exactly one process the writer, enforced at the type level:

  • engram-substrate-proto (new crate): sync bincode framing + SCM_RIGHTS fd passing, no tokio/tonic (fault-path adjacency — the peer.rs discipline).
  • ChunkCacheReader: open/read/contains only — a client that compiles cannot mutate the directory.
  • Host-agent populate server: Hello answers live probes (statfs TMPFS_MAGIC + cache-writability), Populate runs pin → cache.get → open → send-fd → unpin against the one cache instance, so the global singleflight / pins / Hard host disk budget for the chunk cache: periodic enforcement, single evictor, pin-aware arithmetic, dedicated volume #528 budget govern handler traffic. Phase 2 pins the session manifest for the connection (= sandbox) lifetime.
  • Handler: 3-step read path — resident → populate-via-writer → last-resort direct blob served from memory (never written to the cache dir) when the writer is mid-roll (ADR 0044 K2). Readiness Hello runs before the FC-facing UDS binds, so an unready host fails at spawn instead of booting into register memory … userfaultfd.

The daemon is ADR-only (gated)

ADR 0070 (engram-substrated) is delivered as a design document with the crash-story as an explicit gate: a daemon crash closes every uffd → zero-fill guest corruption, and pidfd_getfd needs a live donor. No implementation issue may be filed until fd-retention is chosen + prototyped. The interim protocol/reader/handshake port to the daemon unchanged.

Tests

proto framing + socketpair fd round-trip; reader hit/miss; server singleflight-collapse + Hello probes; pinned-chunk-survives-populate-pressure (unrepresentable pre-0069); degraded-mode fallback-writes-nothing; substrate_populate real-client↔real-server integration (Linux-gated fd passing — wire the new FC-lane test into ci.yml's --test list at merge; the socket-contract property is covered cross-platform, the VM path is dev-vm-validated per the CI test-sizing rule). just check + full-workspace musl clippy green.

VZ/Process

N/A by construction (no out-of-process cache clients off FC/Linux) — the host-agent's in-process cache is already the sole writer there; stated explicitly in ADR 0069.

Note

Includes a small fix(linux-tests) commit (ADR 0067 fallout in cfg(linux)-gated FC test literals, caught by the musl cross-check) that is cherry-picked down to #580 so its FC lane heals; it appears here only because of the stack.

🤖 Generated with Claude Code

https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx

@nikhilunni nikhilunni mentioned this pull request Jul 6, 2026
Merged
@nikhilunni
nikhilunni force-pushed the epic/545-rung1-cancel-evict branch from 8f9e81d to b7f4185 Compare July 6, 2026 13:29
@nikhilunni
nikhilunni force-pushed the epic/547-single-writer branch from 3a20ee9 to b5c653d Compare July 6, 2026 13:30
@nikhilunni
nikhilunni changed the base branch from epic/545-rung1-cancel-evict to epic/545-rungs-2-3 July 6, 2026 13:30
@nikhilunni
nikhilunni force-pushed the epic/545-rungs-2-3 branch 2 times, most recently from a7064cd to a558879 Compare July 6, 2026 22:06
Base automatically changed from epic/545-rungs-2-3 to main July 6, 2026 22:22
@nikhilunni
nikhilunni force-pushed the epic/547-single-writer branch from b5c653d to f845653 Compare July 7, 2026 03:09
nikhilunni added a commit that referenced this pull request Jul 7, 2026
@nikhilunni @claude
feat(substrate): ADR 0075/0076 — interim single-writer chunk-cache + …
f845653 
…substrated design (#547)

Rebased onto main past #585/#590-#596; ADRs renumbered 0072/0073 ->
0075/0076 (main's 0072 = substrate-survey-evidence, 0073 =
binding-epoch). See PR #583 for the full description.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
@nikhilunni
nikhilunni marked this pull request as ready for review July 7, 2026 03:09
@nikhilunni
nikhilunni force-pushed the epic/547-single-writer branch from f845653 to 68375d6 Compare July 7, 2026 03:24
nikhilunni added a commit that referenced this pull request Jul 7, 2026
@nikhilunni @claude
feat(substrate): ADR 0075/0076 — interim single-writer chunk-cache + …
68375d6 
…substrated design (#547)

Rebased onto main past #585/#590-#596; ADRs renumbered 0072/0073 ->
0075/0076 (main's 0072 = substrate-survey-evidence, 0073 =
binding-epoch). See PR #583 for the full description.

Review fix folded in: the client replays `Hello` on every fresh dial
(with_conn), not just at startup — the server's session-chunk pin set
and proto check are per-connection state, and the designed-for
reconnect (a rolled host-agent's successor, holding an empty pin set)
is exactly when silently skipping the replay would leave the handler
unpinned for the rest of the sandbox's life.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
@nikhilunni @claude
feat(substrate): ADR 0075/0076 — interim single-writer chunk-cache + …
5bee563 
…substrated design (#547)

Rebased onto main past #585/#590-#596; ADRs renumbered 0072/0073 ->
0075/0076 (main's 0072 = substrate-survey-evidence, 0073 =
binding-epoch). See PR #583 for the full description.

Review fix folded in: the client replays `Hello` on every fresh dial
(with_conn), not just at startup — the server's session-chunk pin set
and proto check are per-connection state, and the designed-for
reconnect (a rolled host-agent's successor, holding an empty pin set)
is exactly when silently skipping the replay would leave the handler
unpinned for the rest of the sandbox's life.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
@nikhilunni
nikhilunni force-pushed the epic/547-single-writer branch from 68375d6 to 5bee563 Compare July 7, 2026 03:39
@nikhilunni
nikhilunni merged commit 43c87b9 into main Jul 7, 2026
18 checks passed
@nikhilunni
nikhilunni deleted the epic/547-single-writer branch July 7, 2026 03:54
nikhilunni added a commit that referenced this pull request Jul 7, 2026
@nikhilunni @claude
feat(snapshots): ADR 0077 — transactional snapshots, durable_head, Ru…
630cf00 
…ntimeSpec, Revive (#544)

Rebased onto main past #583/#585/#590-#596. Renumbered at land:
ADR 0074 -> 0077 (main's 0074 = parking ladder); migrations
0087-0089 -> 0088-0090 (main's applied high-water = 0087). The
queue_prompt column references died in the rebase (#592 removed it).

Five pre-merge review findings fixed (see the ADR's divergence log):
- get_session projection: the selected_skills removal left a missing
  comma aliasing live_disk_manifest_version AS park_rung — every
  session read un-parked (frozen-VM-advertised-Active on the rung-2
  ascent) and live_disk_manifest read None. Live-PG round-trip test
  pins the projection now.
- fork-at-attach minted an UNPUBLISHED (uuid, v0) into manifest_ref;
  sessions evicted before their first flush were unevictable and
  zero-dirty captures unresumable. Split: fork_identity is adopted at
  v1 by the FIRST publish; manifest_ref stays the resolvable base
  until then (+ regression test).
- durable_head advanced on recoverable=false rows; now gated, and a
  demote of the current head re-points to the newest recoverable.
- prepare_from_row swallowed RuntimeSpec read errors into "no skills";
  now propagates (scanner/resume retry is the recovery).
- the Idle->Created harness-failed park emitted no StatusChanged and
  swallowed transition failures; now emits + propagates.

Accepted (zero users): no selected_skills backfill into
session_runtime_specs; pre-deploy sessions lose their skill selection
on the next re-prepare.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
nikhilunni added a commit that referenced this pull request Jul 7, 2026
@nikhilunni @claude
feat(snapshots): ADR 0077 — transactional snapshots, durable_head, Ru…
3ee7c6e 
…ntimeSpec, Revive (#544)

Rebased onto main past #583/#585/#590-#596. Renumbered at land:
ADR 0074 -> 0077 (main's 0074 = parking ladder); migrations
0087-0089 -> 0088-0090 (main's applied high-water = 0087). The
queue_prompt column references died in the rebase (#592 removed it).

Five pre-merge review findings fixed (see the ADR's divergence log):
- get_session projection: the selected_skills removal left a missing
  comma aliasing live_disk_manifest_version AS park_rung — every
  session read un-parked (frozen-VM-advertised-Active on the rung-2
  ascent) and live_disk_manifest read None. Live-PG round-trip test
  pins the projection now.
- fork-at-attach minted an UNPUBLISHED (uuid, v0) into manifest_ref;
  sessions evicted before their first flush were unevictable and
  zero-dirty captures unresumable. Split: fork_identity is adopted at
  v1 by the FIRST publish; manifest_ref stays the resolvable base
  until then (+ regression test).
- durable_head advanced on recoverable=false rows; now gated, and a
  demote of the current head re-points to the newest recoverable.
- prepare_from_row swallowed RuntimeSpec read errors into "no skills";
  now propagates (scanner/resume retry is the recovery).
- the Idle->Created harness-failed park emitted no StatusChanged and
  swallowed transition failures; now emits + propagates.

Accepted (zero users): no selected_skills backfill into
session_runtime_specs; pre-deploy sessions lose their skill selection
on the next re-prepare.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
@nikhilunni nikhilunni mentioned this pull request Jul 7, 2026
Merged
nikhilunni added a commit that referenced this pull request Jul 7, 2026
@nikhilunni @claude
fix(harness): a Superseded exit must actually exit — drop cmd_tx so t…
6738343 
…he engine terminates (#597)

The ADR 0073 Superseded rejection arm logged "exiting cleanly" and broke
the connection loop — but main holds a cmd_tx clone for the whole
process, so `engine.await` never completed: the harness lingered ALIVE
with its connection loop (and SIGUSR1 handler) dead. agentd's
SpawnHarness reattach arm then saw a live pid on every subsequent
start_agent, nudged it forever, and never respawned a fresh harness —
so every resume from a live-harness checkpoint (evict_local, and the
ADR 0028 host-loss recovery path) wedged prompt delivery permanently:
the outbox looped NotFound -> "harness reattach issued" indefinitely
(prod session 7ed23d9f, 2026-07-06; the #583 verification caught it).

Fix: drop cmd_tx before `engine.await` — the engine's designed
ChannelClosed teardown ("all command senders gone = the connection loop
exited = process teardown") fires, the process exits, agentd reaps it,
and the next SpawnHarness spawns a FRESH harness with the current
binding epoch. The ChannelClosed arm now also SIGINTs claude (it skips
the reap block, and an orphaned twin would contend with the successor's
--resume on the same transcript; claude flushes per-message, so SIGINT
is resume-safe).

The idle-evict path never hit this (its drain produces claude-free
snapshots); only checkpoints capturing a LIVE harness restore into the
stale-epoch -> Superseded -> phantom-exit shape.


Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
nikhilunni added a commit that referenced this pull request Jul 7, 2026
@nikhilunni @claude
feat(snapshots): ADR 0077 — transactional snapshots, durable_head, Ru…
6a37340 
…ntimeSpec, Revive (#544) (#584)

Rebased onto main past #583/#585/#590-#596. Renumbered at land:
ADR 0074 -> 0077 (main's 0074 = parking ladder); migrations
0087-0089 -> 0088-0090 (main's applied high-water = 0087). The
queue_prompt column references died in the rebase (#592 removed it).

Five pre-merge review findings fixed (see the ADR's divergence log):
- get_session projection: the selected_skills removal left a missing
  comma aliasing live_disk_manifest_version AS park_rung — every
  session read un-parked (frozen-VM-advertised-Active on the rung-2
  ascent) and live_disk_manifest read None. Live-PG round-trip test
  pins the projection now.
- fork-at-attach minted an UNPUBLISHED (uuid, v0) into manifest_ref;
  sessions evicted before their first flush were unevictable and
  zero-dirty captures unresumable. Split: fork_identity is adopted at
  v1 by the FIRST publish; manifest_ref stays the resolvable base
  until then (+ regression test).
- durable_head advanced on recoverable=false rows; now gated, and a
  demote of the current head re-points to the newest recoverable.
- prepare_from_row swallowed RuntimeSpec read errors into "no skills";
  now propagates (scanner/resume retry is the recovery).
- the Idle->Created harness-failed park emitted no StatusChanged and
  swallowed transition failures; now emits + propagates.

Accepted (zero users): no selected_skills backfill into
session_runtime_specs; pre-deploy sessions lose their skill selection
on the next re-prepare.


Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
nikhilunni added a commit that referenced this pull request Jul 7, 2026
@nikhilunni @claude
feat(chunk-store): ADR 0078 phase 1 — write-through floor (#548)
c0d684e 
GCS-free resume moves 4+5, rebased onto main past #583/#584/#597. Close
the put_chunk exists()-arm write-through hole + add put_chunk_unchecked
to drop the per-dirty-chunk GCS HEAD on flush paths. ADR renumbered
0075 -> 0078 at land (main's 0075 = substrate-single-writer, #583).

Review fix folded in: the flush path's explicit cache.put after
put_chunk_unchecked was a duplicate 16 MiB local write per flushed
chunk (the store's internal write-through already warms the ONE cache —
prod wires it in host-agent main). Cut to one path; the
flush_write_throughs_chunks_into_local_cache test now mirrors the prod
wiring (store carries the cache).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
nikhilunni added a commit that referenced this pull request Jul 7, 2026
@nikhilunni @claude
feat(chunk-store): ADR 0078 phase 1 — write-through floor (#548)
8bd5563 
GCS-free resume moves 4+5, rebased onto main past #583/#584/#597. Close
the put_chunk exists()-arm write-through hole + add put_chunk_unchecked
to drop the per-dirty-chunk GCS HEAD on flush paths. ADR renumbered
0075 -> 0078 at land (main's 0075 = substrate-single-writer, #583).

The flush path keeps its explicit cache.put alongside the store-internal
write-through: cache and store travel separately into the disk backend,
so the store carrying a cache is prod wiring, not a structural
guarantee — collapsing to one cache identity is ADR 0076 (substrated)
territory. (A review pass tried cutting it; reverted — the CI-proven
shape stands. Note: eviction_finalize_redrive fails on the DEV-VM for
main and this branch alike — environmental, tracked in the dev-vm
pitfalls; CI is the arbiter for that suite.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
nikhilunni added a commit that referenced this pull request Jul 7, 2026
@nikhilunni @claude
feat(chunk-store): ADR 0078 phase 1 — write-through floor (#548)
b26914c 
GCS-free resume moves 4+5, rebased onto main past #583/#584/#597. Close
the put_chunk exists()-arm write-through hole + add put_chunk_unchecked
to drop the per-dirty-chunk GCS HEAD on flush paths. ADR renumbered
0075 -> 0078 at land (main's 0075 = substrate-single-writer, #583).

The flush path keeps its explicit cache.put alongside the store-internal
write-through: cache and store travel separately into the disk backend,
so the store carrying a cache is prod wiring, not a structural
guarantee — collapsing to one cache identity is ADR 0076 (substrated)
territory. (A review pass tried cutting it; reverted — the CI-proven
shape stands. Note: eviction_finalize_redrive fails on the DEV-VM for
main and this branch alike — environmental, tracked in the dev-vm
pitfalls; CI is the arbiter for that suite.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
nikhilunni added a commit that referenced this pull request Jul 7, 2026
@nikhilunni @claude
feat(chunk-store): ADR 0078 phase 1 — write-through floor (#548) (#586)
b3e458c 
GCS-free resume moves 4+5, rebased onto main past #583/#584/#597. Close
the put_chunk exists()-arm write-through hole + add put_chunk_unchecked
to drop the per-dirty-chunk GCS HEAD on flush paths. ADR renumbered
0075 -> 0078 at land (main's 0075 = substrate-single-writer, #583).

The flush path keeps its explicit cache.put alongside the store-internal
write-through: cache and store travel separately into the disk backend,
so the store carrying a cache is prod wiring, not a structural
guarantee — collapsing to one cache identity is ADR 0076 (substrated)
territory. (A review pass tried cutting it; reverted — the CI-proven
shape stands. Note: eviction_finalize_redrive fails on the DEV-VM for
main and this branch alike — environmental, tracked in the dev-vm
pitfalls; CI is the arbiter for that suite.)


Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 8, 2026
Merged
Closed
nikhilunni added a commit that referenced this pull request Jul 9, 2026
@nikhilunni @claude
docs(adr-0075): flip to Accepted — #547 phase 3 bookend close-out
4f732ff 
Phases 1+2 verified shipped in #583 (43c87b9): engram-substrate-proto,
ChunkCacheReader, the populate server (pin→get→open→send-fd→unpin), the
handler's 3-step read path, and the Hello/HelloAck readiness handshake.
Divergence log records: one-PR delivery, per-connection pinning with
Hello-replay-on-reconnect (vs the sketched sandbox-lifetime pin_all),
and the ADR 0080 interaction (materializer = new in-process caller,
single-writer holds by construction; its scratch statvfs guard is
deliberately outside the cache's pin/evict bookkeeping).

ADR 0076 (engram-substrated) deliberately remains Proposed/gated.

Closes the in-repo scope of #547.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jm24mF1hXuphYzXCLVxQ3d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@nikhilunni