Repository navigation
feat(chunk-store): ADR 0072 phase 1 — write-through floor (#548) - #586
Merged
Merged
Conversation
nikhilunni
force-pushed
the
epic/548-gcs-free-resume
branch
from
July 6, 2026 13:30
be8f416 to
5b310e8
Compare
|
The latest Buf updates on your PR. Results from workflow CI / buf (pull_request).
|
nikhilunni
force-pushed
the
epic/544-transactional-snapshots
branch
2 times, most recently
from
July 7, 2026 04:33
630cf00 to
3ee7c6e
Compare
nikhilunni
force-pushed
the
epic/548-gcs-free-resume
branch
from
July 7, 2026 05:30
5b310e8 to
c0d684e
Compare
nikhilunni
marked this pull request as ready for review
July 7, 2026 05:30
nikhilunni
force-pushed
the
epic/548-gcs-free-resume
branch
from
July 7, 2026 06:12
c0d684e to
8bd5563
Compare
GCS-free resume moves 4+5, rebased onto main past #583/#584/#597. Close the put_chunk exists()-arm write-through hole + add put_chunk_unchecked to drop the per-dirty-chunk GCS HEAD on flush paths. ADR renumbered 0075 -> 0078 at land (main's 0075 = substrate-single-writer, #583). The flush path keeps its explicit cache.put alongside the store-internal write-through: cache and store travel separately into the disk backend, so the store carrying a cache is prod wiring, not a structural guarantee — collapsing to one cache identity is ADR 0076 (substrated) territory. (A review pass tried cutting it; reverted — the CI-proven shape stands. Note: eviction_finalize_redrive fails on the DEV-VM for main and this branch alike — environmental, tracked in the dev-vm pitfalls; CI is the arbiter for that suite.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
nikhilunni
force-pushed
the
epic/548-gcs-free-resume
branch
from
July 7, 2026 06:45
8bd5563 to
b26914c
Compare
nikhilunni
added a commit
that referenced
this pull request
Jul 7, 2026
…ocal_snapshots (#548) Rebased onto main past #586/#598. Renumbered at land: migration 0090_drop_hosts_local_snapshots -> 0091 (main took 0090); ADR refs 0075 -> 0078. WIRE_VERSION 10 -> 11 (lockstep coord+host roll; v10 hosts drain from scheduling until the fleet rolls). Review fixes folded in: - the TS tier now follows the retirement: buf-generated fleet_pb.ts regenerated (web + orchestrator) and the four web consumers of the always-zero localSnapshots field removed (the un-regenerated codegen would have failed CI's drift check; the Fleet page rendered a dead '0 snapshots' datum). - snapshot_host_veto's cap arm now emits the promised `cap:<name>` label (bare "cap" made fc_snapshot_version-vs-bundle-stamp spikes undiagnosable from the metric). - pick_from's tier doc no longer claims capacity-BLIND affinity (tier-0 is capacity-checked by construction). - HostUtilization's rustdoc un-interleaved from the disk-floor consts; migration comment corrected (said WIRE 9->10; actual 10->11) before checksum-freeze; README/DESIGN heartbeat field lists updated; ADR gains the possession-freshness divergence note. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
nikhilunni
added a commit
that referenced
this pull request
Jul 7, 2026
…ocal_snapshots (#548) (#587) Rebased onto main past #586/#598. Renumbered at land: migration 0090_drop_hosts_local_snapshots -> 0091 (main took 0090); ADR refs 0075 -> 0078. WIRE_VERSION 10 -> 11 (lockstep coord+host roll; v10 hosts drain from scheduling until the fleet rolls). Review fixes folded in: - the TS tier now follows the retirement: buf-generated fleet_pb.ts regenerated (web + orchestrator) and the four web consumers of the always-zero localSnapshots field removed (the un-regenerated codegen would have failed CI's drift check; the Fleet page rendered a dead '0 snapshots' datum). - snapshot_host_veto's cap arm now emits the promised `cap:<name>` label (bare "cap" made fc_snapshot_version-vs-bundle-stamp spikes undiagnosable from the metric). - pick_from's tier doc no longer claims capacity-BLIND affinity (tier-0 is capacity-checked by construction). - HostUtilization's rustdoc un-interleaved from the disk-floor consts; migration comment corrected (said WIRE 9->10; actual 10->11) before checksum-freeze; README/DESIGN heartbeat field lists updated; ADR gains the possession-freshness divergence note. Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1 of epic #548 (GCS-free resume). ADR 0072 (Proposed). Branches off
main(has the #528/#529 deps).The mechanical write-through floor — moves 4+5 of the ADR. Backend-agnostic, no wire change.
Changes
exists()write-through hole (engram-chunk-store/src/store.rs):put_chunk's idempotency short-circuit now runs the local write-through (warm_local) before returning. A chunk that is remotely present but locally evicted is re-warmed by a put, so the producing host stops re-fetching its own uploads from GCS on the next resume. (Closes the 583+55write_local failed/write-through … failedre-miss class for the exists-arm.)store.rs+ both flush call sites): newput_chunk_unchecked(body)— unconditional PUT + unconditional local warm, noexists(). The NBD disk flush (disk_daemon/backend.rs) and teleport memory catch-up (pooled_backend.rs) switch to it: their input is freshly re-chunked, new by construction, so the per-chunk HEAD was O(dirty) round-trips/flush that always missed.put_chunkkeeps its dedup HEAD for capture/enable-scale uploads (there it saves re-uploading tens of GiB on a re-bake) — the fix is a second entry point, not a behavior change.engram_chunk_put_total{mode=checked|unchecked, outcome}. Thedivergence_source{tier}counter lands in phase 3, wherelocal|peer|gcsare all real (emitted at the resume miss-chain) — not redundantly bolted onto the generic cache hit/miss path now.Tests
Counting-mock
BlobStorageasserts: exists()-arm warms a locally-evicted chunk (move 4);put_chunk_uncheckedissues zero HEADs + uploads + warms (move 5, acceptance #2); a 16-chunk flush issues zero HEADs;put_chunkstill dedups on a second identical put (capture-path regression guard). All green;cargo check -p engram-host-agentclean; clippy + fmt clean.Notes
local_snapshots, wire+schema), 3 (peer-first fill, rides Substrate single-writer discipline: one chunk-cache owner per host (interim) + engram-substrated ADR (design only) #547), 4 (evict-time prestage + rung-3 parked-local), 5 (flip to Accepted) follow as stacked PRs.🤖 Generated with Claude Code