Repository navigation
fix: batch the cache eviction sweep on migration staging paths - #184
Merged
Merged
Conversation
Found by the first successful prod live teleport (108.5s total): a ~94 MiB transfer (59 mem + 4 disk chunks) took 70s to prestage and 12.6s to capture, because ChunkCache::put runs a SYNCHRONOUS eviction sweep — a full two-level readdir of the NVMe cache root + statvfs — per write. 63 staged chunks = 63 full-cache scans. (cache.rs's own comment: "spawn it off-thread if profiling shows it's hurting tail latency" — profiling has spoken.) ChunkCache grows put_no_evict + sweep (the batch-closing pair); the three bulk staging paths — the local-sink re-chunk (source capture), flush_to_local_cache (source disk drain), and the dest prestage pull — write unswept and sweep once per batch. The budget/floor invariant is per-batch, not per-write; singleflight readers are unaffected. Expected: capture 12.6s -> ~2s, prestage 70s -> ~2s on the canary re-run. The remaining 24.8s agent_handshake on the moved guest is a separate (pre-existing resume-path) item. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nikhilunni
added a commit
that referenced
this pull request
Jun 11, 2026
cache.get's miss-path populate (write_local) ran a FULL synchronous eviction sweep — two-level readdir + statvfs — per chunk. #184 fixed the explicit put() callers; this one sat under EVERY other consumer: the uffd handler's fault serving (~150-500ms per post-teleport fault), parallel prefetch (959 chunks measured at 115s), and the migration source's fetch fallback (the 70s synchronous pull on canary 6ab48e73). Debounce to once per interval (config field, default 5s; tests pin 0 for deterministic eviction asserts). Budget enforcement still happens within the interval — it's a soft ceiling probed against a multi-GB cache. Dev-vm real KVM: teleport + migration + NBD suites 7/7; chunk-store units 110/110. Direct-to-main per operator authorization. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Prod-canary finding #2 (the move itself succeeded, lossless): ~94 MiB of staging took 70 s (prestage) + 12.6 s (capture) because every
ChunkCache::putruns a synchronous full-cache readdir+statvfs eviction sweep — 63 chunks = 63 scans of the prod NVMe cache. Newput_no_evict/sweeppair; the three bulk staging paths sweep once per batch. Expected to collapse both legs to seconds; the canary re-run after the roll gives the verified G1 numbers.(Original suspicion — the resume-path prefetch — was wrong: it's backgrounded. Trace + host logs attributed the time precisely before fixing.)
🤖 Generated with Claude Code