Repository navigation
feat(park): ADR 0068 rung 2 — parked-paused eviction + dwell/pressure reaper (#545) - #585
Merged
Merged
Conversation
nikhilunni
force-pushed
the
epic/545-rung1-cancel-evict
branch
from
July 6, 2026 13:29
8f9e81d to
b7f4185
Compare
nikhilunni
force-pushed
the
epic/545-rungs-2-3
branch
from
July 6, 2026 13:29
377db0c to
6abce91
Compare
nikhilunni
force-pushed
the
epic/545-rung1-cancel-evict
branch
from
July 6, 2026 19:58
b7f4185 to
0484b05
Compare
nikhilunni
force-pushed
the
epic/545-rungs-2-3
branch
from
July 6, 2026 21:53
6abce91 to
a7064cd
Compare
nikhilunni
force-pushed
the
epic/545-rungs-2-3
branch
from
July 6, 2026 22:06
a7064cd to
a558879
Compare
nikhilunni
marked this pull request as ready for review
July 6, 2026 22:22
nikhilunni
added a commit
that referenced
this pull request
Jul 6, 2026
…start_agent rung-2 (#585) parked-paused was a LIVE REGRESSION: a returning user to a parked session paid ~40s (WORSE than the ~18s plain evict+resume it targets). The un-pause itself (host.resume of the still-resident VM) is instant, but the harness->hub vsock binding drops across the FC pause, so the outbox delivery driver hit NotFound and fired reattach_harness_in_place -> a full start_agent (agent_handshake + resume prefault). That not only wastes ~40s — it KILLS the harness that is still alive inside the un-paused VM and re-spawns it. The park keeps the harness alive (host.pause is a pure VM freeze — no drain), keeps sandbox_id bound, and does not bump the binding epoch. So the alive harness re-dials the hub on its own (ADR 0073 self-auth loop) with its still- valid epoch within a beat. Fix: on a NotFound the delivery driver now gives the in-guest self-reattach a bounded window of plain retries (~12s over failure_backoff's 2s/4s/6s) BEFORE falling back to start_agent — so a returning user un-pauses in a beat, while the genuine harness-gone desync still self-heals via start_agent after the window. Also fixes the admin-EvictIdle-parks-Active gap: ensure_active's Active arm now un-pauses any park_rung==2 first, so a prompt can't stall on a frozen VM regardless of how the session was parked. Safe by construction (start_agent fallback preserved, never worse); verified by the prod park->un-pause benchmark (18s baseline; target <2s). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
nikhilunni
added a commit
that referenced
this pull request
Jul 7, 2026
…start_agent (#594) rung-2 (#585) parked-paused was a LIVE REGRESSION: a returning user to a parked session paid ~40s (WORSE than the ~18s plain evict+resume it targets). The un-pause itself (host.resume of the still-resident VM) is instant, but the harness->hub vsock binding drops across the FC pause, so the outbox delivery driver hit NotFound and fired reattach_harness_in_place -> a full start_agent (agent_handshake + resume prefault). That not only wastes ~40s — it KILLS the harness that is still alive inside the un-paused VM and re-spawns it. The park keeps the harness alive (host.pause is a pure VM freeze — no drain), keeps sandbox_id bound, and does not bump the binding epoch. So the alive harness re-dials the hub on its own (ADR 0073 self-auth loop) with its still- valid epoch within a beat. Fix: on a NotFound the delivery driver now gives the in-guest self-reattach a bounded window of plain retries (~12s over failure_backoff's 2s/4s/6s) BEFORE falling back to start_agent — so a returning user un-pauses in a beat, while the genuine harness-gone desync still self-heals via start_agent after the window. Also fixes the admin-EvictIdle-parks-Active gap: ensure_active's Active arm now un-pauses any park_rung==2 first, so a prompt can't stall on a frozen VM regardless of how the session was parked. Safe by construction (start_agent fallback preserved, never worse); verified by the prod park->un-pause benchmark (18s baseline; target <2s). Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
nikhilunni
added a commit
that referenced
this pull request
Jul 7, 2026
…substrated design (#547) Rebased onto main past #585/#590-#596; ADRs renumbered 0072/0073 -> 0075/0076 (main's 0072 = substrate-survey-evidence, 0073 = binding-epoch). See PR #583 for the full description. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
nikhilunni
added a commit
that referenced
this pull request
Jul 7, 2026
…substrated design (#547) Rebased onto main past #585/#590-#596; ADRs renumbered 0072/0073 -> 0075/0076 (main's 0072 = substrate-survey-evidence, 0073 = binding-epoch). See PR #583 for the full description. Review fix folded in: the client replays `Hello` on every fresh dial (with_conn), not just at startup — the server's session-chunk pin set and proto check are per-connection state, and the designed-for reconnect (a rolled host-agent's successor, holding an empty pin set) is exactly when silently skipping the replay would leave the handler unpinned for the rest of the sandbox's life. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
nikhilunni
added a commit
that referenced
this pull request
Jul 7, 2026
…substrated design (#547) Rebased onto main past #585/#590-#596; ADRs renumbered 0072/0073 -> 0075/0076 (main's 0072 = substrate-survey-evidence, 0073 = binding-epoch). See PR #583 for the full description. Review fix folded in: the client replays `Hello` on every fresh dial (with_conn), not just at startup — the server's session-chunk pin set and proto check are per-connection state, and the designed-for reconnect (a rolled host-agent's successor, holding an empty pin set) is exactly when silently skipping the replay would leave the handler unpinned for the rest of the sandbox's life. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
nikhilunni
added a commit
that referenced
this pull request
Jul 7, 2026
…substrated design (#547) (#583) Rebased onto main past #585/#590-#596; ADRs renumbered 0072/0073 -> 0075/0076 (main's 0072 = substrate-survey-evidence, 0073 = binding-epoch). See PR #583 for the full description. Review fix folded in: the client replays `Hello` on every fresh dial (with_conn), not just at startup — the server's session-chunk pin set and proto check are per-connection state, and the designed-for reconnect (a rolled host-agent's successor, holding an empty pin set) is exactly when silently skipping the replay would leave the handler unpinned for the rest of the sandbox's life. Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
nikhilunni
added a commit
that referenced
this pull request
Jul 7, 2026
…ntimeSpec, Revive (#544) Rebased onto main past #583/#585/#590-#596. Renumbered at land: ADR 0074 -> 0077 (main's 0074 = parking ladder); migrations 0087-0089 -> 0088-0090 (main's applied high-water = 0087). The queue_prompt column references died in the rebase (#592 removed it). Five pre-merge review findings fixed (see the ADR's divergence log): - get_session projection: the selected_skills removal left a missing comma aliasing live_disk_manifest_version AS park_rung — every session read un-parked (frozen-VM-advertised-Active on the rung-2 ascent) and live_disk_manifest read None. Live-PG round-trip test pins the projection now. - fork-at-attach minted an UNPUBLISHED (uuid, v0) into manifest_ref; sessions evicted before their first flush were unevictable and zero-dirty captures unresumable. Split: fork_identity is adopted at v1 by the FIRST publish; manifest_ref stays the resolvable base until then (+ regression test). - durable_head advanced on recoverable=false rows; now gated, and a demote of the current head re-points to the newest recoverable. - prepare_from_row swallowed RuntimeSpec read errors into "no skills"; now propagates (scanner/resume retry is the recovery). - the Idle->Created harness-failed park emitted no StatusChanged and swallowed transition failures; now emits + propagates. Accepted (zero users): no selected_skills backfill into session_runtime_specs; pre-deploy sessions lose their skill selection on the next re-prepare. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
nikhilunni
added a commit
that referenced
this pull request
Jul 7, 2026
…ntimeSpec, Revive (#544) Rebased onto main past #583/#585/#590-#596. Renumbered at land: ADR 0074 -> 0077 (main's 0074 = parking ladder); migrations 0087-0089 -> 0088-0090 (main's applied high-water = 0087). The queue_prompt column references died in the rebase (#592 removed it). Five pre-merge review findings fixed (see the ADR's divergence log): - get_session projection: the selected_skills removal left a missing comma aliasing live_disk_manifest_version AS park_rung — every session read un-parked (frozen-VM-advertised-Active on the rung-2 ascent) and live_disk_manifest read None. Live-PG round-trip test pins the projection now. - fork-at-attach minted an UNPUBLISHED (uuid, v0) into manifest_ref; sessions evicted before their first flush were unevictable and zero-dirty captures unresumable. Split: fork_identity is adopted at v1 by the FIRST publish; manifest_ref stays the resolvable base until then (+ regression test). - durable_head advanced on recoverable=false rows; now gated, and a demote of the current head re-points to the newest recoverable. - prepare_from_row swallowed RuntimeSpec read errors into "no skills"; now propagates (scanner/resume retry is the recovery). - the Idle->Created harness-failed park emitted no StatusChanged and swallowed transition failures; now emits + propagates. Accepted (zero users): no selected_skills backfill into session_runtime_specs; pre-deploy sessions lose their skill selection on the next re-prepare. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
nikhilunni
added a commit
that referenced
this pull request
Jul 7, 2026
…ntimeSpec, Revive (#544) (#584) Rebased onto main past #583/#585/#590-#596. Renumbered at land: ADR 0074 -> 0077 (main's 0074 = parking ladder); migrations 0087-0089 -> 0088-0090 (main's applied high-water = 0087). The queue_prompt column references died in the rebase (#592 removed it). Five pre-merge review findings fixed (see the ADR's divergence log): - get_session projection: the selected_skills removal left a missing comma aliasing live_disk_manifest_version AS park_rung — every session read un-parked (frozen-VM-advertised-Active on the rung-2 ascent) and live_disk_manifest read None. Live-PG round-trip test pins the projection now. - fork-at-attach minted an UNPUBLISHED (uuid, v0) into manifest_ref; sessions evicted before their first flush were unevictable and zero-dirty captures unresumable. Split: fork_identity is adopted at v1 by the FIRST publish; manifest_ref stays the resolvable base until then (+ regression test). - durable_head advanced on recoverable=false rows; now gated, and a demote of the current head re-points to the newest recoverable. - prepare_from_row swallowed RuntimeSpec read errors into "no skills"; now propagates (scanner/resume retry is the recovery). - the Idle->Created harness-failed park emitted no StatusChanged and swallowed transition failures; now emits + propagates. Accepted (zero users): no selected_skills backfill into session_runtime_specs; pre-deploy sessions lose their skill selection on the next re-prepare. Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacks on #581 (rung 1: the
Evicting → Activeedge + cancellable nomination). Part of epic #545 (parking ladder, ADR 0068).What this adds — rung 2 (parked-paused)
When an idle session is nominated and its host has memory headroom, the eviction pipeline now pauses the VM in place (
park_rung=2) instead of capture+destroy. The harness stays resident; a returning user's prompt un-pauses in <100ms rather than paying the full rebuild (prod p50 12.2s / p95 89.1s).evict_session_to_state, Idle target, before the browser reap / capture): gated onhost_has_memory_headroom, which reads the RAM ledger'sallocatable_mib(Host RAM ledger: charge base-shm, parked residents, and running VMs under one allocatable_mib budget #540 — already nets out other parked VMs' PSS, so we never over-park) and fails closed on any telemetry gap.allow_park=false(drain + the reaper's own descent) skips it.try_cancel_nominated_eviction): un-pause before the CASEvicting → Active, under the held lease; only commit Active if the un-pause succeeded, else leave the sessionEvictingfor the standard resume path (never advertise Active over a paused VM).park_reaper_advance_one, on the existing eviction-scanner tick): descends a parked VM to a full eviction when the dwell cap (ENGRAM_PARK_DWELL_SECS, default 900s) elapses or the host loses headroom (reason=dwell|pressure). Parked rows route here instead of the eviction pipeline (guarded in bothscanner_run_onceandscanner_advance_one) — otherwise the pipeline would re-pause + bumpevict_attemptstoHostLostevery tick.Sessiongainspark_rung/parked_at(projected inget_session+list_evicting_sessions; migration 0086 landed with rung 1). New metrics:parked_paused/unparked_paused/park_descend{reason}.Scope calls
park_rung=3.Testing
Four coordinator unit tests: park+ascent (same live sandbox after un-pause, nothing captured), dwell descent (un-pause → capture → destroy → Idle,
park_rungcleared), and the scanner-routing guard (a parked row is never re-evicted). Nativejust checkgreen (fmt + clippy-D warnings+ nextest);engram-coordinator+engram-postgrescompile clean on the linux dev-vm. FC pause/resume is the pre-existing ADR 0045 Phase F primitive — unchanged.🤖 Generated with Claude Code