Skip to content

feat(park): ADR 0068 rung 2 — parked-paused eviction + dwell/pressure reaper (#545) - #585

Merged
nikhilunni merged 1 commit into
mainfrom
epic/545-rungs-2-3
Jul 6, 2026
Merged

nikhilunni merged 1 commit into
mainfrom
epic/545-rungs-2-3

Conversation

@nikhilunni

Copy link
Copy Markdown
Contributor

Stacks on #581 (rung 1: the Evicting → Active edge + cancellable nomination). Part of epic #545 (parking ladder, ADR 0068).

What this adds — rung 2 (parked-paused)

When an idle session is nominated and its host has memory headroom, the eviction pipeline now pauses the VM in place (park_rung=2) instead of capture+destroy. The harness stays resident; a returning user's prompt un-pauses in <100ms rather than paying the full rebuild (prod p50 12.2s / p95 89.1s).

  • Park decision (evict_session_to_state, Idle target, before the browser reap / capture): gated on host_has_memory_headroom, which reads the RAM ledger's allocatable_mib (Host RAM ledger: charge base-shm, parked residents, and running VMs under one allocatable_mib budget #540 — already nets out other parked VMs' PSS, so we never over-park) and fails closed on any telemetry gap. allow_park=false (drain + the reaper's own descent) skips it.
  • Ascent (try_cancel_nominated_eviction): un-pause before the CAS Evicting → Active, under the held lease; only commit Active if the un-pause succeeded, else leave the session Evicting for the standard resume path (never advertise Active over a paused VM).
  • Reaper (park_reaper_advance_one, on the existing eviction-scanner tick): descends a parked VM to a full eviction when the dwell cap (ENGRAM_PARK_DWELL_SECS, default 900s) elapses or the host loses headroom (reason=dwell|pressure). Parked rows route here instead of the eviction pipeline (guarded in both scanner_run_once and scanner_advance_one) — otherwise the pipeline would re-pause + bump evict_attempts to HostLost every tick.

Session gains park_rung/parked_at (projected in get_session + list_evicting_sessions; migration 0086 landed with rung 1). New metrics: parked_paused / unparked_paused / park_descend{reason}.

Scope calls

  • Rung 3 (parked-local) is sequenced into Epic: GCS-free resume — authoritative host-affinity + peer-first divergence fill #548, not built here. Its primitive — destroy the VM but retain snapshot staging + NBD backing on local NVMe for a host-pinned resume — is the authoritative-affinity machinery of the GCS-free-resume epic; building it separately would duplicate that state two ways. It lands there reusing park_rung=3.
  • Rung 4 (evicted-remote) is today's full eviction, unchanged — now the pressure/dwell floor the ladder descends to.
  • Rung 2 is pause-only in v1 — the park-time diff-banking optimization (make the eventual descent near-free) is a clean follow-up that doesn't change the rung's state shape.

Testing

Four coordinator unit tests: park+ascent (same live sandbox after un-pause, nothing captured), dwell descent (un-pause → capture → destroy → Idle, park_rung cleared), and the scanner-routing guard (a parked row is never re-evicted). Native just check green (fmt + clippy -D warnings + nextest); engram-coordinator+engram-postgres compile clean on the linux dev-vm. FC pause/resume is the pre-existing ADR 0045 Phase F primitive — unchanged.

🤖 Generated with Claude Code

@nikhilunni
nikhilunni force-pushed the epic/545-rung1-cancel-evict branch from 8f9e81d to b7f4185 Compare July 6, 2026 13:29
@nikhilunni
nikhilunni force-pushed the epic/545-rungs-2-3 branch from 377db0c to 6abce91 Compare July 6, 2026 13:29
@nikhilunni
nikhilunni force-pushed the epic/545-rung1-cancel-evict branch from b7f4185 to 0484b05 Compare July 6, 2026 19:58
Base automatically changed from epic/545-rung1-cancel-evict to main July 6, 2026 20:11
@nikhilunni
nikhilunni force-pushed the epic/545-rungs-2-3 branch from 6abce91 to a7064cd Compare July 6, 2026 21:53
@nikhilunni
nikhilunni force-pushed the epic/545-rungs-2-3 branch from a7064cd to a558879 Compare July 6, 2026 22:06
@nikhilunni
nikhilunni marked this pull request as ready for review July 6, 2026 22:22
@nikhilunni
nikhilunni merged commit 28d60aa into main Jul 6, 2026
18 checks passed
@nikhilunni
nikhilunni deleted the epic/545-rungs-2-3 branch July 6, 2026 22:22
nikhilunni added a commit that referenced this pull request Jul 6, 2026
…start_agent

rung-2 (#585) parked-paused was a LIVE REGRESSION: a returning user to a parked
session paid ~40s (WORSE than the ~18s plain evict+resume it targets). The
un-pause itself (host.resume of the still-resident VM) is instant, but the
harness->hub vsock binding drops across the FC pause, so the outbox delivery
driver hit NotFound and fired reattach_harness_in_place -> a full start_agent
(agent_handshake + resume prefault). That not only wastes ~40s — it KILLS the
harness that is still alive inside the un-paused VM and re-spawns it.

The park keeps the harness alive (host.pause is a pure VM freeze — no drain),
keeps sandbox_id bound, and does not bump the binding epoch. So the alive
harness re-dials the hub on its own (ADR 0073 self-auth loop) with its still-
valid epoch within a beat. Fix: on a NotFound the delivery driver now gives the
in-guest self-reattach a bounded window of plain retries (~12s over
failure_backoff's 2s/4s/6s) BEFORE falling back to start_agent — so a returning
user un-pauses in a beat, while the genuine harness-gone desync still self-heals
via start_agent after the window.

Also fixes the admin-EvictIdle-parks-Active gap: ensure_active's Active arm now
un-pauses any park_rung==2 first, so a prompt can't stall on a frozen VM
regardless of how the session was parked.

Safe by construction (start_agent fallback preserved, never worse); verified by
the prod park->un-pause benchmark (18s baseline; target <2s).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
nikhilunni added a commit that referenced this pull request Jul 7, 2026
…start_agent (#594)

rung-2 (#585) parked-paused was a LIVE REGRESSION: a returning user to a parked
session paid ~40s (WORSE than the ~18s plain evict+resume it targets). The
un-pause itself (host.resume of the still-resident VM) is instant, but the
harness->hub vsock binding drops across the FC pause, so the outbox delivery
driver hit NotFound and fired reattach_harness_in_place -> a full start_agent
(agent_handshake + resume prefault). That not only wastes ~40s — it KILLS the
harness that is still alive inside the un-paused VM and re-spawns it.

The park keeps the harness alive (host.pause is a pure VM freeze — no drain),
keeps sandbox_id bound, and does not bump the binding epoch. So the alive
harness re-dials the hub on its own (ADR 0073 self-auth loop) with its still-
valid epoch within a beat. Fix: on a NotFound the delivery driver now gives the
in-guest self-reattach a bounded window of plain retries (~12s over
failure_backoff's 2s/4s/6s) BEFORE falling back to start_agent — so a returning
user un-pauses in a beat, while the genuine harness-gone desync still self-heals
via start_agent after the window.

Also fixes the admin-EvictIdle-parks-Active gap: ensure_active's Active arm now
un-pauses any park_rung==2 first, so a prompt can't stall on a frozen VM
regardless of how the session was parked.

Safe by construction (start_agent fallback preserved, never worse); verified by
the prod park->un-pause benchmark (18s baseline; target <2s).


Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
nikhilunni added a commit that referenced this pull request Jul 7, 2026
…substrated design (#547)

Rebased onto main past #585/#590-#596; ADRs renumbered 0072/0073 ->
0075/0076 (main's 0072 = substrate-survey-evidence, 0073 =
binding-epoch). See PR #583 for the full description.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
nikhilunni added a commit that referenced this pull request Jul 7, 2026
…substrated design (#547)

Rebased onto main past #585/#590-#596; ADRs renumbered 0072/0073 ->
0075/0076 (main's 0072 = substrate-survey-evidence, 0073 =
binding-epoch). See PR #583 for the full description.

Review fix folded in: the client replays `Hello` on every fresh dial
(with_conn), not just at startup — the server's session-chunk pin set
and proto check are per-connection state, and the designed-for
reconnect (a rolled host-agent's successor, holding an empty pin set)
is exactly when silently skipping the replay would leave the handler
unpinned for the rest of the sandbox's life.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
nikhilunni added a commit that referenced this pull request Jul 7, 2026
…substrated design (#547)

Rebased onto main past #585/#590-#596; ADRs renumbered 0072/0073 ->
0075/0076 (main's 0072 = substrate-survey-evidence, 0073 =
binding-epoch). See PR #583 for the full description.

Review fix folded in: the client replays `Hello` on every fresh dial
(with_conn), not just at startup — the server's session-chunk pin set
and proto check are per-connection state, and the designed-for
reconnect (a rolled host-agent's successor, holding an empty pin set)
is exactly when silently skipping the replay would leave the handler
unpinned for the rest of the sandbox's life.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
nikhilunni added a commit that referenced this pull request Jul 7, 2026
…substrated design (#547) (#583)

Rebased onto main past #585/#590-#596; ADRs renumbered 0072/0073 ->
0075/0076 (main's 0072 = substrate-survey-evidence, 0073 =
binding-epoch). See PR #583 for the full description.

Review fix folded in: the client replays `Hello` on every fresh dial
(with_conn), not just at startup — the server's session-chunk pin set
and proto check are per-connection state, and the designed-for
reconnect (a rolled host-agent's successor, holding an empty pin set)
is exactly when silently skipping the replay would leave the handler
unpinned for the rest of the sandbox's life.


Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
nikhilunni added a commit that referenced this pull request Jul 7, 2026
…ntimeSpec, Revive (#544)

Rebased onto main past #583/#585/#590-#596. Renumbered at land:
ADR 0074 -> 0077 (main's 0074 = parking ladder); migrations
0087-0089 -> 0088-0090 (main's applied high-water = 0087). The
queue_prompt column references died in the rebase (#592 removed it).

Five pre-merge review findings fixed (see the ADR's divergence log):
- get_session projection: the selected_skills removal left a missing
  comma aliasing live_disk_manifest_version AS park_rung — every
  session read un-parked (frozen-VM-advertised-Active on the rung-2
  ascent) and live_disk_manifest read None. Live-PG round-trip test
  pins the projection now.
- fork-at-attach minted an UNPUBLISHED (uuid, v0) into manifest_ref;
  sessions evicted before their first flush were unevictable and
  zero-dirty captures unresumable. Split: fork_identity is adopted at
  v1 by the FIRST publish; manifest_ref stays the resolvable base
  until then (+ regression test).
- durable_head advanced on recoverable=false rows; now gated, and a
  demote of the current head re-points to the newest recoverable.
- prepare_from_row swallowed RuntimeSpec read errors into "no skills";
  now propagates (scanner/resume retry is the recovery).
- the Idle->Created harness-failed park emitted no StatusChanged and
  swallowed transition failures; now emits + propagates.

Accepted (zero users): no selected_skills backfill into
session_runtime_specs; pre-deploy sessions lose their skill selection
on the next re-prepare.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
nikhilunni added a commit that referenced this pull request Jul 7, 2026
…ntimeSpec, Revive (#544)

Rebased onto main past #583/#585/#590-#596. Renumbered at land:
ADR 0074 -> 0077 (main's 0074 = parking ladder); migrations
0087-0089 -> 0088-0090 (main's applied high-water = 0087). The
queue_prompt column references died in the rebase (#592 removed it).

Five pre-merge review findings fixed (see the ADR's divergence log):
- get_session projection: the selected_skills removal left a missing
  comma aliasing live_disk_manifest_version AS park_rung — every
  session read un-parked (frozen-VM-advertised-Active on the rung-2
  ascent) and live_disk_manifest read None. Live-PG round-trip test
  pins the projection now.
- fork-at-attach minted an UNPUBLISHED (uuid, v0) into manifest_ref;
  sessions evicted before their first flush were unevictable and
  zero-dirty captures unresumable. Split: fork_identity is adopted at
  v1 by the FIRST publish; manifest_ref stays the resolvable base
  until then (+ regression test).
- durable_head advanced on recoverable=false rows; now gated, and a
  demote of the current head re-points to the newest recoverable.
- prepare_from_row swallowed RuntimeSpec read errors into "no skills";
  now propagates (scanner/resume retry is the recovery).
- the Idle->Created harness-failed park emitted no StatusChanged and
  swallowed transition failures; now emits + propagates.

Accepted (zero users): no selected_skills backfill into
session_runtime_specs; pre-deploy sessions lose their skill selection
on the next re-prepare.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx
nikhilunni added a commit that referenced this pull request Jul 7, 2026
…ntimeSpec, Revive (#544) (#584)

Rebased onto main past #583/#585/#590-#596. Renumbered at land:
ADR 0074 -> 0077 (main's 0074 = parking ladder); migrations
0087-0089 -> 0088-0090 (main's applied high-water = 0087). The
queue_prompt column references died in the rebase (#592 removed it).

Five pre-merge review findings fixed (see the ADR's divergence log):
- get_session projection: the selected_skills removal left a missing
  comma aliasing live_disk_manifest_version AS park_rung — every
  session read un-parked (frozen-VM-advertised-Active on the rung-2
  ascent) and live_disk_manifest read None. Live-PG round-trip test
  pins the projection now.
- fork-at-attach minted an UNPUBLISHED (uuid, v0) into manifest_ref;
  sessions evicted before their first flush were unevictable and
  zero-dirty captures unresumable. Split: fork_identity is adopted at
  v1 by the FIRST publish; manifest_ref stays the resolvable base
  until then (+ regression test).
- durable_head advanced on recoverable=false rows; now gated, and a
  demote of the current head re-points to the newest recoverable.
- prepare_from_row swallowed RuntimeSpec read errors into "no skills";
  now propagates (scanner/resume retry is the recovery).
- the Idle->Created harness-failed park emitted no StatusChanged and
  swallowed transition failures; now emits + propagates.

Accepted (zero users): no selected_skills backfill into
session_runtime_specs; pre-deploy sessions lose their skill selection
on the next re-prepare.


Claude-Session: https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant