Epic #542: binding epoch, self-authenticating attach, durable acked delivery, one idle detector (ADR 0067)

[nikhilunni]

Implements #542 (epic: binding epoch + self-authenticating harness attach + durable acked delivery + one idle detector). ADR 0067 authored Proposed-first and flipped to Accepted with the commit chain, per convention. One commit per phase.

What this does

Phase 1 — binding epoch + self-authenticating attach (migration 0083)
PG owns sessions.binding_epoch, minted coordinator-side at every fresh-spawn bind and carried as an attach token (ENGRAM_SANDBOX_ID + ENGRAM_BINDING_EPOCH, stamped by the backend at spawn — FC/VZ/Process uniformly). The hub validates every attach against a host-durable binding record (<work_dir>/bindings/<session>.json, monotonic in the epoch) — never an in-memory map. A restarted host-agent accepts survivor re-dials with zero rebuild pass (#447 closed by construction; rebind_survivor_sessions deleted); a stale generation is rejected Superseded and exits (the fbd3794c 8-fake-resume loop is unrepresentable).

Phase 2 — coordinator-durable outbox, SendPrompt 202 (migration 0084)
Prompts/answers become session_outbox rows: receipt + echo + one INSERT + NOTIFY + immediate return. The delivery driver resumes the session behind the enqueue, forwards oldest-first, self-heals the unbound desync, and redelivers until the confirming harness event acks the row at the emit choke point. The e35ed1f eaten-message class is gone; the host replay buffers, 10s attach wait, deliver_with_reattach poll, 60s HOLD loop, and the orchestrator's "mention me again" are deleted.

Phase 3 — one idle detector, PG shell pin (migration 0085)
The PG event-log scanner (was the "L3 backstop") becomes THE detector with the host detector's exact semantics (soft 300s/hard 1800s TTLs, 10s cadence, disk brake, default-off pressure-aware gating — now reading heartbeat-persisted hosts.utilization). Shell pins become a shell_pinned_until column stamped by the relays (issue #219 leak class structurally gone). Deleted: the host eviction tick, hub TTL scan, candidates POST path, desync watchdog + rehandshake RPC + HarnessCommand::Rehandshake, shell-pin RPCs. Heartbeat gains harness_attached as a disagreement alarm (metric), never a healer.

Phase 4 — hub down to one leaf mutex; actor-ization moot
With seven of eight maps deleted across the phases, the planned actor rewrite would wrap one never-nested mutex in machinery. Recorded in the ADR as moot-by-deletion.

Deploy notes (⚠️ read before merge)

• WIRE_VERSION 8→9 — lockstep coord+host roll, no fallback ladder.
• Harness bundle re-publish required: HarnessAttach and HarnessCommand are clean bincode breaks (variant indices re-pinned in goldens). publish-bundles on merge covers it; a stale bundle fails attach loudly (UnknownBinding), which is the designed failure.
• Env knob ownership moves: ENGRAM_IDLE_TTL_SECS / ENGRAM_IDLE_HARD_TTL_SECS / ENGRAM_IDLE_EVICT_* now belong on the coordinator deployment (values unchanged; engrams-internal follow-up).
• Migrations 0083–0085 (0080–0082 reserved by open PRs Typed capability-vector host readiness + probe-before-host_lost #564/Enable-time fleet chunk prestage: gate ready on eligible-host staging (INTERIM, issue #538) #565/Create as a plan: per-image boot bundle + one-transaction write-set + overlapped boot legs + prompt-over-wire #566; sqlx needs uniqueness, not contiguity).
• agentd is untouched — no image re-bake needed beyond the bundle.

Divergences from the issue (all recorded in ADR 0067's log)

1. Epoch minted coordinator-side pre-spawn (not inside assign_session_sandbox) — the token rides spawn env, which exists before the sandbox does.
2. The epoch fences process lineage, not VM identity — agentd C1-reattaches live harnesses on teleport, so live moves re-point the record at the same epoch; token sandbox_id is informational.
3. Phases 3/4 swapped (purge before actor), and the actor itself is moot-by-deletion.
4. The fixup-round's outbox/prompt_received coordination honored: the event stays the SLO receipt anchor.

Testing

• just check green (fmt, clippy -D warnings, hakari, full workspace suite).
• New: binding-store unit matrix; fresh-hub survivor routing; loopback survivor re-dial + Superseded rejection regressions (transport-agnostic, run in every lane — the FC lane's e2e_harness covers the real-VM attach chain and now exercises token injection through the real spawn path); outbox store-contract live-PG suite (5 tests, wired into ci.yml's Postgres-gated list, validated against a real PG 16 locally incl. migrations 0083–0085); idle-detector classify matrix + nomination test; orchestrator bun test 541 pass.
• Coverage moves (never deleted): rebind_survivor_sessions tests → fresh-hub disk-routing tests; harness_command_redelivery → outbox re-arm live-PG test + driver; backstop/candidates nomination tests → idle_detector tests.
• Post-rebase reconciliation with merged Host RAM ledger: charge base-shm, parked residents, and running VMs under one allocatable_mib budget #561/Telemetry restoration: host DaemonSet OTLP, span parenting, effectiveness counters (scoped: span parenting + DaemonSet OTLP + counters) #562/Instrument and harden the capture-time [warm] hook: per-stage progress, stall detection, persisted output tail (OSS half) #563 caught and fixed a real direction bug: the detector's memory gate now fails open toward eviction on unmeasured utilization, matching Host RAM ledger: charge base-shm, parked residents, and running VMs under one allocatable_mib budget #561's contract (ported unit coverage proves it).

Baselines this targets (2026-07-01 evidence): 49 sandbox not found/7d → 0 for the binding-desync fraction; SendPrompt ack decoupled from the 12.2s-p50/89s-p95 resume; eaten follow-up messages → redelivered-until-acked.

🤖 Generated with Claude Code

https://claude.ai/code/session_014jJi2vqAaxt3Q5UKxbe4Gx

[github-actions]

The latest Buf updates on your PR. Results from workflow CI / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	⏩ skipped	⏩ skipped	❌ failed (5)	Jul 6, 2026, 3:55 PM