Governance MKP+
Independent watchdog thread enforcing ≤ 5 token post‑trip halt budget, fail‑closed semantics, and streaming/batch symmetry.
AD‑09 [WB] — Watchdog & Streaming Halt Budget
Aliases: None Cluster: Governance Depends on: AD‑07 [CL], AD‑10 [TC], AD‑18 [SE], AD‑08 [SO], AD‑05 [MRP]
0. For humans
What question does this answer?
"If something bad trips mid‑stream, how fast does the system actually stop talking, and who enforces that stop?"
This card defines:
- the streaming halt budget (≤ 5 tokens after any trip);
- the independent watchdog that enforces that budget even if the main eval loop is slow or stuck;
- how this is measured (FHR) and surfaced (dashboards, incidents).
The timing card (AD‑10 [TC]) tells you when we decide to halt; AD‑09 tells you how we guarantee the halt actually happens.
How do I know it's live?
You should be able to:
- Observe in logs that, after any safety/timing breach (e.g. TIG breach, exfil incident), streaming stops within ≤ 5 tokens of the trip event. The delta "tokens emitted after trip" is logged per event.
- See watchdog events in FHR & logs:
watchdog_trip: boolin the timing/RTS fields; aWATCHDOG_TRIPfailure/incident code when it fires. - Open the Grafana / ops pack and see: distributions of
halt_tokens_post_trip, p95/p99 halt latency, p95 streaming overhead; FP/TP rates (how often watchdog fires vs the underlying timing, I/O-integrity, or exfiltration-control detector).
Where do I see it?
- Runtime ledger: per‑turn, you'll see streaming segments; a trip event (e.g.
TIG_BREACH,EXFIL_ALERT); aWATCHDOG_TRIPrecord with token index and reason. - FHR (AD‑08): timing fields
rts_mode,latency_bucket,mi_latency_content_est,tig_breach,watchdog_trip;timing_incidentsincludingTIMING_COVERT_CHANNEL,COMPUTE_BUDGET_EXCEEDED, andWATCHDOG_TRIPwhere relevant. - Dashboards: "Timing & Halt" panel: ring alerts, MRP state, halt tokens, p95 overhead, PR/CUSUM traces, tool aborts.
Quick checklist
- Implemented if: (1) HALT ≤ 5 tokens after any safety/timing breach in both streaming and batch modes; (2) independent watchdog thread exists that can pre‑empt streaming even if main eval stalls, logging
WATCHDOG_TRIP; (3) FHR rows and dashboards expose halt tokens and watchdog activity. - Key artefacts: watchdog thread/process, bench harness, Grafana pack
- Key FHR fields:
watchdog_trip,timing_incidents[],halt_tokens_post_trip,halt_latency_ms - Key failure codes:
WATCHDOG_TRIP,TIMING_COVERT_CHANNEL,COMPUTE_BUDGET_EXCEEDED - Escalation path: WATCHDOG_TRIP → Degraded Safe Mode (AD‑05 [MRP]) → incident runbook (AD‑12 [OP])
1. Purpose & scope
AD‑09 defines the Watchdog & Streaming Halt Budget as the mechanism that guarantees:
Once a protected invariant is breached (timing, exfil, safety gate), the system can stop emitting tokens within a small, fixed token budget, even if the main pipeline is slow, blocked, or adversarially manipulated.
It is the concrete enforcement layer behind:
- Timing‑Integrity Gate & RTS (AD‑10 [TC]): on breach, set
runtime_mode = degraded, halt within ≤ 5 tokens, roll back uncommitted kernel-owned state, and reconcile any dispatched external effect, with awatchdog_tripflag. - Runtime performance SLO: "streaming halts ≤ 5 tokens post‑trip", tracked via a bench harness & Grafana pack.
It is scoped to token emission only:
- It doesn't decide what is a breach (that's AD‑03 [CS] / AD‑10 [TC] and the distributed I/O-integrity / exfiltration-control functions defined at §3).
- It doesn't decide how the conversation continues afterwards (that's MRP / Degraded Safe Mode / care).
- It only enforces "stop talking now" within strict token limits, and makes that behaviour observable.
2. Relation to other ADs
- AD‑10 [TC] — Timing Integrity: defines timing covert‑channel controls, RTS/ETG/MI bounds, and the semantics of "breach". On any breach, the action stack sets
runtime_mode = degraded, halts within ≤ 5 tokens, rolls back uncommitted kernel-owned state, and reconciles dispatched external effects under AD‑18. AD‑09 implements the pre‑emptive watchdog that ensures the halt can actually interrupt streams. - AD‑18 [SE] — Systems Engineering & Runtime: sets TURN‑ACID, router fencing, perf scheduling, and adds "Safety watchdog thread: independent low‑latency watchdog that can pre‑empt streaming to enforce the ≤5‑token halt budget; log WATCHDOG_TRIP."
- AD‑08 [SO] — FHR: logs
watchdog_trip,timing_incidents,timing_op_point_id,mi_method_id, and provideshalt_tokens_post_tripdistributions via dashboards. - AD‑30 [SC] — Session Continuity: watchdog state interacts with memory write restrictions during crisis states.
3. Conceptual model
3.1 Streaming timeline
We model a streaming turn as:
- A sequence of emitted tokens indexed t₁, t₂, …, tₙ.
- A sequence of detector events:
timing_breachfrom TIG/RTS (AD‑10 [TC]);io_integrity_alert/exfil_alertfrom the distributed functions enforced across AD‑18 [SE] and AD‑10 [TC] (not standalone gates; routed via AD‑05 [MRP]); other hard gate violations (e.g. PEL NO‑GO).
Define:
- Trip point τ: the earliest time index when any gating/detector raises a hard trip signal.
- Post‑trip tokens: all tokens emitted with index > index(τ) on the user stream.
The streaming halt budget B is the maximum allowed number of post‑trip tokens:
- Spec default: B = 5; "streaming halts ≤ 5 tokens post‑trip".
3.2 Watchdog vs main eval
- Main eval pipeline: runs the model, sampling, tools, etc; may be busy or slowed by heavy tools, retries, or adversarial prompts.
- Watchdog: independent thread/process with direct visibility into trip events, direct control over the network stream / token emitter. Cannot be overridden by the model; only by operators changing config.
When the trip fires:
- Main eval may still be computing / trying to emit further tokens;
- Watchdog enforces the halt budget on the user‑facing stream, regardless of main eval state.
4. Protocol & state machine
In this card, Degraded Safe Mode is the conservative user-facing behaviour applied while runtime_mode = degraded; it is not a third kernel_profile. An architecture-profile change between full and mkp is a separate governed event.
4.1 States
Per streaming turn, define watchdog states:
- W_IDLE — no stream active.
- W_STREAMING — normal streaming, no breach.
- W_TRIPPED — breach detected; halt budget countdown started.
- W_HALTED — stream forcibly closed; kernel rollback and
runtime_mode = degradedfollow-up, plus external-effect reconciliation where required.
4.2 Transitions
- Start stream — enter W_STREAMING with
post_trip_tokens = 0. - Trip event (any hard breach) — source: TIG/RTS timing breach, distributed I/O-integrity or exfiltration detection, or other G1/H1 gate. Watchdog transitions to W_TRIPPED, stores: τ = trip time;
trip_cause(e.g.TIG_BREACH,IO_INTEGRITY_ALERT,EXFIL_ALERT). - Post‑trip streaming — for each new token candidate: if
post_trip_tokens < B, allow emit and increment; ifpost_trip_tokens >= B, close stream: force W_HALTED; signal the TURN‑ACID runner to roll back uncommitted kernel-owned state, setruntime_mode = degraded, and begin reconciliation for any operation alreadyDISPATCHEDor later. - Halt & recovery — after halt: mark
watchdog_trip = true; emitWATCHDOG_TRIPthrough the AD‑08 event envelope withtrip_cause,post_trip_tokens, and detector references. In degraded runtime mode, use fact‑only literal content; block new tool dispatch; ask before answering; and escalate to a human if configured. An explicit architecture-profile change, if needed, follows AD‑12 governance and is logged separately.
4.3 Interaction with TIG & RTS (AD‑10 [TC])
- TIG/RTS define when latency distributions or MI bounds are breached. On breach they request
runtime_mode = degraded, HALT ≤ 5 tokens, rollback of uncommitted kernel-owned state, and receipt-backed reconciliation of any dispatched external effect. - The watchdog is the mechanism that ensures the "HALT ≤ 5 tokens" part is honoured on the actual stream.
5. Invariants vs configurable degrees of freedom
Hard invariants
To claim AD‑09 compliance, the following must hold:
- Token halt SLO — for any hard trip event on a streaming turn:
post_trip_tokens ≤ Bwith B = 5 in the default profile. This SLO is measured in the bench harness and exposed as:halt_tokens_p95,halt_tokens_p99,halt_latency_ms_p95on dashboards. - Independence — the watchdog must run in a context where it cannot be blocked by model code or tool plugins; it can close the user‑facing stream even if main eval is hung or adversarial.
- Fail‑closed semantics — if watchdog state is uncertain (e.g. loss of heartbeat, stalled detectors), the default action is halt stream and move to Degraded Safe Mode rather than continuing.
- Symmetry: streaming vs batch — the halt budget applies to streaming (tokens post‑trip ≤ B) and batch (once trip is detected, the entire response is treated as uncommitted and replaced by a safe degraded response).
- Observability — each watchdog event must be visible via FHR (
watchdog_trip,timing_incidents,failure_codesincludingWATCHDOG_TRIP) and dashboards (halt_tokens_post_trip, PR/EWMA/CUSUM).
Configurable degrees of freedom
Things a deployment can tune without breaking AD‑09:
- Budget B: can be set stricter (e.g. B = 0 for ultra‑sensitive deployments) or slightly looser in constrained environments, but: default profile must use B ≤ 5; any deviation must be documented as part of the timing profile and recorded in FHR.
- Hysteresis / debounce: to reduce false positives, you can require K consecutive detector ticks or combined signals (e.g. TIG breach + Δ‑triad anomaly). But once a hard trip is raised, the halt SLO still applies.
- Per‑profile overrides: some kernel profiles (e.g. MKP vs full) might enforce tighter budgets or more aggressive halting on sensitive routes; AD‑09 permits stricter variants.
You cannot:
- Eliminate the watchdog in favour of purely synchronous main‑loop halts.
- Relax the halt budget SLO while still claiming full timing‑channel mitigation compliant with AD‑10 [TC].
6. FHR & dashboards
6.1 FHR fields (recap)
AD‑10 [TC] / AD‑08 [SO] add timing fields including:
rts_mode,latency_bucket,mi_latency_content_est,tig_breach: bool,watchdog_trip: bool.timing_incidents[]and failure codes such asTIMING_COVERT_CHANNEL,TIMING_DISTRIBUTION_ANOMALY,COMPUTE_BUDGET_EXCEEDED, andWATCHDOG_TRIP.
Implementations MAY add:
halt_tokens_post_trip— number of tokens emitted after the trip in this event.halt_latency_ms— wall‑time between trip and stream close.
6.2 Dashboards & sampling
Per the SE/observability card:
- Dashboards include "halt tokens, p95 overhead, PR/EWMA/CUSUM, FP rates, tool aborts, ring alerts, MRP states".
- Sampling policy: 100% of incidents (including watchdog trips); 1–5% of normal runs, with privacy filters, hash‑linked to FHR.
7. MKP view
For Minimal Kernel Profile (MKP) (AD‑18 [SE] / AD‑19 [CI]):
- The watchdog must still exist and enforce the same ≤ 5 token post‑trip budget; this is part of MKP's security story.
- MKP may restrict which streams can be long; rely more heavily on batch mode for sensitive routes; but cannot drop the watchdog or the halt SLO.
8. Interfaces & cross‑links
- AD‑10 [TC] — timing breach semantics and TIG/RTS definitions that trigger watchdog.
- AD‑18 [SE] — TURN‑ACID, router fencing, perf scheduling, and watchdog thread specification.
- AD‑08 [SO] — FHR schema hosting watchdog and timing fields; dashboards and sampling.
- AD‑05 [MRP] — Degraded Safe Mode behaviour post‑halt.
- AD‑12 [OP] — incident, rollback, and gate‑flap runbooks that use watchdog events as triggers.
- AD‑19 [CI] — acceptance tests for streaming halt SLO under synthetic breaches.
- AD‑30 [SC] — watchdog state interacts with memory write restrictions during crisis states.
9. Acceptance: when can you say "AD‑09 is implemented"?
AD‑09 counts as implemented if:
- The system enforces HALT ≤ 5 tokens after any safety/timing breach, in both streaming and batch modes.
- An independent watchdog thread exists that can pre‑empt streaming even if main eval stalls and logs
WATCHDOG_TRIP. - FHR rows and dashboards expose halt tokens and watchdog activity through
halt_tokens_post_tripand the watchdog fields.
10. Implementation checklist (engineer view)
To wire AD‑09 end‑to‑end:
- Define trip sources — enumerate which signals are considered hard trips for the watchdog: TIG/RTS breaches; distributed I/O-integrity / exfiltration-control alerts; G1/H1 PEL or organisational NO‑GO gates.
- Implement watchdog process — independent thread/process: subscribes to trip events; tracks token index and stream handle; closes stream once
post_trip_tokens ≥ B. - Integrate with TURN‑ACID — on
WATCHDOG_TRIP: treat the turn as failed commit; roll back transactional state; enter Degraded Safe Mode per AD‑05 [MRP]. - FHR & logging — add
watchdog_trip,timing_incidents, optionalhalt_tokens_post_trip,halt_latency_ms. Ensure incidents logWATCHDOG_TRIPwith trip cause and token count. - Bench harness & tests — extend the bench harness to simulate late trip events during streaming; measure
post_trip_tokensand halt latency; assertpost_trip_tokens ≤ Bacross scenarios. Add CI acceptance tests under AD‑19 [CI]: "Streaming halt SLO": under synthetic breaches, p95 post‑trip tokens ≤ 5. - Dashboards — add visualisations for
WATCHDOG_TRIPfrequency; halt tokens distribution; overhead vs baseline (p95 latency, FP burden with hysteresis/debounce).