DEP: Make PowerAgent pod readiness reflect cap enforcement

Baselined on main at 81fa669fcb. Every behavioral claim and code reference below was read from that tree. This DEP has no prerequisite — the PowerAgent DaemonSet, its chart, POWER_ANNOTATION_KEY, the reconcile loop, and the actuator contract it modifies are all already on main.

Summary

Make a PowerAgent pod's Kubernetes readiness mean that the agent is enforcing power caps.

Today a PowerAgent pod has no readiness probe, so it is Ready from the moment its container starts and stays Ready while every reconcile cycle fails. This DEP adds three things and nothing else (§1–§4 below):

1. the actuator reports whether its writes actually succeeded, and reconcile_once() folds that into a whole-cycle enforcement boolean;
2. a fixed GET /readyz on container port 8081, serving that state; and
3. a readinessProbe on the PowerAgent DaemonSet.

No operator changes, no CRD or status changes, no Planner changes, no chart changes beyond the probe and port. The claim published is narrow and self-reported:

The agent's last reconcile cycle completed every enforcement action it was required to take with no reported failure, and completed recently.

A GPU with no running workload is intentionally skipped, so the claim covers the actions the cycle was required to take, not the live state of every GPU on the node.

Motivation

PowerAgent's failure handling is designed around not dropping caps: a failed write, an unconfirmed GPU identity, a failed PID discovery, a failed release, a failed pod list, and a failed device-count refresh all skip work and continue, preserving last-known-good state rather than exiting. That is the right runtime behavior and this DEP does not change it. The cost is that none of it is observable as state: the only external signal is apply_failures_total, a counter, which says a failure happened — not whether the agent is currently enforcing anything.

The gap is wider than a metrics inconvenience. apply_cap() returns the intended post-clamp value even when the NVML/DCGM write fails, and its sole caller (power_agent.py#L1648) discards the return value. So the loop never learns the outcome of its own writes; there is no in-process notion of "this cycle enforced" to expose. This DEP creates that notion, then exposes it.

Today, therefore, kubectl get pods shows 1/1 Running for an agent that has enforced nothing since startup, and kubectl rollout status reports success for a rollout of non-functional agents. Pod readiness is the standard place for "this instance is doing its job" and is already plumbed everywhere; populating it truthfully is the smallest change that closes the gap.

Current code fit

• run() has no per-cycle result and no watchdog, and sleeps RECONCILE_INTERVAL_S (15s) after each cycle returns — so the cycle period is the interval plus the cycle's own duration.
• reconcile_once() returns None. A failed pod list aborts the cycle early by design; otherwise every GPU is reconciled inside its own try.
• A GPU with no running processes returns early from _reconcile_gpu before any attribution or write, so idle GPUs are untouched.
• Actuator-init and GPU-discovery failures already abort in PowerAgent.__init__, producing CrashLoopBackOff.
• The image already installs prometheus-client and runs Python 3.12. The metrics server can be disabled with --prometheus-port=0, so readiness must not be served from it.
• The chart's DaemonSet has no probes of any kind today.

Proposal

1. Return a whole-cycle enforcement boolean

reconcile_once() returns True iff every enforcement action the cycle was required to take completed, and the inputs it depended on were obtained. Every path that currently swallows a failure sets it false: a failed cap write, a cap skipped on unconfirmed GPU identity or mid-write re-enumeration, a failed PID discovery, a failed release on a disowned GPU, a failed pod list, a failed device-count refresh.

No enforcement failure may be silently absorbed. One boolean is the whole contract because nothing branches on which class; attribution stays in logs and apply_failures_total. Two things are outside it. The persistence-only retries in _flush_pending_retirements and _flush_pending_acquisitions are excluded: each retries a durable record whose hardware write and in-memory ownership already succeeded, so live enforcement is correct and only a restarted agent would notice. And a cgroup read that yields no pod UID is indistinguishable from a non-Kubernetes process, so it is not a failure the cycle can observe at all (see Risks).

A cycle short-circuited by _shutdown returns the result accumulated so far, at both the fast path at the top of reconcile_once and the mid-loop break. It is not special-cased to True. The GPUs the short-circuit skipped were not actions the cycle was required to take, so a shutdown that interrupts an otherwise clean cycle still returns True and does not flip a pod NotReady during its own termination grace period. But a shutdown landing after an earlier GPU already failed must not discard that failure: an unconditional True would publish a fresh successful timestamp at the exact moment _shutdown_cleanup starts restoring default caps, so /readyz would claim enforcement while enforcement is being torn down. Returning the fold costs nothing and needs no special case.

apply_cap() returning normally is not success. Both actuators catch NVMLError/DCGMError, increment apply_failures_total, and return the intended post-clamp int anyway — the documented contract is explicitly "regardless of whether the underlying write succeeded." This cannot be fixed at the call site by inspecting the return value; it requires changing the actuator's result contract so the write outcome reaches the caller. The same holds for restore_default(), which already returns False for an intentional skip that leaves a cap live.

Idle GPUs are never written and contribute nothing either way. A failing cycle must not exit the process: a NotReady agent that keeps reconciling can recover, while a restart drops in-memory GPU ownership.

2. Readiness rule

Readiness is one monotonic timestamp, _last_good_cycle, initialized to 0.0 before the server thread starts and assigned once per cycle thereafter: the current time.monotonic() when the cycle succeeded, 0.0 when it failed or raised. Ready iff it is non-zero and its age is under:

STALE_AFTER_S = 3 * RECONCILE_INTERVAL_S   # 45s at the default interval

One value carries the whole state — 0.0 covers both "no cycle has ever succeeded" and "the last one failed", and the age covers "the last one hung". Because it is a single assignment computed after the try, a reader can never see a torn mix, an exception cannot leave a stale successful timestamp in place, and no lock is required.

The staleness bound catches a hung cycle, which nothing else can: run() has no watchdog, NVML and DCGM calls are unbounded, and a blocked cycle keeps serving whatever the last one published — the probe's failureThreshold cannot see that, because nothing is failing, it is simply not advancing. The bound restarts nothing.

The multiplier of 3 is what makes that safe. Since run() sleeps after each cycle, the timestamp refreshes every RECONCILE_INTERVAL_S plus the cycle's own duration, so a bound of 3 × RECONCILE_INTERVAL_S tolerates a healthy cycle lasting up to 2 × RECONCILE_INTERVAL_S — about 30s at the default. A cycle that exceeds that exposes a 503 for the overshoot, and a node whose cycles routinely run long enough for three consecutive polls to land inside that window — through a throttled pod LIST or a slow DCGM reconnect on a many-GPU host — will go NotReady with nothing failing, and wants a longer RECONCILE_INTERVAL_S rather than a looser bound. A one-off cycle that overruns by a few seconds only shows a brief 503.

No agent-side failure counter. failureThreshold: 3 is the sole debounce. It counts polls, not cycles, and run() sleeps after each cycle, so a failed result stays published for RECONCILE_INTERVAL_S plus the next cycle's duration. Three consecutive failing polls at periodSeconds: 10 span 20s, and the first of them lands 0–10s after the 503 window opens, so the pod goes NotReady 20–30s into that window. At a 15s interval a single failed cycle therefore takes the pod NotReady once the following cycle takes roughly 5–15s, depending on probe phase. The honest contract is that a brief failure is usually absorbed, not guaranteed to be; it depends on elapsed time and probe phase. Accepted rather than fixed — a counter would duplicate a debounce Kubernetes already provides, and the cost is a bounded spurious NotReady that self-heals on the next successful cycle plus one poll.

Note that the STALE_AFTER_S bound and failureThreshold compose rather than overlap: the bound decides when /readyz starts returning 503, and the threshold decides how long a 503 must persist before the pod is marked NotReady. For a hung cycle the two add — 45s of staleness plus 20–30s of failing polls — so the pod is NotReady roughly 65–75s after the last successful cycle.

3. GET /readyz on port 8081

A wsgiref.simple_server WSGI app on a daemon thread, at fixed port 8081 and path /readyz. 200 when §2 holds, 503 otherwise, with the age in the body so a 503 is diagnosable without log access. It takes a handler_class whose log_message is a no-op — WSGIRequestHandler inherits BaseHTTPRequestHandler's, which would write a line to stderr on every poll, roughly 8,600 a day per node at periodSeconds: 10.

Started in run() before its try, so a bind failure propagates and the agent never starts. Placing it inside would run the full _shutdown_cleanup restore sweep on the way out of a failure that never enforced anything. CrashLoopBackOff with a bind error is strictly better than a Running/NotReady process with no path to recovery.

Fixed rather than configurable, and deliberately not on the Prometheus server, which --prometheus-port=0 can disable — readiness must not be optional.

4. readinessProbe on the DaemonSet

readinessProbe:
  httpGet:
    path: /readyz
    port: 8081
  initialDelaySeconds: 15
  periodSeconds: 10
  failureThreshold: 3

No liveness probe. Defining an irrecoverably stuck agent wrongly restarts agents about to recover, and a restart is strictly worse than a NotReady agent that keeps reconciling.

This couples agent health to DaemonSet rollout progress, the one behavioral change for existing installs. An already-NotReady agent consumes the rollout budget under rollingUpdate.maxUnavailable: 1 and can stall a helm upgrade that previously completed. That is correct — an upgrade rolling over a node whose agent is not enforcing should not be reported as progress — but it must be in the release notes. The update strategy itself is unchanged.

Deployment and failure behavior

The kubelet does not probe for the first initialDelaySeconds: 15; the delay does not add to the first cycle, so a pod whose first cycle has already succeeded goes Ready on the first probe. Existing workers are never killed, evicted, or gated by this, and nothing outside the pod's own readiness reads the new state.

• Transient write failure: usually absorbed by the probe debounce and visible in apply_failures_total; see §2 for when it is not.
• Sustained failure, or an unexpected exception: the timestamp resets to 0.0, so /readyz returns 503 from the next poll and the pod is NotReady 20–30s later, after three failing polls. The loop keeps running and recovers without a restart.
• Hung cycle: the 45s bound trips, then three failing polls mark the pod NotReady — roughly 65–75s after the last successful cycle. A healthy cycle running longer than ~30s exposes a 503 for the overshoot and only reaches NotReady if the overshoot covers three consecutive polls; see §2.
• Pod terminating: a cycle short-circuited by _shutdown returns whatever it had accumulated, so a clean cycle keeps readiness steady through the grace period while a cycle that had already failed still reports that failure.
• Actuator-init or GPU-discovery failure: unchanged — fatal in __init__, not modeled by the probe.
• Port 8081 already bound: fatal at startup.
• --prometheus-port=0: /readyz is unaffected.

Explicit non-goals

This proposal does not:

• prove that a requested cap was applied to a particular pod or GPU;
• prevent a worker from starting before its local PowerAgent reconciliation;
• terminate or evict workers when PowerAgent fails;
• add a liveness probe, restart-policy change, or update-strategy change;
• publish per-node Leases, digests, GPU UUID sets, or per-pod enforcement records;
• change any operator, CRD, DGD status, or Planner behavior; or
• assert anything about nodes that have no PowerAgent pod at all.

Alternatives considered

• Leave readiness implicit and alert on apply_failures_total. A counter cannot express "currently not enforcing", cannot be consumed by Kubernetes, requires every operator to build the same alert, and cannot see the failure classes that never reach it.

Requirements

• Loop contract. reconcile_once() returns one boolean covering failed writes, identity skips, PID discovery, releases, pod list, and device-count refresh, with no class silently absorbed and both persistence-only retry queues excluded. The per-GPU fold must not short-circuit, and a _shutdown short-circuit returns the fold so far rather than an unconditional True. A failing cycle never exits the process.
• Write contract. apply_cap() and restore_default() report whether the write took, and the loop consumes that. No post-write readback. GPUs the cycle did not write are not asserted about.
• Readiness state. One monotonic timestamp, assigned once per cycle after the try; 0.0 on failure or exception. Ready iff non-zero and younger than 3 * RECONCILE_INTERVAL_S.
• Endpoint. Fixed GET /readyz on 8081, independent of the Prometheus server, 200/503, age in the body, no per-request access logging, bind failure fatal.
• Chart. readinessProbe with failureThreshold: 3 as the sole debounce, periodSeconds: 10, initialDelaySeconds: 15; container port added; no liveness probe; update strategy unchanged.

Implementation plan

1. Change the apply contract so the write outcome reaches the loop. On the NVML path this means plumbing it out of module-level _apply_cap, which today catches NVMLError, increments apply_failures_total, and returns None; NvmlActuator.apply_cap only re-derives the post-clamp value from constraints. Do the same for DcgmActuator.apply_cap and restore_default().
2. Return the whole-cycle boolean from reconcile_once(), folding per-GPU results without short-circuiting and leaving both persistence-only retry queues out.
3. Track _last_good_cycle in run() and serve it from a wsgiref app on /readyz:8081.
4. Add the probe and container port to the DaemonSet.
5. Note the rollout-coupling change in the chart README and release notes.

Test strategy

• /readyz is 503 before the first successful cycle, 200 immediately after, and 503 once the staleness bound elapses with no completed cycle.
• Each swallowed failure path drives the boolean false; each queued persistence-only retry leaves it true.
• An apply_cap() whose underlying NVML/DCGM write raised drives the boolean false, even though the call returns a plausible int — the regression test for "clean return is not success".
• A restore_default() returning False for an intentional skip fails the cycle.
• A failure on GPU 0 marks the cycle false and GPUs 1…N are still reconciled. A pod-list failure still aborts before the loop and marks it false.
• An idle GPU is never written and does not affect the boolean.
• A GPU falling back to safe_default_watts on a missing or invalid annotation still yields a successful cycle once that default is written.
• An unexpected exception from reconcile_once() resets the timestamp, flips /readyz to 503 on the next poll rather than waiting out the staleness bound, and does not terminate the process.
• A cycle short-circuited by _shutdown, at either the fast path or the mid-loop break, returns the accumulated result: True when nothing had failed, and False when a GPU had already failed before the short-circuit — the regression test for "shutdown does not launder a failure".
• A healthy cycle lasting longer than 2 × RECONCILE_INTERVAL_S makes /readyz return 503 — asserted so the budget in §2 is not silently changed by a future interval or bound edit.
• Stepping the wall clock does not extend or expire readiness.
• /readyz is served with --prometheus-port=0; binding an occupied 8081 is fatal; probe traffic emits no per-request log line.
• Helm: the DaemonSet renders the probe and port, and no liveness probe.

Deliberately not asserted: that one failing cycle cannot make the pod NotReady. Per §2 that is not a property this design has, and a test claiming it would encode a false guarantee.

Risks and limitations

• Readiness is the agent reporting its own write outcomes, not independent proof, and describes the previous cycle: a new worker may start before its cap is applied.
• An out-of-band override is corrected, not reported: every active GPU is rewritten every cycle.
• Readiness tracks the cap the agent resolved, not the one the user asked for. _resolve_cap_for_gpu falls back to safe_default_watts on a missing, unparseable, or contested annotation, and apply_cap clamps to the SKU's settable range — so a node enforcing safe defaults, or clamping a 5000W request to 700W, reports Ready. That is the right readiness answer; catching an unachievable request is admission's job.
• Detection latency is roughly 20–30s for a reported failure and 65–75s for a hung cycle, and the debounce is timing-dependent in both directions (§2), so readiness transitions are not a precise function of failure count. A node failing intermittently but never on consecutive cycles may retain readiness throughout; alert on apply_failures_total.
• The probe couples agent health to DaemonSet rollout progress and can stall a helm upgrade.
• Readiness asserts live enforcement, not durable recoverability: an agent whose persistence writes are failing stays Ready, risking the ownership record on an ungraceful exit.
• A node-wide /proc or permission problem would make cgroup attribution fail silently, since _extract_pod_uid_from_cgroup cannot distinguish an unreadable cgroup from a non-Kubernetes process. Left as-is: the container runs privileged, runAsUser: 0, hostPID: true, so this is not a failure mode this deployment has, and splitting the errno cases would add a flapping surface on every short-lived GPU process.

Deferred

Correlating this readiness with DGD worker placement — detecting a power-annotated worker that landed on a node with no operational PowerAgent, and feeding that into DGD convergence and Planner scaling — is deferred to a follow-up DEP. This DEP is a prerequisite for it and is independently useful without it.

Code references

Pinned to main at 81fa669fcb456054b3aee6557e6e80ced6783a2e.

PowerAgent — apply_cap failure contract · [