#140656 Structured GPU device health reporting in device manager

What would you like to be added?

A structured, backward-compatible health signal in the device plugin API so Kubelet can distinguish DEGRADED (ECC error, thermal throttle, NVLink down) from UNHEALTHY (gone), and surface it via a metric + optional probe.

Today the device manager treats a device as healthy purely based on whether ListAndWatch reports it in the healthy list. There is no queryable per-device health reason, so GPU degradation silently corrupts training workloads and out-of-tree operators (NVIDIA GPU Operator, Fugoku GPU device plugin) reimplement health checks.

Proposed change (additive, v1beta1):

New DeviceHealth message: state (HEALTHY/UNHEALTHY/DEGRADED), reason (e.g. "ecc_error"), last_transition_time.
Optional health_detail field on Device (nil for old plugins -> unchanged behavior).
New Kubelet metric kubelet_device_health_state{resource,device,state}.
Optional HealthCheck unary RPC (Kubelet skips if plugin doesn't implement).

Design doc: https://github.com/richardokonicha/kubernetes/blob/feature/gpu-device-health-reporting/pkg/kubelet/cm/devicemanager/README.md (draft)

Why is this needed?

AI/ML workloads need a standard, upstream health signal instead of every GPU operator reinventing it. Reduces duplication and gives schedulers/operators a uniform signal for drain/cordon/reschedule.

Comments welcome on:

Additive proto approach (backward compat) — acceptable?
Metric name / labels
Whether HealthCheck RPC is worth the surface area vs passive ListAndWatch only

/sig node /kind feature /area kubelet