Add CPU-side Kubernetes image-readiness preflight for VERL training

Problem

In split CPU/GPU deployments, VERL can start and allocate GPUs before knowing whether the CPU-side Kubernetes nodes contain the images required by each dataset row. *This could casue hang for a long time for training **

SWE-Smith :openai images are built and tagged locally by pull_images.py; they are not pushed to a registry. The Job template uses imagePullPolicy: IfNotPresent, so a missing local image can leave the Pod in image-pull backoff until the configured 1,800-second rollout deadline.

Failure mode when an image is missing

The failure is not detected before training starts:

1. Ray initializes and reserves all 8 GPUs.
2. The Controller creates rollout Jobs on the CPU Kubernetes node.
3. Because the required :openai image is not present locally, Kubernetes attempts to pull it due to imagePullPolicy: IfNotPresent.
4. These images are built locally and are not pushed to a registry, so the Pod remains Pending, typically with ErrImagePull or ImagePullBackOff.
5. The rollout produces no trajectory, while the trainer continues waiting for the batch to complete.

**6. The failure may only surface at the configured 1,800-second rollout deadline—up to 30 minutes per affected batch. **

With train_batch_size=8 and rollout.n=8, one unavailable dataset row can create eight blocked rollout Jobs. This makes the problem appear as a long training hang while allocated GPUs remain idle or underutilized.

In our readiness scan, 190/6,343 training rows and 126/474 validation rows required unavailable images. Without preflight filtering, all of these rows could enter this failure path.

Experiment configuration

model: Qwen/Qwen3.5-9B topology: 8-GPU trainer + 1 CPU Kubernetes node train_rows: 6343 validation_rows: 474

trainer.n_gpus_per_node: 8 trainer.total_training_steps: 800 trainer.val_before_train: false trainer.test_freq: 100

data.train_batch_size: 8 actor_rollout_ref.rollout.n: 8 actor_rollout_ref.actor.ppo_mini_batch_size: 8 actor_rollout_ref.rollout.gpu_memory_utilization: 0.50

agentlightning.async_rollout.enabled: true agentlightning.async_rollout.async_train_batch_size: 16 agentlightning.rollout_timeout_seconds: 1800

Rendered image pattern:

image: "{{ input.image_name }}:openai"
imagePullPolicy: IfNotPresent

## Proposal

Add an opt-in agentlightning.k8s.filter_unavailable_images setting, disabled by default.

The Controller publishes a leased CPU-node image inventory. Before ray.init(), the trainer renders the actual Job template and filters unavailable
train and validation rows. The Controller rechecks readiness immediately before creating guarded Jobs.

This changes no dataset files and transfers no images.

## Prototype validation

- Image-readiness tests: 51 passed
- Full test suite: 140 passed
- Disabled mode performs no readiness requests and preserves existing behavior

Would the maintainers accept this in core, or prefer it under contrib/?