caddy-vhosts: requests are routed before healthchecks pass on docker-local, contrary to docs

Disclaimer: This was found by Claude Opus 5.

Description

docs/networking/proxies/caddy.md#L35 states:

Requests are routed as soon as the container is running and passing healthchecks.

On the docker-local scheduler the second half of that sentence is not enforced, and as far as I can tell there is no code path that could enforce it. Caddy begins receiving traffic for the new container as soon as it starts, while healthchecks are still pending.

Why

The caddy integration works by attaching caddy / caddy.reverse_proxy labels to the app container, and those labels are baked in at container creation time. In plugins/scheduler-docker-local/bin/scheduler-deploy-process-container the order is:

• L27 — docker-args-process-deploy trigger runs, and caddy-vhosts appends the caddy labels to DOCKER_ARGS
• L74 — fn-scheduler-docker-local-start-app-container creates the container with those labels
• L77 — docker container start "$cid"
• L101 — plugn trigger check-deploy finally runs the healthchecks

caddy-docker-proxy picks the container up from the docker event within a couple of seconds of L77, so it is already an upstream by the time L101 begins.

This cannot be worked around from the healthcheck side, because caddy-docker-proxy does not consider container health at all. generator/containers.go builds upstreams purely from labels and ingress-network membership — there is no reference to container state or health status in that path. So a Docker HEALTHCHECK on the image does not keep the container out of the upstream pool either.

The readiness healthcheck type looks like the intended mechanism ("temporarily suspends traffic delivery to the container until recovery occurs"), but it appears to be k3s-only: searching this repository, readiness occurs in exactly one file, docs/deployment/schedulers/k3s.md, and nowhere under plugins/scheduler-docker-local. That makes sense for k3s, where a failing readiness probe removes the pod from the Service endpoints; there is no equivalent mechanism on docker-local.

Impact

Every zero-downtime deploy has a window, equal to the app's boot time, where caddy round-robins between the old container and a new one that is not listening yet. Requests landing on the new container fail.

Observed on a Node app with a ~4s boot: 31 × 502 with dial tcp <new-container-ip>:5000: connect: connection refused in a single deploy, all within 4 seconds of container start, ending the moment the app began listening. Apps with slower boots would see a proportionally larger window.

It is survivable — the old container is still up during DOKKU_WAIT_TO_RETIRE — but only because caddy happens to retry onto it if you configure it to. Adding lb_try_duration / lb_try_interval / fail_duration to the reverse_proxy block (via docker-options:add ... deploy "--label caddy.reverse_proxy.lb_try_duration=5s", or caddy:labels:add on newer versions) takes the same deploy from 31 errors to 0, since dial failures against the not-yet-ready container get retried onto the still-running old one. That is a workaround, not a fix, and it silently stops working if an app's boot time ever exceeds the retire window.

Possible resolutions

1. Correct the docs to say requests are routed as soon as the container is running, and note that healthchecks do not gate routing on docker-local.
2. Apply the caddy labels after check-deploy passes rather than at container creation — e.g. attach them in a post-check step. Docker does not allow mutating labels on an existing container, so this likely means creating the container without the routing labels and adding them via a short-lived mechanism, which may not be practical.
3. Ship sensible lb_try_duration / fail_duration defaults in the caddy-vhosts generated labels, so the overlap window is retried through transparently.

Option 1 alone would at least stop the docs from promising something the scheduler does not do.

Version

Behaviour observed on 0.35.12; code references above are against master at 667abddd13a2869edb89b638337e773803ced5be, where the ordering is unchanged.