Running tasks with Vault templates should survive Vault outages using last-rendered secrets

Proposal

When Vault is unreachable, Nomad should keep already-running tasks alive and keep using the last successfully rendered template/env values. Secret sync can stop; the workload shouldn't.

Right now a Vault outage can still cascade into restarts / failed templates / pending allocs for anything that depends on vault + template. vault_retry { attempts = 0 } helps by waiting longer, but it's not the same as "stay up on last known good secrets."

Background

This is related to #11209. That issue described the same class of outage (Vault 503 → jobs with templates restart / get stuck pending). It was closed by #11606 exposing consul-template knobs (vault_retry, etc.).

That improvement is useful and we use it (or plan to). It doesn't fully address the operator need from #11209: don't convert a Vault outage into a full application outage for tasks that already have secrets on disk / in process env.

Use-cases

We run a large number of long-lived Nomad jobs that inject app config/secrets from Vault via templates (env = true, continuous render). Vault is a dependency for changes and for new starts, which is fine. What hurts is:

1. Vault blips / network partitions / maintenance.
2. Template watches or token renewal fail.
3. Tasks that were healthy get restarted or fail.
4. They can't start again until Vault is back → wide blast radius even if the apps themselves were fine with the last env.

We've hit this more than once. Prefer degraded mode: no secret updates, apps keep running, only deploys/restarts that need a fresh render are blocked.

Attempted Solutions

• template { change_mode = "noop" } — helps for secret content changes, not for "Vault is down / render fails."
• Client template.vault_retry with high/unlimited attempts — delays failure, doesn't define last-good-secret semantics; also doesn't help "I need a clear emergency behavior" under prolonged outage.
• Making Vault HA — necessary, but doesn't remove the hard dependency on live Vault for template health.

Possible directions (not prescriptive)

Something like:

• Client or job option: on Vault error after successful initial render, log + keep last file/env, don't fail the running task.
• Or explicit "persist last render" / emergency mode (similar ideas were floated on #11209).
• Clear docs on what is guaranteed during a Vault outage for running vs new allocs.

Happy to provide more detail on our jobspecs / client config if useful (redacted).

Nomad version

Nomad v2.0.3

Reproduction (high level)

1. Job with vault {} + template reading Vault (env = true, once = false).
2. Start job successfully.
3. Take Vault offline / return 503.
4. Observe template/renew paths eventually disrupting running work or blocking restarts; compare to desired "keep last secrets, stay up."