Proposal: make multi-model fallback progress through backup models (#3930)

Summary

Re-scope #3930 around a bounded, progressing multi-model fallback sequence rather than re-submitting the template-only repeated-redirect patch.

Current source evidence

At Console f8410432d450541baec3ae468e4d4a8a07392169, AiRouteServiceImpl.writeAiRouteFallbackResources creates one fallback route. RAND uses all fallback upstreams; SEQ constructs the list from fallbackConfig.getUpstreams().get(0) only. The EnvoyFilter already sets max_internal_redirects: 10, but matches only the original route.

Adding the same policy to the fallback route cannot establish progression to distinct models. With current SEQ resource construction, that route still contains only the first backup.

Recommended scope

Implement deterministic progression for existing SEQ: original -> backup 1 -> backup 2 -> ...; stop on success or exhaustion/limit. Preserve existing RAND semantics; do not claim random selection excludes previously failed providers.

A candidate design is generated per-stage routes/policies, each containing one backup and redirecting only to its successor. Preserve request body/path, configured failure-status behavior, per-stage model mapping and per-route security. Update/deletion must remove stale stages. Internal stage metadata must not trust client-supplied values or reset the attempt bound on re-entry.

Required runtime cases include all-failed termination, finite global bound, one/empty backups, body preservation and exact provider call order. Do not restart an SSE response after downstream bytes have been committed.

Scope decision

The choice between deterministic SEQ progression first and repeated random-pool fallback will be recorded as a blocking typed QUESTION. No Design/TASK or implementation advances until it is resolved by a maintainer. The recommendation is not approval.

Prior work and governance

July prototype eed1c0eda1e0912cb5ad23945d8ba0aa2db2f9c4 only extends template matching to the fallback route. Former Console PR #752 currently returns 404. Preserve it as history, not a complete multi-level fix.

Current contributor: pilichoumao; material AI assistance. Request explicit maintainer scope, Proposal and Design approval. No retroactive approval or maintainer exception is claimed.

Verification direction

Generated Console resources in a pinned real Higress cluster; mock model services return 429, 429, 200. Include all-fail, success-first, one/zero backups, request-body and client-forged-stage cases. Compare baseline/fixed call order/count, client response, access logs, config dumps and metrics. Pin all sources/images, publish machine-checked evidence and prove cleanup. Rendering tests alone are insufficient.