Auto-embedding field is silently and permanently dropped from a collection's schema when remote model validation fails during collection load (server restart)

Bug Description

When a collection has a field with embed pointing to a remote / custom OpenAI-compatible embedding model (e.g. a self-hosted vLLM instance), and the Typesense server restarts, CollectionManager::load() re-validates the remote model for every collection found on disk. If that validation call fails for any reason (the remote endpoint being briefly unreachable, a timeout, or the remote server rejecting an explicit dimensions parameter), Typesense does not retry and does not fail the collection load — it just logs one ERROR line and silently omits the field entirely when reconstructing the collection's in-memory field list. The field then no longer exists: it doesn't show up in GET /collections/:name, vector search on it stops working, and newly indexed documents get no embedding — while every other field and all existing documents load completely normally. There is no retry, no way to observe this failure via the API, and no configuration flag to change this behavior.

Root cause is in CollectionManager::load():

• v30.2: src/collection_manager.cpp lines 134-150
• v31: src/collection_manager.cpp lines 238-254

if(field_obj.count(fields::embed) != 0 && !field_obj[fields::embed].empty()) {
    size_t num_dim = field_obj[fields::num_dim];
    auto& model_config = field_obj[fields::embed][fields::model_config];

    auto res = EmbedderManager::get_instance().validate_and_init_model(model_config, num_dim);
    if(!res.ok()) {
        const std::string& model_name = model_config["model_name"].get<std::string>();
        LOG(ERROR) << "Error initializing model: " << model_name << ", error: " << res.error();
        continue;   // <-- skips constructing/pushing this field for the rest of this process's lifetime
    }
    ...
}

field f(field_obj[fields::name], ...);   // unreachable for the failed field
...
fields.push_back(f);                      // this field is simply never added

When validate_and_init_model(...) returns a non-ok Option, the continue right after the LOG(ERROR) skips straight to the next field in the loop — the field f(...) construction and the subsequent fields.push_back(f) are never reached for that field. It stays absent from the reconstructed Collection object until a manual schema PATCH re-adds it, even though its raw vector data is still safely persisted on disk for every existing document.

Two contributing factors we confirmed in source:

1. The HTTP call inside validate_and_init_model (OpenAIEmbedder::is_model_valid in src/text_embedder_remote.cpp) uses hardcoded constants from include/http_proxy.h (default_timeout_ms = 60000, default_num_try = 2) — not exposed via any CLI flag/env var, so there's no way to add retries/backoff specifically for this startup validation path.
2. dimensions is sent unconditionally once num_dim is known (src/text_embedder_remote.cpp ~line 128: if(has_custom_dims && model_name != "openai/text-embedding-ada-002") { req_body["dimensions"] = num_dims; }). has_custom_dims becomes true as soon as num_dim is non-zero, which it always is after the first successful auto-detection. So every later collection load sends an explicit dimensions value — if the remote OpenAI-compatible server doesn't support Matryoshka-style truncation for that model, it rejects the request, which lands in the same !res.ok() → continue path and drops the field.

Reproduction Steps

This uses a minimal local mock OpenAI-compatible embeddings server so the failure (remote endpoint unreachable at collection-load time) is 100% deterministic instead of depending on a real network race.

#!/usr/bin/env bash
set -e

TYPESENSE_HOST="http://localhost:8108"
API_KEY="xyz"
CONTAINER_NAME="typesense"   # name of your running Typesense container

# 1. Start a minimal OpenAI-compatible embeddings mock server on port 9000
cat > /tmp/mock_embedder.py <<'PY'
import http.server, json

class Handler(http.server.BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get('Content-Length', 0))
        self.rfile.read(length)
        body = json.dumps({"data": [{"embedding": [0.1, 0.2, 0.3, 0.4]}]}).encode()
        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.send_header('Content-Length', str(len(body)))
        self.end_headers()
        self.wfile.write(body)

http.server.HTTPServer(('0.0.0.0', 9000), Handler).serve_forever()
PY
python3 /tmp/mock_embedder.py &
MOCK_PID=$!
sleep 1

# 2. Create a collection with an auto-embedding field pointing at the mock server
curl -s -X POST "$TYPESENSE_HOST/collections" \
  -H "X-TYPESENSE-API-KEY: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "repro_collection",
    "fields": [
      { "name": "texto", "type": "string" },
      {
        "name": "embedding",
        "type": "float[]",
        "embed": {
          "from": ["texto"],
          "model_config": {
            "model_name": "openai/mock-model",
            "api_key": "not-required",
            "url": "http://host.docker.internal:9000"
          }
        }
      }
    ],
    "default_sorting_field": ""
  }'

# 3. Index a document — embedding is generated correctly
curl -s -X POST "$TYPESENSE_HOST/collections/repro_collection/documents" \
  -H "X-TYPESENSE-API-KEY: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "texto": "hello world" }'

echo "--- schema BEFORE restart (embedding field present) ---"
curl -s "$TYPESENSE_HOST/collections/repro_collection" -H "X-TYPESENSE-API-KEY: $API_KEY"

# 4. Simulate the remote embedding model becoming unreachable
kill "$MOCK_PID"

# 5. Restart the Typesense server
docker restart "$CONTAINER_NAME"
sleep 5

echo "--- schema AFTER restart, with the remote model down (embedding field is GONE) ---"
curl -s "$TYPESENSE_HOST/collections/repro_collection" -H "X-TYPESENSE-API-KEY: $API_KEY"

Expected vs Actual

Expected behavior Either: (a) Typesense retries the remote model validation a few times with backoff before giving up, since transient unreachability of a dependency at boot is common; or (b) if validation still can't succeed, the field definition and its already-stored vector data are preserved (marked degraded/unavailable) instead of being deleted from the in-memory schema, and this condition is surfaced through the API (e.g. /health or the collection response) — not just a single log line.

Actual behavior After the restart, GET /collections/repro_collection returns the collection without the embedding field at all:

{
  "name": "repro_collection",
  "num_documents": 1,
  "fields": [
    { "name": "texto", "type": "string", ... }
  ]
}

The only trace of what happened is this line in the server log:

E20260714 11:35:50.249936   490 collection_manager.cpp:141] Error initializing model: openai/mock-model, error: ...

The document itself and all other fields are intact; only the vector field silently disappeared. This repeats on every subsequent restart until a schema PATCH manually re-adds the field — which then triggers a full re-embedding pass over every existing document in the collection.

Environment

• Typesense version: v30.2 (Docker image typesense/typesense:30.2); confirmed the same code path is unchanged in v31 source
• Operating system: Ubuntu/Linux (Docker Compose deployment)
• Client library & version: N/A (reproduced via raw HTTP API / curl)

Schema / Configuration

{
  "name": "repro_collection",
  "fields": [
    { "name": "texto", "type": "string" },
    {
      "name": "embedding",
      "type": "float[]",
      "embed": {
        "from": ["texto"],
        "model_config": {
          "model_name": "openai/mock-model",
          "api_key": "not-required",
          "url": "http://host.docker.internal:9000"
        }
      }
    }
  ],
  "default_sorting_field": ""
}

Additional Context

This was first observed in production with a real self-hosted vLLM server serving Qwen/Qwen3-Embedding-0.6B, where the field disappeared after two independent triggers on different restarts:

1. DNS not yet resolvable for the remote host in the first instants of container boot (CURL failed. Code: 6, strerror: Couldn't resolve host name).
2. The remote server rejecting the dimensions parameter Typesense sends once num_dim is already known, with "Model ... does not support matryoshka representation, changing output dimensions will lead to poor results."

Both land in the same !res.ok() → continue path described above, producing the identical symptom (field silently gone).

We searched existing issues and didn't find one describing this exact "field permanently dropped on load failure via continue" mechanism. Closest related issues:

• #2737 — embeddings recomputed on every restart (a performance complaint, not a field-loss complaint)
• #2652 — local model download timeout during collection creation (fails the whole creation, rather than silently dropping a field during reload)