[Bug]: OpenAI-compatible dense embedder has no request timeout - a stalled embedding endpoint blocks each call for 30-90 min instead of failing fast

Issue Origin

Observed or reproduced in a real environment

Bug Description

The OpenAI-compatible dense embedder builds its clients without a request timeout, and no timeout can be configured for the embedding path.

openviking/models/embedder/openai_embedders.py constructs every client with only api_key, base_url and default_headers, plus a custom httpx pool:

# lines ~143-167 (sync) and ~328-339 (async)
self._client_kwargs: Dict[str, Any] = {"api_key": self.api_key or "no-key"}
...
self.client = openai.OpenAI(
    http_client=openai.DefaultHttpxClient(limits=self._http_limits()),
    **self._client_kwargs,          # <- no timeout, no max_retries
)
...
return openai.AsyncOpenAI(
    http_client=http_client,
    **self._client_kwargs,
)

Consequences:

1. The bound is the OpenAI SDK default. With openai==2.24.0 that is Timeout(connect=5.0, read=600, write=600, pool=600) and max_retries=2 → 3 attempts × 600 s = 1800 s per embeddings.create() call.
2. It stacks with OpenViking's own retry layer. "timeout" is listed in TRANSIENT_API_ERROR_PATTERNS (openviking/utils/model_retry.py:144), so retry_async(..., max_retries=self.max_retries) (openviking/models/embedder/base.py:395, default 3) retries the whole SDK call → worst case ≈ 5400 s (~90 min) for a single embedding.
3. max_retries from config is silently ignored on this path. DenseEmbedderBase.__init__ reads it (base.py:264) and the wrapper uses it, but it is never forwarded to the SDK client, so the SDK's own retries (2) are added on top of the wrapper's (3) instead of being disabled.
4. No knob exists to fix it. DenseEmbedderBase.__init__ reads only max_input_tokens, max_retries, max_concurrent, provider — there is no timeout key for embedding.dense, and the client construction does not forward one either.

Why this hurts specifically here: the embedder is a blocking dependency of memory extraction (session_commit), add_resource ingestion and recall. A slow or wedged embedding endpoint (very common for the self-hosted OpenAI-compatible servers this provider exists for: TEI, vLLM, Ollama, LiteLLM, LM Studio) turns every embedding into a 30–90 minute block instead of failing fast. The circuit breaker (failure_threshold=5, reset_timeout=300 s) can only act after those multi-minute calls finally return, and every half-open probe can hang for another 30 min. In our deployment the queue backed up and memory extraction/recall was effectively stalled for ~2 days, with OpenAI async embedding slow call ... duration_ms=1801254.93 repeating.

Steps to Reproduce

1. Configure a self-hosted OpenAI-compatible embedding server:

embedding:
  dense:
    provider: openai
    api_base: http://<embedding-host>:8081/v1
    api_key: dummy
    model: jinaai/jina-embeddings-v2-base-de
    dimension: 768

1. Make the endpoint accept TCP connections but never answer /v1/embeddings (a backend that silently degraded to CPU under load, or any wedged server that still holds the port).
2. Trigger any embedding: session_commit memory extraction, add_resource, or a recall query.
3. Observe a single embedding call blocking for ~30 min, then a retry, then the circuit breaker warning.

Expected Behavior

Embedding requests are bounded by a configurable timeout with a sane default (the sibling backends in the same package use 60 s), so an unresponsive embedding endpoint produces a fast, retryable error that the existing retry and circuit-breaker layers can act on within seconds — not hours.

Actual Behavior

A single embedding call hangs for up to 1800 s (observed 1801.26 s), and with the retry wrapper up to ~5400 s. Meanwhile the embedding queue backs up, memory extraction and recall stall, and the circuit breaker is nearly useless because its probes can hang for the same 30 minutes.

Minimal Reproducible Example

import openai
from openai._constants import DEFAULT_TIMEOUT, DEFAULT_MAX_RETRIES

print(openai.__version__, DEFAULT_TIMEOUT, DEFAULT_MAX_RETRIES)
# 2.24.0 Timeout(connect=5.0, read=600, write=600, pool=600) 2

# OpenViking builds the client exactly like this (no timeout=):
client = openai.OpenAI(base_url="http://127.0.0.1:9999/v1", api_key="dummy")
# point base_url at a socket that accepts but never replies:
#   nc -lk 9999
# -> the call returns after ~3 * 600 s with APITimeoutError
client.embeddings.create(model="m", input="x")

The same call with timeout=60 fails in ~60 s, or ~180 s with the SDK's own retries.

Error Logs

# openviking.models.embedder.openai_embedders
WARNING - [request_id=...] OpenAI async embedding slow call provider=openai model=jinaai/jina-embeddings-v2-base-de wait_ms=0.00 duration_ms=1801254.93

# openviking.storage.collection_schemas
WARNING - Failed to generate embedding: OpenAI API error: Request timed out. (uri=viking://user/.../memories/profile.md)
WARNING - Embedding circuit breaker is open; re-enqueueing messages

OpenViking Version

0.4.20 (server /health reports v0.4.20). Still present on main at 2b351a9 — grep -c timeout openviking/models/embedder/openai_embedders.py → 0.

Python Version

3.13

Operating System

Linux

Model Backend

Other (self-hosted OpenAI-compatible embedding server, TEI serving jinaai/jina-embeddings-v2-base-de, configured via embedding.dense.api_base)

Additional Context

Timeout handling per backend (current main) — the OpenAI-compatible embedder is the outlier, and it is the one used for every self-hosted endpoint:

So the VLM path already exposes a timeout through config (timeout: 180 works there), while the embedding path cannot be bounded at all — even though it is the path most likely to talk to a self-hosted, occasionally-slow server.

Suggested fix (small and contained):

1. DenseEmbedderBase.__init__: self.timeout = float(self.config.get("timeout", 60.0)) next to the existing max_retries / max_concurrent reads.
2. openai_embedders.py: put the value into _client_kwargs so it applies to OpenAI, AsyncOpenAI, AzureOpenAI and AsyncAzureOpenAI; also forward max_retries (or hard-code max_retries=0) so the SDK does not stack its own retries on top of retry_async — today a single embedding can consume up to 3 (wrapper) × 3 (SDK) attempts.
3. Document the key in docs/en/guides/01-configuration.md (and keep the 60 s default consistent with the sibling backends).

Related issues (same class of missing/unbounded waits, different layers — none of them covers the embedder HTTP client):

• #4341 — AddResource tasks hang forever, wait_for_descendants with no timeout
• #4919 — infinite retry / deadlock of the semantic ingest queue on misclassified permanent errors
• #2698 (closed) — PathLock timeout=None waited forever instead of falling back
• #4159 — silently degrading dependency failures to empty recall results

Happy to open a PR for the three changes above if the approach is acceptable.