Global _HTTP_REQUEST_LOCK serializes all requests; a slow read blocks concurrent calls (HTTP transport)

Summary

On the HTTP transport, _HTTP_REQUEST_LOCK wraps the entire handle_request() dispatch, so every tool runs one-at-a-time even though the server is a ThreadingHTTPServer(daemon_threads=True). A single slow read therefore starves all concurrent requests.

Impact

With the Qdrant backend, aggregate reads like list_wings/status/get_taxonomy full-scan the collection (~180s on a large palace). While one runs it holds the global lock, so concurrent search/initialize/tools/list calls block until it finishes and time out. In practice the MCP client fails to connect while an aggregate call is in flight.

Root cause

The lock's own comment scopes its purpose to preventing concurrent Chroma/HNSW mutation, and the Chroma safety paths are gated by _is_chroma_backend(). So it's a Chroma/SQLite/HNSW-era safeguard that, on the thread-safe Qdrant backend, serializes reads with no shared embedded index to protect. Qdrant's client carries its own re-entrant lock and the server handles concurrent requests transactionally.

Suggested fix

Make the lock write-only: acquire _HTTP_REQUEST_LOCK only for mutating tools; reads plus initialize/ping/tools/list run lock-free. Two read paths touch shared module state and need their own narrow guards rather than the global lock — the _metadata_cache store in _get_cached_metadata, and the _collection_cache rebuild in _get_collection. The cross-process writer flock (_MCP_WRITER_LOCK) stays unchanged, so write-vs-write remains serialized. PR attached.

Verification

Authored and concurrency-tested on v3.5.0: a 5-assertion test passed first try — a search returned in 3.75s while a list_wings full-scan was still running (pre-fix this hung); 5 concurrent searches all <5s; concurrent aggregate reads race-free; two concurrent writes still serialize with no corruption; a read during an in-flight write returned in 0.196s. The attached PR ports that fix onto develop.