FoundationCache holds global lock across heavy model loads, serializing concurrent pipeline initialization

Summary

FoundationCache performs heavy model/file loading while holding its single global threading.Lock, serializing all concurrent pipeline loads and blocking even unrelated cache entries for the duration of a BERT/charlm/pretrain load.

Location

• File: stanza/models/common/foundation_cache.py
• Class: FoundationCache
• Methods: load_bert_with_peft(), load_charlm(), load_pretrain() — each does with self.lock: wrapping bert_embedding.load_bert() / CharacterLanguageModel.load() / Pretrain(filename)

Relevant code path (static analysis of current main):

with self.lock:
    if transformer_name not in self.bert:
        model, tokenizer = bert_embedding.load_bert(transformer_name, ...)  # seconds–minutes of IO/compute under lock
        self.bert[transformer_name] = BertRecord(model, tokenizer, {})

Same pattern in load_charlm (CharacterLanguageModel.load under lock) and load_pretrain (Pretrain(filename) under lock). One lock guards all three dicts (bert, charlms, pretrains).

Problem

The lock's intent (per docstring/comment) is to protect dict mutation ("Uses a lock for thread safety", "future proof ... when the GIL is finally gone"). Instead it is held across:

1. HuggingFace/IO-bound transformer loads (load_bert), often 10s+ including downloads and weight deserialization.
2. CharLM and pretrain file reads.

Consequences:

• Thread A loading bert-large blocks thread B that only wants an already-cached (or different) model, or even a lightweight pretrain vector file, because they share self.lock.
• Startup of multi-pipeline servers (e.g. loading EN + DE + FR pipelines in threads) is fully serialized on model IO, defeating the cache's purpose.
• A slow/hanging download while holding the lock stalls every other foundation-cache user with no ability to make progress on independent entries.

Trigger / Reproduction

Based on static analysis (no multi-threaded model download executed):

1. Share one FoundationCache across threads (standard multi-pipeline setup).
2. Have thread 1 call load_bert("large-model") (cache miss → slow load under lock).
3. Have thread 2 call load_pretrain("other-file") or load_bert("cached-model") concurrently — it blocks on with self.lock until thread 1's IO completes, even though the entries are independent.

Note: this is a static-analysis finding; I did not run concurrent multi-GB model loads.

Expected Behavior

Lock should protect only the cache-dict check-and-publish, not the IO: check outside, load outside, re-check inside and publish (double-checked locking), so independent loads proceed in parallel and only duplicate loads of the same key are coalesced.

Actual Behavior

All foundation loads are globally serialized on one lock, including cross-type blocking (BERT load blocks pretrain/charlm access).

Impact

• Increased wall-clock startup for concurrent pipeline initialization; worst case one stalled download stalls all pipelines.
• Under free-threaded Python (the code's stated future-proofing goal), the contention becomes even more pronounced since IO no longer holds the GIL as a backstop.

Suggested Direction

• Restructure each loader to double-checked locking: snapshot self.bert.get(key) under lock (or lock-free read with copy), release, perform the heavy load_* outside, then re-acquire to publish with a re-check to avoid duplicate work. Consider per-key locks or concurrent.futures-style in-flight futures if duplicate-load suppression matters. No API change needed.

Evidence

• Source via API: foundation_cache.py shows all three loaders wrapping heavy constructors in with self.lock, with a single shared threading.Lock across bert/charlms/pretrains.
• Duplicate check: issue search for foundation cache lock thread returns total_count: 0, and open issues contain no lock-contention report (nearest MWT-serialization issues are unrelated) — no apparent duplicate. Non-security performance/correctness finding, so public issue is appropriate per SECURITY.md (which reserves private disclosure for vulnerabilities).

Describe the bug

Global lock held across model/file IO in FoundationCache, serializing concurrent loads as detailed above.

To Reproduce

Code-inspection path plus threaded-load scenario described in Trigger/Reproduction (static analysis; no live download run).

Expected behavior

Double-checked locking so IO runs outside the critical section.

Environment (please complete the following information):

• OS: N/A (code-path finding, OS-independent)
• Python version: N/A (any; affects threaded use on all versions, worse without GIL)
• Stanza version: current main (verified via API, stanza/models/common/foundation_cache.py)

Additional context

Suggested fix preserves the existing public FoundationCache API and NoTransformerFoundationCache behavior; only lock scope changes.

Classification

• FACT: heavy load_bert/CharacterLanguageModel.load/Pretrain constructors execute inside with self.lock on a single shared lock (verified in source via API).
• INFERENCE: concurrent independent loads are therefore fully serialized and vulnerable to one slow load stalling all.
• HYPOTHESIS: moving IO outside the lock with re-check publish restores parallelism without breaking cache semantics.