Deadlock in concurrent auto-embedding inserts with persistent model cache

Bug description

What happened:

Сoncurrent inserts into an auto-embedding table may hang indefinitely when using sentence-transformers/all-MiniLM-L6-v2

After one or more successful batches, the loader stops making progress. SHOW THREADS shows two Manticore workers stuck in INSERT queries, while other workers and SQL diagnostics remain responsive. Debuggertraces show the affected workers blocked in TextToEmbeddings_c::Convert() and embedding runtime threads waiting on futexes.

A persistent mounted model cache makes the issue reproducible, but it is not the only requirement: the same hang was reproduced with an identical cache stored in a Docker named volume.

What you expected to happen:

Steps to reproduce:

docker run --name manticore -v $(pwd)/cache:/var/lib/manticore/.cache/ -d manticoresearch/manticore
docker exec -it manticore manticore-load --quiet --json --init="CREATE TABLE test_scale_columnar ( 
    id BIGINT,
    title TEXT,
    description TEXT,
    category STRING ATTRIBUTE,
    age INT,
    salary FLOAT,
    discount FLOAT,
    is_active BOOL,
    last_login TIMESTAMP,
    product_codes MULTI,
    large_values MULTI64,
    additional_info JSON,
    vec FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2'
    MODEL_NAME='sentence-transformers/all-MiniLM-L6-v2'
    FROM='title, description'
) engine='columnar'" --load="INSERT INTO test_scale_columnar (id, title, description, category, age, salary, discount, is_active, last_login, product_codes, large_values, additional_info) VALUES (<increment>, 'Document title <increment>', 'Document description <increment>', 'cat_<increment>', <int/18/65>, <float/1000.0/100000.0>, <float/0.0/50.0>, <int/0/1>, <int/1609459200/1704067200>, (<array/1/5/100/999>), (<array/1/3/1000000/9999999>), '{\"rating\":<int/1/5>,\"verified\":<int/0/1>}')" --batch-size=500 --threads=2 --total=5000

If the bug doesn’t reproduce on the first attempt, stop the container, start it again, and run the manticore-load command.

Manticore Search version

Manticore Search 29.9.0+

Operating system version

Docker and CI

Have you tried the latest development version?

Yes — the issue still occurs

How does this bug affect you?

Development or testing is affected

How do you use Manticore Search?

No response

What do you use Manticore for?

No response

Company or project

No response

Approximate scale

No response

Can we learn more about your use case?

• Yes — feel free to @mention me in this issue if you'd like to ask more about how we use Manticore.

Internal Checklist

To be completed by the assignee. Check off tasks that have been completed or are not applicable.

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed