OS error 5 when adding lots of prefixable unique words to an index
Describe the bug
When indexing documents that contain a lot of long words (e.g. 1660534213676785670), the FST that contains the words and that is used by Meilisearch to compute prefix grows a lot. This FST is stored as a single value inside of a LMDB table.
LMDB values are limited in theory to about 2^32 bytes, but in practice depending on the OS it could be less than 2^31 bytes (see these issues for Linux and macOS), so the unbounded FST growth can result in EIO being returned by LMDB as the value reaches the maximum size permitted in practice.
This issue is one of the issues encountered in https://github.com/meilisearch/meilisearch/issues/3654.
To Reproduce Steps to reproduce the behavior:
- Set the maximum number of opened files to a huge number:
ulimit -Sn 10000 - Write the following script to
gen_docs_bigid.py:
import random
random.seed(42)
big_id = 1_660_534_213_676_785_670
big_value = 1_000_000_000_000_785_670
big_random = 10_000_000_000_000_785_670
for i in range(0,100_000_000):
print(f'{{ "id": "{big_id + i + random.randint(big_random, big_random + big_id)}", "value" : "{big_value + random.randint(big_random, big_random + big_id) + i}" }}')
- Run the script:
python3 gen_docs_bigid.py > big_docs.njson - Start Meilisearch with a HTTP payload size of 10G:
meilisearch --http-payload-size-limit 10GB - Add the document to an index:
curl \
-X POST 'http://localhost:7700/indexes/big_fst/documents' \
-H 'Content-Type: application/x-ndjson' \
-T big_docs.njson
- Wait for the indexing to process
- When the task is no longer processing, check in the tasks that it failed with OS error 5
The FST that Meilisearch failed to write to LMDB contains about 200M entries at this point.
Expected behavior Indexing succeeds without errors
Meilisearch version: v1.1.1, but not new in this version
Source: meilisearch/meilisearch