Fulltext Search High memory usage with 200K+ documents during rehydration and cleanup

Hi Team,

We are using the RxDB Premium Full-Text Search plugin with FlexSearch on a large collection (~200K documents) in an Angular 15 hybrid mobile application.

The initial Full-Text Search indexing completes successfully. However, after restarting the application, the application can run out of memory while restoring the persisted FlexSearch state.

I also see a similar memory concern when using RxFulltextSearch.cleanup() after a large indexing operation.

While investigating this, I reviewed the Premium FlexSearch implementation in rx-fulltext-search.js and noticed two places where a large amount of persisted data can potentially be materialized in memory.

Environment

• RxDB: 16.9.0
• rxdb-premium: 16.9.0
• Angular: 15
• Application: Angular 15 hybrid mobile application
• Storage: SQLite with encryption
• Full-text search: RxDB Premium FlexSearch
• Indexed collection size: ~200K documents
• FTS pipeline batch size: 500–1000

1. Rehydration in addFulltextSearch()

The original addFulltextSearch() implementation restores the persisted FlexSearch state using:

const docs = await collection.find().exec();

It then separates the returned documents into type: "index" and type: "append" records and replays the append data into the FlexSearch index.

The relevant implementation is approximately:

const docs = await collection.find().exec();

const indexDocs = docs.filter(
    doc => doc.type === "index"
);

await Promise.all(
    indexDocs.map(doc => {
        doc.dataStr &&
            flexSearch.import(
                doc.name,
                doc.dataStr
            );
    })
);

docs
    .filter(doc => doc.type === "append")
    .forEach(doc => {
        doc.dataAr.forEach(item => {
            flexSearch.add(
                item.id,
                item.searchable
            );
        });
    });

With our ~200K document collection, the persisted append records collectively contain approximately 200K searchable entries.

The concern is that find().exec() materializes the complete FTS persistence collection in memory before the append records are processed.

As an experiment, I changed the rehydration logic to load the append records in smaller batches instead of loading the complete collection at once:

for (;;) {
    const appendDocs = await collection.find({
        selector: {
            type: "append"
        }
    })
    .sort({
        id: "asc"
    })
    .skip(offset)
    .limit(batchSize)
    .exec();

    if (appendDocs.length === 0) {
        break;
    }

    appendDocs.forEach(doc => {
        doc.dataAr.forEach(item => {
            flexSearch.add(
                item.id,
                item.searchable
            );
        });
    });

    offset += appendDocs.length;
}

The intention of this change is to reduce the amount of persisted append data held in memory at any one time.

However, the FlexSearch index itself still needs to be reconstructed in memory.

I would like to confirm whether processing the persisted append records in smaller batches is the recommended approach for large Full-Text Search indexes, or whether RxDB provides another mechanism for memory-efficient rehydration.

2. RxFulltextSearch.cleanup()

I also noticed a similar pattern in the cleanup() implementation:

const appendDocs = await this.collection.find({
    selector: {
        type: "append"
    }
}).exec();

The complete FlexSearch index is then exported:

this.index.export(async (name, dataStr) => {
    await this.collection.upsert({
        type: "index",
        token: this.collection.database.token,
        name: name + "",
        dataStr
    });
});

The append records are then removed:

await this.collection.bulkRemove(
    appendDocs.map(doc => doc.primary)
);

For a large index, this could potentially result in memory being used by:

Existing FlexSearch index
        +
All append documents / dataAr
        +
Serialized FlexSearch export
        +
RxDB storage/encryption buffers

This could create a significant temporary memory peak on a mobile device.

Persisted state

After the initial indexing, before calling cleanup(), our FTS persistence collection contains approximately:

total       = 400
indexCount  = 0
appendCount = 400

This is expected because the persisted FlexSearch snapshot has not yet been created.

My understanding is that cleanup() is responsible for exporting the current FlexSearch index into persisted type: "index" records and removing the corresponding append records.

What I would like to confirm

For a large Full-Text Search index (~200K+ documents):

1. Is cleanup() the intended mechanism for periodically creating the persisted type: "index" snapshot?

2. Is there a recommended way to make both FTS rehydration and cleanup more memory-efficient for large indexes?

3. In particular, would processing persisted append records in smaller chunks/batches be the recommended approach instead of materializing all of them with .find().exec()?

4. For cleanup(), is there a recommended way to avoid holding all append documents/data in memory while creating the FlexSearch snapshot?

5. Does RxDB/FlexSearch provide a supported lower-memory approach for exporting or restoring a large persisted index?

I have attached a single JavaScript reproduction file that creates a large collection, initializes Full-Text Search, closes/reopens the database, and exercises the rehydration and cleanup paths.

Please let me know if any additional information is needed to reproduce and diagnose the issue.

flexsearch-oom.js.txt

Thanks!