Stuck all the incoming messages when triggering PutManyLIDMappings

Performance Issue: Long-running PutManyLIDMappings Blocks StoreLIDPNMapping (from groups) due to Lock Contention

Description

I have encountered a significant performance bottleneck in scenarios involving high-volume group messages, which causes message processing delays. The issue stems from the way LID (Local ID) mapping operations handle concurrent access using s.lidCacheLock.

When new group member data is processed, the PutManyLIDMappings function is called with a large number of mappings (often 6,000+ entries). This bulk operation takes a long time to complete because it involves numerous database write operations.

The core problem is that PutManyLIDMappings acquires and holds the s.lidCacheLock (a simple sync.Mutex inside the Store) for the entire duration of the bulk insertion. During this time, any incoming message that triggers handleEncryptedMessage eventually calls the single-entry mapping function, StoreLIDPNMapping.

Since StoreLIDPNMapping attempts to acquire the same lock, it is completely blocked. This means real-time messages cannot be processed until the bulk database insertion of 6,000+ records is finished, leading to significant message delivery and processing delays.

Affected Functions

1. PutManyLIDMappings (The blocking function)
2. StoreLIDPNMapping (The blocked function, critical for real-time message processing)

Observed Behavior (Detailed Steps)

1. Bulk Operation Starts: A group update event triggers PutManyLIDMappings with N > 6000 data points.
2. Lock Acquired: PutManyLIDMappings calls s.lidCacheLock.Lock().
3. Long Wait: The function proceeds to insert N entries one by one into the store (SQLite/etc.), which is a time-consuming I/O operation.
4. Real-Time Message Arrives: A new message arrives, triggering handleEncryptedMessage.
5. Critical Function Blocked: handleEncryptedMessage calls StoreLIDPNMapping, which then attempts to call s.lidCacheLock.Lock().
6. Deadlock/Bottleneck: The single, quick StoreLIDPNMapping operation waits for the long-running PutManyLIDMappings to finish and release the lock, causing a severe delay in processing the incoming message.

Proposed Solutions / Suggestions

To mitigate this contention and improve real-time performance, I suggest exploring the following options:

1. Granular Locking (Release During I/O): The s.lidCacheLock should be released before the long database write/bulk insert operation and reacquired after it. The lock should only be held for local memory cache updates, not for the duration of the slow database I/O.
2. Asynchronous Bulk Processing: Move the bulk database insertion logic of PutManyLIDMappings into a separate goroutine or process it asynchronously, allowing immediate return and release of the main thread's lock.
3. Use Batching/Transaction for Bulk Insert: Ensure the database operations in PutManyLIDMappings use a single transaction for better performance, although this may not solve the locking issue itself, it will reduce the time the lock is held.

---