Reduce locks contention

As mentioned by @baptistejamin in #372, Sonic has plenty of useless locking which slows down ingestion.

Edit: When I said “useless locking”, I mostly meant “locking for too long”. I have a feeling some locks are completely unnecessary, but most are just locked for too long and force concurrent tasks to wait for no good reason.

For example, StoreKVPool::flush locks the pool in write mode, while it only ever needs read access. By doing so, it prevents concurrent reads, which seem unnecessary. Every time the janitor runs, all KV reads are blocked, increasing response time for in-flight queries.

// Acquire access lock (in blocking write mode), and reference it in context
// Notice: this prevents store to be acquired from any context
let _access = self.store_access_lock.write().unwrap();

if let Some(store) = self.pool.read().unwrap().get(key) {
  // …

I may have missed something and this is just one example, but it’s the kind of broad locking behavior we will get rid of.