Runtime stall with large number of spawn_blocking tasks

Hi! We are debugging latency increase under load in our app. We are still not clear yet on what the actual issue is but one potential hypothesis is that under load, the service spawns many spawn_blocking tasks and there is a mutex protect a VecDeque there. Our server is beefy (100 cores, 512GB Ram), we run a database server.

We configure tokio to have 1024 blocking threads and under load, we spawn up to 60K-100K spawn_blocking tasks (each task is a multi get on rocksdb). We are actively profiling the service to pin-point actual issue but wonder if the community has dealt with this issue before and how do you usually tackle this issue? Do you batch multiple tasks into one spawn_blocking or create a dedicate runtime (another tokio or possibly a rayon) to handle blocking IO tasks. Thank you!