bug(proxy): proxy does not adhere to prescribed resource limits

there is an issue in how we configure our proxy's async runtime. see linkerd2_proxy::rt::build(), here:

match workers.cores().get() {
        1 => { /* elided for brevity... */ }
        cores => {
            info!(%cores, "Using multi-threaded proxy runtime");
            Builder::new_multi_thread()
                .enable_all()
                .thread_name("proxy")
                .worker_threads(cores)
                .max_blocking_threads(cores)
                .build()
                .expect("failed to build threaded runtime!")
        }
    }

when told to establish a runtime with N > 1 cores, we use tokio's multi-threaded runtime. we tell it to use N worker threads, with a maximum of N blocking threads.

however, note the documentation of max_blocking_threads():

Specifies the limit for additional threads spawned by the Runtime.

These threads are used for blocking operations like tasks spawned through spawn_blocking, this includes but is not limited to:

• fs operations
• dns resolution through ToSocketAddrs
• writing to Stdout or Stderr
• reading from Stdin

Unlike the worker_threads, they are not always active and will exit if left idle for too long. You can change this timeout duration with thread_keep_alive.

In old versions max_threads limited both blocking and worker threads, but the current max_blocking_threads does not include async worker threads in the count.

- https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.html#method.max_blocking_threads

emphasis mine.

in other words, this code is not limiting the cores used by linkerd2-proxy to N cores as stated. max_blocking_threads is not inclusive of the threads used to service asynchronous workers, and instead is limiting the number of additional workers spawned.

so the actual upper-bound is N*2 cores used by the runtime.

this manifests in reports like https://github.com/linkerd/linkerd2/issues/14813. while they had configured the proxy to use a maximum of 40 cores, they observed the proxy using 48 cores.

linkerd/linkerd2-proxy#826 upgraded to tokio 1.0, and this breaking change to runtime configuration slipped by during that process. tokio::Runtime::builder::max_threads method was not just renamed! this new method limits the runtime across a different dimension.

for reference, some relevant upstream issues connected to this:

• https://github.com/tokio-rs/tokio/issues/2802#issuecomment-763889643
• https://github.com/tokio-rs/tokio/issues/2720