Comparison of Rust async and Linux thread context switch time.
Comparison of Rust async and Linux thread context switch time.
These are a few programs that try to measure context switch time and task memory use in various ways. In summary:
A context switch takes around 0.2µs between async tasks, versus 1.7µs between kernel threads. But this advantage goes away if the context switch is due to I/O readiness: both converge to 1.7µs. The async advantage also goes away in our microbenchmark if the program is pinned to a single core. So inter-core communication is something to watch out for.
Creating a new task takes ~0.3µs for an async task, versus ~17µs for a new kernel thread.
Memory consumption per task (i.e. for a task that doesn't do much) starts at around a few hundred bytes for an async task, versus around 20KiB (9.5KiB user, 10KiB kernel) for a kernel thread. This is a minimum: more demanding tasks will naturally use more.
It's no problem to create 250,000 async tasks, but I was only able to get my laptop to run 80,000 threads (4 core, two way HT, 32GiB), even after raising every limit I could find. So I don't know what's imposing this limit. See "Running tests with large numbers of threads", below.
These are probably not the limiting factors in your application, but it's nice to know that the headroom is there.
The programs thread-brigade and async-brigade each create 500 tasks
connected by pipes (like a “bucket brigade”) and measure how long it takes to
propagate a single byte from the first to the last. One is implemented with
threads, and the other is implemented with the Tokio crate's async I/O.
$ cd async-brigade/
$ /bin/time cargo run --release
Finished release [optimized] target(s) in 0.02s
Running `/home/jimb/rust/context-switch/target/release/async-brigade`
500 tasks, 10000 iterations:
mean 1.795ms per iteration, stddev 82.016µs (3.589µs per task per iter)
9.83user 8.33system 0:18.19elapsed 99%CPU (0avgtext+0avgdata 17144maxresident)k
0inputs+0outputs (0major+2283minor)pagefaults 0swaps
$
$ cd ../thread-brigade
$ /bin/time cargo run --release
Finished release [optimized] target(s) in 0.02s
Running `/home/jimb/rust/context-switch/target/release/thread-brigade`
500 tasks, 10000 iterations:
mean 2.657ms per iteration, stddev 231.822µs (5.313µs per task per iter)
9.14user 27.88system 0:26.91elapsed 137%CPU (0avgtext+0avgdata 16784maxresident)k
0inputs+0outputs (0major+3381minor)pagefaults 0swaps
$In these runs, I'm seeing 18.19s / 26.91s ≅ 0.68 or a 30% speedup from going async. However, if I pin the threaded version to a single core, the speed advantage of async disappears:
$ taskset --cpu-list 1 /bin/time cargo run --release
Finished release [optimized] target(s) in 0.02s
Running `/home/jimb/rust/context-switch/target/release/thread-brigade`
500 tasks, 10000 iterations:
mean 1.709ms per iteration, stddev 102.926µs (3.417µs per task per iter)
4.81user 12.50system 0:17.37elapsed 99%CPU (0avgtext+0avgdata 16744maxresident)k
0inputs+0outputs (0major+3610minor)pagefaults 0swaps
$I don't know why.
It would be interesting to see whether/how the number of tasks in the brigade affects these numbers.
Per-thread resident memory use in thread-brigade is about 9.5KiB, whereas
per-async-task memory use in async-brigade is around 0.4KiB, a factor of ~20.
See 'Measuring memory use', below.
There are differences in the system calls performed by the two versions:
In thread-brigade, each task does a single recvfrom and a write per
iteration, taking 5.5µs.
In async-brigade, each task does one recvfrom and one write, neither of
which block, and then one more recvfrom, which returns EAGAIN and suspends
the task. Then control returns to the executor. The reactor thread calls
epoll to see which pipes are readable, and tells the executor which task to
run next. All this takes 3.6µs.
In one-thread-brigade, we build the pipes but just have a single thread loop
through them all and do the reads and writes. This gives us a baseline cost
for the I/O operations themselves, which we can subtract off from the times in
the other two programs, in hopes that the remainder reflects the cost of the
context switches alone.
The async-brigade performance isn't affected much if we switch from Tokio's
default multi-thread executor to a single-threaded executor, so it's not
spending much time in kernel context switches. thread-brigade does a kernel
context switch from each task to the next. I think this means that context
switches are more expensive than a recvfrom and epoll system call.
If we run the test with 50000 tasks (and reduce the number of iterations to
100), the speedup doesn't change much, but thread-brigade requires a 466MiB
resident set, whereas async-brigade runs in around 21MiB. That's 10kiB of
memory being actively touched by each task, versus 0.4kiB, about a twentieth.
This isn't just the effect of pessimistically-sized thread stacks: we're looking
at the resident set size, which shouldn't include pages allocated to the stack
that the thread never actually touches. So the way Rust right-sizes futures
seems really effective.
This microbenchmark doesn't do much, but a real application would add to each task's working set, and that difference might become less significant. But I was able to run async-brigade with 250,000 tasks; I wasn't able to get my laptop to run 250,000 threads at all.
The other programs are minor variations, or make other measurements:
async-mem-brigade uses tokio:sync::mpsc channels to send usize values
from one async channel to another. This performs the same number of
task-to-task switches, but avoids the overhead of the pipe I/O. It seems
that Tokio's channels do use futexes on Linux to signal readiness.
one-thread-brigade attempts to measure the cost of the pipe I/O alone, by
creating all the pipes but having a single thread do all the reading and
writing to propagate the byte from the first to the last.
thread-creation and async-creation attempt to measure the time
required to create a thread / async task.
The scripts thread-brigade/rss-per-thread.sh and
async-brigade/rss-per-task.sh run their respective brigade microbenchmarks
with varying numbers of tasks, and measure the virtual and resident memory
consumption at each count. You can then do a linear regression to see the memory
use of a single task. Note that async-brigade/rss-per-task.sh runs 10x as many
tasks, to keep the noise down.
As mentioned above, in my measurements, each thread costs around 9.5KiB, and each async task costs around 0.4KiB, so the async version uses about 1/20th as much memory as the threaded version.
To run this script, you'll need to have the Linux pmap utility installed; this
gives an accurate measurement of resident set size. On Fedora, this is included
in the procps-ng package. (Pull requests for info about other major
distributions welcome.)
It's interesting to play with the number of tasks to see how that affects the relative speed of the async and threaded bucket brigades. But in order to test large numbers of threads, you may need to remove some of your system's guardrails.
On Linux:
You will run out of file descriptors. Each task needs two file descriptors, one for the reading end of the upstream pipe, and one for the writing end of the downstream pipe. The process also needs a few file descriptors for miscellaneous purposes. For 50000 tasks, say:
$ ulimit -n 100010
You may run out of process id numbers. Each thread needs its own pid. So, perhaps something like:
$ sudo sysctl kernel.pid_max=4194304This is overkill, but why worry about this? (The number above is the default in Fedora 33, 4 × 1024 × 1024; apparently systemd was worried about pid rollover.)
You will run out of memory map areas. Each thread has its own stack, with an unmapped guard page at the low end to catch stack overflows. There seem to be other constraints as well. In practice, this seems to work for 50000 tasks:
$ sudo sysctl vm.max_map_count=200000
Process ID numbers can also be limited by the pids cgroup controller.
A cgroup is a collection of processes on which you can impose system resource limits as a group. Every process belongs to exactly one cgroup. When one process creates another, the new process is placed in the same cgroup as its parent.
Cgroups are arranged in a tree, where limits set on a cgroup apply to that
group and all its descendants. Only leaf cgroups actually contain
processes/threads. The cgroups in the hierarchy have names that look like
filesystem paths; the root cgroup is named /.
You can see which cgroup your shell belongs to like this:
$ cat /proc/$$/cgroup
0::/user.slice/user-1000.slice/gargle/howl.scopeThis indicates that my shell is in a cgroup named
/user.slice/user-1000.slice/gargle/howl.scope. The names can get quite
long, so this example is simplified.
On Fedora, at least, the cgroup hierarchy is reflected in the ordinary
filesystem as a directory tree under /sys/fs/cgroup, so my shell's
cgroup appears as a directory here:
$ ls /sys/fs/cgroup/user.slice/user-1000.slice/gargle/howl.scope
cgroup.controllers cpu.stat memory.pressure
cgroup.events io.pressure memory.stat
cgroup.freeze memory.current memory.swap.current
cgroup.max.depth memory.events memory.swap.events
cgroup.max.descendants memory.events.local memory.swap.high
cgroup.procs memory.high memory.swap.max
cgroup.stat memory.low pids.current
cgroup.subtree_control memory.max pids.events
cgroup.threads memory.min pids.max
cgroup.type memory.numa_stat
cpu.pressure memory.oom.group
$You can inspect and manipulate cgroups by looking at these files. Some represent different resources that can be limited, while others relate to the cgroup hierarchy itself.
In particular, the file pids.max shows the limit this cgroup imposes on my
shell:
$ cat /sys/fs/cgroup/user.slice/user-1000.slice/gargle/howl.scope/pids.max
max
$A limit of max means that there's no limit. But limits set on parent
cgroups also apply to their descendants, so we need to check our ancestor
groups:
$ cat /sys/fs/cgroup/user.slice/user-1000.slice/gargle/pids.max
10813
$ cat /sys/fs/cgroup/user.slice/user-1000.slice/pids.max
84184
$ cat /sys/fs/cgroup/user.slice/pids.max
max
$ cat /sys/fs/cgroup/pids.max
cat: /sys/fs/cgroup/pids.max: No such file or directory
$Apparently there's a limit of 10813 pids imposed by my shell's cgroup's parent, and a higher limit of 84184 pids set for me as a user. (On Fedora, these limits are established by systemd configuration files.) To raise that limit, we can simply write another value to these files, as root:
$ sudo sh -c 'echo 100000 > /sys/fs/cgroup/user.slice/user-1000.slice/pids.max'
$ sudo sh -c 'echo max > /sys/fs/cgroup/user.slice/user-1000.slice/gargle/pids.max'The cgroup machinery seems to vary not only from one Linux distribution to
the next, but even from one version to another. So while I hope this is
helpful, you may need to consult other documentation. man cgroups(7) is a
good place to start, but beware, it makes my expla
No open issues yet, or sync has not completed.