CPU oversaturation/oversubscription due to third-party libraries when running containers on Linux

[!WARNING] This issue is not yet ready for a PR. If you are interested in contributing to scikit-learn, please have a look at our contributing guidelines, and in particular the sections for new contributors and the "Needs triage" label.

Background

This is the result of a discussion with @ogrisel and @jeremiedbb, fleshed out and with some additional ideas. I make my suggestion at the end, but will require feedback before I do any of the work.

The problem

When using a container on Linux (Docker, Kubernetes, anything using cgroups), you can set a CPU quota, something like "this process gets 2 cores worth of compute."

Unfortunately, not all third-party C libraries know this. At minimum OpenBLAS does not seem to know about this limit (). So if you have 64 core machine, and run a Kubernetes pod with 8 cores, and OpenBLAS will think it has 64 cores, and it will be much much slower. Similarly libgomp and libomp. MKL builds on OpenMP so maybe no point in filing there until libgomp/libomp are fixed.

• https://github.com/OpenMathLib/OpenBLAS/issues/6059
• https://gcc.gnu.org/bugzilla/show_bug.cgi?id=127651
• https://github.com/llvm/llvm-project/issues/225493

This is a performance problem for scikit-lean users.

Demonstration

My CPU has 12 cores. I set up a cgroup with equivalent of 2 cores, but OpenBLAS still reports 12 cores:

$ systemd-run --user --scope --property CPUQuota="200%" -- python -c "import threadpoolctl; import numpy; print(threadpoolctl.ThreadpoolController().lib_controllers[0].get_num_threads())"
12

Potential fixes

I am going to be fixing one case as a side-effect of https://github.com/joblib/joblib/issues/1846: when using joblib.Parallel, the number of cores will be constrained using threadpoolctl, including in sequential cases. However, this is a strange fix where if you don't use joblib, you still have the same performance problem.

So what else can be done until OpenBLAS any other relevant libraries fix this issue?

Option 1: joblib or scikit-learn set the limit correctly, on import

Instead of doing this in Parallel in the sequential case, do it on import of joblib or maybe sklearn: threadpoolctl.threadpool_limits(limits=joblib.effective_cpu_count(-1)). (Technically this won't fix the above example because of https://github.com/joblib/loky/issues/634 but it will work in Docker or Kubernetes, and everywhere once that issue is fixed).

Pros:

• Fixes the limit for all use cases.

Cons:

• Importing a library changing global configuration. This is not very nice.

Option 2: sklearn sets the limit whenever calling into OpenMP or OpenBLAS

Pros:

• If you do with threadpoolctl.threadpool_limits(limits=joblib.effective_cpu_count(-1)) it undoes the limits when done, so it's not a permanent global config change, it's just DTRT.

Cons:

• This adds performance overhead of all these calls.
• Lots of code changes.

Option 3: Documentation of the problem and how to solve it

Pros:

• No impacts on global behavior.

Cons:

• Most people won't notice the documentation.
• If the situation improves or changes, most people won't re-read the updated docs.

Option 4: Warn on import

If os.process_cpu_count() > joblib.effective_cpu_count(-1) then that means cgroups is restricting CPU count further, so warn the user and tell them what to do. In practice should probably also docs to explain in more detail.

Pros:

• More likely to be noticed than docs.

Cons:

• Users will still have to do work.

My suggestion

After further thought, I think setting the limit on import (option 1) is actually reasonable. The hope is that this is a stop-gap and can eventually be removed once the relevant libraries DTRT, and in practice for the vast majority of people this is the right thing to do.

If that's not acceptable, docs + warning is what I would suggest doing.

I can proceed with either, once there is some consensus.