Repository navigation
os.cpu_count() should return a count assigned to a container or that the process is restricted to #80235
Description
| BPO | 36054 |
|---|---|
| Nosy | @vstinner, @giampaolo, @tiran, @jab, @methane, @matrixise, @corona10, @Zheaoli, @mcnelsonphd, @NargiT, @Redeyed |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2019-02-20.16:56:22.384>
labels = ['interpreter-core', '3.8', '3.7', 'library', 'performance']
title = 'On Linux, os.count() should read cgroup cpu.shares and cpu.cfs (CPU count inside docker container)'
updated_at = <Date 2021-10-11.14:25:34.762>
user = 'https://github.com/keirlawson'bugs.python.org fields:
activity = <Date 2021-10-11.14:25:34.762>
actor = 'corona10'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Library (Lib)']
creation = <Date 2019-02-20.16:56:22.384>
creator = 'keirlawson'
dependencies = []
files = []
hgrepos = []
issue_num = 36054
keywords = []
message_count = 24.0
messages = ['336117', '336126', '336146', '336148', '336149', '338724', '338778', '338780', '338783', '339401', '339404', '339429', '339439', '353690', '364894', '364898', '364901', '364902', '364916', '365071', '365075', '366811', '403622', '403654']
nosy_count = 12.0
nosy_names = ['vstinner', 'giampaolo.rodola', 'christian.heimes', 'jab', 'methane', 'matrixise', 'corona10', 'Manjusaka', 'mcnelsonphd', 'galen', 'nargit', 'RedEyed']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue36054'
versions = ['Python 3.7', 'Python 3.8']
Activity
keirlawson commentedon Feb 20, 2019
There appears to be no way to detect the number of CPUs allotted to a Python program within a docker container. With the following script:
print("os.cpu_count(): " + str(os.cpu_count()))
print("len(os.sched_getaffinity(0)): " + str(len(os.sched_getaffinity(0))))
when run in a container (from an Ubuntu 18.04 host) I get:
docker run -v "$PWD":/src/ -w /src/ --cpus=1 python:3.7 python detect_cpus.py
os.cpu_count(): 4
len(os.sched_getaffinity(0)): 4
Recent vesions of Java are able to correctly detect the CPU allocation:
docker run -it --cpus 1 openjdk:10-jdk
Feb 20, 2019 4:20:29 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
| Welcome to JShell -- Version 10.0.2
| For an introduction type: /help intro
jshell> Runtime.getRuntime().availableProcessors()
$1 ==> 1
matrixise commentedon Feb 20, 2019
I would like to work on this issue.
matrixise commentedon Feb 20, 2019
so, I have also tested with the last docker image of golang.
here is the code of golang:
package main
import "fmt"
import "runtime"
func main() {
cores := runtime.NumCPU()
fmt.Printf("This machine has %d CPU cores.\n", cores)
}
Here is the output
When I try with grep on /proc/cpuinfo
I get this result
I will test with openjdk because it's related to Java and see if I can get the result of 1
matrixise commentedon Feb 20, 2019
ok, I didn't see your test with openjdk:10, sorry
keirlawson commentedon Feb 20, 2019
I believe this is related to this ticket: https://bugs.python.org/issue26692
Looking at Java's implementation it seems like they are checking if cgroups are enabled via /proc/self/cgroup and then if it is parsing the cgroup information out of the filesystem.
matrixise commentedon Mar 24, 2019
I am really sorry but I thought to work on this issue but it's not the case. Feel free to submit a PR.
Zheaoli commentedon Mar 25, 2019
I think that I may work on a PR for this issue. Is there anybody has worked on it ？
matrixise commentedon Mar 25, 2019
Hi Manjusaka,
Could you explain your solution, because I have read the code of
openjdk, (C++) and I am going to be honnest, it was not complex but not
really clear.
Also, if you need my help for the review or for the construction of your
PR, I can help you.
Have a nice day,
Stéphane
Zheaoli commentedon Mar 25, 2019
Hi Stéphane
Thanks a lot!
In my opinion, I would like to make an independent library that name is cgroups. For ease of use and compatibility, I think it's better than combining code with the os module.
Thanks for you working!
Manjusaka
Zheaoli commentedon Apr 3, 2019
Hi Stéphane:
I have checked the JVM implantation about container improvements. I confirm that maybe we need a new Libary for container environment. I don't think that combine it into the os module is a good idea. I will make a PR during this week.
31 remaining items
vstinner commentedon Sep 22, 2023
Is there a reliable way to retrieve this information?
indygreg commentedon Sep 22, 2023
Yes, strictly speaking cgroups shares/weight has nothing to do with parallelism or number of CPUs. This cgroups mechanism is a way of specifying a proportion of the kernel scheduler's time slice that a cgroup can consume. A lot of people mistakenly believe that by specifying cpus=N to their container runtime they are asking for N CPUs. What they are actually asking for is N CPUs worth of time from the scheduler. They are free to consume those resources using as few or many threads as they want (subject to additional cgroups limitations) and the kernel will schedule work on as few or as many physical CPU cores as it chooses.
However, it is extremely popular to size thread/process to match the proportional CPU count given to the container runtime. And container runtimes make it easy to just divide the cgroups share by 1024 to deduce that value. So everyone does that. It mostly just works for the purposes people use it for.
Now if container runtimes exposed the resource settings to containers (by injecting an environment variable perhaps?), that is something I may potentially use. But at the same time, cgroups feels more canonical. We also have to consider that dynamic resource sizing for containers/cgroups is a thing after all! I can't imagine the maintainers of container runtimes will want to try to propagate mutable state into the container. It is far easier for them to do nothing and just tell end-users to query cgroups.
[-]On Linux, os.count() should read cgroup cpu.shares and cpu.cfs (CPU count inside docker container)[/-][+]os.cpu_count() should return a count assigned to a container or that the process is restricted to[/+]vstinner commentedon Sep 22, 2023
If cpu_count() starts parsing cgroups, I would suggest using the minimum between what's announced in cgroups and what's seen by taking in account CPU affinity: that's what does pylint.
See #80235 (comment) and the pointed code:
where cpu_count is
len(sched_getaffinity(0)).vstinner commentedon Sep 22, 2023
What I dislike here is that the ratio converting cgroups CPU shares to a number of logical CPUs is not standardized.
@indygreg proposes to announce this constant on a command line option / env var to solve this issue: #109595 (comment).
Maybe 1024 can be used by default, but it's important that it's possible to configure this value.
corona10 commentedon Oct 12, 2023
FYI, now we provide a way to restrict CPU resources from #109595.
This can be the alternative way to handle this issue.
vstinner commentedon Oct 12, 2023
After reading a lot about Linux cgroups CPU share, I don't think that they should be used in os.cpu_count() or os.process_cpu_count(). I close the issue.
The relationship between a CPU share and a number of CPU is not clear and caused a lot of headaches to many people.
Obviously, some people have different opinion and I suggest them to implement something similar to pylint or create a project on PyPI, get feedback from other users, and then come back with a strong rationale to convince why it's very important to add it to Python :-)
Maybe later we can consider to add a "get_cgroups_cpu_share()" function, but I'm not convinced about using it to compute cpu_count() or process_cpu_count().
robsonpeixoto commentedon Oct 12, 2023
@vstinner os not possible to define
os.cpu_count()based on CPU share. IMHO should only care about quota(CPU limit), because is the only information that can be easily used to define theos.cpu_count().This project can be useful https://github.com/uber-go/automaxprocs
vstinner commentedon Oct 12, 2023
I mean that in general, Linux cgroups CPU limits are not directly related to a whole number of CPU and so it's hard to use such limit to update os.cpu_count() or os.process_cpu_count().
Please create a PyPI project to parse Linux cgroups and see how it goes ;-)
So far, since 2019, I didn't see any clear implementation of "cpu_count" based on Linux cgroups. The closest thing that I saw so far was pylint code, but I don't think that we should do something like that in Python stdlib.
blackliner commentedon May 23, 2024
Is there a TL&DR for anyone coming to this thread from google?
I.e.; What is the canonical way to find the (lower) CPU core number due to cgroup/container resource limitations?
vstinner commentedon May 23, 2024
It was decided to not handle this problem in Python. You should use a 3rd party project for that.
blackliner commentedon May 23, 2024
Can we provide some example, follow up links that could suggest the one or other way to solve this in our application code?
Or can the environment solve this with an approach similar to #109595 via an env variable? Will ask in that thread over there 👍
vstinner commentedon May 23, 2024
You can use PYTHON_CPU_COUNT env var to override os.cpu_count(): https://docs.python.org/dev/using/cmdline.html#envvar-PYTHON_CPU_COUNT
I don't think that we should provide examples in Python documentation. This problem is not solved, there are only heuristics, it depends on your environment and other stuff.
wwarriner commentedon Apr 25, 2025
For what it's worth, in a Slurm job context you can use
os.environ["SLURM_CPUS_ON_NODE"]or withSLURM_CPUS_PER_TASKto get a more accurate number of available cores.https://slurm.schedmd.com/sbatch.html#OPT_SLURM_CPUS_ON_NODE