Repository navigation
CPU & memory resource usage not measured when used with FDS connector & MPI #727
Description
When running FDS using multiple processors (e.g. 20 in my case) the CPU & memory usage is close to zero.
{{ message }}
When running FDS using multiple processors (e.g. 20 in my case) the CPU & memory usage is close to zero.
Activity
wk9874 commentedon Feb 26, 2025
Hmm, have a suspicion that the resource metrics actually arent tracking subprocesses at all :/
wk9874 commentedon Feb 26, 2025
It was tracking all subprocesses, but for some reason without an interval it always reports FDS's usage as 0. Added small 0.1 interval for all CPU metric collections. This shouldnt have a big impact on performance since it happens inside the heartbeat thread which only executes every multiple seconds anyway
alahiff commentedon Feb 27, 2025
Non-MPI jobs look better now (*), but FDS running using MPI still has 0% CPU usage. Will investigate further...
(*) Looking at older FDS jobs they immediately used 100% CPU efficiency, whereas now they ramp up over 800 secs. Will investigate further...
6 remaining items
wk9874 commentedon Feb 28, 2025
Found it! It's because
self.processesrecreates the list of processes each time, creating new process objects for all subprocesses (like FDS). Without an interval the first measurement returns zero as I mentioned, so because its making a new process each time itll always return zero. Just need to make it so thatself.processesis constant, and anytime set_pid is used oradd_processis called, it appends to that list. That should mean I can drop the small interval, which should hopefully fix our problemswk9874 commentedon Feb 28, 2025
Actually looks like it's in Executor:
**
**
wk9874 commentedon Mar 3, 2025
Have fixed this by making it so that in Run and Executor's
processesproperties, it adds to / removes from existing lists instead of overwriting the list each time. Testing performed (each with 5s resource metric interval):topduring this time showed the CPU usage gradually increased to around 20% when I stopped the script:topshowed that the CPU usage of the python process was fluctuating between 20 - 25% in this time (rising overall):@alahiff can you just confirm that you're happy with these? Then I'll put the changes into an alpha for you to test on CUMULUS / CSD3 :)
alahiff commentedon Mar 4, 2025
Tested with 2.0.0a3.
When CPU is used by the Python script itself, the CPU usage reported seems correct. For an external process, like FDS with the FDS connector, the CPU usage is incorrect. FDS used ~100% CPU the whole time it ran, it didn't slowly ramp up.
I'm not sure how you managed to get a plot where CPU usage went to 100% so quickly.
wk9874 commentedon Mar 4, 2025
Damn, I have no idea why thats happening for you...
Can you let me know what you're running this on? Eg locally on laptop, CSD3, CUMULUS etc? Is it Online or Offline? What is the resources_metrics_interval you are using?
Will investigate more, but I can't reproduce that atm
alahiff commentedon Mar 4, 2025
This is on CUMULUS online, i.e. a login node. I haven't adjusted resources_metrics_interval, it's just whatever the default is.