You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Monitoring system resources during training using MLFlow #7405
I found a cool feature in the recent MLFlow release where we can monitor and log system resources (GPU/CPU/MEM/net, HDD, ...) during training. I am using it in the Engine based training as follows:
This way both resources and training logs go the same experiment and run. In a way, this suffices, but takes particularly for resource_monitor linear approach and not Engine/Event paradigm.
I would love to hear if it make sense to think about enhancing this approach.
Thanks
PS: It might make sense to include this in mlflow integration tutorials
Activity
KumoLiu commentedon Jan 19, 2024
A tutorial can be found here.
https://github.com/chenmoneygithub/mlflow/blob/ca5ce50a5aff042a5d8b365e55b4d97934204253/docs/source/system-metrics/index.rst
Record system resource usage with MLFlowHandler
Record system resource usage with MLFlowHandler (Project-MONAI#9052)