You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Investigate why remote_monitor takes time to shutdown after experiment is over #35
I've observed this in long experiments with large query lookback periods. There are long multiple instances of: ssh -o StrictHostKeyChecking=no milindsr@node0.scratch1.cloudmigration-PG0.utah.cloudlab.us "pgrep -f remote_monitor.py" Remote monitor is still running. Will check again in 10 seconds
Not a problem wrt functionality but figuring this out will speed up running of experiments.
My hypothesis was that this was because of remote_monitor and/or PrometheusClient dumping large files and taking time to do so.
However, remote_monitor's monitor_output.json is small and PrometheusClient has been updated to stream results to file instead of dumping them all at once after the end of an experiment.
Activity