Repository navigation
Offline sender sometimes doesn't send heartbeats #719
Description
Here is an example of when the server received heartbeats for a specific run. Note the big gaps in the timestamps:
194.81.223.66 - - [22/Feb/2025:16:35:03 +0000] "PUT /api/runs/V4xXatum2bKBouQdN8JdN5/heartbeat HTTP/1.1" 200 4 "-" "Simvue Python client 2.0.0a1"
194.81.223.66 - - [22/Feb/2025:17:01:03 +0000] "PUT /api/runs/V4xXatum2bKBouQdN8JdN5/heartbeat HTTP/1.1" 200 4 "-" "Simvue Python client 2.0.0a1"
194.81.223.66 - - [22/Feb/2025:17:26:03 +0000] "PUT /api/runs/V4xXatum2bKBouQdN8JdN5/heartbeat HTTP/1.1" 200 4 "-" "Simvue Python client 2.0.0a1"
Sorry, I'm not sure how to reproduce this.
Activity
wk9874 commentedon Feb 24, 2025
No idea what could cause this - do you know what the cronjob interval and heartbeat intervals were @alahiff ? And were you running a lot of simulations at once while this was happening?
I've added threading into the heartbeat sender to speed it up a bit in the case of a very large number of runs, but suspect that wont properly fix this issue. If there were a very large number of concurrent runs / uploads / uploading large files that take a long time then the heartbeats will be stuck behind those uploads finishing, so that might slow it down significantly
alahiff commentedon Feb 27, 2025
Currently being tested.
alahiff commentedon Feb 27, 2025
Currently being tested.
kzscisoft commentedon Jun 11, 2025
@alahiff any updates?