Repository navigation
Files in /var/cache/cloud/processed on virtual router are not cleaned up #3757
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
System VMs
CLOUDSTACK VERSION
4.13.0
CONFIGURATION
Advanced networking
OS / ENVIRONMENT
RHEL 7.x
SUMMARY
We are running some tests that involve creating and deleting lots of instances using the API. Our virtual router ran out of inodes and had over 100,000 files in the /var/cache/cloud/processed directory. I can't find any code in CloudStack that cleans up that directory, but if there is then there should also be an option to cleanup more frequently.
STEPS TO REPRODUCE
Create/delete thousands of VMs.
EXPECTED RESULTS
I'd expect the virtual router to clean up after itself.
ACTUAL RESULTS
The virtual router eventually runs out of inodes.
$ df -i /var
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vda6 117392 117392 0 100% /var
$ ls -f /var/cache/cloud/processed | wc -l
110343
Activity
kioie commentedon Dec 20, 2019
What would trigger a cleanup and what would be the frequency?
andrijapanicsb commentedon Dec 26, 2019
Good one.
yadvr commentedon Oct 29, 2020
Maybe we can live with a weekly or monthly cronjob that checks and clears old files (ls/stat etc. to determine list of files older than week or month).
DaanHoogland commentedon Oct 29, 2020
How about a trigger to delete from the MS and implementing a clean from there in the internal scheduler or from cron using cloudmonkey.
weizhouapache commentedon Aug 17, 2021
since cloudstack 4.15.0.0, there is no parition /var any more. see #4104
VR health check was introduced to cloudstack 4.14.0.0 in #3575, which contains the check for disk space (disk_space_check.py).
Therefore I think we do not need to fix this issue.
what's your opinion ? @rhtyd @nvazquez
yadvr commentedon Aug 17, 2021
@weizhouapache You're right there is no explicit /var partition, however the disk space may still get filled over time. Perhaps a cronjob that cleans the directory weekly/monthly basis would be enough @weizhouapache @nvazquez ? (In a new VR design I'm trying to investigate this may not present a problem in future)
weizhouapache commentedon Aug 17, 2021
@rhtyd I agree to add a cronjob to cleanup disk
there are already some logroate configs to manage log files in /var/log in systemvm/debian/etc/logrotate.d/.
we have fixed some issues with logrorate in pr #4439, #4872, #4869, #1915, #1329 etc
it looks we only need to clean up /var/cache/cloud/processed , maybe only when disk space usage exceeds threshold like 80% ?.
yadvr commentedon Aug 20, 2021
Fixed by #5331