Repository navigation
TRAP caching causes GitHub Actions Cache to churn #2030
Description
I'd like to discuss alternative storage solutions for TRAP caching, as the current implementation causes our monorepo to exhaust our available cache. Each run appears to consume about 100mb of space, and our current merge-to-default velocity means we run out of cache in about a day.
I am aware of the ability to disable trap_caching within the init action, however I would also like to continue using trap caching at some point in the future.
I have successfully leveraged GitHub Packages (i.e., GHCR) for caching Docker Images and other build artifacts in the past. Would that be another possible path forward for this useful feature?
Activity
aeisenberg commentedon Dec 12, 2023
Thanks for raising this issue and the suggestion. We are considering making some changes to the trap caching feature so that it only runs on the default branch. This will ensure that each repo will cache no more than one set of TRAP files. TRAP caching only really makes sense for when a repo runs codeql analysis on a schedule. Most repos only use scheduled runs on the default branch, so this should be workable for most repos.
For your monorepo, is the issue that you are running analyses in PR branches when the code to be analyzed hasn't changed? If so, perhaps you can use path expressions to avoid running codeql in this case.
sayhiben commentedon Dec 12, 2023
Almost all our our trap cache is from our default branch, which we merge to more than 60 times a day on average. Even if this only ran on the default branch, it would continue to exhaust the available space in the GHA cache due to the differing cache keys
If the desire is to continue using the GHA Cache, and there's only a point in caching the most recent run, we'll have to choose an arbitrary cache key and manually invalidate the old cache when we want to update it. Otherwise, each CodeQL run against our default branch will add another 100mb to our usage (edit: i'd expect race conditions to emerge if multiple CodeQL workflow runs attempt to manage the cache simultaneously, however)
Example screenshot from our default branch attached
sayhiben commentedon Dec 12, 2023
Another possible approach might be to save the trap cache as an artifact, then pull the most recent artifact from the workflow run on the default branch when restoring the cache, though I seem to recall that arbitrary artifacts access might need a PAT, which would be something of a dealbreaker...
aeisenberg commentedon Dec 13, 2023
Our hope is that there will only ever be one cached item and it would contain the most recent TRAP on the default branch. Before uploading the new TRAP cache, the old TRAP cache would be removed. (There would be a few moments when there is no cache since the old one was deleted and the new one isn't uploaded yet, but hopefully that window will be small.)
sayhiben commentedon Dec 14, 2023
Ah, then the idea is to update the cache key to remain consistent and use the API to remove the old cache so the key can be updated? That should work
aeisenberg commentedon Dec 14, 2023
Yes. That's right.
phlax commentedon Nov 5, 2024
any progress on this? this is causing us a lot of caching issues given the very limited amount of cache available and the fact that you cannot increase it
we have already disabled codeql on all branches other than
mainand even with that it creates a new cache on ~every commit and squeezes out other caches which are a lot more important and usefulaeisenberg commentedon Nov 5, 2024
I recommend that you disable trap caching. You can set the environment variable
CODEQL_ACTION_CLEANUP_TRAP_CACHEStotrueand the caching feature will be disabled.hsbt commentedon May 1, 2025
Ruby organization faced same issues with @phlax.
CODEQL_ACTION_CLEANUP_TRAP_CACHESis not working now. CodeQL generate 300M+ cache each CI build. We always have trouble with the amount of limit by this problem.aeisenberg commentedon May 1, 2025
Can you please create a new issue for this? Also, please re-run a codeql scan in debug mode and either share the entire log with us, or look for entries like:
Keeping most recent TRAP cacheNo TRAP caches to clean upCleaning up TRAP cachefeature disabledAlso, if you can share the name of the repository, we can look in our telemetry to see if trap cache cleanup did run for you. Add this all in a new issue and our first responder on the team will look into this.
GHA/codeql: try disabling the TRAP cache
GHA/codeql: try disabling the TRAP cache
Fix CodeQL TRAP-cache churn eating the 10GB Actions cache budget
Fix CodeQL TRAP-cache churn eating the 10GB Actions cache budget