Repository navigation
ccache GitHub action generates excessive caches #113858
Description
The ccache action speeds up *nix builds considerably (though not enough that we could do without it).
The issue with it is that every build uploads a new cache, filling up our quota very quickly. GitHub removes caches that weren't used recently, but ccache is aggressive enough that it's crowding out smaller, useful entries. Specifically, the action to mark stale-PRs (which only runs every once in a while), has its state removed before the next run.
- Solve this in
main. (It's hard to test locally, iterating onmainseems to be the best way to do it. Sorry for the noise!) - Backport to bugfix branches
Linked PRs
- GH-113858: GitHub Actions config: Only save ccache on pushes #113859
- gh-113858: Cut down ccache size #113945
- [3.12] gh-113858: GH Actions: Make ccache smaller (GH-113859, GH-113945) #114082
- gh-113858: GH Actions: Limit max ccache size for the asan build #114113
- [3.11] gh-113858: GH Actions: Make ccache smaller (GH-114082) #114188
Activity
encukou commentedon Jan 9, 2024
It's not possible to change the contents of an existing cache, so for this to be useful we need to save a new one occasionally.
With #113859 it should save only on merged commits, not in-progress PRs. Hopefully that'll be enough.
hugovk commentedon Jan 9, 2024
Another idea is to set a
max-size, but the above PR looks like a good thing to try first 👍https://github.com/hendrikmuhs/ccache-action/blob/4cb35b2ff44ce57e1f5f5221e26bd8fefcfe41d0/action.yml#L15-L17
encukou commentedon Jan 9, 2024
Some napkin math suggests that this should be enough, at least outside sprints. Which is good enough :)
I don't think we want to delete the caches manually, so we'll be at GitHub's 10GiB limit regardless of max-size.
I plan to merge tomorrow morning (EU time) so I have the whole day to watch the effect (and revert if needed).
GH-113858: GitHub Actions config: Only save ccache on pushes (GH-113859)
erlend-aasland commentedon Jan 11, 2024
Is there more to do here? (I accidentally clicked "Close with comment" iso. "Comment"; so, please reopen if there is more to do here 😄)
encukou commentedon Jan 11, 2024
Yup, there is :)
20 remaining items
encukou commentedon Jan 24, 2024
Yeah, the current approach could barely get us to where we want to be. IMO it's time to cut the losses and try the other approach.
IMO, the best direction is to put branch names in the cache names, and write logic to clear all ccaches but the freshest one for each branch.
hugovk commentedon Jan 24, 2024
Sure, sounds good.
From the other side of the fence, we could adjust the
stale.ymlcron to run more often than daily: https://github.com/python/cpython/blob/main/.github/workflows/stale.ymlpythonGH-113858: GitHub Actions config: Only save ccache on pushes (p…
pythongh-113858: Cut down ccache size (pythonGH-113945)
pythongh-113858: GH Actions: Limit max ccache size for the asan build (…
hugovk commentedon Mar 12, 2024
Please see #116636 to process the stale issues twice per day, which hopefully will keep its cache file fresh and prevent deletion.
hugovk commentedon Mar 15, 2024
Twice per day isn't enough, let's try 4x: #116857.
hugovk commentedon Mar 17, 2024
4x doesn't help either. The problem now isn't that the stale cache has been deleted (for example, there's ~30 hours of caches available right now), but https://github.com/actions/stale only checks the first page of results from the cache API!
For example, https://api.github.com/repos/python/cpython/actions/caches only has ~3 hours of caches at the moment.
Upstream bug report is at actions/stale#1136
pythonGH-113858: GitHub Actions config: Only save ccache on pushes (p…
pythongh-113858: Cut down ccache size (pythonGH-113945)
pythongh-113858: GH Actions: Limit max ccache size for the asan build (…
[-]ccache GitHub action generates excessive caches[/-][+]`ccache` GitHub action generates excessive caches[/+]hugovk commentedon Jun 19, 2026
Fixed by removing ccache via #139596.
Now about half the cache is made up of CodeQL caches, and I think we only need 1-2 (per language), but that's another issue.
https://github.com/python/cpython/actions/caches?query=key%3Acodeql+