Repository navigation
Pkgcheck git cache should be parallelized #770
Description
Being I have a full gentoo repo locally, the git cache update scanned all the way back to 2019. It's a one time impact, but this is a scalar process consuming from git output.
This may be optimizable in a way allowing it to stay scalar; if not, extracting all git commits that need processing, and then feeding batches into multiprocessing (which processes those batches) should work.
Note if this is implemented, the parallelization should be $CPUS/2; pkgcheck itself seems to be running near 100% whilst git is at 90% for my system. Just assume it'll hold similar.
Activity
radhermit commentedon Jan 19, 2026
Just to note, if you process them in parallel they still need to be merged back in the correct order otherwise package history will be incorrect for the virtual cache repo as iirc only the latest pkg change is saved to limit cache size.
That added complexity and the inability to use pygit2/libgit2 due to log parsing defects and deficiencies during the initial implementation were some of the reasons I didn't do this back then, but it's probably possible to use some external library nowadays instead of the raw
git logcalls this implementation uses since it's been ~7 years.