Repository navigation
Super-speed caching for static module directory trees #629
Description
Hi @xdelaruelle --
After a lot of back-n-forth with my GPT-friend, I would like to propose a new opt-in feature: a user commitment to a static module dir tree (no virtual modules, .modulerc dynamic behaviour changes, etc), which gets you big performance gains (>100x). As you know by now, I have a script that generates 15k (15k today, more coming soon) modules with no dynamic features. And no problem taking an arbitrarily long up-front cache build time.
Do you agree this is something you can accept? The basic contract is that user gives up dynamic features for big performance.
We can talk about what to do if the contract is broken. Fallback to old? Hard fail? User chooses?
In the next comment, I'll copy/paste the full chat log for more details.
Example config names GPT threw out.
1. Add an explicit opt-in mode
Something like a config knob:
- static_cache
- frozen_modulepath
- trust_cache_fully
Example of what we have achieved so far:
Benchmarks, clean baseline modules-pr613 vs new worktree modules-avail-opt:
- full module avail --terse >/dev/null 2>&1
- before: 5.991 s ± 0.157 s
- after: 105.9 ms ± 1.4 ms
- module avail --terse vendor1 >/dev/null 2>&1
- before: 3.750 s ± 0.086 s
- after: 203.7 ms ± 0.8 ms
- module avail --terse vendor1/product1 >/dev/null 2>&1
- before: 3.367 s ± 0.086 s
- after: 142.7 ms ± 3.1 ms
Sanity checks:
- module avail --redirect --terse vendor1 | wc -l -> 50001
- module avail --redirect --terse vendor1/product1 | wc -l -> 10001
- module avail --redirect --terse vendor1/product1/0000* | wc -l -> 10
Activity
smprather commentedon May 27, 2026
This is a chat-log in which I asked for
module availperformance improvement.Consider this an Appendix. Skip to the next comment.
Codex in normal. My prompts in bold italics.
<snip...>
Code changed there:
Benchmarks, clean baseline modules-pr613 vs new worktree modules-avail-opt:
Sanity checks:
Important caveat:
That is probably the right intermediate shape. Next move: port this from modules-avail-opt into modules-main, then widen the index so it can cover richer real trees without dropping to fallback.
› just curious. all of your improvements are cache based. is all the juice squeezed out of the cacheless (or cache-as-it-is-today) code? is there anything in the tcl codebase to speed up this particular hot loop?
• I’m checking a few concrete hot spots before I answer. The big question is whether the normal Tcl path is fundamentally doing too much work per entry, or whether it’s just missing a better index.
• I have enough. Short answer is: no, the non-indexed path is not “fully squeezed.” There are still real gains in the Tcl path, but they’re smaller and riskier than building a better index. I’m pulling one more code slice so I can point to the exact place where today’s cache still does too much work.
• No. There is still headroom in the normal Tcl path.
But the important distinction is:
What I think from reading the code and timing it:
module avail today is not slow because puts is slow. I checked that indirectly: sending output to /dev/null barely changed runtime. The time is mostly CPU spent building and post-processing the result set.
The main hot spots are visible in the current code:
This path does a lot of:
Across 100k files, that adds up fast.
It still:
That means the current cache avoids filesystem reads, but it does not avoid the “touch the whole universe” shape.
For every entry it may call:
Then it sorts the full display list:
On a simple tree like your mock modules, most of that metadata work is wasted.
Even for module avail vendor1/product1, the normal path still builds a large candidate set and filters it late.
So, is all the juice squeezed out of cacheless / current-cache code?
No.
What is still available without “cache exact query answers”:
bucket by prefix / directory / root name instead of replaying the whole cache and filtering afterward.
do not call tag / variant / alias helpers unless output config actually needs them.
stop doing file split / file dirname / file tail in the innermost loops.
What I do not believe:
My practical read:
If you want, next step is I can try a non-cheaty middle ground in the real codebase:
› well, i think, and correct me if i'm wrong, that "replay the module universe" thing is triggered by things like virtual (dynamic) modules. BUT, by creating a cache, aren't we signing a contract to not be dynamic? now that i think about it, what i'm really asking for is a "i agree to be 100% static, and agree to pay any up-front caching time, if you give me big performance improvement". as long as we create that contract for signing, and have a way to sign it, then we should be able to pass all tests, including virtual module testing by not using cache improvements in that case. sound correct?
• Yes. That is the right framing.
You are basically defining a stronger cache contract:
Under that contract, much more becomes legal.
And yes, that should let us keep upstream semantics intact by making it opt-in. In other words:
current semantics, dynamic-safe, all existing tests still relevant
stronger cache mode, only enabled when explicitly configured
That is the clean way to make both true.
Your key point is correct:
current Modules behavior keeps “replaying the universe” partly because some semantics are computed at query time, including things like:
If you declare “I do not need live dynamism,” then the cache can become an actual index or even a precomputed view.
That suggests a good design:
Something like a config knob:
For example:
Start with:
Those are your biggest wins.
If static-fast mode is off:
If on and cache missing/incompatible:
For your use case, hard fail is better. It preserves the contract.
Yes, this is how you pass the normalYes. That is basically correct.
Current Modules behavior treats cache as:
So even with .modulecache, the runtime still preserves dynamic features:
That is why today’s cache still “replays the universe” after loading it. It is not a strict static index contract.
Your proposed model is different and much cleaner:
That is the right shape if your real goal is upstreamable performance without breaking general Modules semantics.
What this contract would mean technically:
Something like:
Example shape:
In static mode, you explicitly give up:
Then we can safely do:
Default mode:
Static mode:
So yes: if the contract is explicit, the semantic risk becomes manageable.
The only correction I’d make is this:
creating a cache alone does not currently mean “I agree to be static.” That is your proposed new contract, not the current one. And I think that distinction is exactly what makes an upstreamable design
possible.
Best upstream framing would be:
That is much easier to defend than trying to smuggle static assumptions into the existing cache model.
If you want, next step is I can sketch the contract precisely:
smprather commentedon May 28, 2026
Codex was not pleased with my chat copy/paste. It wanted a more polished comment 😃.
The important distinction is between:
Today,
.modulecacheis in category 1. It reduces filesystem I/O, but itdoes not mean the module tree has agreed to give up dynamic behavior. Because
of that, runtime still has to preserve semantics for things like aliases,
virtuals, modulerc effects, hidden/forbidden state, variants, and other
metadata-sensitive behavior.
That is why even cached lookups can still end up replaying a large part of the
module universe.
I think there is room for a new explicit mode where the administrator says:
runtime reconstruction
If that contract is explicit, then much more aggressive indexing becomes
reasonable without changing default behavior.
That could look like:
module cachebuild --static.modulecacheThis should let default behavior and existing dynamic tests stay intact, while
opening the door to much faster implementations of:
module availmodule avail foomodule avail foo/barmodule show foo/bar/1.2The main point is that the optimization boundary becomes semantic and
intentional, not implicit.
smprather commentedon May 28, 2026
Actually, we came up with a much more sane configuration interface. If a directory tree contains the new cache information, then that's the implicit static-mode opt-in for that directory tree. That way you can mix/match base dirs in the MODULEPATH. Codex is working on the implementation and testing now.
xdelaruelle commentedon May 31, 2026
Many thanks @smprather for your proposal. I am currently trying to see how this feature idea could be integrated.
Existing
.modulecacheare listing all the files contained in the modulepath, even non-modulefile files. And for modulefiles and modulercs, their content is recorded in the cache file.In situation like yours, it could lead to very large cache file that makes module quite unresponsive as the whole cache needs to be evaluated to proceed to the command.
So an alternate kind of cache, as you propose, seems interesting to add. I call this new kind of cache index.
This is an initial specification that came to mind (with still open questions to discuss):
Introduce a
.moduleindexfile under each modulepath.This file will list all the modulefiles and modulercs contained in the modulepath.
Each line represents a valid modulefile or modulerc with the following information separated by colon character:
open question: do we separated these information with colon character or do we parse the file as a Tcl script, as done with
.modulecachefile.open question: to generate
.moduleindexfiles, should we have dedicated sub-commands (indexbuild, indexclear) or command-line option onto the cache sub-commands?As planned for
.modulecache(see #563), a new option should be added to be able to list sub-commands that should ignore the.moduleindexfile.open question: could the
.moduleindexand.modulecachefile be both used for a given sub-command?open question: based on the answer to the previous question, if
.moduleindexand.modulecachehave no benefit for being both used, could index be an alternative content for.modulecache