Skip to content

Look up parents in a map when computing metrics - #1259

Merged
marco-c merged 3 commits into
masterfrom
fast-parent-lookup
Nov 21, 2026
Merged

marco-c merged 3 commits into
masterfrom
fast-parent-lookup

Conversation

@marco-c

@marco-c marco-c commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1258.

Problem

tree-sitter doesn't store parent pointers: tree_sitter::Node::parent walks down from the root of the tree, so it costs O(depth). Some metrics walk up through all the ancestors of a node with repeated parent() calls (e.g. the C++ LOC metric, for every declaration), which costs O(depth²) per node. On very deep trees, computing the metrics of a single file takes hours. That's what made the "check tree-sitter-mozcpp" CI task always hit its deadline: two generated files in mozilla-central, security/manager/ssl/nsSTSPreloadList.inc (tree ~6,900 levels deep) and third_party/wasm2c/src/prebuilt/lexer-keywords.cc, never finished.

Fix

metrics() now computes the parent of each node once, with a single cursor pass over the tree (Parents). Node gets a private, optional reference to that map, propagated to every node reached from it (children, siblings, cursors), and Node::parent() looks the parent up there. Nodes created without it, outside of metrics(), keep using tree-sitter's lookup, and so do nodes missing from the map: a ParserTrait implementation can return a subtree from get_root(), and its ancestors must still be found (for the root of a whole tree, tree-sitter's lookup returns immediately). So parent() always returns the same node as before.

Node's public field (node.0) is unchanged, but since Node now also has a private field, it can't be constructed directly outside of the crate anymore. From<tree_sitter::Node> is provided instead.

Results

Measured with release builds on a local mozilla-central checkout (20 cores):

master this PR
third_party/wasm2c/src/prebuilt/lexer-keywords.cc didn't finish in 2 min (hours in CI) 0.8 s
security/manager/ssl/nsSTSPreloadList.inc didn't finish in 2 min (hours in CI) 23–29 s, nearly all in tree-sitter's parser (error recovery)
dom/ C/C++ files (5,964), CPU time 134–151 s 97–101 s
dom/, peak RSS ~200 MB ~265 MB
All 42,202 C/C++ files stuck on the 2 files above 6.4 min

The outputs are byte-identical to master's for all 41,970 files that master can process, and the two files above now produce their metrics too.

Tests

  • node::tests::parents_match_tree_sitter: the map gives the same parent as tree-sitter for every node of a C++ sample.
  • node::tests::parents_above_subtree_root: with the arrow function of const f = () => 1; as root, its parent is still the variable declarator, and the JavaScript checker still counts it as a function rather than a closure.
  • node::tests::metrics_on_deep_tree: 500 nested namespaces, each with a declaration (25 s with master in release, 0.2 s with this PR). The test doesn't assert on time, to avoid flakiness, but without the fix it takes ~10× longer even at this size.

Notes

  • This also reverts the exclusion of the two files above from the grammar check tasks, added in build(deps): Update tree-sitter to 0.26.5 and all grammar dependencies #1225 as a stopgap.
  • While doing this, I noticed that Node::has_sibling iterates over the children of the node itself rather than those of its parent, so it behaves like Node::is_child. This PR keeps that behavior unchanged; it probably deserves a separate look.

@codecov-commenter

codecov-commenter commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.09901% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.34%. Comparing base (6fa4e75) to head (182ae1f).

Files with missing lines Patch % Lines
src/node.rs 89.79% 10 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1259      +/-   ##
==========================================
+ Coverage   70.25%   70.34%   +0.09%     
==========================================
  Files          62       62              
  Lines       16325    16414      +89     
==========================================
+ Hits        11469    11547      +78     
- Misses       4856     4867      +11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

tree-sitter doesn't store parent pointers: `tree_sitter::Node::parent`
walks down from the root of the tree, so it costs O(depth). Some metrics
walk up through all the ancestors of a node (e.g. the C++ LOC metric for
every declaration), which then costs O(depth²) per node. On very deep
trees, like the ones of some generated files in mozilla-central, computing
the metrics of a single file took hours.

`metrics` now computes the parent of each node once, with a single cursor
pass over the tree, and the nodes it visits look their parent up in that
map. Nodes created elsewhere keep using tree-sitter's lookup.

Since `Node` now has a private field, it can no longer be constructed
directly outside of the crate; `From<tree_sitter::Node>` is provided
instead.

Fixes #1258.
This reverts commit ac94f0c.

With parents looked up in a map, computing the metrics of
security/manager/ssl/nsSTSPreloadList.inc and
third_party/wasm2c/src/prebuilt/lexer-keywords.cc takes seconds instead of
hours, so they no longer need to be excluded.
The parent map only contains the nodes below the root it was computed
from. A ParserTrait implementation can return a subtree from get_root(),
and then the ancestors above it were no longer found: e.g. with the arrow
function of `const f = () => 1` as root, the JavaScript checker didn't see
the enclosing variable declarator and counted a closure instead of a
function.

On a miss, look the parent up with tree-sitter instead. For the root of
the whole tree, that returns immediately.
@marco-c
marco-c merged commit 4c5328f into master Nov 21, 2026
2 checks passed
@marco-c
marco-c deleted the fast-parent-lookup branch November 21, 2026 01:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Computing metrics takes hours on files with very deep syntax trees

2 participants