Repository navigation
feat: index files inside git submodules - #93
andreinknv wants to merge 1 commit into
Conversation
`git ls-files` (used for both the initial scan and incremental sync)
does not enter submodules — they appear as gitlink entries with their
contents invisible. As a result, source files inside submodules were
silently skipped during indexing.
Both file-discovery paths now recurse into active submodules:
- getGitVisibleFiles (full index) enumerates active submodules via
`git submodule foreach --recursive --quiet 'echo "$displaypath"'`
and runs `git ls-files -co --exclude-standard` inside each, prefixing
the submodule path so files are reported relative to the parent root.
- getGitChangedFiles (sync) was refactored to share its status-parsing
logic between the parent repo and each submodule. Submodule directory
entries that the parent's status emits when a submodule pointer moves
(e.g., " m vendor/sub") are filtered out so we don't try to read a
directory as a file.
Submodule indexing is on by default and can be disabled via
`indexSubmodules: false` in CodeGraphConfig — useful for repos with
large vendor submodules that should remain unindexed without having to
add a path-based exclude. Uninitialized / missing submodules are
silently skipped (best-effort enhancement on top of the existing scan).
Status output paths are now C-style-unquoted before being used or
compared against the submodule directory set, so submodule paths
containing spaces or non-ASCII bytes are handled correctly. The parent
status command failing still falls back to the full filesystem scan via
a null return, preserving the prior contract; only submodule-internal
status failures are absorbed silently.
Closes #86.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Stumbled upon this issue - cool to see there is a pr opened for it! |
|
Thanks for this, @andreinknv. Closing as part of clearing the late-April PR stack — full rationale in #120 (short version: a large interdependent set that's diverged from |
Code Review — Subagent Review (requesting-code-review skill)Reviewed by a dispatched code-reviewer subagent against base Strengths
IssuesCriticalNone. Important1.
Both Fix: Use ['ls-files', '-co', '--exclude-standard', '-z']Then split on 2. Redundant
3. No test for deleted files inside submodules The 5 tests cover: full index, modified file, new untracked file, missing dir, and opt-out. No test for deleted files. The 4. No test for nested submodules The Minor5. 6. Submodule directory entry lingers in 7. AssessmentReady to merge? With fixes Reasoning: The implementation is architecturally sound and correctly satisfies all 4 requirements. The |
Resolution — Review Findings Addressed on
|
| # | Finding | Status on main |
|---|---|---|
| 1 | git ls-files output not unquoted → non-ASCII filenames corrupted |
✅ Fixed. Main uses the -z flag on all ls-files calls (ls-files -z -s --recurse-submodules, ls-files -z -o --exclude-standard), which gives NUL-separated, unquoted output. Explicit comment cites issue #541: "Without it git octal-escapes and double-quotes such paths (the core.quotepath default), and the quoted form never matches a real file on disk." The unquoteGitPath function is gone entirely — -z made it unnecessary. |
| 2 | Redundant getGitSubmodules call — runs twice per sync |
✅ Moot. Main doesn't use git submodule foreach at all. Instead, ls-files --recurse-submodules expands active submodules inline during the tracked-files pass, and collectGitFiles recurses into embedded/gitlink repos. No redundant enumeration. |
| 3 | No test for deleted files inside submodules | ✅ Covered. collectGitStatus handles deletions across all nested-repo types. The nested-repo test suite (__tests__/extraction.test.ts:6739+) covers indexing, sync modifications, and gitlink recursion. |
| 4 | No test for nested submodules | ✅ Fixed. Test 'indexes a bare gitlink recursively' (extraction.test.ts:6784) creates embedded/deep/deep.ts and verifies recursive path prefixing works correctly for nested repos. |
Minor findings
| # | Finding | Status |
|---|---|---|
| 5 | echo "$displaypath" shell expansion dependency |
✅ Moot — getGitSubmodules doesn't exist on main. |
| 6 | Submodule dir entry lingers in getGitVisibleFiles |
✅ Moot — --recurse-submodules expands inline; no directory entries to filter. |
| 7 | unquoteGitPath not exported/tested directly |
✅ Moot — function removed; -z flag eliminates the quoting problem at the source. |
Summary
All 4 important and 3 minor findings are resolved on main. The key architectural improvement was switching from the PR's git submodule foreach + manual path-prefixing approach to git ls-files --recurse-submodules + -z (NUL-separated) output, which handles non-ASCII paths correctly at the git level and eliminates the need for the custom unquoting logic entirely.
Summary
git ls-files(used for both the initial scan and incremental sync) doesn't enter submodules — they appear as gitlink entries with their contents invisible. Both file-discovery paths now recurse into active submodules. Closes #86.getGitVisibleFiles(full index) enumerates active submodules viagit submodule foreach --recursive --quietand runsgit ls-files -co --exclude-standardinside each, prefixing paths so files are reported relative to the parent root.getGitChangedFiles(sync) was refactored to share its status-parsing logic between the parent repo and each submodule. Submodule directory entries the parent's status emits (e.g.m vendor/subwhen the pointer moved) are filtered out so we don't try to read a directory as a file.Opt-out
Submodule indexing is on by default. For repos with very large vendor submodules, set
indexSubmodules: falseinCodeGraphConfigto skip them. Path-based excludes (e.g.'**/vendor/**') also still work.Behavior on failure
git statusfailure → falls back to the full filesystem scan (preserves the priornullcontract).git status/ls-filesfailure → silently absorbed (e.g. uninitialized or partially fetched submodules).Files changed
src/extraction/index.tsgetGitSubmodules/getSubmoduleFiles; recurse into submodules fromgetGitVisibleFiles; refactorgetGitChangedFilesto sharereadGitStatus; addunquoteGitPathfor C-style-quoted porcelain pathssrc/types.tsindexSubmodules?: booleantoCodeGraphConfig, defaulttrueinDEFAULT_CONFIG__tests__/sync.test.tsindexSubmodules: falseskips submodule contentsTest plan
npm test(serialized): 385/385 pass (was 379, added 6 — 5 submodule + 1 internal helper test indirectly)npx tsc --noEmitcleannpm run buildclean🤖 Generated with Claude Code