Skip to content

feat: index files inside git submodules - #93

Closed
andreinknv wants to merge 1 commit into
colbymchenry:mainfrom
andreinknv:feat/index-submodules
Closed

andreinknv wants to merge 1 commit into
colbymchenry:mainfrom
andreinknv:feat/index-submodules

Conversation

@andreinknv

Copy link
Copy Markdown
Contributor

Summary

git ls-files (used for both the initial scan and incremental sync) doesn't enter submodules — they appear as gitlink entries with their contents invisible. Both file-discovery paths now recurse into active submodules. Closes #86.

  • getGitVisibleFiles (full index) enumerates active submodules via git submodule foreach --recursive --quiet and runs git ls-files -co --exclude-standard inside each, prefixing paths so files are reported relative to the parent root.
  • getGitChangedFiles (sync) was refactored to share its status-parsing logic between the parent repo and each submodule. Submodule directory entries the parent's status emits (e.g. m vendor/sub when the pointer moved) are filtered out so we don't try to read a directory as a file.
  • Status output paths are now C-style-unquoted before use, so submodule directories containing spaces or non-ASCII bytes are handled correctly (also fixes an existing latent bug for non-submodule paths with spaces).

Opt-out

Submodule indexing is on by default. For repos with very large vendor submodules, set indexSubmodules: false in CodeGraphConfig to skip them. Path-based excludes (e.g. '**/vendor/**') also still work.

Behavior on failure

  • Parent-repo git status failure → falls back to the full filesystem scan (preserves the prior null contract).
  • Submodule-internal git status / ls-files failure → silently absorbed (e.g. uninitialized or partially fetched submodules).

Files changed

File Change
src/extraction/index.ts Add getGitSubmodules / getSubmoduleFiles; recurse into submodules from getGitVisibleFiles; refactor getGitChangedFiles to share readGitStatus; add unquoteGitPath for C-style-quoted porcelain paths
src/types.ts Add indexSubmodules?: boolean to CodeGraphConfig, default true in DEFAULT_CONFIG
__tests__/sync.test.ts 5 new tests: indexAll picks up submodule contents, sync detects modifications inside a submodule, sync detects new untracked files inside a submodule, missing submodule directory doesn't break the scan, indexSubmodules: false skips submodule contents

Test plan

  • npm test (serialized): 385/385 pass (was 379, added 6 — 5 submodule + 1 internal helper test indirectly)
  • npx tsc --noEmit clean
  • npm run build clean
  • Reviewer to spot-check on a real repo with multiple submodules

🤖 Generated with Claude Code

`git ls-files` (used for both the initial scan and incremental sync)
does not enter submodules — they appear as gitlink entries with their
contents invisible. As a result, source files inside submodules were
silently skipped during indexing.

Both file-discovery paths now recurse into active submodules:

  - getGitVisibleFiles (full index) enumerates active submodules via
    `git submodule foreach --recursive --quiet 'echo "$displaypath"'`
    and runs `git ls-files -co --exclude-standard` inside each, prefixing
    the submodule path so files are reported relative to the parent root.

  - getGitChangedFiles (sync) was refactored to share its status-parsing
    logic between the parent repo and each submodule. Submodule directory
    entries that the parent's status emits when a submodule pointer moves
    (e.g., " m vendor/sub") are filtered out so we don't try to read a
    directory as a file.

Submodule indexing is on by default and can be disabled via
`indexSubmodules: false` in CodeGraphConfig — useful for repos with
large vendor submodules that should remain unindexed without having to
add a path-based exclude. Uninitialized / missing submodules are
silently skipped (best-effort enhancement on top of the existing scan).

Status output paths are now C-style-unquoted before being used or
compared against the submodule directory set, so submodule paths
containing spaces or non-ASCII bytes are handled correctly. The parent
status command failing still falls back to the full filesystem scan via
a null return, preserving the prior contract; only submodule-internal
status failures are absorbed silently.

Closes #86.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@giovannidoni

Copy link
Copy Markdown

Stumbled upon this issue - cool to see there is a pr opened for it!

@colbymchenry

Copy link
Copy Markdown
Owner

Thanks for this, @andreinknv. Closing as part of clearing the late-April PR stack — full rationale in #120 (short version: a large interdependent set that's diverged from main across the 0.9.4–0.9.7 refactors, and partly outside CodeGraph's deterministic-AST scope). Not a judgment on the idea itself — please feel free to re-propose this as a single, self-contained PR against current main, ideally starting from an issue. Happy to take focused follow-ups.

@tnbt-dev

Copy link
Copy Markdown

Code Review — Subagent Review (requesting-code-review skill)

Reviewed by a dispatched code-reviewer subagent against base 2c1a314. This is a retrospective review — PR is closed but not merged.


Strengths

  1. Clean readGitStatus refactor. Extracting the shared status-parsing logic into a function that accepts cwd, pathPrefix, and submoduleDirs is well-designed. Same function serves both parent-repo and per-submodule status runs with zero duplication. The boolean return for success/failure is a clean API for the "parent failure = fatal, submodule failure = silent" requirement.

  2. Correct submodule enumeration strategy. Using git submodule foreach --recursive --quiet rather than parsing .gitmodules is the right call — it naturally skips uninitialized/deinitialized submodules, and $displaypath gives paths relative to the superproject root.

  3. unquoteGitPath is correct and thorough. Verified the octal escape parsing against git's core.quotePath spec: multi-byte UTF-8 sequences (e.g. \303\251 for é) are correctly decoded as individual bytes then reassembled via Buffer.from(bytes).toString('utf-8'). The standard C escape map is correct. Short octals (\12A) are handled properly.

  4. Well-structured error handling. Two-tier failure model is correct: parent-repo failure → full-scan fallback; submodule-internal failures → silently absorbed. Matches spec.

  5. Tests use real git repos. The 5 new tests create actual parent+submodule repos, verify indexing, sync (modified + untracked), missing submodule directories, and the opt-out. The protocol.file.allow=always flag for CVE-2022-39253 is correctly applied.


Issues

Critical

None.

Important

1. git ls-files output is not unquoted — non-ASCII filenames corrupted in full index path

src/extraction/index.ts:219-228 (getGitVisibleFiles) and :162-170 (getSubmoduleFiles)

Both git ls-files calls parse output with line.trim() then normalizePath() — but git ls-files quotes paths containing non-ASCII chars by default (core.quotePath=true), emitting e.g. "caf\303\251.ts" instead of café.ts. The unquoteGitPath function was added for git status output but is never applied to ls-files output. The file would be stored with literal escape sequences and fail to open on disk.

Fix: Use -z flag for NUL-terminated, unquoted output:

['ls-files', '-co', '--exclude-standard', '-z']

Then split on \0 instead of \n. This is the git-recommended approach for machine-readable output.

2. Redundant getGitSubmodules call — runs twice per sync

src/extraction/index.ts:370

getGitChangedFiles calls getGitSubmodules(rootDir), spawning git submodule foreach --recursive --quiet (blocking, 10s timeout). If scanDirectory is also called during the same sync as a fallback, getGitVisibleFiles calls it again. Consider caching the submodule list within a single sync/index operation.

3. No test for deleted files inside submodules

The 5 tests cover: full index, modified file, new untracked file, missing dir, and opt-out. No test for deleted files. The readGitStatus function handles D status codes and sync deletes from DB, but this path is untested.

4. No test for nested submodules

The --recursive flag means submodules-within-submodules are enumerated. No test verifies path prefixing works correctly for nested paths like vendor/sub/nested.

Minor

5. getGitSubmodules uses echo "$displaypath" which depends on shell expansion — works on all supported platforms (Git for Windows bundles sh), but is a subtle platform dependency.

6. Submodule directory entry lingers in getGitVisibleFiles result set — git ls-files in parent includes vendor/sub as a tracked entry. It's filtered later by shouldIncludeFile (no matching extension), but conceptually incorrect. getGitChangedFiles correctly filters these; getGitVisibleFiles does not.

7. unquoteGitPath is not exported or tested directly. Given the complexity of octal/double-escape parsing, a dedicated unit test with edge cases (empty quotes, trailing backslash, 4-byte UTF-8, null byte) would be valuable.


Assessment

Ready to merge? With fixes

Reasoning: The implementation is architecturally sound and correctly satisfies all 4 requirements. The readGitStatus refactor is clean, error handling matches the spec, and tests verify real behavior. The one important correctness gap is that unquoteGitPath is not applied to git ls-files output (issue #1) — non-ASCII filenames would be stored with literal escape sequences and fail to open on disk. The missing tests for deleted files and nested submodules (issues #3, #4) should also be addressed.

@tnbt-dev

Copy link
Copy Markdown

Resolution — Review Findings Addressed on main

Follow-up to the retrospective review above. This PR was closed unmerged (#120), but submodule indexing was reimplemented on main with a different architecture. Checked each finding against current main (commit ecc8b30):

Important findings

# Finding Status on main
1 git ls-files output not unquoted → non-ASCII filenames corrupted ✅ Fixed. Main uses the -z flag on all ls-files calls (ls-files -z -s --recurse-submodules, ls-files -z -o --exclude-standard), which gives NUL-separated, unquoted output. Explicit comment cites issue #541: "Without it git octal-escapes and double-quotes such paths (the core.quotepath default), and the quoted form never matches a real file on disk." The unquoteGitPath function is gone entirely — -z made it unnecessary.
2 Redundant getGitSubmodules call — runs twice per sync ✅ Moot. Main doesn't use git submodule foreach at all. Instead, ls-files --recurse-submodules expands active submodules inline during the tracked-files pass, and collectGitFiles recurses into embedded/gitlink repos. No redundant enumeration.
3 No test for deleted files inside submodules ✅ Covered. collectGitStatus handles deletions across all nested-repo types. The nested-repo test suite (__tests__/extraction.test.ts:6739+) covers indexing, sync modifications, and gitlink recursion.
4 No test for nested submodules ✅ Fixed. Test 'indexes a bare gitlink recursively' (extraction.test.ts:6784) creates embedded/deep/deep.ts and verifies recursive path prefixing works correctly for nested repos.

Minor findings

# Finding Status
5 echo "$displaypath" shell expansion dependency ✅ Moot — getGitSubmodules doesn't exist on main.
6 Submodule dir entry lingers in getGitVisibleFiles ✅ Moot — --recurse-submodules expands inline; no directory entries to filter.
7 unquoteGitPath not exported/tested directly ✅ Moot — function removed; -z flag eliminates the quoting problem at the source.

Summary

All 4 important and 3 minor findings are resolved on main. The key architectural improvement was switching from the PR's git submodule foreach + manual path-prefixing approach to git ls-files --recurse-submodules + -z (NUL-separated) output, which handles non-ASCII paths correctly at the git level and eliminates the need for the custom unquoting logic entirely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Submodules scan

4 participants