tests: suite is not invocation-stable — pytest/rtk-rewritten runs fail to collect while python -m pytest passes #34

Summary

python -m pytest is the documented and CI-used way to run the suite, and it works. But the suite is not invocation-stable: running the same tests through the pytest console script fails to collect, because tests/ is an implicit namespace package that only resolves when CWD lands on sys.path — which only happens under the python -m form.

This is reachable in normal use, not just by hand. rtk rewrite — which lecode runs on every bash tool call — maps python -m pytest to rtk pytest, silently switching the invocation path and turning any focused test run into a collection error.

Environment

Reproduction

Both of these are the same pytest, same arguments, same working directory:

# A — works
$ uv run python -m pytest tests/test_background.py -q
......................                                    [100%]
22 passed in 16.31s

# B — fails to collect
$ uv run pytest tests/test_background.py -q
________________ ERROR collecting tests/test_background.py ________________
tests/test_background.py:10: in <module>
    from tests.fakes import FakeProvider
E   ModuleNotFoundError: No module named 'tests'
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
1 error in 0.05s

The trigger in practice:

$ rtk rewrite 'uv run python -m pytest tests/test_background.py -q'
uv run rtk pytest tests/test_background.py -q

So an agent that asks for the documented command gets rtk pytest, which is form B — a collection error.

Scope

35 of 90 test modules do from tests... import ... and are affected. Modules that don't import tests.* are unaffected and quietly pass under both forms, which makes the failure look flaky rather than deterministic:

$ uv run rtk pytest tests/test_session_storage.py -q
Pytest: 53 passed in 0.23s          # no tests.* import — passes

Note the full-suite run happens to work under both forms, because testpaths/rootdir collection and the tests.fakes import each independently arrange for the repo root to be importable. Only targeted runs break — which is exactly what focused test runs do.

Root cause

tests/ has no __init__.py and there is no pythonpath setting in pyproject.toml:

[tool.pytest.ini_options]
asyncio_mode = "auto"
testpaths = ["tests"]
markers = [...]

With python -m pytest, Python implicitly puts the CWD at the front of sys.path:

$ uv run python -c "import sys; print(repr(sys.path[0]))"
''

The pytest console script does not — sys.path[0] is the script's own directory. So tests is importable in form A and not in form B.

Worth flagging for coherence with the documented contract: docs/agents-and-skills.md and docs/configuration.md both advertise the validation command as

validation = ["uv run ruff check", "uv run python -m pytest"]

Making the suite invocation-agnostic means that string keeps working and the rewritten form stops being a trap.

Suggested fix

Any one of these is sufficient; I verified all three against tests/test_background.py:

1. Pin pythonpath (most direct).

[tool.pytest.ini_options]
pythonpath = ["."]

$ uv run pytest tests/test_background.py -q
Pytest: 22 passed in 17.24s
$ uv run rtk pytest tests/test_background.py -q -o pythonpath=.
Pytest: 22 passed in 16.29s

2. Switch import mode.

$ uv run pytest tests/test_background.py -q --import-mode=importlib
Pytest: 22 passed in 16.30s

3. Add tests/__init__.py — makes tests a regular package on the rootdir path. Slightly more invasive; changes how tests.fakes resolves relative to rootdir.

I'd lean toward (1): one line, no change to collection semantics, and it makes every entry point agree.

Prior art (corrected)

I initially described the rewrite as "semantics-changing" and planned an upstream issue. Checking first, that was wrong on both counts, so recording it here rather than leaving a misleading note.

The rewrite itself is intended. python -m pytest → rtk pytest is covered by an existing upstream test, test_classify_python_m_pytest (src/discover/registry.rs), and is listed as designed behaviour in rtk-ai/rtk#4315. There is nothing to report here.

The binary-swap hazard is already tracked, and explicitly, in rtk-ai/rtk#1053:

A naive fix that normalizes in rewrite_segment() and emits rtk pytest -v would change semantics from “run this exact binary” to “run whatever PATH resolves.” RTK handlers generally discard the caller’s executable path and re-resolve via resolved_command() … This can produce wrong-binary execution or command-not-found failures.

Related, different: rtk-ai/rtk#3035 (a python -m classification bug, closed) documents the same classification path this issue exercises, and notes that the python -m rewrite tests all pass an empty exclusions list — i.e. this rewrite path is untested with real exclusions.

So the remaining point belongs here, not upstream: whatever rtk does, lecode's own suite should not depend on the invocation form, since the rewrite means the invocation form is not under lecode's control. That is what this issue tracks, and pythonpath = ["."] fixes it regardless of which provider the rewritten command resolves to.

Also worth noting that CI cannot currently catch this class of bug: .github/workflows/ci.yml installs a pass-through rtk shim whose rewrite echoes stdin unchanged, so CI never exercises a real rewrite. The suite is green there by construction. That's defensible (rtk has no apt/brew package) but it does mean rewrite-induced invocation drift is invisible to CI by design.

Impact

Focused test runs — the workflow AGENTS.md recommends — are the ones that break. The failure surfaces as an unrelated-looking ModuleNotFoundError: No module named 'tests' rather than anything pointing at the invocation path, which is a bad debugging experience for contributors and for agents running their own verification.