Repository navigation
Description
Noticed from the c64-test-harness side. This is a discovery/layout issue rather than a broken-test issue — the rig tests themselves are fine and do real verification when run the documented way.
The layout inverts the usual convention
| Directory | Contents | Collected by pytest |
|---|---|---|
tools/ (+ libs/*/tools/) |
the actual pytest suite | 154 tests |
tests/ |
five script-style live rig tests | 0 tests |
All five files in tests/ are main() scripts behind if __name__ == "__main__": sys.exit(main()), with no def test_ functions:
test_phase1_dhcp.py 0 test fns 0 collected
test_phase2_http.py 0 test fns 0 collected
test_phase3_https_1mhz.py 0 test fns 0 collected
test_phase3_https.py 0 test fns 0 collected
test_vice_https_macos.py 0 test fns 0 collected
That is deliberate — README lines 333-334 document running them directly:
sudo PYTHONPATH=tools python3 tests/test_phase1_dhcp.py
sudo PYTHONPATH=tools python3 tests/test_phase2_http.pyand they do verify properly that way. test_vice_https_macos.py hard-fails on FAIL: DHCP not acquired after 3 attempts and only prints PASS after a completed TLS handshake + GET, so it is not a vacuous test.
Why it still bites
The files carry test_*.py names in a directory called tests/, which is exactly pytest's discovery convention — but they are the one part of the tree pytest cannot collect. Depending on where you invoke from you get opposite signals:
$ pytest tests/ # rc=5, "no tests collected" -- loud, fine
$ pytest # rc=0, "154 tests collected" -- from tools/, tests/ silently contributes 0The root-level invocation is the problem: it succeeds, reports a healthy-looking 154 passed, and gives no indication that the five network tests in tests/ never ran. Anyone reasonably assumes a green pytest covered them.
There is also no [tool.pytest.ini_options] in pyproject.toml — no testpaths, no norecursedirs — so which behaviour you get depends entirely on the working directory.
Flagging it because #101 (fix/package-verify-vacuous-pass) suggests this class of "looked green, verified nothing" is one you care about. This is a milder cousin: the tests are real, but a green root-level pytest overstates what was covered.
Options
Any one of these closes the gap:
- Move the rig scripts out of pytest's namespace — e.g.
tools/rig/orscripts/, or rename torig_phase1_dhcp.py. Cheapest, and honest about them being manual sudo/rig-dependent scripts. - Add thin pytest wrappers that shell out to
main()andpytest.skip()when the rig is absent (nofeth0, no root). Then a root-levelpytestreports them as skipped rather than omitting them silently. - Pin the boundary in config — set
testpathsinpyproject.tomlso the invocation is unambiguous, and note intests/READMEthat the directory is manual-only.
Option 2 is the most informative if you want a single command to reflect true coverage; option 1 is the least work.
Not an issue
To be explicit about what I am not reporting: the rig tests are not broken, they are not passing vacuously when run as documented, and pytest tests/ does fail loudly (rc=5) rather than pretending. The only misleading path is a bare pytest at the repo root.
🤖 Generated with Claude Code
Activity