P23: coverage after Tests (capability coverage, opt-in)

.github/workflows/ci.yml

@@ -34,7 +34,7 @@ jobs:
       - run: npm run test:hints
       # The runtime under CPython: the visualizer's columns, docstrings,
       # condition comments, readable types and argument checks, the doctest
-      # runner, and which errors get a hint.
+      # runner and its coverage, and which errors get a hint.
       - run: npm run test:runtime
       # The console in Chrome (on the runner) with Pyodide from its CDN,
       # driving the embed built above: Tests → edit → Tests shows the new

CLAUDE.md

@@ -13,7 +13,7 @@ npm run build          # tsc -b && vite build && copy src/style.css -> dist/ (li
 npm run dev            # vite build --watch, for use with npm link/yalc in a host shell
 npm run lint           # oxlint
 npm run test:protocol  # builds the library, then runs scripts/check-embed-protocol.mjs against dist/
-npm run test:runtime   # the runtime under CPython: scripts/check-runtime-{columns,values,doctests,calls,hints,fidelity,replay}.py
+npm run test:runtime   # the runtime under CPython: scripts/check-runtime-{columns,values,doctests,coverage,calls,hints,fidelity,replay,…}.py
 npm run test:hints     # the error hints' wording (de/en) against the built dist/ (run a build first)
 npm run test:console   # the console in Chrome with Pyodide, against dist-embed/ (run build:embed first)
 npm run build:embed    # builds the iframe app + demo page into dist-embed/
@@ -56,7 +56,7 @@ There is no unit test framework. The automated checks are `test:protocol` (plain
 
 Worker messages:
 - `run` → `result`/`error`; while it runs, `output` messages (`{id, text}`, `id` undefined for a program, the input's id for a REPL input) stream what was printed, at most 50 per 50 ms. The final message brings the whole output again.
-- `test` → first as `run` (`result`/`error`, a fresh REPL session from these files), then `test-result` (the doctests' report; `{skipped: true}` if the file failed)
+- `test` → first as `run` (`result`/`error`, a fresh REPL session from these files), then `test-result` (the doctests' report; `{skipped: true}` if the file failed). With `coverage: true` the report has `uncovered` (`doctests.py` records the lines with `sys.monitoring` during the examples only; `check-runtime-coverage.py` compares with Python's `trace`); the UI shows it only when all examples pass.
 - `run-visual` → `trace-result` (`{trace, ast, output, graphics, error, errorMessage, truncated}`)
 - `repl` → `repl-result`
 - `run-visual-repl` (a doctest example) → `trace-result` with `rootPath`/`sources`, or `repl-result` if the input doesn't parse

README.md

@@ -159,6 +159,20 @@ glpython hint that it probably should `return` instead. Texts: `runTests`,
 `testsRunning`, `testsSummary`, `testsNone`, `testsLoadError`,
 `visualizeExample`, `testsPrintHint`.
 
+**Coverage** (capability `coverage`, off by default): when all examples pass,
+the report also lists which lines of the tested file's functions no example
+executed, each with its code («Zeile 11: `return "ungenügend"`»; consecutive
+lines of one function as «Zeilen 8–12:»). Counted are the statements in
+function bodies (also methods and nested functions), as Python's own `trace`
+module counts them during the examples - not `def` lines, docstrings, code
+outside functions, or other files such as libraries. With the editor, the same
+lines are marked in it (line background and a bar in the gutter) until the
+file is edited or the program runs again. When every line ran, a neutral note
+says so and that this doesn't show whether the examples check the edge cases:
+coverage finds untested code, not wrong code. Recorded with `sys.monitoring`,
+each line once, so it doesn't slow the examples down. Texts:
+`testsUncoveredHeading`, `testsUncoveredLines`, `testsAllCovered`.
+
 Call boxes in the visualizer show a function without its comments and
 docstring, which don't change what runs; the file itself is shown as written.
 
@@ -295,7 +309,9 @@ https://example.test/embed/?parentOrigin=https%3A%2F%2Fcourse.example
 Optional query parameters adapt the embed to a host such as the LiaScript
 macro: `features=run,repl,visualize,…` hides the REPL (`repl`), the visualizer
 (`visualize`) or its → chain export (`export`) unless listed (capabilities
-`repl`, `visualization`, `export`), and `size=large` with
+`repl`, `visualization`, `export`); `coverage` in that list switches on the
+coverage after Tests (capability `coverage`), which a missing `features`
+doesn't; and `size=large` with
 `visualizerHeight=<px>` shows bigger code and a visualizer at least that tall.
 
 The UI language is the optional `lang` query parameter (e.g. the LiaScript
@@ -353,14 +369,14 @@ and monospace fonts, no web fonts). A host can override any of them:
 | Variable | Role |
 |---|---|
 | `--glp-bg` | panel background |
-| `--glp-surface` | code in the visualizer, output, REPL |
+| `--glp-surface` | code in the visualizer, output, REPL; coverage after Tests |
 | `--glp-border` | decorative borders and separator lines |
-| `--glp-border-control` | borders of controls, call boxes in box mode |
+| `--glp-border-control` | borders of controls, call boxes in box mode; coverage after Tests (its bar) |
 | `--glp-text`, `--glp-text-muted` | text and code; line numbers, step counter, secondary info, connector lines |
 | `--glp-accent`, `--glp-on-accent` | active mode, scrubber, focus, primary button; text on it |
 | `--glp-current`, `--glp-current-outline` | current subexpression and new value, current line; next step (dashed) |
 | `--glp-error`, `--glp-error-bg` | errors, failing tests, «Erhalten · Erwartet» |
-| `--glp-warning`, `--glp-warning-bg` | hints |
+| `--glp-warning`, `--glp-warning-bg` | hints; the coverage marks in the editor |
 | `--glp-success` | passed tests, runtime ready |
 | `--glp-syntax-keyword`, `-string`, `-number`, `-comment` | syntax colours (names, operators, brackets use `--glp-text`) |
 | `--glp-font-code`, `--glp-font-ui`, `--glp-radius` | fonts and corner radius |

package.json

@@ -29,7 +29,7 @@
     "test:protocol": "npm run build && node scripts/check-embed-protocol.mjs",
     "test:hints": "node scripts/check-error-hints.mjs",
     "test:console": "node scripts/check-console-browser.mjs",
-    "test:runtime": "python3 -B scripts/check-runtime-columns.py && python3 -B scripts/check-runtime-values.py && python3 -B scripts/check-runtime-doctests.py && python3 -B scripts/check-runtime-calls.py && python3 -B scripts/check-runtime-hints.py && python3 -B scripts/check-runtime-fidelity.py && python3 -B scripts/check-runtime-replay.py && python3 -B scripts/check-runtime-imports.py && python3 -B scripts/check-runtime-repl.py",
+    "test:runtime": "python3 -B scripts/check-runtime-columns.py && python3 -B scripts/check-runtime-values.py && python3 -B scripts/check-runtime-doctests.py && python3 -B scripts/check-runtime-coverage.py && python3 -B scripts/check-runtime-calls.py && python3 -B scripts/check-runtime-hints.py && python3 -B scripts/check-runtime-fidelity.py && python3 -B scripts/check-runtime-replay.py && python3 -B scripts/check-runtime-imports.py && python3 -B scripts/check-runtime-repl.py",
     "prepare": "npm run build"
   },
   "peerDependencies": {

scripts/check-console-browser.mjs

@@ -4,6 +4,9 @@
 // the second report used to land out of view below the first one). Tests
 // starts as Run does, so the console then holds only the new report, and
 // the REPL has the definitions just tested (not the ones of the last Run).
+// With "coverage" in features (P23), a report that all passed lists the
+// lines no example executed - which also checks that Pyodide's Python has
+// sys.monitoring - and without it, or with a failing example, it doesn't.
 //
 // Drives the built embed (dist-embed/, from npm run build:embed) in Chrome
 // through its protocol, as the LiaScript macro does; Pyodide comes from its
@@ -21,15 +24,16 @@ const stopServer = () => server.kill()
 process.on('exit', stopServer)
 
 // The parent page, served on the embed's own origin: it loads the embed and
-// passes messages on, version 4.
+// passes messages on, version 4. Its own query (e.g. "?features=…") goes on
+// to the embed.
 const harness = `<!doctype html><meta charset="utf-8"><body style="margin:0">
 <iframe id="embed" style="width:760px;height:900px;border:0"></iframe>
 <script>
   const iframe = document.querySelector('#embed')
   window.ready = false
   window.send = (type, source) => iframe.contentWindow.postMessage({ type: 'glpython:' + type, version: 4, source, mode: 'eval' }, location.origin)
   addEventListener('message', (event) => { if (event.data?.type === 'glpython:ready') window.ready = true })
-  iframe.src = './index.html?parentOrigin=' + encodeURIComponent(location.origin) + '&mode=eval&lang=de'
+  iframe.src = './index.html?parentOrigin=' + encodeURIComponent(location.origin) + '&mode=eval&lang=de' + location.search.replace('?', '&')
 </script>`
 
 const program = (examples) => `def schnitt(a, b):\n    """\n${examples.map(([call, want]) => `    >>> ${call}\n    ${want}\n`).join('')}    """\n    return (a + b) / 2\n`
@@ -113,6 +117,35 @@ try {
     console.error(error.message)
   }
   await page.close()
+
+  // Coverage after Tests.
+  const bewertung = `def bewertung(note):\n    """\n    >>> bewertung(5)\n    'genügend'\n    >>> bewertung(6)\n    'genügend'\n    """\n    if note >= 4:\n        return "genügend"\n    else:\n        return "ungenügend"\n`
+  const istGenuegend = `def ist_genuegend(note):\n    """\n    >>> ist_genuegend(5.5)\n    True\n    >>> ist_genuegend(3.5)\n    False\n    """\n    return note >= 4\n`
+  const bewertungFailing = bewertung.replace(">>> bewertung(6)\n    'genügend'", ">>> bewertung(6)\n    'gut'")
+  for (const [label, query, source, expected] of [
+    ['coverage: a line no example ran', '?features=run,repl,visualize,tests,coverage', bewertung, 'Von keinem Beispiel ausgeführt:\nZeile 11: return "ungenügend"'],
+    ['coverage: every line ran', '?features=run,repl,visualize,tests,coverage', istGenuegend, 'Jede Zeile der Funktionen wurde von mindestens einem Beispiel ausgeführt. Ob die Beispiele auch die Grenzfälle prüfen, zeigt das nicht.'],
+    ['coverage: not with a failing example', '?features=run,repl,visualize,tests,coverage', bewertungFailing, null],
+    ['coverage: not without it in features', '?features=run,repl,visualize,tests', bewertung, null],
+    ['coverage: not without features', '', bewertung, null],
+  ]) {
+    const page = await browser.newPage({ viewport: { width: 800, height: 950 } })
+    await page.route((url) => url.pathname === '/check-console.html', (route) => route.fulfill({ contentType: 'text/html', body: harness }))
+    await page.goto(`${origin}/check-console.html${query}`)
+    await page.waitForFunction(() => window.ready, null, { timeout: 120000 })
+    const frame = page.frameLocator('#embed')
+    await page.evaluate((source) => window.send('test', source), source)
+    await frame.locator('.test-summary').waitFor({ timeout: 60000 })
+    try {
+      const coverage = (await frame.locator('.test-coverage').allInnerTexts()).map((text) => text.replace(/\n+/g, '\n'))
+      assert.deepEqual(coverage, expected === null ? [] : [expected], label)
+      console.log(`${label}: ok`)
+    } catch (error) {
+      failed += 1
+      console.error(error.message)
+    }
+    await page.close()
+  }
 } finally {
   await browser.close()
   stopServer()