Measure rendered styles in the browser, smoke-test deploys, warn before waivers expire

[adewale]

Summary

This PR implements the pythonbyexample recommendations from the September 2026 verification audit (https://github.com/adewale/testing-best-practices/blob/claude/github-testing-practices-review-n5749j/research/PORTFOLIO_VERIFICATION_AUDIT_2026-09.md, pythonbyexample row). There are three implementation commits, plus a merge of main and a pruning commit.

1. P1: rendered-style contracts replace CSS/JS source-text tests.

tests/test_app.py read public/site.css 17 times and read the JS sources repeatedly. It asserted literal text such as "transform: scale(0.96)", "text-wrap: balance" or "execution: 'execute'". docs/lessons-learned.md records these tests breaking on the harmless has-figure change, and they still pass when the rendered page regresses.

scripts/check_browser_layout.mjs already drives a real Worker and Chrome in CI. It now measures the same intentions on the page:

• Pressed states: computed transforms under a forced :active (via CSS.forcePseudoState) for Run, Copy link and the copy button (0.96), and for home cards (0.99).
• Sizing and type: touch targets of at least 40px for runner buttons and nav links. Also underlined nav links, balanced h1, antialiased body text, tabular execution time, and no element with transition: all.
• Layout:
  • Body text follows the reader's browser font size (Page.setFontSizes at 20px).
  • The skip link is off-screen until focused, then on-screen.
  • The share button sits at the toolbar's end, and the copy button is anchored to its cell.
  • Long output wraps without page overflow, and tall output grows the panel instead of being clipped.
  • The fallback textarea does not overflow its panel.
• Contrast: in both light and dark themes, body text ≥ 4.5, the Run button ≥ 4.5, and the output terminal ≥ 7. The page palette switches with the theme, and marginalia figures stay on light paper in dark mode.
• Accessibility fallbacks: the header turns solid under emulated prefers-reduced-transparency and prefers-contrast: more, and nav links get the full text colour under more contrast. The nav is visible on landing.
• Behaviour previously asserted as JS text:
  • Network errors end a run with Run failed: ….
  • An unedited Copy link copies the plain page URL.
  • The copy button falls back to execCommand('copy') without the Clipboard API. Its status is an off-screen polite live region, and its mask glyph changes.
  • Arrow keys ignore modifier keys and focused buttons. ArrowLeft walks back, and at the first example it is a no-op with no error.
  • Search results expose combobox, listbox and option state.
  • The Turnstile widget renders with execution: 'execute' and no size, and is removed and hidden afterwards.
  • Every design token on the About page resolves.
• Arrow-guard fix: the existing checks read the pathname 50 ms later, which cannot see a navigation that has started but not committed. They now count attempted navigations with the Navigation API.
• Removed: the source-text assertions, including 19 tests made only of them. HTML-rendering assertions stay.

2. P1: make deploy smoke-tests the deployed origin. Deploy now ends with a post-deploy-smoke step. It runs scripts/smoke_deployment.py against DEPLOY_URL (default https://www.pythonbyexample.dev). If any GET or POST check fails, it exits non-zero with "the new version is already live: investigate now, or roll back". SMOKE_ARGS passes extra flags explicitly. This runs only inside the manual make deploy, never in CI.

2 (continued). P2: waiver expiry warns 30 days ahead, with a manual fail-early check.

• check_quality_scores.py warns when a waiver has 30 days or fewer left. --fail-within-days N turns that window into an error, and --as-of fixes the date.
• make check-waiver-expiry runs --fail-within-days 30. It is a manual command, not run in CI. (An earlier revision added a weekly Scheduled quality workflow; it was removed when the PR was pruned to the owner's verification-cost standard.)
• The hello-world waiver expires 2026-12-01. Before this change, its first signal would have been a failure on every PR from that day. Now every CI run prints a warning from 2026-11-01, and make check-waiver-expiry fails from that day.

3. P2/P3: stale docs and phrase-order tests.

• docs/lessons-learned.md said "54 tests today cover 9 contract families". tests/test_marginalia_geometry.py has 31 tests in 13 contract classes. The sentence now points at the classes instead of giving a number.
• The same file cited scripts/refresh_golden_fixture.py, which was deleted in 0929c25. It now names what catches loader and parser regressions today.
• tests/test_markdown_migration_prereqs.py loses its spec-wording tests, including the "Red" < "Green" < "Refactor" order check. The README/CI command-contract test stays.

Cost and limits

Pruned to the owner's verification-cost standard on 2026-10-09.

What CI runs for this PR. Only the existing verify job in verify.yml. This PR adds no workflows, jobs, matrix entries or schedules.

• The new checks live inside steps that make verify already runs: browser-layout-test (against the local pywrangler dev Worker, never a deployed origin) and the unit tests.
• Added cost: about 14 s per run, in the browser test. Re-measured on 2026-10-09 against the same local Worker with sandbox Chromium: 20.0 s on this branch versus 6.2 s for origin/main's script. Both runs stopped on the same environment-only failure described below. The 2 new expiry unit tests add well under a second.
• On GitHub, verify passed in 57 s total on the pre-prune head 1d5cd26.
• No raised timeouts or retries. The browser test's waitFor budget (300 × 100 ms) is unchanged from main.

Moved to manual.

• The weekly Scheduled quality workflow (.github/workflows/scheduled-quality.yml, cron 23 6 * * 1 plus workflow_dispatch) is deleted. Its two steps, run by hand:
  • make quality-checks: already part of make verify, so every PR runs it.
  • make check-waiver-expiry: exits 1 while any quality waiver has 30 days or fewer left. Run it monthly or before a release. It is documented in README.md, CONTRIBUTING.md and the Makefile.
• What remains in CI: check_quality_scores.py prints WARNING: quality waiver … during the final 30 days and fails once a waiver has expired.
• Kept as before: make post-deploy-smoke, which runs only inside the manual make deploy. No CI step or schedule calls production.

PBT and fuzz budgets. This PR adds or changes no property-based or fuzz tests. tests/test_parser_properties.py (Hypothesis) is unchanged and keeps the project's existing settings. No mutation testing is added anywhere.

What is not verified.

• With no schedule, nothing alerts anyone before a waiver expires unless someone reads the CI warning or runs make check-waiver-expiry. The hello-world waiver expires on 2026-12-01, so the warning window opens on 2026-11-01.
• In this session's sandbox Chromium (Playwright build 1194, run with --no-sandbox), the browser test fails Search focus indicator is too weak and CodeMirror focus indicator is too weak. Those checks already exist on main, and main's script fails them identically here. All the checks this PR adds passed. CI uses google-chrome, where they pass.
• GitHub Actions has not yet run on the pruned head.
• make deploy and the deploy smoke against production were not run.
• Local setup that is not committed: pywrangler needs uv ≥ 0.12.3, so a newer uv was installed into the session scratchpad. The sandbox uv is 0.11.32.
• The branch was 2 commits behind main (chore: patch Wrangler tooling security dependencies #27, Wrangler dependency patches), so main was merged in with a normal merge commit (68c7387). There was no conflict.

Checks on the pruned head (59ec97f), with CHROME_PATH pointing at a scratch wrapper for /opt/pw-browsers/chromium:

• npm audit --audit-level=high: pass.
• git diff --exit-code -- pylock.toml: pass, after pywrangler dev.
• make verify, run target by target:
  • make build test seo-cache-lint verify-examples quality-checks: pass, 227 tests OK.
  • browser-layout-test: fail, on the environment-only focus checks above.
  • search-ranking-test, lint, check-generated: pass.
• scripts/format_examples.py --check: pass.
• make verify-python-version VERSION=3.13: pass.
• git diff --check: pass.
• make check-waiver-expiry: pass.

Verification (original, pre-prune head 1d5cd26)

• make verify. I ran each target against a local pywrangler dev on port 9796 (to avoid clashing with other local servers) with Chrome for Testing 154:
  • make build test seo-cache-lint verify-examples quality-checks search-ranking-test lint check-generated exited 0, with 227 tests OK.
  • node scripts/check_browser_layout.mjs http://127.0.0.1:9796/examples/values exited 0.
• scripts/format_examples.py --check: exit 0.
• make verify-python-version VERSION=3.13: exit 0, "Verified 109 example(s)".
• git diff --check is clean, npm audit --audit-level=high exits 0, and git diff --exit-code -- pylock.toml shows no change.
• Regression checks fail when the behaviour is reverted. Details are below.
• Manual verification: make post-deploy-smoke DEPLOY_URL=http://127.0.0.1:9796 against the local Worker printed "Deployment smoke OK (9 GETs, 5 POSTs)". The first attempt hit a local-dev ProxyWorker … Network connection lost on POST /examples/subprocesses. Retried by itself, that POST returned 200 three times, and the rerun passed.

Red, then green.

Browser contracts. I applied 11 mutations at once, ran make build, and ran the browser test:

• site.css:
  • body font set to 16px/1.6;
  • removed .share-button { margin-left: auto; }, .button:active, .skip-link:focus, the dark figure-paper rule, the reduced-transparency backdrop-filter: none, and .cell-source { position: relative; };
  • output switched to white-space: pre.
• runner.js: size: 'invisible' instead of execution: 'execute', and the metaKey guard dropped.
• syntax-highlight.js: the execCommand result forced to false.

It exited 1 and named all 11:

Turnstile widget is not rendered in Invisible execute mode (execution=undefined, size set=true)
Source copy did not fall back to execCommand when the Clipboard API is unavailable
Copy button is not anchored to its source cell
Arrow navigation fired with a modifier key or from a focused button
Dark mode does not keep marginalia figures on light paper (rgba(0, 0, 0, 0))
Share button does not sit apart at the end of the toolbar (gap 8px, 100.859375px from the end)
Skip link does not appear on screen when focused
Long output overflows the output panel instead of wrapping ({"textPastPanel":5450.6875,"pageOverflow":5470})
button[type="submit"] does not press down to scale(0.96) when active (none)
Body text ignores the reader's browser font size (16px at a 20px default)
prefers-reduced-transparency: header stays translucent (blur(16px), rgba(0, 0, 0, 0))

After restoring and rerunning make build, the test exited 0 and git status was clean.

The first drafts of two checks missed their mutations: the modifier-key check (pathname after 50 ms) and the wrapping check (scrollWidth). I rewrote them, using the Navigation API and a text range measured against the panel edge, until they failed on the mutation.

Waiver expiry.

• The new tests fail against origin/main's script, because expiry_warning and waiver_expiry_findings do not exist there.
• The old check_expiry_date('2026-12-01', today=2026-11-05) returns None, so it stays silent.
• On this branch, against the real registry:

Invocation	Exit code	Output
no flags (today)	0	no warning
--as-of 2026-11-05	0	WARNING: quality waiver hello-world: expires on 2026-12-01 (26 days left)…
--as-of 2026-11-05 --fail-within-days 30	1
--as-of 2026-12-01	1	expired
--fail-within-days 30 (today)	0

Deploy smoke.

• make -n deploy shows pywrangler deploy followed by post-deploy-smoke.
• make post-deploy-smoke DEPLOY_URL=http://127.0.0.1:9 exited 2 with Deployment smoke FAILED … The new version is already live.
• Against the local Worker it exited 0.
• No deploy was run.

Test counts. 249 on origin/main, 227 now: 19 source-text-only tests and 5 spec-wording tests removed, 2 expiry tests added. Each intermediate commit also passes on its own (230 and 232 tests).

Workflows. This PR no longer adds or changes any workflow. verify.yml is untouched.

Not in this PR

• Source-text tests of src/main.py, the git hooks, Makefile and workflows (test_turnstile_verification_is_session_gated_in_worker, test_worker_entrypoint_uses_fastapi_asgi_bridge, test_dynamic_worker_execution_uses_hash_keyed_get_cache, test_generated_drift_is_blocked_before_commit_and_merge). The recommendation targeted the CSS/JS tests, and 466f616 already added behavioural coverage for most of the main.py paths.
• Brand-colour literals and existence-only selector checks. Assertions on #FF4801, #F5F1EB, #521000, #EBD5C1, .runner-grid, --space-6 and box-shadow: were dropped rather than converted, because they pinned text, not behaviour. Legibility is covered by the contrast checks instead.
• Running make deploy. It touches production.
• A scheduled waiver-expiry run. Removed under the verification-cost standard; see "Cost and limits".

Notes

• Bypass secret for the deploy smoke. With Turnstile challenges enabled in production, the smoke's POST checks need PBE_SMOKE_BYPASS_SECRET exported, as the script's docstring already says. Without it, make deploy will now fail loudly after deploying. SMOKE_ARGS=--skip-post is available only as a deliberate choice.
• Browser test runtime. At authoring time it took about 40 s locally, up from about 21 s; see "Cost and limits" for the re-measurement. It uses the DOM/CSS domains, Emulation.setFocusEmulationEnabled, Page.setFontSizes and media-feature emulation, all of which current google-chrome on ubuntu-latest supports.
• Local-only Chrome setup, not committed. Chrome ran as root here, so it needed --no-sandbox. It also needed to trust this session's egress-proxy CA, which I did with --ignore-certificate-errors-spki-list for the proxy CA keys only. Both lived in a local CHROME_PATH wrapper; the committed script is unchanged in that respect.

🤖 Generated with Claude Code

https://claude.ai/code/session_012eisRQmQvg4TphwztxhQGJ
https://claude.ai/code/session_01UpksNcNFYPn6dR3UscZj4Z