Repository navigation
Refactor the WPT test runner to be a python test runner configuration #51854
Description
While working on the flakes I noticed that there are still some drawbacks with the current WPT test runner (common/wpt.js):
- To enable running tests in parallel the WPT test runner uses workers, which can be harder to debug especially when the use of inspector is needed to look into the JS land. If we just use the python runner, we get the parallelism for free, and debugging individual tests would be easier. We can use a fixture to set up the main context/harness and let the python runner preload that file before running the tests. To debug the test, just
out/Release/node --inspect-brk -r test/common/wpt-fixture.js test/fixtures/wpt/...and connect to the inspector server with a client. - The use of workers for parallelism also mean that we lack coverage of WPT running on the main thread (I've been bitten by this before when adding lazy loading of Web APIs the main context snapshot). If we use the python test runner to run the WPT directly, we can get coverage on both the main thread and the worker thread (via the custom test suite in the CI that uses the
--workerswitch) - The status file for the current WPT runner doesn't allow conditions which is important for marking flakes on a specific platform. To add this to the JS-based WPT runner we need to reinvent the wheel again, and as far as I can tell we might as well just integrate the test runner into the existing python one and get a unified handling for flakes.
Activity
joyeecheung commentedon Feb 23, 2024
cc @panva @legendecas based on git log of
test/common/wpt.jspanva commentedon Feb 23, 2024
I'm just worried even less people will be able to make heads or tails of what it is doing. I for one would not be able to contribute anymore.
joyeecheung commentedon Feb 23, 2024
I personally find the Python runner easier to extend than the current WPT runner (and I wrote the initial implementation of the JS WPT runner - in retrospect I would've just do the harness in a preloaded script, instead of inventing a different runner, had I know that the WPT can start flaking). It's actually surprisingly simple, ignoring language differences.
joyeecheung commentedon Feb 23, 2024
Also, the harness part would still be in JS as a preloaded fixture, only the runner part would be in Python (I think most of the runner code in common/wpt.js is already reinventing what the python runner is capable of anyway...).
legendecas commentedon Feb 24, 2024
I agree that the current WPT runner is trying to reinvent the capability of python runner but there are unique features like generating wpt.fyi reports. But I believe it would be straightforward to re-implement it in python runner as well.
joyeecheung commentedon Jun 5, 2024
Happened to be talking about this during the WinterCG breakout session at Web engine hackfest and it seems there are interest from Deno to work on a common test runner for JS runtimes and upstream it to WPT. If that common test runner could handle the points mentioned here that would be great and we would not need to maintain (alone) yet another test runner / adaptor for WPT (it was proposed mostly because of point 3, trying not to re-invent the wheel of conditional test running based on a status file, and the python test runner looks like a good target to piggy-back. But if there's something else that includes the wheel and another party helps maintain it, that sounds like a better piggy-backing target)
github-actions commentedon May 23, 2026
This issue has been marked as stale due to 210 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.
github-actions commentedon Aug 23, 2026
This issue has been marked as stale due to 90 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.
test: support inspecting WPTs in child processes
test: support inspecting WPTs in child processes
test: support inspecting WPTs in child processes
test: support inspecting WPTs in child processes
panva commentedon Sep 11, 2026
A few updates since this was opened:
.cjsWPT status files have been supported since test,crypto: update WebCryptoAPI WPT #54593. These allow platform- and build-dependent conditions while retaining named subtest expectations.WPT_INSPECT. It runs a selected non-worker WPT variant in a child process with--inspect-brk.-jthen controls WPT concurrency, and individual paths get their own timings, failures, and reruns.#65984 is marked to close this issue when it lands.
test: schedule WPT variants individually
test: support inspecting WPTs in child processes
test: schedule WPT variants individually