Repository navigation
tools: benchmark and compare output against a base - #139
Merged
Merged
Conversation
Add a benchmark that loads the package as it is at a base commit next to the working tree, in one process, and times parse in every style and in auto mode, and compose in every style and rendering, over the case files and the standard library's docstrings. Base and working tree run back to back on small chunks of the inputs, alternating which goes first, and the gate is the median of the per-round ratios. On a loaded machine this keeps identical code within a few percent, where timing each side in its own process swung by more than ten. The run fails when any workload is more than 10% slower. `just bench` runs it against origin/main, and a CI job runs it on Python 3.8 and 3.14 against the base of every pull request.
Add a check that loads the package as it is at a base commit next to the working tree, as the benchmark does, and lists the docstrings whose output differs: the parse result in every style and in auto mode, and every parse result composed in every style and rendering. The inputs are the case files, the standard library's docstrings and 20,000 docstrings generated from lines of every style. A refactoring must leave all of them alone; `just parity` runs it against origin/main.
List how many docstrings differ for each parse and compose command, so that what a change touches shows before reading examples of it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two checks for the rework that follows, which have to show each step costs no speed and, for a refactoring, changes no output.
just benchloads the package at a base commit next to the working tree in one process, and times parse in every style and in auto mode, and compose in every style and rendering, over the case files and the standard library's docstrings. Base and tree run back to back on small chunks, alternating which goes first, and the gate is the median per-round ratio, so load from other processes hits both alike: identical code stays within a few percent even on a busy machine. It fails above 1.10. A CI job runs it on 3.8 and 3.14 against each pull request's base.just parityparses the same inputs plus 20,000 generated docstrings in every style, composes every result in every style and rendering, and lists, then counts per command, the docstrings whose output differs from the base.