Repository navigation
tests: snapshot real docstrings and fix what they catch - #137
Merged
Merged
Conversation
Before reworking the parsers, pin down what they do today on docstrings people actually write, so every behaviour change shows up as a diff. tools/harvest_corpus.py samples 100 docstrings each from numpy, scipy, pandas, jsonargparse, rich, pydantic, google-auth, requests, flask, sphinx and twisted, reading them from pinned wheels with ast. Each becomes a text case file holding the input and the expected output of parsing it in the package's style, auto-detecting it, and composing it back. test_cases.py runs every case file; UPDATE_CASES=1 rewrites the expected output.
Tests that only turn docstring text into a parsed or composed result now live as text case files under tests/cases/, generated from the calls each test made: every input with the commands it went through, and the output the current code gives. The full dump also pins every attribute the old assertions left unchecked. Case files are now read and written as bytes so that \r in an input survives, the compose command takes an optional rendering style, and a new convert command composes a docstring into another style. Tests whose input or check is not text stay in Python: custom parsers and sections, regexes, monkeypatching, timing, hand-built docstrings, parse_from_object, combine_docstrings, and \r\n parsing the same as \n. A None input and Docstring.description, which the dump does not show, get their own tests. Line and branch coverage of the library is unchanged.
Two properties now run over every case file, corpus included, instead of a handful of hand-picked inputs: - Swapping the input's newlines for \r\n or \r gives every command in the case the same output. This replaces test_line_endings, whose inputs become case files. - A composed docstring parses and composes back to the same text. The second one finds nine cases where it does not hold yet. They are listed in UNSTABLE_COMPOSE with the reason and marked as strict xfails, so fixing one forces it off the list.
compose() put a param's type inside the field name, as in ":param int a:". A type such as "float or tuple" or ":class:`Request`" then read back as too many arguments, so the composed docstring raised ParseError. Such a type now gets a :type: or :rtype: field of its own, as the expanded rendering already does. This fixes the three requests docstrings in the corpus that did not survive another parse and compose.
When a param's type gives no default, the parser looks for one in the description. Besides "default is", "default:", "default=" and "defaults to", it took the word after any bare "default", so prose such as "the default engine", "by default only" or "the default value is" gave defaults of "engine", "only" and "value". Composing such a param then wrote a default the docstring never had. Drop the bare form. In the corpus it only ever matched prose.
The type of a param was matched for a trailing ", optional" first and a default second, so in "int, optional, default 1" the "optional" stayed in the type. compose() then added another one, giving "int, optional, optional". Strip "optional" on either side of the default.
inspect.cleandoc() removes the indent common to all lines, so a line indented less than the summary, such as scipy's " .. legacy::" under a four-space docstring, left the short description with a leading space. Every style kept it, while the long description was already stripped. Composing the docstring then dropped the space on the next round. Strip the short description in every style.
A description's continuation lines went through inspect.cleandoc(), which strips the first of them entirely and dedents only the others. A literal block after "Expected values include::" or a block nested under "!!! warning" lost its indent relative to the line above it, and a blank line right after the item's first line was dropped. Dedent all continuation lines alike instead. When every line after the first is indented further than the first, compose() now starts the description below the item's head, as the expanded rendering does. Otherwise reading it back could not tell that extra indent from the continuation indent.
A declaration such as "value (List[\n tuple[int,\n str]])" kept its whole text as the param name with no type, since only the first line was searched for the closing parenthesis. Joining the declaration's second line onto the first already made a type over two lines work, so three lines behaved differently. compose() then wrote the newline into the name, which failed to parse. When the first line has no closing parenthesis, join the remaining lines and look there.
parse() collected the sections in a dict keyed by title, so a second section with the same title replaced the first, which was dropped without a trace. In the corpus, rich's Control.move_to_column has two Returns sections and lost the first. Collect them in a list instead. compose() writes every example under an "Examples:" title, so an "Example:" section followed by an "Examples:" one lost the first on the next round. This was the last case that did not compose back the same, so the list of expected failures goes away.
update-cases ran every test in test_cases.py, so a failing line ending or compose check made the recipe fail even after it had rewritten the cases. Run only the test that rewrites them.
A field ends at the next field or at a blank line, so an unindented line right after ":rtype: int" became part of the type, giving a type of "int\nMore text.". Composed inline, that type made a field name that did not parse back. A type is a single value, so :type: and :rtype: now end at the next unindented line, which goes to the long description. Other fields keep taking such lines as part of their description, since flask writes continuation lines that way.
A field's continuation lines went through inspect.cleandoc(), which drops leading blank lines and strips the first line entirely, so ":param a: One.\n\n Two." lost the blank line between its paragraphs, and a literal block lost its indent. Dedent them all alike instead, as the Google parser now does. compose() starts a description whose continuation lines are all indented further than the first on a line of its own, so that indent survives another parse. The check moves to common.py for both styles.
The head of an item, before its colon, was joined across lines with no separator, so "x more" and "The value: here" gave the name "x moreThe value", and a type wrapped after a comma lost its space. Join the lines with spaces instead, except right inside brackets, so "List[" and "int]" still give "List[int]". With the head on one line, the type now ends at the parenthesis that matches the opening one rather than at the last one on the first line. So "value (int) trailing extra (more)" still has the type "int", and a stray ")" on a later line no longer ends up in the type.
compose() wrote every param as :param or @PARAM, so a docstring's attributes and keyword params read back as plain params. Write the field name from the param's args[0] instead, wherever the style has one, mapping attributes between ReST's :attribute: and Epydoc's @ivar when composing into the other style.
ParseError messages quoted the whole chunk they failed on, so a 100 KB field gave a 100 KB message. Quote at most its first 60 characters.
The reused default parser was rebuilt only when DEFAULT_SECTIONS held different section objects. Numpydoc sections are mutable and compare by identity, so changing a section's title in place left the parser looking the section up under its old title. Compare the titles too. Replacing title_pattern on a Section class still goes unnoticed until DEFAULT_SECTIONS changes: checking every pattern on each call made parsing about 18% slower, while the titles cost about 5%.
From Python 3.13, inspect.cleandoc() measures indentation by stripping spaces only, so a blank line holding a tab or a form feed counts as text with no indent, and nothing gets dedented: "Sum.\n\f\n :param a: x" had one field on 3.12 and none on 3.14. Dedent with our own copy of the cleandoc() from before 3.13, in every style. On 3.8 it gives the same result as inspect.cleandoc() for every case input, and parsing is as fast as before.
This was referenced Nov 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implemented the snapshot case-file test suite and corpus workflow.
tests/cases/.test_cases.pycoverage for:parse <style>detectcompose <style> [<rendering>]convert <from> <to>ast.just update-cases..gitattributeshandling.\n,\r\n, and\r.:type:and:rtype:fields absorbing the next unindented line.typewhen@typecomes first.compose()writing every param as:param/@param.ParseErrormessages quoting entire chunks.optionalhandling.