[mauvilsa]

Adds a new parsing and composing implementation in docstring_parser/_core, enabled with legacy=False on parse, parse_from_object, combine_docstrings and the style-specific parsers. The default is unchanged. The rules and the differences from legacy are in BEHAVIOR.md.

Why a new parser

• Each style has its own rules. The four parsers were written separately, so defaults, optional markers, item keys, attributes, deprecations and examples are handled differently in each, or not at all.
• Text gets lost. Unindented text inside Args: and a Note: or See Also: after Args: disappear without warning. A summary wrapped over two lines is cut in half.
• Real docstrings raise errors. :param Dict[str, int] x: and a bare ValueError under Raises: raise ParseError.
• Compose doesn't round-trip. The legacy composers also drop some items.
• The data is inconsistent. args sometimes holds the type (["param", "x (int)"]), attributes are mixed into params, and a missing description can be "" or None.

On the 2,970 corpus docstrings with a declared style	Legacy	legacy=False
Raise ParseError	19	0
Change when parsed, composed and parsed again	682	0

Why not fix the legacy parsers

• Each fix changes output that users depend on. Recent fixes on main, mostly to compose, changed legacy results for 1,340 of the 3,180 corpus docstrings. Fixing everything above would change far more, in a minor release, with no way to opt out.
• Shared rules need shared code. Patching four parsers separately would keep them drifting apart.
• Some fixes need a different structure. Parsers built to discard unmatched text can't be patched into keeping all of it and round-tripping.
• An opt-in flag gives users a migration path. They can try legacy=False when ready, and it can become the default in the next major version.

What legacy=False guarantees

• One set of rules for all styles.
• No text is silently dropped: anything unrecognized ends up in the descriptions.
• Malformed docstrings don't raise; you get as much structure as can be recognized.
• parse(compose(d)) gives back d in every rendering style, for docstrings parsed in that style. The one known exception is in BEHAVIOR.md.
• The Docstring classes stay the same, with Docstring.attributes, DocstringParam.is_attribute and Docstring.legacy added.

Compatibility

• legacy=True stays the default, and its output is identical to main on the whole corpus.
• legacy=False requires Python 3.10+ and raises RuntimeError on older versions. The package still supports 3.8+, because _core is only imported when legacy=False is used.

Testing

• Golden tests pin both modes over 3,180 real docstrings from 20 projects (docstring_parser/tests/corpus/). scripts/build_corpus.py rebuilds the corpus and scripts/golden_diff.py reviews changes.
• Property-based tests (new hypothesis test dependency) check that only ParseError is raised and that composing round-trips.
• Field and section structure agrees with Sphinx napoleon on about 96% of Google and numpydoc corpus docstrings. Most of the rest are deliberate, such as not turning wrapped lines into parameters.

Reviewing

The diff is large (about 21,000 lines), but most of it is corpus data. The code to review is _core (about 2,000 lines), plus small legacy hooks in the existing modules.

🤖 Generated with Claude Code

[mauvilsa]

As mentioned in #119, the point of this is to illustrate ideas. Common rules, common code, corpus of real data to understand impact of changes, opt-in for new parsers, etc.

[rr-]

Sorry, I started to separately work on this, and landed corpus data along with a couple bug fixes. Will review the approach.

[rr-]

Thanks for this.

I would like to land this in some form, but not in its current shape. Before reviewing it line by line, I think we should agree on the larger design questions.

legacy as a boolean

This is the first thing I would change. A flag named after the old behavior leaves us maintaining two complete parsers until the next major version, with every function signature carrying legacy= along the way. Every bug fix then has to be considered twice, and legacy=False is a name we would eventually need to deprecate once the new behavior becomes the default.

I would rather have one parser. Where its output differs from today’s behavior, we should either describe that as a fix or make it an explicit breaking change in 1.0, which I'd release sooner than later. If a transition period is necessary, it should use a name that describes the behavior being selected, rather than one that describes what is being avoided.

Strict parsing

Never raising is useful, but it should not be the only mode. Silently accepting malformed docstrings hides exactly the problems people use a parser to detect.

I would like strict parsing to be the default, with strict=False providing a lenient mode that returns whatever could be parsed. In both modes, problems should be collected rather than reported one at a time. Strict mode should raise a single ParseError containing all problems, each with its line number. Lenient mode should return those problems alongside the parsed result.

Python 3.10+

I do not see a reason to require Python 3.10. Python 3.8 is past end of life, but the package supports it today, and nothing in _core requires 3.10 features. X | None and match have straightforward alternatives which do not cost us anything.

Custom sections

Looking back at what people have asked for, extensibility has mostly been about sections, not about overriding parser internals:

• Add support to custom sections on Google docstrings #5, and monkey-patching reported there: custom Google sections, including title variants such as ARGUMENTS:.
• Feature Request: User-specified Sections #105 (cyclopts): choosing which sections are parsed through the common parse(). Today the sections argument exists only on GoogleParser and NumpydocParser, and not for ReST or Epydoc. Also, sections without a dedicated attribute, such as References, should be kept with their title so the docstring can be reconstructed.
• feat: support Example section for google docstring #60, FEAT: Add support for numpydoc method section #62, Additional missing sections for Google-style docstrings #108: standard sections that were missing (Google Example, Numpydoc Methods, the rest of napoleon's list).
• "Attributes" in Numpy-style class docstrings are parsed as Parameters #21, Fix: docstring_parser/numpydoc.py maps both 'Parameters' and 'Attributes'... #114, Separate documented attributes from function parameters #118: attributes kept apart from parameters.
• [Google style] Parse "Yields" and "Returns" sections together #78: Yields and Returns in the same docstring.
• google+numpydoc: match section titles literally #134: titles with punctuation, such as C++ Notes or Args (old).

This PR covers part of that: custom section lists are carried over, attributes are separate, and the napoleon sections are recognized.

What I think the new approach should allow:

• One section API for all four styles, with titles and aliases, a key and a kind (text, items, returns, …), usable from parse() as well as from the per-style parsers.
• Adding, removing, renaming and aliasing sections per call, without patching module globals.
• Unrecognized sections kept with their title and position instead of being dropped or merged into the description.

Speed

Using the PR’s own corpus on Python 3.12, legacy=False currently compares with the legacy parser as follows:

Style	Legacy	New	Ratio
ReST	77 ms	94 ms	1.23x
Google	88 ms	124 ms	1.41x
Numpydoc	129 ms	121 ms	0.94x
Epydoc	71 ms	74 ms	1.04x
auto	362 ms	445 ms	1.23x

The target I would like to hold is no more than roughly 10% slower than the current implementation, measured in CI against the base commit. Numpydoc and Epydoc are already within that range, but Google is not yet.

Relation to #137

#137, now merged, adds a snapshot corpus of 1,100 docstrings, together with line-ending and round-trip checks for every case. It also fixes the legacy bugs those checks found. I went with one diff-able .txt file per case rather than JSON, which was Claude's first choice, so that a change in parser output reads as a normal diff in review.

Since it touches the same modules, this PR needs a rebase onto it, and the legacy golden files need to be regenerated. Two corpora are unnecessary, so we should either choose one or merge them rather than keep both. (Sorry!)

The review also found three smaller bugs, all in legacy=False:

• _core/fieldlist.py:262: A custom returns-kind section with its own key, such as Outputs, raises KeyError when composed as ReST or Epydoc. It should fall back to the returns / yields names, as parameter sections do.
• attrdoc.py:116: parse_from_object drops a class attribute x and its docstring when the class docstring documents a parameter named x. Deduplication should only check against attributes.
• _core/numpydoc.py:166: ReST :param: and :keyword: fields composed as Numpydoc produce two separate Parameters sections, causing the keyword kind to be lost when read back.

---

My proposed next step is to agree on the overall shape on each of the above sections first. Once that is settled, the core of this PR can be reworked toward it, but ideally in smaller pieces rather than as one 20,000-line change. Happy to work on it myself too.

[mauvilsa]

legacy as a boolean

This is the first thing I would change. A flag named after the old behavior leaves us maintaining two complete parsers until the next major version, with every function signature carrying legacy= along the way. Every bug fix then has to be considered twice, and legacy=False is a name we would eventually need to deprecate once the new behavior becomes the default.

I would rather have one parser. Where its output differs from today’s behavior, we should either describe that as a fix or make it an explicit breaking change in 1.0, which I'd release sooner than later. If a transition period is necessary, it should use a name that describes the behavior being selected, rather than one that describes what is being avoided.

A legacy boolean parameter is not important. It is a new parser because of the "Why a new parser" reasons above. Since it would be new code, then surely it would have its own bugs. And certainly the idea would be that the newer more consistent parser replaces the old one. What would be important is that people can try it out before it becomes the default.

Strict parsing

Never raising is useful, but it should not be the only mode. Silently accepting malformed docstrings hides exactly the problems people use a parser to detect.

I would like strict parsing to be the default, with strict=False providing a lenient mode that returns whatever could be parsed. In both modes, problems should be collected rather than reported one at a time. Strict mode should raise a single ParseError containing all problems, each with its line number. Lenient mode should return those problems alongside the parsed result.

Sounds good.

Python 3.10+

I do not see a reason to require Python 3.10. Python 3.8 is past end of life, but the package supports it today, and nothing in _core requires 3.10 features. X | None and match have straightforward alternatives which do not cost us anything.

Fine by me want to continue supporting 3.8 and 3.9. It should be very simple to make the new code run on those versions.

Custom sections

Looking back at what people have asked for, extensibility has mostly been about sections, not about overriding parser internals:

* [Add support to custom sections on Google docstrings #5](https://github.com/rr-/docstring_parser/pull/5), and monkey-patching reported there: custom Google sections, including title variants such as `ARGUMENTS:`.

* [Feature Request: User-specified Sections #105](https://github.com/rr-/docstring_parser/issues/105) (cyclopts): choosing which sections are parsed through the common `parse()`. Today the `sections` argument exists only on `GoogleParser` and `NumpydocParser`, and not for ReST or Epydoc. Also, sections without a dedicated attribute, such as `References`, should be kept with their title so the docstring can be reconstructed.

* [feat: support Example section for google docstring #60](https://github.com/rr-/docstring_parser/pull/60), [FEAT: Add support for numpydoc method section #62](https://github.com/rr-/docstring_parser/issues/62), [Additional missing sections for Google-style docstrings #108](https://github.com/rr-/docstring_parser/pull/108): standard sections that were missing (Google `Example`, Numpydoc `Methods`, the rest of napoleon's list).

* ["Attributes" in Numpy-style class docstrings are parsed as Parameters #21](https://github.com/rr-/docstring_parser/issues/21), [Fix: docstring_parser/numpydoc.py maps both 'Parameters' and 'Attributes'... #114](https://github.com/rr-/docstring_parser/pull/114), [Separate documented attributes from function parameters #118](https://github.com/rr-/docstring_parser/pull/118): attributes kept apart from parameters.

* [[Google style] Parse "Yields" and "Returns" sections together #78](https://github.com/rr-/docstring_parser/issues/78): `Yields` and `Returns` in the same docstring.

* [google+numpydoc: match section titles literally #134](https://github.com/rr-/docstring_parser/pull/134): titles with punctuation, such as `C++ Notes` or `Args (old)`.

This PR covers part of that: custom section lists are carried over, attributes are separate, and the napoleon sections are recognized.

What I think the new approach should allow:

* One section API for all four styles, with titles and aliases, a key and a kind (text, items, returns, …), usable from `parse()` as well as from the per-style parsers.

* Adding, removing, renaming and aliasing sections per call, without patching module globals.

* Unrecognized sections kept with their title and position instead of being dropped or merged into the description.

No comment.

Speed

Using the PR’s own corpus on Python 3.12, legacy=False currently compares with the legacy parser as follows:
Style Legacy New Ratio
ReST 77 ms 94 ms 1.23x
Google 88 ms 124 ms 1.41x
Numpydoc 129 ms 121 ms 0.94x
Epydoc 71 ms 74 ms 1.04x
auto 362 ms 445 ms 1.23x

The target I would like to hold is no more than roughly 10% slower than the current implementation, measured in CI against the base commit. Numpydoc and Epydoc are already within that range, but Google is not yet.

Good having that target. The speed is already good, given that the new parser has the added guarantees, which naturally requires more processing.

Relation to #137

#137, now merged, adds a snapshot corpus of 1,100 docstrings, together with line-ending and round-trip checks for every case. It also fixes the legacy bugs those checks found. I went with one diff-able .txt file per case rather than JSON, which was Claude's first choice, so that a change in parser output reads as a normal diff in review.

Since it touches the same modules, this PR needs a rebase onto it, and the legacy golden files need to be regenerated. Two corpora are unnecessary, so we should either choose one or merge them rather than keep both. (Sorry!)

Certainly, no need to have two corpora. What is important is having a corpus, to notice on real data what happens as the code changes.

The review also found three smaller bugs, all in legacy=False:

* `_core/fieldlist.py:262`: A custom returns-kind section with its own key, such as `Outputs`, raises `KeyError` when composed as ReST or Epydoc. It should fall back to the `returns` / `yields` names, as parameter sections do.

* `attrdoc.py:116`: `parse_from_object` drops a class attribute `x` and its docstring when the class docstring documents a parameter named `x`. Deduplication should only check against `attributes`.

* `_core/numpydoc.py:166`: ReST `:param:` and `:keyword:` fields composed as Numpydoc produce two separate `Parameters` sections, causing the keyword kind to be lost when read back.

My proposed next step is to agree on the overall shape on each of the above sections first. Once that is settled, the core of this PR can be reworked toward it, but ideally in smaller pieces rather than as one 20,000-line change. Happy to work on it myself too.

No problem from my side. This pull request was just to illustrate ideas. You can do with this as you see fit. If it is already helpful for you, then great.

I think it would be good if you look in detail at BEHAVIOR.md (excluding the legacy boolean part). Better to clearly document what is the expected behavior of the parser: guarantees, common rules, etc. That way it can be better understood whether some behavior in the code is a bug or intended.

[rr-]

Thanks, this shaped a lot of what landed since. Instead of a second parser behind legacy=False, I reworked the existing ones in place, in small PRs (#139 to #195). Each one is benchmarked against its base and compared on about 29,000 docstrings.

The goals of #136 on main now:

• One lexer and one model (kind, key, name) for all styles (docstring_parser: read every style through Lines #140, common: let every meta say what it documents #141). args is gone (common: drop args from the meta model #143).
• Dropped text goes to the long description, and a test checks this (docstring_parser: keep text that parsers dropped #146).
• parse() raises only ParseError, carrying every problem with its line. try_parse() never raises: it returns the docstring plus problems with stable codes (Add try_parse, which never raises #152, Give each parse problem a code #160).
• SectionSpec works for any style, per call, via parse(sections=...) (sections: declare sections alike in every style #144).
• Attributes and yields are separate from params and returns (List attributes apart from params #154, Keep yields apart from returns #156).

Differences from #136: no legacy flag, since fixes go in with changelog lines. Still Python 3.8+ with no dependencies. Never-raise is opt-in through try_parse().

Since the first draft of this reply, parse_from_object keeps an attribute that shares a name with a param (#162), and undeclared sections keep their title (#146, #166).

Here is where each item of BEHAVIOR.md stands. Rows 1 to 24 follow its "Differences from legacy, at a glance" table, with the ARGS: / Raises: row split in two. Rows 25 to 28 are rules from the other sections that the table doesn't list, and rows 29 to 42 are what a later check of every rule and example in BEHAVIOR.md against main turned up.

#	Item	Status	PR
1	Summary wrapped over two lines: whole first paragraph	fixed	#174
2	Google: unindented text inside Args: is kept	fixed	#146
3	Google: Note:, See Also:, Keyword Args: and the other napoleon sections	fixed	#180
4	Google: case-insensitive titles such as ARGS:	fixed	#177
5	Google: Raises: with a bare ValueError	fixed	#165
6	Google: default from x (int): Defaults to 3.	fixed	#168
7	Google: the name of x (int): is x	fixed	#143
8	ReST: a line starting with :meth:`run` is text	fixed	#170
9	ReST: :param Dict[str, int] x:	fixed	#164
10	ReST: :ivar a: with :vartype a: is one typed attribute	fixed	#172
11	Numpydoc: underline shorter than the title, or made of =	fixed	#178
12	Numpydoc: See also after Returns gives no fake returns	fixed, as a see_also text item with its title	#166, #177
13	Numpydoc: int, optional, default 3 gives type int	fixed	#168
14	Numpydoc: By default the value... gives no default	fixed	#168
15	Numpydoc: x, y : int gives two params	fixed	#182
16	Numpydoc: Attributes only in attributes	fixed	#154
17	Examples: text before the snippet, output in the snippet	fixed	#167
18	Doctest ... continuation lines stay in the snippet	fixed	#167
19	Google: Example: with unindented content is kept as text	fixed for the content; the empty Example: itself stays an item on purpose, so it composes back as written	#146
20	Methods sections, one item per method	fixed	#180, #193
21	Epydoc: @raises and @returns	fixed	#163
22	.. deprecated:: in Google and ReST	fixed	#169
23	is_optional without a type is False	fixed	#168
24	Empty description is None	fixed	#175
25	Google: several names per header, x, y (int):, with ... as a name	fixed	#182
26	Escaped names such as \*\*kwargs and class\_, in every style, ReST types and .. attribute:: included	fixed	#183, #188
27	Google: a bare name is an item, and a wrapped line or a colon inside parens continues the item	fixed	#184
28	Google: list of str: Names. as a return type	fixed	#185
29	ReST compose writes class_ as class\_	fixed	#188
30	ReST: :key: is a param and :kwarg: a keyword	fixed	#189
31	Epydoc: blank lines inside a field are kept	fixed	#189
32	ReST: an empty :rtype: is ignored	fixed	#189
33	Whitespace in types is normalised, list of str gives list of str	fixed	#190
34	Google: Returns: with name (type): text	fixed	#191
35	parse_from_object cleans attribute docstrings like other docstrings	fixed	#192
36	Numpydoc: in Raises, Type : text is split only when what precedes the colon looks like a type	fixed: prose such as If it is bad: an error. is kept whole, as napoleon does	#171
37	Numpydoc: an underline of one or two dashes	choice: an underline needs at least 3 - or = by default, configurable with min_underline and exact_underline; a shorter one gives a bad-underline problem and the section is still read	#187
38	Google: a type wrapped onto the next line is not a header	choice: it stays a header, since multi-line param definitions (#98) shipped in 0.17.0 and this would break them	#98
39	Compose writes int? as , optional	choice: types compose as written, so int? stays int?	#158
40	Custom Numpydoc Section subclasses use their own parse	choice: customising through internals is gone, and SectionSpec(parse_item=...) replaces it	#145
41	Malformed docstrings don't raise	choice: parse() raises ParseError, and try_parse() is the call that never raises	#152, #160
42	API: legacy flag, args, Docstring.attributes and DocstringParam.is_attribute	choice: no flag, kind plus key replace args, and attributes live in Docstring.attrs	#141, #143, #154

Still open besides these: a behaviour doc like the BEHAVIOR.md in #136, and a review of it would be very welcome.

I'd close this PR, if that's OK.

[mauvilsa]

Sounds great, thanks! I am on vacation for a couple of weeks, so I can't really look at the details now. Fine by me if you close it. No need for it to be open for me to comment later.