Repository navigation
pygettext: use an AST parser instead of a tokenizer #104400
Description
Follow up on this forum discussion
This is a part 1/X of improving pygettext. Replacing the tokenizer that powers the message extraction with a parser will simplify the code (no more counting brackets and f-string madness) and make it much easier to extend it with new features later down the road.
This change should also come with a healthy dose of new tests to verify the implementation.
PR coming shortly ;)
Linked PRs
- gh-104400: pygettext: use an AST parser instead of a tokenizer #104402
- gh-104400: Add more tests to pygettext #108173
- [3.13] gh-104400: Add more tests to pygettext (GH-108173) #126361
- [3.12] gh-104400: Add more tests to pygettext (GH-108173) #126362
- gh-104400: Remove
fintl.gettextfrom pygettext #129580 - gh-104400: pygettext: Prepare to replace TokenEater with a NodeVisitor #129672
Activity
terryjreedy commentedon May 11, 2023
We replaced the custom parser in pyclbr with an ast visitor a couple of years ago. It was shorter and clearer and agreed to be a definite improvement. For this type of application, any possible slowdown is irrelevant.
gh-104400: Add more tests to pygettext (GH-108173)
pythongh-104400: Add more tests to pygettext (pythonGH-108173)
pythongh-104400: Add more tests to pygettext (pythonGH-108173)
[3.12] gh-104400: Add more tests to pygettext (GH-108173) (GH-126362)
[3.13] gh-104400: Add more tests to pygettext (GH-108173) (GH-126361)
pythongh-104400: Add more tests to pygettext (pythonGH-108173)
pythongh-104400: Add more tests to pygettext (pythonGH-108173)
gh-104400: Remove ``fintl.gettext`` from pygettext (#129580)
gh-104400: pygettext: Prepare to replace TokenEater with a NodeVisitor (
pythongh-104400: Remove ``fintl.gettext`` from pygettext (python#129580)
pythongh-104400: pygettext: Prepare to replace TokenEater with a Node…
1 remaining item
gh-104400: pygettext: use an AST parser instead of a tokenizer (GH-10…
serhiy-storchaka commentedon Feb 11, 2025
Thanks, @tomasr8. The AST based implementation is much clearer.
The drawback is that it prevents implementing the
--add-commentsoption. I think that the right solution of this is to add optional support of comments in AST -- it could also be useful in other applications. But this is a different issue.Is it all, or you planned other work on this issue?
AA-Turner commentedon Feb 11, 2025
Having comments in the AST would also be useful for applications like Sphinx -- is there an open issue to track this, or should we create one
A
tomasr8 commentedon Feb 11, 2025
Yes, having the comments in the AST would be great - I can look into that and see if there are any issues open about it!
There is actually a way to add translator comments even with the AST. You can run the tokenizer separately and collect all comments and their corresponding line numbers. You can then match the comments to gettext calls. I had a working proof of concept implementation of this for babel but I can repurpose it for pygettext!
Not specifically in this issue, I think we can close it. I have lots of other things I'd like to improve but I think it's better to open separate issues for that (translator comments being one of those)
tomasr8 commentedon Feb 11, 2025
There's a similar issue about type comments: #101494
sergey-miryanov commentedon Feb 12, 2025
We didn't fill an issue, 'cause we stick to too old Python version, but in our applications, we need to support comments in AST and now have to do tricky re.subs to preserve them.
serhiy-storchaka commentedon Feb 12, 2025
This is even more general and flexible way, but it may be hard. With comments in AST we have an issue of representing comments in a way that will not break existing code that uses AST if the tree contains comments. There is a problem of representing comments between string literals which will be concatenated into a single AST node -- it can be ignored for
pygettext, but not for other applications. There is a problem of distinguishing comments that precede a node from a comment that follows it on the same line from comments that follow it on the following lines. Different applications may need to distinguish different types of comments and may have different preferences of the representation.But there is no urge to add the
--add-commentsoption right now.