Repository navigation
Inconsistent ast.parse result for python<3.9 and PEP614 #96427
Description
PEP614 introduces relaxed decorator grammar in 3.9: https://peps.python.org/pep-0614/
Right now, ast.parse with feature_version=(3, 8) and bellow behave differently from specified python versions themself.
For example, main branch is able to parse this code:
Python 3.12.0a0 (heads/main-dirty:e860e521ec, Aug 28 2022, 21:36:18) [Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> code = """
... @decs[0].func
... def some(): pass
... """
>>> ast.parse(code)
<ast.Module object at 0x102db8150>
>>> ast.parse(code, feature_version=(3, 8))
<ast.Module object at 0x102d9d320>While python==3.8.x is not able to:
Python 3.8.9 (default, May 3 2021, 12:15:25)
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> code = """
... @decs[0].func
... def some(): pass
... """
>>> ast.parse(code)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/sobolev/.pyenv/versions/3.8.9/lib/python3.8/ast.py", line 47, in parse
return compile(source, filename, mode, flags,
File "<unknown>", line 2
@decs[0].func
^
SyntaxError: invalid syntaxWhen looking at we-love-parsers/pegen - it does have a version check for this: https://github.com/we-like-parsers/pegen/blame/599255a5ba3930e86367cd3fc78fae91ee184dd5/data/python.gram#L660-L662
Pros and Cons
I think that there are several key points to this, pros:
- Correct parsing of older versions
- Consistency with other parser like
we-love-parsers/pegen
However, there are some serious cons to it:
- Grammar is made quite significanly more complex
- There might be new bugs in new behaviour: locations / performance / etc
I think it is up to @pablogsal and other core devs to decide.
Proposed solution
I've modified an existing we-love-parsers/pegen's solution a bit, here's what it looks like:
decorators[asdl_expr_seq*]: a[asdl_expr_seq*]=decorator+
decorator[expr_ty]:
| a=('@' f=dec_maybe_call NEWLINE { f }) { a }
| a=('@' f=named_expression NEWLINE { f }) {
CHECK_VERSION(expr_ty, 9, "Relaxed decorator syntax is", a) }
dec_maybe_call[expr_ty]:
| a=dec_primary '(' b=[arguments] ')' {
_PyAST_Call(a,
(b) ? ((expr_ty) b)->v.Call.args : NULL,
(b) ? ((expr_ty) b)->v.Call.keywords : NULL,
EXTRA) }
| dec_primary
dec_primary[expr_ty]:
| a=dec_primary '.' b=NAME { _PyAST_Attribute(a, b->v.Name.id, Load, EXTRA) }
| a=NAME
I will send a PR with all the changes / tests / auto-generated files, so we can take a closer look and decide whether we want this or not.
Activity
pythongh-96427: Raise `SyntaxError` for `PEP614` on older python vers…
pablogsal commentedon Aug 30, 2022
I fear that this is too much complexity unfortunately. Unless @isidentical or @lysnikolaou have string opinions about this, I prefer to mark is as won't fix.
This whole idea of rejecting new constructs on compatibility mode is quite a lot of burden for a use case that's not clear (at least to me) and was not really discussed a lot. I don't think this is even fully documented...
In any case, thanks for the issue and the work @sobolevn and sorry that we need to reject it :(
sobolevn commentedon Aug 30, 2022
No worries! I felt almost the same.
At least I had a chance to work with all the parser tooling :)
Let's wait for others to comment.
lysnikolaou commentedon Aug 30, 2022
No strong opinion on my side. Plus I agree with @pablogsal. Should we maybe do a bit more research as to how
feature_versionis used?sobolevn commentedon Aug 30, 2022
I would be glad to. First, I will start with searching for real-life use-cases on github.
sobolevn commentedon Aug 30, 2022
Mypy
My main use-case is
mypyparsing of python files.You can specify target version of python you want to check. For example, you can check
python3.8while actually runningpython3.10It is possible via
mypy yourfile.py --python-version=3.8Here's the source code we use in
mypy: https://github.com/python/mypy/blob/master/mypy/fastparse.py#L275-L303So, the initial problem was:
--python-version=3.8(target version for my project), ideally it should find all problems with my source code. However, it passes and produces a false-negativepython yourfile.pyfails with syntax errorblack
It is also used by
blackfor some similar stuff: https://github.com/psf/black/blob/44d5da00b520a05cd56e58b3998660f64ea59ebd/src/black/parsing.py#L149-L169So, I think we can conlcude that this is a very specific feature for static analysis tools.
Other projects
It is also used by 100+ other projects (including some forks). https://cs.github.com/?q=feature_version%3D%20language%3APython&scopeName=All%20repos&scope=#