Repository navigation

Add work-in-progress implementation of a new Python parser - #20856

Merged

JukkaL merged 194 commits into

masterfrom

new-parser

Feb 25, 2026

Merged

Add work-in-progress implementation of a new Python parser#20856
JukkaL merged 194 commits into
masterfrom
new-parser

Commits on Jan 1, 2026
- [WIP] Add initial nativeparse test case
  JukkaL
  committed
  0c4185f
- [WIP] First steps towards parsing something
  JukkaL
  committed
  8c0f906
- Support multiple defs
  JukkaL
  committed
  d5291be
- Read line/column information
  JukkaL
  committed
  4c72241
- Remove debug print
  JukkaL
  committed
  3798d06
- Fix self check, update docstring
  JukkaL
  committed
  32d855e
- WIP add parse/deserialize benchmark
  JukkaL
  committed
  cf9aefd
- Update for new cache format
  JukkaL
  committed
  9da3988
- Deserialize member expr
  JukkaL
  committed
  c374e0a
- Add data-driven tests
  JukkaL
  committed
  46fe324
- Fix empty line in test output
  JukkaL
  committed
  e8ce8f8
- Deserialize tuple expressions
  JukkaL
  committed
  bab774b
- Deserialize binary operations
  JukkaL
  committed
  ffd4b8d
- Deserialize int expressions
  JukkaL
  committed
  310a445
- Deserialize assignment
  JukkaL
  committed
  2f92446
- Deserialize if statement
  JukkaL
  committed
  845ae1f
- Show informatino about panics in tests
  JukkaL
  committed
  52ed3ac
- Deserialize additional node types
  JukkaL
  committed
  0681829
- Deserialize comparison and bool ops
  JukkaL
  committed
  8046eaf
- Add deserialization test for None, True and False
  JukkaL
  committed
  ddba816
- Deserialize func defs (partial) and return statements
  JukkaL
  committed
  ce7e013
- Deserialize 'pass' and test func defs more
  JukkaL
  committed
  a92202c
- Deserialize parameter defaults
  JukkaL
  committed
  4db2423
- Deserialize keyword args in calls
  JukkaL
  committed
  2430f83
- Test *args and **kwargs in calls
  JukkaL
  committed
  5cf0bef
- Minimal deserialization of class defs
  JukkaL
  committed
  7ff2f48
- Support base classes
  JukkaL
  committed
  7d50e25
- Deserialize floats
  JukkaL
  committed
  321c215
- Deserialize unary expressions
  JukkaL
  committed
  4e27e8f
- Deserialize dict expressions
  JukkaL
  committed
  e5859df
- Deserialize complex literals
  JukkaL
  committed
  b7b6943
- Deserialize slice expressions
  JukkaL
  committed
  df5d22a
- Partial support for deserializing function return types
  JukkaL
  committed
  19e7be1
- Deserialize parameter types
  JukkaL
  committed
  48635fa
- Add test for qualified type names
  JukkaL
  committed
  45f1862
- Test deserialization of generic types
  JukkaL
  committed
  4ff1809
- Deserialize union types
  JukkaL
  committed
  69e5b9f
- Deserialize assignment with type annotation
  JukkaL
  committed
  22f64e0
- Deserialize import statements
  JukkaL
  committed
  2744a20
- Support raise
  JukkaL
  committed
  ff7e14b
- Support break and continue
  JukkaL
  committed
  dcedc1f
- Support generator expressions
  JukkaL
  committed
  48d0379
- Support yield and yield from
  JukkaL
  committed
  56ecbda
- Support list and set comprehensions
  JukkaL
  committed
  dc14c80
- Support dict comprehensions
  JukkaL
  committed
  60a5aeb
- Use the C extension in tests instead of subprocess
  JukkaL
  committed
  e35f8f5
- Support from ... import
  JukkaL
  committed
  fc1fc68
- Support assert statements
  JukkaL
  committed
  db7d848
- Support 'for' statements
  JukkaL
  committed
  1e0d162
- Support 'with' statements
  JukkaL
  committed
  badeb1a
- Support function decorators (non-overloads)
  JukkaL
  committed
  9f7481f
- Fix self check
  JukkaL
  committed
  acba8bb
- Support operator assignment
  JukkaL
  committed
  2d698a9
- Support try statements
  JukkaL
  committed
  ad21b78
- Fix test
  JukkaL
  committed
  8d797d6
- Support ellipsis expressions
  JukkaL
  committed
  da7e92f
- Support 'if' expressions
  JukkaL
  committed
  dbffbf1
- Support type list
  JukkaL
  committed
  558181c
- Support ellipsis types
  JukkaL
  committed
  7853cc1
Commits on Jan 2, 2026
- Support del statements
  JukkaL
  committed
  66d0cd1
- Minimal support for f-strings
  JukkaL
  committed
  ac8fca5
- Support f-string conversion flags
  JukkaL
  committed
  363b0df
- Support f-string format specs and optimize simple f-strings
  JukkaL
  committed
  1902bc5
- Add f-string test case
  JukkaL
  committed
  1ea538e
- Optimize more simple cases
  JukkaL
  committed
  b396a3d
- Fix string literals mixed with f-strings
  JukkaL
  committed
  cf538c6
- Support lambda expressions
  JukkaL
  committed
  05fa03e
- Support assignment expressions (:=)
  JukkaL
  committed
  a72892a
- Support star expressions
  JukkaL
  committed
  7360a99
- Support bytes literals
  JukkaL
  committed
  dc6eab3
- Support 'global' and 'nonlocal' statements
  JukkaL
  committed
  1ee58dd
- Support await expressions
  JukkaL
  committed
  3ceb623
- Support bool literal types
  JukkaL
  committed
  a4ab43c
- Handle syntax errors
  JukkaL
  committed
  b5d73eb
Commits on Jan 3, 2026
- Refactor: split statement parsing in read_statement
  JukkaL
  committed
  34b76f4
- Support overloaded functions (no tests yet)
  JukkaL
  committed
  a510e5a
- Simplify
  JukkaL
  committed
  b9a63f6
- Deserialize function parameter locations
  JukkaL
  committed
  acaee8c
- Refactor function parameter deserialization
  JukkaL
  committed
  8a7c78b
- Improve decorators
  JukkaL
  committed
  8aa6bf0
- Fix decorator locations and add overload test cases
  JukkaL
  committed
  ac18a2b
- Basic support for type ignore comments
  JukkaL
  committed
  9445336
- Fix self check
  JukkaL
  committed
  e2b3b4a
Commits on Jan 4, 2026
- Support big integers
  JukkaL
  committed
  204e664
Commits on Jan 5, 2026
- Support metaclass in class defs
  JukkaL
  committed
  77e10cb
- Support arbitrary class def keywords
  JukkaL
  committed
  85c94c3
- Support unpack types
  JukkaL
  committed
  77288a5
- Support int literal types
  JukkaL
  committed
  2b89afb
- Add negative int literal type test
  JukkaL
  committed
  cd30492
Commits on Jan 7, 2026
- Replace non-ascii character in test
  JukkaL
  committed
  5a620d0
- Support star imports
  JukkaL
  committed
  bbe0164
- Support class decorators
  JukkaL
  committed
  0bd02e1
Commits on Jan 10, 2026
- Support string literal types
  JukkaL
  committed
  545f611
- Support bytes literal types
  JukkaL
  committed
  b6cc04a
Commits on Jan 17, 2026
- Add skip_function_bodies param
  JukkaL
  committed
  2a2d4e9
- Add strip function bodies tests (can't run them yet properly)
  JukkaL
  committed
  5191a54
- Support skipping function bodies in the test runner
  JukkaL
  committed
  a6d9ada
- Fix empty blocks
  JukkaL
  committed
  b7c91e3
- Add --native-parser flags (unused for now)
  JukkaL
  committed
  afc93b1
- WIP: basic integration of native parser
  JukkaL
  committed
  2072f9e
- Strip function bodies if errors are ignored
  JukkaL
  committed
  661dbe8
- Report syntax errors
  JukkaL
  committed
  387f1f2
- Fix class keywords
  JukkaL
  committed
  937a67e
- Fix overloads
  JukkaL
  committed
  999dcfc
Commits on Jan 18, 2026
- Support super() expressions
  JukkaL
  committed
  075788c
- Add support for callable types (still missing error handling)
  JukkaL
  committed
  6317d63
- Support positional-only arguments
  JukkaL
  committed
  a2f8485
- Add some type comment tests (simple cases only)
  JukkaL
  committed
  8e76879
- Support async with and async for
  JukkaL
  committed
  c400120
- Added nested list assignment test
  JukkaL
  committed
  103ecd5
Commits on Jan 19, 2026
- Add test case for legacy parser
  JukkaL
  committed
  0a5373b
- Make --native-parser affect cache
  JukkaL
  committed
  31d5660
- WIP return empty file for directory
  JukkaL
  committed
  49216b4
- WIP TEMPORARY ignore some errors caused be native parser issues
  JukkaL
  committed
  7232133
- WIP HAX allow parsing without file name
  JukkaL
  committed
  a949c69
- TEMPORARY: turn on native parser by default
  JukkaL
  committed
  69e73ac
- Fix elif ast structure
  JukkaL
  committed
  0f56995
- WIP Ignore
  JukkaL
  committed
  8cdcc61
Commits on Jan 24, 2026
- Revert "TEMPORARY: turn on native parser by default"
  JukkaL
  committed
  fcfecce
- Fix decorated functions
  JukkaL
  committed
  3dabbd0
- Revert some ignores
  JukkaL
  committed
  43fc602
- Lint
  JukkaL
  committed
  1e09832
- Add additional syntax error tests
  JukkaL
  committed
  85c335f
- Adjust type ignore
  JukkaL
  committed
  89d6def
Commits on Jan 25, 2026
- Add test for error conditions
  JukkaL
  committed
  010b443
- Propagate state param through parse functions
  JukkaL
  committed
  4a93c7f
- Add syntax error test case
  JukkaL
  committed
  e7fb3e3
- Check callable expression types for errors
  JukkaL
  committed
  89e1ec2
- Support invalid type annotations
  JukkaL
  committed
  0b56ba7
- Remove temporary test case
  JukkaL
  committed
  d57da8d
- Update function param Var node compatibility
  JukkaL
  committed
  b7a9b76
- Set more function type annotation attributes
  JukkaL
  committed
  05716ea
Commits on Jan 26, 2026
- Make special dunder params positional only
  JukkaL
  committed
  05f51f3
Commits on Jan 27, 2026
- FIXME enable --native-parser in testcheck.py
  JukkaL
  committed
  ab52bae
Commits on Jan 31, 2026
- [old parser] Add some conditional overload tests
  JukkaL
  committed
  e0cba2d
- Add helpers
  JukkaL
  committed
  74d5877
- Update native_parse to accept Options
  JukkaL
  committed
  35ee1d3
- Add helper from fastparse.py
  JukkaL
  committed
  9558665
- Add helper functions
  JukkaL
  committed
  103ecc4
- Add fix_function_overloads from fastparse.py
  JukkaL
  committed
  c0d7289
- Fix conditional overload items
  JukkaL
  committed
  e89ddf7
- Make some errors non-blocking
  JukkaL
  committed
  425152d
- Support --implicit-optional
  JukkaL
  committed
  e725ca6
- Update legacy parse tests to pass by adding Keywords
  JukkaL
  committed
  7dc2f8d
- Add legacy parser match statement tests
  JukkaL
  committed
  caf5159
- Deserialize match statements
  JukkaL
  committed
  305b6d3
- WIP PEP 695
  JukkaL
  committed
  ceb6c54
Commits on Feb 1, 2026
- Support except*
  JukkaL
  committed
  548fb1e
- Fix * syntax
  JukkaL
  committed
  c68d176
Commits on Feb 2, 2026
- Revert "Add legacy parser match statement tests"
  JukkaL
  committed
  cf786f1
Commits on Feb 7, 2026
- Ignore serialized imports
  JukkaL
  committed
  e454453
- Add function to deserialize out of band imports (currently unused)
  JukkaL
  committed
  21f27a0
- Use serialized import metadata
  JukkaL
  committed
  17455ac
- Remove now unused ImportCollector
  JukkaL
  committed
  82c448c
- Fix self check
  JukkaL
  committed
  cee5597
- Test import serialization/deserialization
  JukkaL
  committed
  17bcce5
- Deserialize block reachability
  JukkaL
  committed
  0f37b17
- Add reachability tests
  JukkaL
  committed
  76f6a32
- Add reachability tests
  JukkaL
  committed
  0deb566
Commits on Feb 8, 2026
- Fixes to import serialization
  JukkaL
  committed
  e09c3c2
- Fix conditional overloads
  JukkaL
  committed
  363a70a
- Lint
  JukkaL
  committed
  3945f7d
- Simplify redundant tests
  JukkaL
  committed
  c6bc1af
Commits on Feb 10, 2026
- Pass Python version to ast_serialize (fixes self check)
  JukkaL
  committed
  eb72dbf
Commits on Feb 15, 2026
- Merge branch 'master' into new-parser
  JukkaL
  committed
  f2e4706
- Skip native parser tests if ast_serialize is not installed
  JukkaL
  committed
  ae963f0
- Cleanup: Remove temporary benchmark tests
  JukkaL
  committed
  46f810d
- Fix failing native parser test
  JukkaL
  committed
  83914fd
- Fix test case
  JukkaL
  committed
  9e1f65d
- Refactor line/column range set
  JukkaL
  committed
  95319e3
- Add note about ast_serialize repo
  JukkaL
  committed
  60f733c
- Remove some low-value comments from mypy.nativeparse
  JukkaL
  committed
  7ccf14f
- Use native parser in testcheck.py only if env variable TEST_NATIVE_PARSER is set
  JukkaL
  committed
  fd23cb8
- Lint
  JukkaL
  committed
  98d9a1b
- Update semantic analyzer test outputs
  JukkaL
  committed
  3e87c1a
- Fix class keywords in tree transform
  JukkaL
  committed
  840bd1e
Commits on Feb 18, 2026
- Fix an obvious ordering bug to correctly set line
  ilevkivskyi
  committed
  144f6d6
- Always allow new union syntax inside strings/comments
  ilevkivskyi
  committed
  67ff2e5
- Couple tuple-related fixes (#20840)
  ilevkivskyi
  authored
  4270ff8
- Detect partial stub packages (#20844)
  ilevkivskyi
  authored
  3756c17
Commits on Feb 19, 2026
- Add support for inline TypedDicts (#20847)
  ilevkivskyi
  authored
  854eea8
- Fix couple more edge cases type comments/strings (#20848)
  ilevkivskyi
  authored
  fa08d89
Commits on Feb 21, 2026
- Update docstring
  JukkaL
  committed
  3869d79
- Remove test cases that aren't useful any more
  JukkaL
  committed
  512bf15
- Update comments
  JukkaL
  committed
  9c2ca82
- Remove type ignore that isn't needed any more
  JukkaL
  committed
  f7926c1
- Fix self check
  JukkaL
  committed
  f09d7ba
- Lint
  JukkaL
  committed
  ca7d19d
- Add reachability tests
  JukkaL
  committed
  61478c2
- Update docstring
  JukkaL
  committed
  99da49b
- Add docstring and reorganize functions
  JukkaL
  committed
  659ecbf
- Move some type read functions to be close to each other
  JukkaL
  committed
  8b31855
- Clean up test code
  JukkaL
  committed
  0017e18
- Merge branch 'master' into new-parser
  JukkaL
  committed
  11d04c6

Commits

Commits on Jan 1, 2026

Commits on Jan 2, 2026

Commits on Jan 3, 2026

Commits on Jan 4, 2026

Commits on Jan 5, 2026

Commits on Jan 7, 2026

Commits on Jan 10, 2026

Commits on Jan 17, 2026

Commits on Jan 18, 2026

Commits on Jan 19, 2026

Commits on Jan 24, 2026

Commits on Jan 25, 2026

Commits on Jan 26, 2026