Repository navigation
GH-55263: [Python][Annotations] Add internal types and core data structures #55264
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
9404659
5f7cdc8
610c48d
7bd5d7c
2f1a0ba
42ab12d
675ae64
14b924b
4ec84e4
235c90d
046fd11
0454fb9
e6bc69b
a33f327
ca9441c
dbc4e40
d2cc882
dce40ae
f278750
ca7c54e
c032596
3df0567
c990f32
cffdf24
bb8681e
376407d
bc27123
ddd9cb4
92250c5
9636e5c
4abbecc
325ed1d
26fa0e0
7d7a9fd
12ea05f
25a866b
94021e0
ae3cad1
ca5eb2e
d18e949
cf32c2e
516e8a9
401c3c9
368a6dc
91c3f0b
1589c4c
f6892cd
a1163e2
e51b18e
d297339
9dcdac6
2f49f15
3301e77
bf4a592
a9cdd20
905aa33
a94a96e
319b5c5
b642a3b
4b7dc90
41cc576
82f4e53
79a8f15
046f15f
5caf254
c093920
cce4648
61c7c26
6ed844e
6a4e614
da3465a
86fe422
5ea9439
42927be
724d480
1335885
bb913a8
17e3a08
e054197
4be907a
2d3f85c
6ae817e
ff91bcf
c7d1d97
d027724
9fc46cd
a8cc1ae
96ba7b9
0e446fc
faf04a9
4a899eb
91ae655
d2462bd
7f59731
7c1d685
dce7c2a
3675813
8e26b41
08bccc8
ceaf8c8
248bb04
157090e
fc6fb83
4f0e1d1
247ec99
eb7485d
fbf55e6
5a026a0
bc94a18
66d8661
abd654d
97a1f74
d3db98c
72146cb
5a31e48
739800d
f808d64
1616969
cf459f4
1531f93
69b549c
7fa510f
f2a746d
eac50b3
ccf5a11
3d17546
c4b5754
374093b
d085957
cd5e79c
7e9de7d
0f9b12e
bab3dbe
34f5978
0733c1d
b816747
40c4b9f
7b75996
2f5def7
661b373
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -145,6 +145,70 @@ repos: | |
| ( | ||
| ?^python/pyarrow/vendored/| | ||
| ) | ||
| # Linting, formatting and type checking of the type stubs. The type checkers | ||
| # don't run on commit, only on request (the Lint CI job runs them too): | ||
| # pre-commit run --hook-stage manual python-stubs --all-files | ||
| # The type checkers run with uvx, in the environment of | ||
| # python/requirements-stubs.txt. They don't need PyArrow installed: the | ||
| # modules that aren't annotated yet, e.g. pyarrow.compute, are treated as Any | ||
| # (see python/pyproject.toml). | ||
| - repo: https://github.com/astral-sh/ruff-pre-commit | ||
| rev: v0.17.2 | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Are we keeping these up-to-date anywhere? @renovatebot or @dependabot as options, I guess @pre-commit-ci too (although not a fan of anything @pre-commit personally).
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Given pace of typechecker development this might be a good idea. Added 34f5978, but we might need to discuss this in a wider setting? @jorisvandenbossche @AlenkaF @tadeja what do you think?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The pinned version here in pre-commit is not really related to pace of typechecker development, it is just pinning the ruff formatter. But anyway, automating the updates of our pre-commit hooks seems a good idea.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ah yes, sorry, I was going for the #55264 (comment) comment. |
||
| hooks: | ||
| - id: ruff-check | ||
| alias: python-stubs | ||
| name: Python Stubs Lint (ruff) | ||
| args: | ||
| - "--fix" | ||
| files: >- | ||
| ^python/pyarrow-stubs/ | ||
| types: | ||
| - pyi | ||
| - id: ruff-format | ||
| alias: python-stubs | ||
| name: Python Stubs Format (ruff) | ||
| files: >- | ||
| ^python/pyarrow-stubs/ | ||
| types: | ||
| - pyi | ||
| - repo: local | ||
| hooks: | ||
| - id: mypy | ||
| alias: python-stubs | ||
| name: Python Stubs Type Check (mypy) | ||
| stages: | ||
| - manual | ||
| language: python | ||
| entry: bash -c "cd python && exec uvx --with-requirements requirements-stubs.txt mypy" | ||
| pass_filenames: false | ||
| files: >- | ||
| ^python/(pyarrow-stubs(-tests)?/|(pyproject\.toml|requirements-stubs\.txt)$) | ||
| additional_dependencies: &uv | ||
| - uv==0.13.7 | ||
| - id: pyright | ||
| alias: python-stubs | ||
| name: Python Stubs Type Check (pyright) | ||
| stages: | ||
| - manual | ||
| language: python | ||
| entry: uvx --with-requirements python/requirements-stubs.txt pyright -p python/pyproject.toml | ||
| pass_filenames: false | ||
| files: >- | ||
| ^python/(pyarrow-stubs(-tests)?/|(pyproject\.toml|requirements-stubs\.txt)$) | ||
| additional_dependencies: *uv | ||
| - id: ty | ||
| alias: python-stubs | ||
| name: Python Stubs Type Check (ty) | ||
| stages: | ||
| - manual | ||
| language: python | ||
| # Unset the VIRTUAL_ENV of the hook, so that ty uses the environment | ||
| # of uvx. | ||
| entry: env -u VIRTUAL_ENV -u CONDA_PREFIX uvx --with-requirements python/requirements-stubs.txt ty check --project python | ||
| pass_filenames: false | ||
| files: >- | ||
| ^python/(pyarrow-stubs(-tests)?/|(pyproject\.toml|requirements-stubs\.txt)$) | ||
| additional_dependencies: *uv | ||
| - repo: https://github.com/MarcoGorelli/cython-lint | ||
| rev: v0.21.1 | ||
| hooks: | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -25,14 +25,20 @@ if [ -n "${ARROW_PYTHON_VENV:-}" ]; then | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| . "${ARROW_PYTHON_VENV}/bin/activate" | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| fi | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| # Install library stubs. Note some libraries contain their own type hints so they need to be installed. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| pip install fsspec pandas-stubs scipy-stubs types-cffi types-psutil types-requests types-python-dateutil | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| cd "${pyarrow_dir}" | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is currently tested only on: Lines 1535 to 1561 in 931b9cc
Which runs on I think this should be it's own CI job separate from documentation. Can we create a follow up issue to run this in a specific annotations CI job instead of running together with docs?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That'd be really good! Opened a follow-up issue #55676 Just to note - static checks run as part of
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This would be good indeed. Also in terms of environment management for typechecking tests. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| # Install type checkers | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| pip install mypy pyright ty | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| # Install type checkers and library stubs. Note some libraries contain their | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| # own type hints so they need to be installed. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| pip install -r requirements-stubs.txt | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| # Run type checkers | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| cd "${pyarrow_dir}" | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| mypy | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| pyright --stats | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ty check --verbose --output-format concise | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| # The type annotation tests are type checked above; also run them to verify | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| # the expected annotated types against the runtime types. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| pytest -r s pyarrow-stubs-tests | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| # Check that the stubs match the signatures of the installed PyArrow. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| python scripts/run_stubtest.py | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Uh oh!
There was an error while loading. Please reload this page.