types-protobuf fails pre-commit cache

@AlexWaygood

when configuring pre-commit mypy to run with types-protobuf, for example:

  -   repo: https://github.com/pre-commit/mirrors-mypy
      rev: v1.8.0
      hooks:
        -   id: mypy
            additional_dependencies:
              - types-protobuf

it fails with the following:

mypy.....................................................................Failed
- hook id: mypy
- exit code: 2

mypy: can't read file '/home/runner/.cache/pre-commit/repoem44ysms/py_env-python3.10/lib/python3.10/site-packages//google': No such file or directory

i suspect its related to #11249, but not sure - @AlexWaygood @hauntsaninja i would appreciate your insights here

@Itay4

Encountering the same issue as well, this was previously working....

edit: happy to provide logs etc if that helps, but the resulting message I get when I run the pre-commits is the same as @Itay4 's output shown above.

edit2: if it helps, I tried using both Python 3.10 and Python 3.11 virtual envs for running the pre-commit hooks but the same error occurs.

edit3: I cannot find a way to use a "downgraded" of an additional_dependencies entry in mypy as a pre-commit hook so cannot check that previous versions worked, but we certainly did not have this problem previously...

Neither #11249, not #11245, the two PRs included in the latest release, should have an influence here. I'm not sure what exactly is going on here, but types-protobuf doesn't add a google directory, but a google-stubs directory, so it's no wonder that mypy can't find it. Does downgrading to types-protobuf 4.24.0.4 help?

@srittau

Hi @srittau, as I mentioned previously I do know a way to specific specific version within mypy in a pre-commit environment.

Using the default though, exhibits the behavior mentioned by the original author.

If you know a way to "pin" an additional depency version, let me know and I can test it.

@Avasam

This problem was introduced by typeshed-internal/stub_uploader#109. Previously, the google-stubs directory was missing the py.typed file that marked the stubs as partial. I think this was an error according to the typing spec:

For namespace packages (see PEP 420), stub-only packages should use the -stubs suffix on only the root namespace package. All stub-only namespace packages should omit __init__.pyi files. py.typed marker files are not necessary for stub-only packages, but similarly to packages with inline types, if used, they should be in submodules of the namespace to avoid conflicts and for clarity.

Cc @Avasam

If I understand correctly, "google" is a namespace package and the py.typed marker should be found in google-stubs/protobuf/ rather than google-stubs/ ? (that's what I'm getting from the part of the spec srittau is quotig)

In the short term, simply setting partial_stub to false in https://github.com/python/typeshed/blob/main/stubs/protobuf/METADATA.toml (and namespace packages in general) with an appropriate comment should be enough to fix the current issue (no py.typed file should be generated for non-partial stubs, as it is optional and redundant as per the spec).

In the longer term, we may want a way to identify those so stub_uploader adds the py.typed file to the submodules. Like the lack of any file at the top-level.

@Avasam

@Avasam thanks for the explanation, any workarounds for this issue for the interim :-)?

If I understand correctly, "google" is a namespace package and the py.typed marker should be found in google-stubs/protobuf/ rather than google-stubs/ ? (that's what I'm getting from the part of the spec srittau is quotig)

That's my understanding as well. I think making the necessary changes to the stub_uploader should be fairly simple, although I haven't investigated deeply.

protobuff: Prevent namespace package google from having a py.typed fi…

@Avasam

@Avasam thanks for the explanation, any workarounds for this issue for the interim :-)?

@andylamp Check the very bottom of https://github.com/pre-commit/mirrors-mypy 's readme. I'm pretty sure it's just using pip, so the same markers should work to pin to an older version: types-protobuf==X.X.X

Encapsulate PackageData; use pathlib

Hi, would it be possible to delete this version from pypi?

@ilevkivskyi

I'm not sure who has access to the packages on PyPI. I think @ilevkivskyi could do it.

Also finding all my CI pipelines have broken. In my GitLab CI script I have had to pin it to 4.24.0.4 with the following: -

- repo: https://github.com/pre-commit/mirrors-mypy
  rev: v1.8.0
  hooks:
  - id: mypy
    files: ^app
    args:
    - --ignore-missing-imports
    additional_dependencies:
    - types-protobuf==4.24.0.4
    - types-PyYAML
    - types-python-dateutil
    - types-pytz
    - types-requests

Any chance this release can be pulled very soon?

I would suggest that publishing what looks like a nightly build to PyPI (4.24.0.20240106) is probably not recommended in the future?

Stubs are now calendar-versionned, (hence the bump from 4, an index, to 20240106, a date) with the express purpose to communicate that the version number is not semantic (and has never been). There's no such thing as a major/minor/patch/nightly update in typeshed's stubs.
See #10837, #6843 and typeshed-internal/stub_uploader#105 for more details.

I have had to pin it to 4.24.0.4 with the following: -
Any chance this release can be pulled very soon?

Hopefully? But I'd recommend pinning your libraries versions either way if you want to avoid unexpected CI breakage. Even an update that fixes or improves existing stubs, or an update to type-checkers, could cause your CI to fail all of a sudden (like new false-positives, using new features your type-checker version doesn't support, fixing a false-positive you had suppressed, etc.)

Encapsulate PackageData; use pathlib (#119)

Do you still need me to delete some package from PyPI? If yes, could you please post exactly what is needed (i.e. library and version)?

@ilevkivskyi

Hi @ilevkivskyi - For me? I fix the problems with types-protobuf==4.24.0.4 as 4.24.0.20240106 breaks mypy (see #11263).

That's (obviously) not a solution to the problem.

I have a work-around and I don't have a strong feeling on the topic (it's not my repo) so I'm fine - but it means the latest release can't be used by me (and, I suspect, others) because of the problems it seems to cause with mypy.

If someone thinks the problem with types-protobuf is caused by types-protobuf then I guess, yes, pull it. Does someone know what the problem is?

@ilevkivskyi

Hi @ilevkivskyi , the removal idea come from me, IMHO if we know that the release is broken and the fix is not coming in hours, then it should be removed: types-protobuf 4.24.0.20240106
buuut I'm just a guy from the internet :)

See python/typeshed#11254

build(python): fix `docs` and `docfx` builds (#272)

See python/typeshed#11254

build(python): fix `docs` and `docfx` builds (#581)

Refactor collect_package_data to make logic clearer (#120)

Update to those following this issue: with typeshed-internal/stub_uploader#122, the latest version of types-protobuff (4.24.0.20240129) shouldn't have this issue as it doesn't have any py.typed marker.

The marker file should properly be re-introduced in the subpackages (rather than the root of the namespace package), in a future update to stub_uploader.

This issue should be resolved and we haven't had any new complaints since then.
For the missing py.typed marker in namespace packages, please refer to typeshed-internal/stub_uploader#121 (comment) for tracking.

build(python): fix `docs` and `docfx` builds (#581)

build(python): fix `docs` and `docfx` builds (#272)

Navigation Menu

Repository navigation

types-protobuf fails pre-commit cache #11254

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

types-protobuf fails pre-commit cache #11254

Description

Activity

andylamp commented on Jan 7, 2024

srittau commented on Jan 8, 2024

andylamp commented on Jan 8, 2024

srittau commented on Jan 8, 2024

Avasam commented on Jan 8, 2024

andylamp commented on Jan 8, 2024

srittau commented on Jan 8, 2024

Avasam commented on Jan 8, 2024

tonkolviktor commented on Jan 8, 2024

srittau commented on Jan 8, 2024

alanbchristie commented on Jan 9, 2024

Avasam commented on Jan 9, 2024

ilevkivskyi commented on Jan 15, 2024

alanbchristie commented on Jan 16, 2024

tonkolviktor commented on Jan 16, 2024

Avasam commented on Jan 31, 2024

Avasam commented on Mar 14, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions