Daily tests failed on Tue Apr 23 2024

Runs listed here: https://github.com/python/typeshed/actions/workflows/daily.yml

@Daverball

Four distinct errors:

Some regular stubtest failures on TensorFlow about some new parameters being missing
The stubtest_third_party.py script crashed on uwsgi on macos (cc. @Daverball)
Some other weird stubtest failures on macos for pyaudio and JACK-Client: modules failed to import for some reason

Not sure what's up with the macos failures for pyaudio and uwsgi — possibly something's gone wrong with the setup-python build for macos?

@AlexWaygood

@AlexWaygood My best guess is that the MacOS container has updated its OpenSSL version and the uWSGI wheel was built against an earlier incompatible version. We could either try to force an sdist install (which may require installing some development headers) or just skip the MacOS tests on uWSGI for now.

Or the MacOS containers are just borked altogether currently considering the other weird failures related to dynamic libraries.

Fixing the tensorflow problem is above my pay grade. It seems that the source code at https://github.com/tensorflow/tensorflow is only casually related to the source code that gets installed – even when using the correct tag or branch.

Tensorflow errors:

Details


**********************************************************************

Commands run:

/tmp/tmph4cri9bo/bin/pip install tensorflow[]==2.16.* mypy==1.9.0 numpy>=1.20 urllib3>=2
MYPYPATH=/home/runner/work/typeshed/typeshed/stubs/tensorflow:/home/runner/work/typeshed/typeshed/stubs/protobuf:/home/runner/work/typeshed/typeshed/stubs/requests /tmp/tmph4cri9bo/bin/python -m mypy.stubtest --custom-typeshed-dir /home/runner/work/typeshed/typeshed --ignore-missing-stub tensorflow --allowlist /home/runner/work/typeshed/typeshed/stubs/tensorflow/@tests/stubtest_allowlist.txt

**********************************************************************

Command output:

error: tensorflow.keras.layers.Embedding.__init__ is inconsistent, stub argument "lora_rank" differs from runtime argument "weights"
Stub: in file /home/runner/work/typeshed/typeshed/stubs/tensorflow/tensorflow/keras/layers/__init__.pyi:286
def (self: tensorflow.keras.layers.Embedding, input_dim: builtins.int, output_dim: builtins.int, embeddings_initializer: Union[builtins.str, tensorflow.keras.initializers.Initializer, Type[tensorflow.keras.initializers.Initializer], def (Union[tensorflow.TensorShape, typing.Iterable[Union[tensorflow.Tensor, builtins.str, builtins.float, numpy.ndarray[Any, Any], numpy.number[Any], None]], builtins.int, tensorflow.Tensor]) -> tensorflow.Tensor, builtins.dict[builtins.str, Any], None] =, embeddings_regularizer: Union[builtins.str, builtins.dict[builtins.str, Any], tensorflow.keras.regularizers.Regularizer, None] =, embeddings_constraint: Union[builtins.str, builtins.dict[builtins.str, Any], tensorflow.keras.constraints.Constraint, None] =, mask_zero: builtins.bool =, lora_rank: Union[builtins.int, None] =, *, input_length: Union[builtins.int, None] =, activity_regularizer: Union[builtins.str, builtins.dict[builtins.str, Any], tensorflow.keras.regularizers.Regularizer, None] =, trainable: builtins.bool =, dtype: Union[Union[tensorflow.dtypes.DType, builtins.str, numpy.dtype[Any], builtins.int, builtins.dict[builtins.str, Any], Any], None] =, autocast: builtins.bool =, name: Union[builtins.str, None] =)
Runtime: in file /tmp/tmph4cri9bo/lib/python3.11/site-packages/keras/src/layers/core/embedding.py:76
def (self, input_dim, output_dim, embeddings_initializer='uniform', embeddings_regularizer=None, embeddings_constraint=None, mask_zero=False, weights=None, lora_rank=None, **kwargs)

error: tensorflow.keras.layers.Embedding.__init__ is inconsistent, stub does not have argument "lora_rank"
Stub: in file /home/runner/work/typeshed/typeshed/stubs/tensorflow/tensorflow/keras/layers/__init__.pyi:286
def (self: tensorflow.keras.layers.Embedding, input_dim: builtins.int, output_dim: builtins.int, embeddings_initializer: Union[builtins.str, tensorflow.keras.initializers.Initializer, Type[tensorflow.keras.initializers.Initializer], def (Union[tensorflow.TensorShape, typing.Iterable[Union[tensorflow.Tensor, builtins.str, builtins.float, numpy.ndarray[Any, Any], numpy.number[Any], None]], builtins.int, tensorflow.Tensor]) -> tensorflow.Tensor, builtins.dict[builtins.str, Any], None] =, embeddings_regularizer: Union[builtins.str, builtins.dict[builtins.str, Any], tensorflow.keras.regularizers.Regularizer, None] =, embeddings_constraint: Union[builtins.str, builtins.dict[builtins.str, Any], tensorflow.keras.constraints.Constraint, None] =, mask_zero: builtins.bool =, lora_rank: Union[builtins.int, None] =, *, input_length: Union[builtins.int, None] =, activity_regularizer: Union[builtins.str, builtins.dict[builtins.str, Any], tensorflow.keras.regularizers.Regularizer, None] =, trainable: builtins.bool =, dtype: Union[Union[tensorflow.dtypes.DType, builtins.str, numpy.dtype[Any], builtins.int, builtins.dict[builtins.str, Any], Any], None] =, autocast: builtins.bool =, name: Union[builtins.str, None] =)
Runtime: in file /tmp/tmph4cri9bo/lib/python3.11/site-packages/keras/src/layers/core/embedding.py:76
def (self, input_dim, output_dim, embeddings_initializer='uniform', embeddings_regularizer=None, embeddings_constraint=None, mask_zero=False, weights=None, lora_rank=None, **kwargs)

Found 2 errors (checked 2289 modules)
2024-04-23 08:11:07.850458: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-23 08:11:07.853603: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-23 08:11:07.891617: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-23 08:11:08.784575: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

**********************************************************************

Python version: Python 3.11.9 (main, Apr  2 2024, 15:19:53) [GCC 11.4.0]

Ran with the following environment:
absl-py==2.1.0
astunparse==1.6.3
certifi==2024.2.2
charset-normalizer==3.3.2
flatbuffers==24.3.25
gast==0.5.4
google-pasta==0.2.0
grpcio==1.62.2
h5py==3.11.0
idna==3.7
keras==3.3.2
libclang==18.1.1
Markdown==3.6
markdown-it-py==3.0.0
MarkupSafe==2.1.5
mdurl==0.1.2
ml-dtypes==0.3.2
mypy==1.9.0
mypy-extensions==1.0.0
namex==0.0.8
numpy==1.26.4
opt-einsum==3.3.0
optree==0.11.0
packaging==24.0
pip==24.0
protobuf==4.25.3
Pygments==2.17.2
requests==2.31.0
rich==13.7.1
setuptools==69.5.1
six==1.16.0
tensorboard==2.16.2
tensorboard-data-server==0.7.2
tensorflow==2.16.1
tensorflow-io-gcs-filesystem==0.36.0
termcolor==2.4.0
typing_extensions==4.11.0
urllib3==2.2.1
Werkzeug==3.0.2
wheel==0.43.0
wrapt==1.16.0

**********************************************************************

To fix "unused allowlist" errors, remove the corresponding entries from stubs/tensorflow/@tests/stubtest_allowlist.txt


**********************************************************************

Upstream repository: https://github.com/tensorflow/tensorflow
Typeshed source code: https://github.com/python/typeshed/tree/main/stubs/tensorflow

**********************************************************************

Looks like it's due to the release of keras==3.3.2 overnight: https://pypi.org/project/keras/#history.

If stubtest can start failing on types-tensorflow due to a new release of keras, then ideally we would pin keras by adding this to the tool.stubtest table of stubs/tensorflow/METADATA.toml:

stubtest_requirements = ["keras==major.minor.patch"]

But we would have to remember to manually update the pin whenever there's a new keras release (otherwise, our stubs would go out of date)...

Tensorflow has been fixed (#11817), now it's "just" the three bizarre macOS failures on uwsgi, pyaudio and JACK-Client that we need to fix

@AlexWaygood

@AlexWaygood Since the MacOS specific failures are new since today and affect multiple distributions that dynamically load external dynlibs I'd suggest waiting at least until tomorrow's run before investing more time into it. Or manually trigger a run now, to verify it is still broken and wasn't just some flaky behavior with a broken CI runner.

@AlexWaygood

@AlexWaygood Since the MacOS specific failures are new since today and affect multiple distributions that dynamically load external dynlibs I'd suggest waiting at least until tomorrow's run before investing more time into it. Or manually trigger a run now, to verify it is still broken and wasn't just some flaky behavior with a broken CI runner.

@srittau already triggered a rerun and we had the same errors: #11814

I see, for some reason I'm not seeing the manual re-run in the history unless it's also marked as "Scheduled" despite being manually triggered.

Oh I see it was a re-run of the original job, rather than a manual run. That explains it, I was looking in the wrong place.

I see, for some reason I'm not seeing the manual re-run in the history unless it's also marked as "Scheduled" despite being manually triggered.

@AlexWaygood

@AlexWaygood Could this perhaps be a caching issue? I'm not sure to what degree the pip cache on the setup-python action on the CI is actually still used currently, since some of the dependency management has been switched over to uv from pip. This seems like the kind of error that would happen if a package that was built against an older version of a library which stuck around in cache from an earlier run was used with a newer version.

For stubtest_third_party.py we still exclusively use pip, not uv. Could be a caching issue. I don't have time to do much investigating right now, but please feel free to open an experimental PR to try to debug via CI, if you have a hunch :-)

Not sure how easy this would be to test from a PR, since it's trying to be clever and only runs the tests for the distributions that have changed, it might be quicker to just delete all the macOS pip caches here https://github.com/python/typeshed/actions/caches and then trigger a re-run. Given that we find someone that has the necessary privileges to do so.

any PR that edits the daily.yml workflow file itself triggers the fully daily test with stubtest being run on all third-party stubs

Navigation Menu

Repository navigation

Daily tests failed on Tue Apr 23 2024 #11813

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Daily tests failed on Tue Apr 23 2024 #11813

Description

Activity

AlexWaygood commented on Apr 23, 2024

Daverball commented on Apr 23, 2024

Daverball commented on Apr 23, 2024

srittau commented on Apr 23, 2024

AlexWaygood commented on Apr 23, 2024

AlexWaygood commented on Apr 23, 2024

AlexWaygood commented on Apr 23, 2024

Daverball commented on Apr 23, 2024

AlexWaygood commented on Apr 23, 2024

Daverball commented on Apr 23, 2024

AlexWaygood commented on Apr 23, 2024

Daverball commented on Apr 23, 2024

AlexWaygood commented on Apr 23, 2024

Daverball commented on Apr 23, 2024

AlexWaygood commented on Apr 23, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions