Should importlib.util.spec_from_loader use a Protocol for the loader parameter ? #11882

Activity

1. abravalheri commented on May 10, 2024

   

   abravalheri

   on May 10, 2024

   Last edited by abravalheri

   Contributor

   More actions

   Regarding 1:

   After further consideration of the Python docs, we can see that the text references the glossary which describes loader as "an object that loads a module. It must define a method named load_module()" and points to PEP 302 which describes "The Importer Protocol"¹.

   We can see that in the standard library some perfectly valid loaders do not inherit from this abc:

   >>> from importlib.abc import Loader
   >>> import math
   >>> math.__loader__
   <class '_frozen_importlib.BuiltinImporter'>
   >>> isinstance(math.__loader__, Loader)
   False
   >>> import abc
   >>> abc.__loader__
   <class '_frozen_importlib.FrozenImporter'>
   >>> isinstance(abc.__loader__, Loader)
   False
   >>> import importlib.util
   >>> spec = importlib.util.spec_from_loader('math', math.__loader__)
   >>> module = importlib.util.module_from_spec(spec)
   >>> spec.loader.exec_module(module)
   >>> module.sin(module.pi / 2)
   1.0

   We also can see that the type hints that typeshed provides for some pkgutil functions are incorrect:

   >>> import pkgutil
   >>> print(loader := pkgutil.get_loader('os'), f"{isinstance(loader, Loader)=}")
   <class '_frozen_importlib.FrozenImporter'> isinstance(loader, Loader)=False
   >>> print(loader := pkgutil.get_loader('stat'), f"{isinstance(loader, Loader)=}")
   <class '_frozen_importlib.FrozenImporter'> isinstance(loader, Loader)=False
   >>> print(loader := pkgutil.get_loader('math'), f"{isinstance(loader, Loader)=}")
   <class '_frozen_importlib.BuiltinImporter'> isinstance(loader, Loader)=False
   >>> print(loader := pkgutil.get_loader('abc'), f"{isinstance(loader, Loader)=}")
   <class '_frozen_importlib.FrozenImporter'> isinstance(loader, Loader)=False
   >>> print(loader := pkgutil.get_loader('builtins'), f"{isinstance(loader, Loader)=}")
   <class '_frozen_importlib.BuiltinImporter'> isinstance(loader, Loader)=False
   >>> print(loader := pkgutil.get_loader('site'), f"{isinstance(loader, Loader)=}")
   <class '_frozen_importlib.FrozenImporter'> isinstance(loader, Loader)=False
   >>> from importlib.abc import PathEntryFinder, MetaPathFinder
   >>> print(importer := next(pkgutil.iter_importers('stat')), f"{isinstance(importer, (PathEntryFinder, MetaPathFi
   nder))=}")
   <class '_frozen_importlib.BuiltinImporter'> isinstance(importer, (PathEntryFinder, MetaPathFinder))=False
   >>> print(importer := next(pkgutil.iter_importers('math')), f"{isinstance(importer, (PathEntryFinder, MetaPathFi
   nder))=}")
   <class '_frozen_importlib.BuiltinImporter'> isinstance(importer, (PathEntryFinder, MetaPathFinder))=False

   These are all evidences that the definition in typeshed is incorrect and leads to false positives/negatives during type checking.

   I think that in general the best assumption is that importlib.abc abstract classes should not be used for other signatures in typeshed. Instead corresponding protocols should be defined.

   The likely reason is that the PEPs related to the importlib where designed in a time when a formal definition of Protocol was not available for Python and abc was used a conceptual replacement... Some of them serve as informational purposes (specially the generic ones like Loader, PathEntryFinder, MetaPathFinder) and others are just handy ways of sharing optional implementation details.

   Footnotes

   1. Note that even the word "protocol" is used in the standard... We can conclude that the abstract classes are provided as a documentation device and as means of sharing code among a few implementation of the standard library. ↩

   👍React with 👍1Reacted by Avasam
2. srittau commented on May 10, 2024

   

   srittau

   on May 10, 2024

   Collaborator

   More actions

   Using a protocol sounds about right to me for the reasons @abravalheri gave, especially due to the glossary definition and the fact that this is such a simple protocol.

   👍React with 👍1Reacted by Avasam
3. srittau

   closed this as completedin #11890on May 12, 2024