Intersphinx inventory classifies some type-like objects as data

@ambv

BPO	45391
Nosy	@ambv, @bskinn, @miss-islington, @gaborbernat, @Fidget-Spinner
PRs	bpo-45391: mark UnionType as a class in documentation #28757 [3.10] bpo-45391: mark UnionType as a class in documentation (GH-28757) #30004

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-10-06.12:20:50.089>
labels = ['3.10', 'docs']
title = '3.10 objects.inv classifies many types as data'
updated_at = <Date 2021-12-09.13:22:02.376>
user = 'https://github.com/gaborbernat'

bugs.python.org fields:

activity = <Date 2021-12-09.13:22:02.376>
actor = 'kj'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2021-10-06.12:20:50.089>
creator = 'gaborjbernat'
dependencies = []
files = []
hgrepos = []
issue_num = 45391
keywords = ['patch']
message_count = 11.0
messages = ['403300', '403301', '403308', '403310', '403311', '403313', '403318', '403343', '408115', '408119', '408120']
nosy_count = 7.0
nosy_names = ['docs@python', 'lukasz.langa', 'bskinn', 'miss-islington', 'gaborjbernat', 'dom1310df', 'kj']
pr_nums = ['28757', '30004']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue45391'
versions = ['Python 3.10']

Linked PRs

It's a class though:

❯ sphobjinv suggest ./objects.inv UnionType
:py:data:`types.UnionType`

defined as:

UnionType = type(int | str)

The issue with the current state this is that intersphinx fails to find types.UnionType in objects.inv because of leaves under the incorrect namespace (data vs class).

On the PR you mention there's more types with this problem. Can we get a full list?

Not easily, but, e.g. the EllipsisType is one. I would have to create some script which I haven't done yet.

The best would be to create a sphinx plugin that collects entries registered in the doc and displays the discrepancy against the intershpinx object. This way, we could defend against future such issues too. I can give it a go in a few days if no one else wants to do so before that.

If I understand the problem correctly, these mis-attributions of roles (to 'data' instead of 'class' come about when a thing that is technically a class is defined in source using simple assignment, as with UnionType.

Problematic entries will thus have 'data' as role, and their identifiers will be camel-cased.

So, as a quick search to identify likely candidates:

>> import re, sphobjinv as soi
>> from pprint import pprint
>> inv = soi.Inventory(url="https://docs.python.org/3.10/objects.inv")

# Find entries where the first character after the final period
# is uppercase, and the second character after the final period
# is lowercase.
>>> pat = re.compile(r"([.][A-Z][a-z])[^.]*$")

>> pprint([obj.name for obj in inv.objects if obj.role == "data" and pat.search(obj.name)])

['_thread.LockType',
'ast.PyCF_ALLOW_TOP_LEVEL_AWAIT',
'ast.PyCF_ONLY_AST',
'ast.PyCF_TYPE_COMMENTS',
'importlib.resources.Package',
'importlib.resources.Resource',
'socket.SocketType',
'types.AsyncGeneratorType',
'types.BuiltinFunctionType',
'types.BuiltinMethodType',
'types.CellType',
'types.ClassMethodDescriptorType',
'types.CoroutineType',
'types.EllipsisType',
'types.FrameType',
'types.FunctionType',
'types.GeneratorType',
'types.GetSetDescriptorType',
'types.LambdaType',
'types.MemberDescriptorType',
'types.MethodDescriptorType',
'types.MethodType',
'types.MethodWrapperType',
'types.NoneType',
'types.NotImplementedType',
'types.UnionType',
'types.WrapperDescriptorType',
'typing.Annotated',
'typing.Any',
'typing.AnyStr',
'typing.Callable',
'typing.ClassVar',
'typing.Concatenate',
'typing.Final',
'typing.Literal',
'typing.NoReturn',
'typing.Optional',
'typing.ParamSpecArgs',
'typing.ParamSpecKwargs',
'typing.Tuple',
'typing.TypeAlias',
'typing.TypeGuard',
'typing.Union',
'weakref.CallableProxyType',
'weakref.ProxyType',
'weakref.ProxyTypes',
'weakref.ReferenceType']

I would guess those 'ast.PyCF...' objects can be ignored, they appear to be constants?

Identifiers starting with two uppercase letters returns a HUGE list.

>> pat2 = re.compile(r"([.][A-Z][A-Z])[^.]*$")

Filtering down by only those that contain.lower() "type":

>> pprint([obj.name for obj in inv.objects if obj.role == "data" and pat2.search(obj.name) and "type" in obj.name.lower()])

['errno.EPROTOTYPE',
'locale.LC_CTYPE',
'sqlite3.PARSE_DECLTYPES',
'ssl.CHANNEL_BINDING_TYPES',
'token.TYPE_COMMENT',
'token.TYPE_IGNORE',
'typing.TYPE_CHECKING',
'xml.parsers.expat.XMLParserType']

Of these, only 'xml.parsers.expat.XMLParserType' seems to me a likely problem entry.

I think Brian Skinn script is a rough approximation, but definitely entirely accurate. You'd have to match up what sphinx thinks per doc vs what you import for an accurate view.

Here's a gist where I managed to detect roughly 140 errors (some looks like potential false positive, so likely the real number is more around 100):

https://gist.github.com/gaborbernat/5360badab2125b3f81a3bcbce0e94c2a#file-found_issues-output-L1

This does make a few concessions:

ignores the difference between function and method; way to many functions are documented as methods and vice-versa to disallow this (or would be a major overhaul)
https://docs.python.org/3/c-api/structures.html?highlight=meth_class#METH_VARARGS is documented under python domain but IMHO should be C
https://docs.python.org/3/c-api/typeobj.html?highlight=py_tpflags_base_exc_subclass#c.PyTypeObject.tp_flags is documented under python domain but IMHO should be C
does not clarifies where to type classes goes - they seem to be a weird in-between a method and a class, satisfying neither - see related discussion on topic from https://bugs.python.org/issue41973

New changeset e2cfc89 by Bernát Gábor in branch 'main':
bpo-45391: mark UnionType as a class in documentation (GH-28757)
e2cfc89

[3.14] gh-89554: Document typing.ParamSpecArgs and ParamSpecKwargs as…

[3.15] gh-89554: Document typing.ParamSpecArgs and ParamSpecKwargs as…

gh-89554: Document _thread.LockType as a class (#150684)

[3.14] gh-89554: Document _thread.LockType as a class (GH-150684) (#1…

[3.15] gh-89554: Document _thread.LockType as a class (GH-150684) (#1…

[3.13] gh-89554: Document _thread.LockType as a class (#150684) (#150786

pythongh-89554: Apply review suggestions for socket roles

gh-89554: Document socket.SocketType as a class (#150683)

[3.15] gh-89554: Document socket.SocketType as a class (GH-150683) (#…

[3.14] gh-89554: Document socket.SocketType as a class (#150683) (#15…

[3.13][3.14] gh-89554: Document socket.SocketType as a class (#150683) (

pythongh-89554: Document socket.SocketType as a class (python#150683)

Preserve old panel anchors via injected HTML spans

[3.13] gh-89554: Document standard type objects in types as classes (G…

[3.14] gh-89554: Document standard type objects in types as classes (G…

Navigation Menu

Repository navigation

Intersphinx inventory classifies some type-like objects as `data` #89554

Linked PRs

39 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Intersphinx inventory classifies some type-like objects as data #89554

Description

Linked PRs

Activity

gaborbernat commented on Oct 6, 2021

gaborbernat commented on Oct 6, 2021

ambv commented on Oct 6, 2021

gaborbernat commented on Oct 6, 2021

bskinn commented on Oct 6, 2021

bskinn commented on Oct 6, 2021

gaborbernat commented on Oct 6, 2021

gaborbernat commented on Oct 7, 2021

Fidget-Spinner commented on Dec 9, 2021

39 remaining items

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Intersphinx inventory classifies some type-like objects as `data` #89554