Define a JSON type

JSON is such a common interchange format it might make sense to define it as a specific type.

JSON = t.Union[str, int, float, bool, None, t.Mapping[str, 'JSON'], t.List['JSON']]

Not sure if this should go into typing or be introduces as json.JSONType instead (or if it's even worth it considering the variability of the type).

All major type checkers now support recursive type aliases by default, so this should largely work:

JSON: TypeAlias = dict[str, "JSON"] | list["JSON"] | str | int | float | bool | None

Note that because dict is invariant, you might run into some issues e.g. with dict[str, str]. That is, you will not be able to pass something with type dict[str, str] to a function that takes a JSON

For such use cases you can use cast, and if you don't need mutability, something like the following might work:

JSON_ro: TypeAlias = Mapping[str, "JSON_ro"] | Sequence["JSON_ro"] | str | int | float | bool | None

Although this can sometimes result in false negatives, for instance, see python/mypy#13786 (comment)

I tried to do that but a recursive type alias doesn't work in mypy right now, and I'm not sure how to make it work. In the mean time I use JsonDict = Dict[str, Any] (which is not very useful but at least clarifies that the keys are strings), and Any for places where a more general JSON type is expected.

(I'm sure you meant t.Mapping[str, 'JSON'].)

You are right about what I meant and I fixed my comment to not confuse anyone in the future.

And I understand about the lack of recursive object support.

Would this be a better definition?

JSONValue = t.Union[str, int, float, bool, None, t.Dict[str, t.Any], t.List[t.Any]]
JSONType = t.Union[t.Dict[str, JSONValue], t.List[JSONValue]]

If you read RFC 4627 it says a JSON object must be an object or array at the top level (RFC 7159 loosens that to any JSON value, but it's not an accepted standard). If you want to play it safe with what json.loads() takes, then you can just flatten it to:

JSONType = t.Union[str, int, float, bool, None, t.Dict[str, t.Any], t.List[t.Any]]

I guess the real question is how far you want to take this, because if you assume that most JSON objects only go, e.g. 4 levels deeps, you could handcraft accuracy to that level:

_JSONType_0 = t.Union[str, int, float, bool, None, t.Dict[str, t.Any], t.List[t.Any]]
_JSONType_1 = t.Union[str, int, float, bool, None, t.Dict[str, _JSONType_0], t.List[_JSONType_0]]
_JSONType_2 = t.Union[str, int, float, bool, None, t.Dict[str, _JSONType_1], t.List[_JSONType_1]]
_JSONType_3 = t.Union[str, int, float, bool, None, t.Dict[str, _JSONType_2], t.List[_JSONType_2]]
JSONType =  t.Union[str, int, float, bool, None, t.Dict[str, _JSONType_3], t.List[_JSONType_3]]

But then again the union of those objects is pretty broad so this might not be the most useful type hint. :)

I guess that'll work, but I'm not convinced that it's very useful to
do the multiple levels.

The next question is where this would live? Would you add it to
typing.py? Or to the json module? It would have to be added both to
json.pyi as well to the actual implementation module. :-(

OK, so JSONType = t.Union[str, int, float, bool, None, t.Dict[str, t.Any], t.List[t.Any]] seems to be the best solution.

As for where it should go, I don't have a good answer unfortunately. It's like the collections.abc issue; do we have a single place to keep all types -- i.e., typing -- or do we keep types in the specific modules that they relate to (i.e., json in this case)? I guess this would be the first type that wasn't a generic container if we add it to the stdlib somewhere, so there is no precedent to go by.

If we put it in typing then at least all types related to the stdlib are in a single location which is handy for only having to do import typing as t to get at all types. Unfortunately that doesn't work for for third-party libraries so it doesn't seem like the best way to go. So I guess my suggestion is the json module should house it and keep the name JSONType for the module attribute. If you agree I will open an issue on bugs.python.org to add the type to Python 3.6 and then also an accompanying issue for https://github.com/python/typeshed to add a json.pyi and you can close this issue. Otherwise I'll submit a PR to add the type to typing.

I think it's best to add it to the json module; we can't keep adding everything to typing.py (even the io and re types there are already questionable). Code that wants to use these in Python 3.5 or earlier can write

if False:
    from json import JSONType

(Or they can copy the definition into their own code.)

Question: should we name it JsonType or JSONType? There doesn't seem to be a strong convention here in the Python stdlib -- we have XmlListener and HTMLParser... But somehow I am beginning to prefer AcronymsAreWords.

Actually, since the json module consistently uses JSONWhatevs, it should be JSONType.

PEP 8 says to capitalize abbreviations.

Open an issue for the stdlib at http://bugs.python.org/issue26396 and one for typeshed at python/typeshed#84.

I'm marking this for 3.5.2 so we at least have the discussion. I'm still not sure of the solution -- add it to typeshed/.../json.pyi or to typing.py? It can't appear in the stdlib json module until 3.6 but it could appear in typing.py in 3.5.2 (since typing.py is provisional), but I'm not excited about pushing everything to typing.py. So maybe adding it to typeshed/**/json.pyi now and the stdlib json module in 3.6 would be best? If you want to use it then you'd have to write if false: from json import JsonObject.

(I've got a feeling I'm just summarizing where we ended up before but I'm currently in the mood to make the definitive list of things to discuss and/or implement before 3.5.2.)

@gvanrossum

I'm not a fan of having a "partial" JSON type that soon degenerates into Any. Type checkers would enforce things inconsistently. As soon as you descent into a JSON object you'd have to manually annotate the result to get type checking for the component object, and this would be hard to do consistently.

Having a recursive JSON type seems like a better idea to me, but even then I'd like to see how the type works for real-world code before including it in the PEP. I suspect that the majority of code doing JSON parsing actually doesn't perform enough isinstance checks when manipulating JSON objects for the code to type check cleanly. I wouldn't like PEP 484 to require programmers to jump through hoops to get their code to type check. For example, just today I reviewed some JSON parsing code that does not perform enough checks to pass type checking if it had used a strict type for JSON, but I think that the code was fine (@gvanrossum do you recognize what code I'm talking about?) :-)

Anyway, if programmers want to use such as partial type, they can define the alias and use it in their code even without making it official, though they may have to introduce some explicit type annotations when interacting with library code that doesn't use the type.

Two problems with adding a recursive JSON type to the PEP:

IIRC Brett and I tried and failed to come up with a recursive definition that worked in mypy
I don't believe JSON is special enough to deserve a place in the PEP or typing (re and io are borderline but they are way more fundamental than JSON)

@gvanrossum

The summary @gvanrossum gave of where things left off was accurate. Didn't come up with a recursive type that could work.

In response to @JukkaL about usefulness, I view it as useful for specifying what json.load() returns, not what json.dump() accepts. This is what I came across in my own code when I was trying to do proper type hinting but didn't have a way better than Any to express an method parameter that was accepting a JSON object that was received from GitHub's API.

Somehow this was closed but we don't even have consensus!

@AdrianCert

To translate @AdrianCert "worked for me" for others: he likely did not use the open-source mypy and probably checking with the closed-source pylance from inside vscode, which already supports recursive types. That does not count for us, even if we may be using it. As long mypy cannot work with it, it is not sorted.

I only hope that since @gvanrossum joined MSFT, he is still allowed to contribute to mypy, even if I do see how that could be seen as a conflict on interest with pylance.

Now by looking at https://github.com/python/mypy/blame/dc5f89142cb36debc0e804a472f49f51448d76f3/test-data/unit/check-recursive-types.test#L7 I think there is something baking there, but there is no release containing it. It should be included in next release. Not sure how well it works but I will be one of the first to check it. I do need to navigate JSON-like structures a lot these days.

probably checking with the closed-source pylance from inside vscode

I'm also pretty excited about the recent work on recursive types at mypy, but FWIW, pylance uses pyright for its type checking, and pyright is fully open source: https://github.com/microsoft/pyright#faq.

I can still contribute to mypy, I just am pretty rusty with that code base so you won’t hear much from me. Sorry! I also feel this will be sorted soon, it is just difficult to transition a large code base. See python/mypy#731 (comment) .

seems that mypy 0.981 has some (preliminary) support for recursive types using the --enable-recursive-aliases flag. this recursive type definition:

Json = dict[str, 'Json'] | list['Json'] | str | int | float | bool | None

… can be used to type check this code …

d: Json = {'foo': ['bar', {'x': 'y'}]}
reveal_type(d)

… successfully 🥳:

$ mypy --strict --enable-recursive-aliases test.py 
test.py:4: note: Revealed type is "Union[builtins.dict[builtins.str, ...], builtins.list[...], builtins.str, builtins.int, builtins.float, builtins.bool, None]"
Success: no issues found in 1 source file

@wbolster

For anyone with a strong aversion to copy-pasting, I have published the above one-liner from @wbolster as a library https://github.com/kevinheavey/jsonalias

(No this isn't a joke, I need the Json type in multiple projects so I decided to just publish it)

If only I could create one at runtime storing self-referential (scoped, non-string) annotations. Will mypy eventually enable recursive aliases by default?

…

On Fri, 2022-10-28 at 16:08 -0700, Kevin Heavey wrote: For anyone with a strong aversion to copy-pasting, I have published the above one-liner from @wbolster as a library https://github.com/kevinheavey/jsonalias (No this isn't a joke, I need the Json type in multiple projects so I decided to just publish it) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

@pbryan

@pbryan mypy will enable recursive aliases by default in the next release: python/mypy#13516

@wbolster

With the newest versions of Python (3.11.0) and MyPy (0.991), it looks like the one liner proposed by @wbolster works without any additional flags even for complex JSONS.

Json = dict[str, 'Json'] | list['Json'] | str | int | float | bool | None

Examples:

example_1: Json = None

mixed-type object

null = None
false = False
true = True

# https://www.appsloveworld.com/download-sample-json-file-with-multiple-records
example_2: Json = {
        "id": 4051,
        "name": "manoj",
        "email": "manoj@gmail.com",
        "password": "Test@123",
        "about": null,
        "token": "7f471974-ae46-4ac0-a882-1980c300c4d6",
        "country": null,
        "location": null,
        "lng": 0,
        "lat": 0,
        "dob": null,
        "gender": 0,
        "userType": 1,
        "userStatus": 1,
        "profilePicture": "Images/9b291404-bc2e-4806-88c5-08d29e65a5ad.png",
        "coverPicture": "Images/44af97d9-b8c9-4ec1-a099-010671db25b7.png",
        "enablefollowme": false,
        "sendmenotifications": false,
        "sendTextmessages": false,
        "enabletagging": false,
        "createdAt": "2020-01-01T11:13:27.1107739",
        "updatedAt": "2020-01-02T09:16:49.284864",
        "livelng": 77.389849,
        "livelat": 28.6282231,
        "liveLocation": "Unnamed Road, Chhijarsi, Sector 63, Noida, Uttar Pradesh 201307, India",
        "creditBalance": 127,
        "myCash": 0
    }

nested lists and dictionaries of strings with a slice hidden to throw off the type checker

# https://grabthiscode.com/javascript/complex-json-example
example_3: Json = {
"problems": [{
    "Diabetes":[{
        "medications":[{
            "medicationsClasses":[{
                "className":[{
                    "associatedDrug":[{
                        "name":"asprin",
                        "dose":"",
                        "strength":"500 mg"
                    }],
                    "associatedDrug#2":[{
                        "name":"somethingElse",
                        "dose":"",
                        "strength":"500 mg"
                    }]
                }],
                "className2":[{
                    "associatedDrug":[{
                        "name":"asprin",
                        "dose":"",
                        "strength":"500 mg"
                    }],
                    "associatedDrug#2":[{
                        "name":"somethingElse",
                        "dose":"",
                        "strength":"500 mg",
                        "invalid": slice(1, 45)
                    }]
                }]
            }]
        }],
        "labs":[{
            "missing_field": "missing_value"
        }]
    }],
    "Asthma":[{}]
}]}

The type check of all examples together discovers the slice I hid in the last example:

mypy jsonjson.py

jsonjson.py:83: error: Dict entry 3 has incompatible type "str": "slice"; expected "str": "Union[Dict[str, Json], List[Json], str, int, float, None]"  [dict-item]
Found 1 error in 1 file (checked 1 source file)

I'm not sure what MyPy does when data passes through several functions..

All major type checkers now support recursive type aliases by default, so this should largely work:

JSON: TypeAlias = dict[str, "JSON"] | list["JSON"] | str | int | float | bool | None

Note that because dict is invariant, you might run into some issues e.g. with dict[str, str]. That is, you will not be able to pass something with type dict[str, str] to a function that takes a JSON

For such use cases you can use cast, and if you don't need mutability, something like the following might work:

JSON_ro: TypeAlias = Mapping[str, "JSON_ro"] | Sequence["JSON_ro"] | str | int | float | bool | None

Although this can sometimes result in false negatives, for instance, see python/mypy#13786 (comment)

As always, if you return a union type, your callers will have to assert the structure of their JSON in order to type check. If this an issue, return e.g. dict[str, Any] or Any or cast or type-ignore or just bite the bullet and assert.

Given that it is trivial to define these aliases, that as you can see they're not necessarily one-size-fits-all, and that explicit is better than implicit, and that they don't need any type checker special casing, I am not in favour of adding these to CPython's typing.py. However if you like importing stuff, we may ship this and some other useful type definitions in a PyPI package at some point; there also appears to be https://github.com/kevinheavey/jsonalias. If you have opinions about this, please see python/typing_extensions#6.

If you have further questions, please ask at https://github.com/python/typing/discussions
If you encounter issues with specific type checkers, please report to their issue trackers
If you have proposals for type system features that better model some related use case (for example AnyOf - Union for return types #566 ), please open a new issue at https://github.com/python/typing/issues or post to the typing-sig mailing list
If you want to ask about python/typing_extensions#6, please ask about it on that issue
If you have something to say and you're not sure where to post, please fallback to https://github.com/python/typing/discussions

In order to keep this message visible, I may lock this thread. Please strongly consider posting follow-ups at one of the above locations.

Thank you to everyone who contributed support for recursive type aliases to type checkers — cheers and happy typing!

Enhance all dict type annotations

refactor: Improve JsonSerializable type to be recursive (#745)

gjson: type public APIs with a JSON type alias

Navigation Menu

Repository navigation

Define a JSON type #182

72 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Define a JSON type #182

Description

Activity

gvanrossum commented on Feb 19, 2016

brettcannon commented on Feb 19, 2016

gvanrossum commented on Feb 20, 2016

brettcannon commented on Feb 20, 2016

gvanrossum commented on Feb 21, 2016

gvanrossum commented on Feb 21, 2016

brettcannon commented on Feb 21, 2016

gvanrossum commented on Mar 18, 2016

JukkaL commented on Mar 18, 2016

gvanrossum commented on Mar 18, 2016

brettcannon commented on Mar 18, 2016

gvanrossum commented on Mar 21, 2016

72 remaining items

ssbarnea commented on Aug 26, 2022

AlexWaygood commented on Aug 26, 2022

gvanrossum commented on Aug 26, 2022

wbolster commented on Sep 27, 2022

kevinheavey commented on Oct 28, 2022

pbryan commented on Oct 29, 2022

antonagestam commented on Oct 29, 2022

joooeey commented on Nov 19, 2022

hauntsaninja commented on Nov 19, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions