Disallow x["foo"] for NotRequired TypedDict access

Feature

After PEP 655 was implemented, mypy now checks initialization of typed dicts. Access to these dicts however is not checked for required/optional.

Pitch

We have lots of code which makes use of TypedDict with optional parameters, and one of the most common coding errors is to call dict_value["param"]. Because these are not required keys, a sound type checker should disallow this access and instead require dict_value.get("param").

This problem was mentioned near the end of #2632 but no subsequent ticket was filed.

I intend to have a PR for this up in the next few minutes.

Disallow direct item access of NotRequired TypedDict properties: thes…

I just ran across this problem as well. I wouldn't have realized that this was something that mypy wasn't identifying if Pylance hadn't properly identified it with an error message of the form:

"name" is not a required key in "MyTypedDict", so access may result in runtime exception.

I thought that I was perhaps just not configuring mypy correctly for it to identify the problem, but I now see that it's simply not something that mypy identifies yet.

@chuckwondo

@chuckwondo Pylance won't identify erroneous access to optional parameters for my typed dicts, did you have to configure it to highlight such issues?

@Kamforka

@Kamforka, in my VS Code settings, I set "Python > Analysis: Type Checking Mode" from its default value of "off" to "basic"

After doing that, such erroneous access is highlighted.

@chuckwondo

@chuckwondo indeed that was the missing setting!
Thanks!

1. Disallow direct item access of NotRequired TypedDict properties

@bsmedberg-xometry

@bsmedberg-xometry did start a PR proposing an implementation for this issue. However it requires more attention to get it to a place that doesn't mark a lot of errors in reasonable-looking existing code. Additional help from other followers of this issue is requested.

To paraphrase from the PR thread (plus integrating some comments from this issue), an implementation should satisfy:

class A(TypedDict):
    t: NotRequired["str"]

def index_a_t_1(a: A) -> str:
    # ERROR: TypedDict "A" key "t" is not required and might not be present.
    return a["t"]

def index_a_t_2(a: A) -> str:
    assert "t" in a
    # OK, because of type narrowing by: "t" in a
    return a["t"]

def index_a_t_3(a: A) -> str:
    if "t" in a:
        # OK, because of type narrowing by: "t" in a
        return a["t"]
    return "unknown"

def del_a_t_1(a: A) -> None:
    # ERROR: TypedDict "A" key "t" is not required and might not be present.
    del a["t"]

def del_a_t_2(a: A) -> None:
    assert "t" in a
    # OK. Is OK to del a NotRequired key iff type narrowed to Required
    del a["t"]

def get_a_t(a: A) -> str:
    # OK. Always OK to use get(K) or get(K, V) on a NotRequired key
    return a.get("t", "unknown")

def pop_a_t_1(a: A) -> None:
    # OK. Always OK to pop(K, V) a NotRequired key
    pop a("t", None)

def pop_a_t_2(a: A) -> None:
    assert "t" in a
    # OK. Always OK to pop(K, V) a NotRequired key, even if type narrowed to Required by: "t" in a
    pop a("t", None)

# pop(K) behaves in same way as del[K], even though pop(K, V) has different rules above

Edit: Did revise del cases above based on feedback from @JelleZijlstra below. Did also add separate pop cases.

def del_a_t_1(a: A) -> None:
    # OK. Always OK to del a NotRequired key
    del a["t"]

I don't think this should be OK, since del throws an error if the key is not present. Users should use a.pop("t", None) instead.

@JelleZijlstra

I don't think this should be OK, since del throws an error if the key is not present.

@JelleZijlstra Agreed. I've revised the test cases in the comment above to reflect your feedback.

We just ran across this problem as well. In our case, TypedDicts are generated code from schema definitions.

Therefore we are considering a workaround where, instead of a single typeddict, we generate two typeddicts and combine them via a Union:

class MyDict(TypedDict, total=False):
    foo_required: Required[int]
    foo_optional: NotRequired[int]


class MinimalDict(TypedDict, total=False):
    foo_required: Required[int]

MyDict2 = Union[MyDict, MinimalDict]


# mypy (incorrectly) thinks this code is correct
def x(d: MyDict):
    d['foo_optional']
    d['foo_required']

# mypy (correctly) errors because `foo_optional` may not exist
def y(d: MyDict2):
    d['foo_optional']
    d['foo_required']

this still has some edgecases where mypy will behave incorrectly, but I think it might catch the most obvious mistakes until this is fixed in mypy.

Navigation Menu

Repository navigation

Disallow x["foo"] for NotRequired TypedDict access #12094

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Disallow x["foo"] for NotRequired TypedDict access #12094

Description

Activity

chuckwondo commented on Feb 3, 2022

Kamforka commented on Feb 21, 2022

chuckwondo commented on Feb 21, 2022

Kamforka commented on Feb 22, 2022

davidfstr commented on Mar 18, 2023

JelleZijlstra commented on Mar 18, 2023

davidfstr commented on Mar 20, 2023

untitaker commented on May 26, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions