Repository navigation
Should we change PEP 484 to disable implicit Optional when default = None? #275
Description
Activity
I'm for this change. If I recall correctly, this is the only place where the PEP suggests typecheckers understand types to be something other than what's written as the type, so I think this change would be a nice simplification.
I've also started to notice this cause a bit of user confusion. See python/typeshed#500, for example, for all the arguments to builtin functions that were accidentally marked Optional due to this.
I have seen some confusions between optional arguments and optional types, and the convention in question amplifies such confusions. Therefore I am in favour of removing it.
OK, let's bring this up on python-dev once Python 3.6 beta 1 is released
(i.e. after next weekend).On Sat, Sep 3, 2016 at 7:32 AM, Ivan Levkivskyi notifications@github.com
wrote:I have seen some confusions between optional arguments and optional types,
and the convention in question amplifies such confusions. Therefore I am in
favour of removing it.—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#275 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACwrMpUgUvnteOgZfJvI0g_D50N3TOthks5qmYTugaJpZM4J0K87
.--Guido van Rossum (python.org/~guido)
IMO most of the confusion comes from the name being wrong. An argument is «optional» when the parameter has a default in python. An «optional» type is a type that you can choose to add or not (as in gradual typing). If we had
Nullable[T]instead ofOptionalmany of these confusions wold go away (and it's the name used mostly everywhere else, other thanMaybe Tin Haskell).Other than that, there is an inconsistency that may require a change (I just wanted to clarify that the confusions shouldn't be part of the argument)
Reacted by Sebastian Rittau, Max Kühn, Yunhui YZ Zhang, Edward Knight, 5j9, Max Marrone, wxgeo, Robert, Xavier Roynard and Antti Nilakari@dmoisset: I agree that
NullableorNonable(1 char shorter) orMaybe(3 chars shorter) would be better solutions thanOptional.On the other hand, even the short
Maybeis cumbersome. Nonable parameters are a frequent case and I believe the=Noneis sufficiently explicit to warrant introducing an additional PEP rule to cope with it, even if that rule has the character of an exception.Reacted by 5j9, Joel Niemelä and Antti NilakariI like that the
Optional[]can be omitted if aNonedefault is provided. It is less typing, reads naturally (to me) and is unambiguous.
If users want to useOptionalthey can sinceOptional[Optional[T]] == Optional[T].
As syntactic sugar goes, it seems quite sweet :)OOI, which quarters?
Reacted by Sebastian Rittau, Max Kühn, Mateen Ulhaq, Bar Harel, Chris Withers, wxgeo, Aoran Hu and Antti NilakariI haven't had enough experience with using mypy strict optional checking to decide whether the extra verbosity would become awkward. Maybe we should wait until we have a substantial codebase that can be type checked using strict optional checking.
Why do you use the word "strict"? That suggests that there is something "loose" about the syntactic sugar. There isn't; it is precisely defined.
Strict optional checking means that mypy treats
Noneas a regular type, and mypy enforces that you add guards likeif x: foo(x + 1)for values with anOptional[]type. Without strict optional checking,Noneis implicitly compatible with every type, kind of likenullin Java, and anOptional[X]type is equivalent to justX. It has nothing to do with syntax.So how is it relevant to default values for parameters making the type
Optional?The reason for adopting the syntactic sugar was the worry that otherwise type annotations would be too verbose and thus maybe also harder to read. I think that the main arguments for not having the syntactic sugar are consistency and conceptual simplicity.
If we have a codebase that uses
Optional[]consistently, which can be enforced by the strict optional mode of mypy, we could evaluate, at least for that codebase, whether the arguments hold water. We can get a realistic of estimate of the actual verbosity of annotating code with or without the syntactic sugar. We can perhaps also get a better understanding of whether the inconsistency is an actual problem for people who are working on the code.It may all boil down to personal preference, but having some empirical data could be useful for the discussion. Also, it doesn't seem to me that this is an urgent thing to resolve, as this doesn't require major
typingimplementation changes (thoughget_type_hintsmay be affected) and it's going to be easy for a static checker to support both modes via a configuration option. I don't see a need to rush the decision, and we can first wait for tools that can perform useful "strict optional" checking to be built.Whether it is easier to read is, as you say, personal preference, which is rarely influenced by empirical evidence 😄
Happy to wait and see what you come up with.
@JukkaL I ma not sure what kinds of inconsistency you mean, but this one
x = None # type: int # here the type stays int, unlike for function params
will go away after PEP 526. I think there it is better to assume implicit
Optionalor not for both of thesedef f(x: int = None): ... x: int = None
but not separately for each of these.
x = None # type: intAh, that thing. I think that should never have been added. It is just self-contradictory.
Reacted by Higor Carmanini and 5j9Reacted by wxgeo and Aoran Hu@ilevkivskyi There's still the question about annotating instance variables with mutable types in the class body in pre-3.6 code. This is what mypy users can write today:
class A: x = None # type: List[int] # Can't use x = [] (list is mutable) and don't want Optional def __init__(self) -> None: self.x = []This is a common use case and I think we need to have a reasonable way to expressing this without PEP 526 syntax.
33 remaining items
I assume we will keep a flag in mypy, but will just invert the default. Right?
I assume we will keep a flag in mypy, but will just invert the default. Right?
Yes we should do that.
- added a commit that references this issue
on Jul 10, 2018 Closed by python/peps#689.
What if I'm annotating function that does not accept
Nonebut has it as a marker of missed argument? E.g.:def foo(p: int = None): if p is None: print("p not specified")Should I use some another value to achieve this? If so, should it have type
intor other one? E.g.:class _MISSING: pass def f(p: int = _MISSING): ...I'd say it's not desirable to put a default value if you consider that default value is not an acceptable value. In this case, just don't put a default value. If you don't pass anything, mypy will tell you and you'll get a type error, and if you explicitely pass None, mypy will tell you it's wrong.
def foo(p: int): # in the function body, don't try to defend against p being not an int ...
What about other cases such as
dataclasses.field? Default value fordefaultanddefault_factoryhas incompatible type. See stub and specification.- added a commit that references this issue
on Jun 17, 2019 @ewjoachim there are these different scenarios:
# None is not one of possible values, func1 can't be called with explicit `param = None` def func1(param: Class = None): param = param or Class("somethin") # None is one of the possible values, funct2 can be called with explicit `param = None` def func2(param: Optional[Class] = None): pass # None is one of the possible values, funct3 requires param and can be called with explicit `param = None` def func3(param: Optional[Class]): pass # None is one of the possible values, funct3 can be called with `param = None` def func4(param: Optional[Class] = obj): pass
i.e. default values has nothing to do with type annotation i.e. public contract of a function
Not sure I get your point, and I don't agree with "default values has nothing to do with type annotation":
This function has wrong annotations which we can tell BECAUSE the annotations and the default value are clashing:
def func(param: int = "hello"): pass
I was answering to:
What if I'm annotating function that does not accept
Nonebut has it as a marker of missed argument? E.g.:def foo(p: int = None): if p is None: print("p not specified")
and I understood "does not accept
None" as "if you call the function with None (or without an argument), the call is going to be rejected. And I stand by my (... 2-year-old-)opinion: if a function is written to fail if an argument is missing, it's best to not have that argument have a default value.If you want a default value and you want to be able to detect that the default value was used, either use
: Optional[...] = Noneif None doesn't already have another meaning or do something like:missing = object() def func(a: Any = missing): if a is missing: print("default")
or
class Missing: pass missing = Missing() def func(json: Union[None, list, dict, str, int, float, Missing] = missing): if json is missing: print("default")
Maybe I missed an important point ?
Reacted by 5j9, Chris Withers and AJP / James Phillips- added a commit that references this issue
on Dec 20, 2022 Instead of treating this as a special case, it could also be understood as implicit type widening, where the default value is allowed to widen the type, e.g. from
strtostr | None. This works only with type inference, but PEP 484 does recommend:Type checkers are expected to attempt to infer as much information as necessary.
So arguably, it's not really a special case, but a general policy of widening the type based on a default when the typing tools has inferred the type.
This would also work for the case of a marker object, i.e.
missingin @ewjoachim's example.While I can see some strange pitfalls such as
a: int = "foo"this should perhaps not win over the arguably clean syntax offered bya: str = None(assuming that the general policy of implicit type widening would get adopted).Reacted by Sebastian Rittau
CC: @ddfisher @JukkaL @markshannon
PEP 484 currently says:
This was intended as saving some typing in a common case, but I've received strong feedback from some quarters that this is not consistent and a bad idea. There are other places where None is allowed but none of them automatically add Optional (to the contrary).
So far it hasn't mattered much for mypy users because mypy doesn't have Optional support, but that's soon going to change (the
--strict-optionalflag is becoming more reliable) so if we're going to change this, now would be a good time. Thoughts? If we don't change this soon it will probably be too late.