Repository navigation
Clarify status of non-identifier unpacked keyword arguments #96397
Description
Activity
- linked a pull request that will close this issuegh-96397: Document that keywords in calls need not be identifiers #96393
on Aug 29, 2022 Docs referenced from the discussion: https://docs.python.org/3/reference/expressions.html#calls
Secondary issue that could be wrapped in this (from Guido):
positional-only arguments are still described as a CPython-only feature for builtins only.)
Note that the same issue shows up in a few other places, e.g.
setattr()and friends allow any string as the attribute name (unless a specific class restricts the name in their__setattr__()method).(There's also
obj.__dict__which usually returns a dict that does no type checking on keys, and similar forglobals()andlocals(). We could declare that those objects must support arbitrary strings but that limiting the key type to just strings would be acceptable.)I think there's two potentially separate issues:
- Strings that aren't valid identifiers (e.g. keywords, strings with spaces in them). I don't see any reason not to accept these as keyword args; if anything it would be more expensive for an implementation to have to check whether each kwarg is a valid identifier.
- Instances of subclasses of
str. Allowing this seems much riskier, since string subclasses could have all kinds of unexpected behaviors. It may be safer to specify that only instances of exactlystrare required to be accepted as kwargs.
Reacted by Gouvernathor and Jeff AllenUsing non-valid identifiers as attribute names via
__dict__, **kwargs, getattr(), setattr()is being done in live production code right now, at least in cPython (https://pypi.org/project/netCDF4/ and I'm sure others). I have no idea what other implementations are doing, but I suspect it's the same, at least with PyPy. Given that, regardless of original intent -- I think it's better to define this as a language feature rather than an implementation detail.As Guido pointed out, this is also relevant to
__dict__, setattr(), getattr(), locals, globals(anywhere else?) -- but I just looked through the docs, and I can't see where it would fit -- it's seems it belongs in the Language Reference, but I can't find where to put it -- anyone have any suggestions?I had suggested text something like:
"""
In all instances where a python name is specified/stored in a container or variable, the name must be a string, but may be a non-valid python identifier.Examples:
dict and **kwargs may not have non-string keys, but may have non-valid strings as identifiers (e.g. "this-name").
getattr() and setattr() will accept only strings, but the strings to do not have to be valid identifiers. (e.g. setattr(obj, "this-name", value)
"""But I have no idea where to put it in the docs.
- Instances of subclasses of
str. Allowing this seems much riskier, since string subclasses could have all kinds of unexpected behaviors. It may be safer to specify that only instances of exactlystrare required to be accepted as kwargs.
I agree we don't have to allow string subclasses. Right now CPython allows it, but the usage would appear niche. We can leave this unspecified if we want to.
- Instances of subclasses of
As Guido pointed out, this is also relevant to
__dict__, setattr(), getattr(), locals, globals(anywhere else?) -- but I just looked through the docs, and I can't see where it would fit -- it's seems it belongs in the Language Reference, but I can't find where to put it -- anyone have any suggestions?I had suggested text something like:
""" In all instances where a python name is specified/stored in a container or variable, the name must be a string, but may be a non-valid python identifier.
I found this too difficult to understand. What is a "python name" different from a "python identifier"? And what, every container of any kind?
I opted for documenting the behaviour in the specific context of keys in a mapping in a call. I offer to do the same for the name arguments in
getattr(),setattr(), maybe__dict__. Naturally one would want the same conventions to apply (which they seem to) that any string will do, but you can't then expect a parameter (in a function signature) or member (inobj.membernotation) to be possible with that name.getattr() and setattr() will accept only strings, but the strings to do not have to be valid identifiers. (e.g. setattr(obj, "this-name", value) """
But I have no idea where to put it in the docs.
No indeed. And I wouldn't know where to find it, which recommended in my mind putting consistent specialisations of the rule in the docs for call,
__dict__andgetattr().- Instances of subclasses of
str. Allowing this seems much riskier, since string subclasses could have all kinds of unexpected behaviors. It may be safer to specify that only instances of exactlystrare required to be accepted as kwargs.
I agree we don't have to allow string subclasses. Right now CPython allows it, but the usage would appear niche. We can leave this unspecified if we want to.
Yes, but can I suggest this might be important to standardise, one way or the other? In #94938 I broke CPython by violating the rules of hashable, but you can do some things with well-behaved sub-classes might be thought useful:
class C: pass class IdWithUnit(str): def __eq__(self, other): s = self.split()[0] return s.__eq__(other) def __hash__(self): s = self.split()[0] return s.__hash__() def f(a, **kwargs): return f'a={a}, kwargs={kwargs}' awu = IdWithUnit("a ft") assert(awu == "a") assert(str(awu) == "a ft") # An IdWithUnit functions as a keyword assignment assert(f(**{awu: 42}) == "a=42, kwargs={}") # Objects with __dict__ accept IdWithUnit as attribute names c = C() setattr(c, awu, 42) assert(getattr(c, awu) == 42) # The IdWithUnit attribute retains its full name assert(IdWithUnit in map(type, c.__dict__.keys())) assert("a ft" in map(str, c.__dict__.keys())) # Use of an IdWithUnit establishes an attribute with the plain name assert(c.a == 42)- Instances of subclasses of
I found this too difficult to understand. What is a "python name" different from a "python identifier"?
An "identifier" is what you can put in code:
https://docs.python.org/3/reference/lexical_analysis.html#identifiers
I think that's well defined.
A "name" is the thing we can use in **kwargs dict, or pass to get/setattr() or use as a key in a dict. I have no idea how to clearly define it -- maybe: "A key in a namespace mapping" -- but I can't say that's clear either.
And what, every container of any kind?
I used "container" to be generic -- in practice, it's usually a dict -- but that's certainly not part of the language definition -- maybe Mapping? But now that I think about it, it's an argument to setattr() -- not in a container at all :-(
I used those generic terms because it seems that this pops up in multiple places in the language, and currently (and I hope forever), the rule is consistent -- so it seemed better to define the consistent rule in one place, rather than, or in addition to, adding a note to the half a dozen places where it's relevant (and maybe miss some).
I'm getting way out my area of expertise here, but maybe the rules for valid identifiers only apply when parsing/evaluating code. Once the system is passed the parsing stage, they are simply strings, and any string will do. So maybe that could be the way to define this behavior?
An "identifier" is what you can put in code:
https://docs.python.org/3/reference/lexical_analysis.html#identifiers
I think that's well defined.Agreed. And I might say "in code without the quotes" because in the context of Python source, the scanner can pick identifiers out as lexemes.
A "name" is the thing we can use in **kwargs dict, or pass to get/setattr() or use as a key in a dict. I have no idea how to clearly define it -- maybe: "A key in a namespace mapping" -- but I can't say that's clear either.
Also agreed. ;)
I used "container" to be generic -- in practice, it's usually a dict -- but that's certainly not part of the language definition -- maybe Mapping? But now that I think about it, it's an argument to setattr() -- not in a container at all :-(
Ok, I think we're in agreement that while a single statement would be good, it is quite difficult to write and place. Being a bear of very little brain, I chose to start with the specific cases, even at risk of repetition. Maybe a common thread will emerge. It was only about 100 words for keywords. I don't think it's as much as half-a-dozen places.
I'll offer a separate PR documenting the range of names acceptable in
(get|set|del)attr. In fact, I currently think the dunder methods are the place to go into this, because that's where an object would constrain the available choices, butobj.__setattr__doesn't.Note that our current definition of attribute is "something you can address with a dot" so ... would pseudo-attribute be a good name for something you can't address with a dot but can with
getattr(), if the object lets you.I realised I had recently come across pseudo-attributes in a the
pandas.DataFrame, where columns and rows may be addressed by dot notation, if labeled by identifiers, and by index notation["my-name"]generally. On a quick check, these are accessible viagetattr(df, "my-name").6 remaining items
SC ruling: python/steering-council#142 (comment)
The feature is fine, and there’s a distinction between attributes in the object model and names in the syntax that’s IMO clear and meaningful.
- added a commit that references this issue
on Sep 22, 2022 - added 4 commits that reference this issue
on Sep 22, 2022 - added 4 commits that reference this issue
on Sep 29, 2022
In current CPython, unpacked keyword arguments must be strings but don’t have to be valid identifiers.
Discussion: https://discuss.python.org/t/supporting-or-not-invalid-identifiers-in-kwargs/17147
Messages from Guido (creator of the language) and core developers show that this behaviour is desired and will not be changed.
The issue is whether this is a CPython (stable) implementation detail, in which case docs could be clarified with the fix backported, or if it’s time to make it a guaranteed part of the language, binding for other implementations, starting from 3.11 or 3.12.
I have not found where the doc should be improved; arguments and parameters are discussed in glossary, library reference, etc. but I don’t know where is one complete, authoritative explanation of unpacked keyword params.