Repository navigation
FrameLocalsProxy is stricter than dict about what constitutes a match #120906
Description
Activity
Interesting issue. I don't have a definitive answer here but this is something we need to deal with, because we are getting some weird issues here:
import sys class MyString(str): pass def f(): x = 1 local = sys._getframe().f_locals local[MyString('x')] = 2 print(local.keys()) # ['x', 'local', 'x'] print(local) # {'x': 2, 'local': {...}} f()
Internally we check if the input key is an exact unicode, but we do utilize dict for certain features (
reprfor example). This causes an inconsistent behavior.My preferred solution is to enforce any key to be an exact unicode string. The reason for that is, unlike a generic
dict,FrameLocalsProxyshould be a direct proxy for local variables and keys other than unicode string does not quite make sense. However, I know I'm the more aggressive type when it comes to changes so I'll list some concerns as well.frame.f_localsfor an unoptimized frame (module/class) is a real dict and can take any type of keys. There might also be a usage to use a subclass as keys? If the original usage is shown to be helpful, that might tip the scale for me.I'm a bit worried about the can of worms when we allow subclasses of unicode strings - what if it overwrite the
__hash__and__eq__methods? Will it cause more troubles than the benefits?Just to explain the context I ran into it in:
The Cython compiler wraps most of it's strings into an
EncodedStringclass (for reasons that are possibly historic and which I don't quite understand). To clarify - that's a class used internally by the compiler, rather than a class used in Cython extension modules.We have an
inlinemethod of compilation, and here missing variables are taken from the surrounding frame. The frame comes from Python so the variable names are just a normalstr. However the names we're using to look them up are ourEncodedStringclass. These used to work fine when used to look up keys (since they just matched the equivalentstr) but now don't.It's easy enough to work around so I'm relaxed about the solution whatever you decide to do. I think what we were doing was mostly accidental and there wasn't a hidden special use-case behind it.
But for this use-case I'm just reading existing values from the
FrameLocalsProxyand not worried about storing subclassed strings. Obviously you need to consider both sides of it though.We need to discuss this further with @markshannon, but I think the desired behavior would be either
- Report an error when the key is not an exact unicode string
- or allow all string-like (subclasses of unicode) keys and access the actual fast locals even with those strings.
I just remembered that we already had that discussion when I was implementing PEP 667. The PEP suggests that we should allow arbitrary types.
Then the question would be what if the user gives a unicode-like key, do we try to access the fast variables with that key?
To avoid breaking mapping invariants, if a given key compares equal to the name of a local variable, it needs to be handled as a lookup of that variable name. Having a
PyUnicode_CheckExactguard on a string comparison fast path still makes sense, but the fallback needs to be a linear search using the abstract object comparison API rather than assuming the key is absent.But that's not the mapping protocol right? What if a key has a different hash, but the equal value? What if the key is
class Evil: def __eq__(self, other): return True
It's unhashable but only comparing equality will match it to an arbitrary local (the first one).
Are we going to implement the full mapping interface? Checking everything as dict/mapping does? The dark corner we left here might eat us in the future.
What if a key has a different hash, but the equal value
That's a violation of the hash protocol: objects that compare equal must have the same hash (or at least one must be unhashable).
The
Evilexample is incomplete, since__hash__is not inherited if__eq__is overridden.
But ifEvilhad__hash__explicitly set tostr.__hash__, it would be OK to rely on it and skip the call to__eq__.
IMO, it would also be OK to convert the argument to
PyUnicode, and do the lookup with the resulting temporary object.Reacted by Alyssa Coghlan1 remaining item
My worry about supporting a unicode subclass or an arbitrary object as a key is - it does not fit the mapping protocol. If an object overrides it's
__eq__but not__hash__, it can't be used as a key. If an object has a different hash to another one but also equal to that object, they will be different keys in the dict.None of the proposals above have this feature. So basically, we are inventing a very new mapping protocol specifically for
FrameLocalsproxyand it's not easy to explain - we don't have a simple enough rule.That's why I liked "unicode only" solution - that's a simple rule that works and easy to understand. It will cause some trouble, but we are already causing troubles.
We need to consider - what if a key is a unicode subclass, but has different
__eq__and/or__hash__method? It would be very difficult for us to behave exactly as a dict. What if it's not even a subclass of unicode?If we can't do that, I kind of like what @encukou suggested - we convert the non-exact-unicode keys to unicode first, for both read and write. That's a golden rule that everyone can follow, and that will solve many benign cases like this very issue.
I finally looked at the code, rather than going just by the conversation here. But I couldn't just look...
Please see PR #122309, as a suggestion for how to handle this.If an object overrides its
__eq__but not__hash__, it can't be used as a key. It's not hashable.
If an object has a different hash to another one but also equal to that object, that's a violation of the hash protocol, and dict can behave surprisingly (give weird results/exceptions, but not crash). We don't need to match that behavior.The hash protocol:
- limits the types (to hashable ones) and
- allows dicts to do an optimization: they can look at a small subset of entries. We don't need to use that optimization.
Keys that aren't unicode subclasses shouldn't be a problem.
Oh sorry I did not realize you posted in the issue. I don't think I'm getting notification from this issue.
Could you point to me the documentation about the hash protocol? I could not find it.
The data model docs in https://docs.python.org/3/reference/datamodel.html#object.__hash__ also state "The only required property is that objects which compare equal have the same hash value". However, they go into more detail about how to define
__hash__and__eq__to be consistent with each other, as well as how to indicate when subclasses of hashable parent classes are not themselves hashable.Classes that don't abide by those rules just straight up don't work properly, so the fact they won't misbehave as badly with locals proxy instances as they do with regular dicts isn't a problem we need to be concerned about.
This issue is all fixed now, right?
- added a commit that references this issue
on Jul 31, 2024 Indeed it is!
- moved this from Todo to Done in Release and Deferred blockers 🚫
on Aug 1, 2024
Bug report
Bug description:
In Python 3.12 and below this prints
True. In Python 3.13 this printFalse. I think it comes down to the check for exact unicode:cpython/Objects/frameobject.c
Line 112 in f4ddaa3
The change in behaviour isn't a huge problem so if it's intended then I won't spend waste any time complaining about it, but I do think it's worth confirming that it is intended/desired.
CPython versions tested on:
3.13
Operating systems tested on:
Linux
Linked PRs