Docs: note requirement to normalise unicode identifiers passed to globals() and locals()

@ericvsmith

BPO	42680
Nosy	@ericvsmith, @ezio-melotti, @stevendaprano, @outofculture

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-12-19.00:27:18.270>
labels = ['3.8', 'type-bug', '3.7', '3.9', 'expert-unicode']
title = 'unicode identifiers not accessible or assignable through globals()'
updated_at = <Date 2021-01-20.10:46:27.879>
user = 'https://github.com/outofculture'

bugs.python.org fields:

activity = <Date 2021-01-20.10:46:27.879>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode']
creation = <Date 2020-12-19.00:27:18.270>
creator = 'outofculture'
dependencies = []
files = []
hgrepos = []
issue_num = 42680
keywords = []
message_count = 5.0
messages = ['383336', '383338', '383340', '383348', '383462']
nosy_count = 4.0
nosy_names = ['eric.smith', 'ezio.melotti', 'steven.daprano', 'outofculture']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue42680'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9']

This behavior is best described by the code below:

>>> meow = 1
>>> 'meow' in globals()
True
>>> µmeow = 1e-6
>>> 'µmeow' in globals()
False
>>> globals()['woof'] = 1
>>> woof
1
>>> globals()['µwoof'] = 1e-6
>>> µwoof
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'μwoof' is not defined
>>> import sys
>>> sys.getdefaultencoding()
'utf-8'
>>> [(k, bytes(k, 'utf-8')) for k in globals()]
..., ('μmeow', b'\xce\xbcmeow'), ('µwoof', b'\xc2\xb5woof')]
>>> 'µ'.encode('utf-8')
b'\xc2\xb5'

Testing was done on linux and windows, variously using 3.6.12, 3.7.6, 3.8.6 and 3.9.0+.

Oh, I just gave a cursory using locals(), and the same misbehavior is present.

A workaround, for anyone needing to assign or access unicode globals, is to use exec, e.g. exec("µmeow = 1e-6", globals()).

I'm pretty sure this is not a bug, but is working as designed.

The interpreter normalises unicode identifiers, but key lookup in the dict does not.

Sorry, I don't have time right now to give a more detailed answer, but there are two distinct mu characters:

μ U+03BC
µ U+00B5

and my prediction is that the identifier is normalised to the first, the actual Greek mu, but you are looking up the second, the micro sign.

(I'll be able to give a longer response in a couple of hours, if still needed.)

Ah! So then the proper code for me would be e.g.:

>>> globals()[unicodedata.normalize("NFKC", "µmeow")]
1e-06

Yes, it's clear when I read https://www.python.org/dev/peps/pep-3131/ that the normalization is going to happen. Is it also worth adding a note in the documentation for globals() and locals()? That's the first place I looked to try to find out what was wrong.

I think documenting this with globals() and locals() is a good idea. That's the place this is most likely to trip someone up.

Updated issue title and labels to reflect the previous discussion.

Documented as a note in https://docs.python.org/3.15/reference/lexical_analysis.html#non-ascii-characters-in-names

I think that's the proper place, as that where Python does the normalization.
globals() itself is perfectly fine with non-normalized keys:

>>> ﬁx = False
>>> globals()['fix']
False
>>> globals()['ﬁx'] = True
>>> globals()['ﬁx']
True

Let's reopen if someone disagrees.

Navigation Menu

Repository navigation

Docs: note requirement to normalise unicode identifiers passed to globals() and locals() #86846

3 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Docs: note requirement to normalise unicode identifiers passed to globals() and locals() #86846

Description

Activity

outofculture commented on Dec 19, 2020

outofculture commented on Dec 19, 2020

stevendaprano commented on Dec 19, 2020

outofculture commented on Dec 19, 2020

ericvsmith commented on Dec 20, 2020

3 remaining items

ncoghlan commented on Aug 1, 2024

encukou commented on Nov 26, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions