Repository navigation
Implement os.putenv() with setenv() if available #83587
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2020-01-22.22:00:40.936>
created_at = <Date 2020-01-21.08:30:01.989>
labels = ['library', '3.9']
title = 'Implement os.putenv() with setenv() if available'
updated_at = <Date 2020-01-22.22:00:40.935>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2020-01-22.22:00:40.935>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-01-22.22:00:40.936>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2020-01-21.08:30:01.989>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 39406
keywords = ['patch']
message_count = 15.0
messages = ['360365', '360370', '360371', '360378', '360386', '360388', '360419', '360421', '360422', '360424', '360445', '360505', '360509', '360511', '360513']
nosy_count = 3.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'eryksun']
pr_nums = ['18095', '18106', '18126', '18128']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue39406'
versions = ['Python 3.9']
Activity
vstinner commentedon Jan 21, 2020
Currently, os.putenv() is always implemented with putenv(). The problem is that putenv(str) puts directly the string into the environment, the string is not copied. So Python has to keep track of this memory.
In Python 3.9, this string is now cleared at Python exit, without unsetting the environment variable which cause bpo-39395 crash.
I propose to implement os.putenv() with setenv() if available, which avoids bpo-39395 on platforms providing setenv().
vstinner commentedon Jan 21, 2020
setenv() is available on:
It is not available on Windows.
serhiy-storchaka commentedon Jan 21, 2020
Maybe use SetEnvironmentVariable() on Windows?
vstinner commentedon Jan 21, 2020
I didn't know this function. I updated PR 18095.
eryksun commentedon Jan 21, 2020
_wputenvkeeps Python's copy of the environment block in sync with CRT's copy, which in turn callsSetEnvironmentVariableWto sync with process environment block. The CRT's copy of the environment excludes and disallows variable names that start with "=". Because the interpreter initializesos.environfrom the CRT environment, Python has never included or allowed 'hidden' variables such as "=C:".Switching to calling
SetEnvironmentVariableWdirectly won't affectos.systemandos.spawnv[e], since the CRT callsCreateProcessWwithout overriding the process environment. However, have you considered the consequences for extension modules and embedding applications that use the CRT environment (e.g.getenv,_wgetenv) if the interpreter stops syncing with it?The concern that
[_w]putenvdoesn't copy the string does not apply to Windows.serhiy-storchaka commentedon Jan 21, 2020
This indeed a good argument to continue to use _wputenv. Thank you Eryk, you have saved us from making a mistake.
vstinner commentedon Jan 21, 2020
I created bpo-39413 "Implement os.unsetenv() on Windows" to prepare work on this issue. But Eryk raised the same concern about CRT: https://bugs.python.org/issue39413#msg360404
I wasn't aware of that :-/
My main motivation to use SetEnvironmentVariableW() on Windows is to avoid bpo-39395. Do you mean that Windows _putenv() is not affected by bpo-39395: Python can safely removes the string to _putenv() just after the call?
vstinner commentedon Jan 21, 2020
New changeset 623ed61 by Victor Stinner in branch 'master':
bpo-39406: Add PY_PUTENV_DICT macro to posixmodule.c (GH-18106)
623ed61
serhiy-storchaka commentedon Jan 21, 2020
There is also _wputenv_s which affects _spawn, _exec and system.
serhiy-storchaka commentedon Jan 21, 2020
Why posix_putenv_garbage was renamed to putenv_dict?
eryksun commentedon Jan 22, 2020
Windows ucrt copies the buffer that's passed to _[w]putenv. This makes it non-compliant with POSIX but easier to use in this case. Here's an example using ctypes:
>>> ucrt = ctypes.CDLL('ucrtbase') >>> buf = ctypes.create_unicode_buffer('spam=eggs') >>> ucrt._wputenv(buf) 0Directly modifying the buffer has no effect on the _wgetenv result:
>>> buf[5] = 'a' >>> ucrt._wgetenv.restype = ctypes.POINTER(ctypes.c_wchar) >>> ucrt._wgetenv('spam')[:4] 'eggs'Linux putenv complies with POSIX, so changing "eggs" to "aggs" in the buffer does affect the getenv result in Linux.
On the other hand, ucrt's [_w]getenv does not copy the environment string. For example:
>>> p2 = ucrt._wgetenv('spam') >>> p2[:4] 'oggs'[_w]getenv is not thread safe. Even though all calls that access the ucrt environment are internally synchronized on a common lock, accessing the *result* from [_w]getenv still needs to be synchronized with concurrent _[w]putenv calls in a multithreaded process. Better still, use [_w]getenv_s or _[w]dupenv_s, which returns a copy.
The documentation is perhaps misleading. When a variable is set with _[w]putenv[_s], it also sets the variable in the process environment block to stay in sync. This indirectly affects _spawn*, _exec* and system(). common_spawnv (in ucrt\exec\spawnv.cpp), which implements the latter functions, calls CreateProcess with lpEnvironment as NULL (i.e. inherit the current process environment), unless the caller passes a new environment (e.g. envp of spawnve).
vstinner commentedon Jan 22, 2020
New changeset 0852c7d by Victor Stinner in branch 'master':
bpo-39406: os.putenv() avoids putenv_dict on Windows (GH-18126)
0852c7d
vstinner commentedon Jan 22, 2020
New changeset b477d19 by Victor Stinner in branch 'master':
bpo-39406: Implement os.putenv() with setenv() if available (GH-18128)
b477d19
vstinner commentedon Jan 22, 2020
Serhiy:
In bpo-39413, I implemented os.unsetenv() on Windows using SetEnvironmentVariable(), but Eryk reported that SetEnvironmentVariable() does not update the CRT API (_environ, _wenviron, getenv, etc.). So I reverted my change. See bpo-39413 for the details.
Eryk:
Thank you for checking manually. I pushed commit 0852c7d to avoid putenv_dict on Windows.
vstinner commentedon Jan 22, 2020
The initial issue has been fixed, so I close the issue. Big thanks to Eryk Sun for your great help, thanks also Serhiy Storchaka.