Repository navigation
Test Discovery assumes wrong encoding for stdout #16705
Description
Environment data
Commit: c3f126316369cd610563c75b1b1725e0679adfb3
Date: 2021-07-14T22:10:15.214Z (1 day ago)
Electron: 12.0.13
Chrome: 89.0.4389.128
Node.js: 14.16.0
V8: 8.9.255.25-electron.0
OS: Windows_NT x64 10.0.19043
[OK] [Copy]
- VS Code version: Version: 1.58.2 (user setup)
- Extension version (available under the Extensions sidebar): v2021.6.944021595
- OS and version: Windows 10 21H1
- Python version (& distribution if applicable, e.g. Anaconda): 3.8.5
- Type of virtual environment used (N/A | venv | virtualenv | conda | ...): venv
- Relevant/affected Python packages and their versions:
- Relevant/affected Python-related VS Code extensions and their versions:
- Value of the
python.languageServersetting: "Pylance"
Expected behaviour
Test Discovery shows all the tests PYTEST discovers when run in Powershell via pytest.
Actual behaviour
Test Discovery fails with following stack trace:
Test Discovery failed:
Error: Traceback (most recent call last):
File "c:\Users\ich\.vscode\extensions\ms-python.python-2021.6.944021595\pythonFiles\testing_tools\run_adapter.py", line 22, in <module>
main(tool, cmd, subargs, toolargs)
File "c:\Users\ich\.vscode\extensions\ms-python.python-2021.6.944021595\pythonFiles\testing_tools\adapter\__main__.py", line 100, in main
parents, result = run(toolargs, **subargs)
File "c:\Users\ich\.vscode\extensions\ms-python.python-2021.6.944021595\pythonFiles\testing_tools\adapter\pytest\_discovery.py", line 30, in discover
ec = _pytest_main(pytestargs, [_plugin])
File "C:\Program Files (x86)\Python38-32\lib\contextlib.py", line 120, in __exit__
next(self.gen)
File "c:\Users\ich\.vscode\extensions\ms-python.python-2021.6.944021595\pythonFiles\testing_tools\adapter\util.py", line 251, in hide_stdio
yield sio
File "C:\Program Files (x86)\Python38-32\lib\contextlib.py", line 120, in __exit__
next(self.gen)
File "c:\Users\ich\.vscode\extensions\ms-python.python-2021.6.944021595\pythonFiles\testing_tools\adapter\util.py", line 239, in _temp_io
buff = tmp.read()
File "C:\Program Files (x86)\Python38-32\lib\tempfile.py", line 474, in func_wrapper
return func(*args, **kwargs)
File "C:\Program Files (x86)\Python38-32\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 61: character maps to <undefined>Steps to reproduce:
- I am not able to reproduce this as it works fine on my other 2 Windows machines. (One Windows 10 (slightly older build) and one being Win Server 2016
Misc
I asked about this on MS' Discord Server and received the following answer from Brett Cannon (@brettcannon):
It's a bug. Can you please open an issue at https://github.com/microsoft/vscode-python (specifically that code is making assumptions on the encoding of data sent to stdout that it probably shouldn't be making, which is then failing when we try to read the data back from the underlying file)?
Activity
brettcannon commentedon Jul 19, 2021
I think the bug here is there's no checking what
sys.stdoutwas using as an encoding before creating the temp file, and so there's a possible disconnect. I think the fix here is probably reading the encoding from the originalsys.stdoutand then specifying that for the temp file's encoding.botsman commentedon Nov 3, 2021
I am facing the same issue.
Though my case is a little bit complicated, the error and behavior seems to be the same:
I use pytest and in my project I have parametrized test class for a number of instances.
In order to initialize tests dynamically I use parametrized fixture, which initializes test only for instances, which have appropriate config json file.
The error happens for me in this initialization stage where my fixture tries to parse json files and fails with an error
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 187: ordinal not in range(128)The problem is that one of my json configs have letter
é.Interestingly, this error does not occur if I run tests from the console, only when I run tests via VSCode.
I could provide a sample project if it is necessary.
brettcannon commentedon Nov 3, 2021
We are about to embark on rewriting our test adapter code, so hopefully that will resolve this issue.
/cc Kim-Adeline Miguel (@kimadeline)
kimadeline commentedon Nov 3, 2021
#17242
TimoRJensen commentedon Jun 23, 2022
For everyone facing the same problem. It took me ages to realize the workaround for this was rather simple.
I "just" had to change the Windows Display Language from German to English (USA).
eleanorjboyd commentedon Jul 14, 2023
Hello! We have just finished our testing rewrite and are beginning the roll out to users. If you are able, it would be very helpful to know if your issue still exists on the rewrite! To try it yourself, you need to be on the most recent release of the Python extension and add this setting to your users
settings.json"python.experiments.optInto": ["pythonTestAdapter"].You can confirm you have the rewrite enabled by setting
"python.analysis.logLevel": "Trace",in your user settings then check forExperiment 'pythonTestAdapter'is active in your python logs.Let me know if the rewrite fixes your issue. Thanks!
eleanorjboyd commentedon Oct 30, 2023
closing as this should be fixed by changes made during #18346. Thanks