Repository navigation
ValueError: can't have unbuffered text I/O for io.open(1, 'wt', 0) #61606
Description
| BPO | 17404 |
|---|---|
| Nosy | @amauryfa, @pitrou, @rbtcollins, @benjaminp, @hynek, @vadmium, @serhiy-storchaka |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2013-03-12.14:42:40.965>
labels = ['3.10', 'type-bug', '3.8', '3.9', 'expert-IO']
title = "ValueError: can't have unbuffered text I/O for io.open(1,\t'wt', 0)"
updated_at = <Date 2020-11-06.22:51:34.480>
user = 'https://github.com/rbtcollins'bugs.python.org fields:
activity = <Date 2020-11-06.22:51:34.480>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['IO']
creation = <Date 2013-03-12.14:42:40.965>
creator = 'rbcollins'
dependencies = []
files = []
hgrepos = []
issue_num = 17404
keywords = []
message_count = 7.0
messages = ['184025', '184040', '184041', '184045', '184072', '184073', '332739']
nosy_count = 8.0
nosy_names = ['amaury.forgeotdarc', 'pitrou', 'rbcollins', 'benjamin.peterson', 'stutzbach', 'hynek', 'martin.panter', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue17404'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']
Activity
rbtcollins commentedon Mar 12, 2013
The io library rejects unbuffered text I/O, but this is not documented - and in fact can be manually worked around:
binstdout = io.open(sys.stdout.fileno(), 'wt', 0)
sys.stdout = io.TextIOWrapper(binstdout, encoding=sys.stdout.encoding)
will get a sys.stdout that is unbuffered.
Note that writing to a pipe doesn't really need to care about buffering anyway, if the user writes 300 characters, the codec will output a single block and the IO made will be one write:
This test script:
import sys
import io
stream = io.TextIOWrapper(io.open(sys.stdout.fileno(), 'wb', 0), encoding='utf8')
for r in range(10):
stream.write(u'\u1234'*500)
When run under strace -c does exactly 10 writes: so the performance is predictable. IMO it doesn't make sense to prohibit unbuffered text write I/O. readers may be another matter, but that doesn't suffer the same latency issues.
amauryfa commentedon Mar 12, 2013
The proposed workaround seems to work ("wb" instead of "wt"!), with the following restrictions:
it's not really unbuffered: the encoder has its own buffers (OK, in the stdlib only 'idna' encoding will retain data)
it won't work for reading: TextIOWrapper calls the read1() method, which is only defined by BufferedIO objects.
IMO this explains why it's not a supported combination in io.open().
rbtcollins commentedon Mar 12, 2013
Huh, I didn't realise idna would retain data! But that will still be within the TextIOWrapper itself, right?
And a stream opened 'wt' cannot be read from anyway, so the read1 limitation is irrelevant.
amauryfa commentedon Mar 12, 2013
Yes. And I just noticed that the _io module (the C version) will also buffer encoded bytes, up to f._CHUNK_SIZE.
On the other hand, TextIOWrapper is broken for buffering codecs, encode() is never called with final=True
>>> import io >>> buffer = io.BytesIO() # <-- not really buffered, right? >>> output = io.TextIOWrapper(buffer, encoding='idna') >>> output.write("www.somesite.com") 16 >>> print(buffer.getvalue()) b'' # <-- ok, _CHUNK_SIZE buffering >>> output.flush() >>> print(buffer.getvalue()) b'www.somesite.' # <-- the last word is missing! >>> output.close() >>> print(buffer.getvalue()) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: I/O operation on closed file.And it's even worse with python 2.7::
>>> import io as io >>> buffer = io.BytesIO() >>> output = io.TextIOWrapper(buffer, encoding='idna') >>> output.write("www.somesite.com") Traceback (most recent call last): File "<stdin>", line 3, in <module> TypeError: must be unicode, not strserhiy-storchaka commentedon Mar 13, 2013
Since 3.3 TextIOWrapper works with raw IO objects (bpo-12591).
Use write_through=True to disable this.
pitrou commentedon Mar 13, 2013
It won't be technically unbuffered, though.
[-]ValueError: can't have unbuffered text I/O for io.open(1, 'wt', 0)[/-][+]ValueError: can't have unbuffered text I/O for io.open(1, 'wt', 0)[/+]vadmium commentedon Dec 30, 2018
It is documented that buffering=0 is not supported in text mode. Look a handful of paragraphs down from <https://docs.python.org/release/3.7.2/library/functions.html#open\>:
“Pass 0 to switch buffering off (only allowed in binary mode)”
Amaury’s problem with the IDNA buffering encoder has now been reported separately in bpo-35611.
pythongh-85393: Handle partial .write in TextIOWrapper
pythongh-85393: Handle partial .write in TextIOWrapper
pythongh-85393: Handle partial .write in TextIOWrapper
pythongh-85393: Handle partial .write in TextIOWrapper
pythongh-85393: Handle partial .write in TextIOWrapper