Repository navigation
Add support for streaming multipart/form-data #2249
Description
(After initial discussion in #2227.)
As far as I can see, there's no way to stream multipart/form-data uploads when the Content-Length is unknown, since HTTPX tries to get body size some way or another (last resort is to read it all in memory and count). Please, let me know if I overlooked something!
Use case would be to stream something whose size is unknown beforehand, like fetching and in parallel uploading a CSV export from a Postgres database, say.
HTTP/1.1 has chunked encoding for that. As you know, that streams the message itself, not the resource. This encoding is supported for "direct file uploads" in HTTPX, but not for multipart/form-data. This is strange, since chunked encoding is independent of Content-Type. Indeed, if both chunking and compression are in place, you first compress, and then you chunk the compressed bytes.
In HTTP/2 chunked encoding does not exist, byte size is per DATA frame. As you know, a message is allowed to have several. So, in this case the API would support uploading with multiple DATA frames, coming from a generator or iterator, for example.
Activity
lovelydinosaur commentedon Sep 30, 2022
Thanks @fxn... I think you've got this exactly right, yup.
We ought to handle this automatically, so that in the case where we can't determine the file-length (other than reading the file into memory) we use "Transfer-Encoding: chunked". (Our transport layer will deal with handling that correctly between HTTP/1.1 and HTTP/2)
There's a bit of a corner case that I've not yet fixed in the PR dealing with this, which is making sure that we raise an appropriate error if we try to handle a redirect, but can't rewind the file object.
lovelydinosaur commentedon Sep 30, 2022
Ah, that PR should also switch up the behaviour so that we can directly pass a bytes-generator as an argument.