Design Multipart helper to stream by default #114

Activity

1. lovelydinosaur commented on Jul 10, 2019

   

   lovelydinosaur

   on Jul 10, 2019

   Member

   More actions

   Yup, defo.

   One subtlety here is that right now we either support the request content being bytes or a byte iterator. The iterator case can't support body-preserving redirects. So we've actually got a precursor to this issue of supporting a rewindable/seekable type. (To allow rewinding back to the start of the upload.)
2. sethmlarson commented on Jul 10, 2019

   

   sethmlarson

   on Jul 10, 2019

   Last edited by sethmlarson

   ContributorAuthor

   More actions

   Yeah there's some bits and pieces I can lift from urllib3 and some unmerged branches for that specific case. Effectively we add support for passing a seekable object as data and then we implement an file-like interface for the multipart data and internally can rewind the individual file objects when we receive a .seek(0, 0).

   Would something like this be effective?
3. lovelydinosaur

   added

   enhancementNew feature or requestNew feature or request

   on Nov 30, 2019
4. florimondmanca commented on Mar 5, 2020

   

   florimondmanca

   on Mar 5, 2020

   Contributor

   More actions

   Any updated context on this given where we're at now? Is this still an issue/is it easier to tackle, and how?
5. lovelydinosaur commented on Mar 9, 2020

   

   lovelydinosaur

   on Mar 9, 2020

   Member

   More actions

   Ah yes. Streaming multipart would be a great feature for us to add now, in particular because it doesn't add any new API surface area at all, and the content streams are designed to allow us to support it. 😃

   Right now we generate the multipart data on __init__...

   httpx/httpx/_content_streams.py

   Lines 272 to 288 in 6e40c1b

   	def __init__(self, data: dict, files: dict, boundary: bytes = None) -> None:
   	body = BytesIO()
   	if boundary is None:
   	boundary = binascii.hexlify(os.urandom(16))
   	for field in self.iter_fields(data, files):
   	body.write(b"--%s\r\n" % boundary)
   	body.write(field.render_headers())
   	body.write(field.render_data())
   	body.write(b"\r\n")
   	body.write(b"--%s--\r\n" % boundary)
   	self.content_type = "multipart/form-data; boundary=%s" % boundary.decode(
   	"ascii"
   	)
   	self.body = body.getvalue()

   Instead we should be moving that into __iter__...

   httpx/httpx/_content_streams.py

   Lines 308 to 309 in 6e40c1b

   	def __iter__(self) -> typing.Iterator[bytes]:
   	yield self.body

   ...and __aiter__...

   httpx/httpx/_content_streams.py

   Lines 311 to 312 in 6e40c1b

   	async def __aiter__(self) -> typing.AsyncIterator[bytes]:
   	yield self.body

   Additionally we'd want FileField.render_data to read and return chunks-at-a-time, and be called multiple times, rather than being called once and returning the entire file content.
6. florimondmanca commented on Mar 9, 2020

   

   florimondmanca

   on Mar 9, 2020

   Contributor

   More actions

   Okay — looking into it. :-) The bits around estimating the content length without reading the entire data are interesting!
7. florimondmanca

   closed this as completedin #857on Apr 10, 2020