Repository navigation
Serialize File Content #574
Description
Activity
Just had a thought.
We could possibly leave file serialization to HTTP, and handle things on the backend.
For example, adding a HTTP endpoint
# Uploads are given a unique UUID path to route to path("_idom/upload/{uuid}")
Not really sure how we'd be able to tell the WS side of things when an upload is a complete though...
If we do go down this road of serializing via websockets, it's going to be extremely important to stream the file to disk rather than buffer it in memory. Otherwise, this would be a DDOS attack vector.
Ideally, "file stuff" should be handled by individual frameworks... but that design of that might not be technologically feasible.
PyZMQ might be a decent option to communicate between separate ASGI processes.
The initial implementation should probably use form elements to upload files over HTTP. We can overwrite the default
onSubmitbehavior for forms to submit without a redirect. Unfortunately though, I don't think we can supply a built-in solution that will allow for inter-process communication - that will have to be supplied on a case-by-case basis depending on the user's deployment.With that said, I have some thoughts on how this could be accomplished via the websocket in a more natural way, but it would require some pretty deep changes to how we send messages between the client and the server. Right now, there is no real messaging protocol - the server always sends layout updates and the client always sends layout events. We'd need to change this in order to allow the client and server to send different types of messages to each other. For example, all messages between the client and server would probably take the form:
{ "type": str, # some unique name "version": str, # allow client and server to sort out compatibility ... # any other data }- modified the milestones: ,
on Jan 29, 2023 - removed this from the milestone
on Feb 21, 2023 We are likely going to develop two file upload handlers.
For small files, we stream the file directly to memory.
For large files, we stream the file to disk (buffered in memory).- Client begins sending file contents to the server
- Server reassembles the file contents into a buffer
- In the background, the server empties that buffer into a temp file onto the disk, in order to not use up all the system RAM
In order to support this without having this become an attack vector, we should implement a few safeguards:
MAX_BUFFER_SIZE: The maximum amount of bytes to buffer within memory before throttling responsesMAX_FILE_SIZE: The maximum size of a file that can be saved to the diskUPLOAD_DIR: Directory where files get saved- If the transfer ends pre-maturely, the buffer and disk file are deleted
- Look at Django's upload handlers and see if there are other security steps that need to be taken
django.core.files.uploadhandler.MemoryFileUploadHandlerdjango.core.files.uploadhandler.TemporaryFileUploadHandler
We might want to transfer this issue to the
reactpy-routerrepo
Right now we don't serialize file content. Just metadata. Along with this we'll want to allow users to configure the max message size to help users protect against large file upload attacks.
See this comment for implementation details.