Repository navigation
Storage: Fix storage parallel operations copying objects - #8483
HemangChothani wants to merge 3 commits into
Conversation
googlebot
added
the
cla: yes
AVaksman
added
the
kokoro:force-run
yoshi-kokoro
removed
the
kokoro:force-run
frankyn
left a comment
frankyn
left a comment
There was a problem hiding this comment.
Thanks @HemangChothani.
Added a few requests.
One additional nit, in #4684, @houglum mentioned that gsutil uses separate processes with multiple threads to perform the parallel upload.
It'd be ideal to follow along with gsutil's implementation given they both use Python.
Defaults used for processes and threads is defined here: https://github.com/GoogleCloudPlatform/gsutil/blob/0f20832f15adb83047c3c8e74291de7c7d4744e8/gslib/commands/config.py#L362-L367
Additionally, could you perform a time benchmark on using this new method versus gsutil? It doesn't have to be elaborate but to give us a ballpark of how this helper method performs compared to gsutil -m cp.
|
|
||
| _DEFAULT_CHUNKSIZE = 104857600 # 1024 * 1024 B * 100 = 100 MB | ||
| _MAX_MULTIPART_SIZE = 8388608 # 8 MB | ||
| _MAX_THREAD_LIMIT = 10 |
There was a problem hiding this comment.
How is this number determined?
| client, | ||
| predefined_acl, | ||
| ) | ||
| time.sleep(0.3) |
There was a problem hiding this comment.
Could you add more information for using time.sleep(0.3)?
| filepath_list.append(os.path.join(directory, filename)) | ||
| return filepath_list | ||
|
|
||
| def upload_parallel( |
There was a problem hiding this comment.
To follow along with Client.list_blobs(), upload_parallel should be added to Client instead. The current design is a bit confusing as a method under the Blob class given it's used to upload multiple blobs.
|
@HemangChothani, I'm closing this issue after internal discussions. This a feature request and it's not within scope of what should be worked on at this time. We have a priority on fixing reliability bugs in our clients before feature requests. Additionally, this feature in particular requires a design review that has not yet been prioritized. |
Fixes: #4684