This repository was archived by the owner on Mar 31, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 31, 2026. It is now read-only.
Documentation for the "transfer_manager" module #947
Description
Hello,
Thank you very much for the "transfer_manager" module that will probably greatly benefit us. However, I can't find the relevant documentation for this new module in the new documentation site. Do you know where it is?
Or if the documentation is not ready yet, is there an example of the usage of the new module somewhere?
Kind regards,
Anh
Activity
andrewsg commentedon Jan 3, 2023
Hi @tqa236, thanks very much for your interest in our new feature! The documentation isn't yet published, but examples with comments are available: https://github.com/googleapis/python-storage/blob/9998a5e1c9e9e8920c4d40e13e39095585de657a/samples/snippets/storage_transfer_manager.py
As we get ready to publish the documentation, this file will be broken up into smaller files due to a one-sample-per-file rule, but they will remain in the same directory.
Hope this helps. Please feel free to send us any feedback, positive or negative, about the feature here in issues or comments.
tqa236 commentedon Jan 3, 2023
Thank you very much for your prompt reply @andrewsg. I have successfully used the new module thanks to the example. I noticed that, unlike the original PR, the second PR only supports concurrent files download, and not concurrent chunked downloads of a single file anymore. Can I know why it's left out and will it be added back in the future?
andrewsg commentedon Jan 3, 2023
Unfortunately due to contention on the Python GIL, we found that there were no performance benefits for using the feature for chunks of a single file, versus a simple single-threaded download. In fact, the more threads, the slower the download.
Barring the use of gthead or some other monkey-patch solution that is appropriate for an end developer use case but not for a client library, it looks like we will need to implement a multiprocess solution to handle large files appropriately. I'll be working on prototypes for that approach this quarter. It will be a bit challenging to have it play nice with all of the things developers do with the library (recommended or not) so it may take some time.
I'm open to ideas from anyone reading this on approaches that might work, short of multiprocessing. In the meantime, the feature works well for large numbers of files just by using threads.
tqa236 commentedon Jan 4, 2023
Really appreciate the explanation @andrewsg
andrewsg commentedon Mar 7, 2023
@tqa236 FYI we're re-adding chunked downloading, with multiprocessing, in #1002
tqa236 commentedon Mar 7, 2023
thanks for the head ups @andrewsg, it seems like a fairly big PR. Do you have a rough estimate of when it'll be available?
andrewsg commentedon Mar 7, 2023
That depends on how the review goes. If there are no major issues, I'd be happy if it's out by the end of next week. I'm eager to get feedback from people using it!
andrewsg commentedon Mar 29, 2023
@tqa236 This is being released now as 2.8.0. Thank you for your patience.