This repository was archived by the owner on Mar 31, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 31, 2026. It is now read-only.
Storage: Consider externalising http configuration #34
Description
https://cloud.google.com/storage/quotas
self.storage_client: Client = storage.Client.from_service_account_json(config_json_path)
adapter: HTTPAdapter = self.storage_client._connection.http.adapters["https://"]
self.storage_client._connection.http.adapters["https://"] = HTTPAdapter(pool_connections=1000, pool_maxsize=1000, pool_block=adapter._pool_block)
Activity
dhermes commentedon Dec 7, 2017
@ravindranathakila I'm unclear what the action item is for us here?
ravindranathakila commentedon Dec 8, 2017
Do please consider properly externalising the http configuration so that it can be tweaked for high throughput systems. Especially when using storage.Client.from_service_account_json()
Right now, having to set _connection implies it isn't externalised. Unless the library is able to "auto-detect" the ideal configuration given large-files/low-throughput small-files/high-throughput average-files/average-throughput, I suggest you don't hide this.
[-]Google Cloud Storage Supports 1000 Writes Per Second By Default. Do you?[/-][+]Storage: Consider externalising http configuration[/+]sangramql commentedon Jun 20, 2019
@tseaver please suggest the expected implementation and approach.
tseaver commentedon Jun 20, 2019
@sangramql There isn't an "obvious" design for this one.
@ravindranathakila I'm not clear on what you are thinking here. The
_connectionhere doesn't hold any configuration which would affect how blob instances are uploaded / downloaded: it wraps an instance of an object supporting therequests.Sessioninterface (normally agoogle.auth.transport.requests.AuthorizedSessioninstance).The parameter which most affects "througput" is
chunk_size, which can be passed to aBlobconstructor (or set on it afterwards).tseaver commentedon Aug 17, 2020
@ravindranathakila Please feel free to follow up / reopen if you can provide more clarity.
ravindranathakila commentedon Aug 18, 2020
chore: librarian release pull request: 20260113T144551Z (#1704)