Storage: Capture relevant headers to blob properties during download

@william-silversmith

Residual from googleapis/google-cloud-python#9003.

@william-silversmith notes that even with raw_download enabled, he is unable to detect the content_type of a downloaded blob without performing an additional reload request, which is prohibitive for his usecase at scale. E.g.:

blob = bucket.blob( key )
binary = blob.download_as_string(raw_download=True)
if blob.content_encoding == 'gzip':
    return gunzip(binary)
elif blob.content_encoding == 'br':
    return brotli.decompress(binary)
else:
    return binary

Potentially even...

if blob.content_type == 'application/json':
    return json.loads(binary.decode('utf8'))

Thank you Tres!

@william-silversmith

@william-silversmith The headers needed to do this task are buried several layers deep. I'm wondering if a better solution than trying to guess which ones to map onto the blob's properties, we might be better off designing for your usecase by supplying a response_callback argument to the Blob.download_* methods:

The callback would take the blob instance and a requests.Response object as its arguments.
For non-chunked downloads, the callback would be called once; chunked downloads would call it once per chunk.

An example callback implementation for your usecase might be something like:

def _update_blob_content_encoding(blob, response):
    if blob.content_encoding is None:  # latch
        blob.content_encoding = response.headers.get('Content-Encoding')

If we don't use a callback, then the relevant headers to start with would seem to be:

Cache-Control -> Blob.cache_control
Content-Disposition -> Blob.content_disposition
Content-Encoding -> Blob.content_encoding
Content-Length -> Blob.size
Content-Type -> Blob.content_type
Etag -> Blob.etag
X-Goog-Generation: Blob.generation
X-Goog-Hash: Blob.crc32c
X-Goog-Metageneration: Blob.metageneration
X-Goog-Storage-Class: Blob.storage_class

I almost never need more exotic headers than the ones you listed. If it's easier to only expose those headers, that's fine with me. If you want to go for a more general solution, I am fine with a (synchronous) callback. Thanks again!

Any word on when this might drop? Thanks for your help!

Hi Tres, I've made some changes to my own branch of this repo and they seem to work. I tried making a PR but the button was greyed out. Here's a link to the changes in case y'all are interested. In the mean time, I'll direct my repos to use the forked version.

master...seung-lab:master

Unfortunately, it seems PyPI won't let me use a direct reference to github. It would be extremely convenient if some version of this could get merged.

Figured out the PR submission issue.

feat(blob.py): auto-populate standard headers for non-chunked downloa…

Navigation Menu

Repository navigation

Storage: Capture relevant headers to blob properties during download #24

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Storage: Capture relevant headers to blob properties during download #24

Description

Activity

william-silversmith commented on Nov 13, 2019

tseaver commented on Nov 13, 2019

tseaver commented on Nov 13, 2019

william-silversmith commented on Nov 14, 2019

william-silversmith commented on Jan 27, 2020

william-silversmith commented on Jul 14, 2020

william-silversmith commented on Jul 15, 2020

william-silversmith commented on Jul 15, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions