Skip to content

fix(storage): validate CRC32C on HTTP downloads that were not decompressed - #14805

Merged
Dhriti07 merged 6 commits into
googleapis:mainfrom
Dhriti07:fix/storage-http-crc32c-validation
Oct 30, 2026
Merged

fix(storage): validate CRC32C on HTTP downloads that were not decompressed#14805
Dhriti07 merged 6 commits into
googleapis:mainfrom
Dhriti07:fix/storage-http-crc32c-validation

Conversation

@Dhriti07

@Dhriti07 Dhriti07 commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

GCS returns x-goog-stored-content-encoding and x-goog-stored-content-length on every object download, including objects that are not compressed:

HttpStorageRpcHasherHelper.isTranscoded() treated the presence of either header as a sign of decompressive transcoding. So shouldValidate() always returned false, and CRC32C validation was skipped for all HTTP reads (readAllBytes, downloadTo, reader).

In this pr we fix it by checking only the following:

  • the object is stored gzip-compressed (x-goog-stored-content-encoding: gzip) and was served decompressed (no Content-Encoding: gzip), or
  • the HTTP client decompressed the response (existing isDecompressedByClient).

Reference: Transcoding of gzip-compressed files. GCS only decompresses objects stored gzip-compressed with Content-Encoding: gzip.

Related fix: case-insensitive Content-Encoding in isDecompressedByClient

Dhriti Chopra
fix(storage): validate CRC32C on HTTP downloads that were not decompr…
a37b21b 
…essed

GCS returns x-goog-stored-content-encoding and x-goog-stored-content-length on
every object download, but HttpStorageRpcHasherHelper treated the presence of
either header as transcoding, so CRC32C validation was skipped for all HTTP
reads (readAllBytes, downloadTo, reader).

Only skip validation when the object is stored gzip-compressed and was served
decompressed, or when the HTTP client decompressed it.

Fixes googleapis#14773
@Dhriti07
Dhriti07 requested review from a team as code owners October 28, 2026 09:38
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Oct 28, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the transcoding detection logic in HttpStorageRpcHasherHelper to prevent skipping checksum validation on normal downloads. Previously, the presence of x-goog-stored-content-encoding or x-goog-stored-content-length headers (which GCS returns on all downloads) incorrectly flagged responses as transcoded. The updated logic specifically checks if the object was stored as gzip but served decompressed. Corresponding unit tests have been added and updated to ensure proper checksum validation behavior. I have no further feedback to provide as the changes are well-implemented and covered by tests.

Dhriti Chopra and others added 5 commits October 28, 2026 09:49
Dhriti Chopra
fix(storage): don't validate a partial CRC32C after a resumed reader(…
3af785f 
…) download

Now that CRC32C validation runs on HTTP downloads, a reader() download that is
resumed after a broken stream kept the hashing stream from the first (full
object) response. The resumed response is partial and is not validated, so at
EOF the hash of only the first part was compared with the full object checksum,
reporting a false ChecksumMismatchException.

Clear the hashing stream when the current response is not validated.
@Dhriti07
Merge branch 'main' into fix/storage-http-crc32c-validation
10389b0
Dhriti Chopra
test(storage): verify CRC32C validation is skipped for objects served…
baf3664 
… decompressed
Dhriti Chopra
fix(storage): match Content-Encoding case-insensitively when detectin…
2c13579 
…g client decompression

GCS echoes an object's Content-Encoding metadata as-is (e.g. "GZIP"), and the
HTTP client decompresses when the value matches gzip case-insensitively.
isDecompressedByClient used a case-sensitive check, so such responses were
validated against the stored object's checksum after being decompressed,
reporting a false ChecksumMismatchException.
@Dhriti07
Merge branch 'main' into fix/storage-http-crc32c-validation
3607603
@Dhriti07
Dhriti07 requested a review from ShreyasSinha October 29, 2026 07:07
@diwu-sf

diwu-sf commented Oct 29, 2026

Copy link
Copy Markdown

i'm going to close off #14768
this PR addresses our concern about missing data checksumming

@Dhriti07
Dhriti07 merged commit 61aa22f into googleapis:main Oct 30, 2026
203 checks passed
@release-please release-please Bot mentioned this pull request Nov 2, 2026
Merged
jinseopkim0 pushed a commit that referenced this pull request Nov 3, 2026
@release-please
chore(main): release 1.95.0 (#14898)
405578f 
🤖 I have created a release *beep* *boop*
---


<details><summary>1.95.0</summary>

##
[1.95.0](v1.94.0...v1.95.0)
(2026-11-02)


### Features

* **bigquery-jdbc:** expose BigQueryResultSetMetadata and getField(int)
([#14761](#14761))
([f01245a](f01245a))
* **gax:** align resumable upload chunk size to server granularity and
surface effective chunk size
([#14670](#14670))
([6cbb096](6cbb096))
* **google/cloud/designcenter/v1:** add designcenter
([#14823](#14823))
([01d27dc](01d27dc))
* **google/cloud/observability/v1:** add observability
([#14722](#14722))
([624f28e](624f28e))
* **oauth2:** support certificate-bound OAuth2 tokens and mTLS rotation
for Workload Identity Federation
([#14781](#14781))
([dcb29b0](dcb29b0))
* **storage:** add bucket IP filtering samples and tests
([#14328](#14328))
([cb2b726](cb2b726))
* **storage:** add support for DirectPath over Interconnect
([#14006](#14006))
([ad0eb94](ad0eb94))
* **storage:** OpenTelemetry client metrics configuration and options
([#14484](#14484))
([7c29e48](7c29e48))
* **storage:** OpenTelemetry client metrics instrument registry and
views
([#14398](#14398))
([60233b7](60233b7))


### Bug Fixes

* add user agent to session path
([#14835](#14835))
([a653638](a653638))
* **auth:** align mTLS transport and bound token certificate startup
polling and well-known discovery
([#14748](#14748))
([0bae268](0bae268))
* **bigquery-jdbc:** expand handling to OOM errors
([#14759](#14759))
([7d4b921](7d4b921))
* **bigquery-jdbc:** fix naive fallback of parameter count
([#14779](#14779))
([c3e4a5b](c3e4a5b))
* **bigquery-jdbc:** format RANGE bounds consistently across REST and
Arrow
([#14807](#14807))
([2108b8e](2108b8e))
* **bigquery-jdbc:** parse scientific-notation epoch TIMESTAMPs
([#14777](#14777))
([71a31b1](71a31b1))
* **bigquery-jdbc:** preserve pre-1582 and pre-standard-time
DATE/DATETIME digits
([#14787](#14787))
([3dc394f](3dc394f))
* **bigtable:** avoid mutating Query in-place in createPointReadCallable
([#14797](#14797))
([1b1c6a7](1b1c6a7))
* **bigtable:** check isOk and getRejected first in
RetryingVRpc.shouldRetry
([#14795](#14795))
([d821bd3](d821bd3))
* **bigtable:** check remaining deadline on client-initiated retries in
RetryingVRpc
([#14796](#14796))
([a04e24c](a04e24c))
* **bigtable:** Fix empty RowResult semantics in TestProxy.
([#14810](#14810))
([1d2fb9a](1d2fb9a))
* **bigtable:** include reversed in Query equals, hashCode, and toString
([#14799](#14799))
([fe9232b](fe9232b))
* **bigtable:** pass cell value size to RowBuilder.startCell in
RowBuilderShim
([#14800](#14800))
([cfa7949](cfa7949))
* **bigtable:** reject empty split points in RowSetUtil and clarify
QueryPaginator Javadoc
([#14811](#14811))
([1bc507f](1bc507f))
* **bigtable:** remove legacy PollingConfiguration migration in
ClientConfigurationManager
([#14822](#14822))
([95ead45](95ead45))
* **bigtable:** resolve session diversion through a single shared rule
([#14738](#14738))
([00f7bea](00f7bea))
* **bigtable:** respect query limit smaller than pageSize and avoid
mutating Query in QueryPaginator
([#14821](#14821))
([cecd09e](cecd09e))
* **firestore:** graduate pipeline search from preview to GA
([#14741](#14741))
([a993026](a993026))
* **gax:** do not treat a SocketTimeoutException as a thread interrupt
([#14778](#14778))
([7b91805](7b91805))
* **jdbc:** populate `ResultSetMetaData` on empty `DatabaseMetaData`
resultsets
([#14809](#14809))
([4c6a7e3](4c6a7e3))
* **pubsub:** make publish hedging robust to clock jumps, queue errors,
and shutdown
([#14860](#14860))
([9b9e2a5](9b9e2a5))
* **spanner:** attach routing hints to read-write transaction requests
([#14786](#14786))
([54e0585](54e0585))
* **spanner:** cancel queries promptly on close and synchronize error
handling
([#14887](#14887))
([323a8f7](323a8f7))
* **spanner:** do not cache retryable errors in AbstractLazyInitializer
([#14752](#14752))
([6a9ac76](6a9ac76))
* **spanner:** restore setCall overload for stream test compatibility
([#14885](#14885))
([cfe728e](cfe728e))
* **spanner:** synchronize stream lifecycle and retry buffering in
ResumableStreamIterator
([#14770](#14770))
([4edefc5](4edefc5))
* **spanner:** trigger best-effort rollback on indeterminate commit
failures
([#14831](#14831))
([b9d1b95](b9d1b95))
* **storage:** preserve HTTP status for XML multipart upload errors so
retries work
([#14804](#14804))
([3c03034](3c03034))
* **storage:** send destination on gRPC copy when target BlobInfo is
provided
([#14784](#14784))
([43a67a0](43a67a0))
* **storage:** validate CRC32C on HTTP downloads that were not
decompressed
([#14805](#14805))
([61aa22f](61aa22f))


### Documentation

* **bigtable:** document that skipped large rows count towards row limit
([#14801](#14801))
([91d60e0](91d60e0))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ShreyasSinha ShreyasSinha ShreyasSinha approved these changes
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Dhriti07 @diwu-sf @ShreyasSinha