[jjohnson-hdb]

Problem

.xml.gz is the sitemaps.org convention for large sitemaps, and most big catalogs publish the children of their index that way. An origin serves such a child as a gzip file: content-type: application/x-gzip, no Content-Encoding. undici decodes only a Content-Encoding, so fetchSitemap read the compressed bytes with res.text() and handed them to the XML parser. Every such child failed:

Sitemap fetch for https://…/sitemap_…_1.xml.gz returned a non-sitemap response
(status 200, content-type application/x-gzip): Maximum nested tags exceeded — \x1f\x8b…

An index made only of .xml.gz children therefore created zero targets. A weekly or daily refresh would fail the same way on every run. Nothing in the plugin decompressed a gzip file: zlib was used only for page storage, the change probe and sampling.

Fix

• New decodeSitemapBody(bytes, { maxBytes }) in util/sitemap.js:
  • Body starts with the gzip magic bytes 1f 8b: async gunzip, capped at MAX_SITEMAP_DECOMPRESSED_BYTES (100 MB, twice the sitemaps.org 50 MB limit).
  • Anything else: TextDecoder, which is exactly what Response.text() did (UTF-8, BOM stripped).
  • It sniffs the bytes rather than the content-type or a .gz suffix, because origins label these files inconsistently and no XML document starts with 1f 8b.
• fetchSitemap reads arrayBuffer() and decodes through the helper. A truncated, corrupt or oversize gzip file throws, so the child fails loudly rather than being read as a shorter sitemap. That matters for departures: see the existing truncated-gzip note on parseSitemap.
• Transport-encoded bodies (Content-Encoding: gzip) are decoded by undici as before and pass through unchanged.
• README: a short "Gzipped sitemaps (.xml.gz)" section.

Tests

• test/sitemap.test.js, 5 new tests:
  • a gzip file served as application/x-gzip with no Content-Encoding parses to all 5,000 entries; the raw bytes still fail parseSitemap, which proves undici left them compressed
  • a transport-encoded body is unchanged
  • plain XML with a UTF-8 BOM decodes exactly as Response.text() would
  • a truncated gzip file is refused
  • a body that inflates past the cap is refused (gzip-bomb guard)
• node --test in packages/plugin: 2,099 / 2,099 pass. eslint and prettier --check on the touched files are clean.
• A real 1.68 MB production .xml.gz child decodes to 44,554 entries in 194 ms (decodeSitemapBody + parseSitemap).

Verified live

The same two-file change was hot-patched onto a deployed 0.104.0 plugin (6-node cluster; the installed files matched the prerender-v0.104.0 tag byte for byte before patching). After an http_workers restart:

• POST /sitemaps/<index> with revalidate: true finished in about 20 s: sitemapsProcessed: 2/2, created: 46484, failed: [].
• Before the patch, the same walk failed with the error above and created: 0.

That cluster runs this patch outside a release until it upgrades to the release that contains it.

Version

0.106.1 (patch). Open #254 has reserved 0.107.0, so whichever merges second rebases its version.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces support for gzipped sitemaps (.xml.gz) served as raw gzip files without a Content-Encoding header. It implements a decodeSitemapBody utility that sniffs for gzip magic bytes, decompresses the body asynchronously up to a 100 MB limit to prevent gzip bombs, and falls back to plain text decoding. Comprehensive tests are added to cover various decompression scenarios. The feedback recommends improving error handling in the catch blocks of Sitemap.js and sitemap.js by safely accessing the error message using e?.message ?? String(e) to handle non-standard exceptions robustly.

[gemini-code-assist]

When catching exceptions, it is safer to handle non-standard errors or string throws by using e?.message ?? String(e) instead of directly accessing e.message. If a non-standard error or string is thrown, e.message will evaluate to undefined, resulting in an unhelpful error message.

Suggested change

	} catch (e) {
	throw new Error(`Sitemap fetch for ${url}${via} (status ${res.status}): ${e.message}`);
	}
	} catch (e) {
	throw new Error('Sitemap fetch for ' + url + via + ' (status ' + res.status + '): ' + (e?.message ?? String(e)));
	}

References

1. When handling caught exceptions, safely access the error message using optional chaining and fall back to String(e) (e.g., e?.message ?? String(e)) to ensure that the error handling or logging block itself does not throw a TypeError if the caught exception is null, undefined, or a non-object.

[gemini-code-assist]

When catching exceptions, it is safer to handle non-standard errors or string throws by using e?.message ?? String(e) instead of directly accessing e.message. If a non-standard error or string is thrown, e.message will evaluate to undefined, resulting in an unhelpful error message.

Suggested change

	} catch (e) {
	throw new Error(`gzip sitemap body could not be decompressed (${bytes.length} bytes): ${e.message}`);
	}
	} catch (e) {
	throw new Error('gzip sitemap body could not be decompressed (' + bytes.length + ' bytes): ' + (e?.message ?? String(e)));
	}

References

1. When handling caught exceptions, safely access the error message using optional chaining and fall back to String(e) (e.g., e?.message ?? String(e)) to ensure that the error handling or logging block itself does not throw a TypeError if the caught exception is null, undefined, or a non-object.