Repository navigation
gh-pages deploy failing: artifact crossed Pages publish cap on Oct 26 #3231
Description
The pages-build-deployment workflow has been failing on every push to gh-pages since 2026-10-26 14:31 UTC.
Symptom
actions/deploy-pages@v5 warns on upload and the Pages backend rejects the deployment:
Uploaded artifact size of 1442022902 bytes exceeds the allowed size of 1 GB. Deployment might fail.
...
Artifact could not be deployed. Please ensure the content does not contain any hard links, symlinks and total size is less than 10GB.
The "<10 GB" wording in the error is the generic deploy-pages message; the actually enforced ceiling is the GitHub Pages 1 GB site size limit.
When
- Last green run: https://github.com/NVIDIA/cuda-python/actions/runs/40302688363 (2026-10-26 09:56 UTC, artifact 1.359 GB)
- First red run: https://github.com/NVIDIA/cuda-python/actions/runs/40341674789 (2026-10-26 14:47 UTC, artifact 1.442 GB)
Every run since has failed for the same reason.
Artifact size trend (compressed)
| Date (UTC) | Artifact | Outcome |
|---|---|---|
| 2026-10-23 00:37 | 1.364 GB | deployed |
| 2026-10-24 04:19 | 1.292 GB | deployed |
| 2026-10-25 04:19 | 1.245 GB | deployed |
| 2026-10-26 09:56 | 1.359 GB | deployed (last success) |
| 2026-10-26 14:47 | 1.442 GB | rejected (first failure) |
The deploy-pages warning has been firing for days — the site has been riding the edge. The jump between the last green and first red run coincides with three preview commits landing on gh-pages: PRs #3186, #3208, #3133.
What's on gh-pages right now (uncompressed)
pr-preview/— 3.3 GB (37 preview dirs; 32 live PRs, 5 orphan)cuda-bindings/(pinned versions) — 2.25 GBcuda-core/(pinned versions) — 1.0 GBcuda-pathfinder/— 242 MBcoverage/— 89 MB- 26 CTK-tagged doc trees combined — ~100 MB
- Total ~7 GB uncompressed → ~1.44 GB compressed artifact
Immediate unblock
Triggering cleanup-pr-previews.yml via workflow_dispatch sweeps orphan preview dirs for closed PRs (currently #2308, #2925, #3133, #3189, #3208). On its own this is unlikely to drop the artifact under the cap; a durable fix will need to trim one of the top-level buckets.
Mitigation analysis to follow in a comment.
-- Leo's bot
Activity
leofang commentedon Oct 27, 2026
Measuring where the growth is coming from, so each option can be sized.
Per-preview breakdown (sample: pr-preview/pr-3186, 590 MB)
.doctrees/_sources/Published trees (
docs/latest/,docs/cuda-bindings/latest/, etc.) already don't ship.doctreesor_sources— those two directories exist only inside PR previews.What recently inflated the previews
Each preview's
cuda-bindings/latest/module/generated/has two cohorts across the 37 current previews:11 of 37 previews are already in the inflated cohort; the remaining 26 are baseline because they branched before the inflection. Diffing the two sets, all 912 new pages are
cuda.bindings._v2.driver.*autosummary pages (e.g.driver_v2/cuda.bindings._v2.driver.AccessPolicyWindow_v1.html). As in-flight PRs rebase past the v2-driver landing onmain, each one will pick up ~390 MB extra. Any one-shot trim gets eaten back as rebases roll in.Published
docs/cuda-bindings/latest/is already inflated too, which is why the maincuda-bindings/bucket sits at 2.25 GB.Options
.doctrees+_sourcesfrom preview uploads. Saves ~17.6 MB per preview (≈ 650 MB uncompressed across 37, ~100–150 MB compressed). Low effort, buys this week, doesn't address the growth driver.cuda-bindings/latest/module/. Saves the full ~390 MB per inflated preview — 100% of the recent growth. Published docs keep the full API reference.cleanup-pr-previews.ymlmanually to sweep the 5 orphan closed-PR previews ahead of the 04:00 UTC nightly run. Small, immediate.gh-pages. Upload each per-PR docset as a workflow artifact (10 GB cap, 90-day retention) and post the link on the PR. Removes previews from the Pages site budget entirely. xref: CI: Reduce traffic and slow down size-bloating ofgh-pages#3014Suggested sequencing: 4 now to recover the smallest increment, 1 + 2 together as the short-term durable fix, 5 as the long-term direction. Option 3 is complementary to any of these.
CTK-tagged archive pruning is intentionally not on this list — all 26 tags combined are ~100 MB, well under 2% of the site, and a non-goal to touch.
-- Leo's bot
leofang commentedon Oct 27, 2026
@mdboom WDYT?
leofang commentedon Oct 27, 2026
xref: https://github.com/NVIDIA/cuda-python/actions/runs/40424476295
Purge orphaned cuda-core pre-release doc trees
mdboom commentedon Oct 27, 2026
#3095 added the _v2 driver module and its associated docs. It's both (a) more extensively documented than the existing one and (b) our biggest module by far. But I definitely wasn't aware of just /how much/ of an impact it had.
I think option #4 right now, (which @leofang already did), #2 today, with a plan to get to option #5 later is the right approach. #1 seems to be low benefit and #3 is very complex for the benefit we would get.
leofang commentedon Oct 29, 2026
This still happens after #3244 was merged.
Purge gh-pages trees with no matching git tag
Skip docs deploy on tag CI runs
Restore docs/cuda-bindings/13.4.4 to its v13.4.4 release content
Empty docs/pr-preview to bring Pages site back under the 1 GB cap
leofang commentedon Oct 30, 2026
gh-pages corrective commits during this incident
X.Y.Z.devNdirs)13.1.2/13.2.1/13.3.2, 4cuda-bindings/{13.1.2, 13.2.1, 13.3.2, 13.4.0b1},cuda-pathfinder/1.5.7)docs/cuda-bindings/13.4.4/to its real v13.4.4 release content, undoing the overwrite caused by thecuda-pathfinder-v1.8.5tag CI deploydocs/pr-preview/(41 preview trees) to bring the site back under the 1 GB capNet:
docs/tree dropped from ~7 GB uncompressed to ~2.7 GB. At the ~5× sphinx-HTML compression ratio, the published artifact should land around ~550 MB.Side effects:
docs/pr-preview/pr-N/through the normaldoc_previewaction; the sticky comment will refresh at that point.cuda-core,cuda-bindings, andcuda-pathfinderare untouched; version switcher behavior is unchanged.The underlying leak that caused the orphan accumulation and the Oct 29 cuda-bindings/13.4.4 overwrite is not fixed by these commits — it is addressed in #3232 (pending review). Until that lands, each subsequent tag push on any subproject will re-leak at least one directory via the same mechanism.
-- Leo's bot
Skip docs deploy on tag CI runs (#3232)