Skip to content

feat(cli): cloud uploads send a big file in parts, so a cut connection resends one part - #7375

Merged
miguel-heygen merged 6 commits into
mainfrom
feat/cloud-upload-parts-stacked
Nov 7, 2026
Merged

miguel-heygen merged 6 commits into
mainfrom
feat/cloud-upload-parts-stacked

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

What

hyperframes cloud uploads a file over 16 MiB in parts when the cloud offers them, so a cut connection costs one part instead of the whole file.

  • The CLI asks the cloud for a parts plan (multipart: true on create). With a plan it sends each part to its own presigned URL through the same PUT and retry as today, keeps each part's ETag, and tells complete which parts make the file.
  • A part that is cut is sent again from its own start; parts already sent are not.
  • The plan is saved under the CLI config dir by file checksum and size, so a restarted upload of the same file asks the cloud which parts it holds and sends only the rest. The saved plan is removed once complete succeeds, and also when complete refuses the file itself (a 4xx other than 401, 403, 408, 409 or 429), so a refused file is not resumed into the same refusal. A cut, a 5xx or an abort keeps it for the next try.
  • Files at or under 16 MiB, a server that answers without a plan, and a server that refuses the new field (400/422, asked once more without it) all keep the single PUT. This PR is safe to ship before the server side is deployed: until then every upload takes the single PUT, as today.
  • _gen/client.ts and _gen/types.ts gain the part routes and fields the server adds.

Why

Users on links that cut long connections could not finish a big upload: every retry restarted the file, and the CLI ended with "The connection was cut at N s on two tries in a row". Parts are short enough to finish before the cut.

Stack

On top of #7374, which moves the PUT code to uploadPut.ts; review that one first.

Test plan

  • uploadParts.test.ts, against a local HTTP server that cuts every connection after 1.2 s: a 17 MiB file still arrives in 5 parts of up to 4 MiB and complete gets all 5 ETags; a cut part is sent again alone; a restart resends only the missing parts; progress counts finished and live parts. Small files, a plan-less server and a server refusing the field keep the single PUT. 186 tests in src/cloud.
  • A refusal on complete (400 invalid_parameter) makes the next upload of the same file start fresh; a 503 on complete makes it resume. Without the forget on refusal, that test fails (the next upload resumes the stale plan).
  • fails without the fix: with the parts path disabled in upload.ts, the 1.2 s cut test fails (ECONNRESET). A control case in the suite shows the same link failing a single PUT of the same file with "cut at ... on two tries in a row".
  • src/cloud unit tests (186), 3 runs in a row; tsc --noEmit for cli exit 0.
  • bun run format:check exit 0; fallow audit exit 0; oxlint exit 0; comment ratchet and citations exit 0.
  • Not run against a real cloud: the server side is not deployed yet.

heygengenesis[bot]
heygengenesis Bot previously approved these changes Nov 7, 2026

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE · 0 blocker · 1 important · 1 nit · head 9af5d20b

  • important uploadParts.ts:52-68 + upload.ts:385: the saved plan is removed only after complete succeeds. If complete refuses for good (for example the content-type refusal that nameRefusedFile handles), every later run of the same bytes resumes the same asset, with its original filename and content type, and gets the same refusal until the multipart upload expires on the server. Renaming the file doesn't help either, because the key is checksum+size. Fix: forget the saved plan when complete returns a non-409 4xx.
  • nit uploadParts.ts:160: a mint failure (HyperframesApiError 401/404) inside the putWithRetry closure isn't isFinal, so it is retried 4 times with backoff. On a part, the repeatedCut message still says "Every try restarts the file".
Scope

Read upload.ts, uploadParts.ts, uploadPut.ts and the test end to end; skimmed _gen. Ran uploadParts.test.ts locally: 8/8 pass. I checked the "fails without the fix" claim by disabling if (plan) return sendInParts: the 1.2 s cut test fails with "cut at 1 s on two tries in a row", and the cut-part and resume tests fail too. mergeStateStatus CLEAN. No prior reviews.

Base automatically changed from refactor/cloud-upload-put-file to main November 7, 2026 19:50
@miguel-heygen
miguel-heygen dismissed heygengenesis[bot]’s stale review November 7, 2026 19:50

The base branch was changed.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE — 1 resolved, 0 open, 0 new · 9af5d20b → b554a017. A 4xx refusal from complete now forgets the saved plan; 401/403/408/409/429, 5xx and cancels keep it so the upload can resume. The new test covers both sides. Since my last review the merge from main changed no files under src/cloud, and CI is green.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE — 0 resolved, 0 open, 0 new · b554a017 → 8249651a. A resumed upload now goes straight to sendInParts, so a plan read from disk can no longer reach putBytesToPresignedUrl. Behavior is otherwise the same: the guard dropped with "upload_url" in created was only there for the old union type, and CreateAssetUploadResponse.upload_url is required. CI and CodeQL are green.

@miguel-heygen
miguel-heygen added this pull request to the merge queue Nov 7, 2026
Merged via the queue into main with commit 3d7f12a Nov 7, 2026
70 checks passed
@miguel-heygen
miguel-heygen deleted the feat/cloud-upload-parts-stacked branch November 7, 2026 21:30
@miguel-heygen miguel-heygen mentioned this pull request Nov 7, 2026
3 of 5 tasks
SanHsien added a commit to SanHsien/hyperframes that referenced this pull request Nov 9, 2026
Port the adopted upstream fixes from f5bde95 to the current upstream tip:
heygen-com#7375 multipart cloud upload, heygen-com#7382 minParallelChunks, heygen-com#7349 HLG header SEI,
heygen-com#7369 skip unmounted compositions in lint/check, heygen-com#7381 filesystem-path media
in the cloud archive, heygen-com#7383 Safari cut-point pre-roll revert, heygen-com#7373 clock
steering between audio time updates, heygen-com#7384 injector test isolation, heygen-com#7378
preview seek retention, heygen-com#7390 virtual timer schedule per worker, heygen-com#7394 CDN
script failure naming, heygen-com#7395 menu row radius, heygen-com#7119 decode timeout caller
test, heygen-com#7397 authored tween order, heygen-com#7402 capture page clip loading, heygen-com#7412 muted
video audio track at cuts, heygen-com#6150 template root variable test, heygen-com#7442 held tail
seek once.

Deliberate local differences: heygen-com#7067's audio-start hold is not carried into
clock.ts, mountedSources comes from heygen-com#6827 as a helper only, cloud/_gen/types.ts
omits the image-request types the fork does not generate, and the lint
symlink test skips through symlinkOrSkip where the account cannot create links.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant