Skip to content

fix(cli): a cloud upload cut at a fixed age stops early and shows its progress - #7350

Merged
miguel-heygen merged 4 commits into
mainfrom
fix/cloud-upload-cut-connection
Nov 7, 2026
Merged

miguel-heygen merged 4 commits into
mainfrom
fix/cloud-upload-cut-connection

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

What

A cloud upload whose connection is cut at a fixed age no longer repeats the same doomed try four times, and a long upload now says how far it got.

  • Each PUT reports bytes sent at least every 10 s or every 5%: one line per report on stdout, or JSON lines on stderr under --json ({"event":"upload_progress","file","attempt","percent","sent_bytes","total_bytes"}). A second try says it restarted from the first byte.
  • After two tries in a row that die by a connection reset at about the same age with about the same bytes sent, the upload stops and says so: The connection was cut at N s on two tries in a row, after P% of the file was sent ... this link cannot finish it before the cut.
  • When the retries run out, the error says how many tries were made and how far the last one got.

Why

Cloud exports were failing in the prepare stage with Media upload failed after 26 to 35 minutes. The uploader sends one signed PUT, restarts it from byte 0 on any failure, and tries 1 + 3 times with nothing bounding a try. If anything on the path drops every connection at a fixed age and the link cannot finish the file before it, every try dies at that age: failure time = 4 x age + 7 s of backoff, so 26 to 35 minutes means cuts at about 6.5 to 8.7 minutes. A single large file also printed nothing for the whole PUT, so a live slow upload and a dead one looked the same.

Reproduced with the real uploader against a local server that drains 100 KB/s and cuts every connection at 15 s, a 4 MB file: tries 1 to 4 each cut at 15 s after the same 1.5 MB, failure at 67 s (4 x 15 s + 7 s), read ECONNRESET. Controls: no cut finishes in 41 s; a 1 MB file that fits in 15 s finishes in 10 s. After this change the same run stops after the second cut.

How

putWithRetry records each try's age and bytes (PutTry), and compares a connection-reset failure with the previous one. The byte count is what the socket has accepted (bytesWritten, sampled by the existing stall watcher), so the last few MB can be buffered ahead of the wire.

No overall time limit was added: it would kill valid slow uploads. The progress lines let a caller apply its own no-progress stop. A link that cannot finish a file before a fixed cut needs a resumable or multipart upload, which is a server change and not in this PR.

Size

One root in one module (upload.ts), its caller wiring and its tests; nothing else touches the uploader.

Test plan

  • cd packages/cli && npx vitest run src/cloud src/commands/cloud: 216 passed, 5 runs in a row; oxlint, oxfmt and tsc clean.
  • fails without the fix: three new tests (stop after a second identical cut, progress lines and the restart event, tries in the final error) fail on main's upload.ts.
  • comment share of upload.ts 16.4% to 13.8%, render.ts 10.2% to 9.95%, upload.test.ts 0.73% to 0.64%.

@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Fallow audit report

Quality gate failed

Found 3 findings.

Health (3)
Severity Rule Location Description
major fallow/high-crap-score packages/cli/src/cloud/upload.test.ts:391 'server' has CRAP score 37.1 (threshold: 30.0, cyclomatic 11)
major fallow/high-crap-score packages/cli/src/cloud/upload.ts:135 'putWithRetry' has CRAP score 43.1 (threshold: 30.0, cyclomatic 12)
major fallow/high-crap-score packages/cli/src/commands/cloud/render.ts:675 'prepareLocalArchive' has CRAP score 31.6 (threshold: 30.0, cyclomatic 10)

Generated by fallow.

Gate outcomes: failed audit-verdict.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 2e60ba1e. No blockers.

Verified locally (Linux):

  • At this head, src/cloud/upload.test.ts and src/commands/cloud pass (64 tests). Across src/cloud and src/commands/cloud, 215 of 216 pass. The one failure is offload.undecodableTrack.test.ts: my local ffmpeg couldn't build that test's 4-channel PCM .mov input. The diff doesn't touch that test or its code.
  • The fails without the fix claim holds. With main's upload.ts swapped in, all three new tests fail: stops after a second identical cut…, reports the bytes sent while a slow PUT runs… and says how many tries it made…. With the fix restored, they pass.

Code:

  • The early stop is conservative enough. It needs two connection-cut failures in a row (ECONNRESET, EPIPE, ECONNABORTED or a socket hang up) where the earlier try had sent bytes, and the two tries' age and bytes sent must be within 20% of each other. Stalls, 5xx responses and the fetch path (which reports sent = 0) still take the normal retries.
  • Final errors are still thrown unwrapped. Only exhausted retries get wrapped, and the wrapper keeps code and cause. Nothing downstream checks the error class of retryable failures.
  • The progress percent is capped at 99 until the upload finishes, and the existing percent: 100 event has no sentBytes, so the reporter skips it. The ev.attempt === undefined guard keeps the zip's "uploading to S3…" line from being printed again.

Non-blocking notes:

  • Behaviour change: --json mode used to print nothing for the zip upload. It now writes JSON progress lines to stderr. The PR describes this and stdout stays clean, but any caller that treats stderr output as a failure would see the new lines.
  • progressReporter keeps one lastPercent for all tries. After a restart, the next try therefore reports on the 10 s timer until it gets 5 points past the earlier try's high mark. That still meets "at least every 10 s".

@miguel-heygen
miguel-heygen added this pull request to the merge queue Nov 7, 2026
Merged via the queue into main with commit 65c62f3 Nov 7, 2026
118 of 121 checks passed
@miguel-heygen
miguel-heygen deleted the fix/cloud-upload-cut-connection branch November 7, 2026 05:03
SanHsien added a commit to SanHsien/hyperframes that referenced this pull request Nov 7, 2026
…-com#7336-heygen-com#7358

Adopt heygen-com#7304 (without CLI renderDiagnostic, needs heygen-com#6725), heygen-com#7331, heygen-com#7332, heygen-com#7334,
heygen-com#7337, heygen-com#7336, heygen-com#7339, heygen-com#7289, heygen-com#5594, heygen-com#6900, heygen-com#7322, heygen-com#7342, heygen-com#7343, heygen-com#7350 (onProgress
only), heygen-com#7352, heygen-com#7348, heygen-com#7344, heygen-com#7353, heygen-com#7356 and heygen-com#7355. Defer heygen-com#5708 (needs heygen-com#6911) and
heygen-com#7345 (READ_VERSION 4 vs 7); heygen-com#7307 not applicable. Re-read 14 moved pins, pin 6
new PRs (monitored 126). Make checkReportFile module-private and suppress the
cross-package closeTestChrome duplicate for fallow.

Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
Co-authored-by: SophiaHuangCc <129631224+SophiaHuangCc@users.noreply.github.com>
Co-authored-by: Dante <duanjl.china@gmail.com>
Co-authored-by: Muzzy5150 <muzzy5150yt@gmail.com>
Co-authored-by: ump45nose <52391318+ump45nose@users.noreply.github.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant