feat(upload): split profile archive uploads into concurrent parts

[lvaroqui]

Upload profile archives as S3 multipart uploads, with parts sent concurrently and checked against CRC64NVME checksums.

S3 rejects a single upload request above 5 GiB, so large walltime and memory profile archives could not be uploaded (walltime folders above 5 GiB were gzipped on disk to try to fit). Even below that limit, a single connection to S3 only reaches about 20-25 MiB/s on GitHub-hosted runners, so a 1 GiB archive took close to a minute to upload.

How it works

• Part layout: every archive is a multipart upload. The target is two parts per concurrent upload, so that a slot freed by a fast part picks up a remaining one instead of idling while the slowest part finishes. Part sizes are clamped between 16 MiB and 256 MiB, so the part count drifts from the target at both ends: with the default concurrency of 8 (16 parts), archives above 4 GiB get more 256 MiB parts, and archives below 256 MiB get fewer 16 MiB parts, down to a single part.
• Checksums: the CRC64NVME of every part and of the whole archive are computed in one pass over the archive, on the blocking thread pool. They replace the md5 in the upload metadata (bumped to version 12): profileMd5 and profileEncoding become profileArchiveMetadata (encoding, size, crc64nvme, partSize, partCrc64nvmes).
• Upload: the upload endpoint answers with multipartUpload instead of uploadUrl: one presigned UploadPart request per part (parts) and a presigned CompleteMultipartUpload request (complete), each as { url, headers }. The headers are part of the signature and are sent as is, so S3 checks each part and the whole object against the checksums. Parts are uploaded 8 at a time, each with its own retries, then the ETags are sent in part order to complete the upload.
• Completion errors: S3 can report a failed completion with a 200 status, so the response body is checked for an error. Errors S3 documents as transient (InternalError, ServiceUnavailable, SlowDown, RequestTimeout) and dropped connections are retried; others such as InvalidPart fail right away.
• Both archive kinds: on-disk (walltime, memory) archives are streamed from disk part by part, and in-memory gzip (simulation) archives are sliced without copying. Both go through the same retry loop.
• S3 specifics (presigned requests, ETag, completion body and errors) live in the new upload::s3 module.
• CODSPEED_UPLOAD_CONCURRENCY overrides the number of concurrent part uploads.

Other changes

• Walltime profile folders above 5 GiB are no longer gzipped on disk, and the runner no longer caps the archive size: with 256 MiB parts, the S3 limit of 10,000 parts allows archives up to 2.5 TiB.
• The md5 dependency is replaced by crc-fast.

Measurements

Concurrency sweep on a 6 GiB walltime archive (25 parts of 256 MiB), which led to the default of 8:

Concurrency	ubuntu-latest	CodSpeed macro runner (Ryzen 9950X)
1	240.3s · 25.6 MiB/s	59.5s · 103.2 MiB/s
2	83.4s · 73.7 MiB/s	33.9s · 181.4 MiB/s
4	57.2s · 107.4 MiB/s	21.3s · 289.0 MiB/s
8	45.2s · 135.8 MiB/s	8.2s · 746.5 MiB/s
16	48.7s · 126.1 MiB/s	10.3s · 596.5 MiB/s

Smaller archives with the final part layout, concurrency 1 (close to the previous single request) vs 8:

Archive	Runner	Concurrency 1	Concurrency 8
256 MiB	ubuntu-latest	11.6s	2.8s (4.1×)
1 GiB	ubuntu-latest	55.1s	9.0s (6.1×)
256 MiB	macro runner	2.7s	1.3s (2.1×)
1 GiB	macro runner	10.1s	1.6s (6.3×)

Hashing a 6 GiB archive, comparing the md5 the runner computed so far with the alternatives. The CRC64NVME of the whole archive is combined from the part CRCs, so each byte is hashed once and the whole + parts pass costs the same as hashing the whole archive alone:

Variant	ubuntu-latest (EPYC 7763, 2 cores)	Macro runner (9950X, 8 cores)
MD5 whole + parts (before)	24.3s · 0.25 GiB/s	14.5s · 0.41 GiB/s
Composite MD5, parts in parallel	6.7s · 0.89 GiB/s	0.9s · 6.46 GiB/s
CRC64NVME whole + parts (two digests per byte)	1.0s · 5.79 GiB/s	0.2s · 29.11 GiB/s
CRC64NVME parts + combine (this PR)	0.5s · 11.28 GiB/s	0.1s · 44.14 GiB/s
CRC64NVME whole only (reference)	0.5s · 11.35 GiB/s	0.1s · 46.83 GiB/s
CRC32C parts + combine	0.4s · 17.02 GiB/s	0.1s · 46.76 GiB/s
CRC32 parts + combine	0.5s · 11.28 GiB/s	0.1s · 46.48 GiB/s

The backend support for the version 12 upload metadata is not released yet, so the upload cannot be verified end to end against production for now.

Closes COD-3700

[codspeed]

Merging this PR will not alter performance

✅ 24 untouched benchmarks
⏩ 13 skipped benchmarks¹

---

_{Comparing cod-3700-multipart-upload (acfbe6a) with main (24f182e)}

Footnotes

1. 13 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[greptile-apps]

Confidence Score: 5/5

[High impact] The PR appears safe to merge, with a non-blocking safeguard needed for archives above S3’s multipart limit.

Findings

1.  Part count can exceed S3 limit ▶

Fix with agent prompt

### Issue 1
src/upload/profile_archive.rs:132-133
The part size stops growing at 256 MiB, but there is no limit on the number of parts. If a WallTime or Memory archive exceeds about 2.44 TiB, it needs more than S3’s 10,000-part maximum. The runner will hash the archive and prepare the upload before it fails. Checking the part count earlier would avoid that wasted work and give a clearer error.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR replaces single-request profile uploads with concurrent S3 multipart uploads and version 12 CRC64NVME archive metadata.

• Disk archives are streamed by part; in-memory archives are sliced into parts.
• Presigned requests carry signed headers, and completion checks S3’s response body.
• The removed archive limit leaves the S3 part-count ceiling unenforced.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Create profile archive] --> B[Compute whole and part CRC64NVME]
  B --> C[Send version 12 metadata to API]
  C --> D[Receive presigned part and completion requests]
  D --> E[Upload parts concurrently]
  E --> F[Complete multipart upload in part order]

Loading

_{Reviews (5) · Last reviewed commit: "review changes (will squash merge)" · Reviewed by Greptile}

[greptile-apps]

Want your agent to iterate on Greptile's feedback? Start a greploop in Claude Code and it will work through the open comments and keep going until this PR reviews clean.

[adriencaccia]

Seen together, let's remove the single part upload path and always use multipart, albeit with a single part for archives under 64MiB.

[GuillaumeLagrange]

olgtm: the metadata snapshot update should have its morrir conterpart updated in platform repo in order to make sure metadata stays compatible, hit me up for more info.

Also some tests could be trimmed, the /deslop skill that we share in platform repo is quite a good judge for it usually

[GuillaumeLagrange]

OLGTM!
I guess that CI failure is because prod does not yeat accept the new upload metadata. Let's keep this in mind before merging/releasing this and make sure everything's been deployed