Repository navigation
S3 support - #10
Draft
abernardeau-wallix wants to merge 54 commits into
Draft
S3 support#10abernardeau-wallix wants to merge 54 commits into
abernardeau-wallix wants to merge 54 commits into
Conversation
abernardeau-wallix
force-pushed
the
s3-support
branch
2 times, most recently
from
November 5, 2026 22:44
26dedc4 to
ec9bea3
Compare
abernardeau-wallix
marked this pull request as draft
November 5, 2026 23:01
abernardeau-wallix
force-pushed
the
s3-support
branch
2 times, most recently
from
November 6, 2026 15:13
596a3bb to
220e4d4
Compare
Move the object model of the plain-file area (directories, objects, foreign tags, the namespace lock) and the streaming write out of the WebDAV handlers into vk-registry/src/files.rs, so another wire protocol can serve the same objects with its own status codes and bodies. The write is split into staging (stream to uploads/ while hashing) and a commit that promotes and tags under the namespace lock, and the reason a path cannot take an object is returned as a value each front maps to its own status. No behaviour change.
Requests under /s3/ now pass a gate S3 clients understand: a SigV4 Authorization header is verified, refusals are S3 XML errors carrying a request id, and a missing or refused credential is a 403 rather than the 401 challenge. The access key id is a credential the registry already takes (an API key, a hub token, the shared bearer, or the Basic user), authenticated as a bearer would be before any secret is used, and the signing secret is that same string (the password for the Basic user), so nothing new is stored. Presigned URLs and temporary credentials are refused. No operation is served yet: authenticated requests get 404 or 501.
A front over the object layer can now see a body's bytes as they stream (for digests of its own), have annotations written on the object's manifest, and have a precondition on what is at the path checked under the namespace and store locks the tag is written under, so a create-only or compare-and-swap write cannot race another. WebDAV asks for none of it; its behaviour is unchanged.
Serve the plain-file area's objects under /s3/: PutObject, GetObject, HeadObject, GetObjectAttributes and DeleteObject, with HeadBucket, GetBucketLocation and ListBuckets, over the object layer WebDAV uses and under its authorization and disclosure rules. A key maps onto the store's names as they are; anything else is refused naming the rule. A write records the object's MD5 (its ETag) and user metadata on the manifest, checks the signed payload digest and the checksums it is sent, and checks If-None-Match/If-Match under the namespace lock; reads take the four conditional headers and the checksum mode. No content type is kept. Listings, aws-chunked bodies, ranges, copies and multipart answer NotImplemented until they land.
Serve ListObjectsV2 and ListObjects over the files/ tree: a depth-first walk in byte order of the whole key, bounded by one page, that passes over every subtree before its resume point and folds delimited directories without walking them, showing only what a read would serve. A listing is admitted on the bucket or on the directory its prefix names, and HeadBucket and GetBucketLocation on anything readable below the bucket, so a key scoped to one directory can probe and list it.
Decode the aws-chunked bodies AWS SDKs and the CLI send, in one streaming pass under the staging write: signed chunks, signed and unsigned trailers, and plain aws-chunked framing. Chunk and trailer signatures chain from the request's signature and are checked with its signing key; a body sent under a bearer or to an open server has its framing decoded and its signatures left unchecked. The trailer's checksum is checked like a header's. Framing is bounded as it is read, and a body refused anywhere leaves nothing tagged.
Serve one byte range of an object on GetObject and HeadObject: 206 with Content-Range, 416 InvalidRange for no byte of it, If-Range honoured, and several ranges or an unparsable Range answered whole. As S3 takes them, a range with whitespace in it or whose last byte comes before its first does not parse, and a suffix of an empty object is all of it. Accept-Ranges is now sent on every object answer. A plain blob is read after a seek, a zstd-stored one decoded and skipped up to the range. A PutObject of 8 MiB or more is stored uncompressed, so the AWS CLI's parallel 8 MiB ranged downloads stay linear.
Serve CreateBucket and DeleteBucket over files/ directories, CopyObject and DeleteObjects, and answer the configuration probes tools send before using a bucket (versioning, tagging, ACLs, lifecycle and the rest) as S3 answers for a bucket with none set; writes of them stay NotImplemented. A copy tags the source's layer again under the namespace and shared store lock, so no byte moves and the destination records the layer as a member; the source's copy-source conditions and the destination's write conditions are checked there, and its MD5 and metadata carried or replaced. A bucket whose keys are all deleted is empty, so DeleteBucket removes the directories they leave. Delete documents are read by a bounded, dependency-free element scanner that expands no entity.
Serve CreateMultipartUpload, UploadPart and UploadPartCopy, CompleteMultipartUpload, AbortMultipartUpload, ListParts and ListMultipartUploads. An upload is a private directory under uploads/s3/ holding its record and its parts, each replaced whole; a part is received and checked as a PutObject's body is, and a completion concatenates the parts in one pass off the runtime, checking each part's MD5 again, then tags the object as a PutObject is, under the multipart ETag "<md5 of the parts' md5s>-<parts>". The completion's Location is the object's absolute URL, as S3 gives it: the listener's scheme, the Host the request was sent to and its path. The gc sweeps an upload once nothing in it has been written for the grace window, and status counts the uploads in progress.
`accounts create-key` and the `/settings/keys` page now also show a newly minted key as the AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY pair S3 clients take. The CLI's stdout is unchanged (the token alone); the pair goes to stderr, and spells the token out only when stdout is a terminal.
A new optional `s3_host` setting serves `<bucket>.<s3_host>` as the path-style `/s3/<bucket>/<path>`, for S3 clients that only speak virtual-hosted style. The match comes before every other route, so the whole path is the key; any other host, `s3_host` itself and a label that cannot be a bucket are routed as before. Signatures are checked over the request as sent, so nothing changes for credentials.
A host-side harness that points suites written by others at the registry's S3 view: ceph's s3-tests, MSST-S3, versitygw's integration tests, MinIO warp, and an AWS CLI and rclone smoke run, each fetched at a pinned commit or sha256 and run from a shell against a throwaway `vk-registry serve`. A baseline of expected failures per suite, recorded by `run.sh --accept`, makes a run fail only on a regression. Nothing here is part of the build or of `cargo test`.
A ranged GetObject now answers without x-amz-checksum-sha256, as S3 sends a checksum with the whole object only. The AWS CLI checks the checksum it is sent against the bytes it received, so the whole object's on a 206 failed every multi-part download of an object over 8 MiB.
tests/s3-e2e.sh runs vk-registry in accounts mode with TLS on the host's LAN address (a run CA, the guest trusting it) and drives the three clients from a Debian trixie microVM against /s3/, then reads an object back on the host with `vk registry get` across a gc. trixie's sccache lacks its S3 backend, so the image takes the upstream 0.18.0 release, pinned by sha256. Measured with the AWS CLI in a 2-vCPU guest, over TLS and vk's userspace network, CLI start-up (~0.8 s) included: a 100 MiB cp up 4.0-4.3 s and down 1.9-2.4 s; a ranged get of its second 50 MiB 1.7-2.9 s.
A job's token now also reaches the registry's S3 view through the guest proxy, which replaces the guest's SigV4 Authorization (dummy keys or none) with the token. /s3/job/<alias>/<key> maps to the key below the area's prefix, and /s3/job/<alias> alone is the area as a bucket: its listings get the key prefix in their query and continuation token, and the S3 documents answering them, a created bucket's Location and error Resources name the area back by its alias. An area below its bucket takes only reads at the bucket level, since a bucket write or a batch delete would act outside it.
The proxy now keeps an answer's Content-Length whenever the body is passed on as it came, and never maps a HEAD's body; only a rewritten document loses it. The length was dropped with the request's, so a HEAD reached the guest without the object's size, which the AWS CLI reads before a download.
A bucket GET through an area below its bucket is now refused when its query names a listing parameter with any other, as is a delimiter on ?uploads, and parameter names are compared decoded, as the registry takes them; a configuration's query is still kept. Such a query was kept as sent, so a listing with an extra parameter reached the real bucket without the area's key prefix.
A PUT under If-Match, If-None-Match, If-Unmodified-Since or a copy source's x-amz-copy-source-if-* is sent once: repeated after a write that reached the registry, it would fail its own precondition and answer 412 for a write that succeeded.
Writing a key's repository is now what deletes it: a caller that may write but not read the repository deletes, and one that may do neither still finds the key absent. On a write-only grant the object was taken for absent, so DeleteObject answered 204 and DeleteObjects <Deleted> while it stayed.
A SigV4 request carrying an x-amz-* header its SignedHeaders leaves out is refused with 403 AccessDenied naming it in HeadersNotSigned, as S3 refuses it: anything on the way could have added the header. The test signer now signs every x-amz-* header a request carries, as SDKs do.
The query is now parsed once into decoded names and values, with the signature's decoding, and every dispatch and operation reads that; a malformed escape is 400 InvalidArgument, and multipart parameters no longer take a + for a space. Dispatch matched raw names while the signature decoded them, so a name sent escaped was signed as one operation and served as another.
A key DeleteObjects fails to delete now gets an InternalError of its own, logged with the request id, and the batch goes on. The whole request answered 500 after the keys before it were deleted, so the client could not tell what was gone.
The server now refuses at startup an [oidc] public_url whose host is below s3_host, and s3_host with s3 = false; the help and design say the registry's own name must not be below it. Every <label>.<s3_host> is served as a bucket, so a registry whose own name sat there had its /v2/, /login and /browse answered as S3.
The uploads listing now holds its prefix and markers to a bucket listing's rule (400 InvalidArgument), and a request document holding a character XML 1.0 does not allow, literal or in CDATA, is 400 MalformedXML. Echoed unchecked into a ListMultipartUploads answer or a DeleteObjects key, such a character made XML no client can read.
ListBuckets now shows what HeadBucket admits: a bucket the caller may read a directory of. It showed a bucket only to a caller that may read all of it, so a key scoped to files/team/proj/* found no team bucket to start from, though HeadBucket and the listings admitted it.
registry_s3 adds a namespace three components deep with an alias, then has a job write, read, list and probe an object at /s3/job/<alias> unsigned through the guest proxy, and be refused another alias and a bucket its token does not cover. It runs where the registry scenario does.
`GET /s3/<bucket>?versions` lists the bucket as S3 lists one that never had versioning on: the same walk as ListObjects, each object the one `null` version of its key and the latest, no delete markers, paged by `key-marker` with the version id marker taken and echoed. That `null` version names the object itself when sent back: on a DELETE, GET or HEAD with `?versionId=null`, in a `Delete` document entry and as a copy source; any other version id stays 501 NotImplemented. Ceph's s3-tests empty every bucket through the listing and those deletes, so without them the whole suite errors in setup.
Every /s3/ answer now carries x-amz-id-2, the request id again: S3 sends this second id on every response, and SDK test suites assert it is there.
An `Authorization` header of a scheme `/s3/` does not take (SigV2, say), or with no scheme at all, is 400 `InvalidArgument`, as S3 answers a signature version it does not support, rather than 403 as a credential it refused. Basic, Bearer and SigV4 are resolved as before.
A request whose headers ask for server-side encryption (SSE-S3, SSE-KMS or SSE-C, on a read or a copy's source too), object tags, an ACL other than the owner's, a grant, a storage class other than STANDARD, a website redirect or object lock is now 501 NotImplemented, as the PUT of that configuration already is, rather than taken with the header dropped. Until now a client that asked for its object to be encrypted under its own key was answered 200 while the bytes were stored and served plain. A storage class S3 does not have is 400 InvalidStorageClass; an empty x-amz-tagging, no tags, asks for nothing and is taken.
A write's x-amz-checksum header or trailer is now checked in every algorithm S3 has that this registry computes: CRC32C, CRC64NVME, SHA-512 and MD5 join CRC32, SHA-1 and SHA-256, with the crc crate already in the tree. One checksum may be sent, as S3 has it; one in S3's XXHASH family, or any other algorithm, is 501 NotImplemented rather than taken unchecked; a checksum that does not hold is refused in S3's words, naming the algorithm, and an x-amz-sdk-checksum-algorithm naming another algorithm than the header's or the trailer's is 400 InvalidRequest. The checksum that held is kept on the object's manifest and is the object's from then on: sent back on a read asking for it, as ChecksumAlgorithm and ChecksumType in listings, in its attributes, and by a copy. A completed multipart upload keeps the whole-object checksum that held in the same way; one that is neither a full checksum nor a composite one is 400 InvalidRequest. Clients that compare a listing's checksum algorithm with what they uploaded (versitygw's suite crashes without it) now see it.
GET and HEAD take ?partNumber=<n>, as the AWS CLI and SDKs download a large object: the bytes of that part as it was uploaded, 206 with its Content-Range and x-amz-mp-parts-count. A completed multipart upload keeps its parts' sizes on the object's manifest for that, a copy carries them, and GetObjectAttributes answers ObjectParts from them. An object written whole is its one part; a part there is not is 416 InvalidPartNumber, as S3 answers.
A refusal decided before the caller is known now reads up to 8 MiB of the body before answering, the AWS CLI's part size, where 64 KiB left any larger refused upload with a reset instead of the answer. A client holding its body back on Expect: 100-continue is refused before the 100 would go out, so it sends no body and reads the refusal first, as it would from S3.
PutObject answers with x-amz-object-size, as S3 does: the Go SDK reads it back without checking, and versitygw's conditional-delete test crashed on its absence.
DeleteObject takes If-Match, x-amz-if-match-last-modified-time and x-amz-if-match-size, and each <Object> of a DeleteObjects document its ETag, LastModifiedTime and Size: the object is deleted only when every condition given holds, checked under the namespace lock against the object the delete removes. A condition that does not hold is 412 PreconditionFailed (that key's error in a batch), one on a key with no object NoSuchKey, as S3's conditional deletes answer, and a value that does not parse 400. A condition beside a version id is 501, S3 applying them to the current object only. AbortMultipartUpload takes x-amz-if-match-initiated-time: an upload initiated in another second is kept and the abort answers 412; a time still to come is no condition, as S3 takes it.
A completion or an abort now holds the upload's directory locked (flock) from looking the upload up until it is gone. Completions of the same upload sent at once, as SDKs and versitygw's racey test send them, used to read parts another completion was removing and answer 400 InvalidPart; the one that waits now finds the upload completed.
A completed upload now leaves its directory holding completed.json (the parts it named, the object's ETag, the SHA-256 of its bytes and the checksum it answered with) instead of going at once. The same completion sent again, as an SDK retries one whose answer it lost, is 200 with the same result while the key still holds that object; other parts, or an object written over or deleted since, stay 404 NoSuchUpload. The record goes with the idle uploads once past the gc's grace window, and status no longer counts it as an upload in progress.
ListObjects names a NextMarker only beside a delimiter, as S3 does: without one, the page's last key is the next marker, as SDK paginators take it. ListObjectsV2's start-after and continuation-token sent to it are 400 InvalidArgument. ListObjectsV2 sent both a token and start-after starts after the later of the two, so a paginator keeping its start-after beside the token sees no key twice; an empty token is none, echoed empty. A listing echoes Prefix only when one was sent.
…does A key's uploads are listed in the order they were initiated rather than by id as text. An upload-id-marker is taken as S3 takes it: the page starts after that upload, which has to be one of the first key from key-marker on (else 400 InvalidArgument), and without a key-marker it is ignored. NextKeyMarker and NextUploadIdMarker name the page's last upload whether or not more follow, and Prefix is echoed only when one was sent.
A path with a malformed percent escape, a component that is not UTF-8 or holds a control character once decoded, or `..` components climbing above the root (`\` separating them too) is 400 InvalidURI, answered before the credential is looked at, as S3 parses the URI first. It was InvalidArgument or InvalidBucketName once authenticated, and 403 before. A `..` that stays below the root is still refused as a key component.
One table (s3/error.rs) maps each code to its status and AWS's message, as versitygw 1.10.0's s3err mirrors them; handlers name a code, and add the elements AWS sends with it (BucketName, Key, UploadId, ArgumentName and ArgumentValue, Condition, Header, Size and MaxSizeAllowed) where the detail of the old messages now goes. InvalidArgument and InvalidRequest keep the specific sentence AWS has for the case (copy sources, metadata directives, counts, checksum values, listing markers), malformed Authorization headers say AWS's reason in AWS's order, and an unsigned header refusal names all of them. EntityTooLarge is 400, as in S3. Refusals S3 never makes (key charset, store clashes, presigned URLs, read-only store) keep a message saying why. This reverses the earlier choice not to match versitygw's wording.
A request about a bucket that does not exist is 404 NoSuchBucket before anything else is said of it: an unserved subresource (501), a malformed argument, a listing's parameters, GetObjectAttributes, a delete of a key in it, DeleteObjects. A write still makes its bucket, and a request naming an upload is NoSuchBucket only when the upload is not there either. A read of an object (GetObject, HeadObject, a part of one) stays NoSuchKey, as in an empty bucket: OpenDAL takes NoSuchBucket for a broken configuration rather than a miss, so sccache, whose first request is a read, would not start on a bucket its first write is to make. An unserved operation on an object (tagging, retention, legal hold) is NoSuchKey when the object is not there. As HeadBucket, this is said only to a caller that may read in the bucket; any other is answered as before (403, or 501), so it learns nothing of what exists.
- An aws-chunked body whose framing breaks off or is malformed is IncompleteBody (`The request body terminated unexpectedly`), one holding other than its declared length IncompleteBody with S3's Content-Length sentence, a missing or unreadable decoded length 411 MissingContentLength, and aws-chunked beside UNSIGNED-PAYLOAD 400 InvalidArgument. - An upload that is not there is NoSuchUpload before its part number, checksums or copy source are looked at, and before a completion's document is read; ListParts refuses a count that is no number first, a negative one after the upload. A part copy range past the source's end and a completion PartNumber below 1 are InvalidArgument in S3's words. - A read of a key the store cannot name, or a copy from one, is NoSuchKey; a key of slashes only is BadRequest; an empty versionId is InvalidArgument; GetObjectAttributes with no attribute InvalidRequest. - SigV2 is InvalidRequest, an x-amz-content-sha256 S3 does not take is InvalidArgument before the signature, two checksum headers are refused whatever they hold, an unknown tagging directive is InvalidArgument. - PutObject, CopyObject and CompleteMultipartUpload take one condition, If-None-Match: * or If-Match: <etag>: both at once, or If-Match: *, is 501 NotImplemented naming the header, as an If-None-Match other than * already was. An If-Match with no object there is 404 NoSuchKey, still decided under the namespace lock the tag is written under, and a condition that fails is 412 naming it in Condition.
The XXHASH checksums are checked, echoed, kept and listed as the other algorithms are, rather than refused with 501: base64 of the unseeded big-endian digest, as the CRCs, computed with xxhash-rust (no dependencies of its own). With every algorithm S3 has now computed, a checksum header, SDK algorithm or trailer naming any other is 400 InvalidRequest in S3's words, no longer 501.
CreateMultipartUpload takes x-amz-checksum-type beside the algorithm, refuses a type or algorithm S3 does not have, a type without an algorithm and a type the algorithm cannot make, and records both with the upload. Each part of such an upload carries a checksum in its algorithm, or has it computed for a FULL_OBJECT upload, and is recorded with it. The completion is held to them as S3 holds it: the checksum type must be the upload's, each part must name the checksum it was stored with (InvalidArgument, InvalidPart, BadDigest, InvalidRequest in S3's words), and one final checksum at most. A FULL_OBJECT upload's checksum is computed as the parts are assembled, checked and kept.
CopyObject with x-amz-checksum-algorithm gives the object a checksum in that algorithm, computed from the source's bytes unless the source kept one in it, and an algorithm S3 does not have is refused before a copy onto itself is. UploadPartCopy into an upload created with a checksum computes the upload's algorithm as the part is copied and records it. Both results now name the checksum.
A PutObject that carries no checksum gets a full-object CRC64NVME, computed in the pass that already hashes the body, echoed, kept and listed as a sent one is, as S3 does since 2025. A multipart upload created without a checksum algorithm gets the whole object's CRC64NVME from the assembling pass, and the final checksum its completion is sent is passed over, as S3 passes it over. Objects written otherwise (WebDAV, vk registry put) still answer their SHA-256 rather than be read again. The CRC32C and CRC64NVME tables now use sixteen lanes, since every write computes one.
An upload created with a COMPOSITE checksum gets, at completion, its algorithm's checksum of the parts' checksums, <base64>-<parts>, checked against the final header with or without its count, and kept. The parts' checksums are kept with the object too, so a part read with checksum mode answers its own and GetObjectAttributes lists them in ObjectParts. ListParts names each part's checksum and the upload's algorithm and type, as ListMultipartUploads does for each upload. Reads, listings and attributes name each object's ChecksumType as it was made.
…efore another Every chunk of an aws-chunked body that holds data, but the last, holds at least 8192 bytes, as S3 requires: a shorter one followed by more data is 400 InvalidChunkSizeError in S3's words, naming the chunk and its size. SDKs frame in 64 KiB chunks, so only hand-framed bodies change.
…r 5 MiB CompleteMultipartUpload holds every part it names but the last to S3's 5 MiB minimum: a smaller one is 400 EntityTooSmall in S3's words, naming the part, its ETag, its size and the minimum, and the upload is left as it was. Parts of any size are still taken as they are uploaded, as S3 takes them. SDKs, the AWS CLI and sccache upload parts of 5 MiB or more, so only hand-made uploads change.
…tial As S3 routes a request before it authenticates it, query parameters that conflict or do not go with the method or resource, and a copy source where nothing is copied, are refused before the credential is looked at, in S3's words: 400 InvalidArgument (uploads with uploadId, a misplaced x-amz-copy-source, partNumber without uploadId) or InvalidRequest (?uploads or ?versions on a key), and 405 MethodNotAllowed naming the method and resource type, with Allow, for a POST of the service or of a bare object, a PUT naming an upload but no part, and ?torrent. They were 403 to an unsigned request.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.