Repository navigation
feat(vanir): add vanir-sandbox service, gitter commit-pack relay, and input/output hardening - #6403
Draft
another-rex wants to merge 1 commit into
Draft
another-rex wants to merge 1 commit into
another-rex wants to merge 1 commit into
Conversation
… input/output hardening
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
(AI-assisted PR)
This is PR 1 of 2 to isolate Vanir signature generation (
vanir_signatures) into a dedicated, zero-privilege Cloud Run Gen2 sandbox (vanir-sandbox) fed by shallow Git commit packs relayed fromgitter.Keeping this rollout split into two PRs avoids a deployment chicken-and-egg:
gitterPOST /commit-pack,--git-instead-ofssupport ingenerate_signatures.py, the newgo/cmd/vanir_sandboxCloud Run service, input sanitization & output pinning/validation ingo/internal/vanir, thevanir-sandboxDocker target, Terraform resources, and Cloud Build / Cloud Deploy wiring. The existing GKEvanir-signaturesCronJob and Docker target remain backwards-compatible untilvanir-sandboxis provisioned (andSanitizeForVanir+ValidateSignaturesimmediately protect the existing in-containerPythonGenerator).vanir-signaturesto a distroless Go image pointing atVANIR_SANDBOX_URL+GITTER_HOSTand moves the CronJob off thehighendnode pool.Details
1.
gitterShallow Commit-Pack Relay (POST /commit-pack)CommitPackRequesttogo/internal/gitter/pb/repository/repository.protoand regeneratedrepository.pb.go.CreateCommitPackandPOST /commit-packingo/cmd/gitter/:prepareURL, caps requests at1..64validated hex commit hashes, and verifies each object is a commit (git cat-file -t).repoLock.RLock()without mutating the cached repo's.git/configby configuringremote.origin.uploadpackon the temporary bare clone.--filter=blob:none --depth=2 --no-tags, materializes only touched diff blobs viaformat-patch --stdout -1, detachesremote.origin, enablesuploadpack.allowFilter/uploadpack.allowAnySHA1InWant, and returns a.tar.zstarchive (capped at 128 MiB; retries once withForceUpdate: trueon missing commits before returning404).GetCommitPackandErrPackTooLargetogo/internal/gitter.2. Python Helper
--git-instead-ofsSupport--git-instead-ofsflag togo/cmd/vanir_signatures/generate_signatures.pyaccepting a JSON string or file path of[[src_url, local_file_url], ...]pairs, validating non-empty string pairs and passing an immutabletuple[tuple[str, str], ...]toExtractorConfig.git_instead_ofs.3. Untrusted Mode B Sandbox Server (
go/cmd/vanir_sandbox)GET /healthzandPOST /v1/generate(64 MiB request cap, single-request mutex matching Cloud Runconcurrency=1, optional-max-requestsgraceful exit).CAP_SETUID/CAP_SETGIDandCLONE_NEWNET/CLONE_NEWIPC/CLONE_NEWUTSat startup.0711) andos.Mkdirper request; unpacks.tar.zstpacks safely in Go (klauspost/compress/zstd+archive/tar) with decoder memory/window caps, entry count cap, and 256 MiB total size cap, rejecting traversal/symlinks/hardlinks/special nodes and setting0755/0644owned by0:0.generate_signatures.pyunder a unique non-root UID (10000..59999), strippedcmd.Env(GIT_ALLOW_PROTOCOL=filewhen all repos are packed,safe.directory=*), isolatedHOME/TMPDIR, and empty network namespace (CLONE_NEWNET) whenAllowNetworkis false.-reap-uidrunningkill(-1, SIGKILL)as that non-root UID) before atomically reading pre-createdoutput.json(O_NOFOLLOW|O_NONBLOCK|O_CLOEXEC), then sweeps any stray UID temp files and removes request directories.4. Go Orchestrator Hardening (
go/internal/vanir&go/cmd/vanir_signatures)SanitizeForVanir: strips unused free-text fields (Summary,Details,References,Credits), strips non-canonicalAndroid/Pixel/Wearentries while preserving index alignment, allowlists canonical Android fix URLs, validatesRange_GITURLs (blocking non-HTTP(S), credentials,@, option prefixes, internal/loopback/RFC1918/link-local/CGNAT/alternate-IP hosts), normalizesgit://github.comtohttps://github.com, and validates/capsfixedcommit hashes.ValidateSignatures: pins output strictly to the requested vulnerability ID (<id>-prefix), verifies affected index bounds and post-sanitization eligibility, caps at 256 signatures per affected entry and 2 MiB serialized JSON per record, and strictly validates VanirFunction/Linesignature schemas.SanitizeForVanir+ValidateSignatures+ ID pinning directly intoPythonGenerator(with bounded stderr capture andO_NOFOLLOWoutput reads).HTTPSignatureGenerator(idtoken.NewClientOIDC auth, 128 MiB byte-bounded LRU + cancellation-isolatedsingleflightcommit-pack cache, graceful skips on 404/403/413 repos) and-sandbox-url/-gitter-hostflags ingo/cmd/vanir_signatures/main.go.5. Docker, Terraform, Cloud Deploy, Cloud Build, & Docs
vanir-sandboxtarget ingo/Dockerfile(reusingvanir-python-deps; existingvanir-signaturestarget left unchanged for PR 1).deployment/terraform/modules/osv/vanir_sandbox.tf(zero-IAM-rolevanir-sandboxSA, deployeractAsbinding, Cloud Run Gen2 service with no VPC access,concurrency=1,4 vCPU / 8Gi,max_instance_count=30,roles/run.invokerrestricted toworker_service_account_email) and exportedvanir_sandbox_urioutputs.deployment/clouddeploy/vanir-sandbox/manifests and wired build/release/promote/tag steps indeployment/build-and-stage.yamlanddeployment/deploy-prod.yaml.Makefile(vanir-signatures-tests) andAGENTS.md.Testing
go/cmd/gitter/commit_pack_test.go(standalone pack verification after deleting source repo, unrelated blob exclusion, option-injection rejection, 404 handling)go/internal/gitter/http_test.go(TestGetCommitPackincluding 400/403/404/413 error mapping)go/internal/vanir/sanitize_test.go(SanitizeForVanir,ValidateSignatures,HTTPSignatureGeneratorcaching/skipping/index-remapping/singleflight cancellation isolation)go/cmd/vanir_sandbox/main_test.go(end-to-end HTTP test with real.tar.zstcommit pack,--git-instead-ofsfetch, process reaping, symlink/FIFO output rejection, safe tar extraction checks)make vanir-signatures-tests— PASScd go && go test -race -count=1 ./internal/vanir/... ./cmd/vanir_signatures/... ./cmd/vanir_sandbox/... ./internal/gitter/...— PASScd go && go test -race -count=1 ./cmd/gitter -run TestCommitPackHandler— PASScd go && go build ./...— PASS./tools/lint_and_format.sh(pylint10.00/10,yapf,gofmt,golangci-lint0 issues,terraform fmt) — PASSvanir==1.1.0+generate_signatures.py+GIT_ALLOW_PROTOCOL=fileagainst a read-only unpacked commit pack — PASSagy