Skip to content

Speed up package deployment with verified caching and parallel OSS checks - #453

Open
Harrison-Xu wants to merge 1 commit into
mainfrom
codex/faster-package-deployment
Open

Harrison-Xu wants to merge 1 commit into
mainfrom
codex/faster-package-deployment

Conversation

@Harrison-Xu

Copy link
Copy Markdown
Collaborator

Portal submissions could spend almost seven minutes retrying a staging filename that omitted the author's prefix. After skipping unchanged uploads was added, serial OSS checks still took about nine minutes for 763 unchanged files.

This change uses the manifest's existing org staging URL, falls back immediately on a missing candidate, and removes the duplicate download entry point. Package promotion and collection now share SHA256-verified bytes, restore a manifest-keyed cache, prune inactive entries, and download missing packages with four workers.

OSS keeps the existing CRC64 comparisons and validated withdrawal cleanup, while checking/uploading pool files with eight workers. Any pool failure blocks all catalog updates; Packages precedes Release and the CN registry is published last.

Validation:

  • 36 pipeline/cache/download regression tests passed locally.
  • 17 integration tests passed using real ossutil 1.7.7 against a local OSS server, including worker limits, unchanged uploads, withdrawals, and failure barriers.
  • Shell syntax, Python compilation, YAML parsing, and diff checks passed.
  • Local simulated-latency comparison, 37 unchanged files with 80 ms HEAD delay: 9.50 s with one worker versus 2.79 s with eight workers (3.41x). Production timing still needs a deployment run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant