Repository navigation
Requirements for Deploying dstack OS on GCP #125
Description
One of the core design principles of Phala's DStack implementation is the separation of system and application software.
In this implementation, DStack developers focus on developing and releasing system images, while App developers focus on developing applications.
Another option is to package each application with a unique OS Image.
An important distinction between these two approaches is that when system images and application images are released separately, their MRs are also separate. When a system image is released, its MR can be calculated in advance, allowing the system image code to be audited independently.
If application code is bundled into the system image, the MRs become mixed together. When auditing application code, the system layer code must also be audited, which is not acceptable for most simple applications.
To deploy DStack OS on GCP or Azure, we need to address the following requirements:
- Change the current system image from separate files to a packaged disk image format for easier deployment.
- Design a new method to transfer application compose files and configuration files.
Currently, DStack's CVM uses shared directories to transfer application compose files and configuration files. We could consider changing this to download from GCP or Azure storage buckets, or from IPFS. This would transform the deployment process to first upload application configurations to a network storage, then use Cloud API to set the file's network address in VM metadata, and after VM startup, retrieve the file's network address from metadata and download the files. - Add GCP environment MR calculation logic to the dstack-mr command.
Activity
kvinwang commentedon Mar 4, 2025
cc @h4x3rotab
h4x3rotab commentedon Mar 8, 2025
Thanks to @heytdep, who brought the idea and implemented a modularize dstack with GCP support and giving a lot of valuable feedback to us.
heytdep commentedon Mar 11, 2025
@kvinwang @h4x3rotab this sounds good, I agree that we need a distinction between system and app-layer. As you say, custom os images that embed application logic will xor app logic and system measurements which I agree we generally not want to happen.
Regarding the requirements:
2. there's a lot of ways we can go about this, and inferring logic at vm-startup is what gives us the most control. Some alternative approaches could be to have an app-layer manager embedded in system that serves a port to accept vm metadata, once that happens measure the metadata into rtmr[3] and then shut down the listener. Depending on the requirements and if we can have build reproducibility for app-layer we might even not want to virtualize.
3. not sure what you mean here with gcp mr calculation, but if we launch gcp cvms without the shielded vm features then the VF should follow standard implementation. If in the future we want to unify attestation across platforms then supporting the tee<>vtpm attestation might also be worth it, but it's not strictly related to gcp.
(btw, I'm open to helping on this if needed)
kvinwang commentedon Mar 12, 2025
An agent listening for commands in the CVM could function if it can properly manage these scenarios: 1. After a system reboot, the app can automatically recover; 2. Handling of the app code update.
We have a command dstack-mr which can be used to calculate the MRs(MRTD, RTMR[0-2]) for Dstack OS images.
However, the MRs would vary based on different Virtual Firmware versions, hypervisors, or different virtual hardware configurations. So, if we add support for the dstack image to GCP, the dstack-mr needs to be updated.
Actually, I don't know any tech details about GCP, so have no idea about how tee<>vtpm attestation can be involved.
heytdep commentedon Mar 19, 2025
agreed
Ah that makes more sense now. I don't know exactly how gcp works under the hood too (they don't seem to provide much info about that tbh). The tee<>vtmp attestation was just to say that it happens on GCP when they use a different VF spec (or at least I assume they do) if the shielded vm features are on; I didn't take into account VF versioning tho.
h4x3rotab commentedon Apr 4, 2025
This one looks simple.
Seems like we need to introduce a remote config repo. Does IFPS link or http link but with a hash work?
As talked offline, @fnerdman do you have some suggestions or code pointers for this?
fnerdman commentedon Apr 7, 2025
It's all right here: https://cloud.google.com/confidential-computing/confidential-vm/docs/verify-firmware#intel-tdx
Note that this is for GCE, on gcp baremetal, measurements will be the same as on vanilla baremetal
[-]Requirements for Deploying DStack OS on GCP[/-][+]Requirements for Deploying dstack OS on GCP[/+]h4x3rotab commentedon Jun 29, 2026
Scoping note as part of the SEV-SNP issue triage: this is the cloud-deployment / packaging track, not SEV-SNP-specific work. The three requirements here — packaged disk-image format, app compose/config transfer without a host shared dir, and cloud-environment MR calculation in
dstack-mr— apply to Intel TDX on GCP/Azure just as much as to SNP, and they predate the SNP work.Keeping this open and separate. The SEV-SNP cloud backends (GCP/Azure provider-vTPM, AWS VLEK) are tracked in #713 as distinct backends and will consume this track's plumbing rather than the other way around. Cross-links: #713 (SNP design/backends), #744 (GCP support-matrix, resolved).