Skip to content

macOS guests 1/6: VZ runtime, machine images and instance lifecycle - #498

Open
chruffins wants to merge 1 commit into
mainfrom
spike/macos-guests
Open

chruffins wants to merge 1 commit into
mainfrom
spike/macos-guests

Conversation

@chruffins

@chruffins chruffins commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

PR 1 of 6. Experimental macOS guests on Apple silicon through the existing image and instance APIs, using Virtualization.framework via the existing vz-shim. Base: main.

  • Images: cmd/import-macos imports a stopped macOS VM bundle (disk, auxiliary storage, platform config) as a darwin/arm64 machine image, using APFS clones.
  • Instances: create, start, stop, delete, get, list and wait. Each instance gets cloned boot disk and auxiliary storage. CPU and memory default to the image; overrides are validated (at least 2 vCPUs and 4 GiB in whole MiB, within VZ host limits). NAT networking; the IP comes from the host's DHCP leases.
  • Updates: expiration/TTL, and restart policy always or never (on_failure is rejected: there is no workload exit status yet).
  • Identity: one running instance per machine identifier, enforced in hypeman admission and by a lock in the shim. Apple documents concurrent VMs sharing an identifier as undefined behavior.
  • Observability: VM-process CPU/RAM stats (shim plus the VZ XPC process) and VMM logs.
  • Shim lifecycle: a SIGKILLed shim reads as Stopped and its resources are released, so restart policy works. If the API dies mid-boot or mid-restore, start and restore return 409 while the old shim still runs, and stop/delete find it by its socket path even when its PID was never recorded.
  • macos_only mode: a server that runs only macOS guests skips Linux kernel/initrd downloads and ingress, and refuses Linux creates.

Size

About 950 lines of logic. Of the 4,000 added lines in the diff, 1,554 are non-test, non-doc, non-generated Go. Removing comments, blank lines and imports leaves about 1,215. A further ~265 are mechanical:

  • one-line hooks and field plumbing in shared files (~150)
  • the same "unsupported on macOS" early return in snapshot, standby, fork, restore, memory controllers, ingress, exec and builds (~45)
  • existing blocks re-indented under if !MacOSOnly / if stored.MacOS == nil, struct-tag realignment, and a small legacyLayout refactor (~50)
  • non-darwin stubs (~25) and API error-code mapping (~15)

The 1,554 is exact; the split of the rest is a per-hunk classification, about ±30 lines.

 lib/instances/                    macOS lifecycle                  ~321
 lib/images/ + cmd/import-macos    macOS image import               ~235
 cmd/vz-shim/                      boot a macOS VM                  ~134
 lib/vm_metrics/                   stats without /proc              ~100
 cmd/api/                          create defaults, macos_only      ~ 93
 lib/hypervisor/                   macOS boot-mode validation       ~ 41
 lib/hypervisor/vz_shim_pids*.go   find a shim by socket            ~ 24
  1. Lifecycle (lib/instances/macos.go, macos_darwin.go; small parts of logs.go, query.go)
    • Create admission: rejects Linux-only options and resolves CPU/memory from the image plus overrides.
    • Boot disk: APFS clone of the image's disk and auxiliary storage (cloneMacOSStorage).
    • Networking: the guest IP comes from the host's DHCP lease file by MAC (macOSLeaseIP).
    • Identity admission: one running instance per machine identifier and MAC (checkMacOSIdentityAvailable).
    • Killed shim: a refused socket plus a dead recorded PID reads as Stopped and releases resources (macOSShimGone, macOSStateWithoutShim).
    • Logs: VMM logs come from the shim log; app logs return 501.
  2. Image import
    • macos_bundle.go validates a fixed-layout VM bundle (disk, auxiliary storage, config) and rejects unknown files and links.
    • macos_import_darwin.go clones the bundle into image storage and writes metadata; cmd/import-macos is an 18-line wrapper.
    • Shared files: MacOSImage and its Validate, darwin/arm64 platform acceptance, the raw-disk filename in storage, and image resolution at create.
  3. Shim (cmd/vz-shim/macos_*.go)
    • Builds the VZ configuration: macOS platform (hardware model, machine identifier, auxiliary storage), boot loader, display, keyboard, NIC and disk.
    • macos_identity.go holds a file lock on the machine identifier for the shim's lifetime, the second guard against two VMs sharing an identity.
  4. Stats (collector_darwin.go, vm_metrics/manager.go)
    • CPU and memory come from ps, since macOS has no /proc.
    • VZ runs the guest in a separate XPC process, found by its open disk path and added to the shim's numbers.
  5. API and config
    • cmd/api/api/macos.go: RecordCreateFields and ignoreLinuxCreateDefaults, so injected vcpus: 2, 1GB and 10GB don't override the image; a rejection reason for ingress rules targeting a macOS guest.
    • instances.go: macOS skips default bandwidth and disk I/O limits.
    • config.go: macos_only (VZ on Apple silicon only) and listen_address.
  6. Hypervisor validation
    • Adds BootModeMacOS and its config.
    • starter.go validates one writable raw disk, at least 2 vCPUs and 4 GiB in whole MiB within host VZ limits, and no balloon, hotplug, PCI or Rosetta. The shim config carries the machine identity.
  7. Shim lookup (vz_shim_pids*.go)
    • Finds a vz-shim by the socket path in its command line, so stop and delete can kill a shim whose PID was never recorded (for example after an API crash mid-boot).

Linux code touched

Linux behavior is unchanged; these are dispatch hooks.

  • lib/instances: create, start, stop, delete, query (state derivation), logs, update, fork/snapshot/standby admission, admission allocations, process identity.
  • lib/hypervisor: a macOS boot mode in VM config and validation; the QEMU profile rejects it.
  • lib/images: the MacOSImage type, platform parsing, boot storage paths, tagging.
  • cmd/api: macos_only config and startup gates; create, ingress, exec and image guards.
  • Create defaults: the OpenAPI validator writes Linux defaults (vcpus: 2, size: 1GB, overlay_size: 10GB) into every create body. On Darwin hosts only, a middleware records which fields the client actually sent (bodies over 1 MiB pass through unrecorded), so a macOS create ignores only the injected defaults.
  • lib/vm_metrics, lib/resources, lib/guestmemory, lib/paths: macOS stats hook, and skipping balloon and memory reclaim.

Not supported in this PR

Volumes, env/commands, credentials, egress policy, bandwidth and disk I/O shaping, auto-standby, ingress, GPU/PCI devices and memory hotplug are rejected. Guest-agent features arrive in #500; snapshots and clones in #506. Current status is in docs/macos-support.md.

Testing

  • Every layer builds, vets and passes gofmt on Linux, and type-checks for darwin/arm64. The darwin check stubs cgo; native runs are below.
  • Linux tests for touched packages fail the same set as main (tests needing KVM, mkfs.erofs or Docker Hub).
  • Native QA ran on Apple silicon (macOS 27.0.1 host and guests) across six runs plus a VZ restore spike. The simplification pass after run 6 has not been re-run natively yet. This PR's paths covered there: import, create/start/stop/delete, CPU/RAM overrides including explicit vcpus: 2, restart policy after killing the shim, identity and MAC admission, and API crash recovery.

Review focus

  • The create-defaults middleware in cmd/api/api/macos.go and its registration in cmd/api/main.go.
  • The state-derivation hook for a killed shim in lib/instances/query.go (macOSShimGone).
  • The darwin shim lookup in lib/instances/process_identity.go: when no live PID is recorded, it finds the shim by the control socket in its command line.

Stack

Each PR is one commit on the one below it. macOS logic lives in its own files (macos*.go, *_darwin.go, exec_lifecycle.go, port_forward.go); shared Linux files get small hooks such as if stored.MacOS != nil.

  1. macOS guests 1/6: VZ runtime, machine images and instance lifecycle #498 VZ runtime, machine images and instance lifecycle
  2. macOS guests 2/6: OCI machine-image integration #499 OCI machine images
  3. macOS guests 3/6: Darwin guest service, readiness and exec #500 Darwin guest agent, readiness and exec
  4. macOS guests 4/6: snapshots, schedules and stopped forks #506 snapshots, standby and stopped clones
  5. macOS guests 5/6: guest-port forwarding #514 guest-port forwarding
  6. macOS guests 6/6: experimental machine-image builds #503 machine-image builds (not needed for v0; can land later)

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
-->

✱ stlc build

✅ go code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

116 files generated at 9c3d4f7 (pushed)

go get github.com/kernel/hypeman-go-staging@9c3d4f75184af812e24f9580d4e1f8436966c533
✅ python code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

232 files generated at 77843ba (pushed)

✅ typescript code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

138 files generated at 9d02967 (pushed)

Diagnostics: ❗ 0 new / 1 total error, 💡 0 new / 5 total note
LevelCodeMessageTargets
Build metadata
Buildbd_76Xtqqhy-smoky-pantry
Timestamp2026-10-14T20:13:52.838Z
stlc8413509
Spec hash347a7be06316
Config hash659c3687c3f0

This comment is auto-generated by stlc and is kept up to date as you push.
If you push new commits, re-run this workflow to update this comment.
Last updated: 2026-10-14 20:14:22 UTC

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 4849059. Configure here.

Comment thread lib/instances/macos.go
meta, err := m.loadMetadata(entry.Name())
if err != nil {
return fmt.Errorf("check Mac identity admission: %w", err)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Identity admission fails on metadata errors

Medium Severity

checkMacOSIdentityAvailable treats any loadMetadata failure as fatal, including missing metadata for an in-progress create. A concurrent Linux or macOS create that has already made a guest directory but not yet written metadata causes an unrelated macOS start or create to fail admission.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 4849059. Configure here.

}
if exit, ok := e.(*exec.ExitError); !ok || exit.ExitCode() != 1 {
return nil, fmt.Errorf("cannot verify storage is closed: %v", e)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lsof probe leaks process groups

Medium Severity

The import lsof probe runs through exec.CommandContext without a dedicated process group and without SIGKILLing that group on every completion path. Context cancel only stops the wrapper, so lsof descendants can leak after success, ErrWaitDelay, or a wrapper that exits first.

Fix in Cursor Fix in Web

Triggered by learned rule: Subprocess binary probes must use process groups and kill on all completion paths

Reviewed by Cursor Bugbot for commit 4849059. Configure here.

@chruffins chruffins changed the title Experimental local macOS guests through VZ and instance API macOS guests 1/6: native VZ runtime and instance lifecycle Oct 8, 2026
@chruffins

Copy link
Copy Markdown
Contributor Author

Live QA of the reviewed combined PR1–3 sources plus PR4 patches on Apple silicon passed normal authenticated HTTP create from a digest-pinned, previously HTTP-pulled installed macOS OCI artifact (registry offline). Verified actual macOS 27.0.1/26A434 cold boot over pinned-key SSH, inherited 4CPU/8GiB template defaults, duplicate-identity HTTP409, live-guest API restart recovery with unchanged guest boot time, HTTP stop/start with fresh boot time and durable guest-only marker, and unchanged full SHA256 of the immutable materialized base disk+aux afterward. Stop/start here used system-agent-disabled fallback, not verified graceful root-agent shutdown. QA used an isolated data dir/slot; original instance/API/storage were untouched and the QA VM/API ended stopped.

Runtime issue found: the long QA data directory generated an overlong AF_UNIX socket path. VZ reported VM started, then control listener failed listen unix .../vz.sock: bind: invalid argument, resulting in HTTP500 after ~30s. VMM exit and complete create rollback were confirmed. Retrying via a short private data-dir alias passed. Recommend validating the resulting control/vsock socket lengths before VM launch and returning a clear admission error rather than starting then failing. This is not fixed by the workaround and may affect other Unix-socket consumers too.

@chruffins

Copy link
Copy Markdown
Contributor Author

Simplification update at e6e6ab7: shared bounded complete-machine bundle decoding and one macOS request-default preparation; shared concrete create/start network + boot-config preparation. macOS preserves template MAC and avoids Linux allocation/proxy/config-disk dependencies; Linux rollback and request preservation have synthetic tests. Focused instance/API/image race tests passed 3 runs. Identity leases and stopped-storage/export requirements remain.

All PRs remain draft. This is source-level/synthetic validation, not a new live boot or production build proof. Default Codex independent review was attempted but is still blocked by authentication (HTTP401). Published atomically after checking reviewer heads with explicit per-ref force-with-lease; prior heads preserved locally.

@chruffins chruffins changed the title macOS guests 1/6: native VZ runtime and instance lifecycle macOS guests 1/6: VZ runtime, machine images and instance lifecycle Oct 11, 2026
@chruffins
chruffins added this pull request to stack #507 October 11, 2026 18:29
@chruffins
chruffins force-pushed the spike/macos-guests branch 5 times, most recently from 614d302 to c221190 Compare October 13, 2026 00:05
@chruffins
chruffins removed this pull request from stack #507 October 13, 2026 00:06
@chruffins
chruffins added this pull request to stack #508 October 13, 2026 00:06
@chruffins
chruffins force-pushed the spike/macos-guests branch 9 times, most recently from 8d8e740 to 0a99bcb Compare October 14, 2026 17:08
@chruffins
chruffins removed this pull request from stack #508 October 14, 2026 17:08
@chruffins
chruffins added this pull request to stack #515 October 14, 2026 17:10
@chruffins
chruffins marked this pull request as ready for review October 14, 2026 19:12
@hiroTamada
hiroTamada self-requested a review October 14, 2026 19:28
Experimental, administrator-provisioned macOS guests on Apple silicon through the
normal image and instance APIs. Unsupported operations are rejected explicitly.
Linux create, start and API paths are unchanged; macOS enters through hooks.
Per-operation status: docs/macos-support.md.

@hiroTamada hiroTamada left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reviewed at 93803d8. three findings:

  • P1 — lib/instances/macos.go:301–305, lib/instances/process_identity.go:147–149: discover live shims before treating an instance as stopped. if the API crashes after spawning the shim but before recording its PID, the shim can survive without a listening control socket: cmd/vz-shim/main.go:80–93 starts the VM before opening that socket. query.go:76–83 classifies a missing socket as stopped; the new macOSShimGone also treats a refusing socket plus a missing PID as stopped. StopInstance then returns success without termination (manager.go:685–693), and delete skips termination (delete.go:128) before removing storage/metadata. the new PID-discovery fallback is never reached through those shortcuts. please discover the shim before deciding it is stopped, require termination before destructive cleanup, and fail closed on discovery errors. add a regression for a live pre-listener shim with no recorded PID. this is a source-path finding, not a native crash reproduction.

  • P2 — cmd/api/api/macos.go:82–98: preserve explicit resource fields on oversized requests. bodies above 1 MiB pass through without field-presence tracking, but ignoreLinuxCreateDefaults treats missing tracking as proof that default-valued fields were omitted. reproduced with the actual validator and extracted helpers: the same JSON padded with whitespace loses explicit vcpus: 2, size: "1GB", and overlay_size: "10GB". a two-CPU request can therefore use the image's CPU count instead; invalid memory and unsupported overlay settings can be replaced with image defaults rather than rejected. please reject oversized creates with 413 or preserve explicitness instead of guessing omission. add a boundary test checking downstream field values, not just body pass-through.

  • P2 — lib/instances/macos.go:207–208: check network MAC uniqueness separately from machine identity. admission skips existing guests with a different machine identifier, even if both network-enabled guests have the same preserved MAC. the shim lock is also keyed only by machine identifier, and this path bypasses Linux MAC allocation. reproduced the admission gap with different identifiers, the same MAC, and an existing running guest; the same-identifier control was rejected. duplicate MACs could cause DHCP/connectivity conflicts or a later native boot failure. please compare normalized MACs for active network-enabled guests as well as machine identifiers, with a distinct-ID/same-MAC rejection test. native duplicate-MAC behavior has not been verified.

validation: focused image validation, API config, hypervisor parser/validation, VM metrics/parser, and macOS reclaim-exclusion tests passed. API/instance package tests could not compile because required embedded binaries were missing. native VZ/APFS behavior was not tested.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants