feat(supervisor): export OCSF events as OTLP log records

[krishicks]

Summary

Sandbox supervisors export their own OCSF events directly to the collector configured by [openshell.gateway.otlp.ocsf], as OTLP log records. This is the supervisor half of #2892. It is stacked on #4500, which adds the gateway export and the shared exporter.

Related Issue

Closes #4544
Refs #2892
Refs #3224 (close manually once #4500 and #4545 have both landed)

Changes

• Configuration: no new settings. When [openshell.gateway.otlp.ocsf] sets enabled = true, compute drivers pass its resolved endpoint and schema version, and the gateway name, to every supervisor they launch:
  • The Docker, Podman, Kubernetes, and VM drivers set OPENSHELL_OTLP_OCSF_ENDPOINT, OPENSHELL_OTLP_OCSF_SCHEMA_VERSION, and OPENSHELL_GATEWAY_NAME on the supervisor. These are defined in openshell-core::sandbox_env.
  • The gateway passes the settings to the VM driver as --otlp-ocsf-endpoint, --otlp-ocsf-schema-version, and --gateway-name. It now passes --gateway-name whenever trace or OCSF export is enabled, not only with trace export.
  • The trace endpoint still reaches supervisors separately, so each signal can go to its own collector.
• Supervisor export: the supervisor starts the openshell-otel OCSF exporter from feat(server): export gateway OCSF events as OTLP log records #4500 and installs it as a tracing layer. Records use a supervisor resource identity that includes openshell.sandbox.id and openshell.gateway.name. Supervisor spans now carry the gateway name too, so telemetry from sandboxes of different gateways that share a collector can be told apart. Records are flushed when the supervisor shuts down. A setup failure is logged and disables export, so the supervisor still runs. Supervisors keep writing their local OCSF JSONL file.
• Collector credentials:
  • The gateway resolves the headers its own OCSF exporter uses: OTEL_EXPORTER_OTLP_LOGS_HEADERS, falling back to OTEL_EXPORTER_OTLP_HEADERS.
  • The headers go into each sandbox's SupervisorAuthBundle as an optional, redacted ocsf_otlp_headers field. Every driver already delivers this bundle as a protected file that only the supervisor reads.
  • The headers never appear in the sandbox environment or on command lines. The supervisor attaches them to its exporter, which parses them the same way the SDK parses its environment variables.
  • The field is omitted when there are no headers.
• VM shutdown: stopping a VM sandbox now sends SIGTERM to the supervisor and waits up to 5 seconds for it to exit before closing its parent-liveness pipe. Before, closing the pipe first made the supervisor exit(1) at once and drop queued records. The supervisor and the VM still get 5 seconds each, so a stop completes within 10 seconds as before; if the workload is slow to stop, flushing queued records is best effort.
• Docs: the OCSF export section covers supervisor export, its resource attributes, and how collector headers reach supervisors.

Testing

• cargo fmt --check and cargo clippy -D warnings on all targets of the core, OCSF, OTel, server, gateway, supervisor, and Docker, Podman, Kubernetes, and VM driver crates
• cargo nextest run on the same crates: 4,060 passed
• Unit tests for supervisor environment and argument generation in every driver, supervisor CLI parsing and export startup, header parsing, header resolution and precedence on the gateway, launch bundles carrying redacted headers, and an export to a loopback collector that arrives with the authorization header
• A VM stop test whose stand-in supervisor drains only if liveness stays open. It fails with the old ordering.
• cargo nextest run -p openshell-driver-vm after restoring the 10-second stop bound: 250 passed
• Before this branch was rebased and the header and shutdown fixes were added, ran it against a local Docker gateway and an Aspire collector. Supervisor OCSF records arrived as logs under the openshell-supervisor resource.
• E2E tests added/updated

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4545.docs.buildwithfern.com/openshell

[github-actions]

Compatibility review (advisory)

Needs review

VM sandbox stop and delete operations can now take up to 15 additional seconds while the supervisor drains OCSF records. Existing clients with deadlines based on prior behavior may time out, and no migration or deadline guidance is provided.

1. api-behavior — NOT ACKED

VM shutdown previously closed supervisor liveness before a 5-second wait; it now waits up to 15 seconds for the supervisor and then up to 5 seconds for the VM.

• Impact: VM-backed stop and delete calls with deadlines sized to the previous shutdown path can time out before the gateway finishes.
• Action: Preserve the prior completion bound or document the new worst-case stop/delete latency and the minimum client deadline required.
• PR: feat(supervisor): export OCSF events as OTLP log records #4545

Review details

• Execution: complete
• Candidate: PR #4545 (58b0292c9a593da264676f4d557ca6630e1bae06)
• Baseline: b087caa0551b3827687f2f0b2fc4b9d6b0c8b133 (b087caa0551b3827687f2f0b2fc4b9d6b0c8b133)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

[github-actions]

Compatibility review (advisory)

Unaddressed breaking changes

Existing deployments that pin a baseline supervisor image can no longer start sandboxes when OCSF collector headers are configured. VM stop and delete operations can also take up to 20 seconds instead of 10 seconds.

1. configuration — NOT ACKED

With OCSF OTLP headers configured, launch bundles now include ocsf_otlp_headers, which baseline supervisor images reject as an unknown field.

• Impact: Upgraded gateways retaining a baseline supervisor_image fail to start sandboxes because the supervisor cannot decode its authentication bundle.
• Action: Keep launch bundles forward compatible or require and document upgrading supervisor_image together with the gateway.
• PR: feat(supervisor): export OCSF events as OTLP log records #4545

2. api-behavior — NOT ACKED

VM shutdown previously waited up to 5 seconds each for the supervisor and VM, but now waits up to 15 seconds for the supervisor and 5 seconds for the VM.

• Impact: VM-backed stop and delete calls with deadlines based on the previous 10-second bound can time out before the new 20-second shutdown path completes.
• Action: Preserve the previous completion bound or document the 20-second worst case and required client deadlines.
• PR: feat(supervisor): export OCSF events as OTLP log records #4545

Review details

• Execution: complete
• Candidate: PR #4545 (a9952faf58304ea98094a89acb4b0a1119387200)
• Baseline: 1d973d9403b85d0fc4d863869c21dd329cf0cfb2 (1d973d9403b85d0fc4d863869c21dd329cf0cfb2)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

[github-actions]

Compatibility review (advisory)

Unaddressed breaking changes

Gateways using OCSF collector headers cannot launch sandboxes with a baseline pinned supervisor image. VM stop and delete operations can now take up to 20 seconds instead of 10 seconds.

1. configuration — NOT ACKED

With OCSF OTLP headers configured, launch bundles now include ocsf_otlp_headers, which baseline supervisors reject as an unknown field.

• Impact: Upgraded gateways retaining a baseline supervisor_image fail to start sandboxes because the supervisor cannot decode its authentication bundle.
• Action: Make launch bundles forward compatible or require and document upgrading supervisor_image together with the gateway.
• PR: feat(supervisor): export OCSF events as OTLP log records #4545

2. api-behavior — NOT ACKED

VM shutdown previously waited up to 5 seconds each for the supervisor and VM, but now waits up to 15 seconds for the supervisor and 5 seconds for the VM.

• Impact: VM-backed stop and delete calls with deadlines based on the previous 10-second bound can time out before shutdown completes.
• Action: Preserve the previous completion bound or document the 20-second worst case and required client deadlines.
• PR: feat(supervisor): export OCSF events as OTLP log records #4545

Review details

• Execution: complete
• Candidate: PR #4545 (d93586edcfb74efc2c280663faac2bd2c9d3ba5d)
• Baseline: 582bcd83bc26738f8a55cb01463e9e5661cf5384 (582bcd83bc26738f8a55cb01463e9e5661cf5384)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

[TaylorMutch]

Compatibility review (advisory)

Unaddressed breaking changes

Gateways using OCSF collector headers cannot launch sandboxes with a baseline pinned supervisor image. VM stop and delete operations can now take up to 20 seconds instead of 10 seconds.

1. configuration — NOT ACKED

With OCSF OTLP headers configured, launch bundles now include ocsf_otlp_headers, which baseline supervisors reject as an unknown field.

• Impact: Upgraded gateways retaining a baseline supervisor_image fail to start sandboxes because the supervisor cannot decode its authentication bundle.
• Action: Make launch bundles forward compatible or require and document upgrading supervisor_image together with the gateway.
• PR: feat(supervisor): export OCSF events as OTLP log records #4545

2. api-behavior — NOT ACKED

VM shutdown previously waited up to 5 seconds each for the supervisor and VM, but now waits up to 15 seconds for the supervisor and 5 seconds for the VM.

• Impact: VM-backed stop and delete calls with deadlines based on the previous 10-second bound can time out before shutdown completes.
• Action: Preserve the previous completion bound or document the 20-second worst case and required client deadlines.
• PR: feat(supervisor): export OCSF events as OTLP log records #4545

Review details

• Execution: complete
• Candidate: PR #4545 (d93586edcfb74efc2c280663faac2bd2c9d3ba5d)
• Baseline: 582bcd83bc26738f8a55cb01463e9e5661cf5384 (582bcd83bc26738f8a55cb01463e9e5661cf5384)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

2. Seems acceptable to me, since its a worst case scenario.

3. Seems like it could be problematic. Not sure how we want to handle this @krishicks

[TaylorMutch]

@krishicks one thing that caught my eye was that the configuration is always inherited from the gateway. I think it might be better if the supervisor has it's own configuration, which could be inherited from the gateway. That way openshell.supervisor.otlp can be configured specifically, like in the case of the collectors not being colocated in the same infrastructure. We could also have an option to allow being inherited from the gateway. WDYT?

[github-actions]

Compatibility review (advisory)

Unaddressed breaking changes

Gateways using OCSF collector headers cannot launch sandboxes with a baseline pinned supervisor image. VM stop and delete operations can now take up to 20 seconds instead of 10 seconds.

1. configuration — NOT ACKED

With OCSF OTLP headers configured, launch bundles now include ocsf_otlp_headers, which baseline supervisors reject as an unknown field.

• Impact: Upgraded gateways retaining a baseline supervisor_image fail to start sandboxes because the supervisor cannot decode its authentication bundle.
• Action: Make launch bundles forward compatible or require and document upgrading supervisor_image together with the gateway.
• PR: feat(supervisor): export OCSF events as OTLP log records #4545

2. api-behavior — ACKED

VM shutdown previously waited up to 5 seconds each for the supervisor and VM, but now waits up to 15 seconds for the supervisor and 5 seconds for the VM.

• Impact: VM-backed stop and delete calls with deadlines based on the previous 10-second bound can time out before the new 20-second shutdown path completes.
• Action: Preserve the previous completion bound or document the 20-second worst case and required client deadlines.
• PR: feat(supervisor): export OCSF events as OTLP log records #4545

Review details

• Execution: complete
• Candidate: PR #4545 (73ee2cd1fcf4754aa0c5a7cd69f33314f2e25a0b)
• Baseline: 9b890efa31b034ab2a113f1639977eee746a451c (9b890efa31b034ab2a113f1639977eee746a451c)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

[krishicks]

I acknowledge compatibility finding 1: supervisors older than the gateway reject launch bundles that carry ocsf_otlp_headers.

An older supervisor rejects any launch bundle containing a field it doesn't know, so it can't start a sandbox when the bundle carries ocsf_otlp_headers. The gateway only adds that field when both of these hold:

• [openshell.gateway.otlp.ocsf] sets enabled = true, which is new, opt-in, and off by default;
• OTEL_EXPORTER_OTLP_LOGS_HEADERS or OTEL_EXPORTER_OTLP_HEADERS is set on the gateway.

That means existing configs are unaffected. Sandboxes fail to start only if an operator opts into the new feature, sets headers, and also pins supervisor_image to an older version. A supervisor that old couldn't export OCSF events anyway.

[github-actions]

Compatibility review (advisory)

Unaddressed breaking changes

Gateways can no longer launch sandboxes with an older pinned supervisor image when OCSF export headers are configured. The PR author acknowledged this break, but the no-break train requires backward-compatible launch bundles.

1. configuration — ACKED

Candidate gateways add ocsf_otlp_headers to supervisor launch bundles, while baseline supervisors reject unknown bundle fields.

• Impact: Sandboxes using an older pinned supervisor_image fail to start when [openshell.gateway.otlp.ocsf] is enabled and OTLP headers are configured.
• Action: Negotiate bundle capabilities or omit ocsf_otlp_headers for older supervisors, and document upgrading pinned supervisor images.
• PR: feat(supervisor): export OCSF events as OTLP log records #4545

Review details

• Execution: complete
• Candidate: PR #4545 (5520a76e2477e51a8ceee5b00d10eab044264bf4)
• Baseline: 07fd2386ce18fb4e6312b695909d612ffeae578d (07fd2386ce18fb4e6312b695909d612ffeae578d)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run