Repository navigation
feat(ocsf): add trace_id/span_id correlation fields to OCSF event builders #2640
Description
User Story
As a security or compliance reviewer investigating a policy violation, I want each OCSF event to carry the ID of the trace that produced it, so that I can jump from the event in my log aggregator straight to the supervisor's trace instead of matching sandbox IDs and timestamps by hand.
Problem Statement
Open Cybersecurity Schema Framework (OCSF) security events and OpenTelemetry (OTel) traces exist in separate systems with no connection between them. A security reviewer filters the OCSF log aggregator (Loki, Splunk) for DENY events in the last hour and finds a network deny for api.suspicious.com from sandbox sb-abc123. To see how the supervisor handled that connection (which policy matched, what L7 enforcement and middleware ran), they search Jaeger for the sandbox ID and hope the timestamps line up. Nothing links the security event to its trace.
Today only the gateway emits OTel spans. Once #3977 lands, the supervisor does too, and most OCSF events and OTel spans then come from the same code paths. The OCSF builders have no way to record the active trace context.
Impact / Why This Matters
Every deny investigation needs manual correlation across two systems. The workaround is searching the trace backend by sandbox ID within a time window. That breaks down quickly: with #3977 every egress connection starts its own trace, so a busy sandbox produces many traces per second and the time window rarely identifies one. SIEM pipelines also have no join key to automate the correlation.
Proposed Design
Record the active trace context in OCSF events using the OCSF 1.8 Trace profile. When a sampled OTel span is active at emission time, the event carries trace.uid and lists trace in metadata.profiles. Without one, the trace object is omitted. The reviewer pastes trace.uid into Jaeger or Tempo and lands on the supervisor trace that produced the event.
Representation
OCSF 1.8 has no top-level trace_id or span_id attributes. It defines a Trace profile whose trace object carries the W3C trace ID in trace.uid, plus an optional span object:
{"class_uid": 4001, "activity_id": 1, "metadata": {"profiles": ["security_control", "network_proxy", "container", "host", "trace"], ...}, "trace": {"uid": "0af7651916cd43dd8448eb211c80319c"}, ...}The first version populates trace.uid only. The 1.8 span object requires start_time and end_time, which are unknown while the span is still open at emission time. Whether and how to fill trace.span is settled during implementation against the vendored-schema validation tests. The Trace profile schema files get vendored next to the existing ai_operation profile.
Layering
openshell-ocsf stays independent of OpenTelemetry:
openshell-ocsfadds a plain-data trace correlation type and a builder setter that renders thetraceobject and adds the profile.openshell-otelprovides a function that extracts the active OTel context and returns it only when the span context is valid and sampled. An unsampled trace ID points to nothing in the trace backend.- Binaries that emit OCSF events register this extractor with
openshell-ocsfat startup.ocsf_emit!fills in the trace context when the builder has not set one, so individual call sites need no OTel dependency. An explicit setter call overrides the automatic value.
Egress deny spans
The supervisor egress spans from #3977 are DEBUG, and the supervisor's OTLP filter resolves to openshell=info at the default warn log level. Without a change, network denies get no trace ID at default settings. The proposal is an INFO-level span for deny decisions only. Denies are rare and are what reviewers investigate, while allowed connections stay at DEBUG to keep span volume down.
The decision is known only after authorization returns, and every deny OCSF event is emitted after the authorization scope has ended. So the deny span is opened in the deny branch and entered around the OCSF emission at every deny site: L4 policy denials, mapping and SSRF denials, and L7 HTTP denials in the relay and middleware. It carries the decision attributes (policy, reason, destination). A span that wraps only the policy evaluation would leave the emission hook without a trace context.
At the default log level the DEBUG connect span is disabled, so the deny span has no exported parent and becomes a one-span root trace. trace.uid still resolves, but the full connect, authorize, resolve and dial tree appears only at debug level, where the deny span nests under the connect span.
Out of scope: agent trace context
Each egress connection starts its own root trace, so the linked trace shows the supervisor's handling of the connection, not the agent's activity. Linking a deny to the agent's own trace, through the workload's traceparent header relayed via #3196, is a follow-up. The workload controls that header, so it goes in a separate, clearly labelled field and never in the trace object.
Scope
crates/openshell-ocsf/: trace correlation type, builder setter,traceserialization,metadata.profilesentry, vendored Trace profile schema, emission hook, and downgrade stripping of thetraceobject and Trace profile for OCSF 1.3 and 1.1 (the profile first appears in 1.4)crates/openshell-otel/: sampled trace context extractioncrates/openshell-supervisor-network/: INFO-level deny span, entered around the OCSF emission at every deny site- Supervisor and gateway binaries: register the extractor at startup
- No breaking changes (the
traceobject is optional)
Dependencies
- feat(supervisor): export OTLP traces from sandbox supervisors #3977 (supervisor OTLP span export) must land first.
Acceptance Criteria
- OCSF events emitted inside a sampled OTel span include
trace.uid(32 lowercase hex characters) and listtraceinmetadata.profiles. - OCSF events emitted without an active sampled span omit the
traceobject and do not list the profile. - At the default supervisor log level, OCSF network and HTTP deny events carry a
trace.uidthat resolves to an exported trace containing the deny span. - Events downgraded to OCSF 1.3 or 1.1 contain neither the
traceobject nortraceinmetadata.profiles, covered by regression tests next to the existing downgrade tests. openshell-ocsfhas no OpenTelemetry dependency.- The Trace profile schema is vendored, and the schema validation tests cover events with and without
trace. - The published observability docs describe the
tracefield and how to follow it to the trace backend.
Alternatives Considered
Top-level trace_id / span_id fields: Easy to query, but outside the OCSF 1.8 schema. Consumers that validate against OCSF treat them as unknown attributes. The standard Trace profile covers the same need.
The unmapped mechanism: Schema-valid, but unmapped is meant for source data that has no OCSF attribute. Trace context has one, and downstream tooling won't look for it in unmapped.
Per-site .trace_context() calls: Explicit, but every emitting crate would need an OTel dependency, and adoption would drift across the call sites. The emission hook covers every event, and the explicit setter stays available where the current span is the wrong source.
Enrichment in the JSONL layer: Works, but puts the trace context into one output format instead of the event itself, so the shorthand format and any future sink would miss it.
Raising all egress spans to INFO: Gives every OCSF network event a trace ID, but exports one trace per outbound connection at default settings. Scoping the INFO span to denies covers the investigation case at a fraction of the volume.
Agent Investigation
- OCSF builders live in
crates/openshell-ocsf/src/builders/. Each builder declares its profiles throughctx.metadata(&[...]), andapi_activity.rsalready addsai_operationconditionally. - The vendored schemas cover OCSF 1.8.0 but include only the
ai_operationprofile. The Trace profile is available from the OCSF schema server. - OCSF events are emitted via
ocsf_emit!(), which stores the event in a thread-local and emits viatracing::info!(). The shorthand and JSONL layers extract it from there. openshell-otelowns OTel context handling viatracing_opentelemetry::OpenTelemetrySpanExt. The supervisor crates have no OTel dependency on main.ocsf_emit!call sites exist inopenshell-sandbox,openshell-server,openshell-supervisor,openshell-supervisor-network, andopenshell-supervisor-process.- feat(supervisor): export OTLP traces from sandbox supervisors #3977 (closes feat(observability): OpenTelemetry span emission from the sandbox supervisor #2508) adds supervisor span export. Its egress spans (
supervisor.egress.connect,authorize,resolve,dial) are DEBUG, and the OTLP filter isinfo,openshell=<level>with a floor of INFO.
Related: #1055 (Enterprise Observability), #2508 (Supervisor OTel span emission), #3977 (supervisor OTLP span export), #3196 (OTLP relay), #2507 (Gateway OTel export surface)
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsPlanning
Activity
kvnloo commentedon Sep 27, 2026
I checked this against the current crate boundary and the vendored OCSF schema.
One wrinkle: the OCSF 1.8 base schema does not define top-level
trace_id/span_idfields for these events. It does provide the schema-boundedunmappedescape hatch.There is also a dependency-boundary question:
openshell-ocsfcurrently owns OCSF event construction and depends ontracing;openshell-otelalready owns OpenTelemetry context extraction throughtracing_opentelemetry::OpenTelemetrySpanExt.I would prefer not to make the OCSF crate itself responsible for discovering the active OpenTelemetry context.
A narrow design could be:
That keeps OCSF serialization independent of the OpenTelemetry implementation.
If top-level
trace_id/span_idfields are intended as an OpenShell-specific extension instead, I think that should be explicit in the contract because it is outside the vendored base schema.Which representation do you prefer?
rhuss commentedon Oct 4, 2026
Thanks for checking this against the schema, @kvnloo. You're right on both counts. The dependency point also exposes a bug in my original proposal:
tracing::Span::current()only gives you process-local tracing IDs, not W3C trace and span IDs.Representation. I'd go with neither top-level fields nor
unmapped. OCSF 1.8 defines a Trace profile for this case, with atraceobject that holds the W3C trace ID intrace.uid. We only vendor theai_operationprofile today, so it's easy to miss. An event would look like this:{ "metadata": { "profiles": ["security_control", "network_proxy", "container", "host", "trace"] }, "trace": { "uid": "0af7651916cd43dd8448eb211c80319c" } }One catch: the 1.8
spanobject requiresstart_timeandend_time, and we emit the OCSF event while the span is still open. So I'd shiptrace.uidfirst, since that's what a security reviewer pastes into Tempo or Jaeger anyway, and settletrace.spanin the PR against the vendored-schema validation tests.Layering. Agreed,
openshell-ocsfshouldn't know about OpenTelemetry. What I have in mind:openshell-ocsfgets a plain-data correlation type and a builder setter that renders thetraceobject and adds the profile.openshell-otelextracts the active context and returns it only when the span context is valid and sampled. An unsampled trace ID sends the reviewer to an empty search result, which is worse than no ID.openshell-ocsfat startup, andocsf_emit!fills in the context when the builder hasn't set one. The emitting crates don't each need an OTel dependency, and an explicit setter still wins where the current span is the wrong source.Prerequisite. I also got something wrong in the issue: on main, the supervisor crates have no OpenTelemetry instrumentation yet. #3977 adds it, so this should build on that PR. Even with #3977 there are two gaps for the network-deny case:
openshell=infoat the defaultwarnlog level, so at default settings a deny would still carry no exported trace ID. I'd lean toward an INFO span around deny decisions only. Denies are rare and they're what a security reviewer investigates, while allowed connections stay at DEBUG to keep the span volume down.traceparentheader (relayed via feat(observability): supervisor OTLP telemetry relay #3196). The workload controls that header, so it needs a separate, clearly labelled field and must not end up in thetraceobject. I'd handle that as a follow-up.I'll update the issue body to reflect this. Are you planning to pick up the implementation? If so, building on top of #3977 makes sense.
HarryMoss commentedon Oct 6, 2026
I checked this against
mainat5601d71b0now that #3977 has landed, and ran a couple of focused tests around the OCSF/OTel path you described.The correlation gap is still present as expected. With an active sampled OpenTelemetry span,
current_trace_context_carrier()gives me a valid W3Ctraceparent, but an OCSFNetworkActivityemitted in that context still serializes without atraceobject /trace.uid.I also noticed two implementation wrinkles / edge cases that may be worth accounting for when this is implemented.
1. OCSF downgrade path
OpenShell emits OCSF 1.8 internally but supports downgrading JSON output to 1.3 and 1.1. The OCSF Trace profile is not present in those older schemas, while the current downgrade code only strips the newer fields/profiles it already knows about (
ai_model,container,ai_operation, etc.).I tried a 1.8 event containing:
{ "metadata": { "profiles": ["security_control", "trace"] }, "trace": { "uid": "0af7651916cd43dd8448eb211c80319c" } }and downgraded it to 1.3. The
traceobject survived the downgrade.So it looks like adding the Trace profile here will also require the downgrade path to strip both the
traceobject and"trace"frommetadata.profilesfor 1.3/1.1, with a regression test alongside the existing downgrade tests.2. Span lifetime around automatic enrichment
In the current egress path the connect span is made current while authorization runs, but some deny OCSF events are emitted afterwards.
I checked the same lifecycle in a focused test:
So if
ocsf_emit!obtains correlation only from the current active context, the new INFO level deny span needs to remain current through the actual OCSF emission.Wrapping only the policy evaluation would still leave the emission hook with no trace context. The other option would be to capture the correlation while the span is active and carry it explicitly to the event.
A narrow way to implement and test this seems to be:
traceand the Trace profile are removed for OCSF 1.3/1.1;trace.uidthat resolves to the exported supervisor trace.Does that match the intended implementation boundary for the deny span and the older schema downgrade path?
rhuss commentedon Oct 7, 2026
Thanks @HarryMoss, both points check out and they sharpen the plan.
Downgrade. Confirmed: the Trace profile and the
traceobject first appear in OCSF 1.4, andocsf_schema_versiononly accepts1.1and1.3as downgrade targets. So the existing<= 1.3branch informat/downgrade.rsis the right place: strip thetraceobject and thetraceprofile there, with a regression test next to the existing ones.Deny span lifetime. Also confirmed, on all the L4 egress paths: authorization runs inside
in_scope(...)and the deny events are emitted after it returns. There's a second reason the span can't wrap only authorization: the decision isn't known until authorization finishes, so a span that exists only for denies has to be opened in the deny branch anyway. The plan is a small helper that every deny site enters around its OCSF emission, carrying the decision attributes (policy, reason, destination). That includes the L7 deny sites in the relay and middleware. The automatic hook then picks up the context without call-site changes, and the explicit setter stays available for emissions outside such a span.One consequence worth stating: at the default log level the DEBUG connect span is disabled, so the deny span has no exported parent and becomes a one-span trace of its own.
trace.uidstill resolves and the span carries the decision attributes, but the full connect, authorize, resolve and dial tree only shows up at debug level. I think that's the right trade-off for span volume. The acceptance test should check exactly that: at the default level, a deny event'strace.uidresolves to an exported trace containing the deny span.So yes, your four steps match the boundary I have in mind. I'll fold both points into the issue body.
On implementation: I have nothing in progress on this. If you'd like to take it on, @HarryMoss (or @kvnloo, since you looked at it first), go ahead. I'm happy to collaborate, answer design questions as they come up, and review the PR. If neither of you wants to pick it up, just let me know and I'll start on it myself.
HarryMoss commentedon Oct 7, 2026
Thanks @rhuss, I’m happy to take it on and I’ll start looking at it tonight and tomorrow.
@kvnloo, if you’ve already started implementation please let me know, I’m happy to leave it with you rather than duplicate.
kvnloo commentedon Oct 7, 2026
Thanks @HarryMoss, please go ahead and take the lead! I’d started an implementation, but it never got pushed and that workspace is currently inaccessible, so I don’t have a usable branch to hand over. No need to wait on me. Happy to help review/test your PR. Thanks for catching the downgrade and span-lifetime cases, and @rhuss for clarifying the implementation boundary.
HarryMoss commentedon Oct 7, 2026
Thanks @kvnloo, really appreciate that. I’d definitely welcome your review and testing, and if you have any thoughts on the approach below I’d be keen to hear them too.
@rhuss, I’ve got a local implementation of #2640 ready and I’m doing the final code review before committing. It covers:
trace.uidcorrelation, explicit builder overrides, and automatic enrichment without adding OpenTelemetry dependencies toopenshell-ocsf;The focused regressions and affected crate suites pass, including the loopback OTLP tests. Linux ARM64 compilation also passes, I haven’t run a deployed sandbox E2E.
I also noticed #4288 now overlaps the downgrade path and several builder and schema changes. I’m keeping those broader fixes separate from #2640.
I’d be interested in both your thoughts on the implementation. @rhuss, from a merge order point of view, would you prefer I wait for #4288 to land and then rebase and adapt #2640, or prepare #2640 for review in parallel?
rhuss commentedon Oct 8, 2026
@HarryMoss great, thanks for moving so fast, and thanks @kvnloo for handing it over.
On merge order, I'd prepare #2640 for review in parallel rather than wait. #4288 is a much larger change and still in review. The overlap with #2640 is small and mechanical, mainly
format/downgrade.rs, which #4288 rewrites, so whichever lands second rebases. The final order is the maintainers' call, of course. cc @zanetworkerThree things should make the parallel path smooth:
trace, so the object moves tounmapped.downgraded_attributesand thetraceprofile is dropped automatically. If fix(ocsf): emit schema-conformant OCSF 1.8 events #4288 lands first, that commit simply goes away.trace.uidthen survives a downgrade underunmapped, which is arguably better for correlation in older SIEMs and still meets the acceptance criterion. So please have the downgrade tests assert "no top-leveltrace, notraceprofile" rather than "no trace ID anywhere".http_activity.jsonalready has the profile'straceattribute, butnetwork_activity.jsonandbase_event.jsondon't, and thetraceandspanobjects aren't vendored at all. Today's validator only checks top-level non-profile attributes, so it won't notice. fix(ocsf): emit schema-conformant OCSF 1.8 events #4288's validator rejects undefined attributes, so a Network Activity event withtracewould fail there. Running your events through fix(ocsf): emit schema-conformant OCSF 1.8 events #4288's validator once, for example by cherry-picking onto its branch locally, would catch this early. But I guess this also work just when rebasing later on, so probably not that urgent.trace.uidresolves in Tempo, on top of reviewing the code.HarryMoss commentedon Oct 8, 2026
Thanks @rhuss, that sounds good. #2640 is close to being ready for review in parallel, and I’ll keep the downgrade change in its own commit.
I’ve updated the vendored Trace schema coverage and checked fixtures from all nine builders, with and without trace correlation, against #4288’s validator in a separate local checkout. The downgrade tests check that successfully converted 1.1 and 1.3 events have no top-level
traceor Trace profile, without requiring the trace ID to disappear fromunmapped.I’m finishing the rebase and final review before pushing the branch. Once it’s up, I’ll share the branch link and exact commit for your Podman and Tempo check. Thanks for offering to run that, and @kvnloo for offering to review and test.
One production question for later is how we interpret a
trace.uidthat doesn’t resolve in Tempo. A failed lookup alone doesn’t tell us whether the trace is delayed, has been lost, or cannot currently be queried. I’ve added a brief note to the documentation in my local #2640 changes that correlation does not guarantee successful export or retention.Would it be useful to capture that as a separate investigation, particularly around supervisor restarts and collector backpressure? I’d keep it outside #2640 and wouldn’t add requirements to the deployed check you’ve offered.
cc @zanetworker
kvnloo commentedon Oct 8, 2026
Thanks @HarryMoss! The parallel review approach looks good, and validating all nine builders against #4288's stricter schema checks was a great catch.
I agree with keeping the downgrade change isolated and preserving trace correlation under "unmapped" where applicable.
Once your branch is pushed, I'm happy to review the sampled-context extraction, explicit overrides, and L4/L7 deny-span lifecycle.
On unresolved Tempo lookups, I think a separate investigation makes sense. We should distinguish delayed ingestion, dropped exports during supervisor restarts or collector backpressure, and retention/query failures. That shouldn't block #2640.
Thanks again for driving the implementation, and @rhuss for clarifying the merge and validation strategy!
HarryMoss commentedon Oct 9, 2026
Thanks @kvnloo and @rhuss. The branch is now pushed at revision
3fae097a5d4355eeff77746f3ea90e89228455e3.Both commits are signed off, with the downgrade change kept separate.
@kvnloo, this is ready for your review of sampled-context extraction, explicit overrides and the L4/L7 deny-span lifecycle. @rhuss, it’s available for the deployed default-level deny lookup in Collector/Tempo; that check is still pending.
I’ll keep the broader Tempo reliability investigation separate from #2640.
kvnloo commentedon Oct 10, 2026
Thanks @HarryMoss — reviewed
3fae097a5d4355eeff77746f3ea90e89228455e3on Linux x86_64 / Rust 1.95.0. No blocking finding in this bounded review. The head remains unchanged.Executed on your exact branch:
cargo +1.95.0 test --locked -p openshell-ocsf -p openshell-otel: 220 passed, two doc examples ignored. Covers sampled/unsampled/invalid context, explicit overrides, automatic/routed enrichment, all nine builders, and both downgrade contracts.cargo +1.95.0 test --locked -p openshell-supervisor-network --lib deny -- --test-threads=2: 59 passed, including all eight new correlation checks. Real emission helpers and the loopback OTLP receiver confirm default-level L4/L7 deny correlation and allowed/operational controls.telemetry.rs:63–86keeps the INFO span current through actual synchronous OCSF dispatch.Two integration boundaries to retain:
proxy.rs:3026drops the connect span before L7 relay. Later L7 denies can therefore be separate roots even at DEBUG. Your docs already qualify the live-parent condition; retain that qualification in the PR summary. The DEBUG regression proves staged L4 parenting, not a full CONNECT-to-L7 journey.DowngradeOutcome::Downgraded. Successful 1.1/1.3 output must lack top-leveltraceand its profile, while correlation underunmappedmay remain.KeptNativemust retain the whole 1.8 event, including trace/profile.The separate schema review pinned @zanetworker's #4288 at
78621e8bb1fb9af4c91f9cf8dd5c3ebffc62b963: 3 isolated downgrade checks passed, plus one 54-case matrix using Harry's schema files over #4288's unchanged production functions. Results: 18 native no-ops, 24 successful downgrades, 12 correct kept-native outcomes; every result validates against its declared version. Commands werecargo test --locked -p openshell-ocsf --features test-support --test review_trace_downgrade -- --nocaptureandcargo test --locked -p openshell-ocsf --lib review_schema_overlay -- --nocapture. Pinned receipt, source locations, replay fixtures and logs. This independently confirms your earlier nine-builder check; it is a schema overlay, not a compiled full implementation rebase. Suite counts overlap and should not be summed as unique regressions.Credit to Harry for the implementation, @rhuss for the schema/lifetime design and planned deployed check, and @zanetworker for #4288. These local checks do not establish deployed Collector/Tempo delivery; that validation remains with @rhuss. No competing implementation or PR created.
rhuss commentedon Oct 12, 2026
@HarryMoss @kvnloo Picking up the deployed Collector/Tempo delivery check that was left with me. Gateway and supervisor both built from
3fae097a5, run on Podman against an OTel Collector and Tempo, withocsf_json_enabled=trueand the supervisor at the default log level.A sandbox with a read-only policy for
example.comtriggered four denies. Each OCSF-JSON record carriedtrace.uidand thetraceprofile, and every ID resolved in Tempo to a single rootsupervisor.egress.denyspan fromopenshell-supervisorcarrying the sandbox id:www.wikipedia.orgpolicy_dns_ineligiblewww.wikipedia.org:443transparent_tcp_policy_deniedPOST https://example.com/example_readonlyPOST http://example.com/example_readonlyAllowed connections and the allowed GET carried no
trace, as intended. Withocsf_schema_version=1.3, the same denies came out as 1.3 with notraceobject and no Trace profile. No Tempo delivery issues: every lookup resolved immediately.Two non-blocking observations:
openshell.policy.reason, because the HTTP event leavesstatus_detailempty. The reason ("POST / not permitted by policy") appears only inmessage.Not exercised here: the DEBUG parented trace, and the explicit forward-proxy CONNECT path (the workload had no proxy variables, so all egress took the transparent path).
HarryMoss commentedon Oct 12, 2026
Thanks @rhuss for the deployed Collector/Tempo check, and @kvnloo for the independent review, reproducible evidence and compatibility checks. Thanks also @zanetworker for the related schema-conformance and downgrade work in #4288.
Both reviews cover
3fae097a5d4355eeff77746f3ea90e89228455e3. I’ll acknowledge these contributions and link the reviews in the PR description, keeping the tested revisions and coverage limits explicit. I’ll retain the per-decision trace and conditional DEBUG-parenting expectations, and record the non-blocking L7 reason-field observation without expanding #2640.The PR description is prepared. I’m waiting for approval on my existing vouch request, #4222, before submitting it.