feat(observability): correlate agent traces with OpenShell enforcement and OCSF #5210

Activity

1. github-actions

   added

   state:triage-neededOpened without agent diagnostics and needs triageOpened without agent diagnostics and needs triage

   on Nov 3, 2026
2. sftimeless commented on Nov 4, 2026

   

   sftimeless

   on Nov 4, 2026

   More actions

   Heya...

   One aspect of this design that interests me is the fine line between authorization, enforcement, and terminal delivery.

   In our Keel/OpenShell experiments, we could independently verify signed authorization decisions and separately observe OpenShell enforcement outcomes, but couldn't establish a complete cryptographic authorization-to-dispatch chain.

   The proposed correlation and terminal-delivery records would be useful for reconstructing that sequence, provided the evidence preserves which component actually observed each fact.

   Specifically, an external authorization decision, an OpenShell forwarding result, and a downstream service's reported outcome should remain distinguishable, even when they share a request or trace identifier.

   I'd be interested in how the proposed receipt model handles partial forwarding or an uncertain downstream outcome without presenting it as confirmed execution.

   We have existing sanitized ALLOW/DENY test cases in #4557 that could help validate the correlation portion.
3. krishicks commented on Nov 4, 2026

   

   krishicks

   on Nov 4, 2026

   Collaborator

   More actions

   Written by Claude Code, reviewed by @krishicks

   I reviewed #5214 together with the relay-state-* prototype branches and #4789. The direction is right: untrusted W3C context as a correlation hint, and one OpenShell-minted request ID shared across spans, middleware, OCSF metadata.correlation_uid and the response header. Two parts of the prototype's shape should change before it's split into PRs: how many OCSF records each request produces, and the single large enforcement span.

   One HTTP Activity record per request

   Today an inspected request produces one HttpActivity record (L7_REQUEST allow|deny, emitted at the policy decision). The prototype keeps that record and adds summary records for each middleware phase, for credential handling, and for terminal delivery. For a routine allowed request, that's up to four extra records. OCSF isn't sampled, so this becomes a fixed cost on every request.

   Proposal:

   • Keep a single HttpActivity record per inspected request, stamped with correlation_uid.
   • Emit it when the response headers are sent, rather than at the policy decision. At that point the record can include the decision, the action, the request-middleware outcome, the credential outcome, and the effective http_response.code (plus the upstream status when it differs). For a denial this changes nothing, because the 403 is the response. A guard emits the record if the request ends before headers are sent (upstream failure, cancellation), so every request still gets exactly one record.
   • Record routine conditions only on spans. The record gets a field only when something notable happened: fail-open, capacity exhausted, or a withheld, denied, unavailable or stale credential.
   • Emit a second record, with the same correlation_uid, only when something goes wrong after the first record: response middleware denies or fails after streaming starts, or delivery to the client fails or is cancelled partway through. Where it's a security finding, emit a DetectionFinding.

   This replaces the always-emitted summary records and the emission from Drop.

   An allowed request is logged when the upstream answers rather than when it's sent. OCSF export already lags, so the extra delay is small, and a complete record per request is worth more than an earlier, partial one.

   A root span with a child per stage, not one large span

   openshell.http.enforcement declares about 55 attributes up front, and most stay empty for any given request. Proposal: a small root span plus child spans for the stages that actually ran.

   openshell.http.request  (SERVER; parent or link from workload traceparent)
   │  request.id, method, server.address/port, policy.name,
   │  final decision/action/outcome, http.response.status_code
   ├─ openshell.policy.evaluate        decision, reason, policy.hash, endpoint.id, generation
   ├─ openshell.middleware.request     outcome, mutated, fail_open/closed, exhausted
   │    └─ middleware gRPC client → service spans
   ├─ openshell.credential.prepare     mechanism, surface, outcome   (only if injected)
   ├─ openshell.mcp.inspect            message kind, counts, revision (only for JSON-RPC)
   ├─ upstream HTTP (CLIENT)           standard http.* conventions, upstream status
   ├─ openshell.middleware.response    outcome, diagnostics_truncated
   └─ openshell.response.deliver       bytes, partial, delivery outcome
   

   • Stages that don't run produce nothing.
   • Each stage gets its own timing and error status, so error.stage is no longer needed.
   • The upstream call follows standard OpenTelemetry HTTP client conventions.
   • The single OCSF record is emitted with the root span active. Any extra record is emitted inside the span of the stage that caused it, so its span ID points at that stage.
   • Each later piece of work adds its own child span, instead of growing a shared struct.

   This adds spans per request, but spans are sampled and OCSF records aren't, so the net volume that can't be sampled goes down.

   Fitting this together with #4789

   #4789 already puts the trace in the OCSF event itself (trace.uid, Trace profile), uses an extractor registered at startup, and adds deny spans for L4 and CONNECT. I suggest landing it first and building on it:

   • openshell.policy.evaluate takes the place of supervisor.egress.deny on inspected L7 paths, and feat(ocsf): correlate audit events with sampled OpenTelemetry traces #4789's deny span stays the decision span for L4 and CONNECT. That avoids two decision spans for one L7 denial.
   • Read trace context from the current tracing span, as feat(ocsf): correlate audit events with sampled OpenTelemetry traces #4789's extractor does, rather than requiring with_span_context around each emit. The workspace builds tracing-opentelemetry without context activation, so OtelContext::current() is empty unless set by hand.
   • Use one sampling rule for both places a trace ID can appear. Set the OTLP log-record trace context only when the span is sampled, matching trace.uid. Unsampled and file-only records still join through correlation_uid.
   • Reuse feat(ocsf): correlate audit events with sampled OpenTelemetry traces #4789's OcsfEvent::base_mut. Both PRs currently add it in the same place.

   Suggested split

   1. feat(ocsf): correlate audit events with sampled OpenTelemetry traces #4789 lands first.
   2. HTTP correlation core (from relay-state-01 and 02): read traceparent, use the existing per-request middleware UUID as the shared request ID, send x-openshell-request-id on every inspected response and protect it from being set or removed, add the root span and policy.evaluate, and emit the single record when response headers are sent. Document correlation_uid and the header in ocsf-json-export.mdx. The header is for joining records, not for authorization.
   3. Middleware (03, 08, 09, 13c): request and response middleware spans, traceparent on middleware gRPC calls, and notable middleware outcomes on the single record.
   4. Credentials (04): the credential.prepare span.
   5. Resource revisions (06): this needs a decision between typed OCSF fields (for example firewall_rule.uid/version) and unmapped, so schema validation stays meaningful.
   6. MCP (05): the mcp.inspect span.
   7. Terminal delivery (13): the response.deliver span plus the extra record only for problems after headers are sent. This is the largest and riskiest piece, so it should land last.

   Two parts don't depend on this sequence:

   • Relay fixes (10–12): inject only OTEL_EXPORTER_OTLP_TRACES_*, and handle OTLP partial success without retrying. These aren't about correlation and could be reviewed as their own PR.
   • Policy-approval tracing (07): mostly gateway-side, and its grpc/policy.rs changes reach beyond observability (locking and reconciliation). It probably needs its own issue.

   Open questions

   1. unmapped or typed OCSF fields for policy hash, endpoint and revision IDs?
   2. Is it acceptable that the request-ID header and its spoofing protection cover only L7-inspected traffic, not L4 or passthrough traffic?

   👍React with 👍1Reacted by Yash Verma
4. krishicks

   added

   state:acceptedA maintainer decided OpenShell should pursue this issueA maintainer decided OpenShell should pursue this issue

   and removed

   state:triage-neededOpened without agent diagnostics and needs triageOpened without agent diagnostics and needs triage

   on Nov 4, 2026
5. yashdotrv commented on Nov 5, 2026

   

   yashdotrv

   on Nov 5, 2026

   More actions

   I’d be happy to take the HTTP correlation core if no one is already working on it. I noticed the L7 policy event is emitted before the middleware request UUID is created. So, think moving the creation of that existing ID earlier and using the same value in the request span, middleware context, OCSF correlation_uid, and a response header that upstream responses or middleware can’t replace.

   One question about the proposed single HttpActivity record per request: the current request and response middleware paths also emit records per invocation. Should those existing records be consolidated, or is the goal just to avoid the extra prototype summaries? I’d preserve security findings and cover allow, deny, and pre-header failures in tests.

   I saw the suggestion to land #4789 first. Is this slice already owned, and would you prefer I wait until #4789 and #5214 are reconciled? I can keep the first PR to inspected L7 traffic.