[HarryMoss]

Summary

Correlates native HTTP and API OCSF records with sampled OpenTelemetry traces, and attaches emitting-thread trace/span context to OTLP log records. Dedicated supervisors can create a bounded INFO deny span for an HTTP refusal and its companion records.

Network Activity and DNS denials do not receive a new default-level deny span or native Trace body field in this revision. The issue should remain open unless its acceptance criteria and follow-up tracking are agreed before merge.

Related Issue

Part of #2640. The remaining Network Activity correlation needs follow-up; this PR should not close that broader issue.

Changes

• Restricts OCSF Trace enrichment to native HTTP/API classes, validates trace IDs, and vendors only native schema exports. OTLP log context remains available for every class when a sampled span is active.
• Keeps one HTTP decision scope for companion audit records, with available parent context. Destination limits remain burst ten/refill one per second; a process-wide budget adds burst 100/refill 100 per second across destinations.
• Classifies remaining actual refusal findings as Denied: uninspectable traffic/body, fail-closed session capacity, admission shedding, credential-inspection refusals and blocked response delivery. These corrections can increase denial-query counts. Fail-open and ordinary I/O-error records preserve their prior classification.
• Adds refusal/allowed controls, aggregate-budget and audit-preservation coverage, and action-absence assertions on I/O failures. Documents passive parent correlation after rate limiting and the shared-supervisor export limitation.

Changed audit records

These changes describe the complete proposed PR relative to its upstream base. Additional refusal labels can increase counts from queries such as select(.action == "Denied").

Record or outcome	Before	Proposed behavior
FORWARD unparsable URI, HTTP Activity	No action/disposition	Denied/Blocked
FORWARD non-canonical request-target or invalid GraphQL/JSON-RPC, Network Activity	No action/disposition	Denied/Blocked
FORWARD L7 evaluation error, Network Activity	No action/disposition	Enforce: Denied/Blocked; audit: Other/Allowed
Google Cloud metadata emulation 403/405, HTTP Activity	No action/disposition	Denied/Blocked
Fail-closed response middleware invocation, HTTP Activity	Other/Error	Denied/Blocked
Middleware failure, policy-evaluation failure and fail-closed WebSocket stage findings	No action	Denied
Uninspectable traffic/body and session-capacity findings	No action	Denied for a refusal; action remains absent for fail-open
Admission shedding and uninspectable-credential findings	No action	Denied
Response middleware blocked-delivery finding	No action	Denied
HTTP response failure finding	No action	Denied when delivery is refused; action remains absent for ordinary I/O, framing and policy-reload failures
Response diagnostics after a middleware stop	Emitted at the stop point	Emitted with the terminal finding in one decision scope

Testing

The published parent is 2bb7e01. Its NVIDIA Branch Checks passed Rust tests, lint and build-mode checks on Linux x86_64/aarch64 and macOS aarch64; the x86_64 workspace run recorded 9,237 passed and 51 skipped. The advisory compatibility assessment was unavailable.

The follow-up at 4f827abc93d017fd9fb55b1ae171853bc425bbd0 has source tree a8782b1a942ef1f830bae8301bb186fc030b9366, based on that parent. New local validation used macOS arm64, Rust 1.95.0, locked dependencies and repository default profiles:

• Supervisor-network tests: 1,594 passed, zero failed, seven ignored. The ignored tests are five LocalStack integrations, a manual performance baseline and the existing unfinished full-duplex WebSocket fixture.
• Supervisor-network all-target Clippy with -D warnings: passed.
• Rust formatting, whitespace, Markdown lint, docs navigation and complete source-package reproduction: passed.
• Pinned Fern check: zero errors and three existing warnings, concerning gateway configuration MDX parsing, an unauthenticated redirects check and light-mode contrast.

These local results cover the follow-up tree. The parent's CI does not validate these new changes.

Fresh Linux and deployed Collector/backend validation of this follow-up, and reviewer reassessment, remain pending. Local in-memory/loopback exporter tests and historical results do not establish that deployed pass.

Correlation IDs do not guarantee trace export or retention. Limited records can retain an already active sampled trace, caller-propagated gateway IDs are correlation hints rather than identity evidence, and shared supervisors currently initialise no OTLP exporters.

Checklist

• Follows Conventional Commits.
• Commits include DCO sign-offs.
• Operator documentation updated.
• Fresh Linux/deployed validation completed for this follow-up.
• Reviewer reassessment completed.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}

[HarryMoss]

I have read the DCO document and I hereby sign the DCO.

[HarryMoss]

recheck

[HarryMoss]

The README conflict is resolved, and the updated branch passes the local checks. Could a maintainer please vet and approve the CI workflows? Fresh Linux/Collector/Tempo validation remains pending.

[rhuss]

cc-review Summary

@HarryMoss, please have your agent assess each of these comments and check it against the code before acting on it. Some may turn out not to apply, and I would rather you confirm or refute them than take them at face value.

This review comes from an agent-assisted run with several reviewers (cc-review: seven prompt reviewers plus Codex and CodeRabbit, then a skeptic/calibration pass). The findings and the scenarios in them come from the agents reading the code. Nothing was executed or reproduced, and I have not verified each scenario myself.

What Went Well

• crates/openshell-otel/src/correlation.rs current_ocsf_trace_correlation only correlates when the span context is both valid and sampled.
• crates/openshell-ocsf/src/trace.rs keeps openshell-ocsf free of OpenTelemetry through a OnceLock extractor, and enrich never overrides an explicit trace.
• crates/openshell-ocsf/src/events/base_event.rs set_trace adds the trace profile only once.
• crates/openshell-ocsf/src/format/downgrade.rs strips both the trace field and the profile for 1.1 and 1.3, with tests for each target.
• crates/openshell-supervisor-network/src/telemetry/tests.rs exported checks real OTLP/gRPC exports instead of only the JSON shape.

Findings

Severity	File	Description	Source
Important	crates/openshell-supervisor-network/src/telemetry.rs:32	Paired deny event and finding get different trace IDs	gator
Minor	crates/openshell-server/src/cli.rs:669	Gateway trace.uid is caller-propagated but documented as trusted	security
Minor	crates/openshell-supervisor-network/src/l7/middleware.rs:466	Capacity-exhausted finding marked deny when allowed	correctness
Minor	crates/openshell-supervisor-network/src/l7/middleware.rs:487	Load shedding traced as policy deny	goal-alignment
Minor	crates/openshell-supervisor-network/src/l7/middleware.rs:1004	denied: predicates untested	test-quality
Minor	crates/openshell-supervisor-network/src/proxy.rs:1261	Transparent middleware deny not parented on connect span (see below)	correctness
Minor	crates/openshell-supervisor-network/src/proxy.rs:5677	Non-canonical rejection gets no deny span	codex
Minor	crates/openshell-supervisor-network/src/proxy.rs:5956	CONNECT/FORWARD deny-span wiring untested	test-quality
Minor	crates/openshell-supervisor-network/src/proxy.rs:6072	Audit-mode L7 failure traced as enforced deny	codex
Minor	crates/openshell-supervisor-network/src/proxy.rs:6660	parent: is a no-op for non-deny events	architecture
Minor	crates/openshell-supervisor-network/src/proxy/trace_tests.rs:80	Debug deny test lacks count and uid checks	test-quality
Minor	crates/openshell-supervisor-network/src/telemetry.rs:11	Local ocsf_emit! shadows the upstream macro	architecture
Minor	crates/openshell-supervisor-network/src/telemetry.rs:41	Deny classification split between event and flag	architecture
Minor	crates/openshell-supervisor-network/src/telemetry.rs:55	Deny-span policy/host attributes never asserted	test-quality
Minor	crates/openshell-supervisor-network/src/telemetry.rs:64	One exported root span per deny, no rate gate	production
Minor	crates/openshell-ocsf/src/builders/network.rs:121	.trace() setter on 10 builders used only by tests	architecture
Minor	crates/openshell-otel/src/lib.rs:49	Supervisor filter moved into shared public API	architecture
Minor	docs/observability/ocsf-json-export.mdx:281	Docs mention the internal builder API	architecture

Outside the diff: crates/openshell-supervisor-network/src/proxy.rs:1261. The transparent-TCP fail-closed middleware deny calls emit_middleware_uninspectable without the live connect_span, unlike the sibling transparent denies, so at debug level this deny is not linked to the connect/authorize trace even though the docs say it should be. An optional parent parameter, or wrapping the call in connect_span.in_scope(...), would fix it.

Observation: the supervisor.egress.deny span and its openshell.policy.* keys live in telemetry.rs, separate from the supervisor.egress.* spans in proxy/egress.rs, and the attribute values come from string-keyed evidence lookups. Shared constants would keep the two in sync.

Review Details

• Findings posted: 18
• Findings reviewed and not posted: 0
• Gate outcome: FAIL (one Important finding)
• Participating reviewers: architecture, correctness, goal-alignment, production, security, test-quality, gator, codex (CodeRabbit ran with no findings)

[HarryMoss]

Thanks for the review, @rhuss. The follow-up at 910509993e18659404e923a2aa33d772a4ae5655, following the upstream integration, groups companion deny records and terminal response failures under one decision span, corrects caller-propagated correlation wording, and keeps ordinary I/O failures out of enforced-denial spans. Forward evaluation-error spans retain the HTTP policy context, and ocsf.class_uid now exports as an integer.

The final source tree is e231ce833d5999a46d664121858cac22cdc526aa. Its network library suite passed 1,561 tests (2 ignored); five-crate all-target Clippy and formatting passed. OCSF/OTel and docs evidence is separately attributed to unchanged source in the updated PR description.

The docs now explicitly identify Trace applicability on the eight extra classes as an OpenShell extension beyond official OCSF 1.8 HTTP/API applicability. Please confirm whether that extension is acceptable. Fresh Linux and default-level Collector/Tempo validation with allowed/schema controls, NVIDIA CI approval/completion and reviewer reassessment remain pending. This remains Draft; earlier independent deployment results apply only to 3fae097a.

[rhuss]

@krishicks @TaylorMutch wdyt? Could we try to get this PR in? All findings from my review are addressed (I checked each against 9105099), and the branch is current with main, including #5019. What remains needs a maintainer call:

1. Trace profile beyond HTTP/API. OCSF 1.8 lists the Trace profile only for HTTP/API activity. The PR also applies it to eight other classes as a documented OpenShell extension, which strict OCSF consumers may reject. OK as is, or limit it to HTTP/API for now?
2. Public .trace() setters on all ten builders. Production code uses the registered extractor, and only tests call the setters. Keep them or drop them?
3. Schema catalog growth. Following the per-combination layout from feat(ocsf): validate OCSF events against exported JSON Schemas #5019, the PR adds a +trace variant of each of the 18 vendored schemas (~146k generated lines). Fine for now, or should profile variants be generated rather than vendored?

CI also still needs /ok to test.

@HarryMoss is there anything left before moving the PR out of draft? CI would cover the pending Linux build and tests once it is approved. Could you also resolve the threads I confirmed as fixed? That would make the PR easier for maintainers to scan. I'm happy to rerun the Podman + Collector/Tempo deny check on fed4f94 if that helps.

[HarryMoss]

Thanks @rhuss. Yes please, a fresh Podman + Collector/Tempo check on fed4f94257a49d44b5380678ecd32c16b1bf64b3 would be very helpful. The remaining runtime check is default-level deny correlation, including companion findings/domain records sharing the same trace, with allowed-traffic and schema controls. Please record the tested revision and configuration with the results.

Local macOS validation on that exact source tree is complete, including the affected Rust suites, six-crate all-target Clippy, schema checks and docs checks. Fresh Linux CI and the deployed check are still pending.

I have resolved the 16 review threads you confirmed as addressed and left the public .trace() setter discussion open for the maintainer decision.

@krishicks @TaylorMutch, could you please vet/approve CI and advise on the three choices above: the Trace applicability extension beyond HTTP/API, keeping or dropping the public setters, and vendored versus generated profile schemas? I can make the corresponding changes once the intended direction is clear.

I will keep this as a draft pending those decisions and the outstanding validation. Thanks for helping move this forward.

[krishicks]

Written by Claude Code, reviewed by @krishicks

Thanks @HarryMoss for the thorough work, and @rhuss for driving the review. Here are maintainer decisions on the three open questions, plus a few more items. Severity follows rhuss's scale. References are to fed4f94.

Decisions

Trace profile applicability: emit trace only where OCSF 1.8.0 declares it (HTTP Activity and API Activity). Schema conformance comes first for our audit output. A strict consumer rejects the whole record, not just the optional field. The 16 composed +trace schemas are not OCSF exports. schema.ocsf.io ignores profiles=...,trace for network_activity, detection_finding, and the other six classes. For example, the network_activity export with and without trace is byte-identical. trace_schema.py builds those files by grafting http_activity's definition onto each class. That makes the validator check our extension rather than OCSF conformance.

Public .trace() setters: drop them. No production code calls them. #2640 already says the agent's own trace context goes in a separate field, never in trace, so no override use case is coming.

Schema catalog: vendor only what update.sh fetches from the OCSF API. Once applicability matches OCSF, only the two native +trace files remain. Both come straight from the API with no composition step.

Findings

Severity	Item
Important	Restrict trace to HTTP/API Activity
Important	Audit-record changes need calling out
Minor	Make TraceCorrelation validated, drop builder setters
Minor	Set the OTLP log record's trace context
Minor	Derive deny classification from the event
Minor	Guard against the upstream ocsf_emit! in supervisor-network
Minor	Bound deny-span volume
Nit	deny_parent: on a site that never denies
Nit	Test harness cost
Nit	Commit structure

Important: restrict trace to HTTP/API Activity.

• Gate enrichment on the event class in enrich (crates/openshell-ocsf/src/trace.rs:32) and in BaseEventData::set_trace.
• Remove trace_schema.py, the 16 composed schemas, and the update.sh changes.
• Reduce the docs' extension section to one sentence: correlation applies to HTTP and API Activity.
• Knock-on effects in the supervisor:
  • Network Activity and Detection Finding records can no longer carry trace.uid.
  • A supervisor.egress.deny span whose primary record is one of those classes exports a span that no OCSF record points to.
  • Please open deny spans only for decisions with an HTTP Activity record. Drop the wiring that exists only for 4001 or 2004 sites.
• The feat(ocsf): add trace_id/span_id correlation fields to OCSF event builders #2640 acceptance criterion for network deny events needs amending to match. Network Activity correlation can return once OCSF declares the profile there.

Important: changes to audit records need calling out. Besides adding correlation, several existing records change meaning:

• These gain action=Denied and disposition=Blocked:
  • FORWARD parse errors (proxy.rs:2191);
  • non-canonical, GraphQL and JSON-RPC rejections (build_forward_rejection_event, proxy.rs:2135);
  • metadata-server 403/405 responses (google_cloud_metadata.rs:344).
• L7 evaluation errors get a new action and message (proxy.rs:2151).

These look like correct fixes. Those were denials that lacked an action. But they shift SIEM deny counts. Please put them in their own commit and mention them in the PR description and docs.

Minor: make TraceCorrelation validated and drop the builder setters. TraceCorrelation { pub uid: String } (trace.rs:17) accepts any string in a field documented as 32 lowercase hex. Make the field private behind a validating constructor, or build it from [u8; 16]. Then remove .trace() from the ten builders. Tests can drive the registered extractor with a real span, as ocsf_correlation.rs already does.

Minor: set the OTLP log record's trace context. log_record (crates/openshell-otel/src/ocsf.rs:374) puts trace only in the body. Log-to-trace links in OTLP backends use the record's own trace/span fields. These are OTLP fields, not OCSF attributes, so filling them from the active sampled span at emit time adds no conformance issue. It also brings correlation back for Network Activity and findings on the OTLP export path.

Minor: derive deny classification from the event. The decision still comes from three places:

• the event's action;
• the denied: macro arm (telemetry.rs:11);
• the denied argument to emit_decision (telemetry.rs:54).

is_denied (telemetry.rs:34) already reads DetectionFinding.action. Setting ActionId::Denied on findings that describe a refusal lets the flags go. It also records the refusal in the audit record itself, not only in the trace.

Minor: guard against the upstream ocsf_emit! in supervisor-network. A deny emitted with openshell_ocsf::ocsf_emit! instead of egress_emit! silently gets no deny span. opa.rs:148 and policy_dns/runtime.rs:17 still use the upstream macro. A disallowed-macros entry for openshell_ocsf::ocsf_emit in the crate's clippy config would catch new cases.

Minor: bound deny-span volume. Every denied decision exports a root span, and the workload controls how often it is denied. A tight retry loop against a blocked host can fill the batch exporter queue and cause the supervisor's other spans (startup, policy) to be dropped. Please add a per-destination rate limit on deny-span creation. OCSF records stay unaffected.

Nit: deny_parent: on a site that never denies. At proxy.rs:6853 (FORWARD upstream connect failure), the event never carries ActionId::Denied, so the arm does nothing. Use egress_emit!(event).

Nit: test harness cost. Each trace test starts a loopback OTLP gRPC server behind a global lock. Keep one or two end-to-end export tests and use an in-memory exporter for the rest.

Nit: commit structure. This would review and bisect better as separate commits:

1. ocsf/otel correlation plumbing and gateway registration;
2. supervisor deny spans;
3. the audit-record corrections above.

[krishicks]

Written by Claude Code, reviewed by @krishicks

I rebased this branch onto main and applied the decisions from my earlier comment as commits on top of @HarryMoss's work. His commits are squashed into the first commit, which keeps his authorship.

Commits

1. feat(ocsf): correlate audit events with sampled OpenTelemetry traces: the contributor work, rebased.
2. refactor(ocsf): validate trace correlation IDs: TraceCorrelation is built from a 16-byte trace ID, or deserialized with validation (32 lowercase hex characters, not all zero). Its uid field is private, and the builder .trace() setters are gone.
3. fix(ocsf): limit trace correlation to HTTP and API Activity:
   • trace and the trace profile appear only on HTTP Activity (4002) and API Activity (6003). Enrichment, set_trace, and the serializer all enforce this.
   • The composed +trace schemas and trace_schema.py are removed. Only files that update.sh fetches remain.
   • Deny spans open only for decisions with an HTTP Activity record.
4. refactor(supervisor-network): derive deny classification from the event: a decision counts as denied only when one of its records has action Denied. Findings that describe a refusal now set it, and the denied flags are gone.
5. chore(supervisor-network): reject direct ocsf_emit! in the egress crate: a disallowed-macros clippy rule makes every emission go through egress_emit! / emit_decision.
6. feat(supervisor-network): rate-limit deny spans per destination: a token bucket (burst 10, then one per second, up to 1024 destinations) caps how many deny spans a destination can create. The limiter is consulted only when deny spans are enabled. OCSF records are unaffected; records past the limit omit trace.
7. feat(otel): set trace context on OCSF log records: OTLP log records carry the active sampled span's trace and span IDs for every class, so Network Activity and findings still link to traces on the OTLP path.
8. test(supervisor-network): use an in-memory span exporter for deny-span tests: two tests still export over OTLP/gRPC end to end; the rest use an in-memory exporter.
9. docs(ocsf): describe which records report a refusal
10. refactor(supervisor-network): reduce deny spans to one entry point:
    • emit_decision is the only entry point.
    • The deny span describes the denied HTTP record.
    • Records that never share a span are emitted separately again.
11. fix(supervisor-network): keep FORWARD denials in the connect trace:
    • At debug level, FORWARD credential-binding and request middleware denials now join the connect trace.
    • Fail-closed WebSocket middleware findings now report action Denied.

Changed audit records

These records describe refusals and now say so. Counts of action == "Denied" will rise, and the human-readable log lines for these records now print DENIED with a [reason:…] suffix.

Record	Before	After
FORWARD unparsable URI (HTTP Activity)	no action/disposition	Denied/Blocked
FORWARD non-canonical request-target (Network Activity)	no action/disposition	Denied/Blocked
FORWARD invalid GraphQL or JSON-RPC body (Network Activity)	no action/disposition	Denied/Blocked
FORWARD L7 evaluation error (Network Activity)	no action/disposition; "L7 eval failed, denying request"	enforce: Denied/Blocked; audit: Other/Allowed; "L7 evaluation failed"
Google Cloud metadata emulation 403 and 405 (HTTP Activity)	no action/disposition	Denied/Blocked
Response middleware invocation with a fail-closed outcome (HTTP Activity)	Other/Error	Denied/Blocked
Findings openshell.middleware.failure and openshell.middleware.policy_evaluation_failure	no action	Denied
Finding openshell.middleware.http_response_failure	no action	Denied when the supervisor refused delivery; unchanged for I/O, framing, and policy-reload failures
Finding openshell.middleware.websocket_failure, fail-closed stage	no action	Denied
Response diagnostics after a middleware stop	emitted at the stop point	emitted with the terminal finding as one decision

docs/observability/ocsf-json-export.mdx lists these under "Refusal Records".

Verification

I ran a local Docker gateway exporting to an OpenTelemetry Collector, Jaeger, and Aspire, then drove a sandbox through allowed, L7-denied, forward-denied, L4-denied, and rapid-retry traffic:

• HTTP L7 and forward denials carry a trace.uid that matches the OTLP log record's trace ID.
• That ID resolves in Jaeger to a supervisor.egress.deny span with the policy, destination, and OCSF class.
• The retry loop of 16 denials produced 13 deny spans, consistent with the burst-plus-refill limit.
• Gateway authentication records and supervisor configuration records carry OTLP log trace context.

Follow-ups

• Network Activity denials (L4, DNS) get no deny span and no trace context at the default log level. Deny spans for them, linked through OTLP log trace context, can come in a follow-up.
• L7 policy deny records don't set status_detail, so their deny spans lack openshell.policy.reason. Middleware denials already have it.
• L7 denials inside CONNECT tunnels and response-middleware denials are root spans even at debug level, because the connect span has ended by then.
• The feat(ocsf): add trace_id/span_id correlation fields to OCSF event builders #2640 acceptance criterion for network deny events needs amending to match. I'll propose wording on the issue.

[krishicks]

/ok to test a0b27ba

[krishicks]

/ok to test 2bb7e01

[rhuss]

cc-review Summary

Re-review of the reworked branch at 2bb7e01, focused on the ten commits on top of the contributor's work (bb96bb570..2bb7e0155). Thanks @krishicks for turning the decisions into commits so quickly. The agent run found no regression of my earlier findings, and the rework answers my questions from Nov 2.

This comes from an agent-assisted run (cc-review: seven prompt reviewers, then a skeptic/calibration pass; Codex and CodeRabbit did not run this time). The scenarios come from reading the code. Nothing was executed, and I have not reproduced them myself.

What Went Well

• TraceCorrelation keeps uid private and can only be built from 16 bytes or through validated deserialization, which rejects uppercase, wrong-length and all-zero IDs (crates/openshell-ocsf/src/trace.rs).
• The class gate sits in enrich, set_trace and the serializer, so a deserialized trace on Network Activity is never re-emitted (base_event.rs).
• emit_decision is the only deny-span entry point, and the crate-level disallowed-macros rule enforces it (telemetry.rs, clippy.toml).
• deny_span checks tracing::enabled! before taking a limiter token, so a supervisor without deny spans spends nothing (disabled_deny_spans_spend_no_tokens).
• OTLP log records get trace and span IDs only from sampled spans, with tests for both samplers (openshell-otel/src/ocsf.rs).

Findings

Severity	File	Description	Source
Important	crates/openshell-supervisor-network/src/l7/middleware.rs:172	Several refusal findings still lack action: Denied, although the new Refusal Records docs promise it	security (also architecture, correctness, gator)
Important	PR metadata	Closes #2640 would close the issue while its network-deny scenario is not delivered	goal-alignment

The second one is about the PR text, not a line of code, so it is here instead of inline:

Important: The PR body still says Closes #2640. The issue's user story and acceptance criteria are about network deny events carrying trace.uid. With the HTTP/API-only decision, accepts_trace (trace.rs:69) excludes Network Activity, and emit_decision opens deny spans only for HTTP records, so L4 and DNS denials have no trace link at the default log level.

Why this matters: Merging as is closes #2640 while its headline scenario is still open, and nothing tracks the Network Activity follow-up. The Nov 5 comment mentions proposing new wording for the issue; the issue has not changed since Oct 19.

Suggested fix: Amend the acceptance criteria in #2640 and open a follow-up issue for Network Activity correlation before merge, or change the link to Part of #2640.

Source: goal-alignment agent

Nonblocking suggestions

• PR description: The body still describes fed4f94 (builder setters, the extension to eight classes, 18 schemas, its test counts) and lacks the "Changed audit records" table from the Nov 3 request. Replacing it with the Nov 5 summary would match the code.
• Deny-span limiter: The bucket key is the host:port the workload chooses, and there is no global budget, so rotating hostnames could still produce about 10k spans in a burst and about 1k per second after that, more than the default 2048-span batch queue (telemetry.rs:144). Only spans are affected, never OCSF records, and the per-destination design is what I suggested earlier. A small global token bucket in front of it would close the gap.
• Tests: response_trace_upstream_io_failure_is_not_an_enforced_denial and response_trace_downstream_body_failure_is_not_an_enforced_denial (l7/rest.rs:6981-7092) only assert that no deny span exists, which holds whatever action_id the http_response_failure finding carries. Asserting action_id there would guard the "unchanged for I/O failures" row of the audit table.

Review Details

• Findings posted: 2 (one inline, one in this summary), plus 3 nonblocking suggestions
• Findings reviewed and not posted: 0
• Gate outcome: FAIL
• Participating reviewers: security, architecture, correctness, gator, goal-alignment, production, test-quality

[HarryMoss]

Thanks @rhuss, and thanks @krishicks for reworking the branch.

I've pushed 4f827abc on top of 2bb7e015. The remaining refusal findings now set action: Denied, while fail-open outcomes and ordinary I/O errors keep their previous classification. This also adds the global deny-span budget you suggested: burst 100, refilling at 100 per second, alongside the existing destination limits. Audit records continue to be emitted.

Local macOS validation on the exact source tree a8782b1a942ef1f830bae8301bb186fc030b9366 passed: 1,594 supervisor-network tests, 7 ignored, all-target Clippy with -D warnings, formatting and docs checks. The regression coverage includes refusal/fail-open action fields, I/O action absence, aggregate limits, audit preservation and existing sampled context after rate limiting. Fern reported zero errors and the same three recorded warnings.

I've updated the PR description with the current scope, changed-audit-records table and Part of #2640. The remaining Network Activity work stays tracked by #2640.

Fresh Linux and deployed Collector/backend validation are still pending for this revision. Could a maintainer please vet/approve CI for 4f827abc93d017fd9fb55b1ae171853bc425bbd0 and help confirm the fresh runtime check?

One tracking detail needs your guidance: check_issue_acceptance.cjs currently requires a closing issue reference, so Part of #2640 conflicts with that check. Could you please confirm the scoped accepted issue this PR should close, or how the non-closing link should be handled? I have kept #2640 open as requested.

[krishicks]

/ok to test 4f827ab

[krishicks]

Here's a little summary of this change with fancy diagrams.

Before: the records have no trace field, and a grey dashed line marked "?" stands between each record and the trace column. The gateway request span exists but can only be matched by hand. No span records the supervisor's denial.

After: each record has a highlighted trace.uid. A green link with a check mark runs from it to the matching trace ID chip, which sits on the gateway's request span or on a supervisor.egress.deny span showing the policy name and server.address. The footer notes that OTLP log records carry trace and span IDs for every OCSF class.