You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(api): retain error spans and sample remaining traces at 50% - #1034
Use the existing logger to mark warning/error spans with httpsms.must_sample=true; do not copy log messages into spans.
Logger.WithSpan now receives the actual span rather than only its IDs. This removes the separate logger wrapper and all caller-offset machinery.
Filter completed spans before the existing batch exporter: retain errors, recorded exceptions, explicit flags, database spans over 200ms, and truncated telemetry; export 50% of the rest using a trace-ID decision.
Standalone logs and metric measurements are not sampled. Metric exemplars use only the stable sampled trace-ID subset so they do not point to spans discarded by this policy. Log messages, correlation fields, decorations, callers, and destinations are preserved.
Minimal scope
Keep the original GORM logger and dedicated-database logging setup; only update existing span bindings.
No second GORM logger, configuration layer, parameter filters, stack-walking, or log-severity text parsing.
The extra test-double edits only update the WithSpan signature; no service, handler, listener, or validator behavior changes.
Keep the two heartbeat database context-propagation fixes so database spans remain connected to their requests.
Keep focused tests for sampling decisions, retention flags, slow database boundaries, truncation, and unchanged structured log output.
Tradeoffs
All spans are still recorded locally; this reduces exported volume, not instrumentation overhead.
Retained spans outside the normal 50% sample can have missing normal parent or sibling spans.
Slow-database retention uses the span's measured duration and database attributes, independently of GORM's log timing/configuration.
The shared processor is used by the Axiom and Google trace providers.
Validation
go test ./pkg/telemetry ./pkg/services ./pkg/repositories ./pkg/handlers ./pkg/listeners ./pkg/validators ./pkg/middlewares ./pkg/di
go build ./...
No new lint issues across the same packages using golangci-lint run --new-from-rev=origin/main with Go 1.26.0.
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer TIP This summary will be updated as you push new changes.
[High impact] The PR appears safe to merge; no new blocking issues were found.
Summary
The PR retains warning/error spans and applies a trace-ID-based 50% sample to remaining spans.
Trace exporters send flagged spans and half of the rest.
Warnings and errors now leave clues on their active spans.
GORM query errors and slow queries now link to active spans.
No new actionable issues were found. No previous review threads were supplied. Tests were inspected but not run.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Span ends] --> B{Trace ID in 50% sample?}
B -->|Yes| E[Queue for export]
B -->|No| C{Error, exception, warning, or dropped telemetry?}
C -->|Yes| E
C -->|No| D[Do not export]
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Raw error text can expose customer content and API keys in traces, while span retention, production database warning capture, and log caller attribution remain unreliable.
This PR reduces exported API trace volume by retaining warning/error spans and deterministically sampling 50% of remaining spans after completion. All spans remain locally recorded.
Changes:
Adds a shared sampling processor for Axiom and Google trace providers.
Links application and GORM warning/error logs to active spans.
Adds tests for sampling, error retention, logger decoration, and event limits.
File
Description
api/pkg/telemetry/span_logger.go
Adds span-linked warning, error, and fatal logging.
api/pkg/telemetry/sampling_span_processor.go
Filters completed spans using severity and trace-ID sampling.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Unredacted log messages can expose credentials and message bodies in trace exports, while warning spans may still be dropped when event attributes are truncated.
[P2] This copies unrestricted error text into queued spans; HTTPRequestLogger and MessageHandler.PostSend include entire request bodies in their warnings. Large rejected requests therefore increase trace-buffer memory use and export payload size in addition to the existing log traffic. Omit the message from the span event, keeping the severity marker for retention and the full text in the standalone log.
Keep standalone log output intact and simplify telemetry conditions and test fixtures.
Propagate heartbeat update contexts and defer SQL formatting until a log is emitted.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 60fb2b83-6e9d-4656-9f60-6183c61b716d
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
SQL parameters can leak despite redaction, uppercase warning and error events can be dropped, and delegated GORM logs lose the original query call site.
Keep original query callers in standard GORM log output and honor opt-in parameter filtering.
Retain warning and error events regardless of severity capitalization.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 60fb2b83-6e9d-4656-9f60-6183c61b716d
Remove the extra GORM logger, configuration layer, parameter filters, and caller repair.
Retain database spans over 200ms in the export processor without changing logging.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 60fb2b83-6e9d-4656-9f60-6183c61b716d
Bind actual spans through Logger.WithSpan instead of adding a logger wrapper.
Remove caller offsets and unused severity parsing while preserving log output.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 60fb2b83-6e9d-4656-9f60-6183c61b716d
Share the trace-ID decision with exemplar filtering without dropping metric measurements.
Retain spans with dropped links or link attributes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 60fb2b83-6e9d-4656-9f60-6183c61b716d
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
httpsms.must_sample=true; do not copy log messages into spans.Logger.WithSpannow receives the actual span rather than only its IDs. This removes the separate logger wrapper and all caller-offset machinery.Minimal scope
WithSpansignature; no service, handler, listener, or validator behavior changes.Tradeoffs
Validation
go test ./pkg/telemetry ./pkg/services ./pkg/repositories ./pkg/handlers ./pkg/listeners ./pkg/validators ./pkg/middlewares ./pkg/digo build ./...golangci-lint run --new-from-rev=origin/mainwith Go 1.26.0.