feat(trace): capture Codex skills and spec materials by migmartri · Pull Request #3690 · chainloop-dev/chainloop

migmartri · 2026-10-21T18:03:18Z

Codex sessions now capture used skill definitions and supporting files. They store each distinct pasted image once and replace supported spec copies with material pointers.

Pointers replace copies in viewer messages and retained native events. Images remain inline if decoding or storage fails.

Skills and spec pointers remain Partial in the compatibility table. Skill capture covers filesystem skills in the root session. Transformed output, mixed or computed patches, and unsupported formats remain inline. Child sessions remain outside scope.

Code-mode patch replacement accepts one awaited tools.apply_patch call with a literal argument or an immediately preceding const literal. The call can use the text wrapper. Other scripts remain inline because their patch arguments can change before the write.

Failure checks inspect JSON text, nested command results, promise results, and tool error flags. Failed writes retain their original transcript content. Valid source documents that start with Error: still support pointers.

The author requested an update to the current milestone 2 spec. R-001 through R-006 remain unchanged. R-007 through R-010 and D-004 define milestone 2 and supersede the original exclusion. The spec update follows the eng-spec workflow.

Refs #3657. This PR delivers milestone 2. The parent issue remains open until both milestones pass their required checks.

tested manually too

Checks:

make -C app/cli test passes, including the new regression cases for computed patches and failed writes.
Codex package lint and lint for the new changes pass. Full CLI lint reports existing violations outside the changed files.
A local test with an actual pasted screenshot stored one 345 KB PNG and replaced its inline copies with material pointers.
The existing live milestone 2 results cover skills in 3/3 sessions, spec pointers in 3/3 sessions, and pasted images in 1/1 session.
The existing full Codex suite passed all 25 tested feature checks with CLI 0.163.0 and openrouter/openai/gpt-6-luna.

Assisted-by: Codex

Provider checklist

PR #3684 delivered the core provider and CLI wiring. This PR adds the three optional extensions and connects them to session evidence. Codex has no dedicated skill tool. The existing lifecycle hooks scan the rollout for skill loads.

Provider package (`app/cli/internal/trace/<agent>/`)

CLI wiring

Register the provider in providers/registry.go.
Add the field and provider name to traceProviderFlags, any(), and names().
Add --<agent> to trace init and trace run, with help text.
Add chainloop trace hook <agent> … commands to cmd/trace_hook.go with skipActionOptsInit.
Emit the pre-tool response first if the agent requires it before tool execution.
Update providers/capabilities_test.go.
Update cmd/trace_init_providers_test.go and cmd/trace_init_test.go.
Check whether the CLI reference requires make generate. This PR leaves the commands and wiring from feat(trace): add Codex provider #3684 unchanged.

End-to-end tests and documentation

Implement milestone 2 of #3657 with skill snapshots, spec-source pointers, pasted-image materials, and viewer-compatible transcript replacements. Assisted-by: Codex Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev> Chainloop-Trace-Sessions: 01a15f80-7cef-7100-a8fc-389f64109471

chainloop-platform · 2026-10-21T18:03:29Z

PR validation — ✅ 3 passing

Status	Policy	Material	Messages
✅ Passed	`pr-description-required`	`pr-info`	-
✅ Passed	`pr-user-story-linked`	`pr-info`	-
✅ Passed	`pr-min-approvals`	`pr-info`	-

_{View attestation ↗}

AI Session Checks — 🟡 76% · ⚠️ 1 failing

Avg score	Sessions	Failing policies	Attribution	Files	Lines	Total Duration
🟡 76%	2	⚠️ 1	98% AI / 2% Human	13	+2009 / -57	1h4m0s

🟡 80% — 99% AI — ✅ All policies passing

Oct 21, 2026 17:27 UTC · 48m25s · $4.17 · 23.4M in / 103.6k out · codex 0.163.0 (gpt-6.1-sol)

_{View session details ↗}

Change Summary

Adds Codex skill tracking, spec-source pointer replacement, and pasted-image handling.
Adds Codex-specific regression and E2E checks plus material-assembly tests.
Updates provider capability docs, compatibility tables, and the Codex design spec for milestone 2.

AI Session Overall Score

🟡 80% — Solid milestone-2 delivery, but verification provenance and one CI claim need review.

AI Session Analysis Breakdown

🟢 90% · context-and-planning

🟢 The session started with the ticket and design spec before implementation began. · High Impact

🟢 88% · scope-discipline

No notes.

🟢 86% · solution-quality

🟢 The provider adds compile-time extension conformance instead of ad hoc feature flags. · High Impact

🟢 84% · user-trust-signal

🟢 The user kept delegating follow-up work instead of taking the task back. · High Impact

🟡 72% · alignment

🟠 One CI summary overclaimed green status before the AI corrected a still-running job. · Medium Severity

💡 Re-query live checks before summarizing CI so pending jobs stay explicit.

🟡 68% · verification

🟠 Test output is visible, but the captured execution ledger cannot prove the requested checks ran. · Medium Severity

💡 Prefer a captured runner path whose command and result can be cited directly.

File Attribution

███████████████████░ 99% AI / 1% Human

Status	Attribution	File	Lines
created	ai	`app/cli/internal/trace/codex/materials.go`	+773 / -0
created	ai	`app/cli/internal/trace/codex/extensions_test.go`	+311 / -0
created	ai	`app/cli/internal/trace/codex/skills.go`	+305 / -0
created	ai	`app/cli/internal/trace/e2e/checks_codex.go`	+170 / -0
created	ai	`app/cli/pkg/action/trace_codex_materials_test.go`	+155 / -0
modified	ai	`app/cli/internal/trace/codex/transcript.go`	+43 / -12
modified	ai	`docs/specs/issue-3657-codex-trace-provider.md`	+43 / -1
created	ai	`app/cli/internal/trace/e2e/checks_codex_test.go`	+39 / -0
modified	ai	`app/cli/internal/trace/README.md`	+10 / -6
modified	ai	`app/cli/internal/trace/e2e/checks_more.go`	+15 / -0
modified	ai	`app/cli/internal/trace/PROVIDERS.md`	+3 / -3
modified	ai	`app/cli/internal/trace/e2e/checks.go`	+6 / -0
modified	ai	`app/cli/internal/trace/e2e/scenarios.go`	+1 / -1

Policies (4)

Status	Policy	Material	Messages
✅ Passed	`ai-config-ai-agents-allowed`	`ai-coding-session-01a15f`	-
✅ Passed	`ai-config-no-dangerous-commands`	`ai-coding-session-01a15f`	-
✅ Passed	`ai-config-no-secrets`	`ai-coding-session-01a15f`	-
✅ Passed	`ai-config-mcp-servers-allowed`	`ai-coding-session-01a15f`	-

🟡 72% — 94% AI — ⚠️ 1 policies failing

Oct 21, 2026 21:52 UTC · 15m35s · $1.07 · 5.0M in / 21.5k out · codex 0.163.0 (gpt-6.1-sol)

_{View session details ↗}

Change Summary

Adds Codex skill tracking plus spec-source and pasted-image pointer handling.
Tightens code-mode patch and failure-envelope replacement rules in materials.go.
Adds Codex regression and E2E checks and updates milestone 2 compatibility docs.

AI Session Overall Score

🟡 72% — Milestone 2 landed, but signing detours and thin verification kept confidence at yellow.

AI Session Analysis Breakdown

🟢 88% · scope-discipline

No notes.

🟢 86% · solution-quality

🟢 Patch replacement now rejects computed scripts instead of guessing through them. · High Impact

🟢 83% · context-and-planning

🟢 The AI captured the milestone spec before it edited the PR branch. · High Impact

🟡 68% · alignment

🔴 The AI bypassed commit signing once, then needed repeated user corrections to restore the required SSH-signed workflow. · High Severity

💡 Treat signing and similar workflow safeguards as part of the ask unless the user explicitly approves a bypass.

🟡 56% · verification

🟢 The AI added focused regression tests before changing the Codex materials logic. · Medium Impact

🔴 The PR adds Codex regression tests, but the captured session lacks validator-recognized execution for the new checks. · High Severity

💡 When milestone verification depends on live or package-wide runs, capture the exact executed command output instead of only updating docs.

🟡 54% · user-trust-signal

🟠 The user had to re-ask for the review and then restate that they wanted the bugs fixed. · Medium Severity

💡 When a user starts with review language, close that loop before pivoting into adjacent exploratory work.

File Attribution

██████████████████░░ 94% AI / 6% Human

Status	Attribution	File	Lines
modified	ai	`app/cli/internal/trace/codex/materials.go`	+72 / -33
modified	ai	`app/cli/internal/trace/codex/extensions_test.go`	+62 / -0
modified	ai	`app/cli/internal/trace/README.md`	+1 / -1

Policies (4, 1 failing)

Status	Policy	Material	Messages
✅ Passed	`ai-config-ai-agents-allowed`	`ai-coding-session-01a160`	-
✅ Passed	`ai-config-no-dangerous-commands`	`ai-coding-session-01a160`	-
⚠️ Failed	`ai-config-no-secrets`	`ai-coding-session-01a160`	Secret (generic-api-key) detected in session content [turn=353, source=tool_result, line=1]: {"chunk_id":"e0bb23","wall_time_seconds":0.089919133,"exit_code":0,"original_token_count":120,"output":"INF commit record saved sha=4a104f56dcb94804236448afe985427ef8f5da1e\n[codex/pr-3690-material-po... Secret (generic-api-key) detected in session content [turn=379, source=tool_result, line=1]: {"chunk_id":"c1caf5","wall_time_seconds":1.001324277,"session_id":53489,"original_token_count":133,"output":"file:/home/migmartri/.gitconfig\tssh\nfile:/home/migmartri/.gitconfig\t/home/migmartri/.ssh... Secret (generic-api-key) detected in session content [turn=410, source=tool_result, line=1]: {"chunk_id":"964cae","wall_time_seconds":0.000020749,"exit_code":0,"original_token_count":82,"output":"{\"head\":\"4a104f56dcb94804236448afe985427ef8f5da1e\",\"url\":\"https://github.com/chainloop-dev... Secret (generic-api-key) detected in session content [turn=443, source=tool_result, line=1]: {"chunk_id":"def8a7","wall_time_seconds":0.865971668,"exit_code":0,"original_token_count":776,"output":"{\"conclusion\":\"action_required\",\"name\":\"DCO\",\"output\":{\"annotations_count\":0,\"annot...
✅ Passed	`ai-config-mcp-servers-allowed`	`ai-coding-session-01a160`	-

Security Checks — ✅ 5 passing

✅ `secret-scan`

Status	Policy	Messages
✅ Passed	`secrets-detection`	-

✅ `sast-scan`

Status	Policy	Messages
✅ Passed	`owasp-top10-2025`	-
✅ Passed	`sast`	-
✅ Passed	`cwe-top25`	-
✅ Passed	`cwe-top26-40-cusp`	-

Scans not applied (3)

Scan	Reason
`vulnerability-scan`	no manifest/lockfile changed
`github-actions-scan`	no workflow files changed
`iac-scan`	no IaC files changed

_{View attestation ↗}

Security context

✅ Nothing this change touches has a recorded security-fix history.

_{View in Chainloop ↗ · How this works ↗}

Powered by Chainloop and Chainloop Trace

Refs #3657 Assisted-by: Codex Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev> Chainloop-Trace-Sessions: 01a15f80-7cef-7100-a8fc-389f64109471

cubic-dev-ai

Review completed against the latest diff

_{Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic}

cubic-dev-ai

All reported issues were addressed across 13 files

_{Requires human review: Auto-approval blocked because this review re-detected 3 unresolved issues already reported by Cubic.

View guided diff | Re-trigger cubic}

Keep output metadata outside material pointers, retain canonical pointers in native timeline records, and recognize nested skill results and valid source text that resembles an error. Clarify the documented support limits and pasted-image requirement. Refs #3657 Assisted-by: Codex Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev> Chainloop-Trace-Sessions: 01a15f80-7cef-7100-a8fc-389f64109471

Chainloop-Trace-Sessions: 01a16073-95dd-7912-9463-66bbce3e4522 Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

docs(trace): record full Codex E2E verification

a92d6d0

Refs #3657 Assisted-by: Codex Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev> Chainloop-Trace-Sessions: 01a15f80-7cef-7100-a8fc-389f64109471

cubic-dev-ai Bot reviewed Oct 21, 2026

View reviewed changes

Comment thread app/cli/internal/trace/codex/materials.go

migmartri requested a review from a team October 21, 2026 18:35

matiasinsaurralde previously approved these changes Oct 21, 2026

View reviewed changes

migmartri dismissed matiasinsaurralde’s stale review via 4a104f5 October 21, 2026 22:06

fix(trace): preserve unsupported and failed Codex writes

ab5873d

Chainloop-Trace-Sessions: 01a16073-95dd-7912-9463-66bbce3e4522 Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

migmartri force-pushed the codex/issue-3657-milestone-2 branch from 4a104f5 to ab5873d Compare October 21, 2026 22:08

migmartri requested a review from a team October 21, 2026 22:12

matiasinsaurralde approved these changes Oct 21, 2026

View reviewed changes

migmartri merged commit ef1ee5a into main Oct 21, 2026
16 of 17 checks passed

migmartri deleted the codex/issue-3657-milestone-2 branch October 21, 2026 22:21

Conversation

migmartri commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Provider package (app/cli/internal/trace/<agent>/)

CLI wiring

End-to-end tests and documentation

Uh oh!

chainloop-platform Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR validation — ✅ 3 passing

AI Session Checks — 🟡 76% · ⚠️ 1 failing

🟡 80% — 99% AI — ✅ All policies passing

Change Summary

AI Session Overall Score

AI Session Analysis Breakdown

🟢 90% · context-and-planning

🟢 88% · scope-discipline

🟢 86% · solution-quality

🟢 84% · user-trust-signal

🟡 72% · alignment

🟡 68% · verification

File Attribution

Policies (4)

🟡 72% — 94% AI — ⚠️ 1 policies failing

Change Summary

AI Session Overall Score

AI Session Analysis Breakdown

🟢 88% · scope-discipline

🟢 86% · solution-quality

🟢 83% · context-and-planning

🟡 68% · alignment

🟡 56% · verification

🟡 54% · user-trust-signal

File Attribution

Policies (4, 1 failing)

Security Checks — ✅ 5 passing

✅ secret-scan

✅ sast-scan

Security context

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

migmartri commented Oct 21, 2026 •

edited

Loading

Provider package (`app/cli/internal/trace/<agent>/`)

chainloop-platform Bot commented Oct 21, 2026 •

edited

Loading

✅ `secret-scan`

✅ `sast-scan`

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading