Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions app/cli/internal/trace/PROVIDERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,9 +159,9 @@ The handlers find these interfaces with a type assertion. A provider without one

| Interface | Feature | Implemented by | What to do |
|-----------|---------|----------------|------------|
| `trace.SkillTracker` | Skills tracking | Claude, OpenCode | `NewSkillLoads`: return the skill folders that the session loaded since the last call (from the hook input or the transcript). `SessionSkills`: return the skill uses from the copied session data, in the order of the first use. `SkillRoots`: return the plugin, organization, user, and project skill folders of the agent. |
| `trace.SpecCopyReplacer` | Spec source pointers | Claude | Replace each exact copy of a spec source in the raw session with `pointer.New(digest)`. Report the result per finder. |
| `trace.PastedImageFinder` | Pasted image pointers | Claude | Return each distinct pasted image of the raw session, with its digest, media type, and number. |
| `trace.SkillTracker` | Skills tracking | Claude, OpenCode, Codex | `NewSkillLoads`: return the skill folders that the session loaded since the last call (from the hook input or the transcript). `SessionSkills`: return the skill uses from the copied session data, in the order of the first use. `SkillRoots`: return the plugin, organization, user, and project skill folders of the agent. |
| `trace.SpecCopyReplacer` | Spec source pointers | Claude, Codex | Replace each exact copy of a spec source in the raw session with `pointer.New(digest)`. Report the result per finder. |
| `trace.PastedImageFinder` | Pasted image pointers | Claude, Codex | Return each distinct pasted image of the raw session, with its digest, media type, and number. |

Add a compile-time check for each one, for example `var _ trace.SkillTracker = (*Provider)(nil)`.

Expand Down
16 changes: 10 additions & 6 deletions app/cli/internal/trace/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,15 @@ Trace does not support Pi yet (spec issue-3520): `chainloop trace init --pi` and

Codex uses native hooks in `.codex/hooks.json` (Codex CLI 0.163.0). Run `chainloop trace init --codex` or `chainloop trace run --local --codex -- codex exec …`. Review and trust the installed hooks in Codex before starting a session. See the [official hooks documentation](https://learn.chatgpt.com/docs/hooks). Installed hooks have a `Chainloop Trace` status label in Codex. Re-running init adds labels to existing Chainloop hooks. Changing the hooks requires a new trust review.

Codex was tested with CLI 0.163.0 and `openrouter/openai/gpt-6-luna`: 22 feature checks passed with no failures. The tables keep unverified features at **Unknown**. Subagent transcripts, skills tracking, spec source pointers, and pasted image pointers are not implemented.
Codex was tested with CLI 0.163.0 and `openrouter/openai/gpt-6-luna`: all 25 tested feature checks passed in the full Codex suite. The tables keep unverified features at **Unknown**. Subagent transcripts are not implemented.

Milestone 2 live checks passed with the same CLI and model: skills in 3/3 sessions, spec pointers in 3/3 sessions, and pasted images in 1/1 session. The image scenario attaches the same image twice and requires one stored material. Transcript rendering and media redaction checks also passed. Provider and assembly regressions cover malformed images, storage failures, and reuse of an existing image material.

Codex token usage comes from cumulative rollout counters. Cost is an estimate from the embedded standard API prices, with a warning for unknown models. Subscription billing and third-party provider prices can differ. Trace reads rollouts from `$CODEX_HOME/sessions` (default `~/.codex/sessions`) or `archived_sessions`. Their JSONL format is not a stable Codex interface.

In a linked Git worktree, Codex 0.163 reads project hooks from the primary checkout. Install the Codex hooks in that checkout too, then review and trust them in Codex.

Codex evidence converts native messages, tool calls, and tool results to the message shape used by the existing session viewer. Text uses `message.content`, calls use `tool_use` blocks, and results use `tool_result` blocks with matching IDs. Each converted entry retains its original event in `source_event`; native metadata stays unchanged. Codex regression and E2E checks require renderable messages. When tools are used, they also require a completed exchange with readable tool output. A nonempty native transcript alone does not pass. These checks apply to Codex; `raw_session` remains open in the shared evidence schema.
Codex evidence converts native messages, tool calls, and tool results to the message shape used by the existing session viewer. Text uses `message.content`, calls use `tool_use` blocks, and results use `tool_result` blocks with matching IDs. Each converted entry retains its event in `source_event`. Spec and pasted-image pointers replace copies in viewer messages, retained events, and supported native timeline records. Other native metadata stays unchanged. Codex regression and E2E checks require renderable messages. When tools are used, they also require a completed exchange with readable tool output. A nonempty native transcript alone does not pass. These checks apply to Codex; `raw_session` remains open in the shared evidence schema.

## Setup

Expand Down Expand Up @@ -111,16 +113,18 @@ What the session was asked to build, and which skills it used.
| Spec capture reminder | At each user prompt, the agent is reminded to capture new or changed specs. | Yes | No | Yes | Yes | No | Unknown |
| Local spec sources read at push | A capture whose source is a local file holds only a header and a placeholder. Each push reads the file and records its current content, so the evidence follows the edits of the file. Only a regular text file of up to 1 MiB is read. Otherwise the push keeps the body that the agent wrote, or drops the capture with a warning when there is no body (spec issue-3561). | Yes | Yes | Yes | Yes | No | Yes |
| Spec source role, title, and description | Each captured spec source holds the role, the title, and the description that the agent wrote in its header (spec issue-3531). | Yes | Yes | Unknown | Yes | No | Yes |
| Skills tracking | The evidence lists the skills that the session used, with a copy of each skill as it ran. | Yes | No | Partial | Partial | No | No |
| Spec source pointers | Exact copies of spec sources in the transcript (full file reads and file writes) are replaced with a pointer to the spec material, so the evidence does not hold the same content two or three times (spec issue-3556). This includes the copies of pasted images and spec sources in the `api-request-blob` records of a Claude Code transcript (issue 3654). | Yes | No | No | No | No | No |
| Pasted image pointers | Each distinct image that the user pasted into the session, including images in prompts sent while the agent was busy, is stored once as an image spec entry, from the data in the transcript, with the title "Pasted image N" and no role. A repeated paste of the same image adds no entry. The inline image in the transcript is replaced with a pointer to that material. An image whose data cannot be decoded, or that the push could not store, stays inline, and the push continues. When the agent captured the same image, its entry is kept and no second entry is added (spec issue-3569). | Yes | No | No | No | No | No |
| Skills tracking | The evidence lists the skills that the session used, with a copy of each skill as it ran. | Yes | No | Partial | Partial | No | Partial |
| Spec source pointers | Exact copies of spec sources in the transcript (full file reads and file writes) are replaced with a pointer to the spec material, so the evidence does not hold the same content two or three times (spec issue-3556). This includes the copies of pasted images and spec sources in the `api-request-blob` records of a Claude Code transcript (issue 3654). | Yes | No | No | No | No | Partial |
| Pasted image pointers | Each distinct image that the user pasted into the session, including images in prompts sent while the agent was busy, is stored once as an image spec entry, from the data in the transcript, with the title "Pasted image N" and no role. A repeated paste of the same image adds no entry. The inline image in the transcript is replaced with a pointer to that material. An image whose data cannot be decoded, or that the push could not store, stays inline, and the push continues. When the agent captured the same image, its entry is kept and no second entry is added (spec issue-3569). | Yes | No | No | No | No | Yes |

Why not Yes:

- **Spec capture reminder, Codex:** Codex does not report hook messages in its headless output, so the suite cannot inspect the reminder.
- **Spec capture reminder, Cursor:** Cursor has no hook at each user prompt. The agent gets the session-start instruction only.
- **Spec source role, title, and description, OpenCode 1.x:** not verified yet. The end-to-end suite finds the values (spec issue-3635).
- **Skills tracking, spec source pointers, and pasted image pointers, Codex:** these optional integrations are not implemented.
- **Skills tracking, Codex:** tracks filesystem skills loaded through explicit Codex skill instructions or successful shell reads of one identified `SKILL.md`. The definition and supporting files use the shared skill snapshot and packaging pipeline. Multiple distinct skill paths in one tool call cannot be attributed reliably and are not tracked. Skills without a filesystem folder and child sessions are not captured. A snapshot first made at push carries a warning.
- **Spec source pointers, Codex:** supports exact text results, native shell envelopes, recognized code-mode JSON envelopes, and complete file creation through native patches or literal patches in code mode. A code-mode write must be a single awaited `tools.apply_patch` call, optionally wrapped in `text`, with a literal argument or an immediately preceding `const` literal. Stored source files read with `cat` can also match within combined command output and native file-change records. Mixed or computed patches, other scripts, arbitrary output wrappers, transformed content, and other unsupported formats remain inline, so the table remains Partial. Earlier live sessions demonstrated this limit when the model trimmed the source's trailing newline.
- **Pasted image pointers, Codex:** extracts base64 image data URLs from user input, deduplicates their decoded bytes, and replaces stored copies in both viewer messages and retained events. Tool-result images are not pastes. Invalid data, remote URLs, and path-only image records remain inline.
- **Skills tracking, Cursor:** the provider does not track skills. The evidence has no skill entries, which means "not recorded", not "no skill used".
- **Skills tracking, OpenCode:** only the skills that the model starts with the `skill` tool are counted. Skills that the user starts, and skills used in subagents, are not. A plugin installed by an earlier version does not send the skill folder for OpenCode 2: install the hooks again to update it.
- **Spec source pointers, Cursor and OpenCode:** the provider has no finder for the copies in its transcript yet. For OpenCode, the rebuilt transcript holds no tool outputs, so only the write copies would apply.
Expand Down
Loading
Loading