[Bogdan (Dan) Baciu (bogdanbaciu21)]

What Problem This Solves

With projects: "all", the engine's own Claude replay sessions are harvested and can be mined back as user tasks. _is_headless_replay recognizes engine sessions by fixed prompt markers written for earlier prompt wording. None of them match the prompts the Claude backend sends today, and the duration fallback only covers prompts under 200 characters.

The earlier revision of this PR matched the opening lines of the current prompt registry instead. That cannot tell who produced a session: it also dropped an ordinary user session that quotes the grader prompt, and it missed an engine attempt that uses a task-supplied system prompt.

The other four fixes from the original PR landed through #392. This PR now contains only the harvest change.

Proof

Two kinds of evidence: a run through a real claude CLI, and deterministic tests.

Live run, real CLI. The installed claude CLI (2.1.322) wrote real transcripts into a config directory created for the run, driven by the shipped ClaudeCliBackend on two trees: main at 45f65ca and this branch at f6a0b6f. Three engine operations per tree: attempt, judge, and tool attempt (the tool attempt really called the search shim). No API key was set, so the backend's isolation guard probed the CLI and took --safe-mode, which this CLI advertises. The CLI talked to an Anthropic-compatible endpoint (OpenRouter, model deepseek/deepseek-v4.1-flash-1017). A normal user session that quotes the full grader prompt, and mentions the sentinel in the middle of the message, was then added in the CLI's own record format. Each tree's real harvest(..., scope="all") read each directory. The design and the results that would count against this PR were written down before the run.

Sessions written by	Read by the harvester of	Engine sessions on disk	Engine sessions harvested	User session quoting the grader prompt
main backend	main	3	3	kept
main backend	this branch	3	3	kept
this branch's backend	main	3	3	kept
this branch's backend	this branch	3	0	kept

• In all three transcripts written by this branch, the first user message starts with [skillopt-sleep:claude-engine:v1] at offset zero, and each transcript has exactly one user message. The CLI did not wrap or split the stdin prompt.
• On main, all three engine sessions are harvested as user sessions. That is the problem this PR fixes, reproduced with the real CLI.
• The second row is the stated limitation shown live: sessions written without the sentinel are still harvested after this change.
• Six CLI invocations in total. The CLI reports no cost per response; the key's usage counter rose by 0.0019 USD over the run, which is an upper bound because the key was shared.

Deterministic tests. Only the Claude child process (subprocess.run) is faked. Engine prompts are captured from the shipped ClaudeCliBackend and every session goes through the real harvest(..., scope="all"). Sessions are nine seconds long, so the short-session duration fallback cannot be what drops them.

The new file tests/test_harvest_claude_engine_sentinel.py has 16 cases. The second and third columns come from a comparison copy of the file that defines the sentinel string locally and omits the two assertions that the captured prompt is stamped, because those trees do not stamp.

Case	This branch (f6a0b6f)	main (45f65ca)	main plus the earlier registry-marker change
Engine sessions from the real backend path are dropped: attempt, judge, reflect, tool attempt, custom system prompt, miner	6 pass	6 fail	5 pass; custom system prompt fails
Engine session from an overridden judge template is dropped	passes	fails	passes
User session quoting the full grader prompt is kept, default and overridden template	2 pass	2 pass	2 fail
User session quoting the sentinel, or a stamped prompt, later in a message is kept	passes	passes	fails
Multi-turn sessions are kept, including one whose first prompt starts with the sentinel	passes	passes	passes
Fixed markers still drop a pre-sentinel transcript	passes	passes	passes
Sentinel has the same shape as the Codex one	passes	passes	passes
Engine session recorded with CRLF line breaks is dropped	passes	depends on stamping	depends on stamping
A single-turn session that begins with the sentinel line is dropped (the boundary of the rule)	passes	depends on stamping	depends on stamping
Stamping does not change the attempt cache or the token estimate	passes	depends on stamping	depends on stamping

The last three cases exercise the stamp itself, so their result on a tree that does not stamp says nothing about that tree and is not counted as evidence here.

Totals: 16 passed on this branch. Comparison copy: 10 failed, 6 passed on main, 5 failed, 11 passed on main with the earlier change applied.

Why This Change Was Made

The harvester now identifies the producer instead of matching prompt text, the same way #287 does for Codex.

• ClaudeCliBackend starts every prompt it hands to the CLI with CLAUDE_REPLAY_SENTINEL ([skillopt-sleep:claude-engine:v1]) and a blank line. The stamp is added after the template is rendered, on both the text path and the tool path, so it does not depend on any template wording, default or custom.
• _is_headless_replay drops a single-turn session whose prompt starts with the sentinel. A session that quotes an engine prompt, or the sentinel itself, later in a message is kept. Multi-turn sessions are kept, as before.
• The sentinel has the same shape as CODEX_REPLAY_SENTINEL: versioned, written by the backend, and matched only at the start of the prompt. The Codex and Cursor harvesters are untouched.
• The fixed markers that were already there are unchanged, so transcripts written before the sentinel existed are handled exactly as on main.
• Cache keys and the token estimate are still computed from the unstamped prompt, so caching and accounting are unchanged.

Project Fit

• Keeps engine output out of the nightly task pool, so the optimizer is not trained on its own prompts.
• No new configuration and no new dependency. Text-matching of current prompts is not introduced, so rewording or overriding a prompt cannot change what is harvested.

User Impact

Operators who harvest every project stop seeing the engine's own attempt, judge, reflection, miner, and tool sessions come back as mined tasks. People who discuss or paste SkillOpt prompts in their own sessions keep those sessions.

Academic Support

This is a bug fix, so the support is lineage, not literature.

• Exclude SkillOpt-generated Codex replay sessions from harvest #287: the Codex producer marker (CODEX_REPLAY_SENTINEL, written by the backend, matched at the start of the prompt). This PR applies the same mechanism to the Claude backend; a test asserts the two markers share one shape. The Cursor backend on main follows the same pattern.
• fix(sleep): four evidence-integrity fixes from #310 #392: landed the other four fixes from the original version of this PR. Only the harvest change remains here.
• fix(sleep): refuse uncontrolled Claude OAuth replays #336: the Claude replay isolation guard. The live run above goes through it unmodified and records which mode it chose.
• The earlier revision of this PR (registry-derived text markers) is the approach this one replaces. The comparison column in the test table shows what it got wrong.

Testing

Head f6a0b6fa135a9237eaca103a9f2f9b6be5c66a8a, based on main at 45f65ca04f5f6713bca7b9797b495696ba812751. Five hosted runners, each cloning the branch and asserting the exact commit before testing. The same workflow ran on unmodified main the same day, shown beside it. These are contributor-run checks, not official CI.

Platform	Python	This branch, complete suite	main, complete suite	Strict docs
Ubuntu	3.10	2220 passed, 24 skipped, 4 xfailed, 381 subtests passed	2204 passed, 24 skipped, 4 xfailed, 381 subtests passed	n/a
Ubuntu	3.11	2220 passed, 24 skipped, 4 xfailed, 381 subtests passed	2204 passed, 24 skipped, 4 xfailed, 381 subtests passed	n/a
Ubuntu	3.12	2220 passed, 24 skipped, 4 xfailed, 381 subtests passed	2204 passed, 24 skipped, 4 xfailed, 381 subtests passed	passed
macOS arm64	3.12	2 failed, 2218 passed, 24 skipped, 4 xfailed, 381 subtests passed	2 failed, 2202 passed, 24 skipped, 4 xfailed, 381 subtests passed	passed
Windows	3.12	24 failed, 2145 passed, 80 skipped, 4 xfailed, 376 subtests passed	24 failed, 2129 passed, 80 skipped, 4 xfailed, 376 subtests passed	passed

• On every runner this branch passes exactly 16 more tests than main: the 16 cases in tests/test_harvest_claude_engine_sentinel.py.
• The failing tests on macOS and Windows are the same test ids on this branch and on main, and none is in a file this PR touches. macOS, both trees: two tests in tests/test_sleep_adopt_skill_subset.py (test_cmd_adopt_refuses_tampered_manifest_roots and test_tampered_manifest_roots_cannot_escape_caller_destinations). Windows, both trees: those two, four in tests/test_benchmark_task_output_paths.py, one in tests/test_harvest_project_scope.py, and twelve in tests/test_sleep_config_encoding.py (19 test ids, 24 failures counting subtests). They are reported, not fixed, to keep this PR scoped.
• Local Linux x86_64, Python 3.12: git diff --check clean; ruff check on the touched modules gives 8 findings, the same 8 as on main, and the new test file has none.

Limitations & Negative Results

• Position is the only signal. A single-turn session whose one message begins with the sentinel line is treated as engine-produced, whoever typed it. A test pins this.
• Transcripts written before this change carry no sentinel. They are covered only by the fixed markers already on main, which do not match the current prompts, so older engine sessions are not caught retroactively.
• The harvest side relies on the Claude CLI recording the stdin prompt as the first user message with the sentinel still at its start. This was checked against CLI 2.1.322 on Linux, in --safe-mode, through one provider. Other CLI versions, the --bare route, macOS and Windows CLIs were not checked live. If a CLI version wraps the prompt, the session would be harvested as it is on main today, not dropped by mistake.
• The model sees one extra first line. In the live run the stamped attempt, judge and tool attempt all completed normally, but that is three calls on one model, not a comparison of answer quality with and without the line.
• Only the Claude backend is stamped. Codex and Cursor already carry their own markers, and Pi replays run with --no-session. The OpenCode and Copilot backends are not changed here.
• --no-session-persistence was not added to claude -p, because older CLI versions reject the flag.
• macOS and Windows are not green on main today (2 and 24 failures). This PR neither causes nor fixes them.

Reproduce It Yourself

Linux or macOS, shell in a fresh working directory:

git clone --no-checkout https://github.com/microsoft/SkillOpt.git SkillOpt-review-310
cd SkillOpt-review-310
git fetch origin refs/pull/310/head
git checkout --detach f6a0b6fa135a9237eaca103a9f2f9b6be5c66a8a
python3 -m venv .venv
.venv/bin/python -m pip install -e '.[dev,docs]'
.venv/bin/python -m pytest tests/test_harvest_claude_engine_sentinel.py -q
.venv/bin/python -m pytest -q
.venv/bin/python -m mkdocs build --strict

The live check, with your own authenticated claude CLI. Run it once in a checkout of main and once in this branch. It writes real sessions into a throwaway config directory and prints how many of them the harvester returns: 2 on main, 0 here. If you authenticate with an API key instead of a login, the backend takes --bare instead of --safe-mode.

export CLAUDE_CONFIG_DIR="$(mktemp -d)"
# sign in for this directory first (run `claude` once), or export ANTHROPIC_API_KEY
.venv/bin/python - <<'PY'
import os
from skillopt_sleep.backend import ClaudeCliBackend
from skillopt_sleep.harvest import harvest
from skillopt_sleep.types import TaskRecord

backend = ClaudeCliBackend()
task = TaskRecord(id="t", project="/demo", intent="In one sentence, what is a held-out set for?",
                  reference_kind="rubric", reference="Mentions unseen data.")
answer = backend.attempt(task, "Answer briefly.", "")
backend.judge(task, answer)
kept = harvest(os.path.join(os.environ["CLAUDE_CONFIG_DIR"], "projects"), scope="all")
print("isolation:", sorted(backend.isolation_modes_used), "engine sessions harvested:", len(kept))
PY

The offline equivalent, with no CLI and no key, is tests/test_harvest_claude_engine_sentinel.py: the same prompts captured from the shipped backend and run through the real harvester.

[Yifan Yang (Yif-Yang)]

Reviewed 964d9c050935d57b73999901961d6657f25a5788. The five-component selection passes 44 positive cases, including measured tool evidence, judge-score bounds, learned-item preservation and content-twin gate protection. One new harvest regression prevents recommending the whole patch: a normal user session discussing and quoting the current grader prompt is dropped because a registry-derived marker matches anywhere in its text. The same synthetic session is retained on current main.

Please distinguish producer provenance from ordinary quoted prompt content, including custom-template cases, and keep a real harvest regression for both engine sessions and normal lookalikes. Coordinate with #287's Codex producer marker and #272's identity-based exposure tracking; those address related but different boundaries. I have not rewritten or split the feature on your behalf, and the broader issues remain open.

[Yifan Yang (Yif-Yang)]

Thank you, Dan, and apologies for how long this has taken.

The four evidence-integrity fixes from this PR are now on main through #392, which was rebase-merged so your four commits land unchanged and under your name. On current main, each fix's new tests fail without it and pass with it, and the full suite and official CI pass. I added one small maintainer commit: the tool-route test sets a test API key so it still exercises the measured route after #336's Claude isolation guard, plus a CHANGELOG entry crediting you.

The fifth change, the harvest marker, is the only open point, and it is the same one from my Oct 17 review. Markers derived from the prompt registry also drop an ordinary user session that discusses or quotes the grader prompt; the same session is kept on main. If you'd like to continue it here, rebase onto current main so this PR contains only that commit. The acceptance criteria, all in one place:

• distinguish engine-produced sessions from ordinary sessions that merely quote prompt text, including custom templates;
• keep a real harvest regression for both an engine session and a normal look-alike;
• coordinate with Exclude SkillOpt-generated Codex replay sessions from harvest #287's Codex producer marker.

Nothing beyond those.

[Bogdan (Dan) Baciu (bogdanbaciu21)]

Thank you for landing the four fixes through #392 and for keeping the commits and authorship intact.

This PR is now rebased onto current main (45f65ca) and contains one commit, f6a0b6fa135a9237eaca103a9f2f9b6be5c66a8a, for the harvest marker. It replaces the earlier approach instead of patching it. Your Oct 17 point was right: matching prompt text cannot tell who produced a session.

Against the three criteria:

1. Engine-produced versus quoted, including custom templates. The harvester no longer matches registry prompt text. ClaudeCliBackend stamps a producer sentinel, [skillopt-sleep:claude-engine:v1], at the start of every prompt it hands to the CLI, after the template is rendered, on both the text path and the tool path. _is_headless_replay drops a single-turn session only when its prompt starts with that sentinel. A user session that quotes the grader prompt, or quotes the sentinel later in a message, is kept. A custom template needs no registration because the stamp does not depend on the template.
2. Real harvest regressions for both. tests/test_harvest_claude_engine_sentinel.py captures the prompts the shipped backend sends and runs every session through harvest(..., scope="all"). Engine sessions for attempt, judge, reflect, tool attempt, a task-supplied system prompt, the miner, and an overridden judge template are dropped. A normal session quoting the full grader prompt is kept, for the default and for an overridden template. That last case is yours from Oct 17: it fails when the earlier registry-marker change is applied to current main and passes here. The earlier change also missed the task-supplied system prompt.
3. Coordination with Exclude SkillOpt-generated Codex replay sessions from harvest #287. Same mechanism and same shape as CODEX_REPLAY_SENTINEL: a versioned producer marker written by the backend and matched at the start of the prompt. A test asserts the two share one shape. The Codex and Cursor harvesters are untouched.

Things you would otherwise have to find:

• Position is the only signal, so a single-turn session whose one message begins with the sentinel line is treated as engine-produced. A test pins that boundary.
• Transcripts written before the sentinel existed are covered only by the fixed markers already on main, which are unchanged. Older engine sessions are not caught retroactively.
• Cache keys and the token estimate are computed from the unstamped prompt, with a test. The model does see one extra first line; I did not run a live-model comparison.
• I did not check the recorded transcript against a live claude CLI. If a CLI version wraps the stdin prompt, the session would be harvested as it is on main today, not dropped by mistake.

Validation on f6a0b6fa135a9237eaca103a9f2f9b6be5c66a8a, Linux and Python 3.12: 16 passed in the new file; complete suite 2220 passed, 24 skipped, 4 xfailed, 8 warnings, 381 subtests passed (main on the same machine: 2204 passed, 24 skipped, 4 xfailed, 8 warnings, 381 subtests passed); strict docs pass; git diff --check clean. These are contributor-run results on one machine, not official CI. macOS and Windows were not run.

[Bogdan (Dan) Baciu (bogdanbaciu21)]

I updated the description. Proof now comes second and closes the gap I had listed: I ran the real claude CLI (2.1.322) through the shipped backend, and the sentinel lands at the start of the first user message, the branch drops all three engine sessions and keeps a user session quoting the grader prompt, and main harvests all three. Testing now has the five-runner matrix (Ubuntu 3.10, 3.11, 3.12, macOS arm64, Windows) with unmodified main beside it. macOS and Windows fail the same tests on both trees, none in files this PR touches.