feat(sleep): add opt-in leave-one-out edit ablation for accepted gate trials

CHANGELOG.md

@@ -7,6 +7,18 @@ All notable changes to SkillOpt are documented here. This project adheres to
 ## [Unreleased]
 
 ### Added
+- **SkillOpt-Sleep leave-one-out edit ablation** (`gate_edit_ablation`, off by
+  default): each skill or memory candidate the night accepts is replayed
+  without each of its edits, and the per-edit observation (gate score with and
+  without the edit, `observed_change`, changed validation tasks, source task
+  and session IDs) is recorded on the trial in `report.md`, `report.json`,
+  `diagnostics.json`, the CLI's `--json` output, and the evidence log. A row is
+  an observed change on one replay, not a causal attribution. Ablation runs
+  after every gate decision of the night, including all fan-out groups, so
+  decisions and accepted edits are unchanged even when the token budget binds.
+  A single-edit candidate needs no extra replay; an N-edit candidate costs up
+  to N validation replays, which are included in `tokens used` and checked
+  against `max_tokens_per_night` one replay at a time. Follow-up to #174.
 - **OfficeQA Parallel Search MCP** as an opt-in `env.search_mode=parallel`
   provider for `openai_chat`/`qwen_chat`. The default stays `offline` and no
   config selects Parallel automatically. Selecting it sends the generated

docs/sleep/README.md

@@ -35,6 +35,40 @@ candidate; an absent numeric score aborts evaluation. Task-level changes are
 included in `report.md`, `report.json`, `diagnostics.json`, the CLI's `--json`
 output, and the evidence log.
 
+The optional `gate_edit_ablation` config key records, for each accepted edit,
+how the held-out score compares with and without it. It defaults to `false`.
+When set to `true`, each skill or memory candidate the night accepts is
+replayed on the same validation tasks once per edit, each time without that one
+edit. The result is an `edit_ablation` row per edit on that trial: the gate
+score with and without the edit, an `observed_change` of `higher_with_edit`,
+`no_change`, `lower_with_edit`, or `unmeasured`, the validation tasks whose
+score differs, and the edit's `source_task_ids` and `source_session_ids`.
+
+Read every row as an observed change on one replay, not a causal attribution.
+The "with" score is the replay that passed the gate, so it is selected for
+being high, while each "without" score is one further replay; rows therefore
+lean toward `higher_with_edit`. A nondeterministic backend or judge can move
+any row by noise, and where the response cache serves a replay, repeating the
+ablation returns the same result instead of a new sample. Leave-one-out also
+says nothing about interactions: two edits that are each redundant given the
+other both show `no_change`.
+
+The rows are report-only, and ablation runs last: after the final gate
+decision, the held-out test replay and every per-skill fan-out group. No
+decision reads the rows and no ablation replay can spend budget that a decision
+still needs, so the accepted edits and documents are the same with the key on
+or off. A candidate that the final decision rejects is not ablated. A candidate
+with one edit needs no extra replay, because the trial already compared the
+document with and without it; a candidate with N edits costs up to N validation
+replays. Those replays are ordinary model calls: their tokens are included in
+the report's `tokens used` and count toward `max_tokens_per_night`, which is
+checked before each of them. Replays that do not start leave their rows
+`unmeasured` and add one `edit ablation` entry to the budget note. With
+`gate_mode` set to `off` there is no validation replay to compare against, so
+the key has no effect. Rows are included in `report.md`, `report.json`,
+`diagnostics.json`, the CLI's `--json` output, and the evidence log; in
+multi-skill fan-out, each group's rows are in the evidence log.
+
 Rule-based heading checks are intentionally additive. `section_present` keeps
 its legacy strict behavior for existing task sets: an ATX heading cannot append
 text after the requested name. Use `section_contains` when a numbered,
@@ -445,6 +479,9 @@ skill and memory edit steps, before its final validation replay, before the
 held-out test replay, and before each per-skill fan-out group. Steps that did
 not start are listed in a note in `report.md`, for example
 `max_tokens_per_night reached (431870 of 400000 tokens); skipped: memory edits, final validation replay`.
+With `gate_edit_ablation` on, the ablation replays run after all of these
+steps, are checked against the budget one replay at a time, and count toward
+the same spend.
 
 When the final validation replay is skipped, the gate decides from the
 validation replay that already scored the same skill and memory: the baseline
@@ -504,8 +541,8 @@ This runs one additional consolidation per group (including a catch-all group wh
 hinted and unhinted evidence are mixed), so it multiplies backend calls and token
 use; configured dream rollouts and synthetic variants multiply the per-group work
 too. Each group inherits the configured edit budget, gate mode/metric,
-`gate_no_regression`, `dream_rollouts`, `dream_factor`, `recall_k`, and
-`evolve_skill`. Recalled archive tasks are restricted to that same skill hint;
+`gate_no_regression`, `gate_edit_ablation`, `dream_rollouts`, `dream_factor`,
+`recall_k`, and `evolve_skill`. Recalled archive tasks are restricted to that same skill hint;
 shared memory is read-only in fan-out runs. Setting `evolve_skill` to `false`
 therefore disables per-skill proposals as well as the managed skill proposal.
 

plugins/README.md

@@ -249,11 +249,13 @@ python -m skillopt_sleep run --project "$(pwd)" \
 ### Advanced config
 
 The JSON/YAML config under `~/.skillopt-sleep/` supports additional engine keys,
-including `gate_mode`, `gate_metric`, `gate_no_regression`, `dream_rollouts`,
-`dream_factor`, `recall_k`, `evolve_memory`, and `evolve_skill`. These are config
+including `gate_mode`, `gate_metric`, `gate_no_regression`,
+`gate_edit_ablation`, `dream_rollouts`, `dream_factor`, `recall_k`,
+`evolve_memory`, and `evolve_skill`. These are config
 keys, not aliases for the unsupported CLI flags listed above. Shipping defaults
 are conservative: `gate_mode="on"`, `gate_no_regression=false`,
-`dream_rollouts=1`, `dream_factor=0`, and `recall_k=0`.
+`gate_edit_ablation=false`, `dream_rollouts=1`, `dream_factor=0`, and
+`recall_k=0`.
 
 The managed `schedule` command stores only the project, backend, time, and
 optional auto-adopt setting. It does not copy `--source`, `--cursor-home`,

skillopt_sleep/config.py

@@ -68,6 +68,7 @@
     "gate_metric": "mixed",       # hard | soft | mixed (mixed best for tiny holdouts)
     "gate_mixed_weight": 0.5,
     "gate_no_regression": False,    # reject any candidate that lowers a val-task score
+    "gate_edit_ablation": False,    # after all decisions, replay accepted trials minus each edit (report-only)
     "replay_mode": "mock",        # report label; fresh-worktree replay is not implemented
     # ── dream + recall (opt-in; defaults reproduce the prior single-shot loop) ─
     "dream_rollouts": 1,          # >1 => multi-rollout contrastive reflection per task

skillopt_sleep/consolidate.py

@@ -188,6 +188,63 @@ def _task_deltas(
     return out
 
 
+def _ablation_observed_change(
+    score_with: float | None,
+    score_without: float | None,
+    task_rows: List[dict],
+) -> str:
+    """Name how the gate score with one edit compares to the score without it.
+
+    ``higher_with_edit`` means the candidate scored higher with the edit than
+    without it, ``lower_with_edit`` the reverse, and ``no_change`` that the two
+    scores are equal. This is one observation, not a causal attribution: the
+    "with" score is the replay that passed the gate, the "without" score is a
+    single further replay. A comparison with a missing or non-finite score
+    observes nothing, so it is ``unmeasured``.
+    """
+    if score_with is None or score_without is None:
+        return "unmeasured"
+    if not (math.isfinite(score_with) and math.isfinite(score_without)):
+        return "unmeasured"
+    if any(not row.get("scores_are_finite", True) for row in task_rows):
+        return "unmeasured"
+    if score_with > score_without:
+        return "higher_with_edit"
+    if score_with < score_without:
+        return "lower_with_edit"
+    return "no_change"
+
+
+def _ablation_task_changes(task_rows: List[dict]) -> List[dict]:
+    """Keep the validation tasks whose score differs with and without an edit."""
+    return [
+        {
+            "task_id": row["task_id"],
+            "tags": row["tags"],
+            "score_without": row["baseline_score"],
+            "score_with": row["candidate_score"],
+            "status": row["status"],
+        }
+        for row in task_rows
+        if row["status"] != "unchanged"
+    ]
+
+
+def run_edit_ablations(queue: List[Callable[[], bool]]) -> bool:
+    """Run edit ablations that ``consolidate`` deferred, then empty the queue.
+
+    A caller that runs several consolidations under one token budget passes
+    the same list to each as ``edit_ablation_queue`` and calls this once every
+    decision that reads the budget has been made. Returns True when at least
+    one ablation replay was skipped because the budget was spent.
+    """
+    budget_skipped = False
+    while queue:
+        run = queue.pop(0)
+        budget_skipped = run() or budget_skipped
+    return budget_skipped
+
+
 def consolidate(
     backend: Backend,
     tasks: List[TaskRecord],
@@ -198,12 +255,14 @@ def consolidate(
     gate_metric: str = "mixed",
     gate_mixed_weight: float = 0.5,
     gate_no_regression: bool = False,
+    gate_edit_ablation: bool = False,
     gate_mode: str = "on",       # "on" (hard/soft per gate_metric) | "off" (greedy)
     rollouts_k: int = 1,         # >1 => multi-rollout contrastive reflection
     evolve_skill: bool = True,
     evolve_memory: bool = True,
     night: int = 1,
     budget_exhausted: Optional[Callable[[], bool]] = None,
+    edit_ablation_queue: Optional[List[Callable[[], bool]]] = None,
 ) -> ConsolidationResult:
     """Run one consolidation epoch: reflect -> bounded edit -> gate.
 
@@ -221,6 +280,22 @@ def consolidate(
     still rests on complete evidence. When the final replay is skipped, the
     decision uses the validation replay that already scored the final skill
     and memory: the baseline replay, or the trial replay of the last kept edit.
+
+    ``gate_edit_ablation`` replays, for each edit of an accepted skill or
+    memory trial, the candidate without that one edit, and records the
+    observed score difference on the trial under ``edit_ablation``. It runs
+    only after the final decision of an accepted epoch, so no decision reads
+    its rows and none of its replays spends budget a decision still needs. It
+    costs up to one validation replay per edit of a multi-edit trial; the
+    budget is checked before each replay, and replays that do not start are
+    reported as ``unmeasured`` under a single ``edit ablation`` entry in
+    ``budget_skipped``.
+
+    One budget often spans several consolidations (the nightly aggregate run,
+    then each fan-out group). Such a caller passes one list as
+    ``edit_ablation_queue``: this epoch's ablation is appended to it instead
+    of being run here, and the caller runs it with
+    :func:`run_edit_ablations` once every consolidation has decided.
     """
     from skillopt_sleep import evidence as evlog
     ev = evlog.get(backend)
@@ -289,6 +364,129 @@ def _edits_payload(edits: List[EditRecord]) -> List[dict]:
         return [{"op": e.op, "content": e.content, "anchor": e.anchor,
                  "rationale": e.rationale} for e in edits]
 
+    # Accepted trials waiting for their edit ablation. Only bookkeeping happens
+    # while the epoch is still deciding: every ablation replay is deferred
+    # until after the final decision, so it cannot spend budget that the
+    # memory step, the final validation replay or a later epoch still checks.
+    pending_ablations: List[dict] = []
+
+    def _ablate_edits(plan: dict, budget_spent: bool) -> Tuple[List[dict], bool]:
+        """Leave-one-out replay of one accepted trial.
+
+        For each applied edit, rebuild the candidate from the pre-trial
+        document without that edit and score it on the same validation tasks.
+        Two cases need no replay: the document is the same without the edit
+        (another edit of the trial overrides it), and no edit is left once it
+        is removed, which is the pre-trial document ``prior_pairs`` scored.
+
+        The budget is checked before every replay. Returns the rows and
+        whether the budget is spent, so a later trial does not start a replay
+        the budget no longer covers.
+        """
+        which = plan["which"]
+        doc, new_doc, applied = plan["doc"], plan["new_doc"], plan["applied"]
+        with_pairs, with_score = plan["with_pairs"], plan["with_score"]
+        rows: List[dict] = []
+        for index, edit in enumerate(applied):
+            remaining = [e for j, e in enumerate(applied) if j != index]
+            without_doc, still_applied, _unmatched = apply_edits_detailed(
+                doc, remaining
+            )
+            kept = {id(e) for e in still_applied}
+            # Edits that only matched because of the removed one (an anchor it
+            # added) drop out with it; name them so the row is not read as an
+            # observation about a single edit.
+            co_dropped = [
+                j for j, e in enumerate(applied)
+                if j != index and id(e) not in kept
+            ]
+            replayed = False
+            note = ""
+            without_pairs: Optional[List[Tuple[TaskRecord, ReplayResult]]]
+            if not still_applied:
+                without_pairs = plan["prior_pairs"]
+                note = "scored by the replay that preceded this trial"
+            elif without_doc == new_doc:
+                without_pairs = with_pairs
+                note = "document is the same without this edit"
+            else:
+                if not budget_spent and budget_exhausted is not None:
+                    budget_spent = bool(budget_exhausted())
+                    if budget_spent and ev is not None:
+                        ev.log("budget", "skip", step="edit ablation",
+                               night=night)
+                if budget_spent:
+                    without_pairs = None
+                    note = "not replayed: token budget spent"
+                else:
+                    evlog.set_phase(backend, f"gate_ablation:{which}")
+                    without_pairs = replay_batch(
+                        backend,
+                        val_tasks,
+                        without_doc if which == "skill" else plan["skill"],
+                        without_doc if which == "memory" else plan["memory"],
+                    )
+                    replayed = True
+            if without_pairs is None:
+                without_score = None
+                task_rows: List[dict] = []
+            else:
+                h, s = aggregate_scores(without_pairs)
+                without_score = select_gate_score(
+                    h, s, gate_metric, gate_mixed_weight
+                )
+                task_rows = _task_deltas(
+                    val_tasks, without_pairs, with_pairs,
+                    gate_metric, gate_mixed_weight,
+                )
+            observed = _ablation_observed_change(
+                with_score, without_score, task_rows
+            )
+            if observed == "unmeasured":
+                # A missing or non-finite score compares nothing. Report no
+                # score and no per-task direction for it, instead of a mean
+                # over an incomplete slice or the fail-closed "regressed"
+                # label the gate uses for the same condition.
+                without_score = None
+                task_rows = []
+                note = note or "a validation score was missing or non-finite"
+            rows.append({
+                "edit_index": index,
+                "target": which,
+                "op": edit.op,
+                "content": edit.content,
+                "anchor": edit.anchor,
+                "source_task_ids": list(edit.source_task_ids),
+                "source_session_ids": list(edit.source_session_ids),
+                "score_with": _finite_score(with_score),
+                "score_without": (
+                    None if without_score is None
+                    else _finite_score(without_score)
+                ),
+                "observed_change": observed,
+                "replayed": replayed,
+                "note": note,
+                "co_dropped": co_dropped,
+                "task_changes": _ablation_task_changes(task_rows),
+            })
+        if ev is not None:
+            ev.log("gate", "edit_ablation", target=which,
+                   n_edits=len(applied),
+                   n_replays=sum(1 for row in rows if row["replayed"]),
+                   rows=rows)
+        return rows, budget_spent
+
+    def _run_pending_ablations() -> bool:
+        """Ablate every accepted trial of this epoch, in trial order.
+
+        Returns True when a replay was skipped because the budget was spent.
+        """
+        budget_spent = False
+        for plan in pending_ablations:
+            rows, budget_spent = _ablate_edits(plan, budget_spent)
+            plan["trial"]["edit_ablation"] = rows
+        return budget_spent
+
     def _gate_apply(doc: str, edits: List[EditRecord], which: str) -> str:
         nonlocal cand_skill, cand_memory, base_score, current_pairs
         nonlocal all_applied, all_rejected
@@ -344,6 +542,21 @@ def _gate_apply(doc: str, edits: List[EditRecord], which: str) -> str:
                    task_deltas=task_deltas,
                    n_edits=len(applied))
         if improved:
+            if gate_edit_ablation:
+                # Bookkeeping only. The other target's document is captured
+                # now because it can change before the ablation runs.
+                pending_ablations.append({
+                    "trial": gate_trials[-1],
+                    "which": which,
+                    "doc": doc,
+                    "new_doc": new_doc,
+                    "applied": list(applied),
+                    "with_pairs": pairs,
+                    "with_score": cand_score,
+                    "prior_pairs": current_pairs,
+                    "skill": cand_skill,
+                    "memory": cand_memory,
+                })
             base_score = cand_score
             current_pairs = pairs
             all_applied.extend(applied)
@@ -537,6 +750,20 @@ def _gate_apply(doc: str, edits: List[EditRecord], which: str) -> str:
                n_rejected=len(all_rejected),
                n_unmatched=len(all_unmatched), night=night)
 
+    # Read the backend's diagnostics before ablation can add calls of its own.
+    reflect_raw = getattr(backend, "last_reflect_raw", "") or ""
+    call_error = getattr(backend, "last_call_error", "") or ""
+
+    # ── edit ablation (report-only, after every decision of this epoch) ───
+    # A rejected epoch has no accepted edit to describe, so nothing is replayed.
+    if gate_edit_ablation and accepted and pending_ablations:
+        if edit_ablation_queue is not None:
+            # The caller still has budget-dependent decisions to make (later
+            # fan-out groups, the held-out test replay) and runs this itself.
+            edit_ablation_queue.append(_run_pending_ablations)
+        elif _run_pending_ablations():
+            budget_skipped.append("edit ablation")
+
     return ConsolidationResult(
         accepted=accepted,
         gate_action=action,
@@ -551,8 +778,8 @@ def _gate_apply(doc: str, edits: List[EditRecord], which: str) -> str:
         holdout_candidate=final_hard,
         holdout_detail=holdout_detail,
         gate_trials=gate_trials,
-        reflect_raw=getattr(backend, "last_reflect_raw", "") or "",
-        call_error=getattr(backend, "last_call_error", "") or "",
+        reflect_raw=reflect_raw,
+        call_error=call_error,
         holdout_leaked=holdout_leaked,
         budget_skipped=budget_skipped,
     )