feat(sleep): reject candidates that regress on tonight's training tasks (gate_train_check)

CHANGELOG.md

@@ -28,6 +28,16 @@ All notable changes to SkillOpt are documented here. This project adheres to
   search setting. An API key echoed in an error body is redacted before the
   text reaches `fail_reason`, and the round deadline is checked again after
   each response.
+- **SkillOpt-Sleep `gate_train_check`** config key (default `false`) with
+  `gate_train_margin` (default `0.1`): a candidate the validation gate would
+  accept is also replayed on tonight's real training tasks and is rejected
+  when its mean gate score there is lower than the current documents' by more
+  than the margin and at least two more of those tasks got worse than got
+  better. The check only rejects; it never accepts what validation rejected.
+  It is skipped with the gate off or a non-disjoint validation slice. Both
+  arms and every task-level change are kept in the gate records (`train_check`)
+  and a `gate`/`train_check` evidence event, and `report.md` shows the two
+  means and the counts. With the key unset, records and reports are unchanged.
 - **SkillOpt-Sleep paired A/B evalkit** (`python -m skillopt_sleep.evalkit`):
   McNemar plus percentile-bootstrap CIs on a fixed task manifest, with
   task-cluster multi-seed inference and seeded null calibration for exact-test

docs/sleep/README.md

@@ -35,6 +35,43 @@ candidate; an absent numeric score aborts evaluation. Task-level changes are
 included in `report.md`, `report.json`, `diagnostics.json`, the CLI's `--json`
 output, and the evidence log.
 
+The optional `gate_train_check` config key adds a condition to an accept. It
+defaults to `false`. A night's validation slice is a fraction of that night's
+tasks and can be a handful, so a rule that damages the agent can still come out
+ahead on it. When the check is on, a candidate the validation gate would accept
+is also replayed on tonight's real training tasks (recalled tasks included,
+dream variants not). It is rejected when two things hold: its mean score there,
+under the configured `gate_metric`, is lower than the current documents' by
+more than `gate_train_margin` (default `0.1`, a number between 0 and 1), and at
+least two more of those tasks got worse than got better, so one task flipping
+can never decide a night.
+
+These tasks are not held out, since reflection drew on tonight's failures, so
+scoring higher on them proves nothing and the check never accepts anything the
+validation gate rejected. Scoring clearly lower on them, in particular breaking
+tasks the current documents already handle, is not explained by overfitting,
+which is why the comparison is used only to reject. Each candidate is compared
+with the documents it would replace: a skill candidate with the night's
+training replay, a memory candidate with the replay that follows an accepted
+skill. The margin therefore applies per candidate, not per night. The added
+cost is one replay of the training tasks for each candidate that passes
+validation.
+
+A task with a missing or malformed score counts as worse, and a non-finite
+mean rejects the candidate. The check is skipped when `gate_mode` is `off` and
+when the validation slice is not disjoint from training. The gate records
+(`train_check`, in `report.json`, `diagnostics.json`, and the CLI's `--json`
+output) and the evidence log (`gate` / `train_check`) keep both arms and every
+task-level change; `report.md` shows the two means and the counts. These
+records are written only when the check is enabled. `gate_train_check` must be
+a boolean and, when it is `true`, an invalid margin raises before consolidation
+makes any model call.
+
+To measure the gate with and without this key on a public benchmark, run
+`python -m skillopt_sleep.experiments.gate_study` (fault injection of fixed rules
+and the multi-night recipe, both through `dream_consolidate`). `--offline` runs it
+on a committed fixture with no key and no network; live runs need `--live`.
+
 Rule-based heading checks are intentionally additive. `section_present` keeps
 its legacy strict behavior for existing task sets: an ATX heading cannot append
 text after the requested name. Use `section_contains` when a numbered,
@@ -466,8 +503,9 @@ This is not a hard cap. A step that has started always finishes, so a night
 can end above the budget by the cost of one step: the first replay of the
 validation and training tasks, one edit step (a training replay, a reflect
 call and a validation replay of the candidate; with `dream_rollouts` set to K,
-K attempts per training task), the final validation replay, or the held-out
-test replay. Each fan-out group that starts runs its own first replay.
+K attempts per training task; with `gate_train_check`, a replay of the
+training tasks for a candidate that passes validation), the final validation
+replay, or the held-out test replay. Each fan-out group that starts runs its own first replay.
 
 Spend comes from the backend's token counter, the one the report header prints
 as `tokens used`. The header shows the backend's running total and the note
@@ -504,8 +542,8 @@ This runs one additional consolidation per group (including a catch-all group wh
 hinted and unhinted evidence are mixed), so it multiplies backend calls and token
 use; configured dream rollouts and synthetic variants multiply the per-group work
 too. Each group inherits the configured edit budget, gate mode/metric,
-`gate_no_regression`, `dream_rollouts`, `dream_factor`, `recall_k`, and
-`evolve_skill`. Recalled archive tasks are restricted to that same skill hint;
+`gate_no_regression`, `gate_train_check`, `gate_train_margin`,
+`dream_rollouts`, `dream_factor`, `recall_k`, and `evolve_skill`. Recalled archive tasks are restricted to that same skill hint;
 shared memory is read-only in fan-out runs. Setting `evolve_skill` to `false`
 therefore disables per-skill proposals as well as the managed skill proposal.
 

docs/sleep/multi-skill-staging.md

@@ -39,8 +39,8 @@ document. The managed catch-all remains on `proposed_SKILL.md` and is not
 duplicated as a per-skill row.
 
 Each group inherits `recall_k`, `dream_rollouts`, `dream_factor`, `edit_budget`,
-`gate_mode`, `gate_metric`, `gate_mixed_weight`, `gate_no_regression`, and
-`evolve_skill`. Recalled archive tasks are restricted to the same skill hint,
+`gate_mode`, `gate_metric`, `gate_mixed_weight`, `gate_no_regression`,
+`gate_train_check`, `gate_train_margin`, and `evolve_skill`. Recalled archive tasks are restricted to the same skill hint,
 and shared memory is read-only during group runs. Consequently,
 `evolve_skill=false` disables managed and fan-out skill proposals.
 

plugins/README.md

@@ -249,10 +249,11 @@ python -m skillopt_sleep run --project "$(pwd)" \
 ### Advanced config
 
 The JSON/YAML config under `~/.skillopt-sleep/` supports additional engine keys,
-including `gate_mode`, `gate_metric`, `gate_no_regression`, `dream_rollouts`,
-`dream_factor`, `recall_k`, `evolve_memory`, and `evolve_skill`. These are config
-keys, not aliases for the unsupported CLI flags listed above. Shipping defaults
-are conservative: `gate_mode="on"`, `gate_no_regression=false`,
+including `gate_mode`, `gate_metric`, `gate_no_regression`, `gate_train_check`,
+`gate_train_margin`, `dream_rollouts`, `dream_factor`, `recall_k`,
+`evolve_memory`, and `evolve_skill`. These are config keys, not aliases for the
+unsupported CLI flags listed above. Shipping defaults are conservative:
+`gate_mode="on"`, `gate_no_regression=false`, `gate_train_check=false`,
 `dream_rollouts=1`, `dream_factor=0`, and `recall_k=0`.
 
 The managed `schedule` command stores only the project, backend, time, and

plugins/claude-code/skills/skillopt-sleep/SKILL.md

@@ -110,6 +110,7 @@ Beyond the CLI flags, advanced behavior is controlled via config:
 - **`gate_mode`** — `on` (default, validation-gated) or `off` (greedy, accept all edits).
 - **`gate_metric`** — `hard`, `soft`, or `mixed` (default). Controls how the held-out gate scores.
 - **`gate_no_regression`** — `false` by default. Set to `true` to reject a candidate when any validation task's configured gate score decreases.
+- **`gate_train_check`** — `false` by default. Set to `true` to also reject a candidate that scores clearly lower than the current skill and memory on tonight's training tasks (mean lower by more than `gate_train_margin`, default `0.1`, on at least two more tasks than improved).
 - **`dream_rollouts`** — >1 enables multi-rollout contrastive reflection per task.
 - **`recall_k`** — >0 recalls K similar past tasks into the dream (long-term memory).
 - **`evolve_memory`** / **`evolve_skill`** — independently toggle CLAUDE.md vs SKILL.md consolidation.

plugins/codex/README.md

@@ -80,7 +80,8 @@ mining, replay, and reflection can be slow and otherwise quiet. Use
 prefer tasks that match the target skill's path, headings, and content. The
 implemented main-CLI flags work the same across the shared integrations, and
 `--preferences "..."` is available for house rules. Advanced keys such as
-`gate_mode`, `gate_no_regression`, `dream_rollouts`, and `recall_k` belong in the Sleep config; the
+`gate_mode`, `gate_no_regression`, `gate_train_check`, `dream_rollouts`, and `recall_k` belong in
+the Sleep config; the
 nightly CLI does not expose `--gate`, `--rollouts-k`, token/time-budget, or
 optimizer/target-split flags. See the
 [shared CLI reference](../README.md#supported-cli-surface).

plugins/codex/skills/skillopt-sleep/SKILL.md

@@ -134,6 +134,7 @@ without `crontab`, `schedule` prints a line for manual installation.
 - **`gate_mode`** — `on` (validation-gated, default) or `off` (greedy)
 - **`gate_metric`** — `hard` | `soft` | `mixed` (default)
 - **`gate_no_regression`** — `false` by default; set it to `true` to reject a candidate when any validation task's gate score decreases
+- **`gate_train_check`** — `false` by default; set it to `true` to also reject a candidate that scores clearly lower on tonight's training tasks (mean lower by more than `gate_train_margin`, default `0.1`, on at least two more tasks than improved)
 - **`dream_rollouts`** — >1 for multi-rollout contrastive reflection
 - **`recall_k`** — >0 recalls similar past tasks from the archive
 

plugins/copilot/copilot-instructions.snippet.md

@@ -47,6 +47,8 @@ my preferences", or "make the agent improve from past usage", use the MCP tools:
 - `gate_mode` — `on` (default) or `off`
 - `gate_no_regression` — `false` by default; set it to `true` to reject a
   candidate when any validation task's configured gate score decreases
+- `gate_train_check` — `false` by default; set it to `true` to also reject a
+  candidate that scores clearly lower on tonight's training tasks
 - `dream_rollouts` — >1 for more signal
 - `evolve_memory` / `evolve_skill` — toggle which docs consolidate
 

plugins/cursor/skills/skillopt-sleep/SKILL.md

@@ -173,6 +173,9 @@ commits, or generated summaries.
 `gate_no_regression` is a config-only safeguard in
 `~/.skillopt-sleep/config.json`. It defaults to `false`; set it to `true` to
 reject a candidate when any validation task's configured gate score decreases.
+`gate_train_check` is config-only as well. It defaults to `false`; set it to
+`true` to also reject a candidate that scores clearly lower on tonight's
+training tasks.
 
 ## Scheduling
 

plugins/dsh/README.md

@@ -107,7 +107,7 @@ ask the agent: "Use skillopt_status to check the sleep cycle state."
 | `timeoutMs` | `600000` | per-call engine timeout in milliseconds |
 
 Advanced engine keys (`gate_mode`, `gate_metric`, `gate_no_regression`,
-`dream_rollouts`, `recall_k`, `evolve_memory`/`evolve_skill`) go in
+`gate_train_check`, `dream_rollouts`, `recall_k`, `evolve_memory`/`evolve_skill`) go in
 `~/.skillopt-sleep/config.json` — the same file shared by all integrations.
 
 ## Data boundary

plugins/dsh/README.zh.md

@@ -25,6 +25,7 @@
   "gate_mode": "on",
   "gate_metric": "mixed",
   "gate_no_regression": false,
+  "gate_train_check": false,
   "dream_rollouts": 1,
   "recall_k": 0,
   "evolve_memory": true,

plugins/dsh/skills/skillopt-sleep/SKILL.md

@@ -91,7 +91,7 @@ skillopt_schedule project=<dir> hour=3 minute=17 backend=<codex>
 
 Advanced engine keys go in `~/.skillopt-sleep/config.json`:
 `gate_mode` (on/off), `gate_metric` (hard/soft/mixed), `gate_no_regression`,
-`dream_rollouts`, `recall_k`, `evolve_memory` / `evolve_skill`.
+`gate_train_check`, `dream_rollouts`, `recall_k`, `evolve_memory` / `evolve_skill`.
 
 ## Hard rules
 

skillopt_sleep/config.py

@@ -68,6 +68,8 @@
     "gate_metric": "mixed",       # hard | soft | mixed (mixed best for tiny holdouts)
     "gate_mixed_weight": 0.5,
     "gate_no_regression": False,    # reject any candidate that lowers a val-task score
+    "gate_train_check": False,    # reject a candidate that is worse on tonight's training tasks
+    "gate_train_margin": 0.1,     # drop in mean training-task score the check tolerates (0..1)
     "replay_mode": "mock",        # report label; fresh-worktree replay is not implemented
     # ── dream + recall (opt-in; defaults reproduce the prior single-shot loop) ─
     "dream_rollouts": 1,          # >1 => multi-rollout contrastive reflection per task