Skip to content

fix(risk): narrow high to breaking, new-write, and credential changes - #794

Merged
kixelated merged 6 commits into
mainfrom
quest/m0/benchmark/risk-tier
Nov 22, 2026
Merged

kixelated merged 6 commits into
mainfrom
quest/m0/benchmark/risk-tier

Conversation

@kixelated

@kixelated kixelated commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m0/benchmark/risk-tier.md: the maintainer read the week's tiers (2026-11-21, /quest-run) and found high too broad. This PR narrows it in RISK.md and deletes the quest, which unblocks tier guardrails.

Summary

  • RISK.md: high is now a specific list:

    • breaking changes to an export, a record, or parsed output: removing, renaming, or changing the meaning of a field; a reader that refuses what the previous release wrote; or a member a consumer must now supply
    • changes to when an agent may push, merge, post, or spend without asking
    • changes to the reviewer's sandbox or to agent credentials
    • a new kind of data leaving the host or posted publicly
    • the held paths

    Additive exports and fields that consumers keep parsing, bug fixes, skill and audit instructions that leave remote-write authority alone, and routine dependency or pin bumps are medium. The prose also drops the sentence the classifier cited in nearly every high reason ("what agents did under the bad one stays done"). It now says a bug that agents act on while a release is live is a risk every behavior change carries, and that what a revert cannot undo is only what a change newly makes quest do beyond the repository.

  • Classifier instructions are unchanged. Every misjudgment traced to this repository's own RISK.md prose, so a fix there touches no other repository's tiers and needs no quest eval run.

  • quest/m0/benchmark/guardrails.md records the verdict and the re-tier, and no longer requires this quest.

  • vitest.config.ts points every test at a config home that doesn't exist. On a host whose ~/.config/quest/records.json pairs a token with this repository, 9 tests in test/review.test.ts and test/start.test.ts called Kixel Quest's /installation-token and failed, on the base too. CI passed only because its runners have no such file.

Public API and wire: none. RISK.md is read by quest review only, and the test change affects tests only.

Validation

Evidence. quest usage --since 2026-11-14 lists 67 PRs: 21 low, 4 medium, 31 high, and 11 untiered (one of those is this PR). Nearly every model-set high reason ended with some form of "a revert cannot undo actions agents already took".

Re-tier method. .tmp/retier.ts (scratch, not committed) ran the real classify() from src/cli/review.ts on each PR's recorded head, or on its last head when it had no record. It ran against two synthetic bases: origin/main with the current RISK.md (control), and the same tree with this PR's RISK.md. The control reproduced the recorded tiers on all but one PR (#758: medium, then high on rerun). The classifier is Claude Code claude-haiku-5-5 at low effort, as quest review uses.

low medium high
56 tiered PRs, recorded 21 4 31
same, control rerun 21 3 32
same, this RISK.md 21 13 22
10 untiered PRs, control 3 1 6
same, this RISK.md 2 5 3

PRs whose tier moved (recorded, control, then this file):

PR recorded control new title
#793 - high medium chore(deps): Bump the nix group with 4 updates
#791 - low medium chore(deps-dev): Bump prettier (routine bump is medium by design)
#786 high high medium mistake audit, review records quest usage cannot read
#781 high high medium docs(audit): fix what let each finding in
#775 high high medium plant the six audit stubs (with a quest-pinned bump)
#770 high high medium feat(gate): CPU and end time in the gate record
#768 high high medium Merge picks offer "Explain it first"
#758 medium high medium feat(audit): compare forks with their upstreams
#755 high high medium feat(audit): ship the code, cost, docs, and tests audits
#748 high high medium fix(records): resend a session once its PR opens
#739 high high medium feat(github): pullRequestStatus takes cached requirements
#732 - high medium chore(deps): Bump the nix group with 3 updates
#731 high high medium chore(deps): Bump the bun group
#729 - high medium chore(deps): Bump quest-pinned

What stays high, and why. These 17 PRs stay high on the model's reading:

  • Breaking an export or a reader:
    • #737 removes chunks
    • #740 adds a required quota
    • #753 removes exports
    • #754 removes etag
    • #769, #771, and #772 add required Forge members
    • #777 adds a required readToken
    • #747: parseRecord now requires gated/shell, so it refuses records the previous release wrote
    • #762: the status shape requires duplicates on read
  • Changing an existing record field's meaning: #746
  • New spending: #766 adds reviewer fallthrough; #728 adds mandatory evals
  • New remote writes or uploads:
    • #784 writes to contributors' fork PRs
    • #767 uploads without opt-in
    • #763 posts log output publicly
  • Merge authority: #738

Eight more stay high on held paths: #790, #788, #776, #756, #751, #744, #742, and #730.

The window's one later-fixed tiered PR, #751, stays high, and no PR moved to low. The change moves 10 of 56 tiered PRs, not most, so under the quest's Plan the week's reading need not start over.

An earlier draft also made high cover "what quest writes, uploads, or posts beyond the checkout". The re-tier showed that clause pulling every record-content change back to high (#770, #748, #766, #784), so this version limits it to new kinds of data or destinations.

Checks.

  • just check: passes.
  • just test: 1570 passed, 5 skipped.
  • quest check --base origin/main --branch quest/m0/benchmark/risk-tier: 27 documents ok.
  • The 9 host-dependent test failures reproduced on the base commit and with QUEST_API_KEYS unset. They pass with an empty XDG_CONFIG_HOME, and pass with this PR's config.

Choices made

The maintainer said the line stays the agent's judgment; these choices are listed here for review:

  • #747 and #762 were among the maintainer's over-tiered examples, but they stay high. Their readers require the new fields, so a consumer reading what the previous release wrote fails. That is a real break, not an additive field. If those should be medium, the fix is to make such readers accept a missing field, not to lower the tier.
  • #770, #731, #775, #781, and #739 (the other examples) all move to medium.
  • A ! for removing a CLI flag alone stays medium (#750). Callers of a flag are people and skills, not code parsing a contract.
  • The local branch quest/m0/benchmark/risk-tier held #716's already-merged commits. I built this PR from the claim commit instead and left that branch untouched; it is safe to delete.

Caused by

  • moq-dev/quest#777 - its forge reads take the token the developer's records.json pairs with the repository, which tests did not isolate, so they call Kixel Quest on a paired host (found by check)

Follow-ups

None.

Models

Size: XS
Implementer: Claude Code, claude-opus-5-5, unknown effort

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

Review

Rewritten by quest review after each round.

  • Head: f3072a7

  • Base: origin/main at 92a9892

  • Risk: high, it changes RISK.md, which is always high: it sets the tier of every later change

  • Classified: high by the always-high paths

  • Reviewer: Codex, gpt-6.1-sol, medium effort

  • Session: 01a20090-63e8-7ad1-845b-35112d215eac, round 1

  • Score: 9/10, No actionable defects were found; the risk-policy changes are coherent and the test configuration isolates personal tokens, though builds and tests were not run as instructed.

  • Rounds: 5; findings 2 (medium 1, low 1): fixed 2, refuted 0, open 0

  • Findings: none

  • Reviewer: Codex, gpt-6.1-sol, low effort, shadow

  • Session: 01a20090-63e8-7a71-86c8-95a4901d037c, round 1

  • Score: 9/10, No actionable defects were found: the risk-policy changes are internally consistent and the test configuration isolates personal records, though builds and tests were not run as instructed.

  • Rounds: 3; findings 2 (medium 1, low 1): fixed 2, refuted 0, open 0

  • Findings: none

kixelated and others added 5 commits November 21, 2026 15:23
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Read a week of risk tiers before guardrails act on them fix(risk): narrow high to breaking, new-write, and credential changes Nov 22, 2026
@kixelated
kixelated marked this pull request as ready for review November 22, 2026 00:03
@coderabbitai

coderabbitai Bot commented Nov 22, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 7 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 56bc3ea3-cc74-4817-8737-12887ec60a4f
Commits

Reviewing files that changed from the base of the PR and between 058a6d4 and 05b0933.

Files selected for processing: 4

  • RISK.md
  • quest/m0/benchmark/guardrails.md
  • quest/m0/benchmark/risk-tier.md
  • vitest.config.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues — one wording nit inline.

Approach: sound. Tuning the tier prose in RISK.md fixes the cause, since the PR traces the misjudgments to that prose, and it leaves the classifier alone.

Reviewed changes — all four files in the PR.

  • RISK.md: high is now a specific list of breaking changes, changes to agent write authority, sandbox or credential changes, and new kinds of data leaving the host. Medium covers additive changes, bug fixes, and routine bumps. The prose drops the "what agents did under the bad one stays done" sentence.
  • quest/m0/benchmark/guardrails.md: records the 2026-11-21 verdict and the re-tier, and drops the ## Required link to the deleted quest.
  • quest/m0/benchmark/risk-tier.md: the quest is deleted. I found no other references to it in the repo.
  • vitest.config.ts: sets XDG_CONFIG_HOME to a nonexistent path so tests never read a developer's records.json. src/cli/tokens.ts:14 resolves the file from that variable, so the isolation holds. Tests that need a config home set their own.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using claude-sonnet-5-5 | 𝕏

Comment thread RISK.md Resolved
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated enabled auto-merge November 22, 2026 00:14

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes — the delta since the prior pullfrog review.

  • RISK.md wording fix: the medium-tier sentence now reads "instructions that leave remote-write authority alone", applying the earlier suggestion and resolving that thread.

Pullfrog  | View workflow run | Using claude-sonnet-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 22, 2026
Merged via the queue into main with commit a19d184 Nov 22, 2026
15 checks passed
@kixelated
kixelated deleted the quest/m0/benchmark/risk-tier branch November 22, 2026 00:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant