Skip to content

Cut the context a quest-start session re-sends - #808

Merged
kixelated merged 2 commits into
mainfrom
quest/m0/start-spend
Nov 22, 2026
Merged

kixelated merged 2 commits into
mainfrom
quest/m0/start-spend

Conversation

@kixelated

@kixelated kixelated commented Nov 22, 2026 •

Copy link
Copy Markdown
Collaborator

Works on quest/m0/start-spend.md: measures where quest-start sessions re-send context, adds one line to the start skill for the largest lever in scope, and rewrites the quest to its week-later re-measure.

Summary

The quest's Plan guessed that a few large tool results, mostly shell file reads, dominate re-sent cost. Ranking every tool result in the baseline's 324 start sessions (2026-11-14 to 11-21) by tokens times the requests left before the session's next compaction did not bear that out:

  • No outliers: the 20 largest results are 1.6% of the $441 of re-sent tool results, and the largest 1,000 are 29%. (The baseline's $726 ignored compaction.)
  • Reads are already mostly bounded: two thirds of read cost is ranged reads of 1k to 3k tokens. Results over 6k tokens cost $21, re-reads of an already-read range $20, and quest's own gate, review, and status output is about 1%.
  • The multiplier is request count. Of 42,808 requests, 92% make exactly one tool call, and 51% (52% of cost, $1.3k) make a single read-only call, mostly in runs of three or more. About 24% of consecutive single-read pairs read files that neither the first call nor its result named, so they were likely independent.
  • The rest of the bill is outside this quest: cache writes 26%, output 21%, and re-sent assistant turns 26%, mostly retained thinking (model and effort, which Routing evidence owns).

Changes:

  • assets/skills/start.md: one line saying every request re-sends the whole context, so the reads a step already knows it needs go in one response.
  • quest/m0/start-spend.md: rewritten to what remains, the re-measure, scheduled for 2026-11-30. It records the baseline with the new batching metrics (6.7% of requests make more than one tool call, 51% make a single read-only call) and the method to repeat. Resized from M to S.

Public API and wire: none. The start skill's text changes.

Validation

  • Measurement: throwaway Python over the baseline's token-max sessions list and its transcripts, reusing moq-dev/quest#803's session-to-skill attribution and prices. It is kept in this worktree's .tmp/ and not committed, since it reads this host's transcripts. As a check, the cost decomposition sums to the $2.5k start total the baseline reported.

  • A/B of the line on its real surface, headless claude -p at Opus 5.5 medium effort, running the same read-only investigation ("trace quest start from argument parsing to the draft PR, with file:line") under the old and new start skill text:

    Arm Runs Mean requests Tool calls per request Responses with 2+ calls Mean cost
    A (old start.md) 5 6.6 1.12 27% $0.30
    B (new line) 6 5.5 1.27 45% $0.31

    The direction matches the rule, but the sample is small. Cost per run is flat because sessions this short don't build up context; the saving can only show in long sessions, which is what the scheduled re-measure checks.

  • quest skill start from this checkout prints the new line. quest check (26 documents ok), just check, and just test (1,564 passed, 5 skipped) pass.

Decisions

The user was away, so these are my calls, for review:

  • I worded the rule around batching reads rather than the Plan's "read the range a task needs, never a whole large file twice", because the data shows reads are already bounded and re-reads are under 1% of start cost.
  • No PreToolUse check denying unbounded cat or sed: whole-file and over-6k reads are too small a share to justify a hook that also blocks legitimate reads of small files.
  • No source cut to quest's own output: gate, review, and status output is about 1% of re-sent tool results, and the skill and context loads are what moq-dev/quest#807 already trimmed.
  • I kept the line to start.md as the quest scopes it. The rewritten quest proposes the same line for iterate.md only if the re-measure shows it moves the batching share.
  • Observed, no follow-up: the SessionStart and SubagentStart quest hook context reached 1 of 324 start sessions in the window. Main T3 sessions fire no SessionStart hook (this one included), and sub-agents then ran the older quest worktree hook. It costs nothing, since sessions load the same rules through quest context.

Models

Size: M
Implementer: Claude Code, claude-opus-5-5, unknown effort (arm B)

Follow-ups

None. The remaining work is this quest's re-measure, which the rewritten quest holds.

🤖 Generated with Claude Code

(Written by Opus 5.5)

Review

Rewritten by quest review after each round.

  • Head: dfdadb8

  • Base: origin/main at dbe6aab

  • Risk: medium, Adds one instruction to `assets/skills/start.md`, which every `/quest-start` session loads, telling agents to batch reads they already know they need; if it misleads agents, review quality or the PR's validation could suffer for quest users, and a revert of that one line undoes it fully. It leaves push, merge, and post authority untouched, so it is not high; the plan file change is low.

  • Classified: medium by Claude Code, claude-haiku-5-5, low effort, session e574e667-c4ee-445d-b493-436926ec7f68

  • Reviewer: Codex, gpt-6.1-sol, high effort

  • Session: 01a20218-1603-77c1-9033-377b7af9d6e2, round 1

  • Score: 9/10, Static review identified no actionable regressions or stale dependents; builds and tests were not run as instructed.

  • Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

  • Reviewer: Codex, gpt-6.1-sol, xhigh effort, shadow

  • Session: 01a20218-1606-7742-bf73-4ce993476c6b, round 1

  • Score: 9/10, The guidance and revised quest are internally consistent with no actionable regressions found in static review, though builds and tests were not run as instructed.

  • Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

kixelated and others added 2 commits November 21, 2026 22:59
Quest-Claim: 998851c99fc4bb3d
Measured quest-start spend over 2026-11-14 to 11-21: 51% of requests
(52% of cost) make a single read-only call, and only 6.7% make more than
one tool call, while each request re-sends the whole context. Unbounded
reads, re-reads, and quest's own command output are each a few percent.
Add one line to the start skill asking for known reads in one response,
and rewrite the quest to the week-later re-measure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 22, 2026 07:12
@kixelated
kixelated enabled auto-merge November 22, 2026 07:13
@coderabbitai

coderabbitai Bot commented Nov 22, 2026

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The start guide now instructs agents to combine known file, range, or search requests into one response. The spend plan updates its measurement approach, verification measures, recorded choices, and re-measurement schedule for 2026-11-30. The plan title changes from medium to small priority.

Merge Risk: Minimal · up to dfdad

This change adds one batching instruction to the start guide and updates the spend-measurement plan. It does not touch runtime code, so there is no identified merge-blocking risk.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title "Cut the context a quest-start session re-sends" matches the main change: the PR adds a batching instruction to the start skill to reduce the context re-sent per quest-start session, and rew…
Description check Passed The description explains the measurement of where quest-start sessions re-send context, the one-line addition to assets/skills/start.md, the rewrite of quest/m0/start-spend.md, validation steps, and d…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title "Cut the context a quest-start session re-sends" matches the main change: the PR adds a batching instruction to the start skill to reduce the context re-sent per quest-start session, and rewrites the quest plan. The title is specific and concise, and it names the primary goal from the developer's perspective.

Full details: Description check

Explanation

The description explains the measurement of where quest-start sessions re-send context, the one-line addition to assets/skills/start.md, the rewrite of quest/m0/start-spend.md, validation steps, and decisions. It is directly related to the changeset.

Finishing Touches

Simplify code

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: c0cfb1ad-8a67-49bc-89fb-8dd6ed67eb6b
Commits

Reviewing files that changed from the base of the PR and between dbe6aab and dfdadb8.

Files selected for processing: 2

  • assets/skills/start.md
  • quest/m0/start-spend.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Approach: sound. The PR adds one batching instruction to the start skill, aimed at the request count its measurements identify as the cost multiplier, and trims the quest to its re-measure.

Reviewed changes

  • assets/skills/start.md: one line telling the agent to request the reads, ranges, or searches it already knows it needs in a single response. It touches no push, merge, or post authority, and a single revert undoes it.
  • quest/m0/start-spend.md: rewritten to the remaining re-measure, with the baseline's new batching metrics, a ## Scheduled entry for 2026-11-30, and the size changed from M to S.

Pullfrog  | View workflow run | Using claude-sonnet-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 22, 2026
Merged via the queue into main with commit 35d964a Nov 22, 2026
25 checks passed
@kixelated
kixelated deleted the quest/m0/start-spend branch November 22, 2026 07:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant