From 3d7576e07c3aab0c231c78e07c92697e6ed1dcb6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 21 Nov 2026 22:59:05 -0800 Subject: [PATCH 1/2] chore(quest): claim quest/m0/start-spend.md Quest-Claim: 998851c99fc4bb3d From dfdadb8d08b6c201582c05fa6c45389733ff3148 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 21 Nov 2026 23:10:15 -0800 Subject: [PATCH 2/2] start skill: batch the reads a step already knows it needs Measured quest-start spend over 2026-11-14 to 11-21: 51% of requests (52% of cost) make a single read-only call, and only 6.7% make more than one tool call, while each request re-sends the whole context. Unbounded reads, re-reads, and quest's own command output are each a few percent. Add one line to the start skill asking for known reads in one response, and rewrite the quest to the week-later re-measure. Co-Authored-By: Claude Opus 5.5 --- assets/skills/start.md | 1 + quest/m0/start-spend.md | 33 ++++++++++++++++++--------------- 2 files changed, 19 insertions(+), 15 deletions(-) diff --git a/assets/skills/start.md b/assets/skills/start.md index feeaa779..a62bc935 100644 --- a/assets/skills/start.md +++ b/assets/skills/start.md @@ -9,6 +9,7 @@ Claim it with `quest start `; if it refuses, report what it printed and s If you cannot go on before pushing any work, release the claim with `quest start --release `. Implement the quest until it is complete or blocked, then push to its PR and update the description. +Every request re-sends the whole context, and most of a start session's requests make a single read: when you already know several files, ranges, or searches you need, request them in one response. If the quest is complete and no decision is left for the maintainer beyond its follow-ups', run `/quest-review` on it right away. Otherwise ask the user each open decision with your recommendation, record the answers in the PR body's `## Decisions` section, and run `/quest-review` once the quest is complete and the PR reflects them. diff --git a/quest/m0/start-spend.md b/quest/m0/start-spend.md index 69a5cf52..6a313b74 100644 --- a/quest/m0/start-spend.md +++ b/quest/m0/start-spend.md @@ -1,31 +1,34 @@ -# [M] Cut the context a quest-start session re-sends +# [S] Cut the context a quest-start session re-sends ## Goal A `/quest-start` session costs less per merged PR, because it carries less context through its later requests, with no loss in review quality or in what the PR validates. `quest usage` over a week of sessions shows the cut against the baseline below; no fixed target is set. -Non-goals: the writer's model and effort (owned by [Routing evidence](/quest/m0/routing-evidence/README.md)), and trimming the always-loaded rules and skills, done in [moq-dev/quest#807](https://github.com/moq-dev/quest/pull/807), which the baseline shows is not where start sessions spend. +Non-goals: the writer's model and effort (owned by [Routing evidence](/quest/m0/routing-evidence/README.md)), and trimming the always-loaded rules and skills ([moq-dev/quest#807](https://github.com/moq-dev/quest/pull/807)), which the baseline shows is not where start sessions spend. ## Plan -Decided 2026-11-22 (maintainer, `/quest-run`, on [moq-dev/quest#803](https://github.com/moq-dev/quest/pull/803)'s closing re-measure of the token waste epic): keep a follow-up on `quest-start` spend, scoped as this repository's rules size and place it (rejected: close the epic with no follow-up). +Decided 2026-11-22 (maintainer, `/quest-run`, on [moq-dev/quest#803](https://github.com/moq-dev/quest/pull/803)'s closing re-measure of the token waste epic): keep a follow-up on `quest-start` spend (rejected: close the epic with no follow-up). The auto-compact window question went to kixel-ai/dotfiles' quest tree, holding at 400k until this quest's cut is measured. -Kept 2026-11-22 (maintainer, `/quest-run` pick round on [moq-dev/quest#803](https://github.com/moq-dev/quest/pull/803)): ✅ keep this quest (rejected: drop it). The auto-compact window question went to kixel-ai/dotfiles' quest tree: ✅ keep it, holding at 400k until this quest's cut is measured (rejected: drop it). +What remains is the re-measure: [moq-dev/quest#808](https://github.com/moq-dev/quest/pull/808) added one line to `assets/skills/start.md` asking for the reads a step already knows it needs in one response. Re-measure a week of start sessions after it lands, as below, and then keep, reword, or drop that line; if it moves the batching share, propose the same line for `assets/skills/iterate.md` (17.8% of the host's Claude spend in the baseline window). -Baseline, 2026-11-14 to 11-21, this host, from `token-max sessions` and the sessions' transcripts (a session counts as `quest-start` when it, or the session that spawned it, ran `/quest-start` first; dollars are API list prices): +Baseline, 2026-11-14 to 11-21, this host, from `token-max sessions` and the sessions' transcripts (a session counts as `quest-start` when it, or the session that spawned it, ran `/quest-start` first; a request is one API response, deduplicated by message id; dollars are API list prices, 1h cache writes at 2x input): -- 324 sessions (196 main, 128 sub-agents) cost $2.5k of the host's $5.3k Claude spend, 47%. The median session is $4.39, the 90th percentile $18.72, the most expensive $76. -- Per merged PR: $2,504 over the 1,352 PRs merged in the window across the 7 repositories the epic measured, $1.85 each. `quest usage --since 2026-11-14` on moq-dev/quest alone, whose `implementing` role is the closest it reports, shows $89.7 over its 68 merged PRs, $1.32 each. -- Where it sits: 66% of that cost is in sessions whose context peaked above 300k tokens, and 74% is spent after a session's 50th request. Only 4.5% falls in the first 10 requests, where the rules and skills load. Opus at high effort is 62% of it. -- What it re-sends: an estimate of tool results kept in context and re-read by every later request is $726, 29% of the start cost. About three quarters of it is files read through the shell (`sed`, `grep`, `cat`, `ls`), 8.7M tokens through `sed` alone; `quest skill` and `quest context` loads are 7%, `quest gate` output 2%. +- 324 sessions (196 main, 128 sub-agents) cost $2.5k of the host's $5.3k Claude spend, 47%; $1.85 per merged PR across the 7 repositories, and $1.32 for `quest usage --since 2026-11-14`'s `implementing` role on moq-dev/quest's 68 merged PRs. +- Cost by part: cache writes 26%, output 21%, re-sent tool results 17%, re-sent assistant turns (mostly retained thinking and tool-call inputs) 26%, the first request's prompt re-sent 9%. Phase: 72% before the first push, 12% between push and `quest review`, 16% from review on. +- Requests: 42,808, of which 6.7% make more than one tool call, and 51% (52% of cost) make a single read-only call (`Read`, `Grep`, `Glob`, or a shell `sed`, `cat`, `grep`, `rg`, `ls`, `git show`/`log`/`diff`, `gh ... view`), mostly in runs of three or more. Of consecutive single-read pairs, 24% read files that neither the first call nor its result named, so were likely independent; their requests' cache reads alone cost $99. +- Re-sent tool results, counted only until the session's next compaction: $441. No outliers: the 20 largest are 1.6% of it, the largest 1,000 are 29%. Shell reads are 45% (`sed` 35%, `cat` 10%), `grep` 15%, `quest skill` and `quest context` loads 9%, `quest gate` output about 1%. Reads are mostly bounded already: two thirds of read cost is ranged reads, results over 6k tokens are $21, re-reads of a range already read are $20, and 3% of single reads page the file the previous request read. -Agent's choices, for the implementer to revisit: +Verify with, next to the figures above: the share of start-session requests making more than one tool call and the share making a single read-only call; cost per merged PR from `quest usage --since `'s `implementing` role on moq-dev/quest and host-wide from `token-max sessions`; and review findings per PR, so a cheaper session that finds less is visible. -- Start from the measurement, not from the prose: rank the 20 largest tool results in start sessions by tokens times the requests left after them, since a few outliers may dominate. Cut those at their source when a `quest` command's own output is the cause, such as a gate's, a review's, or a status's, by printing only what the next step needs. -- For files read through the shell, add a short rule to `assets/skills/start.md`: read the range a task needs (`Read` with an offset and limit, `rg -l` or `-m` before a full `grep`), never a whole large file twice. Rules a program can check belong in a hook, so first measure whether the guidance moves the share; only if it does not, propose a `PreToolUse` check that denies an unbounded `cat` or `sed` of a file over a line limit, as its own follow-up. -- Do not lower Claude's auto-compact window (400k for Opus) in this quest: it is a setting on the maintainer's host, planned in kixel-ai/dotfiles' quest tree, and it trades re-sent cost against context lost mid-quest. Recommendation for that tree: keep 400k until this quest's cut is measured, then decide with its numbers, since 66% of start cost is in sessions above 300k tokens. -- Verify a week after the change with both measures, next to the baseline here, and with review findings per PR so a cheaper session that finds less is visible: `quest usage --since `'s `implementing` role cost per merged PR on moq-dev/quest, which has no skill, parent, or peak-context field and so stands in for the start-session set; and the host-wide figures, repeated from `token-max sessions` and the transcripts as the baseline was (a session's skill from its first quest skill, a sub-agent under its parent's, peak context the largest input plus cache tokens of any request), which gives cost per merged PR and the share above 300k tokens. Run `just check` and `just test`. -- This quest edits `assets/skills/start.md`, which the maintainer asked for by planning it. +Agent's choices, for review: + +- No source cut in `quest`'s own output: gate, review, and status output is about 1% of re-sent tool results, and the skill and context loads are what trim-skills sized. +- No `PreToolUse` check denying unbounded `cat` or `sed`: whole-file and over-6k reads are too small a share to be worth a hook that blocks legitimate reads of small files. + +## Scheduled + +- 2026-11-30 - a week of start sessions after the skill line lands ## Related