Repository navigation
Add mapbox commands search and commands list - #103
Conversation
mattpodwysocki
left a comment
There was a problem hiding this comment.
Nice work @zmofei. The command itself is in great shape: the index comes from the real clap tree, list covers the hand-written commands too, and typos, other languages and no-match queries all behave well. Build, tests, clippy and fmt are clean.
My two bigger notes are on the CI and eval tooling rather than the command (inline). Smaller things:
- The JSON objects are just
{command, summary}, so an agent can't tell a real match from a weak guess ("create a token" givesfeedback createfirst). Ascorefield would help, and it's easier to add now than to change the shape later. commands search ""and stopword-only queries exit 0 with[]. ANonEmptyStringValueParseror ausageerror would catch a script passing an empty variable.- The doc comment around
tests/output_contract.rs:1305says JSON mode writes nothing to stderr, but the no-match hint does. Fine either way, the comment just needs a tweak. - The new help text in
src/mcp.rsandsrc/config.rslists the client names and config keys by hand. Building them fromCLIENTSandKEYSwould keep them from drifting.
| timeout-minutes: 5 | ||
| permissions: | ||
| actions: read | ||
| pull-requests: write |
There was a problem hiding this comment.
This one makes me a bit nervous. It runs with pull-requests: write, and the body it posts comes from an artifact the PR's own CI run uploads. A fork can edit ci.yml in its PR and upload whatever state and body.md it wants, so the bot can be made to post arbitrary markdown (links, images) under the repo's identity. Finding the PR by head SHA, the size cap and the @ defusing all help, but they don't close that.
Since it also can't be exercised until after merge, could we drop it from this PR? The job summary already covers the "what changed" need. If we want the comment later, I'd rather the trusted side build it from the numbers alone and never pass through markdown from the artifact.
| /// New real queries go in a new file rather than into these: a set the | ||
| /// search has been tuned against stops saying how it does on the next | ||
| /// query nobody has seen. | ||
| const EVAL_SETS: &[(&str, &str, usize, usize)] = &[ |
There was a problem hiding this comment.
Every floor here is exactly today's score, so any change that moves one query by one place fails cargo test in the main build job. That includes a spec regen that tweaks a summary, or someone tidying an about string for an unrelated reason.
I'd like the hard gate to be the targeted tests (a_task_finds_its_command_first and friends), and either give these sets a few queries of slack or make them report-only. Related: the eval plumbing (Python report, shell test, CI job, comment workflow) is close to half this PR. Would you be open to landing the command first and the eval tooling in a follow-up?
d914124 to
c21b32d
Compare
|
Thanks! All addressed in e02dfcd, the only commit since your review.
This branch also carried #106's webpki fix, so I set its base to #106 to keep that out of this diff. |
mattpodwysocki
left a comment
There was a problem hiding this comment.
Thanks @zmofei, this all looks good. I checked each point against the binary: the usage error, the score field and the slack all behave as described, and the workflow is gone. Two small things, neither blocking:
- The no-words check runs per argument, so
mapbox commands search upload a font "?"(orsearch font "$EMPTY") is a usage error even though the query has words in it. Checking the joined query insearch()would only reject queries that are empty overall. Fine to leave if you prefer the stricter behavior. - The docs say the score "reads the same for any query", but it's really how much of the query's wording a command uses, not how likely it is to be the right command. "make a map image" puts the correct
static get-imageat 0.27, while "create a token" putsfeedback createat 0.35. Maybe something like "how much of the query the command's own text covers" so nobody treats it as a confidence threshold?
|
Thanks! Both done in 1dad149. The no-words check now runs on the joined query, so |
Finds a command by what it does: local keyword search over the same command
surface --schema describes, with no request, token or model. A table at a
terminal, [{command, summary}] under json. Top-level help, the agent hint and
an unknown-command error point to it.
Three query sets under tests/fixtures/commands_search are held to score
floors by eval_set_* tests, and a CI job writes each PR's scores and the
queries it fixed and broke against its base to the job summary.
- Text-mode no match writes nothing to stdout. - A real word is no longer typo-matched into another, so "patch a style" stops ranking batch commands first. - The CI summary says why there is nothing to compare: no base, a base from before the eval, or a failed run. The report script never raises, and has its own shell test. - Help rows fit 80 columns again. - Tests for --limit, the human help line, the commands group being left out, and a synonym test that failed for the wrong reason.
Scores read "22 (37%)", with the change against the base when there is one, and a legend under the table says what each column means.
A set ranking 3 or more fewer queries first, or losing 5 or more points, gets the PR one comment, updated on later pushes and marked recovered once the drop is gone. CI's search-eval job only writes the comment as an artifact, since a fork's PR has a read-only token; a workflow_run workflow posts it without checking out or running the PR's code.
Drop search-eval-comment.yml: a fork's CI run could make it post arbitrary markdown. Move the search-eval CI job and its report scripts to a follow-up. The eval sets now fail only when a set drops more than 3 queries below its recorded score, so a reworded summary no longer fails the build. Search results under json carry a score from 0 to 1, a query with no words is a usage error, and the mcp install and config help text is built from SERVERS, CLIENTS and KEYS.
An empty argument beside real words is no longer a usage error; only a query with no words overall is. The score is documented as how much of the query a command's text covers, not as a confidence.
1dad149 to
a29edf3
Compare
mattpodwysocki
left a comment
There was a problem hiding this comment.
Re-approving after the rebase. Checking the joined query and the new score wording both read well, thanks @zmofei.
Adds
mapbox commands search "<task>"andmapbox commands list, so a person or an agent can find a command by what it does instead of chaining--helpcalls. The group iscommandsbecausemapbox searchis the Search Box API.--schemadescribes: no request, no token, no model. It matches the name, summary, description, help-group line and argument help, with prefix, stem and typo matching (a typo is never matched into another real word) and a small one-way synonym table. Results far below the best one are dropped.[{command, summary, score}]underjson.scoreruns from 0 to 1: how much of the query the command's own text covers, not a confidence ("create a token" givesfeedback create0.35). No match prints nothing ([]underjson) with a hint on stderr, and exits 0. A query with no words, such as an empty variable, is a usage error (exit 2).--help, the agent hint, and an unknown command with no near spelling all point to it.tilesets-cli,doctor,config,uninstall,history,mcp installand thefeedbackgroup line now uses the words people search with. Themcp installandconfigtext is built fromSERVERS,CLIENTSandKEYS. Each row still fits one line at 80 columns.Search quality is held by three query sets in
tests/fixtures/commands_search/(260 queries; sources and how much each was tuned against are in its README). Theeval_set_*tests fail when a set drops more than 3 queries below its recorded score, so a reworded summary doesn't fail the build; the targeted tests (a_task_finds_its_command_firstand others) are the hard gate. Current first / top 3: hand-written 84/97 of 100, GitHub-issue phrasing 50/67 of 100, docs phrasing 22/30 of 60. Keyword search works well when the query uses the CLI's own words, as agents do; loose human phrasing tops out around 45% first.The CI job that reports these scores against the base is in a follow-up PR, #108.
Testing: fmt, clippy and the full suite pass locally on macOS.
No downstream or operational impact: new commands only, and no existing command, flag or output changes. CHANGELOG has an entry.