Skip to content

Add mapbox commands search and commands list - #103

Merged
zmofei merged 10 commits into
mainfrom
feat/commands-search
Oct 21, 2026
Merged

zmofei merged 10 commits into
mainfrom
feat/commands-search

Conversation

@zmofei

@zmofei zmofei commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Adds mapbox commands search "<task>" and mapbox commands list, so a person or an agent can find a command by what it does instead of chaining --help calls. The group is commands because mapbox search is the Search Box API.

  • Local keyword search over the same surface --schema describes: no request, no token, no model. It matches the name, summary, description, help-group line and argument help, with prefix, stem and typo matching (a typo is never matched into another real word) and a small one-way synonym table. Results far below the best one are dropped.
  • A table at a terminal, [{command, summary, score}] under json. score runs from 0 to 1: how much of the query the command's own text covers, not a confidence ("create a token" gives feedback create 0.35). No match prints nothing ([] under json) with a hint on stderr, and exits 0. A query with no words, such as an empty variable, is a usage error (exit 2).
  • Top-level --help, the agent hint, and an unknown command with no near spelling all point to it.
  • The query is free text, so telemetry sends only its length (tested).
  • The help text of tilesets-cli, doctor, config, uninstall, history, mcp install and the feedback group line now uses the words people search with. The mcp install and config text is built from SERVERS, CLIENTS and KEYS. Each row still fits one line at 80 columns.

Search quality is held by three query sets in tests/fixtures/commands_search/ (260 queries; sources and how much each was tuned against are in its README). The eval_set_* tests fail when a set drops more than 3 queries below its recorded score, so a reworded summary doesn't fail the build; the targeted tests (a_task_finds_its_command_first and others) are the hard gate. Current first / top 3: hand-written 84/97 of 100, GitHub-issue phrasing 50/67 of 100, docs phrasing 22/30 of 60. Keyword search works well when the query uses the CLI's own words, as agents do; loose human phrasing tops out around 45% first.

The CI job that reports these scores against the base is in a follow-up PR, #108.

Testing: fmt, clippy and the full suite pass locally on macOS.

No downstream or operational impact: new commands only, and no existing command, flag or output changes. CHANGELOG has an entry.

@zmofei zmofei added the locationai-team-skills Opened via the location-ai PR creation skill label Oct 20, 2026
@zmofei zmofei self-assigned this Oct 20, 2026
@zmofei
zmofei marked this pull request as ready for review October 20, 2026 13:39
@zmofei
zmofei requested a review from a team as a code owner October 20, 2026 13:39
@zmofei zmofei mentioned this pull request Oct 20, 2026

@mattpodwysocki mattpodwysocki left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work @zmofei. The command itself is in great shape: the index comes from the real clap tree, list covers the hand-written commands too, and typos, other languages and no-match queries all behave well. Build, tests, clippy and fmt are clean.

My two bigger notes are on the CI and eval tooling rather than the command (inline). Smaller things:

  • The JSON objects are just {command, summary}, so an agent can't tell a real match from a weak guess ("create a token" gives feedback create first). A score field would help, and it's easier to add now than to change the shape later.
  • commands search "" and stopword-only queries exit 0 with []. A NonEmptyStringValueParser or a usage error would catch a script passing an empty variable.
  • The doc comment around tests/output_contract.rs:1305 says JSON mode writes nothing to stderr, but the no-match hint does. Fine either way, the comment just needs a tweak.
  • The new help text in src/mcp.rs and src/config.rs lists the client names and config keys by hand. Building them from CLIENTS and KEYS would keep them from drifting.

timeout-minutes: 5
permissions:
actions: read
pull-requests: write

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one makes me a bit nervous. It runs with pull-requests: write, and the body it posts comes from an artifact the PR's own CI run uploads. A fork can edit ci.yml in its PR and upload whatever state and body.md it wants, so the bot can be made to post arbitrary markdown (links, images) under the repo's identity. Finding the PR by head SHA, the size cap and the @ defusing all help, but they don't close that.

Since it also can't be exercised until after merge, could we drop it from this PR? The job summary already covers the "what changed" need. If we want the comment later, I'd rather the trusted side build it from the numbers alone and never pass through markdown from the artifact.

Comment thread src/commands.rs
/// New real queries go in a new file rather than into these: a set the
/// search has been tuned against stops saying how it does on the next
/// query nobody has seen.
const EVAL_SETS: &[(&str, &str, usize, usize)] = &[

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every floor here is exactly today's score, so any change that moves one query by one place fails cargo test in the main build job. That includes a spec regen that tweaks a summary, or someone tidying an about string for an unrelated reason.

I'd like the hard gate to be the targeted tests (a_task_finds_its_command_first and friends), and either give these sets a few queries of slack or make them report-only. Related: the eval plumbing (Python report, shell test, CI job, comment workflow) is close to half this PR. Would you be open to landing the command first and the eval tooling in a follow-up?

@zmofei
zmofei force-pushed the feat/commands-search branch 2 times, most recently from d914124 to c21b32d Compare October 21, 2026 07:52
@zmofei

zmofei commented Oct 21, 2026 •

Copy link
Copy Markdown
Member Author

Thanks! All addressed in e02dfcd, the only commit since your review.

  • Dropped search-eval-comment.yml. The search-eval job, report script and its test moved to Report commands search eval scores against the base in CI #108 (summary only, no comment).
  • Eval sets now fail only when a set drops more than 3 queries below its recorded score. The targeted tests stay the hard gate.
  • JSON results have a score from 0 to 1, comparable across queries. "create a token" now shows feedback create at 0.35, against 1.0 for a full match.
  • An empty, whitespace-only or punctuation-only query is a usage error (exit 2). A stopwords-only query still returns []: the query comes in as several arguments, so clap can't check it as a whole, and an empty variable is already caught.
  • Fixed the output_contract.rs comment, and the mcp install / config help text is now built from SERVERS, CLIENTS and KEYS.

This branch also carried #106's webpki fix, so I set its base to #106 to keep that out of this diff.

@zmofei
zmofei changed the base branch from main to webpki-advisories October 21, 2026 08:24
mattpodwysocki
mattpodwysocki previously approved these changes Oct 21, 2026

@mattpodwysocki mattpodwysocki left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @zmofei, this all looks good. I checked each point against the binary: the usage error, the score field and the slack all behave as described, and the workflow is gone. Two small things, neither blocking:

  1. The no-words check runs per argument, so mapbox commands search upload a font "?" (or search font "$EMPTY") is a usage error even though the query has words in it. Checking the joined query in search() would only reject queries that are empty overall. Fine to leave if you prefer the stricter behavior.
  2. The docs say the score "reads the same for any query", but it's really how much of the query's wording a command uses, not how likely it is to be the right command. "make a map image" puts the correct static get-image at 0.27, while "create a token" puts feedback create at 0.35. Maybe something like "how much of the query the command's own text covers" so nobody treats it as a confidence threshold?

@zmofei
zmofei changed the base branch from webpki-advisories to main October 21, 2026 16:53
@zmofei
zmofei dismissed mattpodwysocki’s stale review October 21, 2026 16:53

The base branch was changed.

@zmofei

zmofei commented Oct 21, 2026

Copy link
Copy Markdown
Member Author

Thanks! Both done in 1dad149. The no-words check now runs on the joined query, so search upload a font "" "?" works and only a query with no words overall is a usage error. The score is now described as how much of the query the command's text covers, with a note not to use it as a cutoff, in the code comment, docs/commands.md and the changelog. Base is now main since #106 merged.

@zmofei
zmofei added this pull request to stack #109 October 21, 2026 17:01
zmofei added 10 commits October 21, 2026 20:02
Finds a command by what it does: local keyword search over the same command
surface --schema describes, with no request, token or model. A table at a
terminal, [{command, summary}] under json. Top-level help, the agent hint and
an unknown-command error point to it.

Three query sets under tests/fixtures/commands_search are held to score
floors by eval_set_* tests, and a CI job writes each PR's scores and the
queries it fixed and broke against its base to the job summary.
- Text-mode no match writes nothing to stdout.
- A real word is no longer typo-matched into another, so "patch a style"
  stops ranking batch commands first.
- The CI summary says why there is nothing to compare: no base, a base
  from before the eval, or a failed run. The report script never raises,
  and has its own shell test.
- Help rows fit 80 columns again.
- Tests for --limit, the human help line, the commands group being left
  out, and a synonym test that failed for the wrong reason.
Scores read "22 (37%)", with the change against the base when there is
one, and a legend under the table says what each column means.
A set ranking 3 or more fewer queries first, or losing 5 or more points,
gets the PR one comment, updated on later pushes and marked recovered once
the drop is gone. CI's search-eval job only writes the comment as an
artifact, since a fork's PR has a read-only token; a workflow_run workflow
posts it without checking out or running the PR's code.
Drop search-eval-comment.yml: a fork's CI run could make it post arbitrary
markdown. Move the search-eval CI job and its report scripts to a follow-up.

The eval sets now fail only when a set drops more than 3 queries below its
recorded score, so a reworded summary no longer fails the build. Search
results under json carry a score from 0 to 1, a query with no words is a
usage error, and the mcp install and config help text is built from
SERVERS, CLIENTS and KEYS.
An empty argument beside real words is no longer a usage error; only a
query with no words overall is. The score is documented as how much of the
query a command's text covers, not as a confidence.
@zmofei
zmofei force-pushed the feat/commands-search branch from 1dad149 to a29edf3 Compare October 21, 2026 17:03
@zmofei
zmofei requested a review from mattpodwysocki October 21, 2026 17:12

@mattpodwysocki mattpodwysocki left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving after the rebase. Checking the joined query and the new score wording both read well, thanks @zmofei.

@zmofei
zmofei merged commit 94cc573 into main Oct 21, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

locationai-team-skills Opened via the location-ai PR creation skill

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants