diff --git a/AGENTS.md b/AGENTS.md index c99ce88..da285f2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -126,6 +126,24 @@ One table outside that file works the same way: `GROUPS` in an entry outlives its command. A command it misses still shows, under "Other", but CI will not let that ship. +## `commands search` and its query sets + +`mapbox commands search` ranks commands by the words in their names, +summaries, descriptions, help-group lines and argument help, so **editing a +command's `about`, an argument's `help` or a line in `DESCRIPTIONS` changes +search results**, not only `--help`. The `eval_set_*` tests in +`src/commands.rs` run the query sets in `tests/fixtures/commands_search/` and +fail when a set ranks more than a few queries fewer right than its recorded +score. Each file's header says where its queries came from and how much the +search was tuned against them. `cargo test eval_set -- --nocapture` prints +the scores and every miss, and that directory's README explains each set and +how to read its score. + +When a change raises a score, record it in the same commit. Dropping past the +slack is a decision to explain in the PR. Don't tune against the sets by +special-casing a query; fix the kind of miss. New real queries go in a new +file, so there is always one set the search has not been fitted to. + ## What the tests can and cannot tell you `cargo test` answers to fake tokens and a fake server. It runs offline, on a diff --git a/CHANGELOG.md b/CHANGELOG.md index 11c1878..bc64f5d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,6 +50,15 @@ that may never merge. They are not releases and are not listed here. from `tilequery get` and the renderer kept matching the old name. `-o json` is unchanged. +- New `mapbox commands search ""` finds the commands that match a + task, best first, and `mapbox commands list` lists every command with its + summary. Both are local keyword lookups: no request, no token. `json` + gives `[{command, summary, score}]`, where `score` (0 to 1) is how much + of the query the command's own text covers. The query is sent in telemetry as its length + only. Top-level help now points to `commands search` (agents see it in + their hint), and an unknown command with no near spelling suggests it in + the error's `fix`. No existing command or output changes. + - `mapbox usage` reads as a chart: one row per product with no blank lines between, every sparkline as wide as the period, bars scaled from zero so their height is proportional to the value, and the tallest diff --git a/Cargo.lock b/Cargo.lock index dce78d6..a29ad23 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -800,6 +800,7 @@ dependencies = [ "serde_json", "serde_yaml", "sha2", + "strsim", "tar", "terminal-colorsaurus", "terminal_size", diff --git a/Cargo.toml b/Cargo.toml index 51cfd7e..06c0dab 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -65,6 +65,9 @@ clap = { version = "4", features = ["derive", "env", "string", "wrap_help"] } # top-level help, which src/help_layout.rs renders itself and so has to wrap # to the terminal itself. terminal_size = "0.4" +# Typo tolerance for `mapbox commands search`. Already in the graph through +# clap's own "did you mean" suggestions; the same version. +strsim = "0.11" # Renders `mapbox completion ` from the same `Command` tree everything # else here is built from. Kept to the default features on purpose: the # dynamic-completion half is `unstable-dynamic`, and value completion is out diff --git a/README.md b/README.md index 18edc2c..ec12515 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,7 @@ mapbox styles list - [Named profiles](#named-profiles) - [Commands](#commands) - [API commands](#api-commands) + - [Finding a command](#finding-a-command) - [Diagnostics and settings](#diagnostics-and-settings) - [Shell completion](#shell-completion) - [Tileset CLI](#tileset-cli) @@ -308,6 +309,17 @@ account it belongs to, alongside the event. With no token of your own and no way to fetch the CLI's, the event is dropped. `MAPBOX_CLI_NO_TELEMETRY=1` or `mapbox config set telemetry off` turns it off. +### Finding a command + +```sh +mapbox commands search upload a font # the commands that match a task, best first +mapbox commands list # every command with its summary +``` + +Both are local keyword lookups over the CLI's own commands: no model, no +request and no token. Agents get the same matches as JSON. See +[Commands](./docs/commands.md#commands). + ### Diagnostics and settings ```sh diff --git a/docs/commands.md b/docs/commands.md index be5790b..e6958ac 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -134,6 +134,9 @@ nests, and is typed `mapbox styles draft get`. **[History](#history)** — [history.list](#mapbox-history-list) · [history.show](#mapbox-history-show) +**[Commands](#commands)** — [commands.search](#mapbox-commands-search) · +[commands.list](#mapbox-commands-list) + **[Doctor](#doctor)** — [doctor](#mapbox-doctor) **[Usage](#usage)** — [usage](#mapbox-usage) @@ -4736,6 +4739,138 @@ Log no longer available: it expired or was removed to keep diagnostic logs --- +## Commands + +Finding a command by what it does. Both read the same command surface +`mapbox --schema` describes, so they cover every command this build has, +and both are local: no request and no token. `json` gives each command as +`{command, summary}`, plus a `score` from `search`; `mapbox +--schema` has the rest. + +### `mapbox commands search` + +The commands that match a task, best first. It is keyword matching, not a +model: each word of the query is matched against a command's name, summary +and description, its group's description and its arguments' help, with a +command's name counting most. A word can be a prefix (`geo`), a different +ending of the same stem (`geocoding` for `geocoder`) or one typo away +(`uplaod`), unless some command's text uses the word as it is: "match" is a +word, not a typo for "batch". A plural matches its singular. A few common +words stand in for the ones commands use ("remove" for "delete", "rename" +for "update"). +Words like "how", "my", "to" and "mapbox" are ignored. It cannot find a +command whose text uses none of the query's words, so a query names the +action and the thing acted on: "upload a font", not "get my typeface onto +Mapbox". Matches far below the best one are left out. A task no command +does usually still comes back with a few weak guesses, so check the +summaries before running one. + +Under `json`, each match has a `score` from 0 to 1: how much of the query +the command's own text covers. 1 means its name and summary use every word +of the query, a few tenths is a word found elsewhere, and under 0.1 is +barely a match. It is not a confidence, so don't use it as a cutoff: the +right command can score lower than a wrong one on another query. + +The query can be quoted or not. One with no words at all, such as an +empty string, is a usage error (exit 2). No match is not an error: `text` +prints nothing and says so on stderr, `json` prints `[]`, and both exit 0. The +`commands` group itself is left out of the results. Telemetry records the +query's length, never its text. + +#### Parameters + +| Parameter | Effect | +| --- | --- | +| `...` | What you want to do, in a few words. | +| `--limit ` | How many matches to show. Defaults to `5`; `0` shows every match. | + +#### Examples + +```sh +mapbox commands search upload a font + +mapbox commands search "snap a gps trace to roads" + +mapbox commands search tiles --limit 0 +``` + +#### Outputs + + + + +
textjson
+ +``` +COMMAND SUMMARY +mapbox fonts upload Upload a font +mapbox fonts list List fonts for an owner +mapbox fonts delete Delete a font +``` + + + +```json +[ + { + "command": "mapbox fonts upload", + "score": 1.0, + "summary": "Upload a font" + }, + { + "command": "mapbox fonts list", + "score": 0.65, + "summary": "List fonts for an owner" + }, + ... +] +``` + +
+ +### `mapbox commands list` + +Every command with its summary, sorted by name. + +#### Examples + +```sh +mapbox commands list +``` + +#### Outputs + + + + +
textjson
+ +``` +COMMAND SUMMARY +mapbox accounts list-scopes List scopes +mapbox accounts list-tokens List tokens +mapbox accounts retrieve-token Retrieve a token +... +``` + + + +```json +[ + { + "command": "mapbox accounts list-scopes", + "summary": "List scopes" + }, + { + "command": "mapbox accounts list-tokens", + "summary": "List tokens" + }, + ... +] +``` + +
+ ## Doctor ### `mapbox doctor` diff --git a/src/commands.rs b/src/commands.rs new file mode 100644 index 0000000..2b3d4a3 --- /dev/null +++ b/src/commands.rs @@ -0,0 +1,980 @@ +//! `mapbox commands` — finding a command by what it does. +//! +//! `search` ranks every command against a description of a task; `list` is +//! every command with its summary, on one page. Both read the command +//! surface [`crate::schema`] builds, so there is no second description of +//! the CLI to keep in step, and both answer for people and agents alike: a +//! table at a terminal, `[{command, summary}]` under `json`. +//! +//! The search is local keyword matching and nothing more: no model, no +//! request, no index on disk. At a few dozen commands, scoring every one on +//! each call takes no measurable time, and anything smarter would be a +//! dependency to ship for a list a person could read in a minute. The cost +//! is that it only finds words the command's own text uses, or one of a +//! handful of synonyms for them — "upload a worker script" will not find +//! `deploy` — so the summaries are what decide how well it works. +//! +//! The query is a free string, so telemetry records its length and never +//! its text (see [`crate::telemetry_event`]). No token, no request. + +use std::collections::BTreeSet; + +use anyhow::Result; +use clap::{value_parser, Arg, ArgMatches, Command}; +use serde_json::{json, Value}; + +use crate::help_layout; +use crate::output::{self, Mode}; +use crate::schema::{self, CommandEntry}; +use crate::spec::ServiceSpec; + +pub const COMMAND: &str = "commands"; + +const DEFAULT_LIMIT: usize = 5; + +/// How much each part of a command counts toward a match, after +/// `cf cli search`'s: a word in the command's own name says far more than +/// the same word somewhere in one of its arguments' help. +const NAME_WEIGHT: f64 = 8.0; +const SUMMARY_WEIGHT: f64 = 5.0; +const DESCRIPTION_WEIGHT: f64 = 2.0; +/// The group's own one-line description. Many summaries are a few words +/// ("Tilequery", "Forward geocoding"), and the group is often where the +/// plainer word is: `tilesets` "query features at a point". +const GROUP_WEIGHT: f64 = 1.5; +const ARGUMENTS_WEIGHT: f64 = 0.5; +/// How much a term has to score in one command before it counts as +/// answered in full: an exact match anywhere but an argument's help, or a +/// near miss in the name or summary. A word found only in an argument's +/// help, or near-missed in a description or group line, answers it in part, +/// so a passing mention cannot lift a command to full coverage of the query. +const FULL_MATCH: f64 = GROUP_WEIGHT; +/// Taken off for each word of a command's name the query did not ask for, +/// so "delete a style" finds `styles delete` ahead of `styles draft +/// delete`, whose summary and description match it just as well. +const UNASKED_NAME_WORD: f64 = 1.0; + +/// What a `batch` command's score is scaled by when the query did not ask +/// for many. +const UNASKED_BATCH: f64 = 0.5; + +/// What a deprecated command's score is scaled by. Still found, so a script +/// that names one can be told what it was, but below anything it would be +/// better to start using. +const DEPRECATED: f64 = 0.5; + +/// How well a query term matches a word, by kind of match (see +/// [`similarity`]). +const EXACT: f64 = 1.0; +const PREFIX: f64 = 0.7; +const STEM: f64 = 0.5; +const TYPO: f64 = 0.4; + +/// What each word of the query scores when both the command's name and its +/// summary use it. The score a caller sees is the total against this, capped +/// at 1, so it measures how much of the query a command's own text covers, +/// for a query of any length: 1 is a command that says every word in its +/// name and summary, a few tenths is a word found elsewhere, and under 0.1 is +/// barely a match at all. It is not a confidence: "make a map image" puts the +/// right `static get-image` at 0.27, "create a token" the wrong `feedback +/// create` at 0.35. +const STRONG_TERM: f64 = NAME_WEIGHT + SUMMARY_WEIGHT; + +/// A match this far below the best one is noise beside it. +/// +/// Relative only. A fixed minimum also emptied the answer for tasks this CLI +/// cannot do, but it emptied it just as readily for ones it can, worded +/// loosely: "how far can I cycle in twenty minutes" finds `isochrone` with +/// one word. A wrong guess costs the caller a look; no answer when there was +/// one costs them the command. +const MIN_SHARE_OF_BEST: f64 = 0.1; + +/// Words that describe the asking rather than the task. Left in, "how do I +/// list my styles" would rank whatever says "my" most. +/// +/// "mapbox" is one too: every command is Mapbox's, so it only rewards the +/// commands whose text happens to repeat the name. +const STOPWORDS: &[&str] = &[ + "a", "an", "and", "are", "as", "at", "be", "by", "can", "do", "for", "from", "how", "i", "in", + "into", "is", "it", "mapbox", "me", "my", "of", "on", "or", "the", "this", "to", "up", "want", + "what", "with", "you", "your", +]; + +/// Words people use for something the commands' own text says another way, +/// each with the words it stands in for. One way only: "remove" finds +/// `delete`, but "delete" does not go looking for "remove". Keys are in the +/// form [`tokens`] leaves them, plural `s` gone. +const SYNONYMS: &[(&str, &[&str])] = &[ + ("signin", &["login"]), + ("signout", &["logout"]), + ("remove", &["delete"]), + ("discard", &["delete"]), + ("erase", &["delete"]), + ("edit", &["update"]), + ("rename", &["update"]), + ("modify", &["update"]), + ("change", &["update"]), + // Not "many" or "several": they as often describe what comes back + // ("a reverse lookup that returns several results") as what goes in. + ("bulk", &["multiple", "batch"]), + ("picture", &["image"]), + ("png", &["image"]), + ("jpg", &["image"]), + ("thumbnail", &["image"]), + ("screenshot", &["image"]), + ("permission", &["scope"]), + ("saved", &["stored"]), + ("add", &["upload"]), +]; + +/// A synonym's match counts for less than the word itself, so a command +/// that uses the caller's own word still comes first. +const SYNONYM_WEIGHT: f64 = 0.8; + +pub fn command() -> Command { + Command::new(COMMAND) + .about("Find a command by what it does") + .long_about( + "Find a command by what it does. `search` ranks every command against a \ + description of a task; `list` shows every command with its summary. \ + Both are local: no request and no token.", + ) + .subcommand_required(true) + .subcommand( + Command::new("search") + .about("Find the commands that match a task, best first") + .arg( + Arg::new("query") + .required(true) + .num_args(1..) + .help("What you want to do, in a few words: \"upload a font\""), + ) + .arg( + Arg::new("limit") + .long("limit") + .value_parser(value_parser!(usize)) + .default_value(DEFAULT_LIMIT.to_string()) + .help("How many matches to show; 0 shows every match"), + ), + ) + .subcommand(Command::new("list").about("List every command with its summary")) +} + +/// Refuses a query with no words in it as a usage error, so a script +/// passing an empty variable fails rather than looking like a query that +/// found nothing. Checked on the joined query, not per argument: +/// `search font "$EMPTY"` still has a word to search for. +pub fn check_query(matches: &ArgMatches) -> std::result::Result<(), clap::Error> { + let Some(("search", search)) = matches + .subcommand_matches(COMMAND) + .and_then(ArgMatches::subcommand) + else { + return Ok(()); + }; + let has_words = search + .get_many::("query") + .expect("required") + .any(|part| part.chars().any(char::is_alphanumeric)); + if has_words { + Ok(()) + } else { + Err(clap::Error::raw( + clap::error::ErrorKind::InvalidValue, + "the query has no words to search for\n", + )) + } +} + +pub fn search( + app: &Command, + specs: &[ServiceSpec], + matches: &ArgMatches, + mode: Mode, +) -> Result<()> { + let query = matches + .get_many::("query") + .expect("required") + .map(String::as_str) + .collect::>() + .join(" "); + let limit = *matches.get_one::("limit").expect("has a default"); + + let mut found = rank(app, specs, &query); + if limit > 0 { + found.truncate(limit); + } + if found.is_empty() { + output::progress(&format!( + "No command matches \"{query}\". `mapbox commands list` shows every command." + )); + } + let json = found + .iter() + .map(|(score, entry)| { + json!({ + "command": entry.command, + "summary": entry.summary, + // Never 0: everything returned matched something. + "score": ((score * 100.0).round() / 100.0).max(0.01), + }) + }) + .collect(); + let entries: Vec<&CommandEntry> = found.iter().map(|(_, entry)| entry).collect(); + emit(mode, &entries, json) +} + +pub fn list(app: &Command, specs: &[ServiceSpec], mode: Mode) -> Result<()> { + let mut all = schema::build(app, specs, &[]).commands; + all.sort_by(|a, b| a.command.cmp(&b.command)); + let json = all + .iter() + .map(|entry| json!({ "command": entry.command, "summary": entry.summary })) + .collect(); + emit(mode, &all.iter().collect::>(), json) +} + +/// The commands that match `query`, best first, down to +/// [`MIN_SHARE_OF_BEST`], each with its score against [`STRONG_TERM`]. +fn rank(app: &Command, specs: &[ServiceSpec], query: &str) -> Vec<(f64, CommandEntry)> { + let index = index(app, specs); + let order = rank_in(&index, query); + let mut entries: Vec> = + index.into_iter().map(|(_, entry)| Some(entry)).collect(); + order + .into_iter() + .map(|(score, i)| (score, entries[i].take().expect("each position once"))) + .collect() +} + +/// Every command `search` can answer with, and its words. Built once per +/// run, or once for a whole test set. +fn index(app: &Command, specs: &[ServiceSpec]) -> Vec<(Fields, CommandEntry)> { + schema::build(app, specs, &[]) + .commands + .into_iter() + // The caller is already here, and "find a command" would otherwise + // rank this group's own summaries first. + .filter(|entry| entry.service != COMMAND) + .map(|entry| (Fields::of(app, &entry), entry)) + .collect() +} + +/// Positions in `index` of the commands that match `query`, best first, each +/// with its score against [`STRONG_TERM`]. Ties go to the shorter command name, then +/// alphabetically, so the same query always answers the same way. +fn rank_in(index: &[(Fields, CommandEntry)], query: &str) -> Vec<(f64, usize)> { + let vocabulary: BTreeSet<&str> = index + .iter() + .flat_map(|(fields, _)| fields.parts.iter().flat_map(|(_, words)| words)) + .map(String::as_str) + .collect(); + let query = Query::parse(query, &vocabulary); + if query.terms.is_empty() { + return vec![]; + } + let mut scored: Vec<(f64, usize)> = index + .iter() + .enumerate() + .map(|(i, (fields, entry))| (score(&query, fields, entry.deprecated), i)) + .collect(); + scored.sort_by(|(a_score, a), (b_score, b)| { + let (a, b) = (&index[*a].1.command, &index[*b].1.command); + b_score + .total_cmp(a_score) + .then(a.len().cmp(&b.len())) + .then(a.cmp(b)) + }); + let strong = query.terms.len() as f64 * STRONG_TERM; + above_floor(scored) + .into_iter() + .map(|(score, i)| ((score / strong).min(1.0), i)) + .collect() +} + +/// What is left of `scored`, already sorted best first, once the zero scores +/// and the ones too far below the best are dropped. +fn above_floor(scored: Vec<(f64, T)>) -> Vec<(f64, T)> { + let floor = scored + .first() + .map_or(0.0, |(best, _)| best * MIN_SHARE_OF_BEST); + scored + .into_iter() + .take_while(|(score, _)| *score > 0.0 && *score >= floor) + .collect() +} + +struct Query { + terms: Vec, + /// Neighboring words written together, "log out" as "logout", tried + /// only for an exact match, with the two words they were made from. + /// Kept apart from `terms` so that one that matches nothing — almost all + /// of them — does not count as a word of the query going unanswered. + compounds: Vec<(String, [String; 2])>, +} + +struct Term { + word: String, + /// Whether a word a typo away counts. Not for a word some command's text + /// uses as it is: that is a real word, not a misspelling, and matching + /// it one edit off turns "match" into "batch". + typos: bool, +} + +impl Query { + fn parse(query: &str, vocabulary: &BTreeSet<&str>) -> Self { + let all: Vec = tokens(query).collect(); + let mut seen = BTreeSet::new(); + // Before stopwords come out: the "in" of "log in" is one. + let compounds = all + .windows(2) + .map(|pair| (pair.concat(), [pair[0].clone(), pair[1].clone()])) + // "log in log in" asks once. + .filter(|(compound, _)| seen.insert(compound.clone())) + .collect(); + Query { + terms: terms(query) + .into_iter() + .map(|word| Term { + typos: !vocabulary.contains(word.as_str()), + word, + }) + .collect(), + compounds, + } + } +} + +/// A command's text, as the words each weighted part of it holds. +struct Fields { + name: BTreeSet, + parts: Vec<(f64, BTreeSet)>, +} + +impl Fields { + fn of(app: &Command, entry: &CommandEntry) -> Self { + let name = entry + .command + .strip_prefix("mapbox ") + .unwrap_or(&entry.command); + // The help list's line before the command's own `about`: for an API + // group, `about` is the spec's title ("Mapbox Tokens API"), which + // names the API rather than what it does. + let group = help_layout::description(&entry.service) + .map(ToString::to_string) + .or_else(|| { + app.find_subcommand(&entry.service) + .and_then(Command::get_about) + .map(ToString::to_string) + }) + .unwrap_or_default(); + let arguments = entry + .arguments + .iter() + .flat_map(|arg| [Some(arg.name.as_str()), arg.description.as_deref()]) + .flatten() + .collect::>() + .join(" "); + let name = words(name); + Fields { + name: name.clone(), + parts: vec![ + (NAME_WEIGHT, name), + (SUMMARY_WEIGHT, words(&entry.summary)), + ( + DESCRIPTION_WEIGHT, + words(entry.description.as_deref().unwrap_or_default()), + ), + (GROUP_WEIGHT, words(&group)), + (ARGUMENTS_WEIGHT, words(&arguments)), + ], + } + } +} + +/// Each term counts once per part, at its best match there, so a long +/// description does not outrank a short summary by repeating a word. The +/// total is then scaled by how much of the query matched (see +/// [`FULL_MATCH`]): a command that answers the whole task beats one that +/// answers half of it loudly. +fn score(query: &Query, fields: &Fields, deprecated: bool) -> f64 { + let mut total = 0.0; + // A matched compound answers the words it was made from, which on + // their own usually match nothing: "out" of "log out". + let mut answered = BTreeSet::new(); + for (compound, parts) in &query.compounds { + for (weight, words) in &fields.parts { + let best = words + .iter() + .map(|word| exact_or_synonym(compound, word)) + .fold(0.0, f64::max); + if best > 0.0 { + total += weight * best; + answered.extend(parts.iter().cloned()); + } + } + } + let mut matched = 0.0; + for term in &query.terms { + let mut term_total = 0.0; + for (weight, words) in &fields.parts { + let best = words + .iter() + .map(|word| relatedness(term, word)) + .fold(0.0, f64::max); + term_total += weight * best; + } + matched += if answered.contains(&term.word) { + 1.0 + } else { + (term_total / FULL_MATCH).min(1.0) + }; + total += term_total; + } + let mut total = (total - UNASKED_NAME_WORD * unasked_name_words(query, fields) as f64).max(0.0); + // A `batch` command does what a sibling does, for many inputs at once, + // and its text says so at length: "forward or reverse", every option the + // sibling has. Unless the query asks for many, the sibling is the answer. + if fields.name.contains("batch") && !asks_for_many(query) { + total *= UNASKED_BATCH; + } + if deprecated { + total *= DEPRECATED; + } + let coverage = matched / query.terms.len() as f64; + total * coverage * coverage +} + +/// How many words of a command's name no term or compound of the query +/// asked for. +fn unasked_name_words(query: &Query, fields: &Fields) -> usize { + fields + .name + .iter() + .filter(|word| { + query + .terms + .iter() + .all(|term| relatedness(term, word) == 0.0) + && !query + .compounds + .iter() + .any(|(compound, _)| exact_or_synonym(compound, word) > 0.0) + }) + .count() +} + +/// Whether the query says it has many inputs. A typo does not: "patch" and +/// "watch" are one letter from "batch", and a query that means one of them +/// is not asking for a `batch` command. +fn asks_for_many(query: &Query) -> bool { + query.terms.iter().any(|term| { + ["batch", "multiple"].iter().any(|many| { + similarity(&term.word, many, false) > 0.0 || synonyms(&term.word).contains(many) + }) + }) +} + +/// [`similarity`], or a synonym's exact match if that is closer. A synonym +/// is not fuzzy-matched: it is already a guess, and "batch" one letter off +/// is "match". +fn relatedness(term: &Term, word: &str) -> f64 { + let direct = similarity(&term.word, word, term.typos); + if synonyms(&term.word).contains(&word) { + direct.max(SYNONYM_WEIGHT) + } else { + direct + } +} + +/// Compounds are only ever matched whole: "logout", not a word it starts. +fn exact_or_synonym(compound: &str, word: &str) -> f64 { + if compound == word { + 1.0 + } else if synonyms(compound).contains(&word) { + SYNONYM_WEIGHT + } else { + 0.0 + } +} + +fn synonyms(term: &str) -> &'static [&'static str] { + SYNONYMS + .iter() + .find(|(key, _)| *key == term) + .map_or(&[], |(_, synonyms)| synonyms) +} + +/// How well a query term matches one word of a command's text, from 0 to 1. +/// `typos` allows a match one or two edits off. +fn similarity(term: &str, word: &str, typos: bool) -> f64 { + if term == word { + return EXACT; + } + let term_len = term.chars().count(); + // Typing the start of a word: "geo" for "geocoder". + if term_len >= 3 && word.starts_with(term) { + return PREFIX; + } + // The same stem, spelled on for a different ending: "geocoding" and + // "geocoder". + let shared = term + .chars() + .zip(word.chars()) + .take_while(|(a, b)| a == b) + .count(); + if shared >= 5 { + return STEM; + } + if !typos { + return 0.0; + } + // Short words are left out: up to four letters, one edit turns too many + // words into another real one — "shop" into "show" — and up to seven, + // two do — "convert" into "content". + let edits = strsim::damerau_levenshtein(term, word); + let allowed = match term_len.min(word.chars().count()) { + 0..=4 => 0, + 5..=7 => 1, + _ => 2, + }; + if edits <= allowed { + return TYPO; + } + 0.0 +} + +/// The query's words, minus the ones that only describe the asking. +fn terms(query: &str) -> Vec { + let mut seen = BTreeSet::new(); + raw_tokens(query) + // Before the plural `s` comes off, which would leave "this" as "thi". + .filter(|word| !STOPWORDS.contains(&word.as_str())) + .map(singular) + .filter(|word| seen.insert(word.clone())) + .collect() +} + +fn words(text: &str) -> BTreeSet { + tokens(text).collect() +} + +/// Lowercased runs of letters and digits, each with a plural `s` taken off, +/// so "styles" in a query matches "style" in a summary. Cruder than +/// stemming, and enough for the words this CLI's commands use. +fn tokens(text: &str) -> impl Iterator + '_ { + raw_tokens(text).map(singular) +} + +fn raw_tokens(text: &str) -> impl Iterator + '_ { + text.split(|c: char| !c.is_alphanumeric()) + .filter(|word| !word.is_empty()) + .map(str::to_lowercase) +} + +fn singular(word: String) -> String { + match word.strip_suffix('s') { + Some(stem) if stem.chars().count() >= 3 && !stem.ends_with('s') => stem.to_string(), + _ => word, + } +} + +fn emit(mode: Mode, entries: &[&CommandEntry], json: Vec) -> Result<()> { + // No table, not an empty line: `text` with no match prints nothing, so + // `mapbox commands search … > file` leaves an empty file. `json` still + // answers `[]`, the one shape a parser can take. + if entries.is_empty() && !mode.is_json() { + return Ok(()); + } + let width = entries + .iter() + .map(|entry| entry.command.len()) + .max() + .unwrap_or(0) + .max("COMMAND".len()); + let text = std::iter::once(format!("{:width$} SUMMARY", "COMMAND")) + .chain( + entries + .iter() + .map(|entry| format!("{:width$} {}", entry.command, entry.summary)), + ) + .collect::>() + .join("\n"); + output::emit(mode, &text, Value::Array(json))?; + if !mode.is_json() { + output::print_tips(&["`mapbox --help` shows what one takes.".to_string()]); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn top(query: &str) -> Vec { + let specs = crate::spec::effective_services().expect("specs load"); + let app = crate::build_app(&specs); + rank(&app, &specs, query) + .into_iter() + .take(3) + .map(|(_, entry)| entry.command) + .collect() + } + + /// A query set under `tests/fixtures/commands_search/`, and how many + /// rows it answered right when last recorded: first result, and anywhere + /// in the first three. A `-` row is right when nothing comes back. + /// + /// A set fails only when it drops more than [`SLACK`] below these. A + /// change that improves one records the new score in the same commit, so + /// the gain cannot quietly be given back; one that drops past the slack + /// lowers it and says why in its PR. New real queries go in a new file + /// rather than into these: a set the search has been tuned against stops + /// saying how it does on the next query nobody has seen. + const EVAL_SETS: &[(&str, &str, usize, usize)] = &[ + ( + "tuning.tsv", + include_str!("../tests/fixtures/commands_search/tuning.tsv"), + 84, + 97, + ), + ( + "issues.tsv", + include_str!("../tests/fixtures/commands_search/issues.tsv"), + 50, + 67, + ), + ( + "docs.tsv", + include_str!("../tests/fixtures/commands_search/docs.tsv"), + 22, + 30, + ), + ]; + + /// How many queries a set may drop below its recorded score before it + /// fails. Without it, any change that moves one query by one place fails + /// the build, including a spec regeneration that rewords a summary. The + /// targeted tests below are the hard gate for specific queries. + const SLACK: usize = 3; + + /// Holds one set to its recorded scores, less [`SLACK`]. + /// `cargo test eval_set -- --nocapture` prints the scores, and each query + /// that did not rank first. One test per set, so they run in parallel. + fn eval_set_keeps_its_scores(wanted: &str) { + let &(name, text, recorded_first, recorded_top3) = EVAL_SETS + .iter() + .find(|(name, ..)| *name == wanted) + .unwrap_or_else(|| panic!("`{wanted}` is not in EVAL_SETS")); + let specs = crate::spec::effective_services().expect("specs load"); + let app = crate::build_app(&specs); + let index = index(&app, &specs); + // From the index rather than the whole schema: an expected + // `commands` command is one search never answers with. + let known: BTreeSet<&str> = index + .iter() + .map(|(_, entry)| entry.command.as_str()) + .collect(); + + let (mut rows, mut first, mut top3) = (0, 0, 0); + let (mut misses, mut invalid) = (vec![], vec![]); + for line in text + .lines() + .filter(|l| !l.starts_with('#') && !l.trim().is_empty()) + { + let mut columns = line.split('\t'); + let (Some(query), Some(expected)) = (columns.next(), columns.next()) else { + invalid.push(format!("no expected column: {line:?}")); + continue; + }; + let expected: Vec<&str> = match expected { + "-" => vec![], + answers => answers.split('|').collect(), + }; + // A renamed command would leave its row unwinnable, and the + // floor would read that as the search getting worse. + if let Some(command) = expected.iter().find(|c| !known.contains(**c)) { + invalid.push(format!( + "`{command}` is not a command search answers with (\"{query}\")" + )); + continue; + } + let found: Vec = rank_in(&index, query) + .into_iter() + .take(3) + .map(|(_, i)| index[i].1.command.clone()) + .collect(); + let (is_first, in_top3) = if expected.is_empty() { + (found.is_empty(), found.is_empty()) + } else { + ( + found + .first() + .is_some_and(|c| expected.contains(&c.as_str())), + found.iter().any(|c| expected.contains(&c.as_str())), + ) + }; + rows += 1; + first += usize::from(is_first); + top3 += usize::from(in_top3); + if !is_first { + misses.push(format!(" \"{query}\": want {expected:?}, got {found:?}")); + } + } + eprintln!( + "{name}: {first}/{rows} first (recorded {recorded_first}), \ + {top3}/{rows} in top 3 (recorded {recorded_top3})" + ); + for miss in &misses { + eprintln!("{miss}"); + } + assert!(invalid.is_empty(), "{name}:\n{}", invalid.join("\n")); + assert!( + first + SLACK >= recorded_first && top3 + SLACK >= recorded_top3, + "{name} dropped more than {SLACK} below its recorded score: {first} first \ + (recorded {recorded_first}), {top3} in top 3 (recorded {recorded_top3}). \ + If the trade-off is intended — help text made accurate at the cost of a \ + match, say — record {first}/{top3} in EVAL_SETS and say why in the PR. \ + The misses are above.", + ); + if first > recorded_first || top3 > recorded_top3 { + eprintln!( + " above its recorded score: record {}/{} in EVAL_SETS", + first.max(recorded_first), + top3.max(recorded_top3) + ); + } + } + + /// A file added to the directory but not to [`EVAL_SETS`] would be a + /// set that is never scored. + #[test] + fn every_query_file_is_an_eval_set() { + let dir = concat!( + env!("CARGO_MANIFEST_DIR"), + "/tests/fixtures/commands_search" + ); + let mut files: Vec = std::fs::read_dir(dir) + .expect("read the fixture directory") + .map(|entry| entry.expect("a directory entry").file_name()) + .filter_map(|name| name.into_string().ok()) + .filter(|name| name.ends_with(".tsv")) + .collect(); + files.sort(); + let mut sets: Vec<&str> = EVAL_SETS.iter().map(|(name, ..)| *name).collect(); + sets.sort_unstable(); + assert_eq!(files, sets); + } + + #[test] + fn eval_set_tuning() { + eval_set_keeps_its_scores("tuning.tsv"); + } + + #[test] + fn eval_set_issues() { + eval_set_keeps_its_scores("issues.tsv"); + } + + #[test] + fn eval_set_docs() { + eval_set_keeps_its_scores("docs.tsv"); + } + + /// Phrasings a person or an agent might actually type, each held to the + /// command it should find first. A change to the scoring that loses one + /// of these has made the search worse for that caller. + #[test] + fn a_task_finds_its_command_first() { + for (query, expected) in [ + ("upload a font", "mapbox fonts upload"), + ("list my styles", "mapbox styles list"), + ("delete a style", "mapbox styles delete"), + ("geocode an address", "mapbox geocoder forward"), + ("driving route between two points", "mapbox directions"), + ("travel time matrix", "mapbox matrix"), + ("snap gps trace to roads", "mapbox map-match"), + ("static map image", "mapbox static get-image"), + ("log in", "mapbox auth login"), + ("list access tokens", "mapbox accounts list-tokens"), + ("shell completion", "mapbox completion"), + ("install mcp server", "mapbox mcp install"), + ("find coffee shops near me", "mapbox search category"), + ("query features at a point", "mapbox tilesets query"), + ("vector tile", "mapbox tilesets get-mvt"), + ("add an icon to a sprite", "mapbox sprites upload"), + ("download a style", "mapbox styles download"), + ("reachable area in 10 minutes", "mapbox isochrone"), + ("feedback about the cli", "mapbox report"), + ("sign in", "mapbox auth login"), + ("discard a style draft", "mapbox styles draft delete"), + ("rename my style", "mapbox styles update"), + ( + "bulk upload icons to a sprite", + "mapbox sprites upload-batch", + ), + ("add a ttf font", "mapbox fonts upload"), + ("publish a tileset", "mapbox tilesets-cli"), + ("turn off telemetry", "mapbox config set"), + ("check my proxy settings", "mapbox doctor"), + ("uninstall the cli", "mapbox uninstall"), + ] { + let found = top(query); + assert_eq!( + found.first().map(String::as_str), + Some(expected), + "\"{query}\" ranked {found:?}" + ); + } + } + + #[test] + fn a_typo_still_finds_the_command() { + assert_eq!( + top("uplaod font").first().map(String::as_str), + Some("mapbox fonts upload") + ); + } + + #[test] + fn a_query_of_only_stopwords_matches_nothing() { + assert!(top("how do I").is_empty()); + } + + #[test] + fn nonsense_matches_nothing() { + assert!(top("xyzzy").is_empty()); + } + + #[test] + fn matches_far_below_the_best_are_dropped() { + let kept: Vec<&str> = above_floor(vec![(10.0, "best"), (1.0, "tenth"), (0.9, "noise")]) + .into_iter() + .map(|(_, name)| name) + .collect(); + assert_eq!(kept, ["best", "tenth"]); + } + + const WEAK_QUERY: &str = "how far can i cycle in twenty minutes"; + + /// A loosely worded task matched by one word is still answered: a + /// fixed minimum score once emptied this. Not a phrasing from any query + /// set, so it does not lean on one the search was fitted to. + #[test] + fn a_weak_only_match_is_still_an_answer() { + assert_eq!( + top(WEAK_QUERY).first().map(String::as_str), + Some("mapbox isochrone") + ); + } + + /// `geocoder batch`'s description names everything a single lookup + /// does, so without the discount it ranks first for a single query. + #[test] + fn batch_comes_second_unless_asked_for() { + assert_eq!( + top("geocoding results inside a bbox") + .first() + .map(String::as_str), + Some("mapbox geocoder forward") + ); + assert_eq!( + top("bulk geocode addresses").first().map(String::as_str), + Some("mapbox geocoder batch") + ); + } + + /// "patch" and "match" are a letter from "batch", and both are words a + /// command's text uses as they are, so neither asks for many. + #[test] + fn a_word_one_letter_from_batch_does_not_find_a_batch_command() { + for query in ["patch a style", "match a style"] { + let found = top(query); + assert!( + found.first().is_some_and(|c| !c.contains("batch")), + "\"{query}\" ranked {found:?}" + ); + } + } + + #[test] + fn a_word_some_command_uses_is_not_a_typo() { + let vocabulary = BTreeSet::from(["match"]); + let query = Query::parse("match uplaod", &vocabulary); + let typos: Vec = query.terms.iter().map(|term| term.typos).collect(); + assert_eq!(typos, [false, true]); + assert_eq!(relatedness(&query.terms[0], "batch"), 0.0); + } + + /// The caller is already in `commands`, and its own summaries would + /// otherwise answer every query about finding a command. + #[test] + fn the_commands_group_is_left_out() { + let specs = crate::spec::effective_services().expect("specs load"); + let app = crate::build_app(&specs); + let found = rank(&app, &specs, "find a command search list commands"); + assert!( + found.iter().all(|(_, entry)| entry.service != COMMAND), + "{:?}", + found + .iter() + .map(|(_, entry)| &entry.command) + .collect::>() + ); + } + + #[test] + fn a_repeated_pair_is_one_compound() { + let query = Query::parse("log in log in", &BTreeSet::new()); + let compounds: Vec<&str> = query.compounds.iter().map(|(c, _)| c.as_str()).collect(); + assert_eq!(compounds, ["login", "inlog"]); + } + + #[test] + fn stopwords_are_dropped_before_the_plural_is() { + assert!(terms("what is this").is_empty()); + } + + fn term(word: &str) -> Term { + Term { + word: word.to_string(), + typos: true, + } + } + + #[test] + fn a_synonym_counts_for_less_than_the_word_itself() { + assert_eq!(relatedness(&term("remove"), "remove"), EXACT); + assert_eq!(relatedness(&term("remove"), "delete"), SYNONYM_WEIGHT); + assert_eq!(relatedness(&term("delete"), "remove"), 0.0); + } + + /// "bulk" stands in for "batch", and "batch" one letter off is + /// "match": the stand-in is matched exactly, not within a typo. + #[test] + fn a_synonym_is_not_fuzzy_matched() { + assert_eq!(relatedness(&term("bulk"), "batch"), SYNONYM_WEIGHT); + assert_eq!(similarity("batch", "match", true), TYPO); + assert_eq!(relatedness(&term("bulk"), "match"), 0.0); + } + + #[test] + fn a_plural_matches_its_singular() { + assert_eq!( + similarity(&terms("styles")[0], &terms("style")[0], true), + EXACT + ); + } + + #[test] + fn short_words_are_not_fuzzy_matched() { + assert_eq!(similarity("map", "mcp", true), 0.0); + assert_eq!(similarity("shop", "show", true), 0.0); + } + + /// Letters, not bytes: "café" is four letters, too short for a typo. + #[test] + fn lengths_count_letters() { + assert_eq!(similarity("cafè", "café", true), 0.0); + assert_eq!(singular("éés".to_string()), "éés"); + } +} diff --git a/src/config.rs b/src/config.rs index c924efe..dfd9c97 100644 --- a/src/config.rs +++ b/src/config.rs @@ -182,7 +182,7 @@ pub fn command() -> Command { }; Command::new(COMMAND) - .about("Get or set a persisted mapbox setting") + .about(format!("Get or set settings: {}", KEYS.join(", "))) .long_about( "Get or set a mapbox setting that persists across shells and sessions, \ written to a file beside the stored credentials rather than an \ @@ -196,7 +196,7 @@ pub fn command() -> Command { ) .subcommand( Command::new("set") - .about("Persist a setting") + .about("Turn a setting on or off, and keep it that way") .arg(key_arg()) .arg(Arg::new("value").required(true).value_parser([ON, OFF])), ) diff --git a/src/doctor.rs b/src/doctor.rs index ab91fb3..3e9faf5 100644 --- a/src/doctor.rs +++ b/src/doctor.rs @@ -107,13 +107,15 @@ const PROXY_VARS: &[(&str, &str)] = &[ pub fn command() -> Command { Command::new(COMMAND) - .about("Show the environment the next command would run in") + .about("Show the token, proxy and settings the next command would use") .long_about( "A read-only snapshot of what the next command would see: which token wins \ and its state, which proxy variables are in effect, and where the \ update-check and telemetry switches currently stand. `--verify` additionally \ checks that api.mapbox.com is reachable — the one part of this that makes a \ - request.", + request.\n\n\ + Start here when requests fail for no obvious reason: a missing or unexpected \ + token, or a network, proxy or firewall in the way.", ) .arg( Arg::new("verify") diff --git a/src/help_layout.rs b/src/help_layout.rs index 99acdde..4e58596 100644 --- a/src/help_layout.rs +++ b/src/help_layout.rs @@ -45,6 +45,7 @@ const GROUPS: &[(&str, &[&str])] = &[ ( "CLI", &[ + "commands", "config", "history", "doctor", @@ -80,7 +81,10 @@ const DESCRIPTIONS: &[(&str, &str)] = &[ "places", "Full detail for a place, by its Search Box result id", ), - ("feedback", "Submit and list feedback about Mapbox data"), + ( + "feedback", + "Report wrong or missing map data, and list what was reported", + ), ("directions", "Route between waypoints"), ( "matrix", @@ -416,7 +420,9 @@ fn render(sections: &[Section], width: usize, palette: &Palette) -> String { let lit = palette.literal; page.push_str(&format!( "{muted}Run{muted:#} {lit}mapbox --help{lit:#} \ - {muted}for a command's own options.{muted:#}", + {muted}for a command's own options.{muted:#}\n\ + {muted}Run{muted:#} {lit}mapbox commands search \"\"{lit:#} \ + {muted}to find a command by what it does.{muted:#}", muted = palette.muted, )); page @@ -634,7 +640,9 @@ fn grouped(rows: Vec<(String, String)>) -> Vec<(&'static str, Vec<(String, Strin groups } -fn description(name: &str) -> Option<&'static str> { +/// Also read by `commands search`, for which these are the plainest words +/// a group has. +pub(crate) fn description(name: &str) -> Option<&'static str> { DESCRIPTIONS .iter() .find(|(n, _)| *n == name) diff --git a/src/history.rs b/src/history.rs index b0cb9d8..0133cac 100644 --- a/src/history.rs +++ b/src/history.rs @@ -36,7 +36,7 @@ pub fn command() -> Command { .subcommand_required(true) .subcommand( Command::new("list") - .about("List the most recent runs, newest first") + .about("List the commands you ran recently, newest first") .arg( Arg::new("limit") .long("limit") @@ -47,7 +47,7 @@ pub fn command() -> Command { ) .subcommand( Command::new("show") - .about("Show everything recorded about one run") + .about("Show everything recorded about one run, the last one by default") .arg( Arg::new("id") .help("The run's id, or a prefix of it; the newest run when left out"), diff --git a/src/main.rs b/src/main.rs index 4c1823e..578a8d0 100644 --- a/src/main.rs +++ b/src/main.rs @@ -26,6 +26,7 @@ mod agent_skills; mod api_command_surface; mod auth; mod cli_token; +mod commands; mod completion; mod config; mod confirm; @@ -71,8 +72,9 @@ const REPO_URL: &str = repo_url!(); /// by chaining `--help` calls, one command at a time; `--schema` answers the /// same question for every command in one document. const AGENT_HELP_HINT: &str = "\ -Agents: `mapbox --schema` describes every command, its arguments and the request it makes -as one JSON document. Use it instead of chaining --help calls."; +Agents: `mapbox commands search \"\"` finds the command for a task. +`mapbox --schema` describes every command, its arguments and the request it +makes as one JSON document. Use these instead of chaining --help calls."; /// Help headings for the global options, which otherwise share one /// "Options:" list with every command's own. @@ -784,6 +786,10 @@ fn build_app(specs: &[ServiceSpec]) -> Command { // Beside `config`, which turns the history it reads on and off. app = app.subcommand(history::command()); + // Reads the same tree `--schema` does, so it is built from the commands + // above rather than listing them itself. + app = app.subcommand(commands::command()); + // Reads what the other hand-written commands above also read — the // token store, the proxy environment, the config and telemetry // switches — so it belongs beside them rather than the API surface @@ -943,6 +949,11 @@ fn cli() -> u8 { }, }; + if let Err(e) = commands::check_query(&matches) { + run_record::set_unparsed(&app, &raw_argv, e.kind()); + return report_parse_result(e, &raw_argv); + } + // Before anything resolves a token: describing a command makes no // request, and asking for a login first would be a poor way to answer a // question about what a login is for. @@ -1032,6 +1043,9 @@ fn report_parse_result(err: clap::Error, raw_argv: &[std::ffi::OsString]) -> u8 run_record::add_stdout_bytes(err.render().to_string().len()); } let _ = err.print(); + if let Some(fix) = search_fix(&err, &rendered) { + output::print_tips(&[fix.to_string()]); + } return code; } @@ -1054,8 +1068,11 @@ fn report_parse_result(err: clap::Error, raw_argv: &[std::ffi::OsString]) -> u8 // answers that, and inventing one would be worse than the message // alone — but clap's own suggester sometimes already has the answer, // and `-o text` shows it; carry it over rather than losing it here. - _ => match clap_tip(&rendered) { - Some(tip) => Remedy::default().with_fix(&tip), + _ => match clap_tip(&rendered) + .as_deref() + .or(search_fix(&err, &rendered)) + { + Some(fix) => Remedy::default().with_fix(fix), None => Remedy::default(), }, }); @@ -1064,6 +1081,16 @@ fn report_parse_result(err: clap::Error, raw_argv: &[std::ffi::OsString]) -> u8 code } +/// Where to go after a command name nothing resembles, which is usually a +/// guess at what the command is called — the question `commands search` +/// answers. A fix rather than an action: the query is the caller's to write. +/// Nothing when clap already suggested a near spelling, which is the better +/// answer. +fn search_fix(err: &clap::Error, rendered: &str) -> Option<&'static str> { + (err.kind() == ErrorKind::InvalidSubcommand && clap_tip(rendered).is_none()) + .then_some("To find a command by what it does: mapbox commands search \"\"") +} + /// The `tip: …` line clap's own suggester renders for an unrecognized /// subcommand or flag, if there is one. /// @@ -1388,6 +1415,14 @@ fn run(app: &Command, specs: &[ServiceSpec], matches: &ArgMatches, mode: Mode) - Some(("unset", unset_matches)) => config::unset(unset_matches, mode)?, _ => unreachable!("`config` sets subcommand_required(true)"), }, + // Ahead of the generic service arm too, and handed `app` for the + // same reason `generate-skills` is: it reads the command tree already + // in memory. No token and no request. + Some((commands::COMMAND, commands_matches)) => match commands_matches.subcommand() { + Some(("search", search_matches)) => commands::search(app, specs, search_matches, mode)?, + Some(("list", _)) => commands::list(app, specs, mode)?, + _ => unreachable!("`commands` sets subcommand_required(true)"), + }, // Ahead of the generic service arm too: it reads local history and // makes no request. Some((history::COMMAND, history_matches)) => match history_matches.subcommand() { diff --git a/src/mcp.rs b/src/mcp.rs index ac11282..5b22d92 100644 --- a/src/mcp.rs +++ b/src/mcp.rs @@ -329,6 +329,15 @@ fn spawn_live(client: &Client, args: &[&str]) -> io::Result<(ExitStatus, String) Err(last_err.unwrap_or_else(|| io::Error::from(io::ErrorKind::NotFound))) } +/// "A, B or C", for help text built from [`SERVERS`] and [`CLIENTS`]. +fn joined(items: impl Iterator, last: &str) -> String { + let items: Vec<&str> = items.collect(); + match items.split_last() { + Some((tail, init)) if !init.is_empty() => format!("{} {last} {tail}", init.join(", ")), + _ => items.concat(), + } +} + pub fn command() -> ClapCommand { let server_flags: Vec<&'static str> = SERVERS.iter().map(|s| s.flag).collect(); let client_flags: Vec<&'static str> = CLIENTS.iter().map(|c| c.flag).collect(); @@ -354,6 +363,12 @@ pub fn command() -> ClapCommand { .subcommand( ClapCommand::new(INSTALL) .about("Register a server with a client. With neither flag, every known server for every detected client") + .long_about(format!( + "Register the {} servers with a coding agent: {}. With neither flag, registers \ + every known server with every detected client.", + joined(SERVERS.iter().map(|s| s.label), "and"), + joined(CLIENTS.iter().map(|c| c.label), "or"), + )) .arg( Arg::new(SERVER_ARG) .long(SERVER_ARG) diff --git a/src/output/mod.rs b/src/output/mod.rs index dc84b18..bab5b68 100644 --- a/src/output/mod.rs +++ b/src/output/mod.rs @@ -324,7 +324,7 @@ pub fn progress(message: &str) { /// caller needs no guard. A blank line first — these are notes about /// whatever was just printed, not more of it, and butted against the last /// line they'd read as one. -fn print_tips(tips: &[String]) { +pub(crate) fn print_tips(tips: &[String]) { if tips.is_empty() { return; } diff --git a/src/schema.rs b/src/schema.rs index 2f8387d..731f012 100644 --- a/src/schema.rs +++ b/src/schema.rs @@ -386,6 +386,14 @@ fn commands(app: &Command, specs: &[ServiceSpec], path: &[String]) -> Vec Command { Command::new(COMMAND) - .about("Run the Mapbox Tilesets CLI (`tilesets`, installed separately)") + .about("Run the separate Tilesets CLI: upload data, publish tilesets") .long_about(format!( - "Forward arguments to the Mapbox Tilesets CLI.\n\n\ + "Forward arguments to the Mapbox Tilesets CLI, the command-line client for \ + Mapbox Tiling Service (MTS): upload GeoJSON as a tileset source, validate a \ + recipe, create and publish a tileset, and check on its jobs. It also lists, \ + updates and deletes tilesets and tileset sources.\n\n\ Everything after `{COMMAND}` is passed through to the `tilesets` binary \ untouched — including flags, so `{COMMAND} --help` shows the Tilesets \ CLI's own help rather than this one. `tilesets` ships separately as the \ diff --git a/src/uninstall.rs b/src/uninstall.rs index 42449e3..be90736 100644 --- a/src/uninstall.rs +++ b/src/uninstall.rs @@ -31,7 +31,7 @@ const DRY_RUN_HELP: &str = "Describe what this would delete, then exit without d pub fn command() -> Command { Command::new(COMMAND) - .about("Remove the installed `mapbox` binary") + .about("Uninstall this CLI by removing the installed `mapbox` binary") .long_about( "Remove the `mapbox` binary this process is running from.\n\n\ Deletes only the executable. Stored credentials, `auth logout`'s file, \ diff --git a/tests/fixtures/commands_search/README.md b/tests/fixtures/commands_search/README.md new file mode 100644 index 0000000..6ffea84 --- /dev/null +++ b/tests/fixtures/commands_search/README.md @@ -0,0 +1,91 @@ +# `commands search` query sets + +Queries that `mapbox commands search` is scored against. The `eval_set_*` +tests in `src/commands.rs` run every file here and fail when a set ranks +more than `SLACK` (3) queries fewer right than its recorded score, so a +reworded summary that moves one query does not fail the build. +`cargo test eval_set -- --nocapture` prints each set's score and every +query that did not rank first. + +## Format + +Tab-separated, one query per line, `#` lines are comments: + +| Column | Meaning | +| --- | --- | +| query | What a person or an agent types, as one string. | +| expected | The command that should rank first, as `mapbox …`. `\|` separates several acceptable answers; `-` means the CLI cannot do this and nothing should come back. | +| kind | What sort of query it is (below). | +| source | Where the phrasing came from (`issues.tsv` and `docs.tsv` only). | + +A query counts as right when its first result is one of the expected +commands, or, for `-`, when the result is empty. Top 3 counts it when an +expected command is anywhere in the first three. + +## The sets + +The sets differ mostly in how much the search has been tuned against them. +The more it was, the better the score looks and the less it says about a +query nobody has seen. + +| File | Rows | Source | Tuned against | +| --- | --- | --- | --- | +| `tuning.tsv` | 100 | Written by hand while building the search | Heavily: scoring and help text were changed until these ranked well | +| `issues.tsv` | 100 | Mapbox GitHub issue and help-center titles, lightly rewritten | Partly: its misses were read to choose which kinds of fix to make | +| `docs.tsv` | 60 | Task phrasings from Mapbox docs, blog posts and tutorials | Least: unseen until the last round, then used once to choose between two changes | + +Kinds: + +- `agent`: worded the way the CLI's own text is, as an agent that has read + `--help` would ("get sprite json"). +- `human`: a person's own words ("which fonts do I have"). +- `synonym`: a word the command's text says differently ("rename my style"). +- `typo`: a realistic misspelling ("uplaod a font"). +- `symptom`: a bug report rather than a task ("estimate area returns 0 km2"). + The search is not built for these; they are a secondary measure. +- `none`: a task the CLI cannot do ("build a store locator"). + +## Reading the scores + +- A drop in `tuning.tsv` or `issues.tsv` means something that worked broke. +- `docs.tsv` is the best estimate of how the search does on unseen + phrasing. A gain there is the one worth believing. +- Keyword search has a ceiling: a query that shares no words with any + command's text cannot be found. Most of the remaining misses are of that + kind, and tuning will not fix them. +- A `-` row can only be won when no word of the query matches anything, + because matches are cut relative to the best one, never at a fixed + minimum (see `MIN_SHARE_OF_BEST`). A task the CLI cannot do almost always + shares a word with some command, so those rows are lost today, and the + rows a set can actually win are fewer than its total: + + | File | Rows | `-` rows | `-` rows won | Rows that can win | + | --- | --- | --- | --- | --- | + | `tuning.tsv` | 100 | 2 | 2 | 100 | + | `issues.tsv` | 100 | 11 | 0 | 89 | + | `docs.tsv` | 60 | 11 | 0 | 49 | + + They stay in the score, rather than being scored apart, so a set's score + keeps one meaning across changes, and a change that starts answering + "nothing" for them shows up as a gain in the same number. `tuning.tsv` + wins its two only because "bake a cake" and "deploy to kubernetes" share + no word with any command. + +## Limits + +- Stack Overflow, GIS StackExchange and Reddit could not be fetched while + `issues.tsv` and `docs.tsv` were gathered, so neither has queries from + them, and both lean toward developer phrasing. +- Many `docs.tsv` sources are a page title seen in search results, not a page + that was read in full. +- The expected command is a judgment call for some rows, which is why some + list more than one. + +## Changing them + +- Record a set's new score in `EVAL_SETS` when a change improves it, in the + same commit. Dropping past the slack is a decision to explain in the PR. +- Fix the kind of miss rather than special-casing a query. +- Put new real queries, such as ones the team actually typed, in a new file + rather than these, so there is always a set the search has not been + fitted to. Add it to `EVAL_SETS` with its own test. diff --git a/tests/fixtures/commands_search/docs.tsv b/tests/fixtures/commands_search/docs.tsv new file mode 100644 index 0000000..ceaa388 --- /dev/null +++ b/tests/fixtures/commands_search/docs.tsv @@ -0,0 +1,64 @@ +# Task phrasings from Mapbox docs, blog posts and tutorials, unseen until the +# last tuning round and then used once to choose between two changes. The +# least tuned-against of the three. Columns: query, expected first result, +# kind, source. +turn a list of customer addresses into coordinates mapbox geocoder batch|mapbox geocoder forward human https://hackersandslackers.com/preparing-data-for-mapbox-geocoding/ +find the address nearest to a gps point mapbox geocoder reverse|mapbox search reverse human https://docs.mapbox.com/help/dive-deeper/geocoding.md +look up a place name and get its coordinates mapbox geocoder forward|mapbox search forward human https://designcode.io/react-hooks-handbook-geocoding-mapbox +geocode addresses for a spreadsheet mapbox geocoder batch|mapbox geocoder forward human https://interworks.com/blog/2020/07/28/geocoding-in-tableau-prep-with-mapbox/ +search for an address in japan mapbox search forward|mapbox geocoder forward human https://docs.mapbox.com/help/troubleshooting/japan-specific-considerations-search-api.md +reverse lookup that returns several results instead of one mapbox geocoder reverse|mapbox search reverse human https://docs.mapbox.com/android/search/api/core/2.7.0/sdk/com.mapbox.search/-reverse-geo-options/limit.html +route that avoids toll roads mapbox directions human https://docs.mapbox.com/api/navigation/directions/ +cycling route with distance and duration mapbox directions human https://docs.mapbox.com/help/dive-deeper/directions.md +route through several stops in order mapbox directions human https://rdrr.io/cran/mapboxapi/man/mb_directions.html +driving time with live traffic mapbox directions human https://docs.mapbox.com/api/navigation/directions/ +add a pin to a static map mapbox static get-image human https://blog.mapbox.com/static-api-with-overlays-932ffc5fcf3d +map image url to use in an img tag mapbox static get-image human https://scour.ing/@tomasz/p/https://dev.to/mapbox/server-rendered-maps-for-web-and-mobile-51f7 +make a map picture for printing mapbox static get-image human https://docs.mapbox.com/help/dive-deeper/static-maps.md +static map with a custom icon marker mapbox static get-image human https://docs.mapbox.com/android/java/guides/static-image/ +export my studio style to a json file mapbox styles get|mapbox styles download human https://docs.mapbox.com/atlas/guides/styles/ +create a style from a json file mapbox styles create human https://docs.mapbox.com/style-spec/guides/ +change a published style without opening studio mapbox styles update human https://docs.mapbox.com/style-spec/guides/ +save work in progress on a style before publishing mapbox styles draft update human https://docs.mapbox.com/atlas/guides/styles/ +add svg icons to my style mapbox sprites upload|mapbox sprites upload-batch human https://docs.mapbox.com/studio-manual/examples/custom-icon +add a whole folder of svg icons at once mapbox sprites upload-batch human https://docs.mapbox.com/style-spec/reference/sprite.md +which fontstacks can i use for labels mapbox fonts list human https://docs.mapbox.com/api/maps/fonts/ +remove a font i uploaded by mistake mapbox fonts delete human https://docs.mapbox.com/api/maps/fonts/ +turn populated places data into vector tiles mapbox tilesets-cli human https://docs.mapbox.com/help/tutorials/get-started-mts-and-tilesets-cli.md +check a recipe is valid before using it mapbox tilesets-cli human https://pypi.org/project/mapbox-tilesets/1.5.0 +why did my tileset fail to publish mapbox tilesets-cli human https://docs.mapbox.com/mapbox-tiling-service/guides/errors/ +find out which source features were dropped mapbox tilesets-cli human https://docs.mapbox.com/mapbox-tiling-service/guides/frequentlyaskedquestions.md +pick the minimum and maximum zoom for my tiles mapbox tilesets-cli human https://docs.mapbox.com/mapbox-tiling-service/examples/basic-recipe-using-zoom-levels/ +which poi and building layers are within 30 meters of a point mapbox tilesets query human https://docs.mapbox.com/api/maps/tilequery.md +get the nearest features from a tileset around a location mapbox tilesets query human https://docs.mapbox.com/android/java/guides/services/tilequery/ +make a token that only works on my website - none https://docs.mapbox.com/help/dive-deeper/access-tokens/ +which scopes do i need to upload fonts mapbox accounts list-scopes human https://docs.mapbox.com/api/maps/fonts/ +what is the difference between public and secret tokens mapbox accounts list-scopes|mapbox accounts list-tokens human https://docs.mapbox.com/accounts/guides/tokens/ +see every token on my account and when it was created mapbox accounts list-tokens human https://docs.mapbox.com/accounts/guides/tokens/ +switch between two saved accounts mapbox auth profiles|mapbox auth login human https://docs.mapbox.com/cli/reference.md +take mapbox off my laptop mapbox uninstall human https://docs.mapbox.com/cli/reference/uninstall/ +tab completion for fish mapbox completion human https://docs.mapbox.com/cli/reference.md +disable the update check mapbox config set human https://docs.mapbox.com/cli/reference.md +what did i run yesterday mapbox history list human https://docs.mapbox.com/cli/reference.md +connect cursor to mapbox tools mapbox mcp install human https://cdn.jsdelivr.net/gh/mapbox/mcp-devkit-server@main/README.md +how many requests did i make last week mapbox usage human https://docs.mapbox.com/accounts/guides/statistics/ +which product is using most of my quota mapbox usage human https://docs.mapbox.com/help/faq/what-statistics-are-available-for-my-mapbox-account/ +how far can i walk in ten minutes mapbox isochrone human https://www.mapbox.com/insights/isochrones +delivery catchment area around a store mapbox isochrone human https://www.mapbox.com/blog/using-isochrones-to-understand-reachability-in-web-and-mobile-applications +durations from one warehouse to many customers mapbox matrix human https://docs.mapbox.com/api/navigation/matrix.md +pick the closest depot by drive time mapbox matrix|mapbox directions human https://blog.afi.io/blog/real-time-traffic-with-the-mapbox-api/ +fix gps jitter in a recorded ride mapbox map-match human https://docs.mapbox.com/playground/map-matching/ +find bookstores along my route mapbox search category human https://docs.mapbox.com/mapbox-search-js/api/core/search.md +get details and coordinates for a venue from a search result mapbox places human https://mapbox.com/blog/introducing-search-box +tell mapbox a road is missing from the map mapbox feedback create human https://docs.mapbox.com/help/dive-deeper/mapbox-data.md +check on a map problem i reported earlier mapbox feedback get|mapbox feedback list human https://docs.mapbox.com/help/dive-deeper/mapbox-data.md +show map labels in another language in gl js - none https://docs.mapbox.com/help/dive-deeper/change-language +show a popup when someone clicks a point - none https://docs.mapbox.com/mapbox-gl-js/example/popup-on-click/ +how much does mapbox cost per month - none https://docs.mapbox.com/accounts/overview/pricing/ +delete my mapbox account - none https://docs.mapbox.com/help/account/settings +add 3d buildings to my map - none https://docs.mapbox.com/help/dive-deeper/z_add-3d-buildings-studio.md +make a choropleth map - none https://docs.mapbox.com/help/tutorials/choropleth-studio-gl-pt-1.md +offline maps in an ios app - none https://docs.mapbox.com/help/tutorials/ios-offline-maps.md +toggle layers with checkboxes in react - none https://docs.mapbox.com/help/tutorials/react-toggle-layers.md +turn by turn navigation in my ios app - none https://docs.mapbox.com/help/tutorials/ios-navigation.md +geofencing in an android app - none https://docs.mapbox.com/help/tutorials/android-geofencing.md diff --git a/tests/fixtures/commands_search/issues.tsv b/tests/fixtures/commands_search/issues.tsv new file mode 100644 index 0000000..7c1dc79 --- /dev/null +++ b/tests/fixtures/commands_search/issues.tsv @@ -0,0 +1,104 @@ +# Lightly rewritten titles of Mapbox GitHub issues and help-center pages. Used +# to pick which kinds of fix to make, so partly tuned against. `symptom` rows +# are bug reports rather than tasks; `none` rows are tasks this CLI cannot do. +# Columns: query, expected first result, kind, source. +save static api response to a png file mapbox static get-image human https://github.com/mapbox/mapbox-sdk-js/issues/317 +static image request entity too large mapbox static get-image symptom https://github.com/mapbox/mapbox-sdk-js/issues/324 +add text to a static map image mapbox static get-image human https://docs.mapbox.com/help/tutorials/static-images-text +put a static map in an email receipt mapbox static get-image human https://docs.mapbox.com/help/tutorials/receipt-map +geojson polygon overlay on a static image mapbox static get-image human https://github.com/mapbox/mapbox-sdk-js/issues/283 +hide multiple layers in static images mapbox static get-image human https://github.com/mapbox/mapbox-gl-js/issues/12607 +offset a custom marker in a static map mapbox static get-image human https://github.com/mapbox/mapbox-sdk-js/issues/431 +static map with language and worldview mapbox static get-image human https://github.com/mapbox/mapbox-gl-js/issues/11005 +mpaboxstatic coordinates auto requires zoom mapbox static get-image symptom https://github.com/mapbox/mapbox-sdk-js/issues/268 +static tiles of a custom style look wrong mapbox static get-tile|mapbox static get-image symptom https://github.com/mapbox/mapbox-gl-js/issues/13504 +serving raster tiles source mapbox tilesets get-tile|mapbox static get-tile human https://github.com/mapbox/mapbox-gl-js/issues/9280 +create a style with a geojson source mapbox styles create human https://github.com/mapbox/mapbox-sdk-js/issues/210 +create a custom basemap style mapbox styles create human https://docs.mapbox.com/help/tutorials/create-a-custom-style +list styles with fresh results not cached mapbox styles list symptom https://github.com/mapbox/mapbox-sdk-js/issues/389 +transfer styles between accounts mapbox styles download|mapbox styles create human https://docs.mapbox.com/help/dive-deeper/transfer-styles-between-accounts +add a map style offline mapbox styles download human https://github.com/mapbox/mapbox-gl-js/issues/9861 +creating a style and rendering it mapbox styles create|mapbox static get-image human https://github.com/mapbox/mapbox-sdk-js/issues/322 +map does not show the latest styling changes mapbox styles get|mapbox styles draft get symptom https://docs.mapbox.com/help/troubleshooting/stale-data +sprite property in style adding custom images mapbox sprites upload human https://github.com/mapbox/mapbox-gl-js/issues/9854 +deleting icons dont work right away mapbox sprites delete symptom https://github.com/mapbox/mapbox-sdk-js/issues/388 +svg image errors when uploading an icon mapbox sprites upload symptom https://docs.mapbox.com/help/troubleshooting/studio-svg-upload-errors +custom sprite in a custom style does not work mapbox sprites upload|mapbox sprites get-json symptom https://github.com/mapbox/mapbox-gl-js/issues/13268 +add a custom font to a map style mapbox fonts upload human https://docs.mapbox.com/help/how-to-videos/add-custom-font-studio-video +manage your fontstacks mapbox fonts list human https://docs.mapbox.com/help/dive-deeper/manage-fontstacks +custom font appears in preview but not in published style mapbox fonts list|mapbox fonts upload symptom https://github.com/mapbox/mapbox-gl-js/issues/13577 +no geojson features processed make sure line delimited mapbox tilesets-cli symptom https://github.com/mapbox/tilesets-cli/issues/155 +how do i deal with a tileset bigger than 50 gb mapbox tilesets-cli human https://github.com/mapbox/tilesets-cli/issues/164 +cant re-publish a tileset after adding new sources mapbox tilesets-cli symptom https://github.com/mapbox/tilesets-cli/issues/162 +delete a tileset source mapbox tilesets-cli human https://github.com/mapbox/tilesets-cli/issues/171 +list only shows the last 100 tilesets mapbox tilesets-cli symptom https://github.com/mapbox/tilesets-cli/issues/135 +estimate area returns 0 km2 mapbox tilesets-cli symptom https://github.com/mapbox/tilesets-cli/issues/161 +upload large source files to mapbox tiling service mapbox tilesets-cli human https://docs.mapbox.com/help/troubleshooting/upload-large-source-files-mts +keep a tileset updated with incremental updates mapbox tilesets-cli human https://docs.mapbox.com/help/tutorials/mts-incremental-updates +cluster point data with the tiling service mapbox tilesets-cli human https://docs.mapbox.com/help/tutorials/cluster-point-data-with-mts +view the recipe of a tileset mapbox tilesets-cli human https://github.com/mapbox/tilesets-cli/issues/126 +status of an unpublished tileset mapbox tilesets-cli human https://github.com/mapbox/tilesets-cli/issues/127 +adjust the zoom extent of my tileset mapbox tilesets-cli human https://docs.mapbox.com/help/troubleshooting/adjust-tileset-zoom-extent +csv file errors on upload mapbox tilesets-cli symptom https://docs.mapbox.com/help/troubleshooting/csv-upload-errors +how to remove an incomplete upload mapbox tilesets-cli human https://github.com/mapbox/mapbox-cli-py/issues/90 +retreiving tilesets mapbox tilesets-cli|mapbox tilesets get-mvt symptom https://github.com/mapbox/mapbox-cli-py/issues/130 +upload data to mapbox mapbox tilesets-cli human https://docs.mapbox.com/help/troubleshooting/uploads +feature ids in tilequery response mapbox tilesets query human https://github.com/mapbox/mapbox-sdk-js/issues/265 +point in polygon query mapbox tilesets query human https://docs.mapbox.com/help/tutorials/point-in-polygon-query-with-mapbox-boundaries +get the city and street where my marker is mapbox geocoder reverse|mapbox search reverse human https://github.com/mapbox/mapbox-gl-js/issues/12302 +unable to change location name to coordinates mapbox geocoder forward|mapbox search forward human https://github.com/mapbox/mapbox-sdk-js/issues/424 +country code in reverse geocode response mapbox geocoder reverse|mapbox search reverse human https://github.com/mapbox/mapbox-sdk-js/issues/423 +geocode with proximity set to ip mapbox geocoder forward|mapbox search forward human https://github.com/mapbox/mapbox-sdk-js/issues/466 +geocoding bbox parameter mapbox geocoder forward|mapbox search forward human https://github.com/mapbox/mapbox-cli-py/issues/70 +perform batch geocoding mapbox geocoder batch human https://docs.mapbox.com/help/how-to-videos/alteryx-macro-search-for-batch-geocoding-video +v6 geocoding batch api mapbox geocoder batch human https://github.com/mapbox/mapbox-sdk-js/issues/485 +geocoding with structured input and secondary address mapbox geocoder forward|mapbox search forward human https://github.com/mapbox/mapbox-sdk-js/issues/489 +temporary versus permanent geocoding mapbox geocoder forward|mapbox geocoder batch human https://docs.mapbox.com/help/dive-deeper/understand-temporary-vs-permanent-geocoding +specific address shows incorrect location wrong zipcode mapbox feedback create|mapbox geocoder forward symptom https://github.com/mapbox/mapbox-gl-js/issues/10373 +address geocoding format guide mapbox geocoder forward|mapbox search forward human https://docs.mapbox.com/help/troubleshooting/address-geocoding-format-guide +add point of interest poi search to a map mapbox search category|mapbox search forward human https://docs.mapbox.com/help/tutorials/poi-search-react +retrieve api returning 404 mapbox places symptom https://github.com/mapbox/mapbox-gl-js/issues/13402 +filter by counties for us mapbox geocoder forward|mapbox search forward human https://github.com/mapbox/mapbox-gl-js/issues/9950 +forward geocoding doesnt return bbox property mapbox geocoder forward symptom https://github.com/mapbox/mapbox-gl-js/issues/10897 +403 forbidden for geocoding api request mapbox auth whoami|mapbox accounts list-scopes symptom https://github.com/mapbox/mapbox-gl-js/issues/8155 +city name needs to be changed mapbox feedback create human https://github.com/mapbox/mapbox-gl-js/issues/11593 +reporting data issues mapbox feedback create human https://docs.mapbox.com/help/dive-deeper/reporting-data-issues +report an error from a map search or route result mapbox feedback create human https://docs.mapbox.com/help/troubleshooting/reporting-a-problem +mapbox returns driving instead of walking directions mapbox directions symptom https://github.com/mapbox/mapbox-cli-py/issues/71 +add directions route lines to a web map mapbox directions human https://docs.mapbox.com/help/tutorials/getting-started-directions-api +calculation of distances with different values mapbox directions|mapbox matrix human https://github.com/mapbox/mapbox-gl-js/issues/10939 +generate an optimized route mapbox directions|mapbox matrix human https://docs.mapbox.com/help/tutorials/optimization-api +matrix returning null values mapbox matrix symptom https://github.com/mapbox/mapbox-sdk-js/issues/312 +add travel time isochrones to a web map mapbox isochrone human https://docs.mapbox.com/help/tutorials/get-started-isochrone-api +isochrone api polygon mapbox isochrone human https://traveltime.com/blog/mapbox-isochrone-api-polygon +map matching api always returns an http error mapbox map-match symptom https://github.com/mapbox/mapbox-sdk-js/issues/286 +need a matching full example mapbox map-match human https://github.com/mapbox/mapbox-sdk-js/issues/284 +mapmatching for multi feature inputs mapbox map-match human https://github.com/mapbox/mapbox-cli-py/issues/37 +get started with the map matching api mapbox map-match human https://docs.mapbox.com/help/tutorials/get-started-map-matching-api +what is token with styles:list scope mapbox accounts list-scopes human https://github.com/mapbox/mapbox-sdk-js/issues/281 +create a temporary token - none https://github.com/mapbox/mapbox-sdk-js/issues/291 +troubleshooting access tokens mapbox accounts list-tokens|mapbox accounts retrieve-token|mapbox auth whoami human https://docs.mapbox.com/help/troubleshooting/token-errors +receiving error 401 unauthorized mapbox auth whoami|mapbox accounts retrieve-token symptom https://github.com/mapbox/mapbox-gl-js/issues/13325 +get token details by token id mapbox accounts retrieve-token human https://github.com/mapbox/mapbox-sdk-js/issues/463 +access token missing command mapbox accounts list-tokens|mapbox accounts retrieve-token symptom https://github.com/mapbox/mapbox-cli-py/issues/82 +trouble setting access token mapbox auth login|mapbox auth whoami human https://github.com/mapbox/mapbox-cli-py/issues/69 +mapboxaccesstoken or mapbox_access_token mapbox auth whoami|mapbox doctor symptom https://github.com/mapbox/mapbox-cli-py/issues/53 +token without the right permissions gives unhelpful error mapbox accounts list-scopes|mapbox auth whoami symptom https://github.com/mapbox/tilesets-cli/issues/110 +add cli for analytics api mapbox usage human https://github.com/mapbox/mapbox-cli-py/issues/93 +manage your web map costs mapbox usage human https://docs.mapbox.com/help/troubleshooting/manage-web-map-costs +disable event tracking mapbox config set human https://github.com/mapbox/mapbox-gl-js/issues/13226 +give more guidance on config file location mapbox config list|mapbox doctor human https://github.com/mapbox/mapbox-cli-py/issues/65 +getaddrinfo enotfound api.mapbox.com mapbox doctor symptom https://github.com/mapbox/mapbox-sdk-js/issues/325 +working with firewalls mapbox doctor human https://docs.mapbox.com/help/dive-deeper/firewalls +ai assisted development with mapbox mapbox mcp install|mapbox agent-skills install human https://docs.mapbox.com/help/getting-started/ai-assisted-development +build your own agent with the mapbox mcp server mapbox mcp install|mapbox mcp list human https://docs.mapbox.com/help/tutorials/build-your-own-agent +how to regain access to your account - none https://docs.mapbox.com/help/troubleshooting/account-lockout +switch from google maps javascript api to mapbox gl js - none https://docs.mapbox.com/help/tutorials/google-to-mapbox +use mapbox gl js in a react app - none https://docs.mapbox.com/help/tutorials/use-mapbox-gl-js-with-react +add address autofill to a form on a website - none https://docs.mapbox.com/help/tutorials/add-address-autofill-to-your-website +add custom markers in mapbox gl js - none https://docs.mapbox.com/help/tutorials/custom-markers-gl-js +make a heatmap with mapbox gl js - none https://docs.mapbox.com/help/tutorials/make-a-heatmap-with-mapbox-gl-js +mapbox logo disappears on streets view - none https://github.com/mapbox/mapbox-gl-js/issues/13561 +improve the performance of mapbox gl js maps - none https://docs.mapbox.com/help/troubleshooting/mapbox-gl-js-performance +build a store locator - none https://docs.mapbox.com/help/tutorials/building-a-store-locator +how to draw arrow on line layer - none https://github.com/mapbox/mapbox-gl-js/issues/12891 diff --git a/tests/fixtures/commands_search/tuning.tsv b/tests/fixtures/commands_search/tuning.tsv new file mode 100644 index 0000000..e9a40db --- /dev/null +++ b/tests/fixtures/commands_search/tuning.tsv @@ -0,0 +1,104 @@ +# Written by hand while building the search, and tuned against: the scoring and +# the help text were changed until these ranked well, so this set flatters it. +# Columns: query, expected first result (`|` between acceptable answers, `-` +# for none), kind. +how do I log in to mapbox mapbox auth login human +sign in mapbox auth login synonym +log out of my account mapbox auth logout human +remove my saved credentials mapbox auth logout human +refresh my access token mapbox auth refresh human +who am I logged in as mapbox auth whoami human +which token will the next command use mapbox auth whoami agent +show stored credential profiles mapbox auth profiles human +list my access tokens mapbox accounts list-tokens human +what scopes can a token have mapbox accounts list-scopes human +token permissions mapbox accounts list-scopes synonym +get details of one token mapbox accounts retrieve-token human +create a new map style mapbox styles create human +list all styles in my account mapbox styles list human +get the json of a style mapbox styles get agent +update an existing style mapbox styles update human +delete a style mapbox styles delete human +download a style as a zip mapbox styles download human +back up my style offline mapbox styles download synonym +edit the draft of a style mapbox styles draft update human +get a style draft mapbox styles draft get agent +discard a style draft mapbox styles draft delete synonym +rename my style mapbox styles update synonym +find styles I deleted mapbox styles list human +add an icon to a sprite mapbox sprites upload human +upload many images to a sprite at once mapbox sprites upload-batch human +remove an image from the sprite mapbox sprites delete human +delete several sprite images mapbox sprites delete-batch human +get sprite json mapbox sprites get-json agent +what icons are in my sprite mapbox sprites get-json human +upload a custom font mapbox fonts upload human +list fonts mapbox fonts list agent +delete a font mapbox fonts delete agent +add a ttf font mapbox fonts upload human +which fonts do I have mapbox fonts list human +get a vector tile mapbox tilesets get-mvt agent +fetch a raster tile from a tileset mapbox tilesets get-tile agent +query features near a coordinate mapbox tilesets query agent +what features are at this point mapbox tilesets query human +upload a tileset mapbox tilesets-cli human +create a tileset from geojson mapbox tilesets-cli human +download mvt mapbox tilesets get-mvt agent +render a static map image mapbox static get-image agent +make a png of a map mapbox static get-image human +static raster tile mapbox static get-tile agent +map picture for an email mapbox static get-image synonym +thumbnail of my style mapbox static get-image synonym +search for an address mapbox search forward|mapbox geocoder forward human +find a poi by name mapbox search forward agent +what is at this coordinate mapbox search reverse|mapbox geocoder reverse|mapbox tilesets query human +find restaurants nearby mapbox search category human +list poi categories mapbox search list-category agent +category ids for search mapbox search list-category agent +get details for a place mapbox places human +look up a place by its id mapbox places agent +geocode an address mapbox geocoder forward|mapbox search forward agent +reverse geocode a coordinate mapbox geocoder reverse|mapbox search reverse agent +batch geocode many addresses mapbox geocoder batch agent +turn coordinates into an address mapbox geocoder reverse|mapbox search reverse human +convert an address to lat long mapbox geocoder forward|mapbox search forward human +directions from a to b mapbox directions human +driving route mapbox directions agent +walking directions between two points mapbox directions human +travel time between many locations mapbox matrix human +distance matrix mapbox matrix agent +area reachable in 15 minutes mapbox isochrone human +drive time polygon mapbox isochrone synonym +snap gps points to roads mapbox map-match human +match a gps trace mapbox map-match agent +clean up a noisy gps track mapbox map-match synonym +report a wrong address on the map mapbox feedback create human +list data feedback I submitted mapbox feedback list human +get a feedback item mapbox feedback get agent +send feedback about the cli mapbox report human +file a bug about this tool mapbox report synonym +install an mcp server mapbox mcp install agent +list mcp servers mapbox mcp list agent +set up mapbox for claude code mapbox mcp install|mapbox agent-skills install human +install agent skills mapbox agent-skills install agent +update installed skills mapbox agent-skills update agent +remove skills mapbox agent-skills uninstall human +list available skills mapbox agent-skills list agent +write a skill describing this cli mapbox generate-skills agent +shell completion for zsh mapbox completion human +enable tab completion mapbox completion human +uninstall the cli mapbox uninstall human +turn off telemetry mapbox config set human +show current settings mapbox config list human +reset a setting to its default mapbox config unset human +show recent commands I ran mapbox history list human +details of the last run mapbox history show human +check my proxy settings mapbox doctor human +how much have I used this month mapbox usage human +api usage by day mapbox usage agent +uplaod a font mapbox fonts upload typo +direcitons mapbox directions typo +isochorne mapbox isochrone typo +dleete a style mapbox styles delete typo +bake a cake - none +deploy to kubernetes - none diff --git a/tests/output_contract.rs b/tests/output_contract.rs index 22e66c2..8f25d72 100644 --- a/tests/output_contract.rs +++ b/tests/output_contract.rs @@ -443,12 +443,13 @@ fn help_names_env_variables_without_their_values() { assert!(!text.contains("=]"), "an env note still shows `=`: {text}"); } -const AGENT_HINT: &str = "Agents: `mapbox --schema`"; +const AGENT_HINT: &str = "Agents: `mapbox commands search"; -/// An agent reading top-level help is pointed at `--schema` rather than left -/// to chain `--help` calls. Only there: subcommand help stays as it was. +/// An agent reading top-level help is pointed at `commands search` and +/// `--schema` rather than left to chain `--help` calls. Only there: +/// subcommand help stays as it was. #[test] -fn an_agent_is_pointed_at_the_schema() { +fn an_agent_is_pointed_at_search_and_the_schema() { let out = command() .env("CLAUDECODE", "1") .arg("--help") @@ -456,6 +457,11 @@ fn an_agent_is_pointed_at_the_schema() { .expect("run mapbox"); assert!(out.status.success()); assert!(stdout(&out).contains(AGENT_HINT), "{}", stdout(&out)); + assert!( + stdout(&out).contains("`mapbox --schema`"), + "{}", + stdout(&out) + ); let sub = command() .env("CLAUDECODE", "1") @@ -1294,3 +1300,216 @@ fn text_into_a_pipe_carries_no_color_and_no_banner() { } } } + +/// `commands search` answers on stdout under `json`: the matches as +/// `[{command, summary, score}]`, best first. A match leaves stderr empty; +/// no match adds a hint there (below). +#[test] +fn commands_search_prints_its_matches_as_json() { + let out = run(&["-o", "json", "commands", "search", "upload", "a", "font"]); + assert!(out.status.success(), "{}", stderr(&out)); + assert_eq!(stderr(&out), ""); + let matches = json(&stdout(&out)); + let matches = matches.as_array().expect("an array"); + assert_eq!(matches[0]["command"], "mapbox fonts upload"); + assert!(matches.len() <= 5, "{matches:?}"); + for entry in matches { + let keys: Vec<&String> = entry.as_object().expect("an object").keys().collect(); + assert_eq!(keys, ["command", "score", "summary"], "{entry}"); + } + let scores: Vec = matches + .iter() + .map(|entry| entry["score"].as_f64().expect("a number")) + .collect(); + assert_eq!(scores[0], 1.0, "{scores:?}"); + assert!( + scores.windows(2).all(|pair| pair[0] >= pair[1]) && scores.iter().all(|s| *s > 0.0), + "{scores:?}" + ); +} + +/// An empty query is most likely a script's empty variable, so it fails as +/// a usage error rather than answering like a query that found nothing. +#[test] +fn commands_search_refuses_a_query_with_no_words() { + for query in ["", " ", "?!"] { + let out = run(&["-o", "json", "commands", "search", query]); + assert_eq!(out.status.code(), Some(2), "{query:?}: {}", stderr(&out)); + assert_eq!(json(&stderr(&out))["code"], "usage", "{query:?}"); + assert_eq!(stdout(&out), "", "{query:?}"); + } +} + +/// The check is on the whole query: an empty variable beside real words +/// still leaves something to search for. +#[test] +fn commands_search_accepts_an_empty_part_beside_words() { + let out = run(&[ + "-o", "json", "commands", "search", "upload", "a", "font", "", "?", + ]); + assert!(out.status.success(), "{}", stderr(&out)); + assert_eq!(json(&stdout(&out))[0]["command"], "mapbox fonts upload"); +} + +/// A person gets a table on stdout and the next step on stderr, so +/// `> file` keeps the table alone. +#[test] +fn commands_search_prints_a_table_and_a_tip_as_text() { + let out = run(&["-o", "text", "commands", "search", "upload a font"]); + assert!(out.status.success(), "{}", stderr(&out)); + let text = stdout(&out); + assert!(text.starts_with("COMMAND"), "{text}"); + assert!(text.contains("mapbox fonts upload"), "{text}"); + assert!(stderr(&out).contains("Tip:"), "{}", stderr(&out)); +} + +/// No match is an empty answer, not a failure: the query was fine, the CLI +/// just has nothing for it. +#[test] +fn commands_search_with_no_match_is_an_empty_list() { + let out = run(&["-o", "json", "commands", "search", "xyzzy"]); + assert!(out.status.success(), "{}", stderr(&out)); + assert_eq!(json(&stdout(&out)), serde_json::json!([])); +} + +/// Under `text` the empty answer is no output at all, not a blank line, so +/// `> file` leaves an empty file. Where to look instead goes to stderr, and +/// the `--help` tip, which is about a match, does not. +#[test] +fn commands_search_with_no_match_prints_nothing_as_text() { + let out = run(&["-o", "text", "commands", "search", "xyzzy"]); + assert!(out.status.success(), "{}", stderr(&out)); + assert_eq!(stdout(&out), ""); + assert!( + stderr(&out).contains("mapbox commands list"), + "{}", + stderr(&out) + ); + assert!(!stderr(&out).contains("Tip:"), "{}", stderr(&out)); +} + +/// `--limit` caps the matches at its value, five by default, and `0` +/// lifts the cap. "style" matches more than five commands. +#[test] +fn commands_search_limit_caps_the_matches() { + let count = |extra: &[&str]| { + let mut args = vec!["-o", "json", "commands", "search", "style"]; + args.extend_from_slice(extra); + let out = run(&args); + assert!(out.status.success(), "{}", stderr(&out)); + json(&stdout(&out)).as_array().expect("an array").len() + }; + assert_eq!(count(&[]), 5); + assert_eq!(count(&["--limit", "1"]), 1); + assert!(count(&["--limit", "0"]) > 5); +} + +#[test] +fn commands_list_names_every_command_once() { + let out = run(&["-o", "json", "commands", "list"]); + assert!(out.status.success(), "{}", stderr(&out)); + let listed = json(&stdout(&out)); + let listed: Vec<&str> = listed + .as_array() + .expect("an array") + .iter() + .map(|entry| entry["command"].as_str().expect("a command")) + .collect(); + let schema = json(&stdout(&run(&["--schema"]))); + let mut described: Vec<&str> = schema["commands"] + .as_array() + .expect("commands") + .iter() + .map(|entry| entry["command"].as_str().expect("a command")) + .collect(); + described.sort_unstable(); + assert_eq!(listed, described); + let mut unique = listed.clone(); + unique.dedup(); + assert_eq!(unique, listed, "a command is listed twice"); +} + +/// A command name nothing resembles is pointed at `commands search`, in +/// both modes. One clap has a near spelling for keeps clap's suggestion, +/// which is the better answer. +#[test] +fn an_unknown_command_is_pointed_at_search() { + let out = run(&["-o", "json", "upload-font"]); + assert_eq!(out.status.code(), Some(2)); + let error = json(&stderr(&out)); + assert!( + error["fix"] + .as_str() + .is_some_and(|fix| fix.contains("mapbox commands search")), + "{error}" + ); + + let out = run(&["-o", "text", "upload-font"]); + assert!( + stderr(&out).contains("mapbox commands search"), + "{}", + stderr(&out) + ); + + let out = run(&["-o", "json", "styles", "lst"]); + let error = json(&stderr(&out)); + assert!( + error["fix"] + .as_str() + .is_some_and(|fix| fix.contains("'list'")), + "{error}" + ); + assert!( + !error["fix"] + .as_str() + .is_some_and(|fix| fix.contains("commands search")), + "{error}" + ); + + let out = run(&["-o", "text", "styles", "lst"]); + assert!(stderr(&out).contains("'list'"), "{}", stderr(&out)); + assert!( + !stderr(&out).contains("commands search"), + "{}", + stderr(&out) + ); +} + +/// The closing line of top-level help points a person at `commands search` +/// too, not only an agent reading the hint above it. +#[test] +fn top_level_help_points_a_person_at_search() { + let out = run(&["--help"]); + assert!(out.status.success()); + let text = stdout(&out); + assert!( + text.lines() + .any(|line| line.starts_with("Run mapbox commands search \"\"")), + "{text}" + ); +} + +/// Each command's row in top-level help fits one line at 80 columns: a +/// summary that wraps reads as two rows at a glance. The options may wrap, +/// since their environment notes are long by nature. +#[test] +fn every_command_fits_one_help_line_at_80_columns() { + let out = command() + .env("COLUMNS", "80") + .env("CLAUDECODE", "1") + .arg("--help") + .output() + .expect("run mapbox"); + let text = stdout(&out); + let (hint, page) = text.split_at(text.find("Usage:").expect("a usage line")); + for line in hint.lines() { + assert!(line.chars().count() <= 80, "over 80 columns: {line:?}"); + } + let commands = &page[..page.find("Authentication:").expect("an options section")]; + for line in commands.lines() { + assert!( + !line.starts_with(" "), + "a command's summary wraps: {line:?}\n{commands}" + ); + } +} diff --git a/tests/telemetry_events.rs b/tests/telemetry_events.rs index 8ce38a9..4cdd528 100644 --- a/tests/telemetry_events.rs +++ b/tests/telemetry_events.rs @@ -1276,3 +1276,35 @@ fn an_unexplained_refusal_starts_a_cooldown_too() { assert_eq!(state["rejected"], false); assert!(state["cooldowns"]["telemetry"]["until"].is_u64(), "{state}"); } + +/// A search query is whatever the caller typed — a customer's name, a +/// token pasted by mistake — so it leaves as its length and never as text. +#[test] +fn a_search_query_is_sent_as_its_length() { + let home = scratch("commands-search"); + let (out, event) = run_sent( + &home, + &[ + "-o", + "json", + "commands", + "search", + "secret-customer-name", + "upload", + ], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + assert_eq!(event["command"], serde_json::json!(["commands", "search"])); + assert!( + !event.to_string().contains("secret-customer-name"), + "the query's text was sent: {event}" + ); + assert_eq!( + event["params"], + serde_json::json!([{ "name": "query", "length": "secret-customer-name upload".len() }]) + ); +}