From ba0f1324a13c62ff4ff0e207370b7e107ba9eb94 Mon Sep 17 00:00:00 2001 From: Mofei Zhu Date: Tue, 29 Sep 2026 15:42:28 +0300 Subject: [PATCH 01/33] Add mapbox workflow: install and run multi-step workflows (beta) A workflow is a workflow.yaml of command and script steps, installed from a local directory or a GitHub repository's workflow/// into ~/.mapbox/workflows/, then run by name. Command steps run this binary again with --output json, so they resolve credentials, timeouts and path encoding exactly as a typed command does. The first published workflow is workflow/beta/copy-style, which copies a style between accounts using two credential profiles. --- CHANGELOG.md | 10 + docs/commands.md | 208 +++++++ src/main.rs | 15 + src/schema.rs | 8 + src/workflow/definition.rs | 598 ++++++++++++++++++ src/workflow/mod.rs | 494 +++++++++++++++ src/workflow/runner.rs | 601 ++++++++++++++++++ src/workflow/store.rs | 644 ++++++++++++++++++++ src/workflow/template.rs | 351 +++++++++++ tests/source_guards.rs | 10 +- tests/workflow.rs | 295 +++++++++ workflow/README.md | 94 +++ workflow/beta/copy-style/README.md | 21 + workflow/beta/copy-style/scripts/prepare.py | 67 ++ workflow/beta/copy-style/workflow.yaml | 65 ++ 15 files changed, 3480 insertions(+), 1 deletion(-) create mode 100644 src/workflow/definition.rs create mode 100644 src/workflow/mod.rs create mode 100644 src/workflow/runner.rs create mode 100644 src/workflow/store.rs create mode 100644 src/workflow/template.rs create mode 100644 tests/workflow.rs create mode 100644 workflow/README.md create mode 100644 workflow/beta/copy-style/README.md create mode 100644 workflow/beta/copy-style/scripts/prepare.py create mode 100644 workflow/beta/copy-style/workflow.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index 11c1878..f9d037f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,16 @@ that may never merge. They are not releases and are not listed here. ## Unreleased +- `mapbox workflow`, beta: install and run workflows, which are named, + multi-step recipes of `mapbox` commands and scripts defined in a + `workflow.yaml`. `install` copies one from a local directory or from a + GitHub repository's `workflow///` (by default `mapbox/cli`; + `GH_TOKEN`/`GITHUB_TOKEN` for a private one) into `~/.mapbox/workflows/`, + and `list`, `show`, `run` and `uninstall` work on what is installed. The + first one published is `copy-style`, which copies a style between + accounts. The command and the `version: 1` format are beta and may change. + Nothing changes for a script that does not use them. + - New `mapbox report ""` sends feedback about the CLI itself to the Mapbox CLI team, with the CLI's version and OS appended. It is filed under Mapbox's account, not yours. It is sent with the CLI's own token, diff --git a/docs/commands.md b/docs/commands.md index be5790b..1fc0936 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -138,6 +138,12 @@ nests, and is typed `mapbox styles draft get`. **[Usage](#usage)** — [usage](#mapbox-usage) +**[Workflows](#workflows)** — [workflow.list](#mapbox-workflow-list) · +[workflow.show](#mapbox-workflow-show) · +[workflow.install](#mapbox-workflow-install) · +[workflow.uninstall](#mapbox-workflow-uninstall) · +[workflow.run](#mapbox-workflow-run) + **[Tilesets CLI](#tilesets-cli)** — [tilesets-cli](#mapbox-tilesets-cli-args) Then [Errors](#errors) — the shape a failure takes in each mode. @@ -4966,6 +4972,208 @@ list drops the `--daily` suggestion once it's already in effect. --- +## Workflows + +A workflow is a named, multi-step recipe of `mapbox` commands and scripts, +defined in a `workflow.yaml`. **Beta:** the command and the `version: 1` +format may change. The format and the rules a workflow directory follows are +in [workflow/README.md](../workflow/README.md). + +None ships inside the binary. `install` copies one into +`~/.mapbox/workflows//`, and every other subcommand works on what is +installed there. A `beta` workflow says so on stderr when it is installed or +run. + +--- + +### `mapbox workflow list` + +Lists the installed workflows. One that no longer loads, because a file was +edited by hand or this CLI no longer reads its format, is listed with its +error instead of a summary. + +#### Outputs + + + + +
textjson
+ +``` +copy-style (beta) + Copy a style from one account to another +``` + + + +```json +[ + { + "name": "copy-style", + "path": "/Users/me/.mapbox/workflows/copy-style", + "source": "github:mapbox/cli@main", + "stage": "beta", + "summary": "Copy a style from one account to another" + } +] +``` + +
+ +--- + +### `mapbox workflow show` + +Describes an installed workflow: its inputs, with their types and defaults, +and its steps. When a step names a command or an argument this build no +longer has, the problems are listed at the end and under `problems`. + +#### Parameters + +| Parameter | Effect | +| --- | --- | +| `NAME` | Name of an installed workflow. | + +--- + +### `mapbox workflow install` + +Installs a workflow from GitHub or from a local directory. Everything is read +and checked before anything is written: the layout, the `workflow.yaml`, and +every command step against this build's commands. A workflow that fails any +of these is `invalid_workflow`, with every problem listed at once. + +#### Parameters + +| Parameter | Effect | +| --- | --- | +| `SOURCE` | A workflow name, looked up in the repository's `workflow///`, or a path to a local workflow directory: anything with a `/` or starting with `.`. | +| `--repo` | GitHub repository to install from, as `OWNER/REPO`. Defaults to `mapbox/cli`. | +| `--ref` | Branch, tag or commit to install from. Defaults to `main`. | +| `--force` | Replace a workflow that is already installed. | +| `--dry-run` | Check the workflow and list the files it would write, then exit. | + +The repository is read as one tarball through the GitHub API. For a private +repository, such as `mapbox/cli` today, set `GH_TOKEN` or `GITHUB_TOKEN`. It is +sent only to `api.github.com`. Without one, a private repository answers 404, +exactly as a ref that does not exist does, and the error says both. + +**An installed workflow stops the install** unless `--force` is given. The new +copy is staged and renamed into place, so an interrupted install leaves the +old copy or the new one. Only regular files are taken. A symlink in a local +directory is refused, and an archive entry whose path would leave the +workflow's directory stops the read. + +#### Examples + +```sh +export GITHUB_TOKEN="$(gh auth token)" +mapbox workflow install copy-style + +mapbox workflow install copy-style --ref v0.4.0 + +mapbox workflow install ./workflow/beta/copy-style --force +``` + +#### Outputs + + + + +
textjson
+ +``` +Installed `copy-style` from +/Users/me/dev/cli/workflow/beta/copy-style +into /Users/me/.mapbox/workflows/copy-style. +Run it with `mapbox workflow run copy-style`. +``` + + + +```json +{ + "dry_run": false, + "files": [ + "README.md", + "scripts/prepare.py", + "workflow.yaml" + ], + "name": "copy-style", + "path": "/Users/me/.mapbox/workflows/copy-style", + "source": "/Users/me/dev/cli/workflow/beta/copy-style", + "stage": "beta" +} +``` + +
+ +--- + +### `mapbox workflow uninstall` + +Removes an installed workflow's directory. At a terminal it asks first, and +`--yes` skips the question. Only a plain workflow name is accepted, and only a +directory `install` wrote is removed. + +#### Parameters + +| Parameter | Effect | +| --- | --- | +| `NAME` | Name of an installed workflow. | +| `--dry-run` | Say what it would remove, then exit without removing it. | + +--- + +### `mapbox workflow run` + +Runs an installed workflow's steps in order. The first step that fails stops +the run with `workflow_failed`, naming the step, and exit code 1. + +#### Parameters + +| Parameter | Effect | +| --- | --- | +| `NAME` | Name of an installed workflow. | +| `--input`, `-i` | `KEY=VALUE` for one of the workflow's inputs, repeatable. The value is read as the input's declared type. | +| `--dry-run` | Check the workflow and its inputs and print the plan, then exit without running a step. | + +**stdout holds only the result**: the workflow's `outputs`, or the last +step's output if it declares none, rendered like any other result. Each +step's progress line and anything a step writes to stderr go to stderr. + +Each command step is this binary run again with `--output json`, so it +resolves its token and applies its timeouts as the same command typed by +hand would, and appears in `mapbox history` as its own run. The globals given +to `workflow run` (`--profile`, `--username`, `--use-login`, `--timeout`, +`--yes`, `--debug`, `--token`) reach every command step that does not set its +own. A token typed as `--token` goes to the step's environment, not its +command line. + +#### Examples + +```sh +mapbox workflow run copy-style \ + --input style_id=cmm28c5rm00bj01qz9hwp69qc \ + --input from_profile=source \ + --input to_profile=target + +mapbox workflow run copy-style -i style_id=cmm28c5rm00bj01qz9hwp69qc \ + -i from_profile=source -i to_profile=target --dry-run +``` + +`--dry-run` prints the plan: the inputs as they were read, and each step with +its arguments as written, since nothing has run to fill them in. + +``` +Would run `copy-style`: + 1. source mapbox styles get + 2. body python3 scripts/prepare.py + 3. created mapbox styles create +``` + +--- + ## Tilesets CLI ### `mapbox tilesets-cli ` diff --git a/src/main.rs b/src/main.rs index 4c1823e..8131742 100644 --- a/src/main.rs +++ b/src/main.rs @@ -54,6 +54,7 @@ mod telemetry_sink; mod tilesets_cli; mod uninstall; mod update_check; +mod workflow; use output::{CliError, Mode}; use remedy::Remedy; @@ -793,6 +794,7 @@ fn build_app(specs: &[ServiceSpec]) -> Command { app = app.subcommand(account_usage::command()); + app = app.subcommand(workflow::command()); app.subcommand(tilesets_cli::command()) } @@ -1395,6 +1397,19 @@ fn run(app: &Command, specs: &[ServiceSpec], matches: &ArgMatches, mode: Mode) - Some(("show", show_matches)) => history::show(show_matches, mode)?, _ => unreachable!("`history` sets subcommand_required(true)"), }, + // Ahead of the generic service arm too. Its own requests go to + // GitHub, and each command step is a child `mapbox` that resolves + // its credentials itself — so nothing is loaded here. + Some((workflow::COMMAND, workflow_matches)) => workflow::run( + app, + workflow_matches, + workflow::RunFlags { + globals: matches, + debug, + assume_yes, + }, + mode, + )?, // Also ahead of the generic service arm: read-only except for the // opt-in `--verify` request, and needs no credential load of its own // — it reports what one would resolve to, not what a fresh one diff --git a/src/schema.rs b/src/schema.rs index 2f8387d..69ee671 100644 --- a/src/schema.rs +++ b/src/schema.rs @@ -394,6 +394,14 @@ fn commands(app: &Command, specs: &[ServiceSpec], path: &[String]) -> Vec>; + +/// A checked workflow. Nothing constructs one except [`parse`]. +#[derive(Debug, Clone)] +pub struct Workflow { + pub name: String, + pub summary: String, + pub description: Option, + pub inputs: BTreeMap, + pub steps: Vec, + pub outputs: Option, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Input { + #[serde(rename = "type")] + pub kind: InputType, + #[serde(default)] + pub description: Option, + #[serde(default)] + pub required: bool, + #[serde(default)] + pub default: Option, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum InputType { + String, + Number, + Boolean, +} + +impl InputType { + pub fn as_str(self) -> &'static str { + match self { + InputType::String => "string", + InputType::Number => "number", + InputType::Boolean => "boolean", + } + } + + pub fn accepts(self, value: &Value) -> bool { + matches!( + (self, value), + (InputType::String, Value::String(_)) + | (InputType::Number, Value::Number(_)) + | (InputType::Boolean, Value::Bool(_)) + ) + } +} + +#[derive(Debug, Clone)] +pub struct Step { + pub id: String, + pub name: Option, + pub action: Action, + pub stdin: Option, +} + +#[derive(Debug, Clone)] +pub enum Action { + Command { + /// `styles get` as `["styles", "get"]`. + path: Vec, + args: Map, + }, + Script { + /// Relative to `scripts/`. + script: PathBuf, + interpreter: String, + args: Vec, + }, +} + +impl Step { + /// How progress and plans name the step. + pub fn label(&self) -> String { + match &self.action { + Action::Command { path, .. } => format!("mapbox {}", path.join(" ")), + Action::Script { + script, + interpreter, + .. + } => format!("{interpreter} {SCRIPTS_DIR}/{}", slash_path(script)), + } + } +} + +/// The file as written, before anything is checked. +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct Raw { + version: u64, + name: String, + summary: String, + #[serde(default)] + description: Option, + #[serde(default)] + inputs: BTreeMap, + steps: Vec, + #[serde(default)] + outputs: Option, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct RawStep { + id: String, + #[serde(default)] + name: Option, + #[serde(default)] + command: Option, + #[serde(default)] + script: Option, + #[serde(default)] + interpreter: Option, + #[serde(default)] + args: Option, + #[serde(default)] + stdin: Option, +} + +/// A workflow name is one directory name, lower-case and dash-separated — +/// `uninstall` turns it into a path it deletes, so nothing else may pass. +pub fn is_workflow_name(name: &str) -> bool { + !name.is_empty() + && name.len() <= 64 + && name.starts_with(|c: char| c.is_ascii_lowercase() || c.is_ascii_digit()) + && name + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-') +} + +/// Input names and step ids, which expressions refer to. +fn is_identifier(name: &str) -> bool { + name.starts_with(|c: char| c.is_ascii_lowercase()) + && name + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '_') +} + +/// A path that stays inside the directory it is relative to. +pub fn is_contained(path: &Path) -> bool { + !path.as_os_str().is_empty() + && path + .components() + .all(|component| matches!(component, Component::Normal(_))) +} + +pub fn slash_path(path: &Path) -> String { + path.components() + .map(|part| part.as_os_str().to_string_lossy()) + .collect::>() + .join("/") +} + +/// The interpreter a script runs under when the step names none. A short, +/// fixed list: guessing further is how a script ends up run by the wrong +/// program. +fn default_interpreter(script: &Path) -> Option<&'static str> { + match script.extension()?.to_str()? { + "sh" => Some("sh"), + "py" => Some("python3"), + "js" | "mjs" => Some("node"), + _ => None, + } +} + +/// Checks `files` as the workflow called `name`, and returns it or every +/// problem found — all of them at once, so fixing a workflow is not one +/// install attempt per mistake. +pub fn parse(name: &str, files: &Files) -> Result> { + let mut problems = vec![]; + + for path in files.keys() { + let allowed = path == Path::new(DEFINITION_FILE) + || path == Path::new(README_FILE) + || path.starts_with(SCRIPTS_DIR); + if !allowed { + problems.push(format!( + "`{}` does not belong in a workflow: only {DEFINITION_FILE}, {README_FILE} \ + and {SCRIPTS_DIR}/ do", + slash_path(path) + )); + } + } + + let Some(bytes) = files.get(Path::new(DEFINITION_FILE)) else { + problems.push(format!("there is no {DEFINITION_FILE}")); + return Err(problems); + }; + let raw: Raw = match serde_yaml::from_slice(bytes) { + Ok(raw) => raw, + Err(e) => { + problems.push(format!("{DEFINITION_FILE} is not valid: {e}")); + return Err(problems); + } + }; + + if raw.version != SCHEMA_VERSION { + problems.push(format!( + "`version: {}` is not one this CLI reads; it reads `version: {SCHEMA_VERSION}`", + raw.version + )); + } + if raw.name != name { + problems.push(format!( + "`name: {}` must match the directory it is in, `{name}`", + raw.name + )); + } + if !is_workflow_name(&raw.name) { + problems.push(format!( + "`name: {}` must be lower-case letters, digits and dashes", + raw.name + )); + } + if raw.summary.trim().is_empty() || raw.summary.contains('\n') { + problems.push("`summary` must be one non-empty line".to_string()); + } + + for (input, spec) in &raw.inputs { + if !is_identifier(input) { + problems.push(format!( + "input `{input}` must be lower-case letters, digits and underscores" + )); + } + if let Some(default) = &spec.default { + if spec.required { + problems.push(format!( + "input `{input}` is required and has a default; it can only be one" + )); + } + if !spec.kind.accepts(default) { + problems.push(format!( + "input `{input}`'s default is not a {}", + spec.kind.as_str() + )); + } + } + } + + if raw.steps.is_empty() { + problems.push("`steps` is empty".to_string()); + } + + let mut steps = vec![]; + let mut seen: BTreeSet = BTreeSet::new(); + let mut referenced_scripts: BTreeSet = BTreeSet::new(); + for raw_step in raw.steps { + let id = raw_step.id.clone(); + if !is_identifier(&id) { + problems.push(format!( + "step `{id}`: the id must be lower-case letters, digits and underscores" + )); + } + if seen.contains(&id) { + problems.push(format!("step `{id}` appears twice")); + } + + for value in raw_step.args.iter().chain(raw_step.stdin.iter()) { + check_references( + value, + &format!("step `{id}`"), + &raw.inputs, + &seen, + &mut problems, + ); + } + + let action = match (raw_step.command, raw_step.script) { + (Some(command), None) => { + if raw_step.interpreter.is_some() { + problems.push(format!( + "step `{id}`: `interpreter` only applies to a `script` step" + )); + } + let path: Vec = command.split_whitespace().map(String::from).collect(); + if path.is_empty() { + problems.push(format!("step `{id}`: `command` is empty")); + } + let args = match raw_step.args { + None => Map::new(), + Some(Value::Object(args)) => args, + Some(_) => { + problems.push(format!( + "step `{id}`: a command's `args` is a mapping of argument name to value" + )); + Map::new() + } + }; + Some(Action::Command { path, args }) + } + (None, Some(script)) => { + let script = PathBuf::from(script); + let under_scripts = Path::new(SCRIPTS_DIR).join(&script); + if !is_contained(&script) { + problems.push(format!( + "step `{id}`: `script: {}` must be a path inside {SCRIPTS_DIR}/", + script.display() + )); + } else if !files.contains_key(&under_scripts) { + problems.push(format!( + "step `{id}`: there is no {SCRIPTS_DIR}/{}", + slash_path(&script) + )); + } + referenced_scripts.insert(under_scripts); + + let interpreter = match raw_step + .interpreter + .or_else(|| default_interpreter(&script).map(String::from)) + { + Some(interpreter) + if !interpreter.is_empty() + && !interpreter.contains(char::is_whitespace) => + { + interpreter + } + Some(interpreter) => { + problems.push(format!( + "step `{id}`: `interpreter: {interpreter}` must be one program name" + )); + String::new() + } + None => { + problems.push(format!( + "step `{id}`: name an `interpreter` for {}; only .sh, .py and .js \ + have a default", + slash_path(&script) + )); + String::new() + } + }; + let args = match raw_step.args { + None => vec![], + Some(Value::Array(args)) => args, + Some(_) => { + problems.push(format!( + "step `{id}`: a script's `args` is a list of values" + )); + vec![] + } + }; + Some(Action::Script { + script, + interpreter, + args, + }) + } + (Some(_), Some(_)) => { + problems.push(format!( + "step `{id}` has both `command` and `script`; a step is one or the other" + )); + None + } + (None, None) => { + problems.push(format!("step `{id}` needs a `command` or a `script`")); + None + } + }; + + seen.insert(id.clone()); + if let Some(action) = action { + steps.push(Step { + id, + name: raw_step.name, + action, + stdin: raw_step.stdin, + }); + } + } + + if let Some(outputs) = &raw.outputs { + check_references(outputs, "`outputs`", &raw.inputs, &seen, &mut problems); + } + + for path in files.keys().filter(|path| path.starts_with(SCRIPTS_DIR)) { + if !referenced_scripts.contains(path) { + problems.push(format!( + "`{}` is not used by any step; a workflow ships only the scripts it runs", + slash_path(path) + )); + } + } + + if !problems.is_empty() { + return Err(problems); + } + Ok(Workflow { + name: raw.name, + summary: raw.summary, + description: raw.description, + inputs: raw.inputs, + steps, + outputs: raw.outputs, + }) +} + +/// Every expression in `value` must name a declared input or a step that +/// has already run by the time `value` is read. +fn check_references( + value: &Value, + owner: &str, + inputs: &BTreeMap, + earlier_steps: &BTreeSet, + problems: &mut Vec, +) { + let references = match template::references(value) { + Ok(references) => references, + Err(e) => { + problems.push(format!("{owner}: {e}")); + return; + } + }; + for reference in references { + match &reference { + Reference::Input(name) if !inputs.contains_key(name) => problems.push(format!( + "{owner}: `{}` names no declared input", + template::display(&reference) + )), + Reference::Step { id, .. } if !earlier_steps.contains(id) => problems.push(format!( + "{owner}: `{}` names no earlier step", + template::display(&reference) + )), + _ => {} + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const MINIMAL: &str = "\ +version: 1 +name: demo +summary: A demo +inputs: + style_id: { type: string, required: true } +steps: + - id: fetch + command: styles get + args: { style-id: '${{ inputs.style_id }}' } + - id: shape + script: shape.py + stdin: '${{ steps.fetch.output }}' +outputs: + id: '${{ steps.shape.output.id }}' +"; + + fn files(entries: &[(&str, &str)]) -> Files { + entries + .iter() + .map(|(path, body)| (PathBuf::from(path), body.as_bytes().to_vec())) + .collect() + } + + fn problems(entries: &[(&str, &str)]) -> Vec { + parse("demo", &files(entries)).expect_err("should be refused") + } + + #[test] + fn a_minimal_workflow_parses() { + let workflow = parse( + "demo", + &files(&[ + (DEFINITION_FILE, MINIMAL), + ("scripts/shape.py", ""), + (README_FILE, ""), + ]), + ) + .unwrap(); + assert_eq!(workflow.steps.len(), 2); + assert_eq!(workflow.steps[0].label(), "mapbox styles get"); + assert_eq!(workflow.steps[1].label(), "python3 scripts/shape.py"); + } + + #[test] + fn the_name_must_match_the_directory() { + let found = parse( + "other", + &files(&[(DEFINITION_FILE, MINIMAL), ("scripts/shape.py", "")]), + ) + .unwrap_err(); + assert!(found.iter().any(|p| p.contains("must match")), "{found:?}"); + } + + #[test] + fn stray_and_unused_files_are_refused() { + let found = problems(&[ + (DEFINITION_FILE, MINIMAL), + ("scripts/shape.py", ""), + ("scripts/old.py", ""), + ("notes.txt", ""), + ]); + assert!( + found.iter().any(|p| p.contains("scripts/old.py")), + "{found:?}" + ); + assert!(found.iter().any(|p| p.contains("notes.txt")), "{found:?}"); + } + + #[test] + fn a_missing_script_is_refused() { + let found = problems(&[(DEFINITION_FILE, MINIMAL)]); + assert!( + found.iter().any(|p| p.contains("no scripts/shape.py")), + "{found:?}" + ); + } + + #[test] + fn a_script_path_cannot_leave_scripts() { + let yaml = MINIMAL.replace("script: shape.py", "script: ../workflow.yaml"); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found.iter().any(|p| p.contains("inside scripts/")), + "{found:?}" + ); + } + + #[test] + fn a_reference_must_point_backwards() { + let yaml = MINIMAL.replace( + "args: { style-id: '${{ inputs.style_id }}' }", + "args: { style-id: '${{ steps.shape.output.id }}' }", + ); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found.iter().any(|p| p.contains("no earlier step")), + "{found:?}" + ); + } + + #[test] + fn an_undeclared_input_is_refused() { + let yaml = MINIMAL.replace("inputs.style_id }}'", "inputs.nope }}'"); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found.iter().any(|p| p.contains("no declared input")), + "{found:?}" + ); + } + + #[test] + fn an_unknown_field_is_refused() { + let yaml = MINIMAL.replace("summary: A demo", "summary: A demo\nretries: 3"); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!(found.iter().any(|p| p.contains("retries")), "{found:?}"); + } + + #[test] + fn an_unknown_extension_needs_an_interpreter() { + let yaml = MINIMAL.replace("shape.py", "shape.rb"); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.rb", "")]); + assert!(found.iter().any(|p| p.contains("interpreter")), "{found:?}"); + } + + #[test] + fn workflow_names_are_one_plain_directory_name() { + for good in ["copy-style", "a", "v2-sync"] { + assert!(is_workflow_name(good), "{good}"); + } + for bad in ["", "..", "a/b", "/tmp", "Copy", "-x", "a_b", "a.b"] { + assert!(!is_workflow_name(bad), "{bad}"); + } + } +} diff --git a/src/workflow/mod.rs b/src/workflow/mod.rs new file mode 100644 index 0000000..48fbcc8 --- /dev/null +++ b/src/workflow/mod.rs @@ -0,0 +1,494 @@ +//! `mapbox workflow` — install and run workflows: named, multi-step recipes +//! of `mapbox` commands and scripts. Beta. +//! +//! A workflow is a directory holding a `workflow.yaml` and the scripts it +//! runs (see [`definition`] for the schema and the layout rules). None ships +//! inside the binary. `install` copies one in, from a local directory or a +//! GitHub repository's `workflow///`, and `run` runs only what +//! is installed. +//! +//! What this refuses to be: a scheduler, a retry engine, or a language. +//! Steps run in order and the first failure stops the run. Logic belongs in +//! a script step. + +pub mod definition; +pub mod runner; +pub mod store; +pub mod template; + +use anyhow::Result; +use clap::{Arg, ArgAction, ArgMatches, Command}; +use serde_json::{json, Value}; + +use crate::confirm; +use crate::executor; +use crate::output::{self, field_lines, Mode}; + +pub const COMMAND: &str = "workflow"; + +const NAME_ARG: &str = "name"; +const SOURCE_ARG: &str = "source"; +const REPO_ARG: &str = "repo"; +const REF_ARG: &str = "ref"; +const FORCE_ARG: &str = "force"; +const INPUT_ARG: &str = "input"; + +pub fn command() -> Command { + let name = || { + Arg::new(NAME_ARG) + .value_name("NAME") + .required(true) + .help("Name of an installed workflow") + }; + Command::new(COMMAND) + .about("Install and run multi-step workflows of mapbox commands and scripts (beta)") + .long_about( + "Install and run workflows: named, multi-step recipes of mapbox commands and \ + scripts, defined in a workflow.yaml.\n\n\ + A workflow runs only once it is installed, from a local directory or from a \ + GitHub repository's workflow/// directory. Workflows and this \ + command are beta: their format may change.", + ) + .subcommand_required(true) + .subcommand(Command::new("list").about("List installed workflows")) + .subcommand( + Command::new("show") + .about("Describe an installed workflow: its inputs and steps") + .arg(name()), + ) + .subcommand( + Command::new("install") + .about("Install a workflow from a local directory or from GitHub") + .arg( + Arg::new(SOURCE_ARG) + .value_name("SOURCE") + .required(true) + .help( + "A workflow name, fetched from GitHub, or a path to a local \ + workflow directory (anything with a `/` or starting with `.`)", + ), + ) + .arg( + Arg::new(REPO_ARG) + .long(REPO_ARG) + .value_name("OWNER/REPO") + .help(format!( + "GitHub repository to install from [default: {}]. Reads \ + GH_TOKEN or GITHUB_TOKEN for a private one", + store::DEFAULT_REPO + )), + ) + .arg( + Arg::new(REF_ARG) + .long(REF_ARG) + .value_name("REF") + .help(format!( + "Branch, tag or commit to install from [default: {}]", + store::DEFAULT_REF + )), + ) + .arg( + Arg::new(FORCE_ARG) + .long(FORCE_ARG) + .action(ArgAction::SetTrue) + .help("Replace a workflow that is already installed"), + ) + .arg(executor::dry_run_arg( + "Check the workflow and list the files it would write, then exit", + )), + ) + .subcommand( + Command::new("uninstall") + .about("Remove an installed workflow") + .arg(name()) + .arg(executor::dry_run_arg( + "Say what it would remove, then exit without removing it", + )), + ) + .subcommand( + Command::new("run") + .about("Run an installed workflow") + .arg(name()) + .arg( + Arg::new(INPUT_ARG) + .long(INPUT_ARG) + .short('i') + .value_name("KEY=VALUE") + .action(ArgAction::Append) + .help("A value for one of the workflow's inputs, repeatable"), + ) + .arg(executor::dry_run_arg( + "Check the workflow and its inputs and print the plan, then exit \ + without running a step", + )), + ) +} + +/// What the globals say, for the subcommands that need them. +pub struct RunFlags<'a> { + pub globals: &'a ArgMatches, + pub debug: bool, + pub assume_yes: bool, +} + +pub fn run(app: &Command, matches: &ArgMatches, flags: RunFlags, mode: Mode) -> Result<()> { + match matches.subcommand() { + Some(("list", _)) => list(mode), + Some(("show", m)) => show(app, name(m), mode), + Some(("install", m)) => install(app, m, flags.debug, mode), + Some(("uninstall", m)) => { + uninstall(name(m), executor::wants_dry_run(m), flags.assume_yes, mode) + } + Some(("run", m)) => run_workflow(app, m, flags.globals, mode), + _ => unreachable!("`workflow` sets subcommand_required(true)"), + } +} + +fn name(matches: &ArgMatches) -> &str { + matches.get_one::(NAME_ARG).expect("required") +} + +fn stage_label(stage: Option<&str>) -> String { + stage.unwrap_or("local").to_string() +} + +fn warn_if_beta(workflow: &str, stage: Option<&str>) { + if stage == Some("beta") { + output::progress(&format!( + "`{workflow}` is a beta workflow: its inputs and outputs may change." + )); + } +} + +fn source_label(meta: &store::Meta) -> String { + match &meta.git_ref { + Some(git_ref) => format!("{}@{git_ref}", meta.source), + None => meta.source.clone(), + } +} + +fn list(mode: Mode) -> Result<()> { + let installed = store::list()?; + let mut rows = vec![]; + let mut text = String::new(); + for (name, loaded) in &installed { + match loaded { + Ok(found) => { + rows.push(json!({ + "name": name, + "stage": stage_label(found.meta.stage.as_deref()), + "summary": found.workflow.summary, + "source": source_label(&found.meta), + "path": found.root, + })); + text.push_str(&format!( + "{name} ({})\n {}\n", + stage_label(found.meta.stage.as_deref()), + found.workflow.summary + )); + } + Err(e) => { + rows.push(json!({ "name": name, "error": format!("{e:#}") })); + text.push_str(&format!("{name} (broken)\n {e:#}\n")); + } + } + } + if installed.is_empty() { + text = "No workflows installed. Install one with `mapbox workflow install `.\n" + .to_string(); + } + output::emit(mode, text.trim_end(), Value::Array(rows)) +} + +fn show(app: &Command, name: &str, mode: Mode) -> Result<()> { + let found = store::load(name)?; + let workflow = &found.workflow; + let problems = runner::command_problems(app, workflow); + + let inputs: Vec = workflow + .inputs + .iter() + .map(|(name, input)| { + json!({ + "name": name, + "type": input.kind.as_str(), + "required": input.required, + "default": input.default, + "description": input.description, + }) + }) + .collect(); + let steps: Vec = workflow + .steps + .iter() + .map(|step| json!({ "id": step.id, "name": step.name, "run": step.label() })) + .collect(); + + let mut text = field_lines( + &[ + ("Name", workflow.name.clone()), + ("Stage", stage_label(found.meta.stage.as_deref())), + ("Summary", workflow.summary.clone()), + ("Source", source_label(&found.meta)), + ("Path", found.root.display().to_string()), + ], + output::result_in_color(), + ); + if let Some(description) = &workflow.description { + text.push_str(&format!("\n\n{}", description.trim_end())); + } + text.push_str("\n\nInputs:"); + if workflow.inputs.is_empty() { + text.push_str(" none"); + } + for (name, input) in &workflow.inputs { + let mut line = format!("\n {name} ({}", input.kind.as_str()); + if input.required { + line.push_str(", required"); + } + if let Some(default) = &input.default { + line.push_str(&format!(", default {default}")); + } + line.push(')'); + if let Some(description) = &input.description { + line.push_str(&format!(" {description}")); + } + text.push_str(&line); + } + text.push_str("\n\nSteps:"); + for (index, step) in workflow.steps.iter().enumerate() { + let title = step.name.as_deref().unwrap_or(&step.id); + text.push_str(&format!("\n {}. {title} {}", index + 1, step.label())); + } + if !problems.is_empty() { + text.push_str("\n\nThis CLI cannot run it:"); + for problem in &problems { + text.push_str(&format!("\n - {problem}")); + } + } + + output::emit( + mode, + &text, + json!({ + "name": workflow.name, + "stage": stage_label(found.meta.stage.as_deref()), + "summary": workflow.summary, + "description": workflow.description, + "source": source_label(&found.meta), + "path": found.root, + "inputs": inputs, + "steps": steps, + "problems": problems, + }), + ) +} + +/// A source with a path separator or a leading `.` is a directory; anything +/// else is a name to look up on GitHub. A workflow name can hold neither, +/// so the two never overlap. +fn is_local_source(source: &str) -> bool { + source.starts_with('.') || source.contains('/') || source.contains('\\') +} + +fn install(app: &Command, matches: &ArgMatches, debug: bool, mode: Mode) -> Result<()> { + let source = matches.get_one::(SOURCE_ARG).expect("required"); + let repo = matches.get_one::(REPO_ARG); + let git_ref = matches.get_one::(REF_ARG); + let force = matches.get_flag(FORCE_ARG); + let dry_run = executor::wants_dry_run(matches); + + let package = if is_local_source(source) { + if repo.is_some() || git_ref.is_some() { + return Err(output::CliError::new( + "invalid_arguments", + "`--repo` and `--ref` name a GitHub source; this one is a local directory.", + ) + .into()); + } + store::read_local(std::path::Path::new(source))? + } else { + store::fetch_github( + source, + repo.map_or(store::DEFAULT_REPO, String::as_str), + git_ref.map_or(store::DEFAULT_REF, String::as_str), + debug, + )? + }; + let workflow = &package.workflow; + let problems = runner::command_problems(app, workflow); + if !problems.is_empty() { + return Err(store::invalid_workflow(&workflow.name, &problems)); + } + + let files: Vec = package + .files + .keys() + .map(|path| definition::slash_path(path)) + .collect(); + let target = store::root_path()?.join(&workflow.name); + let summary = json!({ + "name": workflow.name, + "stage": stage_label(package.meta.stage.as_deref()), + "source": source_label(&package.meta), + "path": target, + "files": files, + "dry_run": dry_run, + }); + + if dry_run { + if target.exists() && !force { + return Err(store::already_installed(&workflow.name, &target)); + } + warn_if_beta(&workflow.name, package.meta.stage.as_deref()); + let text = format!( + "Would install `{}` from {} into {}:\n{}", + workflow.name, + source_label(&package.meta), + target.display(), + files + .iter() + .map(|file| format!(" {file}")) + .collect::>() + .join("\n") + ); + return output::emit(mode, &text, summary); + } + + let target = store::install(&package, force)?; + warn_if_beta(&workflow.name, package.meta.stage.as_deref()); + let text = format!( + "Installed `{}` from {} into {}.\nRun it with `mapbox workflow run {}`.", + workflow.name, + source_label(&package.meta), + target.display(), + workflow.name + ); + output::emit(mode, &text, summary) +} + +fn uninstall(name: &str, dry_run: bool, assume_yes: bool, mode: Mode) -> Result<()> { + let found = store::installed_dir(name)?; + let Some(dir) = found else { + // `load` words the error, with the commands to try instead. + return store::load(name).map(|_| ()); + }; + if dry_run { + return output::emit( + mode, + &format!("Would remove {}.", dir.display()), + json!({ "name": name, "path": dir, "dry_run": true }), + ); + } + confirm::destructive_local_action( + &format!("Remove the workflow at {}?", dir.display()), + assume_yes, + )?; + let dir = store::uninstall(name)?; + output::emit( + mode, + &format!("Removed `{name}` from {}.", dir.display()), + json!({ "name": name, "path": dir, "dry_run": false }), + ) +} + +fn run_workflow( + app: &Command, + matches: &ArgMatches, + globals: &ArgMatches, + mode: Mode, +) -> Result<()> { + let found = store::load(name(matches))?; + let workflow = &found.workflow; + let problems = runner::command_problems(app, workflow); + if !problems.is_empty() { + return Err(store::invalid_workflow(&workflow.name, &problems)); + } + + let pairs: Vec = matches + .get_many::(INPUT_ARG) + .into_iter() + .flatten() + .cloned() + .collect(); + let inputs = runner::read_inputs(workflow, &pairs)?; + + if executor::wants_dry_run(matches) { + let plan = runner::plan(workflow, &inputs); + let text = + std::iter::once(format!("Would run `{}`:", workflow.name)) + .chain( + workflow.steps.iter().enumerate().map(|(index, step)| { + format!(" {}. {} {}", index + 1, step.id, step.label()) + }), + ) + .collect::>() + .join("\n"); + return output::emit(mode, &text, plan); + } + + warn_if_beta(&workflow.name, found.meta.stage.as_deref()); + let inherited = runner::Inherited::from_matches(globals); + let result = runner::run(app, workflow, &found.root, &inputs, &inherited)?; + output::emit_value(mode, &result, None, None, None) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::path::Path; + + #[test] + fn a_name_and_a_path_never_look_alike() { + for local in [ + "./copy-style", + "../x", + "workflow/beta/copy-style", + "/abs", + ".", + "a\\b", + ] { + assert!(is_local_source(local), "{local}"); + } + assert!(!is_local_source("copy-style")); + } + + /// Every workflow this repository publishes is one `install` accepts. + /// The rules are the same as for anybody's, so this is also what + /// catches a stray file or a step naming a command that was renamed. + #[test] + fn every_published_workflow_is_valid() { + let app = crate::build_app(&crate::spec::effective_services().expect("bundled specs")); + let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("workflow"); + let mut checked = 0; + for stage in std::fs::read_dir(&root) + .expect("workflow/ exists") + .flatten() + { + if !stage.path().is_dir() { + continue; + } + let stage_name = stage.file_name().to_string_lossy().into_owned(); + assert!( + store::STAGES.contains(&stage_name.as_str()), + "workflow/{stage_name}/ is not a stage; add it to store::STAGES or move it" + ); + for dir in std::fs::read_dir(stage.path()) + .expect("read stage") + .flatten() + { + let package = store::read_local(&dir.path()) + .unwrap_or_else(|e| panic!("{}: {e:#}", dir.path().display())); + let problems = runner::command_problems(&app, &package.workflow); + assert!( + problems.is_empty(), + "{}: {problems:#?}", + dir.path().display() + ); + assert_eq!(package.meta.stage.as_deref(), Some(stage_name.as_str())); + checked += 1; + } + } + assert!(checked > 0, "no workflows found under workflow/"); + } +} diff --git a/src/workflow/runner.rs b/src/workflow/runner.rs new file mode 100644 index 0000000..aa96576 --- /dev/null +++ b/src/workflow/runner.rs @@ -0,0 +1,601 @@ +//! Runs a checked workflow, one step at a time. +//! +//! A `command` step is this binary run again as a child, with `--output json` +//! and the step's arguments turned into flags. That is the point rather than +//! a shortcut: the child resolves its token, applies its timeouts, encodes +//! its path segments and records its run exactly as a command typed by hand +//! would, so a workflow cannot reach the API any way a person could not. +//! +//! Only the child's stdout is captured, as the step's output. Its stderr and +//! stdin are the terminal's, so progress, warnings and a confirmation prompt +//! reach the person running the workflow — unless the step feeds `stdin` +//! itself. +//! +//! A `script` step runs its file from the installed workflow's `scripts/` +//! under the interpreter the definition settled on. It is given `MAPBOX_CLI`, +//! the path to this binary, so it can call commands of its own. + +use std::collections::BTreeMap; +use std::ffi::OsString; +use std::io::Write; +use std::path::Path; +use std::process::{Command as Process, Stdio}; + +use anyhow::{anyhow, Context as _, Result}; +use clap::{Arg, ArgAction, Command}; +use serde_json::{json, Map, Value}; + +use super::definition::{Action, InputType, Step, Workflow, SCRIPTS_DIR}; +use super::template::{self, Context}; +use crate::auth; +use crate::output::{self, CliError}; + +/// Global options a step may not set, because the runner owns them or +/// because they would put a credential in a file. +const RESERVED: &[(&str, &str)] = &[ + ("output", "the runner reads every step as JSON"), + ("quiet", "the runner sets it"), + ( + "schema", + "a workflow runs commands rather than describing them", + ), + ( + "dry-run", + "use `mapbox workflow run --dry-run` for the whole workflow", + ), + ( + "token", + "a token written into a workflow is a secret in a file; use `profile`", + ), +]; + +/// The globals of the `workflow run` line that each command step inherits +/// unless it sets its own. +#[derive(Debug, Default, Clone)] +pub struct Inherited { + /// Only a token typed as `--token`; one from the environment reaches the + /// child by being in its environment already. + pub typed_token: Option, + pub options: Vec<(&'static str, String)>, + pub flags: Vec<&'static str>, +} + +impl Inherited { + pub fn from_matches(matches: &clap::ArgMatches) -> Self { + let mut inherited = Inherited { + typed_token: auth::typed_token(matches), + ..Default::default() + }; + for option in ["profile", "username", crate::http::TIMEOUT_ARG] { + let typed = + matches.value_source(option) == Some(clap::parser::ValueSource::CommandLine); + if let (true, Some(value)) = (typed, matches.get_one::(option)) { + inherited.options.push((option, value.clone())); + } + } + for flag in ["use-login", "debug", crate::confirm::ARG] { + if matches.get_flag(flag) { + inherited.flags.push(flag); + } + } + inherited + } +} + +/// Turns `--input key=value` pairs into the workflow's inputs, typed as the +/// definition declares, with defaults filled in. +pub fn read_inputs(workflow: &Workflow, pairs: &[String]) -> Result> { + let mut given: Map = Map::new(); + for pair in pairs { + let (key, raw) = pair + .split_once('=') + .ok_or_else(|| invalid_input(format!("`{pair}` is not KEY=VALUE"), workflow))?; + let Some(spec) = workflow.inputs.get(key) else { + return Err(invalid_input( + format!("`{}` has no input called `{key}`", workflow.name), + workflow, + )); + }; + let value = match spec.kind { + InputType::String => Value::String(raw.to_string()), + InputType::Number => raw + .parse::() + .map(Value::from) + .or_else(|_| raw.parse::().map(Value::from)) + .map_err(|_| { + invalid_input(format!("input `{key}` is a number, not `{raw}`"), workflow) + })?, + InputType::Boolean => match raw { + "true" => Value::Bool(true), + "false" => Value::Bool(false), + _ => { + return Err(invalid_input( + format!("input `{key}` is `true` or `false`, not `{raw}`"), + workflow, + )) + } + }, + }; + if given.insert(key.to_string(), value).is_some() { + return Err(invalid_input( + format!("input `{key}` is given twice"), + workflow, + )); + } + } + + let mut missing = vec![]; + for (name, spec) in &workflow.inputs { + if given.contains_key(name) { + continue; + } + match &spec.default { + Some(default) => { + given.insert(name.clone(), default.clone()); + } + None if spec.required => missing.push(name.as_str()), + None => { + given.insert(name.clone(), Value::Null); + } + } + } + if !missing.is_empty() { + let flags: Vec = missing + .iter() + .map(|name| format!("--input {name}=…")) + .collect(); + return Err(invalid_input( + format!("missing required input: {}", flags.join(" ")), + workflow, + )); + } + Ok(given) +} + +fn invalid_input(message: String, workflow: &Workflow) -> anyhow::Error { + CliError::new("invalid_input", message) + .with_remedy( + crate::remedy::Remedy::default() + .with_action(Some(format!("mapbox workflow show {}", workflow.name))), + ) + .into() +} + +/// What `--dry-run` prints: the plan, with the inputs resolved and every +/// later reference left as written, since nothing has run to fill it. +pub fn plan(workflow: &Workflow, inputs: &Map) -> Value { + let steps: Vec = workflow + .steps + .iter() + .map(|step| { + let (kind, args) = match &step.action { + Action::Command { args, .. } => ("command", Value::Object(args.clone())), + Action::Script { args, .. } => ("script", Value::Array(args.clone())), + }; + json!({ "id": step.id, "kind": kind, "run": step.label(), "args": args, "stdin": step.stdin }) + }) + .collect(); + json!({ "workflow": workflow.name, "inputs": inputs, "steps": steps, "outputs": workflow.outputs }) +} + +/// Runs every step and returns what `outputs` resolves to, or the last +/// step's output when the workflow declares none. +pub fn run( + app: &Command, + workflow: &Workflow, + root: &Path, + inputs: &Map, + inherited: &Inherited, +) -> Result { + let exe = std::env::current_exe().context("Could not find this program's own path")?; + let mut outputs: BTreeMap = BTreeMap::new(); + let total = workflow.steps.len(); + + for (index, step) in workflow.steps.iter().enumerate() { + let title = step.name.as_deref().unwrap_or(&step.id); + output::progress(&format!( + "[{}/{total}] {title} ({})", + index + 1, + step.label() + )); + + let context = Context { + inputs, + steps: &outputs, + }; + let stdin = step + .stdin + .as_ref() + .map(|value| template::resolve(value, &context)) + .transpose() + .map_err(|e| step_failure(step, e))?; + + let mut process = match &step.action { + Action::Command { path, args } => { + let args = match template::resolve(&Value::Object(args.clone()), &context) + .map_err(|e| step_failure(step, e))? + { + Value::Object(args) => args, + _ => unreachable!("an object resolves to an object"), + }; + let argv = + command_argv(app, path, &args, inherited).map_err(|e| step_failure(step, e))?; + let mut process = Process::new(&exe); + process.args(argv); + process + } + Action::Script { + script, + interpreter, + args, + } => { + let args = template::resolve(&Value::Array(args.clone()), &context) + .map_err(|e| step_failure(step, e))?; + let mut process = Process::new(interpreter); + process.arg(root.join(SCRIPTS_DIR).join(script)); + for value in args.as_array().into_iter().flatten() { + process.arg(template::as_text(value).unwrap_or_default()); + } + process + .env("MAPBOX_CLI", &exe) + .env("MAPBOX_WORKFLOW_ROOT", root); + process + } + }; + if let Some(token) = &inherited.typed_token { + process.env(auth::CLAP_TOKEN_ENV, token); + } + + let value = execute(&mut process, stdin.as_ref(), step)?; + outputs.insert(step.id.clone(), value); + } + + let last = workflow + .steps + .last() + .and_then(|step| outputs.get(&step.id)) + .cloned() + .unwrap_or(Value::Null); + match &workflow.outputs { + Some(declared) => template::resolve( + declared, + &Context { + inputs, + steps: &outputs, + }, + ) + .map_err(|e| CliError::new("workflow_failed", format!("`outputs`: {e:#}")).into()), + None => Ok(last), + } +} + +fn step_failure(step: &Step, err: anyhow::Error) -> anyhow::Error { + CliError::new( + "workflow_failed", + format!("Step `{}` ({}): {err:#}", step.id, step.label()), + ) + .into() +} + +/// Spawns one step and reads its stdout as its output: JSON when it parses, +/// the text otherwise, nothing when there is none. +fn execute(process: &mut Process, stdin: Option<&Value>, step: &Step) -> Result { + // A script with nothing to read gets nothing, rather than a terminal it + // might block on. A command keeps the terminal, which is where a + // confirmation prompt reads its answer. + let input = match (stdin, &step.action) { + (Some(_), _) => Stdio::piped(), + (None, Action::Script { .. }) => Stdio::null(), + (None, Action::Command { .. }) => Stdio::inherit(), + }; + let mut child = process + .stdin(input) + .stdout(Stdio::piped()) + .stderr(Stdio::inherit()) + .spawn() + .map_err(|e| { + let program = process.get_program().to_string_lossy().into_owned(); + let message = if e.kind() == std::io::ErrorKind::NotFound { + format!("`{program}` is not installed or not on PATH") + } else { + format!("could not start `{program}`: {e}") + }; + step_failure(step, anyhow!(message)) + })?; + + // Written from a thread: a child that fills its stdout pipe before it + // has read all of stdin would otherwise wait on us while we wait on it. + let writer = stdin.map(|value| { + let body = match value { + Value::String(text) => text.clone(), + other => other.to_string(), + }; + let mut pipe = child.stdin.take().expect("stdin was piped"); + std::thread::spawn(move || { + // A child that exits without reading is reported by its status. + let _ = pipe.write_all(body.as_bytes()); + }) + }); + let finished = child + .wait_with_output() + .map_err(|e| step_failure(step, anyhow!("could not wait for it: {e}")))?; + if let Some(writer) = writer { + let _ = writer.join(); + } + + if !finished.status.success() { + let how = match finished.status.code() { + Some(code) => format!("exited with {code}"), + None => "was stopped by a signal".to_string(), + }; + return Err(step_failure( + step, + anyhow!("{how}; the workflow stopped here"), + )); + } + + let text = String::from_utf8(finished.stdout).map_err(|_| { + step_failure( + step, + anyhow!("its output is binary, which cannot be passed to another step"), + ) + })?; + let text = text.trim(); + if text.is_empty() { + return Ok(Value::Null); + } + Ok(serde_json::from_str(text).unwrap_or_else(|_| Value::String(text.to_string()))) +} + +enum ArgShape { + Flag, + Option { repeatable: bool }, + Positional { repeatable: bool }, +} + +/// The leaf command a step names, when it names one a workflow may run. +fn leaf<'a>(app: &'a Command, path: &[String]) -> Result<&'a Command, String> { + let named = format!("mapbox {}", path.join(" ")); + if path.first().map(String::as_str) == Some(super::COMMAND) { + return Err(format!("`{named}`: a workflow cannot run another workflow")); + } + let mut current = app; + for segment in path { + current = current + .find_subcommand(segment) + .ok_or_else(|| format!("`{named}` is not a command"))?; + } + if current.has_subcommands() { + return Err(format!("`{named}` is a group; name one of its commands")); + } + Ok(current) +} + +/// The argument a step's key names: one of the command's own, by its long +/// flag or, for a positional, its name — the same names `mapbox --schema` +/// lists — or a global option. +fn find_arg<'a>( + app: &'a Command, + command: &'a Command, + key: &str, +) -> Result<(&'a Arg, ArgShape), String> { + if let Some((_, why)) = RESERVED.iter().find(|(name, _)| *name == key) { + return Err(format!("`{key}` cannot be set by a step: {why}")); + } + let own = command + .get_arguments() + .find(|arg| arg.get_long() == Some(key) || (arg.is_positional() && arg.get_id() == key)); + let arg = own + .or_else(|| { + app.get_arguments() + .find(|arg| arg.is_global_set() && arg.get_long() == Some(key)) + }) + .ok_or_else(|| format!("`{key}` is not an argument of this command"))?; + let repeatable = matches!(arg.get_action(), ArgAction::Append); + let shape = if matches!(arg.get_action(), ArgAction::SetTrue | ArgAction::SetFalse) { + ArgShape::Flag + } else if arg.is_positional() { + ArgShape::Positional { repeatable } + } else { + ArgShape::Option { repeatable } + }; + Ok((arg, shape)) +} + +/// Every `command` step's problems against this binary's command tree — +/// checked at install and again before a run, since the CLI may have been +/// upgraded underneath an installed workflow. +pub fn command_problems(app: &Command, workflow: &Workflow) -> Vec { + let mut problems = vec![]; + for step in &workflow.steps { + let Action::Command { path, args } = &step.action else { + continue; + }; + let command = match leaf(app, path) { + Ok(command) => command, + Err(e) => { + problems.push(format!("step `{}`: {e}", step.id)); + continue; + } + }; + for (key, value) in args { + match find_arg(app, command, key) { + Err(e) => problems.push(format!("step `{}`: {e}", step.id)), + Ok((_, ArgShape::Flag)) if !(value.is_boolean() || value.is_string()) => problems + .push(format!( + "step `{}`: `{key}` is a flag; give it true or false", + step.id + )), + Ok(_) => {} + } + } + } + problems +} + +/// A step's arguments as the child's command line. +/// +/// Options are spelled `--name=value`, so a value that starts with a dash — +/// a western longitude — is never read as a flag. Positionals come last, +/// after `--`, for the same reason. +fn command_argv( + app: &Command, + path: &[String], + args: &Map, + inherited: &Inherited, +) -> Result> { + let command = leaf(app, path).map_err(|e| anyhow!(e))?; + let mut argv: Vec = path.iter().map(OsString::from).collect(); + let mut positionals: Vec<(usize, OsString)> = vec![]; + + for (key, value) in args { + let (arg, shape) = find_arg(app, command, key).map_err(|e| anyhow!(e))?; + let values: Vec<&Value> = match value { + Value::Array(items) if !matches!(shape, ArgShape::Flag) => items.iter().collect(), + Value::Null => continue, + other => vec![other], + }; + match shape { + ArgShape::Flag => match value { + Value::Bool(true) => argv.push(format!("--{key}").into()), + Value::Bool(false) => {} + Value::String(text) if text == "true" => argv.push(format!("--{key}").into()), + Value::String(text) if text == "false" => {} + _ => return Err(anyhow!("`{key}` is a flag; give it true or false")), + }, + ArgShape::Option { repeatable } | ArgShape::Positional { repeatable } + if values.len() > 1 && !repeatable => + { + return Err(anyhow!("`{key}` takes one value, not a list")); + } + ArgShape::Option { .. } => { + for value in values { + let text = template::as_text(value).unwrap_or_default(); + argv.push(format!("--{key}={text}").into()); + } + } + ArgShape::Positional { .. } => { + let index = arg.get_index().unwrap_or(usize::MAX); + for value in values { + positionals.push((index, template::as_text(value).unwrap_or_default().into())); + } + } + } + } + + for (option, value) in &inherited.options { + if !args.contains_key(*option) { + argv.push(format!("--{option}={value}").into()); + } + } + for flag in &inherited.flags { + if !args.contains_key(*flag) { + argv.push(format!("--{flag}").into()); + } + } + argv.push(format!("--{}={}", output::ARG, output::JSON).into()); + argv.push(format!("--{}", output::banner::ARG).into()); + + if !positionals.is_empty() { + positionals.sort_by_key(|(index, _)| *index); + argv.push("--".into()); + argv.extend(positionals.into_iter().map(|(_, value)| value)); + } + Ok(argv) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::workflow::definition::{parse, Files, DEFINITION_FILE}; + use std::path::PathBuf; + + fn app() -> Command { + let specs = crate::spec::effective_services().expect("bundled specs"); + crate::build_app(&specs) + } + + fn workflow(steps: &str) -> Workflow { + let yaml = format!( + "version: 1\nname: t\nsummary: t\ninputs:\n n: {{ type: number, default: 2 }}\n flag: {{ type: boolean, default: false }}\nsteps:\n{steps}" + ); + let files: Files = [(PathBuf::from(DEFINITION_FILE), yaml.into_bytes())].into(); + parse("t", &files).expect("valid") + } + + #[test] + fn a_step_is_checked_against_the_real_command_tree() { + let found = command_problems( + &app(), + &workflow( + " - id: a\n command: styles nope\n\ + \x20 - id: b\n command: styles\n\ + \x20 - id: c\n command: styles get\n args: { bogus: 1, token: x }\n\ + \x20 - id: d\n command: workflow list\n", + ), + ); + let joined = found.join("\n"); + assert!( + joined.contains("`mapbox styles nope` is not a command"), + "{joined}" + ); + assert!(joined.contains("is a group"), "{joined}"); + assert!(joined.contains("`bogus` is not an argument"), "{joined}"); + assert!(joined.contains("a secret in a file"), "{joined}"); + assert!(joined.contains("cannot run another workflow"), "{joined}"); + } + + #[test] + fn arguments_become_flags_and_positionals_come_last() { + let app = app(); + let args = serde_json::json!({ + "style-id": "-abc", + "optimize": true, + "download": false, + "profile": "work", + }); + let inherited = Inherited { + options: vec![("profile", "ignored".into()), ("timeout", "30".into())], + flags: vec!["yes"], + ..Default::default() + }; + let argv = command_argv( + &app, + &["styles".into(), "get".into()], + args.as_object().unwrap(), + &inherited, + ) + .unwrap(); + let argv: Vec = argv + .iter() + .map(|a| a.to_string_lossy().into_owned()) + .collect(); + assert_eq!( + argv, + [ + "styles", + "get", + "--optimize", + "--profile=work", + "--timeout=30", + "--yes", + "--output=json", + "--quiet", + "--", + "-abc" + ] + ); + } + + #[test] + fn inputs_are_typed_and_defaulted() { + let wf = workflow(" - id: a\n command: styles list\n"); + let inputs = read_inputs(&wf, &["n=5".into()]).unwrap(); + assert_eq!(inputs["n"], serde_json::json!(5)); + assert_eq!(inputs["flag"], serde_json::json!(false)); + assert!(read_inputs(&wf, &["n=five".into()]).is_err()); + assert!(read_inputs(&wf, &["nope=1".into()]).is_err()); + assert!(read_inputs(&wf, &["n=1".into(), "n=2".into()]).is_err()); + assert!(read_inputs(&wf, &["n".into()]).is_err()); + } +} diff --git a/src/workflow/store.rs b/src/workflow/store.rs new file mode 100644 index 0000000..03d8d05 --- /dev/null +++ b/src/workflow/store.rs @@ -0,0 +1,644 @@ +//! Where installed workflows live, and how they get there. +//! +//! Nothing is bundled into the binary: a workflow runs only once it has been +//! installed, from a local directory or from a GitHub repository laid out as +//! `workflow///`, into `/workflows//`. A copy +//! is taken rather than a link kept, so what runs is what was checked at +//! install time and nothing edited since. +//! +//! The same rules as `agent-skills` hold for the write, for the same +//! reasons: everything is read and checked in memory first, an existing +//! workflow stops the install unless `--force`, only regular files are +//! taken, and a path that would leave the workflow's directory stops the +//! read altogether. The new directory is staged and renamed into place. + +use std::io::Read; +use std::path::{Component, Path, PathBuf}; +use std::time::{SystemTime, UNIX_EPOCH}; + +use anyhow::{anyhow, Context, Result}; +use serde::{Deserialize, Serialize}; + +use super::definition::{self, is_contained, is_workflow_name, Files, Workflow}; +use crate::auth; +use crate::executor; +use crate::http; +use crate::output::CliError; +use crate::remedy::Remedy; + +const DIR_NAME: &str = "workflows"; + +/// Written beside an installed workflow's files, and what marks a directory +/// as one this command installed — `uninstall` removes nothing without it. +const META_FILE: &str = ".install.json"; + +/// Where the tarball comes from. The API rather than codeload directly, +/// because the API is what honors a token for a private repository; it +/// answers with a redirect to a signed codeload URL, and `reqwest` drops the +/// `Authorization` header when it follows a redirect to another host. +const GITHUB_API: &str = "https://api.github.com"; +pub const DEFAULT_REPO: &str = "mapbox/cli"; +pub const DEFAULT_REF: &str = "main"; + +/// The directory inside a repository that holds workflows, by stage. +const REPO_PREFIX: &str = "workflow"; + +/// The stages a workflow may be published under. `beta` is the only one so +/// far; a `stable/` beside it needs nothing but an entry here. +pub const STAGES: &[&str] = &["beta"]; + +/// A ceiling on what is read, from a tarball or a local directory. Far above +/// any real workflow, and there so a hostile archive cannot make this +/// allocate without bound. +const MAX_BYTES: u64 = 64 * 1024 * 1024; + +/// The files `read_local` passes over rather than refusing. Finder writes +/// `.DS_Store` into any directory it opens. +const IGNORED: &[&str] = &[".DS_Store"]; + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Meta { + /// Where it came from, as a person would write it: a directory, or + /// `github:owner/repo`. + pub source: String, + #[serde(default, rename = "ref")] + pub git_ref: Option, + #[serde(default)] + pub stage: Option, + pub installed_at: u64, +} + +/// A workflow read and checked, not yet written anywhere. +pub struct Package { + pub workflow: Workflow, + pub files: Files, + pub meta: Meta, +} + +/// A workflow on disk. +pub struct Installed { + pub root: PathBuf, + pub workflow: Workflow, + pub meta: Meta, +} + +/// `/workflows`, resolved and not created. +pub fn root_path() -> Result { + auth::config_dir_path() + .map(|dir| dir.join(DIR_NAME)) + .ok_or_else(|| anyhow!("Could not determine home directory")) +} + +pub fn invalid_workflow(name: &str, problems: &[String]) -> anyhow::Error { + let listed = problems + .iter() + .map(|problem| format!(" - {problem}")) + .collect::>() + .join("\n"); + CliError::new( + "invalid_workflow", + format!("`{name}` is not a valid workflow:\n{listed}"), + ) + .into() +} + +/// A workflow name from the command line, checked before it is ever joined +/// to a path. +pub fn check_name(name: &str) -> Result<()> { + if is_workflow_name(name) { + return Ok(()); + } + Err(CliError::new( + "invalid_name", + format!("`{name}` is not a workflow name: lower-case letters, digits and dashes"), + ) + .with_remedy(Remedy::default().with_action(Some("mapbox workflow list".to_string()))) + .into()) +} + +fn now() -> u64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|elapsed| elapsed.as_secs()) + .unwrap_or(0) +} + +/// Every regular file under `dir`, by path relative to it. +/// +/// A symlink is refused rather than followed: it is a way to install +/// something from outside the directory that was named. +fn read_tree(dir: &Path, skip: &[&str]) -> Result { + let mut files = Files::new(); + let mut total: u64 = 0; + let mut pending = vec![dir.to_path_buf()]; + while let Some(current) = pending.pop() { + let entries = std::fs::read_dir(¤t) + .with_context(|| format!("Could not read {}", current.display()))?; + for entry in entries { + let entry = entry.with_context(|| format!("Could not read {}", current.display()))?; + let path = entry.path(); + let relative = path.strip_prefix(dir).expect("under the directory walked"); + let name = entry.file_name(); + if current == dir && skip.iter().any(|s| name == *s) { + continue; + } + if IGNORED.iter().any(|ignored| name == *ignored) { + continue; + } + let kind = entry + .file_type() + .with_context(|| format!("Could not read {}", path.display()))?; + if kind.is_dir() { + pending.push(path); + } else if kind.is_file() { + let bytes = std::fs::read(&path) + .with_context(|| format!("Could not read {}", path.display()))?; + total += bytes.len() as u64; + if total > MAX_BYTES { + return Err(anyhow!( + "{} holds more than a workflow should", + dir.display() + )); + } + files.insert(relative.to_path_buf(), bytes); + } else { + return Err(CliError::new( + "invalid_workflow", + format!( + "{} is not a regular file; a workflow holds only files and directories", + path.display() + ), + ) + .into()); + } + } + } + Ok(files) +} + +fn package(name: &str, files: Files, meta: Meta) -> Result { + let workflow = definition::parse(name, &files).map_err(|p| invalid_workflow(name, &p))?; + Ok(Package { + workflow, + files, + meta, + }) +} + +/// A workflow from a directory on this machine. The directory's own name is +/// the workflow's, as it is in a repository. +pub fn read_local(dir: &Path) -> Result { + let dir = dir + .canonicalize() + .with_context(|| format!("There is no directory {}", dir.display()))?; + if !dir.is_dir() { + return Err(anyhow!("{} is not a directory", dir.display())); + } + let name = dir + .file_name() + .map(|n| n.to_string_lossy().into_owned()) + .unwrap_or_default(); + let stage = dir + .parent() + .and_then(Path::file_name) + .map(|n| n.to_string_lossy().into_owned()) + .filter(|parent| STAGES.contains(&parent.as_str())); + let files = read_tree(&dir, &[])?; + package( + &name, + files, + Meta { + source: dir.display().to_string(), + git_ref: None, + stage, + installed_at: now(), + }, + ) +} + +/// `owner/repo`, and nothing that could move the request elsewhere. +pub fn check_repo(repo: &str) -> Result<()> { + let part = |p: &str| { + !p.is_empty() + && p != "." + && p != ".." + && p.chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '-' | '_' | '.')) + }; + match repo.split_once('/') { + Some((owner, name)) if part(owner) && part(name) => Ok(()), + _ => Err(CliError::new( + "invalid_repo", + format!("`{repo}` is not a GitHub repository; write it as OWNER/REPO"), + ) + .into()), + } +} + +/// A branch, tag or commit. `/` is allowed, since a branch may hold one; +/// URL syntax and `..` are not, since the ref goes into a URL path. +pub fn check_ref(git_ref: &str) -> Result<()> { + let ok = !git_ref.is_empty() + && !git_ref + .split('/') + .any(|part| part.is_empty() || part == "." || part == "..") + && git_ref + .chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '-' | '_' | '.' | '/')); + if ok { + return Ok(()); + } + Err(CliError::new( + "invalid_ref", + format!("`{git_ref}` is not a branch, tag or commit this can fetch"), + ) + .into()) +} + +fn github_token() -> Option { + ["GH_TOKEN", "GITHUB_TOKEN"] + .iter() + .filter_map(|name| std::env::var(name).ok()) + .find(|value| !value.trim().is_empty()) +} + +/// A workflow out of a GitHub repository's tarball. +pub fn fetch_github(name: &str, repo: &str, git_ref: &str, debug: bool) -> Result { + fetch_github_from(GITHUB_API, name, repo, git_ref, debug) +} + +fn fetch_github_from( + base: &str, + name: &str, + repo: &str, + git_ref: &str, + debug: bool, +) -> Result { + check_name(name)?; + check_repo(repo)?; + check_ref(git_ref)?; + + let url = format!("{base}/repos/{repo}/tarball/{git_ref}"); + if debug { + eprintln!("[debug] GET {url}"); + } + let token = github_token(); + let mut request = http::client()? + .get(&url) + .header("Accept", "application/vnd.github+json"); + if let Some(token) = &token { + request = request.bearer_auth(token); + } + let response = http::send(request) + .map_err(|e| executor::transport_failure("Could not reach GitHub", e))?; + + let status = response.status(); + if !status.is_success() { + // A private repository answers 404 to a request without a token, the + // same as a ref that does not exist, so both are named. + let message = match (status.as_u16(), &token) { + (404, None) => format!( + "GitHub found no `{git_ref}` in {repo}. If the repository is private, set \ + GITHUB_TOKEN to a token that can read it." + ), + (404, Some(_)) => format!( + "GitHub found no `{git_ref}` in {repo}, or the token in GH_TOKEN/GITHUB_TOKEN \ + cannot read it." + ), + _ => format!("GitHub answered {status} for {url}."), + }; + let mut remedy = Remedy::default(); + if token.is_none() { + remedy = remedy.with_fix("export GITHUB_TOKEN=\"$(gh auth token)\""); + } + return Err(CliError::http(status.as_u16(), &message) + .with_remedy(remedy) + .into()); + } + + let mut bytes = vec![]; + response + .take(MAX_BYTES) + .read_to_end(&mut bytes) + .map_err(|e| anyhow!("Could not read the archive from GitHub: {e}"))?; + let (stage, files) = extract(&bytes, name, repo)?; + package( + name, + files, + Meta { + source: format!("github:{repo}"), + git_ref: Some(git_ref.to_string()), + stage: Some(stage), + installed_at: now(), + }, + ) +} + +fn unsafe_entry(path: &Path) -> anyhow::Error { + CliError::new( + "unsafe_archive", + format!( + "Refusing to read `{}` out of the archive: it points outside the directory it \ + would be written to.", + path.display() + ), + ) + .into() +} + +/// The files of `workflow///` in a repository tarball, and the +/// stage it was found under. +fn extract(archive: &[u8], name: &str, repo: &str) -> Result<(String, Files)> { + let decoder = flate2::read::GzDecoder::new(archive); + let mut tar = tar::Archive::new(decoder.take(MAX_BYTES)); + + let mut found: std::collections::BTreeMap<(String, String), Files> = Default::default(); + for entry in tar + .entries() + .context("The archive from GitHub is not readable as a tarball")? + { + let mut entry = entry.context("The archive from GitHub ended unexpectedly")?; + // Only regular files: a link is a way out of the destination no path + // check would catch. + if entry.header().entry_type() != tar::EntryType::Regular { + continue; + } + let path = entry + .path() + .context("The archive holds an entry whose path is not valid UTF-8")? + .into_owned(); + // GitHub wraps every archive in one directory named after the + // repository and the commit, so it is stripped by position. + let mut components = path.components(); + let Some(Component::Normal(_)) = components.next() else { + return Err(unsafe_entry(&path)); + }; + let inner: PathBuf = components.collect(); + if !is_contained(&inner) { + return Err(unsafe_entry(&path)); + } + let Ok(within) = inner.strip_prefix(REPO_PREFIX) else { + continue; + }; + let mut parts = within.components(); + let (Some(stage), Some(workflow)) = (parts.next(), parts.next()) else { + continue; + }; + let relative: PathBuf = parts.collect(); + if relative.as_os_str().is_empty() { + continue; + } + let stage = stage.as_os_str().to_string_lossy().into_owned(); + let workflow = workflow.as_os_str().to_string_lossy().into_owned(); + if !STAGES.contains(&stage.as_str()) { + continue; + } + let key = (stage, workflow); + if key.1 != name { + found.entry(key).or_default(); + continue; + } + let mut bytes = vec![]; + entry + .read_to_end(&mut bytes) + .with_context(|| format!("Could not read {} out of the archive", path.display()))?; + found.entry(key).or_default().insert(relative, bytes); + } + + let mut matching: Vec<(String, Files)> = vec![]; + let mut available: Vec = vec![]; + for ((stage, workflow), files) in found { + if workflow == name { + matching.push((stage, files)); + } else { + available.push(format!("{workflow} ({stage})")); + } + } + match matching.len() { + 1 => Ok(matching.pop().expect("one")), + 0 => { + let listed = if available.is_empty() { + format!("{repo} publishes no workflows under `{REPO_PREFIX}/`.") + } else { + format!("It publishes: {}.", available.join(", ")) + }; + Err(CliError::new( + "workflow_not_found", + format!("{repo} has no workflow called `{name}`. {listed}"), + ) + .into()) + } + _ => { + let stages: Vec<&str> = matching.iter().map(|(stage, _)| stage.as_str()).collect(); + Err(CliError::new( + "ambiguous_workflow", + format!( + "{repo} publishes `{name}` under more than one stage ({}), which it \ + should not.", + stages.join(", ") + ), + ) + .into()) + } + } +} + +/// Writes `package` into place, replacing an existing install only when +/// `force` says so. Returns where it went. +pub fn install(package: &Package, force: bool) -> Result { + let name = &package.workflow.name; + check_name(name)?; + let root = root_path()?; + std::fs::create_dir_all(&root) + .with_context(|| format!("Could not create {}", root.display()))?; + let target = root.join(name); + if target.exists() && !force { + return Err(already_installed(name, &target)); + } + + let staging = root.join(format!(".{name}.staging")); + let _ = std::fs::remove_dir_all(&staging); + let result = (|| -> Result<()> { + for (relative, bytes) in &package.files { + let path = staging.join(relative); + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent) + .with_context(|| format!("Could not create {}", parent.display()))?; + } + std::fs::write(&path, bytes) + .with_context(|| format!("Could not write {}", path.display()))?; + } + std::fs::write( + staging.join(META_FILE), + serde_json::to_string_pretty(&package.meta)?, + ) + .context("Could not record where the workflow came from")?; + + if target.exists() { + std::fs::remove_dir_all(&target) + .with_context(|| format!("Could not replace {}", target.display()))?; + } + std::fs::rename(&staging, &target) + .with_context(|| format!("Could not move the workflow into {}", target.display())) + })(); + let _ = std::fs::remove_dir_all(&staging); + result.map(|()| target) +} + +pub fn already_installed(name: &str, target: &Path) -> anyhow::Error { + CliError::new( + "already_installed", + format!("`{name}` is already installed at {}", target.display()), + ) + .with_remedy(Remedy::default().with_fix("Pass --force to replace it.")) + .into() +} + +fn not_installed(name: &str) -> anyhow::Error { + CliError::new( + "workflow_not_installed", + format!("No workflow called `{name}` is installed."), + ) + .with_remedy( + Remedy::default() + .with_action(Some(format!("mapbox workflow install {name}"))) + .with_action(Some("mapbox workflow list".to_string())), + ) + .into() +} + +/// An installed workflow's directory, if there is one. Only a directory +/// holding [`META_FILE`] counts. +pub fn installed_dir(name: &str) -> Result> { + check_name(name)?; + let dir = root_path()?.join(name); + Ok(dir.join(META_FILE).is_file().then_some(dir)) +} + +/// Reads an installed workflow back and checks it again: a file edited by +/// hand since the install is refused here rather than failing mid-run. +pub fn load(name: &str) -> Result { + let root = installed_dir(name)?.ok_or_else(|| not_installed(name))?; + load_dir(name, root) +} + +fn load_dir(name: &str, root: PathBuf) -> Result { + let meta: Meta = serde_json::from_slice( + &std::fs::read(root.join(META_FILE)) + .with_context(|| format!("Could not read {}", root.join(META_FILE).display()))?, + ) + .with_context(|| format!("{} is not readable", root.join(META_FILE).display()))?; + let files = read_tree(&root, &[META_FILE])?; + let workflow = definition::parse(name, &files).map_err(|p| invalid_workflow(name, &p))?; + Ok(Installed { + root, + workflow, + meta, + }) +} + +/// Every installed workflow, by name, with the ones that no longer load +/// kept as their error. +pub fn list() -> Result)>> { + let root = root_path()?; + let Ok(entries) = std::fs::read_dir(&root) else { + return Ok(vec![]); + }; + let mut out = vec![]; + for entry in entries.flatten() { + let name = entry.file_name().to_string_lossy().into_owned(); + if !is_workflow_name(&name) || !entry.path().join(META_FILE).is_file() { + continue; + } + let loaded = load_dir(&name, entry.path()); + out.push((name, loaded)); + } + out.sort_by(|a, b| a.0.cmp(&b.0)); + Ok(out) +} + +/// Removes an installed workflow. The name is checked and the directory +/// must carry [`META_FILE`], so nothing but a workflow this command put +/// there is ever removed. +pub fn uninstall(name: &str) -> Result { + let dir = installed_dir(name)?.ok_or_else(|| not_installed(name))?; + std::fs::remove_dir_all(&dir).with_context(|| format!("Could not remove {}", dir.display()))?; + Ok(dir) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn archive(entries: &[(&str, &[u8])]) -> Vec { + let mut builder = tar::Builder::new(flate2::write::GzEncoder::new( + vec![], + flate2::Compression::fast(), + )); + for (path, bytes) in entries { + let mut header = tar::Header::new_gnu(); + header.set_size(bytes.len() as u64); + header.set_mode(0o644); + header.set_entry_type(tar::EntryType::Regular); + builder.append_data(&mut header, path, *bytes).unwrap(); + } + builder.into_inner().unwrap().finish().unwrap() + } + + #[test] + fn only_the_named_workflow_comes_out() { + let bytes = archive(&[ + ("mapbox-cli-abc/README.md", b"x"), + ( + "mapbox-cli-abc/workflow/beta/copy-style/workflow.yaml", + b"a", + ), + ("mapbox-cli-abc/workflow/beta/copy-style/scripts/p.py", b"b"), + ("mapbox-cli-abc/workflow/beta/other/workflow.yaml", b"c"), + ]); + let (stage, files) = extract(&bytes, "copy-style", "mapbox/cli").unwrap(); + assert_eq!(stage, "beta"); + assert_eq!( + files.keys().cloned().collect::>(), + [ + PathBuf::from("scripts/p.py"), + PathBuf::from("workflow.yaml") + ] + ); + } + + #[test] + fn a_missing_workflow_lists_what_is_there() { + let bytes = archive(&[("r-abc/workflow/beta/other/workflow.yaml", b"c")]); + let err = extract(&bytes, "copy-style", "mapbox/cli").unwrap_err(); + assert!(err.to_string().contains("other (beta)"), "{err}"); + } + + #[test] + fn an_unknown_stage_is_not_published() { + let bytes = archive(&[("r-abc/workflow/experimental/copy-style/workflow.yaml", b"c")]); + assert!(extract(&bytes, "copy-style", "mapbox/cli").is_err()); + } + + #[test] + fn repos_and_refs_cannot_carry_url_syntax() { + assert!(check_repo("mapbox/cli").is_ok()); + for bad in [ + "mapbox", + "mapbox/cli/x", + "../x", + "a/b?c", + "a/b#c", + "/cli", + "a/..", + ] { + assert!(check_repo(bad).is_err(), "{bad}"); + } + for good in ["main", "v0.3.0", "feat/copy-style", "063415e"] { + assert!(check_ref(good).is_ok(), "{good}"); + } + for bad in ["", "a?b", "a#b", "../main", "a//b", "a b", "a%2e"] { + assert!(check_ref(bad).is_err(), "{bad}"); + } + } +} diff --git a/src/workflow/template.rs b/src/workflow/template.rs new file mode 100644 index 0000000..51ce29d --- /dev/null +++ b/src/workflow/template.rs @@ -0,0 +1,351 @@ +//! `${{ … }}` expressions: how one step's values reach the next. +//! +//! Deliberately a lookup language and nothing more — no operators, no +//! functions, no conditionals. An expression names a value: an input, or a +//! path into an earlier step's output. Anything that needs logic belongs in a +//! script step, which is a real language and is tested as one. +//! +//! A string that is exactly one expression takes the value's own JSON type, +//! so an object can be handed to `stdin` whole and a number stays a number. +//! An expression inside a longer string is interpolated as text. + +use std::collections::BTreeMap; + +use anyhow::{anyhow, Result}; +use serde_json::{Map, Value}; + +const OPEN: &str = "${{"; +const CLOSE: &str = "}}"; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Segment { + Key(String), + Index(usize), +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Reference { + Input(String), + Step { id: String, path: Vec }, +} + +/// What an expression can see while a workflow runs. +pub struct Context<'a> { + pub inputs: &'a Map, + /// The outputs of the steps that have finished, by id. + pub steps: &'a BTreeMap, +} + +/// Every reference in `value`, for checking before anything runs. +pub fn references(value: &Value) -> Result, String> { + let mut found = vec![]; + walk_strings(value, &mut |text| { + for piece in pieces(text)? { + if let Piece::Expression(reference) = piece { + found.push(reference); + } + } + Ok(()) + })?; + Ok(found) +} + +/// `value` with every expression replaced by what it names. +pub fn resolve(value: &Value, context: &Context) -> Result { + match value { + Value::String(text) => resolve_string(text, context), + Value::Array(items) => items + .iter() + .map(|item| resolve(item, context)) + .collect::>() + .map(Value::Array), + Value::Object(fields) => fields + .iter() + .map(|(key, item)| Ok((key.clone(), resolve(item, context)?))) + .collect::>() + .map(Value::Object), + other => Ok(other.clone()), + } +} + +/// How a resolved value is spelled where only text fits: an argv entry, an +/// interpolated string. +pub fn as_text(value: &Value) -> Option { + match value { + Value::Null => None, + Value::String(text) => Some(text.clone()), + Value::Bool(_) | Value::Number(_) => Some(value.to_string()), + Value::Array(_) | Value::Object(_) => Some(value.to_string()), + } +} + +fn resolve_string(text: &str, context: &Context) -> Result { + let pieces = pieces(text).map_err(|e| anyhow!(e))?; + if let [Piece::Expression(reference)] = pieces.as_slice() { + return lookup(reference, context); + } + let mut out = String::new(); + for piece in pieces { + match piece { + Piece::Text(literal) => out.push_str(&literal), + Piece::Expression(reference) => { + let value = lookup(&reference, context)?; + // Interpolating a null would write "null" or nothing into the + // middle of a URL or a name, and neither is what was meant. + let text = as_text(&value).ok_or_else(|| { + anyhow!( + "`{}` is empty, so it cannot be written into \"{text}\"", + display(&reference) + ) + })?; + out.push_str(&text); + } + } + } + Ok(Value::String(out)) +} + +fn lookup(reference: &Reference, context: &Context) -> Result { + match reference { + Reference::Input(name) => Ok(context.inputs.get(name).cloned().unwrap_or(Value::Null)), + Reference::Step { id, path } => { + let mut current = context + .steps + .get(id) + .ok_or_else(|| anyhow!("`{}`: step `{id}` has not run", display(reference)))?; + for segment in path { + let next = match segment { + Segment::Key(key) => current.get(key.as_str()), + Segment::Index(index) => current.get(*index), + }; + current = next.ok_or_else(|| { + anyhow!( + "`{}`: step `{id}`'s output has nothing at that path", + display(reference) + ) + })?; + } + Ok(current.clone()) + } + } +} + +/// An expression as the author wrote it, for error messages. +pub fn display(reference: &Reference) -> String { + match reference { + Reference::Input(name) => format!("inputs.{name}"), + Reference::Step { id, path } => { + let mut out = format!("steps.{id}.output"); + for segment in path { + match segment { + Segment::Key(key) => { + out.push('.'); + out.push_str(key); + } + Segment::Index(index) => out.push_str(&format!("[{index}]")), + } + } + out + } + } +} + +fn walk_strings( + value: &Value, + visit: &mut dyn FnMut(&str) -> Result<(), String>, +) -> Result<(), String> { + match value { + Value::String(text) => visit(text), + Value::Array(items) => items.iter().try_for_each(|item| walk_strings(item, visit)), + Value::Object(fields) => fields + .values() + .try_for_each(|item| walk_strings(item, visit)), + _ => Ok(()), + } +} + +#[derive(Debug, PartialEq)] +enum Piece { + Text(String), + Expression(Reference), +} + +fn pieces(text: &str) -> Result, String> { + let mut out = vec![]; + let mut rest = text; + while let Some(start) = rest.find(OPEN) { + if start > 0 { + out.push(Piece::Text(rest[..start].to_string())); + } + let after = &rest[start + OPEN.len()..]; + let end = after + .find(CLOSE) + .ok_or_else(|| format!("unclosed `{OPEN}` in \"{text}\""))?; + out.push(Piece::Expression(parse(after[..end].trim())?)); + rest = &after[end + CLOSE.len()..]; + } + if !rest.is_empty() { + out.push(Piece::Text(rest.to_string())); + } + Ok(out) +} + +/// `inputs.` or `steps..output` followed by `.key` and `[n]`. +fn parse(expression: &str) -> Result { + let invalid = || { + format!( + "`{expression}` is not an expression this understands. Use \ + `inputs.` or `steps..output`, optionally followed by \ + `.key` or `[index]`" + ) + }; + + let mut segments = vec![]; + let mut chars = expression.chars().peekable(); + let mut first = true; + while chars.peek().is_some() { + if !first { + match chars.next() { + Some('.') => {} + Some('[') => { + let digits: String = chars.by_ref().take_while(|c| *c != ']').collect(); + let index = digits.parse::().map_err(|_| invalid())?; + segments.push(Segment::Index(index)); + continue; + } + _ => return Err(invalid()), + } + } + first = false; + let mut key = String::new(); + while let Some(c) = chars.peek() { + if c.is_ascii_alphanumeric() || *c == '_' || *c == '-' { + key.push(*c); + chars.next(); + } else { + break; + } + } + if key.is_empty() { + return Err(invalid()); + } + segments.push(Segment::Key(key)); + } + + let mut segments = segments.into_iter(); + match (segments.next(), segments.next(), segments.next()) { + (Some(Segment::Key(root)), Some(Segment::Key(name)), None) if root == "inputs" => { + Ok(Reference::Input(name)) + } + (Some(Segment::Key(root)), Some(Segment::Key(id)), Some(Segment::Key(output))) + if root == "steps" && output == "output" => + { + Ok(Reference::Step { + id, + path: segments.collect(), + }) + } + _ => Err(invalid()), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + fn context_resolve(value: Value) -> Result { + let inputs = json!({ "style_id": "abc", "zoom": 12, "empty": null }); + let steps = BTreeMap::from([( + "source".to_string(), + json!({ "owner": "alice", "layers": [{ "id": "water" }] }), + )]); + resolve( + &value, + &Context { + inputs: inputs.as_object().unwrap(), + steps: &steps, + }, + ) + } + + #[test] + fn a_whole_expression_keeps_its_type() { + assert_eq!( + context_resolve(json!("${{ inputs.zoom }}")).unwrap(), + json!(12) + ); + assert_eq!( + context_resolve(json!("${{ steps.source.output.layers[0] }}")).unwrap(), + json!({ "id": "water" }) + ); + } + + #[test] + fn an_embedded_expression_is_interpolated_as_text() { + assert_eq!( + context_resolve(json!( + "styles/${{ steps.source.output.owner }}/${{inputs.style_id}}" + )) + .unwrap(), + json!("styles/alice/abc") + ); + } + + #[test] + fn objects_and_arrays_are_resolved_inside() { + assert_eq!( + context_resolve(json!({ "a": ["${{ inputs.style_id }}", 1] })).unwrap(), + json!({ "a": ["abc", 1] }) + ); + } + + #[test] + fn an_empty_value_cannot_be_interpolated() { + assert!(context_resolve(json!("x-${{ inputs.empty }}")).is_err()); + assert_eq!( + context_resolve(json!("${{ inputs.empty }}")).unwrap(), + Value::Null + ); + } + + #[test] + fn a_missing_path_names_the_expression() { + let err = context_resolve(json!("${{ steps.source.output.nope }}")).unwrap_err(); + assert!( + err.to_string().contains("steps.source.output.nope"), + "{err}" + ); + } + + #[test] + fn references_are_found_for_checking() { + assert_eq!( + references(&json!({ "a": "${{ inputs.x }}", "b": ["${{ steps.s.output.k[2] }}"] })) + .unwrap(), + vec![ + Reference::Input("x".into()), + Reference::Step { + id: "s".into(), + path: vec![Segment::Key("k".into()), Segment::Index(2)] + } + ] + ); + } + + #[test] + fn anything_else_is_refused() { + for bad in [ + "${{ inputs }}", + "${{ inputs.a.b }}", + "${{ steps.s }}", + "${{ steps.s.result }}", + "${{ env.HOME }}", + "${{ inputs.a + 1 }}", + "${{ steps.s.output[x] }}", + "${{ inputs.a", + ] { + assert!(references(&json!(bad)).is_err(), "{bad} was accepted"); + } + } +} diff --git a/tests/source_guards.rs b/tests/source_guards.rs index c91aae3..6cef6ed 100644 --- a/tests/source_guards.rs +++ b/tests/source_guards.rs @@ -49,6 +49,10 @@ fn sources() -> Vec<(String, String)> { /// - `generate_skills` — the staged skill directory it renames into place. /// - `skill_dest` — a test scratch directory. /// - `uninstall` — the binary itself, which is the whole command. +/// - `workflow/store.rs` — its staging directory, the installed workflow +/// `install --force` replaces, and `workflow uninstall`. The name is +/// checked to be one plain directory name first, and only a directory +/// holding its `.install.json` marker is removed. const MAY_DELETE: &[&str] = &[ "agent_skills.rs", "auth.rs", @@ -58,6 +62,7 @@ const MAY_DELETE: &[&str] = &[ "skill_dest.rs", "telemetry_sink.rs", "uninstall.rs", + "workflow/store.rs", ]; /// A new module that deletes files has to say so here first. @@ -240,7 +245,10 @@ const CARRIES_A_REQUEST_ID: &[&str] = &["account_usage.rs", "auth.rs", "executor /// `agent_skills.rs` talks to GitHub codeload, which identifies requests with /// `x-github-request-id`. That is not something Mapbox support can look up, /// so an id there would point at the wrong company — worse than none. -const NO_REQUEST_ID_TO_CARRY: &[&str] = &["agent_skills.rs"]; +/// +/// `workflow/store.rs` fetches a tarball from the GitHub API, for the same +/// reason. +const NO_REQUEST_ID_TO_CARRY: &[&str] = &["agent_skills.rs", "workflow/store.rs"]; /// `output/error.rs` defines `CliError::http` rather than calling it over a /// wire. diff --git a/tests/workflow.rs b/tests/workflow.rs new file mode 100644 index 0000000..c8eb052 --- /dev/null +++ b/tests/workflow.rs @@ -0,0 +1,295 @@ +//! End-to-end tests for `mapbox workflow`. +//! +//! The unit tests in `src/workflow/` cover the schema, the expressions and +//! how a step's arguments become a command line. What they cannot show is +//! what a real run does: that a command step is a real child `mapbox` whose +//! JSON reaches the next step, that a script's stdin and stdout carry values +//! between steps, that only the workflow's result lands on stdout, and what +//! `install` and `uninstall` leave on disk. +//! +//! Nothing here reaches the network. The command steps run `config list` and +//! `auth whoami`, which make no request, and every install is from a local +//! directory. + +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; + +use serde_json::{json, Value}; + +/// A token-shaped fake for account `example-user`. +const TOKEN: &str = "pk.eyJ1IjoiZXhhbXBsZS11c2VyIiwiYSI6IngifQ.SIGNATURE"; + +fn scratch(name: &str) -> PathBuf { + let home = PathBuf::from(env!("CARGO_TARGET_TMPDIR")).join(format!("workflow-{name}")); + let _ = std::fs::remove_dir_all(&home); + std::fs::create_dir_all(&home).expect("create the scratch home"); + home +} + +fn run(home: &Path, args: &[&str]) -> Output { + Command::new(env!("CARGO_BIN_EXE_mapbox")) + .env_remove("MAPBOX_ACCESS_TOKEN") + .env_remove("MapboxAccessToken") + .env_remove("MAPBOX_USERNAME") + .env_remove("MAPBOX_OUTPUT") + .env_remove("GH_TOKEN") + .env_remove("GITHUB_TOKEN") + .env("MAPBOX_NO_UPDATE_CHECK", "1") + .env("MAPBOX_QUIET", "1") + .env("HOME", home) + .env("XDG_CONFIG_HOME", home.join(".config")) + .env("MAPBOX_CONFIG_DIR", home.join(".mapbox")) + .args(args) + .output() + .expect("run mapbox") +} + +fn stdout_json(out: &Output) -> Value { + serde_json::from_slice(&out.stdout).unwrap_or_else(|e| { + panic!( + "stdout is not one JSON document ({e}):\n{}\nstderr:\n{}", + String::from_utf8_lossy(&out.stdout), + String::from_utf8_lossy(&out.stderr) + ) + }) +} + +/// Writes a workflow at `/src/beta//` and returns its path. +fn write_workflow(home: &Path, name: &str, yaml: &str, scripts: &[(&str, &str)]) -> PathBuf { + let dir = home.join("src").join("beta").join(name); + std::fs::create_dir_all(dir.join("scripts")).unwrap(); + std::fs::write(dir.join("workflow.yaml"), yaml).unwrap(); + for (file, body) in scripts { + std::fs::write(dir.join("scripts").join(file), body).unwrap(); + } + dir +} + +const DEMO: &str = r#"version: 1 +name: demo +summary: Pass values from a command to a script and back +inputs: + greeting: { type: string, default: hello } +steps: + - id: settings + command: config list + - id: whoami + command: auth whoami + - id: shout + script: shout.sh + args: ["${{ inputs.greeting }}"] + stdin: ${{ steps.settings.output[0] }} +outputs: + account: ${{ steps.whoami.output.account }} + first_key: ${{ steps.settings.output[0].key }} + shouted: ${{ steps.shout.output.shouted }} + seen: ${{ steps.shout.output.seen }} +"#; + +/// Echoes its argument upper-cased beside what it read on stdin, and says +/// something on stderr, which must not reach the result. +const SHOUT: &str = r#"read -r line +echo "shouting" >&2 +printf '{"shouted":"%s","seen":%s}\n' "$(printf %s "$1" | tr a-z A-Z)" "$line" +"#; + +fn install_demo(home: &Path) { + let dir = write_workflow(home, "demo", DEMO, &[("shout.sh", SHOUT)]); + let out = run( + home, + &["workflow", "install", dir.to_str().unwrap(), "-o", "json"], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); +} + +#[test] +fn a_run_carries_values_between_steps_and_prints_only_the_result() { + let home = scratch("run"); + install_demo(&home); + + let out = run( + &home, + &[ + "--token", + TOKEN, + "workflow", + "run", + "demo", + "-i", + "greeting=hi", + "-o", + "json", + ], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + + let first_key = json!("update-check"); + assert_eq!( + stdout_json(&out), + json!({ + "account": "example-user", + "first_key": first_key, + "shouted": "HI", + "seen": { "key": first_key, "value": true }, + }) + ); + + let stderr = String::from_utf8_lossy(&out.stderr); + assert!( + stderr.contains("[1/3] settings (mapbox config list)"), + "{stderr}" + ); + assert!( + stderr.contains("[3/3] shout (sh scripts/shout.sh)"), + "{stderr}" + ); + assert!(stderr.contains("shouting"), "{stderr}"); + assert!(stderr.contains("beta workflow"), "{stderr}"); +} + +#[test] +fn a_failing_step_stops_the_run() { + let home = scratch("fail"); + let yaml = "version: 1\nname: broken\nsummary: Fails in the middle\nsteps:\n\ + \x20 - id: first\n script: fail.sh\n\ + \x20 - id: second\n command: config list\n"; + let dir = write_workflow(&home, "broken", yaml, &[("fail.sh", "exit 3\n")]); + assert!(run(&home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); + + let out = run(&home, &["workflow", "run", "broken", "-o", "json"]); + assert_eq!(out.status.code(), Some(1)); + assert!( + out.stdout.is_empty(), + "{}", + String::from_utf8_lossy(&out.stdout) + ); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!(stderr.contains("exited with 3"), "{stderr}"); + assert!(!stderr.contains("[2/2]"), "the second step ran: {stderr}"); +} + +#[test] +fn dry_run_runs_nothing() { + let home = scratch("dry-run"); + let marker = home.join("ran"); + let yaml = "version: 1\nname: touch\nsummary: Leaves a file behind\n\ + inputs:\n path: { type: string, required: true }\nsteps:\n\ + \x20 - id: touch\n script: touch.sh\n args: ['${{ inputs.path }}']\n"; + let dir = write_workflow(&home, "touch", yaml, &[("touch.sh", "touch \"$1\"\n")]); + assert!(run(&home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); + + let path = format!("path={}", marker.display()); + let out = run( + &home, + &[ + "workflow", + "run", + "touch", + "-i", + &path, + "--dry-run", + "-o", + "json", + ], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + assert_eq!(stdout_json(&out)["steps"][0]["run"], "sh scripts/touch.sh"); + assert!(!marker.exists(), "--dry-run ran the step"); + + let out = run(&home, &["workflow", "run", "touch", "-i", &path]); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + assert!(marker.exists(), "the real run did not run the step"); +} + +#[test] +fn install_refuses_an_invalid_workflow_and_writes_nothing() { + let home = scratch("invalid"); + let yaml = "version: 1\nname: bad\nsummary: Names a command that is not one\nsteps:\n\ + \x20 - id: a\n command: styles nope\n"; + let dir = write_workflow(&home, "bad", yaml, &[("unused.sh", "")]); + + let out = run( + &home, + &["workflow", "install", dir.to_str().unwrap(), "-o", "json"], + ); + assert_eq!(out.status.code(), Some(1)); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!(stderr.contains("invalid_workflow"), "{stderr}"); + assert!(stderr.contains("scripts/unused.sh"), "{stderr}"); + assert!(!home.join(".mapbox/workflows/bad").exists()); +} + +#[test] +fn install_replaces_only_with_force_and_uninstall_removes() { + let home = scratch("lifecycle"); + install_demo(&home); + let installed = home.join(".mapbox/workflows/demo"); + assert!(installed.join("workflow.yaml").is_file()); + assert!(installed.join(".install.json").is_file()); + + let source = home.join("src/beta/demo"); + let again = run(&home, &["workflow", "install", source.to_str().unwrap()]); + assert_eq!(again.status.code(), Some(1)); + assert!(String::from_utf8_lossy(&again.stderr).contains("already installed")); + + let forced = run( + &home, + &["workflow", "install", source.to_str().unwrap(), "--force"], + ); + assert!( + forced.status.success(), + "{}", + String::from_utf8_lossy(&forced.stderr) + ); + + let listed = stdout_json(&run(&home, &["workflow", "list", "-o", "json"])); + assert_eq!(listed[0]["name"], "demo"); + assert_eq!(listed[0]["stage"], "beta"); + + let removed = run(&home, &["workflow", "uninstall", "demo", "-o", "json"]); + assert!( + removed.status.success(), + "{}", + String::from_utf8_lossy(&removed.stderr) + ); + assert!(!installed.exists()); + + let missing = run(&home, &["workflow", "run", "demo"]); + assert_eq!(missing.status.code(), Some(1)); + assert!(String::from_utf8_lossy(&missing.stderr).contains("mapbox workflow install demo")); +} + +#[test] +fn uninstall_takes_only_a_workflow_name() { + let home = scratch("escape"); + std::fs::create_dir_all(home.join("keep")).unwrap(); + for name in ["../keep", "/tmp", ".."] { + let out = run(&home, &["workflow", "uninstall", name]); + assert_eq!(out.status.code(), Some(1), "{name}"); + assert!( + String::from_utf8_lossy(&out.stderr).contains("not a workflow name"), + "{name}" + ); + } + assert!(home.join("keep").exists()); +} diff --git a/workflow/README.md b/workflow/README.md new file mode 100644 index 0000000..913a767 --- /dev/null +++ b/workflow/README.md @@ -0,0 +1,94 @@ +# Workflows + +A workflow is a named, multi-step recipe of `mapbox` commands and scripts. None ships inside the binary: `mapbox workflow install` copies one in, and `mapbox workflow run` runs only what is installed. See [docs/commands.md](../docs/commands.md#workflows) for the commands. + +Workflows and the `workflow` command are beta. The format below is `version: 1`, and may change before it is stable. + +## Layout + +``` +workflow/ + beta/ # the stage; the only one so far + copy-style/ # the workflow's name + workflow.yaml # required + scripts/ # the scripts its steps run + prepare.py + README.md # optional +``` + +- The directory name is the workflow's name: lower-case letters, digits and dashes, and the same as `name` in `workflow.yaml`. +- The stage is the parent directory. A `beta` workflow says so on stderr when it is installed or run. +- A workflow holds `workflow.yaml`, `README.md` and `scripts/`, and nothing else. +- Every file in `scripts/` is run by some step. A workflow ships only the scripts it runs. + +The same rules apply to any workflow `install` reads, from this repository, another one (`--repo OWNER/REPO`, laid out the same way) or a local directory. `every_published_workflow_is_valid` in `src/workflow/mod.rs` holds this directory to them in `cargo test`. + +## `workflow.yaml` + +```yaml +version: 1 # required; the schema version +name: copy-style # required; the directory's name +summary: Copy a style from one account to another # required; one line +description: | # optional; shown by `workflow show` + Longer text. + +inputs: # optional; given as --input KEY=VALUE + style_id: + type: string # string | number | boolean + required: true + description: ID of the style to copy + name: + type: string # not required and no default: null + zoom: + type: number + default: 12 + +steps: # required; run in order, first failure stops + - id: source # required; lower-case, digits, underscores + name: Read the style # optional; the progress line + command: styles get # a mapbox command + args: # names as `mapbox --schema` lists them + style-id: ${{ inputs.style_id }} + profile: work # global options too + use-login: true # a flag takes true or false + + - id: body + script: prepare.py # a file in scripts/ + interpreter: python3 # optional for .sh, .py and .js + args: ["--zoom", "${{ inputs.zoom }}"] + stdin: # optional; sent to the step as JSON + style: ${{ steps.source.output }} + +outputs: # optional; the default is the last step's output + id: ${{ steps.source.output.id }} +``` + +A step is a `command` or a `script`, never both. + +### Expressions + +`${{ inputs. }}` and `${{ steps..output }}`, followed by any number of `.key` and `[index]`. Nothing else: no operators and no functions. Logic belongs in a script. + +- A value that is exactly one expression keeps its JSON type, so a whole object can go to `stdin` and a number stays a number. +- An expression inside a longer string is written in as text. One that is null there is an error rather than an empty string. +- An expression may name only a declared input or an earlier step. That is checked before any step runs. + +### Command steps + +Each command step runs this `mapbox` binary again, with `--output json`, and its JSON result becomes the step's output. It resolves its token, timeouts and path encoding exactly as the same command typed by hand would, and appears in `mapbox history` on its own. + +- `args` keys are the argument names `mapbox --schema ` lists: a flag's long name, or a positional's name. Global options such as `profile`, `username` and `use-login` are accepted too. +- `output`, `quiet`, `schema` and `dry-run` belong to the runner. `token` is refused, because a token written into a workflow is a secret in a file: log in under a profile and name the `profile` instead. +- Global options given to `mapbox workflow run` (`--profile`, `--username`, `--use-login`, `--timeout`, `--yes`, `--debug`, `--token`) reach every command step that does not set its own. +- `stdin` is sent to the command, for `--data @-`. Without it, the command reads the terminal, which is where a confirmation prompt gets its answer. +- A step cannot run `mapbox workflow`. + +### Script steps + +A script runs from the installed copy of `scripts/`, in the directory `mapbox workflow run` was started from. + +- It gets `args` as its arguments and `stdin` on standard input, as JSON unless the value is a string. With no `stdin` it reads nothing. +- Whatever it writes to stdout is its output: JSON when it parses as JSON, the text otherwise. It writes progress to stderr. +- A non-zero exit stops the workflow. +- `MAPBOX_CLI` is the path to this `mapbox` binary, for a script that runs commands of its own. `MAPBOX_WORKFLOW_ROOT` is the installed workflow's directory. +- The interpreter must be installed on the machine. `.sh` runs under `sh`, `.py` under `python3` and `.js` under `node`, and any other extension needs `interpreter`. diff --git a/workflow/beta/copy-style/README.md b/workflow/beta/copy-style/README.md new file mode 100644 index 0000000..f3f22da --- /dev/null +++ b/workflow/beta/copy-style/README.md @@ -0,0 +1,21 @@ +# copy-style + +Copies a style from one Mapbox account to another. + +```sh +mapbox auth login --profile source +mapbox auth login --profile target +mapbox workflow install copy-style +mapbox workflow run copy-style \ + --input style_id=