diff --git a/CHANGELOG.md b/CHANGELOG.md index 5e92da1..42322de 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,47 @@ that may never merge. They are not releases and are not listed here. ## Unreleased +- `mapbox workflow` (beta): install and run workflows, which are named, + multi-step recipes of + `mapbox` commands and scripts defined in a `workflow.yaml`. `install` copies one from a local directory or from a + GitHub repository's `workflow//` (by default `mapbox/mapbox-cli`; + `GH_TOKEN`/`GITHUB_TOKEN` for a private one) into `~/.mapbox/workflows/`, + recording the commit a GitHub ref resolved to, + and `list`, `show`, `run` and `uninstall` work on what is installed. + `run` takes each of the workflow's inputs as a flag, as in + `mapbox workflow run copy-style --style-id --from-profile source + --to-profile target`. `run --dry-run` runs only the steps a workflow + marks `dry_run`, which write nothing, so the plan can show real data. At a + terminal each step shows a spinner, the details its script reports on + stderr (a `::progress ` line updates the spinner's text, a `::warn ` line + is a warning, listed again at the end), and `✓` or `✗` with the time it + took; off a terminal, each step is a plain `[n/total]` line followed by + its details. `--quiet` drops the details but keeps warnings. A workflow's + optional `result` template is what text mode prints; `-o json` prints its + outputs. A command step's `save: ` keeps its stdout, such as a + style's ZIP, as a file for later steps, in an owner-only working directory + removed when the run ends, Ctrl-C included (the run then fails with + `cancelled`); a command step may be marked `dry_run` only when its command + is known to change nothing: an API `GET`, or a local read such as + `config list`. The + generated agent skill leaves `workflow` out. A script step needs its interpreter on `PATH`: on Windows, + `.py` needs a `python3`, which the python.org installer does not add. + A workflow may name its own `version`, independent of the CLI's, and the + oldest CLI it `requires`; `install` refuses one that needs a newer CLI. + `install` lists the scripts a workflow runs, and from a repository other + than `mapbox/mapbox-cli`, or a `--ref` not on its `main`, asks first, + refusing with `confirmation_required` where there is no terminal and no + typed `--yes` (`MAPBOX_YES` does not answer it). A failed command step's + own error is under `cause` in the run's `workflow_failed`, and on Unix a + SIGTERM to the run is passed on to the running step. + The first one published is + `copy-style`, which copies a style between accounts with its custom fonts + and icons, and on a partial failure lists what it created and the + commands that remove it. The command, the `schema: 1` format and the published + workflows may change or be removed in any release, and every `workflow` + subcommand says so on stderr. Nothing changes for a script that does not + use them. + ## 1.0.0 - 2026-10-22 - New `mapbox report ""` sends feedback about the CLI itself to diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4e69488..6eaf3ff 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -100,9 +100,9 @@ or given a new default; an exit code or stdout/stderr change; a `--schema` field removed or repurposed. Not breaking: new commands, new API response fields, new `--schema` fields, and any wording. -A command marked beta is outside this promise until the label is removed: -it says so in its help and on every run, and it may change or be removed in -any release. +A command marked beta, such as `mapbox workflow`, is outside this promise +until the label is removed: it says so in its help and on every run, and it +may change or be removed in any release. Deprecated commands and endpoints print a warning before the request goes out, and are marked `[deprecated]` in `--help` and `--schema`. diff --git a/Cargo.lock b/Cargo.lock index 134718b..c395af2 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -97,6 +97,15 @@ dependencies = [ "generic-array", ] +[[package]] +name = "block2" +version = "0.6.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cdeb9d870516001442e364c5220d3574d2da8dc765554b4a617230d33fa58ef5" +dependencies = [ + "objc2", +] + [[package]] name = "bumpalo" version = "3.20.3" @@ -241,6 +250,17 @@ dependencies = [ "typenum", ] +[[package]] +name = "ctrlc" +version = "3.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e0b1fab2ae45819af2d0731d60f2afe17227ebb1a1538a236da84c93e9a60162" +dependencies = [ + "dispatch2", + "nix", + "windows-sys 0.61.2", +] + [[package]] name = "digest" version = "0.10.7" @@ -272,6 +292,18 @@ dependencies = [ "windows-sys 0.48.0", ] +[[package]] +name = "dispatch2" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e0e367e4e7da84520dedcac1901e4da967309406d1e51017ae1abfb97adbd38" +dependencies = [ + "bitflags", + "block2", + "libc", + "objc2", +] + [[package]] name = "displaydoc" version = "0.2.7" @@ -790,9 +822,11 @@ dependencies = [ "base64 0.22.1", "clap", "clap_complete", + "ctrlc", "dirs", "flate2", "jsonc-parser", + "nix", "open", "rand", "reqwest", @@ -856,6 +890,33 @@ version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "743fb55ba31b18fb1ecef6bdc9aa2743314978ac084044301a7eee33fb99a20d" +[[package]] +name = "nix" +version = "0.31.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d" +dependencies = [ + "bitflags", + "cfg-if", + "cfg_aliases", + "libc", +] + +[[package]] +name = "objc2" +version = "0.6.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08849bbd4767dfae9457696856ae1c84fe4e0281bbe4a7abff2d0e06fb7981f8" +dependencies = [ + "objc2-encode", +] + +[[package]] +name = "objc2-encode" +version = "4.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ef25abbcd74fb2609453eb695bd2f860d389e457f67dc17cafc8b8cbc89d0c33" + [[package]] name = "once_cell" version = "1.21.4" diff --git a/Cargo.toml b/Cargo.toml index f17a08f..6de3457 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -114,6 +114,19 @@ jsonc-parser = { version = "0.34.0", features = ["serde", "serde_json"] } # OSC 11 is known at once instead of after the timeout. Asked only by a run # that shows help; see src/output/theme.rs. terminal-colorsaurus = "1" +# Lets `workflow run` outlive a Ctrl-C, SIGTERM or SIGHUP long enough to remove its working +# directory, which a process killed by SIGINT never gets to do. Installing +# a signal handler takes `unsafe` this crate does not have. Installed by +# that command only; every other command keeps the default. +ctrlc = { version = "3", features = ["termination"] } + +[target.'cfg(unix)'.dependencies] +# For `workflow run`: passing a SIGTERM on to the running step, which a +# terminal's Ctrl-C reaches by itself and a `kill` does not. Sending a signal +# takes `unsafe` this crate does not have. Already in the graph through +# `ctrlc`; the same version. `process` and `term` for telling a Ctrl-C at the +# terminal, which reached the step already, from a `kill`. +nix = { version = "0.31", default-features = false, features = ["process", "signal", "term"] } [target.'cfg(windows)'.dependencies] # For `src/detach.rs`: keeping the parent's standard handles out of a detached diff --git a/README.md b/README.md index bba364a..d0c5e84 100644 --- a/README.md +++ b/README.md @@ -25,6 +25,7 @@ mapbox styles list - [Diagnostics and settings](#diagnostics-and-settings) - [Shell completion](#shell-completion) - [Tileset CLI](#tileset-cli) + - [Workflows (beta)](#workflows-beta) - [For AI agents](#for-ai-agents) - [Quick start: Mapbox agent setup](#quick-start-mapbox-agent-setup) - [Agent skills](#agent-skills) @@ -388,6 +389,17 @@ apply here; `tilesets` has its own flags, so use `--force`/`-f` for its prompts. If a tileset command answers for the wrong account, `mapbox auth whoami` shows which token is in play. +### Workflows (beta) + +```sh +mapbox workflow install copy-style +mapbox workflow run copy-style --style-id --from-profile source --to-profile target +``` + +A workflow is a named, multi-step recipe of `mapbox` commands and scripts. +`mapbox workflow` is beta and may change or be removed in any release. See +[Workflows](./docs/commands.md#workflows). + ## For AI agents ### Quick start: Mapbox agent setup diff --git a/docs/commands.md b/docs/commands.md index e6958ac..725c3c4 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -141,6 +141,12 @@ nests, and is typed `mapbox styles draft get`. **[Usage](#usage)** — [usage](#mapbox-usage) +**[Workflows](#workflows)** — [workflow.list](#mapbox-workflow-list) · +[workflow.show](#mapbox-workflow-show) · +[workflow.install](#mapbox-workflow-install) · +[workflow.uninstall](#mapbox-workflow-uninstall) · +[workflow.run](#mapbox-workflow-run) + **[Tilesets CLI](#tilesets-cli)** — [tilesets-cli](#mapbox-tilesets-cli-args) Then [Errors](#errors) — the shape a failure takes in each mode. @@ -561,7 +567,7 @@ either. | `--profile ` | Which stored credentials to use. | | `--output`, `-o` | `auto` \| `text` \| `json`. | | `--id ` | On a command that returns a list, print just the row with that `id` or `name`. | -| `--quiet`, `-q` | Don't print the `mapbox · v` banner, which goes to stderr and only when stderr is a terminal. Also `MAPBOX_QUIET`. | +| `--quiet`, `-q` | Don't print the `mapbox · v` banner, which goes to stderr and only when stderr is a terminal, the note after a download, or a workflow step's details. Also `MAPBOX_QUIET`. | | `--no-color` | Turn off color in help, errors and text output. Same as `NO_COLOR`; wins over `FORCE_COLOR`. | | `--timeout ` | How long one request may take, connection included. Defaults to 60 seconds, or 900 for a body read from `--file` or from a `--data @`/`@-`. Also `MAPBOX_TIMEOUT`. | @@ -5101,6 +5107,336 @@ list drops the `--daily` suggestion once it's already in effect. --- +## Workflows + +A workflow is a named, multi-step recipe of `mapbox` commands and scripts, +defined in a `workflow.yaml`. The format and the rules a workflow directory +follows are in [workflow/README.md](../workflow/README.md). + +**Beta.** The commands, the `schema: 1` format and the published workflows +may change or be removed in any release. Every `mapbox workflow` subcommand opens with this line on +stderr: + +``` +Beta: `mapbox workflow` may change or be removed in any release. +``` + +In text mode, `list`, `show` and `install` end with tips on stderr, naming +the command to run next. `mapbox generate-skills` leaves the `workflow` +commands out of the skill it writes, so that an agent is not taught a command +nobody should rely on yet. + +None ships inside the binary. `install` copies one into +`~/.mapbox/workflows//`, and every other subcommand works on what is +installed there. + +**On Windows**, CI runs workflows with command steps and `sh` script steps, +but no published workflow has been run there. A script step runs the +interpreter its extension names, and Windows ships none of them: `.sh` needs +an `sh` on `PATH`, such as Git Bash's, and `.py` needs `python3`, which the +python.org installer does not add (it adds `python` and `py`). So +`copy-style` runs on Windows only where `python3` is on `PATH`. A missing +interpreter fails the step with `` `python3` is not installed or not on +PATH ``. + +--- + +### `mapbox workflow list` + +Lists the installed workflows. One that no longer loads, because a file was +edited by hand or this CLI no longer reads its format, is listed with its +error instead of a summary. + +#### Outputs + + + + +
textjson
+ +``` +NAME VERSION SUMMARY +copy-style 0.1.0 Copy a style, with its fonts and icons, from one account to another +``` + + + +```json +[ + { + "commit": "4c8dd090573afc073465ab66f105088156cf92cb", + "name": "copy-style", + "path": "/Users/me/.mapbox/workflows/copy-style", + "source": "github:mapbox/mapbox-cli@main", + "summary": "Copy a style, with its fonts and icons, from one account to another", + "version": "0.1.0" + } +] +``` + +
+ +--- + +### `mapbox workflow show` + +Describes an installed workflow, in the same layout for every one: its name +and summary, its description, its inputs with their types and defaults, its +steps, and the installed copy's version, the CLI it requires, and where it +came from. When a step names a command or +an argument this build no longer has, the problems are listed after the steps +and under `problems`. Script steps are marked as running as you, and each +step's `kind` is `command` or `script`. How a description is written is in +[workflow/README.md](../workflow/README.md#description). + +#### Parameters + +| Parameter | Effect | +| --- | --- | +| `NAME` | Name of an installed workflow. | + +--- + +### `mapbox workflow install` + +Installs a workflow from GitHub or from a local directory. Everything is read +and checked before anything is written: the layout, the `workflow.yaml`, and +every command step against this build's commands. A workflow that fails any +of these is `invalid_workflow`, with every problem listed at once. So is one +whose `requires` names a newer CLI than this one, with that as the only +problem. + +#### Parameters + +| Parameter | Effect | +| --- | --- | +| `SOURCE` | A workflow name, looked up in the repository's `workflow//`, or a path to a local workflow directory: anything with a `/` or starting with `.`. | +| `--repo` | GitHub repository to install from, as `OWNER/REPO`. Defaults to `mapbox/mapbox-cli`. | +| `--ref` | Branch, tag or commit to install from. Defaults to `main`. | +| `--force` | Replace a workflow that is already installed. | +| `--dry-run` | Check the workflow and list the files it would write, then exit. | + +The ref is resolved to a commit first, and that commit is read as one tarball +through the GitHub API, so a branch that moves in between cannot mix the two. +The commit is recorded beside the workflow in `.install.json` and reported as +`commit` by `install`, `list` and `show` (`null` for a local install). +`mapbox/mapbox-cli` is public and needs no token. For a private repository +named with `--repo`, set `GH_TOKEN` or `GITHUB_TOKEN`. It is sent only to +`api.github.com`. Without one, a private repository answers 404, +exactly as a ref that does not exist does, and the error says both. + +**An installed workflow stops the install** unless `--force` is given. The new +copy is staged and renamed into place, and the old one is moved aside until +then and moved back if that fails, so a failed install leaves the old copy. +One interrupted between the two moves leaves the old copy aside, and the next +`mapbox workflow` command puts it back. Only regular files are taken. A symlink in a local +directory is refused, and an archive entry whose path would leave the +workflow's directory stops the read. + +**Only install a workflow you trust.** Its steps run as you, with your +credentials and every stored profile, and a script step runs whatever its +interpreter does. `install` lists the scripts a workflow runs. From +`mapbox/mapbox-cli` or a local directory it installs without asking, and so +does a `--ref` whose commit is on `mapbox/mapbox-cli`'s `main`. Anything else, +including another `--repo`, a pull request's `pull/N/head` or a commit off +`main`, asks first at a terminal; without one it refuses with +`confirmation_required` unless `--yes` is given. `MAPBOX_YES` does not answer +this question; only the typed flag does. + +#### Examples + +```sh +mapbox workflow install copy-style + +mapbox workflow install copy-style --ref workflow-copy-style-0.1.0 + +export GITHUB_TOKEN="$(gh auth token)" +mapbox workflow install sync-tilesets --repo my-org/private-workflows + +mapbox workflow install ./workflow/copy-style --force +``` + +#### Outputs + + + + +
textjson
+ +``` +Installed copy-style + +Version 0.1.0 +Source ~/dev/cli/workflow/copy-style +Path ~/.mapbox/workflows/copy-style +Files README.md, scripts/copy_style.py, workflow.yaml +Scripts python3 scripts/copy_style.py +``` + + + +```json +{ + "commit": null, + "dry_run": false, + "files": [ + "README.md", + "scripts/copy_style.py", + "workflow.yaml" + ], + "name": "copy-style", + "path": "/Users/me/.mapbox/workflows/copy-style", + "scripts": [ + "python3 scripts/copy_style.py" + ], + "source": "/Users/me/dev/cli/workflow/copy-style", + "version": "0.1.0" +} +``` + +
+ +--- + +### `mapbox workflow uninstall` + +Removes an installed workflow's directory. At a terminal it asks first, and +`--yes` skips the question. Only a directory `install` wrote under +`~/.mapbox/workflows/` is removed. + +#### Parameters + +| Parameter | Effect | +| --- | --- | +| `NAME` | Name of an installed workflow, or the directory it was installed from, which names it by its directory name. The directory itself is not touched. | +| `--dry-run` | Say what it would remove, then exit without removing it. | + +--- + +### `mapbox workflow run` + +Runs an installed workflow's steps in order. The first step that fails stops +the run with `workflow_failed`, naming the step, and exit code 1. A failed +command step's own error is under `cause`, and its message, status and advice +are the run's. On Unix, a SIGTERM to the run is passed on to the running step +as an interrupt after two seconds, so a hung step does not outlive a CI +timeout. It reaches the step's own process only, so a shell step waiting on a +long command of its own runs until that command ends. A run that is the +terminal's foreground passes nothing on, since a Ctrl-C there reached the +step already. + +#### Parameters + +| Parameter | Effect | +| --- | --- | +| `NAME` | Name of an installed workflow. | +| `--` | One flag per input the workflow declares, spelled with dashes (`style_id` is `--style-id`). Required, typed and defaulted as its `workflow.yaml` says. `mapbox workflow run --help` lists them. | +| `--dry-run` | Check the workflow and its inputs and print the plan. Runs only the steps marked `dry_run`, and skips the rest. A marked script step really runs, as you and with your credentials. | + +The flags come from the installed workflow's definition, read before the +command line is parsed, so a missing input, a misspelled flag or a value of the +wrong type is a usage error (exit 2), as on any other command. They are not in +`--schema` or in the shell completion, which describe the command tree without +reading `~/.mapbox/workflows/`. `workflow show` lists them for any installed +workflow. + +**stdout holds only the result**: the workflow's `outputs`, or the last +step's output if it declares none, rendered like any other result. Each +step's progress and anything a step writes to stderr go to stderr. At a +terminal, each step shows its title, the details its script reports, and a +spinner that becomes `✓` or `✗` with the time it took; anywhere else, each +step is one `[n/total]` line followed by its details, with no escape codes. +`--quiet` keeps each step's title, how it ended and its warnings, and drops +its details. `workflow/README.md` describes how a script reports progress, +details and warnings. + +A workflow that declares `result` prints it in text mode instead of the +outputs as fields; `-o json` always prints the outputs. + +Each command step is this binary run again with `--output json`, so it +resolves its token and applies its timeouts as the same command typed by +hand would, and appears in `mapbox history` as its own run. The globals given +to `workflow run` (`--profile`, `--username`, `--use-login`, `--timeout`, +`--yes`, `--debug`, `--token`) reach every command step that does not set its +own. A token typed as `--token` goes to the step's environment, not its +command line. + +#### Examples + +```sh +mapbox workflow run copy-style \ + --style-id cmm28c5rm00bj01qz9hwp69qc \ + --from-profile source \ + --to-profile target + +mapbox workflow run copy-style --style-id cmm28c5rm00bj01qz9hwp69qc \ + --from-profile source --to-profile target --dry-run +``` + +`--dry-run` prints the plan: the inputs as they were read, and each step with +its arguments as written. A step marked `dry_run` in its `workflow.yaml` runs +for real, so the plan can show what the rest would do with real data: a +command step only if its command is known to change nothing (an API command +that sends a `GET`, or a local read such as `config list`; never +`tilesets-cli`), and a script step with +`MAPBOX_WORKFLOW_DRY_RUN=1` in its environment and the promise to write +nothing. The CLI cannot check that promise: a script step runs as you, with +your credentials, in a dry run as in a real one. Their outputs are under `results` in `-o json`. Every other step is +skipped. `copy-style` marks all of its steps, so its dry run downloads the +style and lists what it would upload: + +``` +$ mapbox workflow run copy-style --style-id cmums8rlh000301s96498hnju \ + --from-profile default --to-profile default --dry-run +🗺️ mapbox · v0.3.0 +──────────────────────────────────────── +Beta: `mapbox workflow` may change or be removed in any release. + +▸ Check the source login mapbox auth status + ✓ 0.0s + +▸ Check the target login mapbox auth status + ✓ 0.0s + +▸ Download the style mapbox styles download + Saved style.zip (965 KB) + ✓ 0.3s + +▸ List the target account's fonts mapbox fonts list + ✓ 0.1s + +▸ Plan the copy + Found 561 icons and 1 custom font + Font Yellow Banana Regular is already in zhuwenlong; it will be skipped + ✓ 0.2s + +▸ Copy the fonts, the style and its icons + Would copy the style into zhuwenlong as "CLI copy test: Helsinki Evening · CLI Blog Demo": + skip font Yellow Banana Regular (already in zhuwenlong) + create the style + upload 561 icons in 23 batches + point its sprite and glyphs at zhuwenlong + ✓ 0.1s + +Dry run — only the steps marked dry_run ran. Would run copy-style: + +Inputs + from_profile default + name (none) + style_id cmums8rlh000301s96498hnju + to_profile default + +Steps + 1. Check the source login mapbox auth status (ran in this dry run) + 2. Check the target login mapbox auth status (ran in this dry run) + 3. Download the style mapbox styles download (ran in this dry run) + 4. List the target account's fonts mapbox fonts list (ran in this dry run) + 5. Plan the copy python3 scripts/copy_style.py (ran its script in this dry run) + 6. Copy the fonts, the style and its icons python3 scripts/copy_style.py (ran its script in this dry run) +``` + +--- + ## Tilesets CLI ### `mapbox tilesets-cli ` @@ -5303,10 +5639,21 @@ faults and are not: | `invalid_file` | A file could not be read: one named by `--file`, or one named by `--data @`. Also a `@` that is not valid UTF-8, which a JSON body has to be. | | `binary_response` | The response was bytes and stdout is a terminal. Redirect it to a file. | | `missing_subcommand` | A command group was named with no operation. | -| `cancelled` | A delete was declined at the confirmation prompt. Nothing was sent. | +| `cancelled` | A delete was declined at the confirmation prompt, and nothing was sent; or `workflow run` was interrupted (Ctrl-C, or on Unix SIGTERM or SIGHUP), and the steps after the current one did not run. | | `cli_token_unavailable` | A command that prefers the CLI's own token (`report`) could not use it — it could not be fetched, or it is revoked and no new one is published yet — and you have no token of your own to fall back to. | | `cli_token_forbidden` | This API refused the CLI's own token, even after fetching it afresh, and you have no token of your own to fall back to. | -| `usage` | Anything else clap rejected. Exit code 2, not 1. | +| `usage` | Anything else clap rejected, or flags a command refuses together, such as `workflow install --ref` with a local directory. Exit code 2, not 1. | +| `confirmation_required` | `workflow install` from a source the CLI does not vouch for (another repository, or a `--ref` not on `mapbox/mapbox-cli`'s `main`), with no terminal to ask on and no typed `--yes`. Nothing was installed. | +| `invalid_workflow` | A workflow failed `install`'s checks, or no longer passes them at `show` or `run`: its layout, its `workflow.yaml`, a command step against this build's commands, or a `requires` naming a newer CLI. Every problem is listed. | +| `workflow_failed` | A step of `workflow run` failed, and the steps after it did not run. When a command step failed, `cause` holds that command's own error document, so its `code` (an `http_`, say) can still be branched on. | +| `workflow_not_found` | The repository has no `workflow//`. The message lists the workflows it does have. | +| `already_installed` | `workflow install` or `agent-skills install` would replace what is already there. `--force` replaces a workflow. | +| `workflow_not_installed` | `show`, `run` or `uninstall` named a workflow that is not installed. | +| `invalid_name` | A workflow name that is not lower-case letters, digits and dashes. | +| `invalid_repo` | `--repo` is not `OWNER/REPO`. | +| `invalid_ref` | `--ref` holds characters a branch, tag or commit cannot. | +| `invalid_input` | A workflow input's value does not fit its type. | +| `unsafe_archive` | A path in the downloaded archive would leave the workflow's directory. Nothing was installed. | | `error` | Unclassified. | Exit codes: `0` success, `1` runtime failure, `2` usage error. diff --git a/src/auth.rs b/src/auth.rs index a0c6a61..c16f754 100644 --- a/src/auth.rs +++ b/src/auth.rs @@ -882,6 +882,28 @@ pub fn with_auth_fix( } } +/// [`with_auth_fix`]'s advice for a workflow step whose run was given +/// `--token`. The step is handed that token as [`CLAP_TOKEN_ENV`], so the +/// advice it wrote blames the environment; this puts the flag back. +pub fn with_typed_token_fix(err: anyhow::Error) -> anyhow::Error { + let mut cli = match err.downcast::() { + Ok(cli) => cli, + Err(other) => return other, + }; + let remedy = match cli.status { + Some(401) => remedy_for(Some(TokenSource::Flag), CLAP_TOKEN_ENV, false), + Some(403) => match remedy::missing_scope(&cli.message).map(str::to_owned) { + Some(scope) => scope_remedy_for(Some(TokenSource::Flag), CLAP_TOKEN_ENV, false, &scope), + None => return cli.into(), + }, + _ => return cli.into(), + }; + cli.fix = None; + cli.next_actions.clear(); + cli.docs.clear(); + cli.with_remedy(remedy).into() +} + /// The advice for a credential the API refused: the prose, one command to /// run, and where tokens are documented. /// diff --git a/src/confirm.rs b/src/confirm.rs index 3edfb43..57faf0d 100644 --- a/src/confirm.rs +++ b/src/confirm.rs @@ -12,6 +12,10 @@ //! `--yes` says the answer is yes before it is asked, which is the only way a //! caller *at* a terminal can get the non-interactive behavior on purpose. //! +//! One question is the exception, and refuses rather than proceeds when nobody +//! can answer it: installing a workflow from a repository the CLI does not +//! vouch for. See [`untrusted_install`]. +//! //! The question is deliberately not `--output`-aware. It goes to stderr as //! prose in both modes, because the only way to see it is to be a person at a //! terminal, and a person reads prose. stdout still carries nothing but the @@ -89,12 +93,12 @@ fn is_yes(answer: &str) -> bool { /// EOF part-way through the question, which is what a piped stdout at a /// terminal produces when nothing answers. The code is stable so a caller can /// branch on it instead of matching the prose. -fn cancelled() -> anyhow::Error { +fn cancelled(outcome: &str) -> anyhow::Error { // No `next_actions`, deliberately. The only command that would answer // this is the delete that was just declined, with `--yes` bolted on — // handing that back to whoever typed `n` is the one suggestion this CLI // must not make. The flag is named in the prose, where a person decides. - CliError::new("cancelled", "Cancelled — nothing was sent.") + CliError::new("cancelled", format!("Cancelled — {outcome}.")) .with_remedy( Remedy::default().with_fix("Pass --yes (or set MAPBOX_YES=1) to skip this question."), ) @@ -119,14 +123,14 @@ pub fn destructive_request(method: &str, url: &str, assume_yes: bool) -> Result< // No "this cannot be undone": it is not true of every delete here, and a // warning the reader can catch out is worse than none. The method and the // resource are the part that lets them catch out the *command*. - ask(&format!("About to {method} {url}")) + ask(&format!("About to {method} {url}"), "nothing was sent") } /// Asks a yes/no question on stderr/stdin, for a destructive local action /// that (unlike `destructive_request`) is not an HTTP request — `uninstall`, /// so far. The caller has already decided the question is worth asking /// (typically via [`decide`]); this is just the I/O. -fn ask(question: &str) -> Result<()> { +fn ask(question: &str, outcome: &str) -> Result<()> { let mut stderr = std::io::stderr(); // Ignored deliberately: a broken stderr cannot be reported *on* stderr, // and the read below will fail too, which is where the refusal comes from. @@ -139,7 +143,7 @@ fn ask(question: &str) -> Result<()> { // Ok(0) is EOF — the terminal went away mid-question. Treated as a // no, like every other answer that is not a yes. Ok(_) if is_yes(&answer) => Ok(()), - _ => Err(cancelled()), + _ => Err(cancelled(outcome)), } } @@ -153,12 +157,65 @@ pub fn destructive_local_action(question: &str, assume_yes: bool) -> Result<()> if assume_yes || !(std::io::stdin().is_terminal() && std::io::stderr().is_terminal()) { return Ok(()); } - ask(question) + ask(question, "nothing was sent") +} + +/// Whether `--yes` was typed, rather than set as [`ENV`]. +/// +/// [`untrusted_install`] takes only this. A CI job that exports +/// `MAPBOX_YES=1` so its deletes go through has said nothing about which +/// repositories' code it trusts. +pub fn typed_yes(matches: &clap::ArgMatches) -> bool { + matches.value_source(ARG) == Some(clap::parser::ValueSource::CommandLine) +} + +/// Asks before installing a workflow from a source the CLI does not vouch +/// for, and refuses when there is nobody to ask. +/// +/// Everywhere else no terminal means go ahead. Here that would install code +/// that later runs as the caller, with their credentials, on nobody's +/// say-so — a mistyped `--repo` in CI would be enough. So without a terminal +/// it takes `--yes`, typed: see [`typed_yes`]. +pub fn untrusted_install(question: &str, typed_yes: bool) -> Result<()> { + let at_terminal = std::io::stdin().is_terminal() && std::io::stderr().is_terminal(); + match untrusted_decision(typed_yes, at_terminal) { + Some(Decision::Proceed) => Ok(()), + Some(Decision::Ask) => ask(question, "nothing was installed"), + None => Err(CliError::new( + "confirmation_required", + format!("{question}\nNothing was installed: there is no terminal to confirm it on."), + ) + .with_remedy(Remedy::default().with_fix( + "Pass --yes once you trust what it runs. MAPBOX_YES does not answer this one.", + )) + .into()), + } +} + +/// [`untrusted_install`]'s choice, with no I/O in it: `None` is a refusal. +fn untrusted_decision(typed_yes: bool, at_terminal: bool) -> Option { + if typed_yes { + Some(Decision::Proceed) + } else if at_terminal { + Some(Decision::Ask) + } else { + None + } } #[cfg(test)] mod tests { - use super::{decide, destructive_local_action, is_yes, Decision}; + use super::{decide, destructive_local_action, is_yes, untrusted_decision, Decision}; + + /// Unlike a delete, no terminal is a refusal: code from an unvetted + /// repository is not installed on nobody's say-so. + #[test] + fn an_untrusted_install_needs_yes_or_someone_to_ask() { + assert_eq!(untrusted_decision(true, false), Some(Decision::Proceed)); + assert_eq!(untrusted_decision(true, true), Some(Decision::Proceed)); + assert_eq!(untrusted_decision(false, true), Some(Decision::Ask)); + assert_eq!(untrusted_decision(false, false), None); + } /// The whole point of the flag: it makes a terminal behave like CI. #[test] diff --git a/src/generate_skills.rs b/src/generate_skills.rs index a192a21..f53f2b8 100644 --- a/src/generate_skills.rs +++ b/src/generate_skills.rs @@ -135,8 +135,21 @@ pub fn command() -> Command { .arg(executor::dry_run_arg(DRY_RUN_HELP)) } +/// `--schema`'s walk, without the commands a skill must not teach. +/// +/// `workflow` is beta and may change or be removed in any release, and an +/// agent that reads about a command in its skill takes it as one to reach +/// for. `--schema` still lists it, with the notice in its description. +fn skill_schema(app: &Command, specs: &[ServiceSpec]) -> schema::Schema { + let mut schema = schema::build(app, specs, &[]); + schema + .commands + .retain(|entry| entry.service != crate::workflow::COMMAND); + schema +} + pub fn run(app: &Command, specs: &[ServiceSpec], matches: &ArgMatches, mode: Mode) -> Result<()> { - let schema = schema::build(app, specs, &[]); + let schema = skill_schema(app, specs); let header = Header::new(requested_services(&schema, matches)?); let files = render(app, &schema, &header); @@ -1475,7 +1488,7 @@ mod tests { fn generated() -> (Command, Vec, Vec) { let specs = specs(); let app = crate::build_app(&specs); - let schema = schema::build(&app, &specs, &[]); + let schema = skill_schema(&app, &specs); let header = Header::new(None); let files = render(&app, &schema, &header); (app, specs, files) @@ -1574,10 +1587,18 @@ mod tests { // The same walk `schema::every_command_in_the_tree_is_described` // does, for the same reason: the command tree is the authority on // what a command is. - let mut runnable = crate::runnable_commands(&app); + // Less `workflow`, which `skill_schema` leaves out on purpose. + let mut runnable: Vec = crate::runnable_commands(&app) + .into_iter() + .filter(|command| !command.starts_with("mapbox workflow ")) + .collect(); runnable.sort(); assert_eq!(described, runnable); + assert!( + !described.iter().any(|command| command.contains("workflow")), + "the skill teaches `workflow`, which is beta" + ); let mut unique = described.clone(); unique.dedup(); @@ -1731,7 +1752,7 @@ mod tests { // And through the real renderer, where a wrapper that swallowed part // of the body would still pass the two assertions above. let (app, specs, files) = generated(); - let schema = schema::build(&app, &specs, &[]); + let schema = skill_schema(&app, &specs); let real_body = render_body(&app, &schema, &by_service(&schema, &header), &header); for name in ["SKILL.md", "AGENTS.md"] { let file = files @@ -1844,7 +1865,7 @@ mod tests { fn a_service_filter_narrows_the_skill_and_is_recorded() { let specs = specs(); let app = crate::build_app(&specs); - let schema = schema::build(&app, &specs, &[]); + let schema = skill_schema(&app, &specs); let header = Header::new(Some(vec!["styles".to_string()])); let files = render(&app, &schema, &header); @@ -1881,7 +1902,7 @@ mod tests { fn an_unknown_service_is_an_error_that_lists_the_real_ones() { let specs = specs(); let app = crate::build_app(&specs); - let schema = schema::build(&app, &specs, &[]); + let schema = skill_schema(&app, &specs); let matches = app .clone() @@ -2023,7 +2044,7 @@ mod tests { fn positionals_keep_their_order_and_flags_are_sorted() { let (_, specs, _) = generated(); let app = crate::build_app(&specs); - let schema = schema::build(&app, &specs, &[]); + let schema = skill_schema(&app, &specs); let mut checked = 0usize; for entry in &schema.commands { @@ -2062,7 +2083,7 @@ mod tests { fn a_table_names_a_value_rather_than_suggesting_one() { let (_, specs, files) = generated(); let app = crate::build_app(&specs); - let schema = schema::build(&app, &specs, &[]); + let schema = skill_schema(&app, &specs); let output = schema .global_options @@ -2110,7 +2131,7 @@ mod tests { fn a_description_is_cut_only_when_it_would_not_fit() { let (_, specs, _) = generated(); let app = crate::build_app(&specs); - let schema = schema::build(&app, &specs, &[]); + let schema = skill_schema(&app, &specs); let output = schema .global_options diff --git a/src/help_layout.rs b/src/help_layout.rs index 4e58596..f1c7757 100644 --- a/src/help_layout.rs +++ b/src/help_layout.rs @@ -51,6 +51,7 @@ const GROUPS: &[(&str, &[&str])] = &[ "doctor", "completion", "tilesets-cli", + "workflow", "uninstall", "report", "help", @@ -95,6 +96,7 @@ const DESCRIPTIONS: &[(&str, &str)] = &[ "Areas reachable from a point within a time or distance", ), ("map-match", "Snap a GPS trace to the road network"), + ("workflow", "Install and run multi-step workflows (beta)"), ("accounts", "List access tokens and their scopes"), ( "generate-skills", diff --git a/src/main.rs b/src/main.rs index c40f694..36c147f 100644 --- a/src/main.rs +++ b/src/main.rs @@ -55,6 +55,7 @@ mod telemetry_sink; mod tilesets_cli; mod uninstall; mod update_check; +mod workflow; use output::{CliError, Mode}; use remedy::Remedy; @@ -598,7 +599,7 @@ fn build_app(specs: &[ServiceSpec]) -> Command { .value_parser(FalseyValueParser::new()) .help_heading(OUTPUT_OPTIONS) .global(true) - .help("Hide the version banner and download notes"), + .help("Hide the version banner, download notes and workflow step details"), ) .arg( Arg::new(http::TIMEOUT_ARG) @@ -799,6 +800,7 @@ fn build_app(specs: &[ServiceSpec]) -> Command { app = app.subcommand(account_usage::command()); + app = app.subcommand(workflow::command()); app.subcommand(tilesets_cli::command()) } @@ -920,7 +922,9 @@ fn cli() -> u8 { output::banner::show_before_help(&raw_argv); } - let mut app = build_app(&specs); + // Before the parse, since a workflow's inputs are flags only once its + // definition has been read. Everything else sees the tree unchanged. + let mut app = workflow::with_run_target(build_app(&specs), &raw_argv); // Here rather than in `build_app`, which tests call directly and which // should not depend on who happens to be running them. if agent_detect::detect_agent().is_some() { @@ -967,7 +971,17 @@ fn cli() -> u8 { Ok(()) => 0, Err(e) => { output::emit_error(mode, &e); - 1 + // A command line only the command itself can judge, such as two + // flags that conflict for one kind of value, is a usage error + // like any clap rejects. + let usage = e + .downcast_ref::() + .is_some_and(|e| e.code == "usage"); + if usage { + 2 + } else { + 1 + } } } } @@ -1428,6 +1442,19 @@ fn run(app: &Command, specs: &[ServiceSpec], matches: &ArgMatches, mode: Mode) - Some(("show", show_matches)) => history::show(show_matches, mode)?, _ => unreachable!("`history` sets subcommand_required(true)"), }, + // Ahead of the generic service arm too. Its own requests go to + // GitHub, and each command step is a child `mapbox` that resolves + // its credentials itself — so nothing is loaded here. + Some((workflow::COMMAND, workflow_matches)) => workflow::run( + app, + workflow_matches, + workflow::RunFlags { + globals: matches, + debug, + assume_yes, + }, + mode, + )?, // Also ahead of the generic service arm: read-only except for the // opt-in `--verify` request, and needs no credential load of its own // — it reports what one would resolve to, not what a fresh one @@ -2339,6 +2366,13 @@ mod tests { assert!(assume_yes_with(yes), "MAPBOX_YES={yes:?} was read as a no"); } + // A yes to deletes, but not to trusting a workflow's source. + let matches = build_app(&[]) + .try_get_matches_from(["mapbox", "workflow", "install", "x"]) + .expect("a hand-written command parses without the specs"); + assert!(matches.get_flag(confirm::ARG)); + assert!(!confirm::typed_yes(&matches)); + std::env::remove_var(confirm::ENV); let matches = build_app(&[]) .try_get_matches_from(["mapbox", "auth", "logout"]) diff --git a/src/output/error.rs b/src/output/error.rs index cf325c3..d00dc62 100644 --- a/src/output/error.rs +++ b/src/output/error.rs @@ -53,8 +53,18 @@ pub struct CliError { /// to support — a 404 on a mistyped style id is theirs to fix, and an id /// under it would be noise on the common case. pub request_id: Option, + /// The failure this one wraps, as the error document it was printed as: + /// a workflow step's own error, so that a caller can still branch on its + /// `code`. JSON only; the text rendering already says it in `message`. + pub cause: Option, } +/// Set on a child this binary runs as a workflow step: the file its error +/// document goes to instead of stderr, for the run to report as the step's +/// cause. Its text form on the terminal would only repeat what the run is +/// about to say, and its JSON form would be a second, stray document. +pub const STEP_ERROR_ENV: &str = "MAPBOX_INTERNAL_STEP_ERROR_FILE"; + impl CliError { pub fn new(code: impl Into, message: impl Into) -> Self { CliError { @@ -67,6 +77,7 @@ impl CliError { next_actions: Vec::new(), docs: Vec::new(), request_id: None, + cause: None, } } @@ -141,6 +152,7 @@ impl CliError { next_actions: Vec::new(), docs: Vec::new(), request_id: None, + cause: None, } } @@ -229,9 +241,22 @@ fn error_payload(e: &CliError) -> Value { if let Some(request_id) = &e.request_id { obj["request_id"] = json!(request_id); } + if let Some(cause) = &e.cause { + obj["cause"] = cause.clone(); + } obj } +/// The error document `err` renders as under `json`. +fn payload(err: &anyhow::Error) -> Value { + match err.downcast_ref::() { + Some(e) => error_payload(e), + // `{:#}` flattens anyhow's context chain into one line, so a + // wrapped error keeps the context that explains it. + None => json!({ "code": GENERIC_CODE, "message": format!("{err:#}") }), + } +} + /// Renders a failure to stderr. /// /// Flat, not wrapped in an `{"error": …}` object. Under `json`, stderr never @@ -244,13 +269,16 @@ pub fn emit_error(mode: Mode, err: &anyhow::Error) { let code = cli.map_or(GENERIC_CODE, |e| e.code.as_str()); crate::run_record::set_error(code, &format!("{err:#}")); + // Printed as usual if the file cannot be written, so the failure is + // never lost. + if let Some(path) = std::env::var_os(STEP_ERROR_ENV) { + if std::fs::write(path, payload(err).to_string()).is_ok() { + return; + } + } + if mode.is_json() { - let payload = match cli { - Some(e) => error_payload(e), - // `{:#}` flattens anyhow's context chain into one line, so a - // wrapped error keeps the context that explains it. - None => json!({ "code": GENERIC_CODE, "message": format!("{err:#}") }), - }; + let payload = payload(err); // Serializing a `json!` object cannot fail; fall back rather than // panic while already on the error path. let pretty = matches!(mode, Mode::Json { pretty: true }); diff --git a/src/output/mod.rs b/src/output/mod.rs index bab5b68..e4cf115 100644 --- a/src/output/mod.rs +++ b/src/output/mod.rs @@ -26,7 +26,7 @@ mod render; pub mod style; pub mod theme; -pub use error::{emit_error, CliError}; +pub use error::{emit_error, CliError, STEP_ERROR_ENV}; pub use render::field_lines; pub(crate) use render::LINE; @@ -252,6 +252,38 @@ pub fn emit_value( footer: Option<&str>, service: Option<&str>, page: Option<&str>, +) -> Result<()> { + emit_rendered( + mode, + value, + footer, + service, + page, + "`-o json` for the response as the API sent it.", + ) +} + +/// [`emit_value`] for a result this CLI put together rather than an API +/// response, such as a workflow's outputs, so the tip does not claim the +/// JSON is what an API sent. +pub fn emit_result(mode: Mode, value: &Value) -> Result<()> { + emit_rendered( + mode, + value, + None, + None, + None, + "`-o json` prints it as JSON.", + ) +} + +fn emit_rendered( + mode: Mode, + value: &Value, + footer: Option<&str>, + service: Option<&str>, + page: Option<&str>, + json_tip: &str, ) -> Result<()> { if let Mode::Json { pretty } = mode { write_stdout(&encode(value, pretty)?)?; @@ -287,7 +319,7 @@ pub fn emit_value( let mut tips = vec![if rendered.shortened { "Values are shortened to fit; `-o json` prints each row whole.".to_string() } else { - "`-o json` for the response as the API sent it.".to_string() + json_tip.to_string() }]; tips.extend(next); // Last, because it is about the response as a whole rather than diff --git a/src/output/style.rs b/src/output/style.rs index 33e8e11..a862a7d 100644 --- a/src/output/style.rs +++ b/src/output/style.rs @@ -60,6 +60,12 @@ pub fn accent() -> String { theme::open(theme::accent()) } +/// The palette's error color, as an escape. Bold where 24-bit color is not +/// known to render. +pub fn error() -> String { + theme::open(theme::error()) +} + /// The palette's muted color, as an escape. Empty where 24-bit color is not /// known to render: plain text, not a theme's `dim`. pub fn muted() -> String { @@ -105,6 +111,14 @@ fn console_renders_ansi() -> bool { !cfg!(windows) || env_value("WT_SESSION").is_some() || env_value("TERM_PROGRAM").is_some() } +/// Whether a line on a stream can be redrawn in place, for a spinner. +/// +/// Not a color question, so `NO_COLOR` does not turn it off; a terminal that +/// would print the escapes literally does. +pub fn redraws(stream_is_terminal: bool) -> bool { + stream_is_terminal && env_value("TERM").as_deref() != Some("dumb") && console_renders_ansi() +} + /// A variable's value, treating an empty one as unset — `NO_COLOR=` is how a /// shell clears it, not a request for no color. fn env_value(name: &str) -> Option { diff --git a/src/output/theme.rs b/src/output/theme.rs index 0c947f8..377b529 100644 --- a/src/output/theme.rs +++ b/src/output/theme.rs @@ -180,6 +180,11 @@ pub fn accent() -> Style { accent_in(current()) } +/// A step or a request that failed. Always bold. +pub fn error() -> Style { + colored(current(), |p| p.error).bold() +} + /// Secondary text. pub fn muted() -> Style { muted_in(current()) diff --git a/src/run_record.rs b/src/run_record.rs index d979e7c..8d41c14 100644 --- a/src/run_record.rs +++ b/src/run_record.rs @@ -391,6 +391,11 @@ fn leaf<'a>( // Its forwarded words come back as subcommands of their own. return (path, command.find_subcommand(name).unwrap_or(command), sub); } + // The word after `workflow run` names a workflow on this machine, and + // a record keeps command paths, never what was typed as a value. + if path == [crate::workflow::COMMAND, "run"] { + return (path, command.find_subcommand(name).unwrap_or(command), sub); + } match command.find_subcommand(name) { Some(found) => command = found, None => break, @@ -422,7 +427,8 @@ pub(crate) fn tree_path(app: &Command, words: impl IntoIterator) match command.find_subcommand(&word) { Some(found) => { path.push(found.get_name().to_string()); - if found.get_name() == TILESETS { + // See `leaf`: what follows `workflow run` is a workflow's name. + if found.get_name() == TILESETS || path == [crate::workflow::COMMAND, "run"] { break; } command = found; diff --git a/src/schema.rs b/src/schema.rs index 731f012..b46f2ad 100644 --- a/src/schema.rs +++ b/src/schema.rs @@ -402,6 +402,14 @@ fn commands(app: &Command, specs: &[ServiceSpec], path: &[String]) -> Vec(cache: Option<&'a Cache>, current: &str, now: u64) -> Optio /// pre-release flag is carried at all, since a dev-channel version is /// spelled exactly that way. Ordering two *different* pre-releases is not /// attempted, because nothing publishes two. -fn parse_version(text: &str) -> Option<(u64, u64, u64, bool)> { +pub(crate) fn parse_version(text: &str) -> Option<(u64, u64, u64, bool)> { let text = text.trim().trim_start_matches('v'); // Build metadata never affects precedence, so it goes before anything // else is read. diff --git a/src/workflow/definition.rs b/src/workflow/definition.rs new file mode 100644 index 0000000..59b52af --- /dev/null +++ b/src/workflow/definition.rs @@ -0,0 +1,884 @@ +//! The `workflow.yaml` schema, and the layout rules a workflow directory is +//! held to. +//! +//! A workflow reaches this module as files in memory — read from a local +//! directory or out of a GitHub tarball — and is checked here before any of +//! it is written or run. The rules are the same for the workflows in this +//! repository's `workflow/` and for anybody else's, so `install` is where a +//! broken one is refused rather than `run`, halfway through. +//! +//! What this cannot check is whether a `command` step names a real command +//! with real arguments: that needs the command tree, and lives in +//! [`super::runner::command_problems`]. + +use std::collections::{BTreeMap, BTreeSet}; +use std::path::{Component, Path, PathBuf}; + +use serde::Deserialize; +use serde_json::{Map, Value}; + +use super::template::{self, Reference}; + +pub const SCHEMA_VERSION: u64 = 1; +pub const DEFINITION_FILE: &str = "workflow.yaml"; +pub const SCRIPTS_DIR: &str = "scripts"; +pub const README_FILE: &str = "README.md"; + +/// A workflow's files, by path relative to its own directory. +pub type Files = BTreeMap>; + +/// A checked workflow. Nothing constructs one except [`parse`]. +#[derive(Debug, Clone)] +pub struct Workflow { + pub name: String, + /// The workflow's own version, independent of the CLI's: workflows are + /// released on their own, and may move to a repository of their own. + /// Optional, so a workflow that has never been released still installs. + pub version: Option, + /// The oldest CLI that can run it, from `requires: ">=X.Y.Z"`. Checked by + /// [`super::runner::command_problems`], since it is the only check that + /// covers what a script step asks of `$MAPBOX_CLI`. + pub requires: Option, + pub summary: String, + pub description: Option, + pub inputs: BTreeMap, + pub steps: Vec, + pub outputs: Option, + /// What `run` prints in text mode instead of the outputs as fields: a + /// template over the same inputs and steps. `-o json` still prints the + /// outputs. + pub result: Option, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Input { + #[serde(rename = "type")] + pub kind: InputType, + #[serde(default)] + pub description: Option, + #[serde(default)] + pub required: bool, + #[serde(default)] + pub default: Option, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum InputType { + String, + Number, + Boolean, +} + +impl InputType { + pub fn as_str(self) -> &'static str { + match self { + InputType::String => "string", + InputType::Number => "number", + InputType::Boolean => "boolean", + } + } + + pub fn accepts(self, value: &Value) -> bool { + matches!( + (self, value), + (InputType::String, Value::String(_)) + | (InputType::Number, Value::Number(_)) + | (InputType::Boolean, Value::Bool(_)) + ) + } +} + +#[derive(Debug, Clone)] +pub struct Step { + pub id: String, + pub name: Option, + pub action: Action, + pub stdin: Option, + /// Runs under `--dry-run` too. A script is told so through + /// [`DRY_RUN_ENV`] and promises to write nothing; a command step may be + /// marked only if its command is on the read-only list that + /// [`super::runner::command_problems`] checks. + pub dry_run: bool, + /// A command step's stdout, written as this file in the run's + /// [`super::workdir`] rather than read as its output. How a binary + /// response, such as a style's ZIP, reaches a later step. + pub save: Option, +} + +/// Set to `1` for a `dry_run` step while `--dry-run` runs it. +pub const DRY_RUN_ENV: &str = "MAPBOX_WORKFLOW_DRY_RUN"; + +#[derive(Debug, Clone)] +pub enum Action { + Command { + /// `styles get` as `["styles", "get"]`. + path: Vec, + args: Map, + }, + Script { + /// Relative to `scripts/`. + script: PathBuf, + interpreter: String, + args: Vec, + }, +} + +impl Step { + /// How progress and plans name the step. + pub fn label(&self) -> String { + match &self.action { + Action::Command { path, .. } => format!("mapbox {}", path.join(" ")), + Action::Script { + script, + interpreter, + .. + } => format!("{interpreter} {SCRIPTS_DIR}/{}", slash_path(script)), + } + } +} + +/// The file as written, before anything is checked. +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct Raw { + #[serde(default)] + schema: Option, + /// A string `0.1.0` alongside `schema`; on its own, the integer that + /// named the schema before `schema` existed. See [`schema_and_version`]. + #[serde(default)] + version: Option, + #[serde(default)] + requires: Option, + name: String, + summary: String, + #[serde(default)] + description: Option, + #[serde(default)] + inputs: BTreeMap, + steps: Vec, + #[serde(default)] + outputs: Option, + #[serde(default)] + result: Option, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct RawStep { + id: String, + #[serde(default)] + name: Option, + #[serde(default)] + command: Option, + #[serde(default)] + script: Option, + #[serde(default)] + interpreter: Option, + #[serde(default)] + args: Option, + #[serde(default)] + stdin: Option, + #[serde(default)] + dry_run: bool, + #[serde(default)] + save: Option, +} + +/// A workflow name is one directory name, lower-case and dash-separated — +/// `uninstall` turns it into a path it deletes, so nothing else may pass. +pub fn is_workflow_name(name: &str) -> bool { + !name.is_empty() + && name.len() <= 64 + && name.starts_with(|c: char| c.is_ascii_lowercase() || c.is_ascii_digit()) + && name + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-') +} + +/// Input names and step ids, which expressions refer to. +fn is_identifier(name: &str) -> bool { + name.starts_with(|c: char| c.is_ascii_lowercase()) + && name + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '_') +} + +/// `major.minor.patch`, with an optional pre-release. A release is meant to +/// be tagged `workflow--`, so only what a ref may hold, and no +/// leading `v` or `+build`, which would give one version two spellings. +fn is_workflow_version(version: &str) -> bool { + version.starts_with(|c: char| c.is_ascii_digit()) + && !version.ends_with('-') + && version + .chars() + .all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-') + && crate::update_check::parse_version(version).is_some() +} + +/// The version in `requires: ">=X.Y.Z"`. A lower bound only: which future +/// CLI breaks a workflow is not knowable when it is written. +fn minimum_cli(requires: &str) -> Option<&str> { + let version = requires.trim().strip_prefix(">=")?.trim(); + is_workflow_version(version).then_some(version) +} + +/// The workflow's own version, once `schema` is checked. Before `schema` +/// existed, `version: 1` named the schema; that spelling still reads, as an +/// unversioned workflow, so a commit from then still installs. +fn schema_and_version( + schema: Option, + version: Option<&Value>, + problems: &mut Vec, +) -> Option { + match (schema, version) { + (Some(schema), version) => { + if schema != SCHEMA_VERSION { + problems.push(format!( + "`schema: {schema}` is not one this CLI reads; it reads \ + `schema: {SCHEMA_VERSION}`" + )); + } + match version { + None => None, + Some(Value::String(version)) if is_workflow_version(version) => { + Some(version.clone()) + } + Some(other) => { + let shown = other + .as_str() + .map_or_else(|| other.to_string(), String::from); + problems.push(format!( + "`version: {shown}` must be the workflow's own version, like `0.1.0`" + )); + None + } + } + } + (None, Some(Value::Number(legacy))) if legacy.as_u64() == Some(SCHEMA_VERSION) => None, + (None, _) => { + problems.push(format!("`schema: {SCHEMA_VERSION}` is missing")); + None + } + } +} + +/// A path that stays inside the directory it is relative to. +pub fn is_contained(path: &Path) -> bool { + !path.as_os_str().is_empty() + && path + .components() + .all(|component| matches!(component, Component::Normal(_))) +} + +pub fn slash_path(path: &Path) -> String { + path.components() + .map(|part| part.as_os_str().to_string_lossy()) + .collect::>() + .join("/") +} + +/// The interpreter a script runs under when the step names none. A short, +/// fixed list: guessing further is how a script ends up run by the wrong +/// program. +fn default_interpreter(script: &Path) -> Option<&'static str> { + match script.extension()?.to_str()? { + "sh" => Some("sh"), + "py" => Some("python3"), + "js" | "mjs" => Some("node"), + _ => None, + } +} + +/// Checks `files` as the workflow called `name`, and returns it or every +/// problem found — all of them at once, so fixing a workflow is not one +/// install attempt per mistake. +pub fn parse(name: &str, files: &Files) -> Result> { + let mut problems = vec![]; + + for path in files.keys() { + let allowed = path == Path::new(DEFINITION_FILE) + || path == Path::new(README_FILE) + || path.starts_with(SCRIPTS_DIR); + if !allowed { + problems.push(format!( + "`{}` does not belong in a workflow: only {DEFINITION_FILE}, {README_FILE} \ + and {SCRIPTS_DIR}/ do", + slash_path(path) + )); + } + } + + let Some(bytes) = files.get(Path::new(DEFINITION_FILE)) else { + problems.push(format!("there is no {DEFINITION_FILE}")); + return Err(problems); + }; + let raw: Raw = match serde_yaml::from_slice(bytes) { + Ok(raw) => raw, + Err(e) => { + problems.push(format!("{DEFINITION_FILE} is not valid: {e}")); + return Err(problems); + } + }; + + let version = schema_and_version(raw.schema, raw.version.as_ref(), &mut problems); + let requires = raw.requires.as_deref().and_then(|requires| { + let minimum = minimum_cli(requires); + if minimum.is_none() { + problems.push(format!( + "`requires: {requires}` must be `\">=X.Y.Z\"`, quoted, naming the oldest \ + mapbox that can run it" + )); + } + minimum.map(String::from) + }); + if raw.name != name { + problems.push(format!( + "`name: {}` must match the directory it is in, `{name}`", + raw.name + )); + } + if !is_workflow_name(&raw.name) { + problems.push(format!( + "`name: {}` must be lower-case letters, digits and dashes", + raw.name + )); + } + if raw.summary.trim().is_empty() || raw.summary.contains('\n') { + problems.push("`summary` must be one non-empty line".to_string()); + } + + for (input, spec) in &raw.inputs { + if !is_identifier(input) { + problems.push(format!( + "input `{input}` must be lower-case letters, digits and underscores" + )); + } + if let Some(default) = &spec.default { + if spec.required { + problems.push(format!( + "input `{input}` is required and has a default; it can only be one" + )); + } + if !spec.kind.accepts(default) { + problems.push(format!( + "input `{input}`'s default is not a {}", + spec.kind.as_str() + )); + } + } + } + + if raw.steps.is_empty() { + problems.push("`steps` is empty".to_string()); + } + + let mut steps = vec![]; + let mut seen: BTreeSet = BTreeSet::new(); + let mut runs_in_dry_run: BTreeSet = BTreeSet::new(); + let mut referenced_scripts: BTreeSet = BTreeSet::new(); + for raw_step in raw.steps { + let id = raw_step.id.clone(); + if !is_identifier(&id) { + problems.push(format!( + "step `{id}`: the id must be lower-case letters, digits and underscores" + )); + } + if seen.contains(&id) { + problems.push(format!("step `{id}` appears twice")); + } + + for value in raw_step.args.iter().chain(raw_step.stdin.iter()) { + check_references( + value, + &format!("step `{id}`"), + &raw.inputs, + &seen, + &mut problems, + ); + } + + if raw_step.dry_run { + // A dry run skips every other step, so there is no output of + // theirs for this one to read. + for value in raw_step.args.iter().chain(raw_step.stdin.iter()) { + for reference in template::references(value).unwrap_or_default() { + if let Reference::Step { id: read, .. } = &reference { + if seen.contains(read) && !runs_in_dry_run.contains(read) { + problems.push(format!( + "step `{id}` runs under --dry-run but reads `{}`, from a \ + step that does not", + template::display(&reference) + )); + } + } + } + } + } + + let action = match (raw_step.command, raw_step.script) { + (Some(command), None) => { + if let Some(save) = &raw_step.save { + let path = Path::new(save); + if !is_contained(path) || path.components().count() != 1 { + problems.push(format!( + "step `{id}`: `save: {save}` must be a plain file name" + )); + } + } + if raw_step.interpreter.is_some() { + problems.push(format!( + "step `{id}`: `interpreter` only applies to a `script` step" + )); + } + let path: Vec = command.split_whitespace().map(String::from).collect(); + if path.is_empty() { + problems.push(format!("step `{id}`: `command` is empty")); + } + let args = match raw_step.args { + None => Map::new(), + Some(Value::Object(args)) => args, + Some(_) => { + problems.push(format!( + "step `{id}`: a command's `args` is a mapping of argument name to value" + )); + Map::new() + } + }; + Some(Action::Command { path, args }) + } + (None, Some(script)) => { + if raw_step.save.is_some() { + problems.push(format!( + "step `{id}`: `save` only applies to a `command` step; a script \ + writes its own files" + )); + } + let script = PathBuf::from(script); + let under_scripts = Path::new(SCRIPTS_DIR).join(&script); + if !is_contained(&script) { + problems.push(format!( + "step `{id}`: `script: {}` must be a path inside {SCRIPTS_DIR}/", + script.display() + )); + } else if !files.contains_key(&under_scripts) { + problems.push(format!( + "step `{id}`: there is no {SCRIPTS_DIR}/{}", + slash_path(&script) + )); + } + referenced_scripts.insert(under_scripts); + + let interpreter = match raw_step + .interpreter + .or_else(|| default_interpreter(&script).map(String::from)) + { + Some(interpreter) + if !interpreter.is_empty() + && !interpreter.contains(char::is_whitespace) => + { + interpreter + } + Some(interpreter) => { + problems.push(format!( + "step `{id}`: `interpreter: {interpreter}` must be one program name" + )); + String::new() + } + None => { + problems.push(format!( + "step `{id}`: name an `interpreter` for {}; only .sh, .py and .js \ + have a default", + slash_path(&script) + )); + String::new() + } + }; + let args = match raw_step.args { + None => vec![], + Some(Value::Array(args)) => args, + Some(_) => { + problems.push(format!( + "step `{id}`: a script's `args` is a list of values" + )); + vec![] + } + }; + Some(Action::Script { + script, + interpreter, + args, + }) + } + (Some(_), Some(_)) => { + problems.push(format!( + "step `{id}` has both `command` and `script`; a step is one or the other" + )); + None + } + (None, None) => { + problems.push(format!("step `{id}` needs a `command` or a `script`")); + None + } + }; + + seen.insert(id.clone()); + if raw_step.dry_run { + runs_in_dry_run.insert(id.clone()); + } + if let Some(action) = action { + steps.push(Step { + id, + name: raw_step.name, + action, + stdin: raw_step.stdin, + dry_run: raw_step.dry_run, + save: raw_step.save, + }); + } + } + + if let Some(outputs) = &raw.outputs { + check_references(outputs, "`outputs`", &raw.inputs, &seen, &mut problems); + } + if let Some(result) = &raw.result { + let result = Value::String(result.clone()); + check_references(&result, "`result`", &raw.inputs, &seen, &mut problems); + } + + for path in files.keys().filter(|path| path.starts_with(SCRIPTS_DIR)) { + if !referenced_scripts.contains(path) { + problems.push(format!( + "`{}` is not used by any step; a workflow ships only the scripts it runs", + slash_path(path) + )); + } + } + + if !problems.is_empty() { + return Err(problems); + } + Ok(Workflow { + name: raw.name, + version, + requires, + summary: raw.summary, + description: raw.description, + inputs: raw.inputs, + steps, + outputs: raw.outputs, + result: raw.result, + }) +} + +/// Every expression in `value` must name a declared input or a step that +/// has already run by the time `value` is read. +fn check_references( + value: &Value, + owner: &str, + inputs: &BTreeMap, + earlier_steps: &BTreeSet, + problems: &mut Vec, +) { + let references = match template::references(value) { + Ok(references) => references, + Err(e) => { + problems.push(format!("{owner}: {e}")); + return; + } + }; + for reference in references { + match &reference { + Reference::Input(name) if !inputs.contains_key(name) => problems.push(format!( + "{owner}: `{}` names no declared input", + template::display(&reference) + )), + Reference::Step { id, .. } if !earlier_steps.contains(id) => problems.push(format!( + "{owner}: `{}` names no earlier step", + template::display(&reference) + )), + _ => {} + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const MINIMAL: &str = "\ +schema: 1 +name: demo +summary: A demo +inputs: + style_id: { type: string, required: true } +steps: + - id: fetch + command: styles get + args: { style-id: '${{ inputs.style_id }}' } + - id: shape + script: shape.py + stdin: '${{ steps.fetch.output }}' +outputs: + id: '${{ steps.shape.output.id }}' +"; + + fn files(entries: &[(&str, &str)]) -> Files { + entries + .iter() + .map(|(path, body)| (PathBuf::from(path), body.as_bytes().to_vec())) + .collect() + } + + fn problems(entries: &[(&str, &str)]) -> Vec { + parse("demo", &files(entries)).expect_err("should be refused") + } + + #[test] + fn a_minimal_workflow_parses() { + let workflow = parse( + "demo", + &files(&[ + (DEFINITION_FILE, MINIMAL), + ("scripts/shape.py", ""), + (README_FILE, ""), + ]), + ) + .unwrap(); + assert_eq!(workflow.steps.len(), 2); + assert_eq!(workflow.steps[0].label(), "mapbox styles get"); + assert_eq!(workflow.steps[1].label(), "python3 scripts/shape.py"); + } + + #[test] + fn the_name_must_match_the_directory() { + let found = parse( + "other", + &files(&[(DEFINITION_FILE, MINIMAL), ("scripts/shape.py", "")]), + ) + .unwrap_err(); + assert!(found.iter().any(|p| p.contains("must match")), "{found:?}"); + } + + #[test] + fn stray_and_unused_files_are_refused() { + let found = problems(&[ + (DEFINITION_FILE, MINIMAL), + ("scripts/shape.py", ""), + ("scripts/old.py", ""), + ("notes.txt", ""), + ]); + assert!( + found.iter().any(|p| p.contains("scripts/old.py")), + "{found:?}" + ); + assert!(found.iter().any(|p| p.contains("notes.txt")), "{found:?}"); + } + + #[test] + fn a_missing_script_is_refused() { + let found = problems(&[(DEFINITION_FILE, MINIMAL)]); + assert!( + found.iter().any(|p| p.contains("no scripts/shape.py")), + "{found:?}" + ); + } + + #[test] + fn a_script_path_cannot_leave_scripts() { + let yaml = MINIMAL.replace("script: shape.py", "script: ../workflow.yaml"); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found.iter().any(|p| p.contains("inside scripts/")), + "{found:?}" + ); + } + + #[test] + fn a_reference_must_point_backwards() { + let yaml = MINIMAL.replace( + "args: { style-id: '${{ inputs.style_id }}' }", + "args: { style-id: '${{ steps.shape.output.id }}' }", + ); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found.iter().any(|p| p.contains("no earlier step")), + "{found:?}" + ); + } + + #[test] + fn save_is_a_plain_file_name_on_a_command_step() { + for bad in ["../x.zip", "dir/x.zip", "/tmp/x.zip"] { + let yaml = MINIMAL.replace( + " command: styles get\n", + &format!(" command: styles get\n save: '{bad}'\n"), + ); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found + .iter() + .any(|p| p.contains("must be a plain file name")), + "{bad}: {found:?}" + ); + } + + let yaml = MINIMAL.replace( + " script: shape.py\n", + " script: shape.py\n save: out.json\n", + ); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found + .iter() + .any(|p| p.contains("only applies to a `command` step")), + "{found:?}" + ); + } + + /// A dry run skips the steps not marked `dry_run`, so a step that is + /// marked cannot read their output. + #[test] + fn a_dry_run_step_reads_only_dry_run_steps() { + let yaml = MINIMAL.replace( + " script: shape.py\n", + " script: shape.py\n dry_run: true\n", + ); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found + .iter() + .any(|p| p.contains("runs under --dry-run but reads `steps.fetch")), + "{found:?}" + ); + } + + #[test] + fn result_may_name_only_earlier_steps_and_declared_inputs() { + let yaml = format!("{MINIMAL}result: 'Made ${{{{ steps.nope.output.id }}}}'\n"); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found + .iter() + .any(|p| p.starts_with("`result`") && p.contains("no earlier step")), + "{found:?}" + ); + } + + #[test] + fn an_undeclared_input_is_refused() { + let yaml = MINIMAL.replace("inputs.style_id }}'", "inputs.nope }}'"); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!( + found.iter().any(|p| p.contains("no declared input")), + "{found:?}" + ); + } + + #[test] + fn an_unknown_field_is_refused() { + let yaml = MINIMAL.replace("summary: A demo", "summary: A demo\nretries: 3"); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.py", "")]); + assert!(found.iter().any(|p| p.contains("retries")), "{found:?}"); + } + + #[test] + fn an_unknown_extension_needs_an_interpreter() { + let yaml = MINIMAL.replace("shape.py", "shape.rb"); + let found = problems(&[(DEFINITION_FILE, &yaml), ("scripts/shape.rb", "")]); + assert!(found.iter().any(|p| p.contains("interpreter")), "{found:?}"); + } + + #[test] + fn workflow_names_are_one_plain_directory_name() { + for good in ["copy-style", "a", "v2-sync"] { + assert!(is_workflow_name(good), "{good}"); + } + for bad in ["", "..", "a/b", "/tmp", "Copy", "-x", "a_b", "a.b"] { + assert!(!is_workflow_name(bad), "{bad}"); + } + } + + fn parse_yaml(yaml: &str) -> Result> { + parse( + "demo", + &files(&[(DEFINITION_FILE, yaml), ("scripts/shape.py", "")]), + ) + } + + #[test] + fn version_and_requires_are_optional_and_read_when_given() { + let bare = parse_yaml(MINIMAL).unwrap(); + assert_eq!((bare.version, bare.requires), (None, None)); + + let yaml = MINIMAL.replace( + "name: demo\n", + "name: demo\nversion: 0.2.0-beta.1\nrequires: \">= 0.4.0\"\n", + ); + let given = parse_yaml(&yaml).unwrap(); + assert_eq!(given.version.as_deref(), Some("0.2.0-beta.1")); + assert_eq!(given.requires.as_deref(), Some("0.4.0")); + } + + /// Before `schema`, the schema was `version: 1`. A commit from then + /// still installs, with no version of its own. + #[test] + fn the_old_schema_spelling_still_reads() { + let yaml = MINIMAL.replace("schema: 1\n", "version: 1\n"); + assert_eq!(parse_yaml(&yaml).unwrap().version, None); + } + + #[test] + fn a_schema_is_required_and_must_be_known() { + for (yaml, expected) in [ + (MINIMAL.replace("schema: 1\n", ""), "`schema: 1` is missing"), + ( + MINIMAL.replace("schema: 1\n", "version: 0.1.0\n"), + "`schema: 1` is missing", + ), + (MINIMAL.replace("schema: 1\n", "schema: 2\n"), "`schema: 2`"), + ] { + let found = parse_yaml(&yaml).unwrap_err(); + assert!(found.iter().any(|p| p.contains(expected)), "{found:?}"); + } + } + + #[test] + fn a_version_is_a_plain_semver() { + for bad in [ + "1", + "1.0", + "v0.1.0", + "0.1.0+build", + "0.1", + "0.1.0-", + "0.1.0/x", + ] { + let yaml = MINIMAL.replace("name: demo\n", &format!("name: demo\nversion: '{bad}'\n")); + let found = parse_yaml(&yaml).unwrap_err(); + assert!( + found + .iter() + .any(|p| p.contains("the workflow's own version")), + "{bad}: {found:?}" + ); + } + // Unquoted, YAML reads `1` as a number, which is the old schema + // spelling's shape; beside `schema` it is refused all the same. + let yaml = MINIMAL.replace("name: demo\n", "name: demo\nversion: 1\n"); + assert!(parse_yaml(&yaml).is_err()); + } + + #[test] + fn requires_is_a_quoted_lower_bound() { + for bad in ["0.4.0", "<=0.4.0", ">=0.4", ">=v0.4.0"] { + let yaml = MINIMAL.replace("name: demo\n", &format!("name: demo\nrequires: '{bad}'\n")); + let found = parse_yaml(&yaml).unwrap_err(); + assert!( + found.iter().any(|p| p.contains("`requires:")), + "{bad}: {found:?}" + ); + } + } +} diff --git a/src/workflow/mod.rs b/src/workflow/mod.rs new file mode 100644 index 0000000..b0a33d0 --- /dev/null +++ b/src/workflow/mod.rs @@ -0,0 +1,1107 @@ +//! `mapbox workflow` — install and run workflows: named, multi-step recipes +//! of `mapbox` commands and scripts. Beta: every subcommand says so on +//! stderr. +//! +//! A workflow is a directory holding a `workflow.yaml` and the scripts it +//! runs (see [`definition`] for the schema and the layout rules). None ships +//! inside the binary. `install` copies one in, from a local directory or a +//! GitHub repository's `workflow//`, and `run` runs only what +//! is installed. +//! +//! What this refuses to be: a scheduler, a retry engine, or a language. +//! Steps run in order and the first failure stops the run. Logic belongs in +//! a script step. + +pub mod definition; +pub mod progress; +mod prose; +pub mod runner; +pub mod store; +pub mod template; +pub mod workdir; + +use std::ffi::OsString; +use std::io::IsTerminal; +use std::path::Path; + +use anyhow::Result; +use clap::builder::{StringValueParser, TypedValueParser}; +use clap::{Arg, ArgAction, ArgMatches, Command}; +use serde_json::{json, Value}; + +use crate::confirm; +use crate::executor; +use crate::output::{self, field_lines, style, Mode}; + +pub const COMMAND: &str = "workflow"; + +/// Printed by every subcommand. Nothing here is a promise yet, and a +/// person who found the command in `--help` should not build on it. One +/// line on purpose: `--help` and the docs carry the rest. +const NOTICE: &str = "`mapbox workflow` may change or be removed in any release."; + +const NAME_ARG: &str = "name"; +const SOURCE_ARG: &str = "source"; +const REPO_ARG: &str = "repo"; +const REF_ARG: &str = "ref"; +const FORCE_ARG: &str = "force"; +const RUN: &str = "run"; + +/// Every `workflow` subcommand's help says it is beta, not only the group's: +/// `mapbox workflow run x --help` is often the first page someone reads. +fn beta(about: &str) -> String { + format!("{about} (beta)") +} + +const BETA_NOTE: &str = "Beta: it may change or be removed in any release."; + +pub fn command() -> Command { + let name = || { + Arg::new(NAME_ARG) + .value_name("NAME") + .required(true) + .help("Name of an installed workflow") + }; + Command::new(COMMAND) + .about("Install and run multi-step workflows of mapbox commands and scripts (beta)") + .long_about( + "Install and run workflows: named, multi-step recipes of mapbox commands and \ + scripts, defined in a workflow.yaml.\n\n\ + A workflow runs only once it is installed, from a local directory or from a \ + GitHub repository's workflow// directory.\n\n\ + Beta: the commands, the workflow format and the published workflows may \ + change or be removed in any release.", + ) + .subcommand_required(true) + .subcommand(Command::new("list").about(beta("List installed workflows"))) + .subcommand( + Command::new("show") + .about(beta("Describe an installed workflow: its inputs and steps")) + .arg(name()), + ) + .subcommand( + Command::new("install") + .about(beta( + "Install a workflow from a local directory or from GitHub", + )) + .arg( + Arg::new(SOURCE_ARG) + .value_name("SOURCE") + .required(true) + .help( + "A workflow name, fetched from GitHub, or a path to a local \ + workflow directory (anything with a `/` or starting with `.`)", + ), + ) + .arg( + Arg::new(REPO_ARG) + .long(REPO_ARG) + .value_name("OWNER/REPO") + .help(format!( + "GitHub repository to install from [default: {}]. Reads \ + GH_TOKEN or GITHUB_TOKEN for a private one", + store::DEFAULT_REPO + )), + ) + .arg( + Arg::new(REF_ARG) + .long(REF_ARG) + .value_name("REF") + .help(format!( + "Branch, tag or commit to install from [default: {}]", + store::DEFAULT_REF + )), + ) + .arg( + Arg::new(FORCE_ARG) + .long(FORCE_ARG) + .action(ArgAction::SetTrue) + .help("Replace a workflow that is already installed"), + ) + .arg(executor::dry_run_arg( + "Check the workflow and list the files it would write, then exit", + )), + ) + .subcommand( + Command::new("uninstall") + .about(beta("Remove an installed workflow")) + .arg(Arg::new(NAME_ARG).value_name("NAME").required(true).help( + "Name of an installed workflow, or the directory it was installed \ + from", + )) + .arg(executor::dry_run_arg( + "Say what it would remove, then exit without removing it", + )), + ) + .subcommand( + Command::new(RUN) + .about(beta("Run an installed workflow")) + .long_about(format!( + "Run an installed workflow, giving each of its inputs as a flag: \ + `mapbox workflow run copy-style --style-id --from-profile source \ + --to-profile target`.\n\n\ + `mapbox workflow run --help` lists that workflow's flags.\n\n\ + `--dry-run` runs only the steps the workflow marks `dry_run` and prints \ + the plan for the rest. Those steps really run, scripts included, as \ + you and with your credentials; the workflow promises they write \ + nothing, and the CLI holds a command step to that but cannot check \ + a script.\n\n{BETA_NOTE}", + )) + .subcommand_value_name("NAME") + .subcommand_help_heading("Workflows") + .subcommand_required(true) + // A workflow is a subcommand only once `with_run_target` has + // read it from disk. Any other name still parses, so that + // `run` can say it is not installed, or why it cannot load. + .allow_external_subcommands(true), + ) +} + +/// The `run` subcommand for one installed workflow: its inputs as flags, +/// typed and required as the definition says, so that `--help`, a missing +/// input and a misspelled flag read as they do for every other command. +fn run_command(workflow: &definition::Workflow) -> Command { + let mut command = Command::new(workflow.name.clone()) + .about(beta(&workflow.summary)) + .arg(executor::dry_run_arg( + "Run only the steps marked dry_run, which promise to write nothing, and print \ + the plan for the rest. A marked script still runs, as you", + )); + if let Some(description) = &workflow.description { + command = command.long_about(format!( + "{}\n\n{}\n\n{BETA_NOTE}", + workflow.summary, + prose::render(description, false) + )); + } + for (name, input) in &workflow.inputs { + let flag = runner::input_flag(name); + let mut help = input.description.clone().unwrap_or_default(); + if let Some(default) = &input.default { + help.push_str(&format!( + " [default: {}]", + template::as_text(default).unwrap_or_default() + )); + } + let mut arg = Arg::new(name.clone()) + .long(flag.clone()) + .value_name(flag.to_uppercase().replace('-', "_")) + .required(input.required) + .help(help.trim().to_string()) + // Apart from the globals clap lists beside them, and required + // ones first, as `workflow show` orders them. + .help_heading("Inputs") + .display_order(usize::from(!input.required)); + if flag != *name { + arg = arg.alias(name.clone()); + } + arg = match input.kind { + definition::InputType::String => arg, + definition::InputType::Number => { + arg.value_parser(StringValueParser::new().try_map(|raw: String| { + runner::parse_number(&raw) + .map(|_| raw.clone()) + .ok_or_else(|| format!("`{raw}` is not a number")) + })) + } + // `--flag` alone means true, so a boolean reads like any flag. + definition::InputType::Boolean => arg + .value_parser(["true", "false"]) + .num_args(0..=1) + .default_missing_value("true"), + }; + command = command.arg(arg); + } + command +} + +/// The workflow a command line runs, when it is `… workflow run …`. +/// +/// Read off argv because it has to be known before clap parses: the +/// workflow's flags come from its `workflow.yaml`, and parsing is what they +/// are needed for. +fn run_target(argv: &[OsString]) -> Option { + let words: Vec<&str> = argv.iter().skip(1).filter_map(|arg| arg.to_str()).collect(); + let at = words.windows(2).position(|pair| pair == [COMMAND, RUN])?; + let name = *words.get(at + 2)?; + definition::is_workflow_name(name).then(|| name.to_string()) +} + +/// `app` with the workflow this command line runs attached under `run`, so +/// that clap parses its inputs as flags. Anything that stops it loading is +/// left for `run` to report, with the command tree unchanged: every other +/// command pays nothing for this, and a broken workflow cannot break the +/// parse. +pub fn with_run_target(app: Command, argv: &[OsString]) -> Command { + let Some(name) = run_target(argv) else { + return app; + }; + let Ok(found) = store::load(&name) else { + return app; + }; + if !runner::command_problems(&app, &found.workflow).is_empty() { + return app; + } + let run = run_command(&found.workflow); + app.mut_subcommand(COMMAND, |workflow| { + workflow.mut_subcommand(RUN, |parent| parent.subcommand(run)) + }) +} + +/// What the globals say, for the subcommands that need them. +pub struct RunFlags<'a> { + pub globals: &'a ArgMatches, + pub debug: bool, + pub assume_yes: bool, +} + +pub fn run(app: &Command, matches: &ArgMatches, flags: RunFlags, mode: Mode) -> Result<()> { + let terminal = std::io::stderr().is_terminal(); + let color = style::enabled(terminal); + output::progress(&format!( + "{} {}", + style::paint("Beta:", style::BOLD, color), + style::dim_prose(NOTICE, color) + )); + // At a terminal the notice sits right above the result; a blank line + // keeps it from reading as the result's first line. + if terminal { + output::progress(""); + } + match matches.subcommand() { + Some(("list", _)) => list(mode), + Some(("show", m)) => show(app, name(m), mode), + Some(("install", m)) => install(app, m, flags.debug, mode), + Some(("uninstall", m)) => { + uninstall(name(m), executor::wants_dry_run(m), flags.assume_yes, mode) + } + Some((RUN, m)) => match m.subcommand() { + Some((name, workflow_matches)) => { + run_workflow(app, name, workflow_matches, flags.globals, mode) + } + None => unreachable!("`run` sets subcommand_required(true)"), + }, + _ => unreachable!("`workflow` sets subcommand_required(true)"), + } +} + +fn name(matches: &ArgMatches) -> &str { + matches.get_one::(NAME_ARG).expect("required") +} + +/// A path as a person reads it: under the home directory as `~/…`. Text +/// output only; JSON keeps the absolute path a program can open. +pub(crate) fn tilde(path: &Path) -> String { + match dirs::home_dir().and_then(|home| path.strip_prefix(home).ok().map(Path::to_path_buf)) { + Some(rest) if rest.as_os_str().is_empty() => "~".to_string(), + Some(rest) => format!("~/{}", rest.display()), + None => path.display().to_string(), + } +} + +fn source_label(meta: &store::Meta) -> String { + match &meta.git_ref { + Some(git_ref) => format!("{}@{git_ref}", meta.source), + None => meta.source.clone(), + } +} + +fn source_text(meta: &store::Meta) -> String { + match (&meta.git_ref, &meta.commit) { + (Some(_), Some(commit)) => format!( + "{} (commit {})", + source_label(meta), + // `.install.json` is a file anyone can edit, so not a byte slice. + commit.chars().take(12).collect::() + ), + (Some(_), None) => source_label(meta), + (None, _) => tilde(Path::new(&meta.source)), + } +} + +/// How a cell in [`columns`] is painted. +#[derive(Clone, Copy)] +enum Tone { + Plain, + Bold, + Muted, +} + +impl Tone { + fn escape(self) -> String { + match self { + Tone::Plain => String::new(), + Tone::Bold => style::BOLD.to_string(), + Tone::Muted => style::muted(), + } + } +} + +/// Rows of cells, each column padded to its widest cell. Padding is +/// measured before painting, so escapes never skew the alignment. +fn columns(rows: &[Vec<(String, Tone)>], color: bool) -> String { + let widths: Vec = (0..rows.iter().map(Vec::len).max().unwrap_or(0)) + .map(|column| { + rows.iter() + .filter_map(|row| row.get(column)) + .map(|(cell, _)| cell.chars().count()) + .max() + .unwrap_or(0) + }) + .collect(); + rows.iter() + .map(|row| { + let last = row.len().saturating_sub(1); + row.iter() + .enumerate() + .map(|(column, (cell, tone))| { + let pad = if column == last { + String::new() + } else { + " ".repeat(widths[column] - cell.chars().count() + 2) + }; + format!("{}{pad}", style::paint(cell, &tone.escape(), color)) + }) + .collect::() + }) + .collect::>() + .join("\n") +} + +fn heading(text: &str, color: bool) -> String { + style::paint(text, style::BOLD, color) +} + +/// The `run` line a person would type, with every required input spelled +/// out so the next command is one they only have to fill in. +fn run_example(workflow: &definition::Workflow) -> String { + let mut line = format!("mapbox workflow run {}", workflow.name); + for (name, input) in &workflow.inputs { + if input.required { + line.push_str(&format!(" --{} …", runner::input_flag(name))); + } + } + line +} + +/// Tips go with the text rendering only, as every other command's do. +fn tips(mode: Mode, tips: &[String]) { + if !mode.is_json() { + output::print_tips(tips); + } +} + +/// `-` for a workflow with no version of its own, so the column stays +/// aligned. +fn version_cell(workflow: &definition::Workflow) -> (String, Tone) { + match &workflow.version { + Some(version) => (version.clone(), Tone::Plain), + None => ("-".to_string(), Tone::Muted), + } +} + +fn list(mode: Mode) -> Result<()> { + let installed = store::list()?; + let color = output::result_in_color(); + let mut rows = vec![]; + let mut table = vec![vec![ + ("NAME".to_string(), Tone::Bold), + ("VERSION".to_string(), Tone::Bold), + ("SUMMARY".to_string(), Tone::Bold), + ]]; + for (name, loaded) in &installed { + match loaded { + Ok(found) => { + rows.push(json!({ + "name": name, + "version": found.workflow.version, + "summary": found.workflow.summary, + "source": source_label(&found.meta), + "commit": found.meta.commit, + "path": found.root, + })); + table.push(vec![ + (name.clone(), Tone::Plain), + version_cell(&found.workflow), + (found.workflow.summary.clone(), Tone::Plain), + ]); + } + Err(e) => { + rows.push(json!({ "name": name, "error": format!("{e:#}") })); + table.push(vec![ + (name.clone(), Tone::Plain), + ("-".to_string(), Tone::Muted), + (format!("cannot load: {e:#}"), Tone::Muted), + ]); + } + } + } + + if installed.is_empty() { + output::emit(mode, "No workflows installed.", Value::Array(rows))?; + tips( + mode, + &["Install one with `mapbox workflow install ` or `mapbox workflow install ./`." + .to_string()], + ); + return Ok(()); + } + output::emit(mode, &columns(&table, color), Value::Array(rows))?; + tips( + mode, + &["`mapbox workflow show ` for a workflow's inputs and steps.".to_string()], + ); + Ok(()) +} + +fn show(app: &Command, name: &str, mode: Mode) -> Result<()> { + let found = store::load(name)?; + let workflow = &found.workflow; + let problems = runner::command_problems(app, workflow); + let color = output::result_in_color(); + + let inputs: Vec = workflow + .inputs + .iter() + .map(|(name, input)| { + json!({ + "name": name, + "type": input.kind.as_str(), + "required": input.required, + "default": input.default, + "description": input.description, + }) + }) + .collect(); + let steps: Vec = workflow + .steps + .iter() + .map(|step| { + let kind = match step.action { + definition::Action::Command { .. } => "command", + definition::Action::Script { .. } => "script", + }; + json!({ "id": step.id, "name": step.name, "kind": kind, "run": step.label() }) + }) + .collect(); + + // Fixed order for every workflow: what it is, what it needs, what it + // does, then where this copy came from. + let mut text = format!( + "{}\n{}", + heading(&workflow.name, color), + style::paint(&workflow.summary, &style::muted(), color) + ); + if let Some(description) = &workflow.description { + text.push_str(&format!("\n\n{}", prose::render(description, color))); + } + + text.push_str(&format!("\n\n{}", heading("Inputs", color))); + if workflow.inputs.is_empty() { + text.push_str("\n none"); + } else { + // Required inputs first: they are the ones a run cannot do without. + let mut ordered: Vec<_> = workflow.inputs.iter().collect(); + ordered.sort_by_key(|(name, input)| (!input.required, name.as_str())); + let rows: Vec> = ordered + .into_iter() + .map(|(name, input)| { + let mut kind = input.kind.as_str().to_string(); + if input.required { + kind.push_str(", required"); + } + if let Some(default) = &input.default { + kind.push_str(&format!(", default {default}")); + } + vec![ + (format!(" {name}"), Tone::Plain), + (kind, Tone::Muted), + (input.description.clone().unwrap_or_default(), Tone::Plain), + ] + }) + .collect(); + text.push('\n'); + text.push_str(&columns(&rows, color)); + } + + text.push_str(&format!("\n\n{}\n", heading("Steps", color))); + let rows: Vec> = workflow + .steps + .iter() + .enumerate() + .map(|(index, step)| { + vec![ + ( + format!( + " {}. {}", + index + 1, + step.name.as_deref().unwrap_or(&step.id) + ), + Tone::Plain, + ), + (step.label(), Tone::Muted), + ] + }) + .collect(); + text.push_str(&columns(&rows, color)); + if !scripts(workflow).is_empty() { + text.push_str(&format!( + "\n {}", + style::paint( + "Its scripts run as you, with your credentials.", + &style::muted(), + color + ) + )); + } + + if !problems.is_empty() { + text.push_str(&format!("\n\n{}", heading("This CLI cannot run it", color))); + for problem in &problems { + text.push_str(&format!("\n - {problem}")); + } + } + + text.push_str(&format!("\n\n{}\n", heading("Installed", color))); + let mut installed = vec![]; + if let Some(version) = &workflow.version { + installed.push(vec![ + (" Version".to_string(), Tone::Plain), + (version.clone(), Tone::Muted), + ]); + } + if let Some(requires) = &workflow.requires { + installed.push(vec![ + (" Requires".to_string(), Tone::Plain), + (format!("mapbox {requires} or later"), Tone::Muted), + ]); + } + installed.push(vec![ + (" Source".to_string(), Tone::Plain), + (source_text(&found.meta), Tone::Muted), + ]); + installed.push(vec![ + (" Path".to_string(), Tone::Plain), + (tilde(&found.root), Tone::Muted), + ]); + text.push_str(&columns(&installed, color)); + + output::emit( + mode, + &text, + json!({ + "name": workflow.name, + "version": workflow.version, + "requires": workflow.requires, + "summary": workflow.summary, + "description": workflow.description, + "source": source_label(&found.meta), + "commit": found.meta.commit, + "path": found.root, + "inputs": inputs, + "steps": steps, + "problems": problems, + }), + )?; + if problems.is_empty() { + tips(mode, &[format!("Run it with `{}`.", run_example(workflow))]); + } + Ok(()) +} + +/// A source with a path separator or a leading `.` is a directory; anything +/// else is a name to look up on GitHub. A workflow name can hold neither, +/// so the two never overlap. +fn is_local_source(source: &str) -> bool { + source.starts_with('.') || source.contains('/') || source.contains('\\') +} + +/// What a workflow's script steps run, once each, as `python3 scripts/x.py`. +fn scripts(workflow: &definition::Workflow) -> Vec { + let mut found: Vec = vec![]; + for step in &workflow.steps { + let label = step.label(); + if matches!(step.action, definition::Action::Script { .. }) && !found.contains(&label) { + found.push(label); + } + } + found +} + +/// Whether an install needs no question: this repository's workflows as +/// reviewed onto `main`, and a directory on this machine, which the person +/// already has. Anything else is code from somewhere the CLI cannot vouch +/// for, and runs as the person with their credentials. +/// +/// A `--ref` in this repository is not vouched for by the name alone: +/// `pull/N/head` and a fork's commit resolve here too. `is_on_main` is asked +/// only then, since it costs a request. +fn is_vouched_for( + source: &str, + repo: Option<&String>, + git_ref: Option<&String>, + is_on_main: impl FnOnce() -> bool, +) -> bool { + if is_local_source(source) { + return true; + } + let ours = repo.is_none_or(|repo| repo.eq_ignore_ascii_case(store::DEFAULT_REPO)); + ours && (git_ref.is_none() || is_on_main()) +} + +fn install(app: &Command, matches: &ArgMatches, debug: bool, mode: Mode) -> Result<()> { + let source = matches.get_one::(SOURCE_ARG).expect("required"); + let repo = matches.get_one::(REPO_ARG); + let git_ref = matches.get_one::(REF_ARG); + let force = matches.get_flag(FORCE_ARG); + let dry_run = executor::wants_dry_run(matches); + + let package = if is_local_source(source) { + if repo.is_some() || git_ref.is_some() { + return Err(output::CliError::new( + "usage", + "`--repo` and `--ref` name a GitHub source; this one is a local directory.", + ) + .into()); + } + store::read_local(std::path::Path::new(source))? + } else { + store::fetch_github( + source, + repo.map_or(store::DEFAULT_REPO, String::as_str), + git_ref.map_or(store::DEFAULT_REF, String::as_str), + debug, + )? + }; + let workflow = &package.workflow; + let problems = runner::command_problems(app, workflow); + if !problems.is_empty() { + return Err(store::invalid_workflow(&workflow.name, &problems)); + } + + let files: Vec = package + .files + .keys() + .map(|path| definition::slash_path(path)) + .collect(); + let scripts = scripts(workflow); + store::recover_name(&workflow.name)?; + let target = store::root_path()?.join(&workflow.name); + let summary = json!({ + "name": workflow.name, + "version": workflow.version, + "source": source_label(&package.meta), + "commit": package.meta.commit, + "path": target, + "files": files, + "scripts": scripts, + "dry_run": dry_run, + }); + + let color = output::result_in_color(); + let mut field_rows = vec![]; + if let Some(version) = &workflow.version { + field_rows.push(("Version", version.clone())); + } + field_rows.extend([ + ("Source", source_text(&package.meta)), + ("Path", tilde(&target)), + ("Files", files.join(", ")), + ]); + if !scripts.is_empty() { + field_rows.push(("Scripts", scripts.join(", "))); + } + let fields = field_lines(&field_rows, color); + + // Checked here as well as in `store::install`, so the error can name + // the line to retry with; the store's own check covers a race. + if target.exists() && !force { + let retry = format!("mapbox workflow install {source} --force"); + return Err(store::already_installed( + &workflow.name, + &target, + Some(retry), + )); + } + + if dry_run { + let text = format!( + "Dry run — nothing was written. Would install {}:\n\n{fields}", + heading(&workflow.name, color) + ); + return output::emit(mode, &text, summary); + } + + let vouched = is_vouched_for(source, repo, git_ref, || { + package + .meta + .commit + .as_deref() + .is_some_and(|commit| store::is_on_main(store::DEFAULT_REPO, commit, debug)) + }); + if !vouched { + let runs = if scripts.is_empty() { + String::new() + } else { + format!(", including {}", scripts.join(", ")) + }; + confirm::untrusted_install( + &format!( + "`{}` comes from {}, which this CLI does not vouch for. Its steps will run \ + as you, with your credentials{runs}. Install it only if you trust where \ + it comes from.", + workflow.name, + source_label(&package.meta) + ), + confirm::typed_yes(matches), + )?; + } + + store::install(&package, force)?; + let text = format!("Installed {}\n\n{fields}", heading(&workflow.name, color)); + output::emit(mode, &text, summary)?; + tips( + mode, + &[ + format!("Run it with `{}`.", run_example(workflow)), + format!( + "`mapbox workflow show {}` describes its inputs.", + workflow.name + ), + ], + ); + Ok(()) +} + +/// The workflow `uninstall` means. A path is accepted, as `install` takes +/// one, and names the workflow by its directory — which must still be a +/// plain workflow name, so nothing but an installed workflow is removed. +fn uninstall_name(given: &str) -> &str { + if !is_local_source(given) { + return given; + } + Path::new(given) + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or(given) +} + +fn uninstall(given: &str, dry_run: bool, assume_yes: bool, mode: Mode) -> Result<()> { + let name = uninstall_name(given); + let found = store::installed_dir(name)?; + let Some(dir) = found else { + // `load` words the error, with the commands to try instead. + return store::load(name).map(|_| ()); + }; + if dry_run { + return output::emit( + mode, + &format!( + "Dry run — nothing was removed. Would remove {}.", + tilde(&dir) + ), + json!({ "name": name, "path": dir, "dry_run": true }), + ); + } + confirm::destructive_local_action( + &format!("Remove the workflow at {}?", tilde(&dir)), + assume_yes, + )?; + let dir = store::uninstall(name)?; + output::emit( + mode, + &format!("Removed {name} ({}).", tilde(&dir)), + json!({ "name": name, "path": dir, "dry_run": false }), + ) +} + +fn run_workflow( + app: &Command, + name: &str, + matches: &ArgMatches, + globals: &ArgMatches, + mode: Mode, +) -> Result<()> { + store::check_name(name)?; + let found = store::load(name)?; + let workflow = &found.workflow; + let problems = runner::command_problems(app, workflow); + if !problems.is_empty() { + return Err(store::invalid_workflow(&workflow.name, &problems)); + } + + let attached = app + .find_subcommand(COMMAND) + .and_then(|group| group.find_subcommand(RUN)) + .and_then(|run| run.find_subcommand(name)) + .is_some(); + if !attached { + // Only when argv was spelled in a way `run_target` does not read. + return Err(output::CliError::new( + "usage", + format!( + "Could not read `{name}`'s inputs from this command line. Write it as \ + `{}`.", + run_example(workflow) + ), + ) + .into()); + } + + let given: Vec<(String, String)> = workflow + .inputs + .keys() + .filter_map(|input| { + matches + .get_one::(input) + .map(|value| (input.clone(), value.clone())) + }) + .collect(); + let inputs = runner::read_inputs(workflow, &given)?; + + let inherited = runner::Inherited::from_matches(globals); + + if executor::wants_dry_run(matches) { + let mut plan = runner::plan(workflow, &inputs); + // Steps that promise to write nothing run for real, so the plan can + // say what the rest would do with actual data rather than guess. + let rehearsed = workflow.steps.iter().any(|step| step.dry_run); + if rehearsed { + let results = runner::dry_run(app, workflow, &found.root, &inputs, &inherited)?; + plan["results"] = json!(results); + } + let color = output::result_in_color(); + let input_rows: Vec> = inputs + .iter() + .map(|(name, value)| { + let shown = template::as_text(value).unwrap_or_else(|| "(none)".to_string()); + vec![(format!(" {name}"), Tone::Plain), (shown, Tone::Plain)] + }) + .collect(); + let step_rows: Vec> = workflow + .steps + .iter() + .enumerate() + .map(|(index, step)| { + vec![ + ( + format!( + " {}. {}", + index + 1, + step.name.as_deref().unwrap_or(&step.id) + ), + Tone::Plain, + ), + ( + match (step.dry_run, &step.action) { + (true, definition::Action::Script { .. }) => { + format!("{} (ran its script in this dry run)", step.label()) + } + (true, _) => format!("{} (ran in this dry run)", step.label()), + (false, _) => step.label(), + }, + Tone::Muted, + ), + ] + }) + .collect(); + let mut text = if rehearsed { + // Not "they wrote nothing": that is the workflow's promise, which + // the CLI holds a command step to but cannot check of a script. + format!( + "Dry run — only the steps marked dry_run ran. Would run {}:", + heading(&workflow.name, color) + ) + } else { + format!( + "Dry run — nothing was run. Would run {}:", + heading(&workflow.name, color) + ) + }; + if !input_rows.is_empty() { + text.push_str(&format!( + "\n\n{}\n{}", + heading("Inputs", color), + columns(&input_rows, color) + )); + } + text.push_str(&format!( + "\n\n{}\n{}", + heading("Steps", color), + columns(&step_rows, color) + )); + return output::emit(mode, &text, plan); + } + + let finished = runner::run(app, workflow, &found.root, &inputs, &inherited)?; + match finished.text { + // The workflow's own wording for a person; `-o json` still gets the + // outputs, which is what a script reads. + Some(text) if !mode.is_json() => output::emit(mode, text.trim_end(), finished.outputs), + _ => output::emit_result(mode, &finished.outputs), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn uninstall_reads_a_path_as_its_directory_name() { + assert_eq!(uninstall_name("copy-style"), "copy-style"); + assert_eq!(uninstall_name("./workflow/copy-style"), "copy-style"); + assert_eq!(uninstall_name("./workflow/copy-style/"), "copy-style"); + // No directory name at all: left as typed, for the name check to refuse. + assert_eq!(uninstall_name(".."), ".."); + } + + #[test] + fn every_subcommand_says_it_is_beta_in_its_help() { + let mut workflow = command(); + for sub in workflow.get_subcommands_mut() { + let name = sub.get_name().to_string(); + assert!(sub.render_help().to_string().contains("beta"), "{name} -h"); + assert!( + sub.render_long_help() + .to_string() + .to_lowercase() + .contains("beta"), + "{name} --help" + ); + } + } + + #[test] + fn columns_line_up_with_color_on_or_off() { + let rows = vec![ + vec![ + ("NAME".to_string(), Tone::Bold), + ("SUMMARY".to_string(), Tone::Bold), + ], + vec![ + ("copy-style".to_string(), Tone::Plain), + ("Copy a style".to_string(), Tone::Plain), + ], + ]; + let plain = columns(&rows, false); + assert_eq!(plain, "NAME SUMMARY\ncopy-style Copy a style"); + assert_eq!(style::strip(&columns(&rows, true)), plain); + } + + #[test] + fn a_path_under_home_is_shown_with_a_tilde() { + let home = dirs::home_dir().expect("a home directory"); + assert_eq!( + tilde(&home.join(".mapbox/workflows/x")), + "~/.mapbox/workflows/x" + ); + assert_eq!(tilde(Path::new("/opt/x")), "/opt/x"); + } + + #[test] + fn only_another_repository_or_a_ref_off_main_needs_a_yes() { + let other = "someone/workflows".to_string(); + let ours = "Mapbox/Mapbox-CLI".to_string(); + let unasked = || -> bool { panic!("asked GitHub with no --ref") }; + assert!(is_vouched_for("copy-style", None, None, unasked)); + assert!(is_vouched_for("copy-style", Some(&ours), None, unasked)); + assert!(is_vouched_for("./workflow/copy-style", None, None, unasked)); + assert!(!is_vouched_for("copy-style", Some(&other), None, || true)); + + // Both resolve through this repository without being on its `main`. + for git_ref in ["pull/1/head", "0123456789abcdef0123456789abcdef01234567"] { + let git_ref = git_ref.to_string(); + assert!(!is_vouched_for("copy-style", None, Some(&git_ref), || { + false + })); + assert!(!is_vouched_for( + "copy-style", + Some(&ours), + Some(&git_ref), + || false + )); + assert!(is_vouched_for("copy-style", None, Some(&git_ref), || true)); + assert!(!is_vouched_for( + "copy-style", + Some(&other), + Some(&git_ref), + || true + )); + } + } + + /// `MAPBOX_YES` answers a delete, not the trust question, so only the + /// typed flag counts, on either side of the subcommand. + #[test] + fn only_a_typed_yes_answers_the_trust_question() { + let install = |args: &[&str]| -> bool { + let matches = crate::build_app(&[]) + .try_get_matches_from(args) + .expect("parses"); + let (_, workflow) = matches.subcommand().expect("workflow"); + let (_, install) = workflow.subcommand().expect("install"); + confirm::typed_yes(install) + }; + assert!(install(&["mapbox", "workflow", "install", "x", "--yes"])); + assert!(install(&["mapbox", "-y", "workflow", "install", "x"])); + // Whatever the environment holds while this runs, it is not typed. + assert!(!install(&["mapbox", "workflow", "install", "x"])); + } + + #[test] + fn scripts_are_listed_once_each() { + let files: definition::Files = [ + ( + std::path::PathBuf::from(definition::DEFINITION_FILE), + b"schema: 1\nname: t\nsummary: t\nsteps:\n - id: a\n script: x.py\n \ + - id: b\n command: config list\n - id: c\n script: x.py\n" + .to_vec(), + ), + (std::path::PathBuf::from("scripts/x.py"), vec![]), + ] + .into(); + let workflow = definition::parse("t", &files).unwrap(); + assert_eq!(scripts(&workflow), ["python3 scripts/x.py"]); + } + + #[test] + fn a_name_and_a_path_never_look_alike() { + for local in [ + "./copy-style", + "../x", + "workflow/copy-style", + "/abs", + ".", + "a\\b", + ] { + assert!(is_local_source(local), "{local}"); + } + assert!(!is_local_source("copy-style")); + } + + /// Every workflow this repository publishes is one `install` accepts. + /// The rules are the same as for anybody's, so this is also what + /// catches a stray file or a step naming a command that was renamed. + #[test] + fn every_published_workflow_is_valid() { + let app = crate::build_app(&crate::spec::effective_services().expect("bundled specs")); + let root = Path::new(env!("CARGO_MANIFEST_DIR")).join(store::REPO_PREFIX); + let mut checked = 0; + for dir in std::fs::read_dir(&root) + .expect("workflow/ exists") + .flatten() + .filter(|entry| entry.path().is_dir()) + { + let package = store::read_local(&dir.path()) + .unwrap_or_else(|e| panic!("{}: {e:#}", dir.path().display())); + let problems = runner::command_problems(&app, &package.workflow); + assert!( + problems.is_empty(), + "{}: {problems:#?}", + dir.path().display() + ); + checked += 1; + } + assert!(checked > 0, "no workflows found under workflow/"); + } +} diff --git a/src/workflow/progress.rs b/src/workflow/progress.rs new file mode 100644 index 0000000..a637494 --- /dev/null +++ b/src/workflow/progress.rs @@ -0,0 +1,329 @@ +//! How a workflow run shows its steps on stderr. +//! +//! Two modes, chosen once per run. Where the line can be redrawn, each step +//! is its title, then what it reports, indented, then one status line: a +//! spinner while it runs, a mark and a duration once it ends. Only that last +//! line is ever redrawn, so a detail that wraps cannot throw the drawing off. Anywhere else +//! — a pipe, a CI log, an agent reading stderr — each step is one plain +//! `[n/total]` line and its reports follow as they arrive, with nothing that +//! only makes sense on a screen. +//! +//! A script reports through its stderr. A line starting with +//! [`PROGRESS_PREFIX`] replaces the text beside the spinner and is dropped +//! where there is none. One starting with [`WARN_PREFIX`] is a warning: shown +//! where it happened and listed again once the run ends, so it is not lost +//! among the details. Any other line is a detail, kept in both modes unless +//! the run is `--quiet`. +//! +//! A command step is never animated. It keeps the terminal, where a +//! confirmation prompt has to be able to reach the person running it, so +//! nothing here may draw over its output. + +use std::io::{IsTerminal, Write}; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Mutex}; +use std::thread::JoinHandle; +use std::time::{Duration, Instant}; + +use crate::output::style; + +/// Marks a script's stderr line as progress rather than a detail. +pub const PROGRESS_PREFIX: &str = "::progress "; +/// Marks a script's stderr line as a warning. +pub const WARN_PREFIX: &str = "::warn "; + +const FRAMES: [&str; 10] = ["⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"]; +const TICK: Duration = Duration::from_millis(80); +const DETAIL_INDENT: &str = " "; + +pub struct Display { + live: bool, + color: bool, + quiet: bool, + total: usize, + warnings: Arc>>, +} + +impl Display { + pub fn for_stderr(total: usize, quiet: bool) -> Self { + let terminal = std::io::stderr().is_terminal(); + Display { + live: style::redraws(terminal), + color: style::enabled(terminal), + quiet, + total, + warnings: Arc::default(), + } + } + + /// Lists every warning a step reported, once the steps are done. + pub fn finish_run(&self) { + let warnings = self.warnings.lock().expect("warnings"); + if warnings.is_empty() { + return; + } + let heading = if warnings.len() == 1 { + "Warning" + } else { + "Warnings" + }; + eprintln!("{}", style::paint(heading, style::BOLD, self.color)); + for warning in warnings.iter() { + eprintln!(" {} {warning}", style::paint("!", style::BOLD, self.color)); + } + eprintln!(); + } + + /// A step that will not run, because this is a dry run and it did not + /// promise to write nothing. + pub fn skipped(&self, index: usize, title: &str, label: &str) { + let line = if self.live { + format!( + "{} {} {}", + style::paint("○", &style::muted(), self.color), + style::paint(title, &style::muted(), self.color), + style::paint("skipped in a dry run", &style::muted(), self.color), + ) + } else { + format!( + "{} {} {}", + self.counter(index), + style::paint(title, style::BOLD, self.color), + style::paint( + &format!("({label}, skipped in a dry run)"), + &style::muted(), + self.color + ), + ) + }; + eprintln!("{line}"); + } + + pub fn start(&self, index: usize, title: &str, label: &str, animate: bool) -> StepView { + let view = StepView { + state: Arc::new(Mutex::new(State { + progress: String::new(), + frame: 0, + spinning: self.live && animate, + })), + stop: Arc::new(AtomicBool::new(false)), + ticker: None, + started: Instant::now(), + live: self.live, + color: self.color, + quiet: self.quiet, + warnings: Arc::clone(&self.warnings), + }; + + if self.live { + eprintln!("{}", self.header(title, (!animate).then_some(label))); + } + if self.live && animate { + draw_spinner(&view.state.lock().expect("progress state"), self.color); + let state = Arc::clone(&view.state); + let stop = Arc::clone(&view.stop); + let color = self.color; + let ticker = std::thread::spawn(move || { + while !stop.load(Ordering::Relaxed) { + std::thread::sleep(TICK); + let mut state = state.lock().expect("progress state"); + if stop.load(Ordering::Relaxed) { + break; + } + state.frame = (state.frame + 1) % FRAMES.len(); + draw_spinner(&state, color); + } + }); + return StepView { + ticker: Some(ticker), + ..view + }; + } + + if !self.live { + eprintln!( + "{} {} {}", + self.counter(index), + style::paint(title, style::BOLD, self.color), + style::paint(&format!("({label})"), &style::muted(), self.color), + ); + } + view + } + + /// `label` only for a step without a spinner, which has nowhere else to + /// say what is running. + fn header(&self, title: &str, label: Option<&str>) -> String { + let mut line = format!( + "{} {}", + style::paint("▸", &style::accent(), self.color), + style::paint(title, style::BOLD, self.color), + ); + if let Some(label) = label { + line.push_str(&format!( + " {}", + style::paint(label, &style::muted(), self.color) + )); + } + line + } + + fn counter(&self, index: usize) -> String { + style::paint( + &format!("[{}/{}]", index + 1, self.total), + &style::muted(), + self.color, + ) + } +} + +struct State { + progress: String, + frame: usize, + spinning: bool, +} + +/// One running step's place on stderr. Shared with the thread reading the +/// step's stderr, so every write goes through the one lock. +pub struct StepView { + state: Arc>, + stop: Arc, + ticker: Option>, + started: Instant, + live: bool, + color: bool, + quiet: bool, + warnings: Arc>>, +} + +impl StepView { + /// A handle the stderr reader can own. + pub fn reporter(&self) -> Reporter { + Reporter { + state: Arc::clone(&self.state), + live: self.live, + color: self.color, + quiet: self.quiet, + warnings: Arc::clone(&self.warnings), + } + } + + /// Replaces the status line with how the step ended. Nothing in the plain + /// mode: the next step's line, or the error, already says so. + pub fn finish(mut self, succeeded: bool) { + self.stop.store(true, Ordering::Relaxed); + if let Some(ticker) = self.ticker.take() { + let _ = ticker.join(); + } + if !self.live { + return; + } + let state = self.state.lock().expect("progress state"); + if state.spinning { + clear_line(); + } + let took = elapsed(self.started.elapsed()); + let (mark, paint, text) = if succeeded { + ("✓", &style::accent(), took) + } else { + ("✗", &style::error(), format!("failed after {took}")) + }; + // A blank line after, so each step reads as its own block. + eprintln!( + " {} {}\n", + style::paint(mark, paint, self.color), + style::paint(&text, &style::muted(), self.color), + ); + let _ = std::io::stderr().flush(); + } +} + +/// What the thread reading a step's stderr holds: the same view, without the +/// right to finish it. +pub struct Reporter { + state: Arc>, + live: bool, + color: bool, + quiet: bool, + warnings: Arc>>, +} + +impl Reporter { + /// One line the step wrote to stderr. + pub fn report(&self, line: &str) { + let mut state = self.state.lock().expect("progress state"); + if let Some(progress) = line.strip_prefix(PROGRESS_PREFIX) { + if state.spinning { + state.progress = progress.trim().to_string(); + draw_spinner(&state, self.color); + } + return; + } + let shown = match line.strip_prefix(WARN_PREFIX) { + Some(warning) => { + let warning = warning.trim().to_string(); + self.warnings + .lock() + .expect("warnings") + .push(warning.clone()); + if self.live { + format!("{} {warning}", style::paint("!", style::BOLD, self.color)) + } else { + format!("warning: {warning}") + } + } + None if self.quiet => return, + None => line.to_string(), + }; + if state.spinning { + clear_line(); + } + if self.live { + eprintln!("{DETAIL_INDENT}{shown}"); + } else { + eprintln!("{shown}"); + } + if state.spinning { + draw_spinner(&state, self.color); + } + } +} + +fn draw_spinner(state: &State, color: bool) { + let progress = if state.progress.is_empty() { + "working" + } else { + &state.progress + }; + eprint!( + "\r\x1b[2K {} {}", + style::paint(FRAMES[state.frame], &style::accent(), color), + style::paint(progress, &style::muted(), color), + ); + let _ = std::io::stderr().flush(); +} + +fn clear_line() { + eprint!("\r\x1b[2K"); +} + +fn elapsed(duration: Duration) -> String { + let seconds = duration.as_secs_f64(); + if seconds < 60.0 { + format!("{seconds:.1}s") + } else { + let whole = duration.as_secs(); + format!("{}m {:02}s", whole / 60, whole % 60) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_duration_reads_in_seconds_then_minutes() { + assert_eq!(elapsed(Duration::from_millis(2840)), "2.8s"); + assert_eq!(elapsed(Duration::from_secs(72)), "1m 12s"); + } +} diff --git a/src/workflow/prose.rs b/src/workflow/prose.rs new file mode 100644 index 0000000..0215d35 --- /dev/null +++ b/src/workflow/prose.rs @@ -0,0 +1,237 @@ +//! Renders a workflow's `description` for `workflow show`. +//! +//! A deliberately small subset of Markdown, the parts a description needs +//! and nothing a terminal cannot show: paragraphs separated by a blank line, +//! lines indented by two or more spaces as commands, `- ` items as a list, +//! and `code` spans. Anything else is a paragraph, so no description can +//! fail to render — it can only render plainly. +//! +//! Paragraphs and list items are reflowed to [`WIDTH`], so an author's own +//! line breaks inside a YAML `|` block do not leave ragged lines. Commands +//! are kept exactly as written. + +use crate::output::style; + +/// Columns a paragraph is reflowed to. There is no terminal-size lookup in +/// this crate, and 80 columns is what every terminal shows. +pub const WIDTH: usize = 80; + +const COMMAND_INDENT: &str = " "; + +enum Block { + Paragraph(String), + Commands(Vec), + List(Vec), +} + +fn is_command(line: &str) -> bool { + line.starts_with(" ") +} + +fn list_item(line: &str) -> Option<&str> { + line.strip_prefix("- ").or_else(|| line.strip_prefix("* ")) +} + +fn blocks(text: &str) -> Vec { + let mut out: Vec = vec![]; + for chunk in text.split("\n\n") { + // Only blocks from this chunk may be continued: a blank line always + // ends a list or a run of commands. + let first = out.len(); + let mut paragraph: Vec<&str> = vec![]; + for line in chunk.lines().filter(|line| !line.trim().is_empty()) { + let open = out.len() > first && paragraph.is_empty(); + match out.last_mut() { + // An indented line under a list item continues the item. + Some(Block::List(items)) if open && is_command(line) => { + let last = items.last_mut().expect("a list has an item"); + last.push(' '); + last.push_str(line.trim()); + continue; + } + _ => {} + } + if let Some(item) = list_item(line) { + flush(&mut paragraph, &mut out); + let continues = out.len() > first; + match out.last_mut() { + Some(Block::List(items)) if continues => items.push(item.trim().to_string()), + _ => out.push(Block::List(vec![item.trim().to_string()])), + } + } else if is_command(line) { + flush(&mut paragraph, &mut out); + let continues = out.len() > first; + match out.last_mut() { + Some(Block::Commands(commands)) if continues => { + commands.push(line.trim().to_string()) + } + _ => out.push(Block::Commands(vec![line.trim().to_string()])), + } + } else { + paragraph.push(line.trim()); + } + } + flush(&mut paragraph, &mut out); + } + out +} + +fn flush(paragraph: &mut Vec<&str>, out: &mut Vec) { + if !paragraph.is_empty() { + out.push(Block::Paragraph(paragraph.join(" "))); + paragraph.clear(); + } +} + +/// Words laid into lines of at most `width` columns. A word longer than a +/// line gets a line to itself rather than being broken. +fn wrap(text: &str, width: usize) -> Vec { + let mut lines = vec![]; + let mut line = String::new(); + for word in text.split_whitespace() { + let fits = line.is_empty() || line.chars().count() + 1 + word.chars().count() <= width; + if !fits { + lines.push(std::mem::take(&mut line)); + } + if !line.is_empty() { + line.push(' '); + } + line.push_str(word); + } + if !line.is_empty() { + lines.push(line); + } + lines +} + +/// Paints the `code` spans in one line, backticks included. `inside` +/// carries an open span over to the next line, since wrapping may split one. +fn paint_spans(line: &str, inside: &mut bool, color: bool) -> String { + if !color { + return line.to_string(); + } + let mut out = String::new(); + if *inside { + out.push_str(&style::accent()); + } + for c in line.chars() { + if c == '`' && !*inside { + out.push_str(&style::accent()); + out.push(c); + } else if c == '`' { + out.push(c); + out.push_str(style::RESET); + } else { + out.push(c); + } + if c == '`' { + *inside = !*inside; + } + } + if *inside { + out.push_str(style::RESET); + } + out +} + +fn reflowed(text: &str, first: &str, rest: &str, color: bool) -> Vec { + let width = WIDTH.saturating_sub(first.chars().count()).max(20); + let mut inside = false; + wrap(text, width) + .iter() + .enumerate() + .map(|(index, line)| { + let lead = if index == 0 { first } else { rest }; + format!("{lead}{}", paint_spans(line, &mut inside, color)) + }) + .collect() +} + +pub fn render(text: &str, color: bool) -> String { + blocks(text) + .into_iter() + .map(|block| match block { + Block::Paragraph(text) => reflowed(&text, "", "", color).join("\n"), + Block::Commands(commands) => commands + .iter() + .map(|command| { + format!( + "{COMMAND_INDENT}{}", + style::paint(command, &style::accent(), color) + ) + }) + .collect::>() + .join("\n"), + Block::List(items) => items + .iter() + .flat_map(|item| reflowed(item, " • ", " ", color)) + .collect::>() + .join("\n"), + }) + .collect::>() + .join("\n\n") +} + +#[cfg(test)] +mod tests { + use super::*; + + const DESCRIPTION: &str = "\ +Reads a style with one credential profile and creates a copy of it with +another. Log in to each account first: + + mapbox auth login --profile source + mapbox auth login --profile target + +Not copied: +- tilesets and fonts that belong to the source account, which the target + can read only if they are public +- the sprite +"; + + #[test] + fn paragraphs_reflow_commands_stay_and_lists_hang() { + assert_eq!( + render(DESCRIPTION, false), + "\ +Reads a style with one credential profile and creates a copy of it with another. +Log in to each account first: + + mapbox auth login --profile source + mapbox auth login --profile target + +Not copied: + + • tilesets and fonts that belong to the source account, which the target can + read only if they are public + • the sprite" + ); + } + + #[test] + fn a_command_right_under_a_paragraph_is_still_a_command() { + assert_eq!( + render("Log in first:\n mapbox auth login\nThen run it.", false), + "Log in first:\n\n mapbox auth login\n\nThen run it." + ); + } + + #[test] + fn color_changes_nothing_but_the_escapes() { + let colored = render("See `warnings` and `a b`.\n\n mapbox x", true); + assert!(colored.contains(&style::accent()), "{colored:?}"); + assert_eq!( + style::strip(&colored), + render("See `warnings` and `a b`.\n\n mapbox x", false) + ); + } + + #[test] + fn a_span_split_by_wrapping_stays_painted_on_both_lines() { + let long = format!("{} `one two three`", "word ".repeat(15).trim_end()); + let colored = render(&long, true); + let lines: Vec<&str> = colored.lines().collect(); + assert_eq!(lines.len(), 2, "{colored:?}"); + assert_eq!(style::strip(&colored), render(&long, false)); + } +} diff --git a/src/workflow/runner.rs b/src/workflow/runner.rs new file mode 100644 index 0000000..2115995 --- /dev/null +++ b/src/workflow/runner.rs @@ -0,0 +1,1138 @@ +//! Runs a checked workflow, one step at a time. +//! +//! A `command` step is this binary run again as a child, with `--output json` +//! and the step's arguments turned into flags. That is the point rather than +//! a shortcut: the child resolves its token, applies its timeouts, encodes +//! its path segments and records its run exactly as a command typed by hand +//! would, so a workflow cannot reach the API any way a person could not. +//! +//! Only the child's stdout is captured, as the step's output. Its stderr and +//! stdin are the terminal's, so progress, warnings and a confirmation prompt +//! reach the person running the workflow — unless the step feeds `stdin` +//! itself. +//! +//! A `script` step runs its file from the installed workflow's `scripts/` +//! under the interpreter the definition settled on. It is given `MAPBOX_CLI`, +//! the path to this binary, so it can call commands of its own. + +use std::cell::OnceCell; +use std::collections::BTreeMap; +use std::ffi::OsString; +use std::io::{BufRead, BufReader, Write}; +use std::path::Path; +use std::process::{Command as Process, Stdio}; +use std::sync::atomic::{AtomicBool, AtomicU32, Ordering}; + +use anyhow::{anyhow, Context as _, Result}; +use clap::{Arg, ArgAction, Command}; +use serde_json::{json, Map, Value}; + +use super::definition::{Action, InputType, Step, Workflow, DRY_RUN_ENV, SCRIPTS_DIR}; +use super::progress::{Display, StepView}; +use super::template::{self, Context}; +use super::workdir::{self, Workdir}; +use crate::auth; +use crate::output::{self, CliError}; + +/// Global options a step may not set, because the runner owns them or +/// because they would put a credential in a file. +const RESERVED: &[(&str, &str)] = &[ + ("output", "the runner reads every step as JSON"), + ("quiet", "the runner sets it"), + ( + "schema", + "a workflow runs commands rather than describing them", + ), + ( + "dry-run", + "use `mapbox workflow run --dry-run` for the whole workflow", + ), + ( + "token", + "a token written into a workflow is a secret in a file; use `profile`", + ), + ( + "yes", + "only the person running it can skip a confirmation, with `mapbox workflow run --yes`", + ), +]; + +/// The globals of the `workflow run` line that each command step inherits +/// unless it sets its own. +#[derive(Debug, Default, Clone)] +pub struct Inherited { + /// Only a token typed as `--token`; one from the environment reaches the + /// child by being in its environment already. + pub typed_token: Option, + pub options: Vec<(&'static str, String)>, + pub flags: Vec<&'static str>, + /// `--quiet`: steps show their title and how they ended, and warnings, + /// but not their details. + pub quiet: bool, +} + +impl Inherited { + pub fn from_matches(matches: &clap::ArgMatches) -> Self { + let mut inherited = Inherited { + typed_token: auth::typed_token(matches), + quiet: matches.get_flag(output::banner::ARG), + ..Default::default() + }; + for option in ["profile", "username", crate::http::TIMEOUT_ARG] { + let typed = + matches.value_source(option) == Some(clap::parser::ValueSource::CommandLine); + // Raw, as typed: `--timeout` is parsed into a `Duration`, and + // `get_one::` on it panics. + let value = matches + .get_raw(option) + .and_then(|mut values| values.next()) + .and_then(|value| value.to_str()); + if let (true, Some(value)) = (typed, value) { + inherited.options.push((option, value.to_string())); + } + } + for flag in ["use-login", "debug", crate::confirm::ARG] { + if matches.get_flag(flag) { + inherited.flags.push(flag); + } + } + inherited + } +} + +/// The flag an input is given as: its name, dashed, as every other flag in +/// this CLI is spelled. `--style_id` is accepted too, as an alias. +pub fn input_flag(name: &str) -> String { + name.replace('_', "-") +} + +/// Turns the values given for a workflow's inputs, by input name, into its +/// inputs typed as the definition declares, with defaults filled in. The +/// parser has already checked each value's shape; this still does, so it +/// can be trusted on its own. +pub fn read_inputs(workflow: &Workflow, given: &[(String, String)]) -> Result> { + let mut inputs: Map = Map::new(); + for (key, raw) in given { + let flag = input_flag(key); + let Some(spec) = workflow.inputs.get(key) else { + return Err(invalid_input( + format!("`{}` has no input called `{key}`", workflow.name), + workflow, + )); + }; + let value = match spec.kind { + InputType::String => Value::String(raw.to_string()), + InputType::Number => parse_number(raw).ok_or_else(|| { + invalid_input(format!("`--{flag}` is a number, not `{raw}`"), workflow) + })?, + InputType::Boolean => match raw.as_str() { + "true" => Value::Bool(true), + "false" => Value::Bool(false), + _ => { + return Err(invalid_input( + format!("`--{flag}` is `true` or `false`, not `{raw}`"), + workflow, + )) + } + }, + }; + inputs.insert(key.to_string(), value); + } + + let mut missing = vec![]; + for (name, spec) in &workflow.inputs { + if inputs.contains_key(name) { + continue; + } + match &spec.default { + Some(default) => { + inputs.insert(name.clone(), default.clone()); + } + None if spec.required => missing.push(format!("--{}", input_flag(name))), + None => { + inputs.insert(name.clone(), Value::Null); + } + } + } + if !missing.is_empty() { + return Err(invalid_input( + format!("missing required input: {}", missing.join(" ")), + workflow, + )); + } + Ok(inputs) +} + +pub fn parse_number(raw: &str) -> Option { + raw.parse::() + .map(Value::from) + .ok() + .or_else(|| raw.parse::().ok().map(Value::from)) +} + +fn invalid_input(message: String, workflow: &Workflow) -> anyhow::Error { + CliError::new("invalid_input", message) + .with_remedy( + crate::remedy::Remedy::default() + .with_action(Some(format!("mapbox workflow show {}", workflow.name))), + ) + .into() +} + +/// What `--dry-run` prints: the plan, with the inputs resolved and every +/// later reference left as written, since nothing has run to fill it. +pub fn plan(workflow: &Workflow, inputs: &Map) -> Value { + let steps: Vec = workflow + .steps + .iter() + .map(|step| { + let (kind, args) = match &step.action { + Action::Command { args, .. } => ("command", Value::Object(args.clone())), + Action::Script { args, .. } => ("script", Value::Array(args.clone())), + }; + json!({ + "id": step.id, + "kind": kind, + "run": step.label(), + "args": args, + "stdin": step.stdin, + "dry_run": step.dry_run, + }) + }) + .collect(); + json!({ "workflow": workflow.name, "inputs": inputs, "steps": steps, "outputs": workflow.outputs }) +} + +/// What a run ends with. +pub struct Finished { + /// What `outputs` resolves to, or the last step's output when the + /// workflow declares none. + pub outputs: Value, + /// What `result` resolves to, when the workflow declares one. + pub text: Option, +} + +/// Runs every step. +pub fn run( + app: &Command, + workflow: &Workflow, + root: &Path, + inputs: &Map, + inherited: &Inherited, +) -> Result { + let outputs = run_steps(app, workflow, root, inputs, inherited, false)?; + let context = Context { + inputs, + steps: &outputs, + }; + let text = match &workflow.result { + Some(template) => Some( + template::resolve(&Value::String(template.clone()), &context) + .map(|value| template::as_text(&value).unwrap_or_default()) + .map_err(|e| CliError::new("workflow_failed", format!("`result`: {e:#}")))?, + ), + None => None, + }; + + let last = workflow + .steps + .last() + .and_then(|step| outputs.get(&step.id)) + .cloned() + .unwrap_or(Value::Null); + let outputs = match &workflow.outputs { + Some(declared) => template::resolve(declared, &context) + .map_err(|e| CliError::new("workflow_failed", format!("`outputs`: {e:#}")))?, + None => last, + }; + Ok(Finished { outputs, text }) +} + +/// Runs only the steps marked `dry_run`, each with [`DRY_RUN_ENV`] set, and +/// returns their outputs by step id. `outputs` is not resolved: the steps it +/// reads may not have run. +pub fn dry_run( + app: &Command, + workflow: &Workflow, + root: &Path, + inputs: &Map, + inherited: &Inherited, +) -> Result> { + run_steps(app, workflow, root, inputs, inherited, true) +} + +fn run_steps( + app: &Command, + workflow: &Workflow, + root: &Path, + inputs: &Map, + inherited: &Inherited, + dry_run: bool, +) -> Result> { + let exe = std::env::current_exe().context("Could not find this program's own path")?; + catch_interrupt(); + let workdir = Workdir::create()?; + let mut outputs: BTreeMap = BTreeMap::new(); + let total = workflow.steps.len(); + + let display = Display::for_stderr(total, inherited.quiet); + for (index, step) in workflow.steps.iter().enumerate() { + // A Ctrl-C between steps reached no child, so nothing else stops the + // next one from starting. + if INTERRUPTED.load(Ordering::SeqCst) { + return Err(cancelled(&format!("before step `{}`", step.id))); + } + let title = step.name.as_deref().unwrap_or(&step.id); + if dry_run && !step.dry_run { + display.skipped(index, title, &step.label()); + continue; + } + + let context = Context { + inputs, + steps: &outputs, + }; + let stdin = step + .stdin + .as_ref() + .map(|value| template::resolve(value, &context)) + .transpose() + .map_err(|e| step_failure(step, e))?; + + let mut process = match &step.action { + Action::Command { path, args } => { + let args = match template::resolve(&Value::Object(args.clone()), &context) + .map_err(|e| step_failure(step, e))? + { + Value::Object(args) => args, + _ => unreachable!("an object resolves to an object"), + }; + let argv = + command_argv(app, path, &args, inherited).map_err(|e| step_failure(step, e))?; + let mut process = Process::new(&exe); + process.args(argv); + process + } + Action::Script { + script, + interpreter, + args, + } => { + let args = template::resolve(&Value::Array(args.clone()), &context) + .map_err(|e| step_failure(step, e))?; + let mut process = Process::new(interpreter); + process.arg(root.join(SCRIPTS_DIR).join(script)); + for value in args.as_array().into_iter().flatten() { + process.arg(template::as_text(value).unwrap_or_default()); + } + process + .env("MAPBOX_CLI", &exe) + .env("MAPBOX_WORKFLOW_ROOT", root) + .env(workdir::ENV, workdir.path()); + if dry_run { + process.env(DRY_RUN_ENV, "1"); + } + process + } + }; + if let Some(token) = &inherited.typed_token { + process.env(auth::CLAP_TOKEN_ENV, token); + } + + let is_script = matches!(step.action, Action::Script { .. }); + // A command step hands its error document back through a file, so + // the run can carry its code rather than print it raw. + let error_file = + (!is_script).then(|| workdir.path().join(format!(".error-{}.json", step.id))); + if let Some(path) = &error_file { + process.env(output::STEP_ERROR_ENV, path); + } + let view = display.start(index, title, &step.label(), is_script); + let result = execute( + &mut process, + stdin.as_ref(), + step, + &view, + workdir.path(), + error_file.as_deref(), + ); + let result = match &inherited.typed_token { + Some(_) => result.map_err(auth::with_typed_token_fix), + None => result, + }; + let interrupted = interrupted(result.is_err()); + view.finish(result.is_ok() && !interrupted); + if interrupted { + return Err(cancelled(&format!("during step `{}`", step.id))); + } + outputs.insert(step.id.clone(), result?); + } + + display.finish_run(); + Ok(outputs) +} + +static INTERRUPTED: AtomicBool = AtomicBool::new(false); + +/// The running step's process id, or 0 between steps. +static CURRENT_CHILD: AtomicU32 = AtomicU32::new(0); + +/// How long a step gets to stop on its own before the signal is passed on. +#[cfg(unix)] +const FORWARD_AFTER: std::time::Duration = std::time::Duration::from_secs(2); + +/// Ctrl-C reaches the running step's child as well as this process, and so +/// does the SIGHUP of a closed terminal; a SIGTERM reaches only this one. +/// The child stops; this process only notes it and fails once the child has +/// gone, so the run's [`Workdir`] is dropped rather than left in place by a +/// process that died holding it. +/// +/// For the SIGTERM a CI timeout sends, a step still running after +/// [`FORWARD_AFTER`] is sent a SIGINT: SIGINT rather than SIGTERM so a +/// script can still report what it did, as `copy-style` does on a +/// `KeyboardInterrupt`. `ctrlc` does not say which signal arrived, so +/// nothing is passed on while this process is the terminal's foreground: a +/// Ctrl-C there reached the step already, and a second one could cut that +/// report short. Only the step's own process is signaled, not anything it +/// started. +fn catch_interrupt() { + // On failure Ctrl-C keeps its default, which stops the run as before + // but leaves the directory behind; not worth failing the run over. + let _ = ctrlc::set_handler(|| { + INTERRUPTED.store(true, Ordering::SeqCst); + #[cfg(unix)] + { + if is_terminal_foreground() { + return; + } + std::thread::sleep(FORWARD_AFTER); + // A process id can be reused once its process is reaped. The + // window is the moment between the reap and the store of 0 + // after it, which is as narrow as this gets without `unsafe`. + let pid = CURRENT_CHILD.load(Ordering::SeqCst); + if let Ok(pid) = i32::try_from(pid) { + if pid > 0 { + let _ = nix::sys::signal::kill( + nix::unistd::Pid::from_raw(pid), + nix::sys::signal::Signal::SIGINT, + ); + } + } + } + }); +} + +/// Whether this process's group is its terminal's foreground, the one a +/// Ctrl-C reaches. No terminal, as in CI, is a no. +#[cfg(unix)] +fn is_terminal_foreground() -> bool { + use nix::unistd::{getpgrp, tcgetpgrp}; + let foreground = tcgetpgrp(std::io::stdin()).or_else(|_| tcgetpgrp(std::io::stderr())); + foreground == Ok(getpgrp()) +} + +fn cancelled(when: &str) -> anyhow::Error { + CliError::new( + "cancelled", + format!("Interrupted {when}; the steps after it did not run."), + ) + .into() +} + +/// Whether Ctrl-C was pressed. `ctrlc` runs the handler on a thread of its +/// own, so a child stopped by the same Ctrl-C can be reaped before the flag +/// is set; a step that failed waits a moment for it, so the run says it was +/// interrupted rather than that the step failed. +fn interrupted(step_failed: bool) -> bool { + let grace = if step_failed { + std::time::Duration::from_millis(250) + } else { + std::time::Duration::ZERO + }; + let deadline = std::time::Instant::now() + grace; + loop { + if INTERRUPTED.load(Ordering::SeqCst) { + return true; + } + if std::time::Instant::now() >= deadline { + return false; + } + std::thread::sleep(std::time::Duration::from_millis(10)); + } +} + +fn size(bytes: usize) -> String { + match bytes { + 0..=1023 => format!("{bytes} bytes"), + 1024..=1_048_575 => format!("{} KB", bytes / 1024), + _ => format!("{:.1} MB", bytes as f64 / 1_048_576.0), + } +} + +fn step_failure(step: &Step, err: anyhow::Error) -> anyhow::Error { + CliError::new( + "workflow_failed", + format!("Step `{}` ({}): {err:#}", step.id, step.label()), + ) + .into() +} + +/// A command step's failure, built from the error document the child wrote. +/// Still `workflow_failed`, so a caller knows the run stopped; the child's +/// own document is its `cause`, so a caller can branch on its `code`, and its +/// status and advice are lifted onto this one for a person to read. +fn command_failure(step: &Step, cause: Value) -> anyhow::Error { + let text = |key: &str| cause.get(key).and_then(Value::as_str).map(String::from); + let list = |key: &str| -> Vec { + cause + .get(key) + .and_then(Value::as_array) + .map(|items| { + items + .iter() + .filter_map(|item| item.as_str().map(String::from)) + .collect() + }) + .unwrap_or_default() + }; + let mut error = CliError::new( + "workflow_failed", + format!( + "Step `{}` ({}): {}", + step.id, + step.label(), + text("message").unwrap_or_else(|| "failed".to_string()) + ), + ); + error.status = cause + .get("status") + .and_then(Value::as_u64) + .and_then(|status| u16::try_from(status).ok()); + error.fix = text("fix"); + error.next_actions = list("next_actions"); + error.docs = list("docs"); + error.request_id = text("request_id"); + error.cause = Some(cause); + error.into() +} + +/// Spawns one step and reads its stdout as its output: JSON when it parses, +/// the text otherwise, nothing when there is none. +fn execute( + process: &mut Process, + stdin: Option<&Value>, + step: &Step, + view: &StepView, + workdir: &Path, + error_file: Option<&Path>, +) -> Result { + // A script with nothing to read gets nothing, rather than a terminal it + // might block on. A command keeps the terminal, which is where a + // confirmation prompt reads its answer. + let input = match (stdin, &step.action) { + (Some(_), _) => Stdio::piped(), + (None, Action::Script { .. }) => Stdio::null(), + (None, Action::Command { .. }) => Stdio::inherit(), + }; + // A script's stderr is read line by line, for its progress and details. + // A command's is the terminal's, so a confirmation prompt still shows. + let errors = match step.action { + Action::Script { .. } => Stdio::piped(), + Action::Command { .. } => Stdio::inherit(), + }; + let mut child = process + .stdin(input) + .stdout(Stdio::piped()) + .stderr(errors) + .spawn() + .map_err(|e| { + let program = process.get_program().to_string_lossy().into_owned(); + let message = if e.kind() == std::io::ErrorKind::NotFound { + format!("`{program}` is not installed or not on PATH") + } else { + format!("could not start `{program}`: {e}") + }; + step_failure(step, anyhow!(message)) + })?; + + // Written from a thread: a child that fills its stdout pipe before it + // has read all of stdin would otherwise wait on us while we wait on it. + let writer = stdin.map(|value| { + let body = match value { + Value::String(text) => text.clone(), + other => other.to_string(), + }; + let mut pipe = child.stdin.take().expect("stdin was piped"); + std::thread::spawn(move || { + // A child that exits without reading is reported by its status. + let _ = pipe.write_all(body.as_bytes()); + }) + }); + let reader = child.stderr.take().map(|pipe| { + let reporter = view.reporter(); + std::thread::spawn(move || { + for line in BufReader::new(pipe).lines().map_while(Result::ok) { + reporter.report(&line); + } + }) + }); + CURRENT_CHILD.store(child.id(), Ordering::SeqCst); + let finished = child.wait_with_output(); + CURRENT_CHILD.store(0, Ordering::SeqCst); + let finished = + finished.map_err(|e| step_failure(step, anyhow!("could not wait for it: {e}")))?; + if let Some(writer) = writer { + let _ = writer.join(); + } + // Every line the step wrote is shown before its outcome is. + if let Some(reader) = reader { + let _ = reader.join(); + } + + if !finished.status.success() { + let cause = error_file + .and_then(|path| std::fs::read(path).ok()) + .and_then(|bytes| serde_json::from_slice::(&bytes).ok()) + .filter(|cause| cause.get("code").is_some_and(Value::is_string)); + if let Some(cause) = cause { + return Err(command_failure(step, cause)); + } + let how = match finished.status.code() { + Some(code) => format!("exited with {code}"), + None => "was stopped by a signal".to_string(), + }; + return Err(step_failure( + step, + anyhow!("{how}; the workflow stopped here"), + )); + } + + if let Some(save) = &step.save { + let path = workdir.join(save); + std::fs::write(&path, &finished.stdout) + .map_err(|e| step_failure(step, anyhow!("could not save {}: {e}", path.display())))?; + let bytes = finished.stdout.len(); + view.reporter() + .report(&format!("Saved {save} ({})", size(bytes))); + return Ok(json!({ "path": path, "bytes": bytes })); + } + + let text = String::from_utf8(finished.stdout).map_err(|_| { + step_failure( + step, + anyhow!( + "its output is binary, which cannot be passed to another step; give it \ + `save: ` to keep it as a file instead" + ), + ) + })?; + let text = text.trim(); + if text.is_empty() { + return Ok(Value::Null); + } + Ok(serde_json::from_str(text).unwrap_or_else(|_| Value::String(text.to_string()))) +} + +enum ArgShape { + Flag, + Option { repeatable: bool }, + Positional { repeatable: bool }, +} + +/// The leaf command a step names, when it names one a workflow may run. +fn leaf<'a>(app: &'a Command, path: &[String]) -> Result<&'a Command, String> { + let named = format!("mapbox {}", path.join(" ")); + if path.first().map(String::as_str) == Some(super::COMMAND) { + return Err(format!("`{named}`: a workflow cannot run another workflow")); + } + let mut current = app; + for segment in path { + current = current + .find_subcommand(segment) + .ok_or_else(|| format!("`{named}` is not a command"))?; + } + if current.has_subcommands() { + return Err(format!("`{named}` is a group; name one of its commands")); + } + Ok(current) +} + +/// The argument a step's key names: one of the command's own, by its long +/// flag or, for a positional, its name — the same names `mapbox --schema` +/// lists — or a global option. +fn find_arg<'a>( + app: &'a Command, + command: &'a Command, + key: &str, +) -> Result<(&'a Arg, ArgShape), String> { + if let Some((_, why)) = RESERVED.iter().find(|(name, _)| *name == key) { + return Err(format!("`{key}` cannot be set by a step: {why}")); + } + let own = command + .get_arguments() + .find(|arg| arg.get_long() == Some(key) || (arg.is_positional() && arg.get_id() == key)); + let arg = own + .or_else(|| { + app.get_arguments() + .find(|arg| arg.is_global_set() && arg.get_long() == Some(key)) + }) + .ok_or_else(|| format!("`{key}` is not an argument of this command"))?; + let repeatable = matches!(arg.get_action(), ArgAction::Append); + let shape = if matches!(arg.get_action(), ArgAction::SetTrue | ArgAction::SetFalse) { + ArgShape::Flag + } else if arg.is_positional() { + ArgShape::Positional { repeatable } + } else { + ArgShape::Option { repeatable } + }; + Ok((arg, shape)) +} + +/// Whether a CLI at `current` satisfies `requires`, comparing numbers only: +/// a dev build of 0.4.0 is taken as 0.4.0, since a workflow requiring it was +/// most likely written against that build. +fn too_old(requires: &str, current: &str) -> Option { + let parse = crate::update_check::parse_version; + let (Some(needed), Some(this)) = (parse(requires), parse(current)) else { + return None; + }; + ((this.0, this.1, this.2) < (needed.0, needed.1, needed.2)).then(|| { + format!("it needs mapbox {requires} or later, and this is {current}; update the CLI") + }) +} + +/// Every `command` step's problems against this binary's command tree — +/// checked at install and again before a run, since the CLI may have been +/// upgraded underneath an installed workflow. +pub fn command_problems(app: &Command, workflow: &Workflow) -> Vec { + // Alone when it fails: an older CLI's other complaints are symptoms of + // the same thing, and a script step's needs show up nowhere else. + if let Some(problem) = workflow + .requires + .as_deref() + .and_then(|requires| too_old(requires, env!("CARGO_PKG_VERSION"))) + { + return vec![problem]; + } + + let mut problems = vec![]; + let read_only = OnceCell::new(); + + // An input is given as a flag beside the globals, so it cannot take one + // of their names — clap would refuse to build the command at all. + let taken: Vec<&str> = app + .get_arguments() + .filter(|arg| arg.is_global_set()) + .filter_map(Arg::get_long) + .chain([crate::executor::DRY_RUN_ARG, "help", "version"]) + .collect(); + for name in workflow.inputs.keys() { + let flag = input_flag(name); + if taken.contains(&flag.as_str()) { + problems.push(format!( + "input `{name}` would be given as `--{flag}`, which is already an option \ + of every command; rename it" + )); + } + } + for step in &workflow.steps { + let Action::Command { path, args } = &step.action else { + continue; + }; + let command = match leaf(app, path) { + Ok(command) => command, + Err(e) => { + problems.push(format!("step `{}`: {e}", step.id)); + continue; + } + }; + if step.dry_run + && !read_only + .get_or_init(read_only_commands) + .contains(&canonical(app, path)) + { + problems.push(format!( + "step `{}`: `dry_run` needs a command known to change nothing, and `mapbox {}` \ + is not one", + step.id, + path.join(" ") + )); + } + for (key, value) in args { + match find_arg(app, command, key) { + Err(e) => problems.push(format!("step `{}`: {e}", step.id)), + Ok((_, ArgShape::Flag)) if !(value.is_boolean() || value.is_string()) => problems + .push(format!( + "step `{}`: `{key}` is a flag; give it true or false", + step.id + )), + Ok(_) => {} + } + } + } + problems +} + +/// Hand-written commands a `dry_run` step may run: each reads local state or +/// sends only a `GET`. Kept as a list rather than inferred, because what a +/// command lacks says nothing about what it does — `tilesets-cli` has no +/// `--dry-run` either, and hands its arguments to a program that may write. +const READ_ONLY_COMMANDS: &[&str] = &[ + "auth whoami", + "auth profiles", + "config get", + "config list", + "history list", + "history show", +]; + +/// Every command a dry run may send: [`READ_ONLY_COMMANDS`] and every API +/// operation whose method is `GET`. +fn read_only_commands() -> Vec { + let mut commands: Vec = READ_ONLY_COMMANDS.iter().map(|c| c.to_string()).collect(); + // The tree being checked was built from these, so a failure here has + // already been reported by the parse; nothing is then allowed. + if let Ok(specs) = crate::spec::effective_services() { + commands.extend( + specs + .iter() + .flat_map(|spec| &spec.operations) + .filter(|op| op.method.eq_ignore_ascii_case("GET")) + .map(|op| op.command()), + ); + } + commands +} + +/// `path` with each alias replaced by the command's own name, so `auth +/// status` is checked as the `auth whoami` it is. +fn canonical(app: &Command, path: &[String]) -> String { + let mut current = app; + let mut names = vec![]; + for segment in path { + let Some(next) = current.find_subcommand(segment) else { + break; + }; + names.push(next.get_name().to_string()); + current = next; + } + names.join(" ") +} + +/// A step's arguments as the child's command line. +/// +/// Options are spelled `--name=value`, so a value that starts with a dash — +/// a western longitude — is never read as a flag. Positionals come last, +/// after `--`, for the same reason. +fn command_argv( + app: &Command, + path: &[String], + args: &Map, + inherited: &Inherited, +) -> Result> { + let command = leaf(app, path).map_err(|e| anyhow!(e))?; + let mut argv: Vec = path.iter().map(OsString::from).collect(); + let mut positionals: Vec<(usize, OsString)> = vec![]; + + for (key, value) in args { + let (arg, shape) = find_arg(app, command, key).map_err(|e| anyhow!(e))?; + let values: Vec<&Value> = match value { + Value::Array(items) if !matches!(shape, ArgShape::Flag) => items.iter().collect(), + Value::Null => continue, + other => vec![other], + }; + match shape { + ArgShape::Flag => match value { + Value::Bool(true) => argv.push(format!("--{key}").into()), + Value::Bool(false) => {} + Value::String(text) if text == "true" => argv.push(format!("--{key}").into()), + Value::String(text) if text == "false" => {} + _ => return Err(anyhow!("`{key}` is a flag; give it true or false")), + }, + ArgShape::Option { repeatable } | ArgShape::Positional { repeatable } + if values.len() > 1 && !repeatable => + { + return Err(anyhow!("`{key}` takes one value, not a list")); + } + ArgShape::Option { .. } => { + for value in values { + let text = template::as_text(value).unwrap_or_default(); + argv.push(format!("--{key}={text}").into()); + } + } + ArgShape::Positional { .. } => { + let index = arg.get_index().unwrap_or(usize::MAX); + for value in values { + positionals.push((index, template::as_text(value).unwrap_or_default().into())); + } + } + } + } + + for (option, value) in &inherited.options { + if !args.contains_key(*option) { + argv.push(format!("--{option}={value}").into()); + } + } + for flag in &inherited.flags { + if !args.contains_key(*flag) { + argv.push(format!("--{flag}").into()); + } + } + argv.push(format!("--{}={}", output::ARG, output::JSON).into()); + argv.push(format!("--{}", output::banner::ARG).into()); + + if !positionals.is_empty() { + positionals.sort_by_key(|(index, _)| *index); + argv.push("--".into()); + argv.extend(positionals.into_iter().map(|(_, value)| value)); + } + Ok(argv) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::workflow::definition::{parse, Files, DEFINITION_FILE}; + use std::path::PathBuf; + + fn app() -> Command { + let specs = crate::spec::effective_services().expect("bundled specs"); + crate::build_app(&specs) + } + + fn workflow(steps: &str) -> Workflow { + let yaml = format!( + "schema: 1\nname: t\nsummary: t\ninputs:\n n: {{ type: number, default: 2 }}\n flag: {{ type: boolean, default: false }}\nsteps:\n{steps}" + ); + let files: Files = [(PathBuf::from(DEFINITION_FILE), yaml.into_bytes())].into(); + parse("t", &files).expect("valid") + } + + /// A dry run sends every step it runs, so a command step can be in one + /// only if its command is known to change nothing. + #[test] + fn only_a_command_known_to_change_nothing_runs_in_a_dry_run() { + let found = command_problems( + &app(), + &workflow( + " - id: read\n command: styles get\n args: { style-id: x }\n dry_run: true\n\ + \x20 - id: local\n command: config list\n dry_run: true\n\ + \x20 - id: alias\n command: auth status\n dry_run: true\n\ + \x20 - id: write\n command: styles create\n dry_run: true\n\ + \x20 - id: passthrough\n command: tilesets-cli\n dry_run: true\n", + ), + ) + .join("\n"); + for refused in ["write", "passthrough"] { + assert!( + found.contains(&format!( + "step `{refused}`: `dry_run` needs a command known to change nothing" + )), + "{found}" + ); + } + for allowed in ["read", "local", "alias"] { + assert!(!found.contains(&format!("step `{allowed}`")), "{found}"); + } + } + + /// The list names real commands; one renamed out from under it would + /// silently stop being allowed. + #[test] + fn every_read_only_command_exists() { + let app = app(); + for command in READ_ONLY_COMMANDS { + let path: Vec = command.split(' ').map(String::from).collect(); + assert_eq!(canonical(&app, &path), *command, "{command}"); + assert!(leaf(&app, &path).is_ok(), "{command}"); + } + } + + #[test] + fn a_step_is_checked_against_the_real_command_tree() { + let found = command_problems( + &app(), + &workflow( + " - id: a\n command: styles nope\n\ + \x20 - id: b\n command: styles\n\ + \x20 - id: c\n command: styles get\n args: { bogus: 1, token: x }\n\ + \x20 - id: d\n command: workflow list\n\ + \x20 - id: e\n command: styles delete\n args: { style-id: x, yes: true }\n", + ), + ); + let joined = found.join("\n"); + assert!( + joined.contains("step `e`") && joined.contains("only the person running it"), + "{joined}" + ); + assert!( + joined.contains("`mapbox styles nope` is not a command"), + "{joined}" + ); + assert!(joined.contains("is a group"), "{joined}"); + assert!(joined.contains("`bogus` is not an argument"), "{joined}"); + assert!(joined.contains("a secret in a file"), "{joined}"); + assert!(joined.contains("cannot run another workflow"), "{joined}"); + } + + /// A step is handed a typed `--token` through the environment, so its + /// own advice for a 401 names the variable rather than the flag. + #[test] + fn a_typed_token_is_blamed_on_the_flag_not_the_environment() { + let step = &workflow(" - id: list\n command: styles list\n").steps[0]; + let cause = serde_json::json!({ + "code": "http_401", + "message": "Not Authorized - Invalid Token", + "status": 401, + "fix": "The token came from MAPBOX_ACCESS_TOKEN. Run `mapbox auth login` to sign in with an account instead.", + "next_actions": ["mapbox auth login"], + }); + let err = auth::with_typed_token_fix(command_failure(step, cause)); + let err = err.downcast::().expect("a CliError"); + let fix = err.fix.as_deref().unwrap_or_default(); + assert!(fix.contains("`--token`"), "{fix}"); + assert!(!fix.contains("came from MAPBOX_ACCESS_TOKEN"), "{fix}"); + assert_eq!(err.next_actions, ["mapbox auth whoami"]); + assert_eq!(err.cause.as_ref().unwrap()["code"], "http_401"); + + let cause = serde_json::json!({ + "code": "http_403", + "message": "This API requires a token with styles:download scope.", + "status": 403, + "fix": "The token from MAPBOX_ACCESS_TOKEN lacks the `styles:download` scope.", + }); + let err = auth::with_typed_token_fix(command_failure(step, cause)); + let fix = err.downcast::().unwrap().fix.unwrap_or_default(); + assert!( + fix.contains("passed with `--token` lacks the `styles:download`"), + "{fix}" + ); + } + + /// Through the real parser, which is where `--timeout` stops being a + /// string. + #[test] + fn typed_globals_are_inherited_as_typed() { + for args in [ + [ + "mapbox", + "--timeout", + "30", + "--profile", + "work", + "workflow", + "list", + ], + [ + "mapbox", + "workflow", + "list", + "--timeout", + "30", + "--profile", + "work", + ], + ] { + let matches = app().try_get_matches_from(args).expect("parses"); + let inherited = Inherited::from_matches(&matches); + assert_eq!( + inherited.options, + [ + ("profile", "work".to_string()), + ("timeout", "30".to_string()) + ], + "{args:?}" + ); + } + } + + #[test] + fn arguments_become_flags_and_positionals_come_last() { + let app = app(); + let args = serde_json::json!({ + "style-id": "-abc", + "optimize": true, + "download": false, + "profile": "work", + }); + let inherited = Inherited { + options: vec![("profile", "ignored".into()), ("timeout", "30".into())], + flags: vec!["yes"], + ..Default::default() + }; + let argv = command_argv( + &app, + &["styles".into(), "get".into()], + args.as_object().unwrap(), + &inherited, + ) + .unwrap(); + let argv: Vec = argv + .iter() + .map(|a| a.to_string_lossy().into_owned()) + .collect(); + assert_eq!( + argv, + [ + "styles", + "get", + "--optimize", + "--profile=work", + "--timeout=30", + "--yes", + "--output=json", + "--quiet", + "--", + "-abc" + ] + ); + } + + #[test] + fn inputs_are_typed_and_defaulted() { + let wf = workflow(" - id: a\n command: styles list\n"); + let given = |pairs: &[(&str, &str)]| -> Vec<(String, String)> { + pairs + .iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect() + }; + let inputs = read_inputs(&wf, &given(&[("n", "5")])).unwrap(); + assert_eq!(inputs["n"], serde_json::json!(5)); + assert_eq!(inputs["flag"], serde_json::json!(false)); + assert!(read_inputs(&wf, &given(&[("n", "five")])).is_err()); + assert!(read_inputs(&wf, &given(&[("nope", "1")])).is_err()); + } + + #[test] + fn an_input_cannot_take_a_global_flag() { + let yaml = "schema: 1\nname: t\nsummary: t\ninputs:\n profile: { type: string }\n dry_run: { type: boolean }\nsteps:\n - id: a\n command: styles list\n"; + let files: Files = [(PathBuf::from(DEFINITION_FILE), yaml.as_bytes().to_vec())].into(); + let found = command_problems(&app(), &parse("t", &files).unwrap()).join("\n"); + assert!(found.contains("`--profile`"), "{found}"); + assert!(found.contains("`--dry-run`"), "{found}"); + } + + #[test] + fn requires_compares_release_numbers() { + assert!(too_old("0.4.0", "0.3.9").is_some()); + assert!(too_old("0.4.0", "0.4.0").is_none()); + assert!(too_old("0.4.0", "0.4.0-dev.abc1234").is_none()); + assert!(too_old("0.4.0", "1.0.0").is_none()); + } + + /// A workflow for a newer CLI is refused with that reason alone, even + /// though its unknown command would be refused too. + #[test] + fn a_newer_cli_requirement_is_the_only_problem_reported() { + let yaml = "schema: 1\nname: t\nrequires: '>=999.0.0'\nsummary: t\nsteps:\n - id: a\n command: no such\n"; + let files: Files = [(PathBuf::from(DEFINITION_FILE), yaml.as_bytes().to_vec())].into(); + let found = command_problems(&app(), &parse("t", &files).unwrap()); + assert_eq!(found.len(), 1, "{found:?}"); + assert!( + found[0].contains("needs mapbox 999.0.0 or later"), + "{found:?}" + ); + } +} diff --git a/src/workflow/store.rs b/src/workflow/store.rs new file mode 100644 index 0000000..69e9439 --- /dev/null +++ b/src/workflow/store.rs @@ -0,0 +1,1117 @@ +//! Where installed workflows live, and how they get there. +//! +//! Nothing is bundled into the binary: a workflow runs only once it has been +//! installed, from a local directory or from a GitHub repository laid out as +//! `workflow//`, into `/workflows//`. A copy +//! is taken rather than a link kept, so what runs is what was checked at +//! install time and nothing edited since. +//! +//! The same rules as `agent-skills` hold for the write, for the same +//! reasons: everything is read and checked in memory first, an existing +//! workflow stops the install unless `--force`, only regular files are +//! taken, and a path that would leave the workflow's directory stops the +//! read altogether. The new directory is staged and renamed into place, and +//! one it replaces is moved aside until then rather than removed first. + +use std::io::Read; +use std::path::{Component, Path, PathBuf}; +use std::time::{SystemTime, UNIX_EPOCH}; + +use anyhow::{anyhow, Context, Result}; +use serde::{Deserialize, Serialize}; + +use super::definition::{self, is_contained, is_workflow_name, Files, Workflow}; +use crate::auth; +use crate::executor; +use crate::http; +use crate::output::CliError; +use crate::remedy::Remedy; + +const DIR_NAME: &str = "workflows"; + +/// Written beside an installed workflow's files, and what marks a directory +/// as one this command installed — `uninstall` removes nothing without it. +const META_FILE: &str = ".install.json"; + +/// Where the tarball comes from. The API rather than codeload directly, +/// because the API is what honors a token for a private repository; it +/// answers with a redirect to a signed codeload URL, and `reqwest` drops the +/// `Authorization` header when it follows a redirect to another host. +const GITHUB_API: &str = "https://api.github.com"; +/// This repository. Not `REPO_URL`'s `mapbox/cli`: on GitHub that name +/// redirects to a different repository. +pub const DEFAULT_REPO: &str = "mapbox/mapbox-cli"; +pub const DEFAULT_REF: &str = "main"; + +/// The directory inside a repository that holds its workflows, one +/// directory each. +pub const REPO_PREFIX: &str = "workflow"; + +/// A ceiling on what is read, from a tarball or a local directory. Far above +/// any real workflow, and there so a hostile archive cannot make this +/// allocate without bound. +const MAX_BYTES: u64 = 64 * 1024 * 1024; + +fn too_large(what: &str, limit: u64) -> anyhow::Error { + anyhow!( + "{what} is larger than {} MiB, more than a workflow should be", + limit / (1024 * 1024) + ) +} + +/// All of `reader`, or an error once it passes `limit`. One byte more than +/// the limit is read, so an oversized source is said to be one rather than +/// cut short into something that reads as corrupt. +fn read_capped(reader: impl Read, limit: u64, what: &str) -> Result> { + let mut bytes = vec![]; + reader + .take(limit + 1) + .read_to_end(&mut bytes) + .with_context(|| format!("Could not read {what}"))?; + if bytes.len() as u64 > limit { + return Err(too_large(what, limit)); + } + Ok(bytes) +} + +/// The files `read_local` passes over rather than refusing. Finder writes +/// `.DS_Store` into any directory it opens. +const IGNORED: &[&str] = &[".DS_Store"]; + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Meta { + /// Where it came from, as a person would write it: a directory, or + /// `github:owner/repo`. + pub source: String, + #[serde(default, rename = "ref")] + pub git_ref: Option, + /// The commit `git_ref` named when it was installed, since a branch + /// moves. Absent for a local install, and for one made before it was + /// recorded. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub commit: Option, + pub installed_at: u64, +} + +/// A workflow read and checked, not yet written anywhere. +pub struct Package { + pub workflow: Workflow, + pub files: Files, + pub meta: Meta, +} + +/// A workflow on disk. +pub struct Installed { + pub root: PathBuf, + pub workflow: Workflow, + pub meta: Meta, +} + +/// `/workflows`, resolved and not created. +pub fn root_path() -> Result { + auth::config_dir_path() + .map(|dir| dir.join(DIR_NAME)) + .ok_or_else(|| anyhow!("Could not determine home directory")) +} + +pub fn invalid_workflow(name: &str, problems: &[String]) -> anyhow::Error { + let listed = problems + .iter() + .map(|problem| format!(" - {problem}")) + .collect::>() + .join("\n"); + CliError::new( + "invalid_workflow", + format!("`{name}` is not a valid workflow:\n{listed}"), + ) + .into() +} + +/// A workflow name from the command line, checked before it is ever joined +/// to a path. +pub fn check_name(name: &str) -> Result<()> { + if is_workflow_name(name) { + return Ok(()); + } + Err(CliError::new( + "invalid_name", + format!("`{name}` is not a workflow name: lower-case letters, digits and dashes"), + ) + .with_remedy(Remedy::default().with_action(Some("mapbox workflow list".to_string()))) + .into()) +} + +fn now() -> u64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|elapsed| elapsed.as_secs()) + .unwrap_or(0) +} + +/// Every regular file under `dir`, by path relative to it. +/// +/// A symlink is refused rather than followed: it is a way to install +/// something from outside the directory that was named. +fn read_tree(dir: &Path, skip: &[&str], limit: u64) -> Result { + let mut files = Files::new(); + let mut total: u64 = 0; + let what = dir.display().to_string(); + let mut pending = vec![dir.to_path_buf()]; + while let Some(current) = pending.pop() { + let entries = std::fs::read_dir(¤t) + .with_context(|| format!("Could not read {}", current.display()))?; + for entry in entries { + let entry = entry.with_context(|| format!("Could not read {}", current.display()))?; + let path = entry.path(); + let relative = path.strip_prefix(dir).expect("under the directory walked"); + let name = entry.file_name(); + if current == dir && skip.iter().any(|s| name == *s) { + continue; + } + if IGNORED.iter().any(|ignored| name == *ignored) { + continue; + } + let kind = entry + .file_type() + .with_context(|| format!("Could not read {}", path.display()))?; + if kind.is_dir() { + pending.push(path); + } else if kind.is_file() { + // Capped at what is left, so one huge file is refused + // without being read whole first. + let mut bytes = vec![]; + std::fs::File::open(&path) + .and_then(|file| file.take(limit - total + 1).read_to_end(&mut bytes)) + .with_context(|| format!("Could not read {}", path.display()))?; + total += bytes.len() as u64; + if total > limit { + return Err(too_large(&what, limit)); + } + files.insert(relative.to_path_buf(), bytes); + } else { + return Err(CliError::new( + "invalid_workflow", + format!( + "{} is not a regular file; a workflow holds only files and directories", + path.display() + ), + ) + .into()); + } + } + } + Ok(files) +} + +fn package(name: &str, files: Files, meta: Meta) -> Result { + let workflow = definition::parse(name, &files).map_err(|p| invalid_workflow(name, &p))?; + Ok(Package { + workflow, + files, + meta, + }) +} + +/// A workflow from a directory on this machine. The directory's own name is +/// the workflow's, as it is in a repository. +pub fn read_local(dir: &Path) -> Result { + let dir = dir + .canonicalize() + .with_context(|| format!("There is no directory {}", dir.display()))?; + if !dir.is_dir() { + return Err(anyhow!("{} is not a directory", dir.display())); + } + let name = dir + .file_name() + .map(|n| n.to_string_lossy().into_owned()) + .unwrap_or_default(); + let files = read_tree(&dir, &[], MAX_BYTES)?; + package( + &name, + files, + Meta { + source: dir.display().to_string(), + git_ref: None, + commit: None, + installed_at: now(), + }, + ) +} + +/// `owner/repo`, and nothing that could move the request elsewhere. +pub fn check_repo(repo: &str) -> Result<()> { + let part = |p: &str| { + !p.is_empty() + && p != "." + && p != ".." + && p.chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '-' | '_' | '.')) + }; + match repo.split_once('/') { + Some((owner, name)) if part(owner) && part(name) => Ok(()), + _ => Err(CliError::new( + "invalid_repo", + format!("`{repo}` is not a GitHub repository; write it as OWNER/REPO"), + ) + .into()), + } +} + +/// A branch, tag or commit. `/` is allowed, since a branch may hold one; +/// URL syntax and `..` are not, since the ref goes into a URL path. +pub fn check_ref(git_ref: &str) -> Result<()> { + let ok = !git_ref.is_empty() + && !git_ref + .split('/') + .any(|part| part.is_empty() || part == "." || part == "..") + && git_ref + .chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '-' | '_' | '.' | '/')); + if ok { + return Ok(()); + } + Err(CliError::new( + "invalid_ref", + format!("`{git_ref}` is not a branch, tag or commit this can fetch"), + ) + .into()) +} + +fn github_token() -> Option { + ["GH_TOKEN", "GITHUB_TOKEN"] + .iter() + .filter_map(|name| std::env::var(name).ok()) + .find(|value| !value.trim().is_empty()) +} + +/// A workflow out of a GitHub repository's tarball. +pub fn fetch_github(name: &str, repo: &str, git_ref: &str, debug: bool) -> Result { + fetch_github_from(GITHUB_API, name, repo, git_ref, debug) +} + +fn fetch_github_from( + base: &str, + name: &str, + repo: &str, + git_ref: &str, + debug: bool, +) -> Result { + check_name(name)?; + check_repo(repo)?; + check_ref(git_ref)?; + + // The commit first, then the tarball of that commit rather than of the + // ref, so what is recorded is what was downloaded even if the branch + // moves in between. + let token = github_token(); + let commit = get( + &format!("{base}/repos/{repo}/commits/{git_ref}"), + "application/vnd.github.sha", + token.as_deref(), + (repo, git_ref), + debug, + )? + .text() + .map_err(|e| anyhow!("Could not read the commit from GitHub: {e}"))?; + let commit = commit.trim().to_string(); + if !is_commit_id(&commit) { + return Err(anyhow!( + "GitHub answered `{git_ref}` in {repo} with something that is not a commit id" + )); + } + let response = get( + &format!("{base}/repos/{repo}/tarball/{commit}"), + "application/vnd.github+json", + token.as_deref(), + (repo, git_ref), + debug, + )?; + + let bytes = read_capped(response, MAX_BYTES, "the archive from GitHub")?; + let files = extract(&bytes, name, repo, MAX_BYTES)?; + package( + name, + files, + Meta { + source: format!("github:{repo}"), + git_ref: Some(git_ref.to_string()), + commit: Some(commit), + installed_at: now(), + }, + ) +} + +/// Whether `commit` is on `repo`'s `main`, so has been through its review. +/// +/// What a ref names is not enough: GitHub resolves `pull/N/head`, and a +/// commit that only exists in a fork, through the repository's own +/// `commits/{ref}`. Any failure to tell counts as no, which means asking. +pub fn is_on_main(repo: &str, commit: &str, debug: bool) -> bool { + is_on_main_from(GITHUB_API, repo, commit, debug) +} + +fn is_on_main_from(base: &str, repo: &str, commit: &str, debug: bool) -> bool { + if !is_commit_id(commit) { + return false; + } + // Page 2 of one commit per page, because only the first page carries + // the changed files, which for an old commit can be megabytes of diff. + let url = format!("{base}/repos/{repo}/compare/{DEFAULT_REF}...{commit}?per_page=1&page=2"); + let body = get( + &url, + "application/vnd.github+json", + github_token().as_deref(), + (repo, commit), + debug, + ) + .and_then(|response| read_capped(response, MAX_BYTES, "the comparison from GitHub")) + .and_then(|body| Ok(serde_json::from_slice::(&body)?)); + match body { + // `behind` and `identical`: `commit` is an ancestor of `main`. + Ok(body) => matches!(body["status"].as_str(), Some("behind" | "identical")), + Err(e) => { + if debug { + eprintln!("[debug] could not tell whether {commit} is on {repo}'s main: {e:#}"); + } + false + } + } +} + +/// A full SHA-1 or SHA-256 commit id, as `application/vnd.github.sha` +/// answers. Checked because it goes into the next URL and into +/// `.install.json`. +fn is_commit_id(value: &str) -> bool { + matches!(value.len(), 40 | 64) && value.bytes().all(|b| b.is_ascii_hexdigit()) +} + +/// A GET to GitHub's API, with an error that says what `asked` — the +/// repository and the ref — when it fails. +fn get( + url: &str, + accept: &str, + token: Option<&str>, + asked: (&str, &str), + debug: bool, +) -> Result { + let (repo, git_ref) = asked; + if debug { + eprintln!("[debug] GET {url}"); + } + let mut request = http::client()?.get(url).header("Accept", accept); + if let Some(token) = token { + request = request.bearer_auth(token); + } + let response = http::send(request) + .map_err(|e| executor::transport_failure("Could not reach GitHub", e))?; + + let status = response.status(); + if status.is_success() { + return Ok(response); + } + // A private repository answers 404 to a request without a token, the + // same as a ref that does not exist, so both are named. + let message = match (status.as_u16(), token) { + (404, None) => format!( + "GitHub found no `{git_ref}` in {repo}. If the repository is private, set \ + GITHUB_TOKEN to a token that can read it." + ), + (404, Some(_)) => format!( + "GitHub found no `{git_ref}` in {repo}, or the token in GH_TOKEN/GITHUB_TOKEN \ + cannot read it." + ), + _ => format!("GitHub answered {status} for {url}."), + }; + let mut remedy = Remedy::default(); + if token.is_none() { + remedy = remedy.with_fix("export GITHUB_TOKEN=\"$(gh auth token)\""); + } + Err(CliError::http(status.as_u16(), &message) + .with_remedy(remedy) + .into()) +} + +fn unsafe_entry(path: &Path) -> anyhow::Error { + CliError::new( + "unsafe_archive", + format!( + "Refusing to read `{}` out of the archive: it points outside the directory it \ + would be written to.", + path.display() + ), + ) + .into() +} + +/// The files of `workflow//` in a repository tarball. +fn extract(archive: &[u8], name: &str, repo: &str, limit: u64) -> Result { + // Unpacked whole first: a small archive can inflate to far more than it + // is, and the cap has to apply to what it inflates to. + let unpacked = read_capped( + flate2::read::GzDecoder::new(archive), + limit, + "the archive from GitHub, unpacked,", + )?; + let mut tar = tar::Archive::new(unpacked.as_slice()); + + let mut files = Files::new(); + let mut available: std::collections::BTreeSet = Default::default(); + for entry in tar + .entries() + .context("The archive from GitHub is not readable as a tarball")? + { + let mut entry = entry.context("The archive from GitHub ended unexpectedly")?; + // Only regular files: a link is a way out of the destination no path + // check would catch. + if entry.header().entry_type() != tar::EntryType::Regular { + continue; + } + let path = entry + .path() + .context("The archive holds an entry whose path is not valid UTF-8")? + .into_owned(); + // GitHub wraps every archive in one directory named after the + // repository and the commit, so it is stripped by position. + let mut components = path.components(); + let Some(Component::Normal(_)) = components.next() else { + return Err(unsafe_entry(&path)); + }; + let inner: PathBuf = components.collect(); + if !is_contained(&inner) { + return Err(unsafe_entry(&path)); + } + let Ok(within) = inner.strip_prefix(REPO_PREFIX) else { + continue; + }; + let mut parts = within.components(); + let Some(workflow) = parts.next() else { + continue; + }; + let relative: PathBuf = parts.collect(); + if relative.as_os_str().is_empty() { + continue; + } + if workflow.as_os_str() != name { + available.insert(workflow.as_os_str().to_string_lossy().into_owned()); + continue; + } + let mut bytes = vec![]; + entry + .read_to_end(&mut bytes) + .with_context(|| format!("Could not read {} out of the archive", path.display()))?; + files.insert(relative, bytes); + } + + if files.is_empty() { + let listed = if available.is_empty() { + format!("{repo} publishes no workflows under `{REPO_PREFIX}/`.") + } else { + format!( + "It publishes: {}.", + available.into_iter().collect::>().join(", ") + ) + }; + return Err(CliError::new( + "workflow_not_found", + format!("{repo} has no workflow called `{name}`. {listed}"), + ) + .into()); + } + Ok(files) +} + +/// Writes `package` into place, replacing an existing install only when +/// `force` says so. Returns where it went. +pub fn install(package: &Package, force: bool) -> Result { + let name = &package.workflow.name; + check_name(name)?; + let root = root_path()?; + std::fs::create_dir_all(&root) + .with_context(|| format!("Could not create {}", root.display()))?; + let target = root.join(name); + let previous = previous_path(&root, name); + recover(&root, name)?; + if target.exists() && !force { + return Err(already_installed(name, &target, None)); + } + + let staging = staging_path(&root, name); + let _ = std::fs::remove_dir_all(&staging); + let result = (|| -> Result<()> { + for (relative, bytes) in &package.files { + let path = staging.join(relative); + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent) + .with_context(|| format!("Could not create {}", parent.display()))?; + } + std::fs::write(&path, bytes) + .with_context(|| format!("Could not write {}", path.display()))?; + } + std::fs::write( + staging.join(META_FILE), + serde_json::to_string_pretty(&package.meta)?, + ) + .context("Could not record where the workflow came from")?; + + swap_in(&staging, &target, &previous, |from, to| { + std::fs::rename(from, to) + }) + })(); + let _ = std::fs::remove_dir_all(&staging); + result.map(|()| target) +} + +fn previous_path(root: &Path, name: &str) -> PathBuf { + root.join(format!(".{name}.previous")) +} + +fn staging_path(root: &Path, name: &str) -> PathBuf { + root.join(format!(".{name}.staging")) +} + +/// Puts back the copy an install moved aside, when it stopped before moving +/// the new one in: that copy is then the workflow. Called wherever an +/// installed workflow is looked for, so a crash in between never reads as +/// "not installed". +pub fn recover_name(name: &str) -> Result<()> { + check_name(name)?; + recover(&root_path()?, name) +} + +fn recover(root: &Path, name: &str) -> Result<()> { + let (target, previous) = (root.join(name), previous_path(root, name)); + if !target.exists() && previous.exists() { + std::fs::rename(&previous, &target) + .with_context(|| format!("Could not restore {}", target.display()))?; + } + Ok(()) +} + +/// Moves `staged` into `target`. An existing `target` is moved aside to +/// `previous` rather than removed first, and moved back if the new copy +/// cannot go in, so a failed `--force` leaves the old workflow installed. +/// `rename` is a parameter so a test can make the last move fail. +fn swap_in( + staged: &Path, + target: &Path, + previous: &Path, + rename: impl Fn(&Path, &Path) -> std::io::Result<()>, +) -> Result<()> { + let _ = std::fs::remove_dir_all(previous); + let replacing = target.exists(); + if replacing { + rename(target, previous) + .with_context(|| format!("Could not move {} aside", target.display()))?; + } + if let Err(e) = rename(staged, target) { + if replacing { + rename(previous, target).with_context(|| { + format!( + "Could not move the workflow into {}, nor put the old one back; it is at {}", + target.display(), + previous.display() + ) + })?; + } + return Err(anyhow::Error::new(e).context(format!( + "Could not move the workflow into {}", + target.display() + ))); + } + if replacing { + let _ = std::fs::remove_dir_all(previous); + } + Ok(()) +} + +/// `retry` is the `install` line that would replace it, when the caller +/// knows what was typed. +pub fn already_installed(name: &str, target: &Path, retry: Option) -> anyhow::Error { + CliError::new( + "already_installed", + format!("`{name}` is already installed at {}.", super::tilde(target)), + ) + .with_remedy( + Remedy::default() + .with_fix("Pass --force to replace it.") + .with_action(retry), + ) + .into() +} + +fn not_installed(name: &str) -> anyhow::Error { + CliError::new( + "workflow_not_installed", + format!("No workflow called `{name}` is installed."), + ) + .with_remedy( + Remedy::default() + .with_action(Some(format!("mapbox workflow install {name}"))) + .with_action(Some("mapbox workflow list".to_string())), + ) + .into() +} + +/// An installed workflow's directory, if there is one. Only a directory +/// holding [`META_FILE`] counts. +pub fn installed_dir(name: &str) -> Result> { + check_name(name)?; + let root = root_path()?; + // A failed restore leaves the copy where it was; `install` reports it. + let _ = recover(&root, name); + let dir = root.join(name); + Ok(dir.join(META_FILE).is_file().then_some(dir)) +} + +/// Reads an installed workflow back and checks it again: a file edited by +/// hand since the install is refused here rather than failing mid-run. +pub fn load(name: &str) -> Result { + let root = installed_dir(name)?.ok_or_else(|| not_installed(name))?; + load_dir(name, root) +} + +fn load_dir(name: &str, root: PathBuf) -> Result { + let meta: Meta = serde_json::from_slice( + &std::fs::read(root.join(META_FILE)) + .with_context(|| format!("Could not read {}", root.join(META_FILE).display()))?, + ) + .with_context(|| format!("{} is not readable", root.join(META_FILE).display()))?; + let files = read_tree(&root, &[META_FILE], MAX_BYTES)?; + let workflow = definition::parse(name, &files).map_err(|p| invalid_workflow(name, &p))?; + Ok(Installed { + root, + workflow, + meta, + }) +} + +/// Every installed workflow, by name, with the ones that no longer load +/// kept as their error. +pub fn list() -> Result)>> { + let root = root_path()?; + let Ok(entries) = std::fs::read_dir(&root) else { + return Ok(vec![]); + }; + // First, so a workflow an interrupted install left aside is listed. + let entries: Vec<_> = entries.flatten().map(|entry| entry.file_name()).collect(); + for name in &entries { + let name = name.to_string_lossy(); + if let Some(moved) = name + .strip_prefix('.') + .and_then(|rest| rest.strip_suffix(".previous")) + .filter(|moved| is_workflow_name(moved)) + { + let _ = recover(&root, moved); + } + } + let Ok(entries) = std::fs::read_dir(&root) else { + return Ok(vec![]); + }; + let mut out = vec![]; + for entry in entries.flatten() { + let name = entry.file_name().to_string_lossy().into_owned(); + if !is_workflow_name(&name) || !entry.path().join(META_FILE).is_file() { + continue; + } + let loaded = load_dir(&name, entry.path()); + out.push((name, loaded)); + } + out.sort_by(|a, b| a.0.cmp(&b.0)); + Ok(out) +} + +/// Removes an installed workflow. The name is checked and the directory +/// must carry [`META_FILE`], so nothing but a workflow this command put +/// there is ever removed. +pub fn uninstall(name: &str) -> Result { + let dir = installed_dir(name)?.ok_or_else(|| not_installed(name))?; + std::fs::remove_dir_all(&dir).with_context(|| format!("Could not remove {}", dir.display()))?; + // What an earlier install left behind, which would otherwise come back + // as this workflow through `recover`. + let root = root_path()?; + for leftover in [previous_path(&root, name), staging_path(&root, name)] { + let _ = std::fs::remove_dir_all(leftover); + } + Ok(dir) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn archive(entries: &[(&str, &[u8])]) -> Vec { + let mut builder = tar::Builder::new(flate2::write::GzEncoder::new( + vec![], + flate2::Compression::fast(), + )); + for (path, bytes) in entries { + let mut header = tar::Header::new_gnu(); + header.set_size(bytes.len() as u64); + header.set_mode(0o644); + header.set_entry_type(tar::EntryType::Regular); + builder.append_data(&mut header, path, *bytes).unwrap(); + } + builder.into_inner().unwrap().finish().unwrap() + } + + fn scratch_dir(name: &str) -> PathBuf { + let dir = std::env::temp_dir().join(format!( + "mapbox-store-{name}-{:016x}", + rand::random::() + )); + std::fs::create_dir_all(&dir).unwrap(); + dir + } + + fn holding(dir: &Path, text: &str) -> PathBuf { + std::fs::create_dir_all(dir).unwrap(); + std::fs::write(dir.join("workflow.yaml"), text).unwrap(); + dir.to_path_buf() + } + + fn content(dir: &Path) -> String { + std::fs::read_to_string(dir.join("workflow.yaml")).unwrap() + } + + #[test] + fn a_replacement_goes_in_and_the_old_copy_goes() { + let root = scratch_dir("swap"); + let (staged, target, previous) = ( + holding(&root.join(".demo.staging"), "new"), + holding(&root.join("demo"), "old"), + root.join(".demo.previous"), + ); + swap_in(&staged, &target, &previous, |a, b| std::fs::rename(a, b)).expect("swapped"); + assert_eq!(content(&target), "new"); + assert!(!previous.exists() && !staged.exists()); + let _ = std::fs::remove_dir_all(root); + } + + /// The old copy must survive a `--force` whose last step fails. + #[test] + fn a_failed_final_move_leaves_the_old_workflow_installed() { + let root = scratch_dir("swap-fails"); + let (staged, target, previous) = ( + holding(&root.join(".demo.staging"), "new"), + holding(&root.join("demo"), "old"), + root.join(".demo.previous"), + ); + let failing = |from: &Path, to: &Path| { + if from == staged { + Err(std::io::Error::other("disk full")) + } else { + std::fs::rename(from, to) + } + }; + let err = swap_in(&staged, &target, &previous, failing).expect_err("the move fails"); + assert!(format!("{err:#}").contains("disk full"), "{err:#}"); + assert_eq!(content(&target), "old"); + assert!(!previous.exists()); + let _ = std::fs::remove_dir_all(root); + } + + /// An install that stopped between its two renames left the old copy + /// aside and none in place; it is the workflow, so it comes back. + #[test] + fn a_copy_left_aside_is_put_back() { + let root = scratch_dir("recover"); + holding(&previous_path(&root, "demo"), "old"); + recover(&root, "demo").expect("recovered"); + assert_eq!(content(&root.join("demo")), "old"); + assert!(!previous_path(&root, "demo").exists()); + + // With a copy in place, one aside is a leftover and stays put. + holding(&previous_path(&root, "demo"), "older"); + recover(&root, "demo").expect("nothing to do"); + assert_eq!(content(&root.join("demo")), "old"); + let _ = std::fs::remove_dir_all(root); + } + + /// A loopback stand-in for GitHub's API that answers each request in + /// turn, closing the connection after each so every one is its own, and + /// returns the request lines it was sent. + fn serve( + responses: Vec<(&'static str, Vec)>, + ) -> (std::thread::JoinHandle>, String) { + use std::io::Write as _; + let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("a loopback port"); + let base = format!("http://{}", listener.local_addr().unwrap()); + let server = std::thread::spawn(move || { + let mut seen = vec![]; + for (content_type, body) in responses { + let Ok((mut stream, _)) = listener.accept() else { + break; + }; + let mut head = vec![]; + let mut byte = [0u8; 1]; + while !head.ends_with(b"\r\n\r\n") { + match std::io::Read::read(&mut stream, &mut byte) { + Ok(1) => head.push(byte[0]), + _ => break, + } + } + let head = String::from_utf8_lossy(&head).into_owned(); + seen.push(head.lines().next().unwrap_or_default().to_string()); + let _ = write!( + stream, + "HTTP/1.1 200 OK\r\nContent-Type: {content_type}\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", + body.len() + ); + let _ = stream.write_all(&body); + } + seen + }); + (server, base) + } + + /// A branch moves, so the install records the commit it resolved to, + /// and downloads that commit rather than whatever the branch is by then. + #[test] + fn a_github_install_records_the_commit_it_downloaded() { + let commit = "0123456789abcdef0123456789abcdef01234567"; + let tarball = archive(&[( + "mapbox-mapbox-cli-0123456/workflow/demo/workflow.yaml", + b"schema: 1\nname: demo\nsummary: d\nsteps:\n - id: a\n command: config list\n", + )]); + let (server, base) = serve(vec![ + ( + "application/vnd.github.sha", + format!("{commit}\n").into_bytes(), + ), + ("application/x-gzip", tarball), + ]); + + let package = + fetch_github_from(&base, "demo", "mapbox/mapbox-cli", "main", false).expect("fetched"); + assert_eq!(package.meta.git_ref.as_deref(), Some("main")); + assert_eq!(package.meta.commit.as_deref(), Some(commit)); + assert_eq!( + server.join().unwrap(), + [ + "GET /repos/mapbox/mapbox-cli/commits/main HTTP/1.1".to_string(), + format!("GET /repos/mapbox/mapbox-cli/tarball/{commit} HTTP/1.1"), + ] + ); + } + + /// A PR's head or a fork's commit resolves through this repository, so + /// only GitHub's own comparison against `main` vouches for it. + #[test] + fn only_a_commit_on_main_is_on_main() { + let commit = "0123456789abcdef0123456789abcdef01234567"; + for (status, expected) in [ + ("behind", true), + ("identical", true), + ("ahead", false), + ("diverged", false), + ] { + let body = format!(r#"{{"status":"{status}"}}"#).into_bytes(); + let (server, base) = serve(vec![("application/json", body)]); + assert_eq!( + is_on_main_from(&base, "mapbox/mapbox-cli", commit, false), + expected, + "{status}" + ); + assert_eq!( + server.join().unwrap(), + [format!( + "GET /repos/mapbox/mapbox-cli/compare/main...{commit}?per_page=1&page=2 HTTP/1.1" + )] + ); + } + + // Nothing listening: not knowing is a no. + let closed = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let base = format!("http://{}", closed.local_addr().unwrap()); + drop(closed); + assert!(!is_on_main_from(&base, "mapbox/mapbox-cli", commit, false)); + // And a ref rather than a commit is never sent. + assert!(!is_on_main_from( + &base, + "mapbox/mapbox-cli", + "pull/1/head", + false + )); + } + + #[test] + fn only_a_full_commit_id_is_taken() { + assert!(is_commit_id(&"a".repeat(40))); + assert!(is_commit_id(&"0".repeat(64))); + for bad in ["0123456", "main", "../../x", &"g".repeat(40)] { + assert!(!is_commit_id(bad), "{bad}"); + } + } + + /// An install from before the commit was recorded still loads. + #[test] + fn an_older_install_record_still_reads() { + let meta: Meta = serde_json::from_str( + r#"{"source":"github:mapbox/mapbox-cli","ref":"main","installed_at":1}"#, + ) + .expect("parses"); + assert_eq!(meta.commit, None); + } + + #[test] + fn only_the_named_workflow_comes_out() { + let bytes = archive(&[ + ("mapbox-cli-abc/README.md", b"x"), + ("mapbox-cli-abc/workflow/copy-style/workflow.yaml", b"a"), + ("mapbox-cli-abc/workflow/copy-style/scripts/p.py", b"b"), + ("mapbox-cli-abc/workflow/other/workflow.yaml", b"c"), + ]); + let files = extract(&bytes, "copy-style", "mapbox/cli", MAX_BYTES).unwrap(); + assert_eq!( + files.keys().cloned().collect::>(), + [ + PathBuf::from("scripts/p.py"), + PathBuf::from("workflow.yaml") + ] + ); + } + + #[test] + fn a_missing_workflow_lists_what_is_there() { + let bytes = archive(&[("r-abc/workflow/other/workflow.yaml", b"c")]); + let err = extract(&bytes, "copy-style", "mapbox/cli", MAX_BYTES).unwrap_err(); + assert!(err.to_string().contains("It publishes: other."), "{err}"); + } + + #[test] + fn a_file_directly_under_workflow_is_not_a_workflow() { + let bytes = archive(&[("r-abc/workflow/README.md", b"c")]); + assert!(extract(&bytes, "copy-style", "mapbox/cli", MAX_BYTES).is_err()); + } + + #[test] + fn repos_and_refs_cannot_carry_url_syntax() { + assert!(check_repo("mapbox/cli").is_ok()); + for bad in [ + "mapbox", + "mapbox/cli/x", + "../x", + "a/b?c", + "a/b#c", + "/cli", + "a/..", + ] { + assert!(check_repo(bad).is_err(), "{bad}"); + } + for good in ["main", "v0.3.0", "feat/copy-style", "063415e"] { + assert!(check_ref(good).is_ok(), "{good}"); + } + for bad in ["", "a?b", "a#b", "../main", "a//b", "a b", "a%2e"] { + assert!(check_ref(bad).is_err(), "{bad}"); + } + } + + /// An archive entry written as-is. `tar::Builder` refuses a `..` or an + /// absolute path, which is exactly what a hostile archive holds. + fn raw_entry( + builder: &mut tar::Builder>>, + path: &str, + kind: tar::EntryType, + bytes: &[u8], + ) { + let mut header = tar::Header::new_gnu(); + header.as_gnu_mut().unwrap().name[..path.len()].copy_from_slice(path.as_bytes()); + header.set_size(bytes.len() as u64); + header.set_mode(0o644); + header.set_entry_type(kind); + if kind == tar::EntryType::Symlink { + header.set_link_name("/etc/passwd").unwrap(); + } + header.set_cksum(); + builder.append(&header, bytes).unwrap(); + } + + fn raw_archive(entries: &[(&str, tar::EntryType)]) -> Vec { + let mut builder = tar::Builder::new(flate2::write::GzEncoder::new( + vec![], + flate2::Compression::fast(), + )); + for (path, kind) in entries { + let bytes: &[u8] = if *kind == tar::EntryType::Regular { + b"x" + } else { + b"" + }; + raw_entry(&mut builder, path, *kind, bytes); + } + builder.into_inner().unwrap().finish().unwrap() + } + + fn code(err: &anyhow::Error) -> &str { + err.downcast_ref::() + .map_or("", |e| e.code.as_str()) + } + + #[test] + fn an_entry_that_leaves_its_directory_stops_the_read() { + use tar::EntryType::Regular; + for bad in [ + "r-abc/../workflow/copy-style/workflow.yaml", + "r-abc/workflow/copy-style/../../../x", + "/workflow/copy-style/workflow.yaml", + ] { + let bytes = raw_archive(&[ + ("r-abc/workflow/copy-style/workflow.yaml", Regular), + (bad, Regular), + ]); + let err = extract(&bytes, "copy-style", "mapbox/cli", MAX_BYTES).unwrap_err(); + assert_eq!(code(&err), "unsafe_archive", "{bad}: {err:#}"); + } + } + + #[test] + fn links_in_an_archive_are_passed_over() { + let bytes = raw_archive(&[ + ( + "r-abc/workflow/copy-style/workflow.yaml", + tar::EntryType::Regular, + ), + ( + "r-abc/workflow/copy-style/scripts/x.py", + tar::EntryType::Symlink, + ), + ( + "r-abc/workflow/copy-style/scripts/y.py", + tar::EntryType::Link, + ), + ]); + let files = extract(&bytes, "copy-style", "mapbox/cli", MAX_BYTES).unwrap(); + assert_eq!( + files.keys().cloned().collect::>(), + [PathBuf::from("workflow.yaml")] + ); + } + + #[cfg(unix)] + #[test] + fn a_symlink_in_a_local_directory_is_refused() { + let dir = holding(&scratch_dir("symlink").join("demo"), "x"); + std::fs::create_dir_all(dir.join("scripts")).unwrap(); + std::os::unix::fs::symlink("/etc/passwd", dir.join("scripts/x.py")).unwrap(); + let err = read_tree(&dir, &[], MAX_BYTES).unwrap_err(); + assert!(err.to_string().contains("not a regular file"), "{err:#}"); + } + + /// Past the cap is an error saying so, not a read cut short into + /// something that then fails as corrupt. + #[test] + fn past_the_size_cap_is_refused_as_too_large() { + assert_eq!(read_capped(&[0u8; 8][..], 8, "x").unwrap().len(), 8); + let err = read_capped(&[0u8; 9][..], 8, "x").unwrap_err(); + assert!(err.to_string().contains("larger than"), "{err:#}"); + + let bytes = archive(&[("r-abc/workflow/copy-style/workflow.yaml", &[b'x'; 4096])]); + let err = extract(&bytes, "copy-style", "mapbox/cli", 2048).unwrap_err(); + assert!(err.to_string().contains("larger than"), "{err:#}"); + + let dir = holding(&scratch_dir("large").join("demo"), &"x".repeat(4096)); + let err = read_tree(&dir, &[], 2048).unwrap_err(); + assert!(err.to_string().contains("larger than"), "{err:#}"); + assert!(read_tree(&dir, &[], MAX_BYTES).is_ok()); + } +} diff --git a/src/workflow/template.rs b/src/workflow/template.rs new file mode 100644 index 0000000..51ce29d --- /dev/null +++ b/src/workflow/template.rs @@ -0,0 +1,351 @@ +//! `${{ … }}` expressions: how one step's values reach the next. +//! +//! Deliberately a lookup language and nothing more — no operators, no +//! functions, no conditionals. An expression names a value: an input, or a +//! path into an earlier step's output. Anything that needs logic belongs in a +//! script step, which is a real language and is tested as one. +//! +//! A string that is exactly one expression takes the value's own JSON type, +//! so an object can be handed to `stdin` whole and a number stays a number. +//! An expression inside a longer string is interpolated as text. + +use std::collections::BTreeMap; + +use anyhow::{anyhow, Result}; +use serde_json::{Map, Value}; + +const OPEN: &str = "${{"; +const CLOSE: &str = "}}"; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Segment { + Key(String), + Index(usize), +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Reference { + Input(String), + Step { id: String, path: Vec }, +} + +/// What an expression can see while a workflow runs. +pub struct Context<'a> { + pub inputs: &'a Map, + /// The outputs of the steps that have finished, by id. + pub steps: &'a BTreeMap, +} + +/// Every reference in `value`, for checking before anything runs. +pub fn references(value: &Value) -> Result, String> { + let mut found = vec![]; + walk_strings(value, &mut |text| { + for piece in pieces(text)? { + if let Piece::Expression(reference) = piece { + found.push(reference); + } + } + Ok(()) + })?; + Ok(found) +} + +/// `value` with every expression replaced by what it names. +pub fn resolve(value: &Value, context: &Context) -> Result { + match value { + Value::String(text) => resolve_string(text, context), + Value::Array(items) => items + .iter() + .map(|item| resolve(item, context)) + .collect::>() + .map(Value::Array), + Value::Object(fields) => fields + .iter() + .map(|(key, item)| Ok((key.clone(), resolve(item, context)?))) + .collect::>() + .map(Value::Object), + other => Ok(other.clone()), + } +} + +/// How a resolved value is spelled where only text fits: an argv entry, an +/// interpolated string. +pub fn as_text(value: &Value) -> Option { + match value { + Value::Null => None, + Value::String(text) => Some(text.clone()), + Value::Bool(_) | Value::Number(_) => Some(value.to_string()), + Value::Array(_) | Value::Object(_) => Some(value.to_string()), + } +} + +fn resolve_string(text: &str, context: &Context) -> Result { + let pieces = pieces(text).map_err(|e| anyhow!(e))?; + if let [Piece::Expression(reference)] = pieces.as_slice() { + return lookup(reference, context); + } + let mut out = String::new(); + for piece in pieces { + match piece { + Piece::Text(literal) => out.push_str(&literal), + Piece::Expression(reference) => { + let value = lookup(&reference, context)?; + // Interpolating a null would write "null" or nothing into the + // middle of a URL or a name, and neither is what was meant. + let text = as_text(&value).ok_or_else(|| { + anyhow!( + "`{}` is empty, so it cannot be written into \"{text}\"", + display(&reference) + ) + })?; + out.push_str(&text); + } + } + } + Ok(Value::String(out)) +} + +fn lookup(reference: &Reference, context: &Context) -> Result { + match reference { + Reference::Input(name) => Ok(context.inputs.get(name).cloned().unwrap_or(Value::Null)), + Reference::Step { id, path } => { + let mut current = context + .steps + .get(id) + .ok_or_else(|| anyhow!("`{}`: step `{id}` has not run", display(reference)))?; + for segment in path { + let next = match segment { + Segment::Key(key) => current.get(key.as_str()), + Segment::Index(index) => current.get(*index), + }; + current = next.ok_or_else(|| { + anyhow!( + "`{}`: step `{id}`'s output has nothing at that path", + display(reference) + ) + })?; + } + Ok(current.clone()) + } + } +} + +/// An expression as the author wrote it, for error messages. +pub fn display(reference: &Reference) -> String { + match reference { + Reference::Input(name) => format!("inputs.{name}"), + Reference::Step { id, path } => { + let mut out = format!("steps.{id}.output"); + for segment in path { + match segment { + Segment::Key(key) => { + out.push('.'); + out.push_str(key); + } + Segment::Index(index) => out.push_str(&format!("[{index}]")), + } + } + out + } + } +} + +fn walk_strings( + value: &Value, + visit: &mut dyn FnMut(&str) -> Result<(), String>, +) -> Result<(), String> { + match value { + Value::String(text) => visit(text), + Value::Array(items) => items.iter().try_for_each(|item| walk_strings(item, visit)), + Value::Object(fields) => fields + .values() + .try_for_each(|item| walk_strings(item, visit)), + _ => Ok(()), + } +} + +#[derive(Debug, PartialEq)] +enum Piece { + Text(String), + Expression(Reference), +} + +fn pieces(text: &str) -> Result, String> { + let mut out = vec![]; + let mut rest = text; + while let Some(start) = rest.find(OPEN) { + if start > 0 { + out.push(Piece::Text(rest[..start].to_string())); + } + let after = &rest[start + OPEN.len()..]; + let end = after + .find(CLOSE) + .ok_or_else(|| format!("unclosed `{OPEN}` in \"{text}\""))?; + out.push(Piece::Expression(parse(after[..end].trim())?)); + rest = &after[end + CLOSE.len()..]; + } + if !rest.is_empty() { + out.push(Piece::Text(rest.to_string())); + } + Ok(out) +} + +/// `inputs.` or `steps..output` followed by `.key` and `[n]`. +fn parse(expression: &str) -> Result { + let invalid = || { + format!( + "`{expression}` is not an expression this understands. Use \ + `inputs.` or `steps..output`, optionally followed by \ + `.key` or `[index]`" + ) + }; + + let mut segments = vec![]; + let mut chars = expression.chars().peekable(); + let mut first = true; + while chars.peek().is_some() { + if !first { + match chars.next() { + Some('.') => {} + Some('[') => { + let digits: String = chars.by_ref().take_while(|c| *c != ']').collect(); + let index = digits.parse::().map_err(|_| invalid())?; + segments.push(Segment::Index(index)); + continue; + } + _ => return Err(invalid()), + } + } + first = false; + let mut key = String::new(); + while let Some(c) = chars.peek() { + if c.is_ascii_alphanumeric() || *c == '_' || *c == '-' { + key.push(*c); + chars.next(); + } else { + break; + } + } + if key.is_empty() { + return Err(invalid()); + } + segments.push(Segment::Key(key)); + } + + let mut segments = segments.into_iter(); + match (segments.next(), segments.next(), segments.next()) { + (Some(Segment::Key(root)), Some(Segment::Key(name)), None) if root == "inputs" => { + Ok(Reference::Input(name)) + } + (Some(Segment::Key(root)), Some(Segment::Key(id)), Some(Segment::Key(output))) + if root == "steps" && output == "output" => + { + Ok(Reference::Step { + id, + path: segments.collect(), + }) + } + _ => Err(invalid()), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + fn context_resolve(value: Value) -> Result { + let inputs = json!({ "style_id": "abc", "zoom": 12, "empty": null }); + let steps = BTreeMap::from([( + "source".to_string(), + json!({ "owner": "alice", "layers": [{ "id": "water" }] }), + )]); + resolve( + &value, + &Context { + inputs: inputs.as_object().unwrap(), + steps: &steps, + }, + ) + } + + #[test] + fn a_whole_expression_keeps_its_type() { + assert_eq!( + context_resolve(json!("${{ inputs.zoom }}")).unwrap(), + json!(12) + ); + assert_eq!( + context_resolve(json!("${{ steps.source.output.layers[0] }}")).unwrap(), + json!({ "id": "water" }) + ); + } + + #[test] + fn an_embedded_expression_is_interpolated_as_text() { + assert_eq!( + context_resolve(json!( + "styles/${{ steps.source.output.owner }}/${{inputs.style_id}}" + )) + .unwrap(), + json!("styles/alice/abc") + ); + } + + #[test] + fn objects_and_arrays_are_resolved_inside() { + assert_eq!( + context_resolve(json!({ "a": ["${{ inputs.style_id }}", 1] })).unwrap(), + json!({ "a": ["abc", 1] }) + ); + } + + #[test] + fn an_empty_value_cannot_be_interpolated() { + assert!(context_resolve(json!("x-${{ inputs.empty }}")).is_err()); + assert_eq!( + context_resolve(json!("${{ inputs.empty }}")).unwrap(), + Value::Null + ); + } + + #[test] + fn a_missing_path_names_the_expression() { + let err = context_resolve(json!("${{ steps.source.output.nope }}")).unwrap_err(); + assert!( + err.to_string().contains("steps.source.output.nope"), + "{err}" + ); + } + + #[test] + fn references_are_found_for_checking() { + assert_eq!( + references(&json!({ "a": "${{ inputs.x }}", "b": ["${{ steps.s.output.k[2] }}"] })) + .unwrap(), + vec![ + Reference::Input("x".into()), + Reference::Step { + id: "s".into(), + path: vec![Segment::Key("k".into()), Segment::Index(2)] + } + ] + ); + } + + #[test] + fn anything_else_is_refused() { + for bad in [ + "${{ inputs }}", + "${{ inputs.a.b }}", + "${{ steps.s }}", + "${{ steps.s.result }}", + "${{ env.HOME }}", + "${{ inputs.a + 1 }}", + "${{ steps.s.output[x] }}", + "${{ inputs.a", + ] { + assert!(references(&json!(bad)).is_err(), "{bad} was accepted"); + } + } +} diff --git a/src/workflow/workdir.rs b/src/workflow/workdir.rs new file mode 100644 index 0000000..2171c93 --- /dev/null +++ b/src/workflow/workdir.rs @@ -0,0 +1,79 @@ +//! The directory one workflow run keeps its files in. +//! +//! A command step's `save` writes there, and a script finds it through +//! [`ENV`]. It is created under the system's temporary directory with a +//! random name, readable by its owner only, and removed when the run ends — +//! including by Ctrl-C, SIGTERM or SIGHUP, which the runner turns into an +//! ordinary failure for this reason. A run that is killed outright, or whose +//! Windows console is closed, still leaves it behind. + +use std::path::{Path, PathBuf}; + +use anyhow::{Context as _, Result}; + +/// The run's directory, in a script step's environment. +pub const ENV: &str = "MAPBOX_WORKFLOW_WORKDIR"; + +pub struct Workdir { + path: PathBuf, +} + +impl Workdir { + pub fn create() -> Result { + let name = format!("mapbox-workflow-{:016x}", rand::random::()); + let path = std::env::temp_dir().join(name); + // Not `recursive`: a name that already exists is somebody else's + // directory, and this one is about to be removed. + builder() + .create(&path) + .with_context(|| format!("Could not create {}", path.display()))?; + Ok(Workdir { path }) + } + + pub fn path(&self) -> &Path { + &self.path + } +} + +/// Owner-only on Unix, where `/tmp` is shared and a saved style or a +/// script's output is nobody else's business. Windows' temporary directory +/// is already under the user's own profile. +fn builder() -> std::fs::DirBuilder { + #[allow(unused_mut)] + let mut builder = std::fs::DirBuilder::new(); + #[cfg(unix)] + std::os::unix::fs::DirBuilderExt::mode(&mut builder, 0o700); + builder +} + +impl Drop for Workdir { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.path); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[cfg(unix)] + #[test] + fn only_its_owner_can_read_it() { + use std::os::unix::fs::PermissionsExt as _; + let workdir = Workdir::create().expect("create"); + let mode = std::fs::metadata(workdir.path()) + .expect("stat") + .permissions() + .mode(); + assert_eq!(mode & 0o777, 0o700); + } + + #[test] + fn it_is_removed_when_dropped() { + let workdir = Workdir::create().expect("create"); + let path = workdir.path().to_path_buf(); + std::fs::write(path.join("x"), "x").expect("write"); + drop(workdir); + assert!(!path.exists()); + } +} diff --git a/tests/source_guards.rs b/tests/source_guards.rs index c91aae3..0b06146 100644 --- a/tests/source_guards.rs +++ b/tests/source_guards.rs @@ -49,6 +49,13 @@ fn sources() -> Vec<(String, String)> { /// - `generate_skills` — the staged skill directory it renames into place. /// - `skill_dest` — a test scratch directory. /// - `uninstall` — the binary itself, which is the whole command. +/// - `workflow/store.rs` — its staging directory, the installed workflow +/// `install --force` replaces, and `workflow uninstall`. The name is +/// checked to be one plain directory name first, and only a directory +/// holding its `.install.json` marker is removed. +/// - `workflow/workdir.rs` — the directory one workflow run keeps its files +/// in, which it created itself under the system temp directory with a +/// random name. No caller names it. const MAY_DELETE: &[&str] = &[ "agent_skills.rs", "auth.rs", @@ -58,6 +65,8 @@ const MAY_DELETE: &[&str] = &[ "skill_dest.rs", "telemetry_sink.rs", "uninstall.rs", + "workflow/store.rs", + "workflow/workdir.rs", ]; /// A new module that deletes files has to say so here first. @@ -240,7 +249,10 @@ const CARRIES_A_REQUEST_ID: &[&str] = &["account_usage.rs", "auth.rs", "executor /// `agent_skills.rs` talks to GitHub codeload, which identifies requests with /// `x-github-request-id`. That is not something Mapbox support can look up, /// so an id there would point at the wrong company — worse than none. -const NO_REQUEST_ID_TO_CARRY: &[&str] = &["agent_skills.rs"]; +/// +/// `workflow/store.rs` fetches a tarball from the GitHub API, for the same +/// reason. +const NO_REQUEST_ID_TO_CARRY: &[&str] = &["agent_skills.rs", "workflow/store.rs"]; /// `output/error.rs` defines `CliError::http` rather than calling it over a /// wire. diff --git a/tests/workflow.rs b/tests/workflow.rs new file mode 100644 index 0000000..03dca54 --- /dev/null +++ b/tests/workflow.rs @@ -0,0 +1,880 @@ +//! End-to-end tests for `mapbox workflow`. +//! +//! The unit tests in `src/workflow/` cover the schema, the expressions and +//! how a step's arguments become a command line. What they cannot show is +//! what a real run does: that a command step is a real child `mapbox` whose +//! JSON reaches the next step, that a script's stdin and stdout carry values +//! between steps, that only the workflow's result lands on stdout, and what +//! `install` and `uninstall` leave on disk. +//! +//! Nothing here reaches the network. The command steps run `config list` and +//! `auth whoami`, which make no request, and every install is from a local +//! directory. + +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; + +use serde_json::{json, Value}; + +/// A token-shaped fake for account `example-user`. +const TOKEN: &str = "pk.eyJ1IjoiZXhhbXBsZS11c2VyIiwiYSI6IngifQ.SIGNATURE"; + +fn scratch(name: &str) -> PathBuf { + let home = PathBuf::from(env!("CARGO_TARGET_TMPDIR")).join(format!("workflow-{name}")); + let _ = std::fs::remove_dir_all(&home); + std::fs::create_dir_all(&home).expect("create the scratch home"); + home +} + +fn run(home: &Path, args: &[&str]) -> Output { + run_with(home, args, &[]) +} + +/// [`run`], with `env` set on top of what it sets. +fn run_with(home: &Path, args: &[&str], env: &[(&str, &str)]) -> Output { + command(home, args, env).output().expect("run mapbox") +} + +fn command(home: &Path, args: &[&str], env: &[(&str, &str)]) -> Command { + let mut command = Command::new(env!("CARGO_BIN_EXE_mapbox")); + command + // Telemetry is not under test here, and would post to Mapbox. Every + // step's child inherits these. + .env("MAPBOX_CLI_NO_TELEMETRY", "1") + .env_remove("MAPBOX_CLI_TOKEN") + .env("MAPBOX_INTERNAL_CLI_TOKEN_URL", "http://127.0.0.1:9/") + .env("MAPBOX_INTERNAL_TOKENS_URL", "http://127.0.0.1:9/") + .env_remove("MAPBOX_ACCESS_TOKEN") + .env_remove("MapboxAccessToken") + .env_remove("MAPBOX_USERNAME") + .env_remove("MAPBOX_OUTPUT") + .env_remove("GH_TOKEN") + .env_remove("GITHUB_TOKEN") + .env("MAPBOX_NO_UPDATE_CHECK", "1") + .env("MAPBOX_QUIET", "1") + .env("HOME", home) + .env("XDG_CONFIG_HOME", home.join(".config")) + .env("MAPBOX_CONFIG_DIR", home.join(".mapbox")) + .envs(env.iter().copied()) + .args(args); + command +} + +fn stdout_json(out: &Output) -> Value { + serde_json::from_slice(&out.stdout).unwrap_or_else(|e| { + panic!( + "stdout is not one JSON document ({e}):\n{}\nstderr:\n{}", + String::from_utf8_lossy(&out.stdout), + String::from_utf8_lossy(&out.stderr) + ) + }) +} + +/// Writes a workflow at `/src//` and returns its path. +fn write_workflow(home: &Path, name: &str, yaml: &str, scripts: &[(&str, &str)]) -> PathBuf { + let dir = home.join("src").join(name); + std::fs::create_dir_all(dir.join("scripts")).unwrap(); + std::fs::write(dir.join("workflow.yaml"), yaml).unwrap(); + for (file, body) in scripts { + std::fs::write(dir.join("scripts").join(file), body).unwrap(); + } + dir +} + +const DEMO: &str = r#"schema: 1 +name: demo +summary: Pass values from a command to a script and back +inputs: + greeting: { type: string, default: hello } +steps: + - id: settings + command: config list + - id: whoami + command: auth whoami + - id: shout + script: shout.sh + args: ["${{ inputs.greeting }}"] + stdin: ${{ steps.settings.output[0] }} +outputs: + account: ${{ steps.whoami.output.account }} + first_key: ${{ steps.settings.output[0].key }} + shouted: ${{ steps.shout.output.shouted }} + seen: ${{ steps.shout.output.seen }} +"#; + +/// Echoes its argument upper-cased beside what it read on stdin, and says +/// something on stderr, which must not reach the result. +const SHOUT: &str = r#"read -r line +echo "shouting" >&2 +printf '{"shouted":"%s","seen":%s}\n' "$(printf %s "$1" | tr a-z A-Z)" "$line" +"#; + +fn install_demo(home: &Path) { + let dir = write_workflow(home, "demo", DEMO, &[("shout.sh", SHOUT)]); + let out = run( + home, + &["workflow", "install", dir.to_str().unwrap(), "-o", "json"], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); +} + +#[test] +fn a_run_carries_values_between_steps_and_prints_only_the_result() { + let home = scratch("run"); + install_demo(&home); + + // Not quiet, so the script's stderr is shown as a detail. + let out = run_with( + &home, + &[ + "--token", + TOKEN, + "workflow", + "run", + "demo", + "--greeting", + "hi", + "-o", + "json", + ], + &[("MAPBOX_QUIET", "0")], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + + let first_key = json!("update-check"); + assert_eq!( + stdout_json(&out), + json!({ + "account": "example-user", + "first_key": first_key, + "shouted": "HI", + "seen": { "key": first_key, "value": true }, + }) + ); + + let stderr = String::from_utf8_lossy(&out.stderr); + assert!( + stderr.contains("[1/3] settings (mapbox config list)"), + "{stderr}" + ); + assert!( + stderr.contains("[3/3] shout (sh scripts/shout.sh)"), + "{stderr}" + ); + assert!(stderr.contains("shouting"), "{stderr}"); + assert!(stderr.contains("may change or be removed"), "{stderr}"); +} + +/// A `.py` script with no `interpreter` runs under `python3` from `PATH`, +/// as `copy-style`'s do. Unix only: Windows has no `python3` by default. +#[cfg(unix)] +#[test] +fn a_python_step_runs_under_python3_by_default() { + let home = scratch("python"); + let yaml = "schema: 1\nname: py\nsummary: Runs Python\nsteps:\n\ + \x20 - id: hello\n script: hello.py\n"; + let script = "import json, sys\nprint(json.dumps({\"major\": sys.version_info[0]}))\n"; + let dir = write_workflow(&home, "py", yaml, &[("hello.py", script)]); + assert!(run(&home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); + + let out = run(&home, &["workflow", "run", "py", "-o", "json"]); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + assert_eq!(stdout_json(&out), json!({ "major": 3 })); +} + +#[test] +fn a_failing_step_stops_the_run() { + let home = scratch("fail"); + let yaml = "schema: 1\nname: broken\nsummary: Fails in the middle\nsteps:\n\ + \x20 - id: first\n script: fail.sh\n\ + \x20 - id: second\n command: config list\n"; + let dir = write_workflow(&home, "broken", yaml, &[("fail.sh", "exit 3\n")]); + assert!(run(&home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); + + let out = run(&home, &["workflow", "run", "broken", "-o", "json"]); + assert_eq!(out.status.code(), Some(1)); + assert!( + out.stdout.is_empty(), + "{}", + String::from_utf8_lossy(&out.stdout) + ); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!(stderr.contains("exited with 3"), "{stderr}"); + assert!(!stderr.contains("[2/2]"), "the second step ran: {stderr}"); +} + +/// `--repo` and `--ref` mean nothing for a local directory. Like any flag +/// combination a command refuses, that is a usage error: exit 2. +#[test] +fn github_flags_with_a_local_directory_are_a_usage_error() { + let home = scratch("usage"); + let yaml = + "schema: 1\nname: local\nsummary: Local\nsteps:\n - id: a\n command: config list\n"; + let dir = write_workflow(&home, "local", yaml, &[]); + let out = run( + &home, + &[ + "workflow", + "install", + dir.to_str().unwrap(), + "--ref", + "main", + "-o", + "json", + ], + ); + assert_eq!(out.status.code(), Some(2)); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!(stderr.contains("\"code\":\"usage\""), "{stderr}"); +} + +/// A failed command step keeps the child's own error: its code under +/// `cause` for a caller to branch on, its advice lifted onto the run's +/// error, and in text mode its message rather than a raw JSON document. +#[test] +fn a_failed_command_step_carries_its_own_error() { + let home = scratch("step-error"); + // No username anywhere, so the child fails before any request. + let yaml = "schema: 1\nname: listing\nsummary: Lists styles\nsteps:\n\ + \x20 - id: list\n command: styles list\n"; + let dir = write_workflow(&home, "listing", yaml, &[]); + assert!(run(&home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); + + let out = run(&home, &["workflow", "run", "listing", "-o", "json"]); + assert_eq!(out.status.code(), Some(1)); + let stderr = String::from_utf8_lossy(&out.stderr); + let documents: Vec<&str> = stderr.lines().filter(|l| l.starts_with('{')).collect(); + assert_eq!(documents.len(), 1, "one error document only: {stderr}"); + let error: Value = serde_json::from_str(documents[0]).unwrap(); + assert_eq!(error["code"], "workflow_failed"); + assert_eq!(error["cause"]["code"], "missing_path_parameters"); + assert_eq!(error["next_actions"], json!(["mapbox auth whoami"])); + + let out = run(&home, &["workflow", "run", "listing", "-o", "text"]); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!( + stderr.contains("Error: Step `list` (mapbox styles list): Missing required path"), + "{stderr}" + ); + assert!(!stderr.contains("{\"code\""), "{stderr}"); +} + +/// Ctrl-C at a terminal reaches the whole foreground process group: the +/// step's child and `mapbox` alike. The run must still remove its working +/// directory, which a process that simply died would leave behind. +#[cfg(unix)] +#[test] +fn ctrl_c_stops_the_run_and_removes_its_directory() { + use std::os::unix::process::CommandExt as _; + use std::time::{Duration, Instant}; + + let home = scratch("interrupt"); + let marker = home.join("workdir"); + let yaml = "schema: 1\nname: slow\nsummary: Waits to be interrupted\n\ + inputs:\n marker: { type: string, required: true }\nsteps:\n\ + \x20 - id: wait\n script: wait.sh\n args: ['${{ inputs.marker }}']\n\ + \x20 - id: after\n command: config list\n"; + let script = + "printf %s \"$MAPBOX_WORKFLOW_WORKDIR\" > \"$1.tmp\"\nmv \"$1.tmp\" \"$1\"\nsleep 30\n"; + let dir = write_workflow(&home, "slow", yaml, &[("wait.sh", script)]); + assert!(run(&home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); + + let marker_arg = marker.display().to_string(); + let child = command( + &home, + &[ + "workflow", + "run", + "slow", + "--marker", + &marker_arg, + "-o", + "json", + ], + &[], + ) + .process_group(0) + .stdout(std::process::Stdio::piped()) + .stderr(std::process::Stdio::piped()) + .spawn() + .expect("start mapbox"); + + let started = Instant::now(); + while !marker.exists() { + assert!( + started.elapsed() < Duration::from_secs(20), + "the step never started" + ); + std::thread::sleep(Duration::from_millis(50)); + } + let workdir = PathBuf::from(std::fs::read_to_string(&marker).unwrap()); + assert!(workdir.is_dir(), "{}", workdir.display()); + + let group = format!("-{}", child.id()); + assert!(Command::new("kill") + .args(["-INT", "--", &group]) + .status() + .expect("run kill") + .success()); + let out = child.wait_with_output().expect("wait for mapbox"); + + assert!( + started.elapsed() < Duration::from_secs(20), + "the run outlived the step" + ); + assert_eq!(out.status.code(), Some(1)); + assert!(!workdir.exists(), "{} was left behind", workdir.display()); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!(stderr.contains("cancelled"), "{stderr}"); + assert!( + stderr.contains("Interrupted during step `wait`"), + "{stderr}" + ); + assert!(!stderr.contains("[2/2]"), "the next step ran: {stderr}"); +} + +/// A SIGTERM, as a CI timeout sends, reaches `mapbox` alone. It is passed on +/// to the running step, which would otherwise keep the run alive until it +/// finished by itself. +#[cfg(unix)] +#[test] +fn sigterm_is_passed_on_to_the_running_step() { + use std::os::unix::process::CommandExt as _; + use std::time::{Duration, Instant}; + + let home = scratch("sigterm"); + let marker = home.join("started"); + let yaml = "schema: 1\nname: hang\nsummary: Hangs until stopped\n\ + inputs:\n marker: { type: string, required: true }\nsteps:\n\ + \x20 - id: hang\n script: hang.sh\n args: ['${{ inputs.marker }}']\n"; + // `exec`, so the signal reaches `sleep` itself rather than a shell + // waiting on it. + let script = "touch \"$1\"\nexec sleep 60\n"; + let dir = write_workflow(&home, "hang", yaml, &[("hang.sh", script)]); + assert!(run(&home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); + + let marker_arg = marker.display().to_string(); + let child = command( + &home, + &[ + "workflow", + "run", + "hang", + "--marker", + &marker_arg, + "-o", + "json", + ], + &[], + ) + .process_group(0) + .stdout(std::process::Stdio::piped()) + .stderr(std::process::Stdio::piped()) + .spawn() + .expect("start mapbox"); + + let started = Instant::now(); + while !marker.exists() { + assert!( + started.elapsed() < Duration::from_secs(20), + "the step never started" + ); + std::thread::sleep(Duration::from_millis(50)); + } + assert!(Command::new("kill") + .args(["-TERM", &child.id().to_string()]) + .status() + .expect("run kill") + .success()); + let out = child.wait_with_output().expect("wait for mapbox"); + + assert!( + started.elapsed() < Duration::from_secs(30), + "the step ran on after SIGTERM" + ); + assert_eq!(out.status.code(), Some(1)); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!( + stderr.contains("Interrupted during step `hang`"), + "{stderr}" + ); +} + +#[test] +fn dry_run_runs_nothing() { + let home = scratch("dry-run"); + let marker = home.join("ran"); + let yaml = "schema: 1\nname: touch\nsummary: Leaves a file behind\n\ + inputs:\n path: { type: string, required: true }\nsteps:\n\ + \x20 - id: touch\n script: touch.sh\n args: ['${{ inputs.path }}']\n"; + let dir = write_workflow(&home, "touch", yaml, &[("touch.sh", "touch \"$1\"\n")]); + assert!(run(&home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); + + let path = marker.display().to_string(); + let out = run( + &home, + &[ + "workflow", + "run", + "touch", + "--path", + &path, + "--dry-run", + "-o", + "json", + ], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + assert_eq!(stdout_json(&out)["steps"][0]["run"], "sh scripts/touch.sh"); + assert!(!marker.exists(), "--dry-run ran the step"); + + let out = run(&home, &["workflow", "run", "touch", "--path", &path]); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + assert!(marker.exists(), "the real run did not run the step"); +} + +/// Steps marked `dry_run` run under `--dry-run`, told so through the +/// environment, and their output is in the plan; the rest do not run. +#[test] +fn a_dry_run_runs_only_the_steps_that_support_it() { + let home = scratch("dry-run-steps"); + let marker = home.join("wrote"); + let yaml = "schema: 1\nname: rehearse\nsummary: Plans, writes, reports\n\ + inputs:\n path: { type: string, required: true }\nsteps:\n\ + \x20 - id: plan\n script: env.sh\n dry_run: true\n\ + \x20 - id: write\n script: touch.sh\n args: ['${{ inputs.path }}']\n\ + \x20 - id: report\n script: echo.sh\n dry_run: true\n\ + \x20 stdin: ${{ steps.plan.output }}\n"; + let dir = write_workflow( + &home, + "rehearse", + yaml, + &[ + ( + "env.sh", + "printf '{\"dry_run\":\"%s\"}\\n' \"$MAPBOX_WORKFLOW_DRY_RUN\"\n", + ), + ("touch.sh", "touch \"$1\"\n"), + ("echo.sh", "cat\n"), + ], + ); + assert!(run(&home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); + + let path = marker.display().to_string(); + let out = run( + &home, + &[ + "workflow", + "run", + "rehearse", + "--path", + &path, + "--dry-run", + "-o", + "json", + ], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + let plan = stdout_json(&out); + assert_eq!(plan["results"]["plan"], json!({ "dry_run": "1" })); + assert_eq!(plan["results"]["report"], json!({ "dry_run": "1" })); + assert!(plan["results"].get("write").is_none(), "{plan}"); + assert_eq!(plan["steps"][1]["dry_run"], false); + assert!( + !marker.exists(), + "--dry-run ran a step that does not support it" + ); + + let out = run( + &home, + &["workflow", "run", "rehearse", "--path", &path, "-o", "json"], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + assert_eq!(stdout_json(&out), json!({ "dry_run": "" })); + assert!(marker.exists(), "the real run did not run every step"); +} + +const REPORTING: &str = "schema: 1\nname: report\nsummary: Reports as it goes\n\ + inputs:\n who: { type: string, default: world }\nsteps:\n\ + \x20 - id: work\n script: work.sh\n\ + outputs:\n said: ${{ steps.work.output.said }}\n\ + result: 'Said ${{ steps.work.output.said }} to ${{ inputs.who }}.'\n"; + +const WORK: &str = "echo '::progress halfway' >&2\n\ + echo 'Did the thing' >&2\n\ + echo '::warn Something to know' >&2\n\ + echo '{\"said\":\"hello\"}'\n"; + +fn install_reporting(home: &Path) { + let dir = write_workflow(home, "report", REPORTING, &[("work.sh", WORK)]); + assert!(run(home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); +} + +/// Away from a terminal, a script's details and warnings are kept and its +/// progress lines dropped, with no escape codes around any of them. +#[test] +fn a_script_reports_details_and_warnings_and_drops_progress_off_a_terminal() { + let home = scratch("step-reports"); + install_reporting(&home); + + let out = run_with( + &home, + &["workflow", "run", "report", "-o", "json"], + &[("MAPBOX_QUIET", "0")], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!(stderr.contains("[1/1] work"), "{stderr}"); + assert!( + stderr.lines().any(|line| line == "Did the thing"), + "{stderr}" + ); + assert!( + stderr + .lines() + .any(|line| line == "warning: Something to know"), + "{stderr}" + ); + assert!( + stderr.contains("Warning\n ! Something to know"), + "a warning is listed again once the run ends: {stderr}" + ); + assert!(!stderr.contains("halfway"), "{stderr}"); + assert!(!stderr.contains('\x1b'), "{stderr}"); +} + +/// `--quiet` drops a step's details, but never its warnings. +#[test] +fn quiet_drops_details_but_keeps_warnings() { + let home = scratch("step-reports-quiet"); + install_reporting(&home); + + let out = run(&home, &["workflow", "run", "report", "-o", "json"]); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!(!stderr.contains("Did the thing"), "{stderr}"); + assert!(stderr.contains("Something to know"), "{stderr}"); +} + +/// `result` is how text mode shows the outcome; `-o json` still prints the +/// outputs, which is what a script reads. +#[test] +fn result_is_the_text_and_outputs_are_the_json() { + let home = scratch("result-text"); + install_reporting(&home); + + let text = run( + &home, + &[ + "workflow", "run", "report", "--who", "Helsinki", "-o", "text", + ], + ); + assert!( + text.status.success(), + "{}", + String::from_utf8_lossy(&text.stderr) + ); + assert_eq!( + String::from_utf8_lossy(&text.stdout).trim_end(), + "Said hello to Helsinki." + ); + + let json = run(&home, &["workflow", "run", "report", "-o", "json"]); + assert_eq!(stdout_json(&json), json!({ "said": "hello" })); +} + +/// `save` keeps a command step's stdout as a file in the run's working +/// directory, which a later script reads, and which is gone once the run ends. +#[test] +fn save_keeps_a_command_output_as_a_file_for_the_run() { + let home = scratch("save"); + // The path reaches the result through `outputs`, not through the script: + // on Windows it is full of backslashes, which a script writing its own + // JSON would leave unescaped. + let yaml = "schema: 1\nname: keep\nsummary: Saves then reads\nsteps:\n\ + \x20 - id: settings\n command: config list\n save: settings.json\n\ + \x20 - id: read\n script: read.sh\n stdin: ${{ steps.settings.output.path }}\n\ + outputs:\n path: ${{ steps.settings.output.path }}\n saved: ${{ steps.read.output }}\n"; + let script = "read -r path\n\ + [ \"$(dirname \"$path\")\" = \"$MAPBOX_WORKFLOW_WORKDIR\" ] || exit 7\n\ + cat \"$path\"\n"; + let dir = write_workflow(&home, "keep", yaml, &[("read.sh", script)]); + assert!(run(&home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); + + let out = run(&home, &["workflow", "run", "keep", "-o", "json"]); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + let result = stdout_json(&out); + assert!( + result["saved"].is_array(), + "the saved file is the command's JSON: {result}" + ); + let path = PathBuf::from(result["path"].as_str().unwrap()); + assert!( + !path.parent().unwrap().exists(), + "the run's working directory outlived the run: {}", + path.display() + ); +} + +#[test] +fn install_refuses_an_invalid_workflow_and_writes_nothing() { + let home = scratch("invalid"); + let yaml = "schema: 1\nname: bad\nsummary: Names a command that is not one\nsteps:\n\ + \x20 - id: a\n command: styles nope\n"; + let dir = write_workflow(&home, "bad", yaml, &[("unused.sh", "")]); + + let out = run( + &home, + &["workflow", "install", dir.to_str().unwrap(), "-o", "json"], + ); + assert_eq!(out.status.code(), Some(1)); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!(stderr.contains("invalid_workflow"), "{stderr}"); + assert!(stderr.contains("scripts/unused.sh"), "{stderr}"); + assert!(!home.join(".mapbox/workflows/bad").exists()); +} + +#[test] +fn install_replaces_only_with_force_and_uninstall_removes() { + let home = scratch("lifecycle"); + install_demo(&home); + let installed = home.join(".mapbox/workflows/demo"); + assert!(installed.join("workflow.yaml").is_file()); + assert!(installed.join(".install.json").is_file()); + + let source = home.join("src/demo"); + let again = run(&home, &["workflow", "install", source.to_str().unwrap()]); + assert_eq!(again.status.code(), Some(1)); + assert!(String::from_utf8_lossy(&again.stderr).contains("already installed")); + + let forced = run( + &home, + &["workflow", "install", source.to_str().unwrap(), "--force"], + ); + assert!( + forced.status.success(), + "{}", + String::from_utf8_lossy(&forced.stderr) + ); + + let listed = stdout_json(&run(&home, &["workflow", "list", "-o", "json"])); + assert_eq!(listed[0]["name"], "demo"); + + let removed = run(&home, &["workflow", "uninstall", "demo", "-o", "json"]); + assert!( + removed.status.success(), + "{}", + String::from_utf8_lossy(&removed.stderr) + ); + assert!(!installed.exists()); + + let missing = run(&home, &["workflow", "run", "demo"]); + assert_eq!(missing.status.code(), Some(1)); + assert!(String::from_utf8_lossy(&missing.stderr).contains("mapbox workflow install demo")); +} + +#[test] +fn uninstall_removes_nothing_outside_the_installed_workflows() { + let home = scratch("escape"); + std::fs::create_dir_all(home.join("keep")).unwrap(); + for name in ["../keep", "/tmp", "..", "../.."] { + let out = run(&home, &["workflow", "uninstall", name]); + assert_eq!(out.status.code(), Some(1), "{name}"); + } + assert!(home.join("keep").exists()); +} + +#[test] +fn uninstall_takes_the_directory_install_was_given() { + let home = scratch("uninstall-path"); + install_demo(&home); + let source = home.join("src/demo"); + + // Read from the JSON error rather than matched in its text: JSON escapes a + // Windows path's backslashes, so the raw text never holds the path as typed. + let again = run( + &home, + &[ + "workflow", + "install", + source.to_str().unwrap(), + "-o", + "json", + ], + ); + let stderr = String::from_utf8_lossy(&again.stderr); + let error: Value = stderr + .lines() + .rev() + .find_map(|line| serde_json::from_str(line).ok()) + .unwrap_or_else(|| panic!("no JSON error on stderr:\n{stderr}")); + assert_eq!( + error["next_actions"][0], + json!(format!( + "mapbox workflow install {} --force", + source.display() + )), + "{stderr}" + ); + + let out = run(&home, &["workflow", "uninstall", source.to_str().unwrap()]); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + assert!(!home.join(".mapbox/workflows/demo").exists()); + assert!( + source.join("workflow.yaml").is_file(), + "the source directory was touched" + ); +} + +#[test] +fn inputs_are_flags_like_any_other_command() { + let home = scratch("flags"); + install_demo(&home); + + let help = run(&home, &["workflow", "run", "demo", "--help"]); + let text = String::from_utf8_lossy(&help.stdout); + assert!( + help.status.success(), + "{}", + String::from_utf8_lossy(&help.stderr) + ); + assert!(text.contains("--greeting "), "{text}"); + assert!(text.contains("[default: hello]"), "{text}"); + + // Clap's own usage error, exit 2, as for a typo on any command. + let typo = run(&home, &["workflow", "run", "demo", "--greting", "hi"]); + assert_eq!(typo.status.code(), Some(2)); + assert!(String::from_utf8_lossy(&typo.stderr).contains("--greeting")); + + // Globals still work after the workflow's name. + let out = run( + &home, + &[ + "workflow", + "run", + "demo", + "--greeting=yo", + "--token", + TOKEN, + "-o", + "json", + ], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + assert_eq!(stdout_json(&out)["shouted"], "YO"); +} + +#[test] +fn a_missing_required_input_names_its_flag() { + let home = scratch("required"); + let yaml = "schema: 1\nname: needs\nsummary: Needs an input\n\ + inputs:\n style_id: { type: string, required: true }\nsteps:\n\ + \x20 - id: a\n command: config list\n"; + let dir = write_workflow(&home, "needs", yaml, &[]); + std::fs::remove_dir(dir.join("scripts")).unwrap(); + assert!(run(&home, &["workflow", "install", dir.to_str().unwrap()]) + .status + .success()); + + let out = run(&home, &["workflow", "run", "needs"]); + assert_eq!(out.status.code(), Some(2)); + assert!(String::from_utf8_lossy(&out.stderr).contains("--style-id ")); + + // The underscore spelling is accepted as well. + let out = run( + &home, + &["workflow", "run", "needs", "--style_id", "x", "-o", "json"], + ); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); +} + +#[test] +fn a_workflow_that_is_not_installed_says_how_to_install_it() { + let home = scratch("not-installed"); + let out = run(&home, &["workflow", "run", "copy-style", "--style-id", "x"]); + assert_eq!(out.status.code(), Some(1)); + assert!(String::from_utf8_lossy(&out.stderr).contains("mapbox workflow install copy-style")); +} + +#[test] +fn history_records_the_command_and_not_the_workflow_name() { + let home = scratch("history"); + install_demo(&home); + assert!(run(&home, &["workflow", "run", "demo", "--token", TOKEN]) + .status + .success()); + + let listed = stdout_json(&run(&home, &["history", "list", "-o", "json"])); + assert_eq!(listed[0]["command"], json!(["workflow", "run"]), "{listed}"); + assert!(!listed.to_string().contains("demo"), "{listed}"); +} diff --git a/workflow/README.md b/workflow/README.md new file mode 100644 index 0000000..f5a07e4 --- /dev/null +++ b/workflow/README.md @@ -0,0 +1,177 @@ +# Workflows + +A workflow is a named, multi-step recipe of `mapbox` commands and scripts. None ships inside the binary: `mapbox workflow install` copies one in, and `mapbox workflow run` runs only what is installed. See [docs/commands.md](../docs/commands.md#workflows) for the commands. + +> **Beta.** The `workflow` command, the format below (`schema: 1`) and the published workflows may change or be removed in any release. Every `mapbox workflow` subcommand says so on stderr. + +**Only install a workflow you trust.** Its steps run as you, with your credentials and every stored profile; a script step runs whatever its interpreter does, in a `--dry-run` too when it is marked `dry_run`. `install` lists a workflow's scripts, and asks before installing one from a repository other than this one, or from a `--ref` that is not on this repository's `main`, such as a pull request. Only a typed `--yes` answers that question; `MAPBOX_YES` does not. + +## Layout + +``` +workflow/ + README.md # this file + copy-style/ # the workflow's name + workflow.yaml # required + scripts/ # the scripts its steps run + copy_style.py + README.md # optional +``` + +- The directory name is the workflow's name: lower-case letters, digits and dashes, and the same as `name` in `workflow.yaml`. +- A workflow holds `workflow.yaml`, `README.md` and `scripts/`, and nothing else. +- Every file in `scripts/` is run by some step. A workflow ships only the scripts it runs. + +The same rules apply to any workflow `install` reads, from this repository, another one (`--repo OWNER/REPO`, laid out the same way) or a local directory. `every_published_workflow_is_valid` in `src/workflow/mod.rs` holds this directory to them in `cargo test`. + +## `workflow.yaml` + +```yaml +schema: 1 # required; the format's version +name: copy-style # required; the directory's name +version: 0.1.0 # optional; the workflow's own version +requires: ">=0.4.0" # optional; the oldest mapbox that can run it +summary: Copy a style from one account to another # required; one line +description: | # optional; shown by `workflow show` + Longer text. + +inputs: # optional; each one is a flag of `run` + style_id: + type: string # string | number | boolean + required: true + description: ID of the style to copy + name: + type: string # not required and no default: null + zoom: + type: number + default: 12 + +steps: # required; run in order, first failure stops + - id: source # required; lower-case, digits, underscores + name: Read the style # optional; the progress line + command: styles get # a mapbox command + args: # names as `mapbox --schema` lists them + style-id: ${{ inputs.style_id }} + profile: work # global options too + use-login: true # a flag takes true or false + dry_run: true # optional; also runs under --dry-run (a command that changes nothing) + + - id: bundle + command: styles download + args: { style-id: "${{ inputs.style_id }}", profile: work, use-login: true } + save: style.zip # optional; keep stdout as a file, for a binary response + + - id: body + script: prepare.py # a file in scripts/ + interpreter: python3 # optional for .sh, .py and .js + dry_run: true # optional; runs under --dry-run too, told so in its environment + args: ["--zoom", "${{ inputs.zoom }}"] + stdin: # optional; sent to the step as JSON + style: ${{ steps.source.output }} + +outputs: # optional; the default is the last step's output + id: ${{ steps.source.output.id }} + +result: | # optional; what text mode prints instead of the outputs + Created ${{ steps.source.output.id }}. +``` + +A step is a `command` or a `script`, never both. + +### Versions + +A workflow's `version` is its own, not the CLI's: workflows are released on their own schedule. It is optional, and a workflow without one installs and runs the same. `list`, `show` and `install` report it. A release is tagged `workflow--`, such as `workflow-copy-style-0.1.0`, and installed with `--ref` set to that tag. + +`requires` names the oldest CLI that can run the workflow, as `">=X.Y.Z"`, quoted, since YAML reads an unquoted `>` as the start of a block. An older CLI refuses it at `install`, and `show` and `run` say why. Set it when a step, or a script through `$MAPBOX_CLI`, needs a command or flag that older releases lack: install checks each command step against the CLI's commands, but cannot see what a script calls. + +Before `schema` existed, the format was named by `version: 1`. That spelling still reads, as a workflow with no version. + +### Inputs as flags + +`mapbox workflow run ` takes each input as a flag, spelled with dashes: `style_id` is `--style-id`, and `--style_id` works too. A required input is a required flag, a number must parse as one, and a boolean is a flag that means true on its own (`--overwrite` or `--overwrite false`). `mapbox workflow run --help` lists them. + +An input cannot be named after an option every command already has, such as `profile`, `output` or `dry_run`. `install` refuses a workflow that does. + +### `description` + +`workflow show` renders it in a fixed layout, after the name and summary and before the inputs and steps. It reads a small subset of Markdown, so every workflow's page looks the same: + +- A blank line separates paragraphs. A paragraph is reflowed to 80 columns, so line breaks inside one do not matter. +- A line indented by two or more spaces is a command. It is highlighted and kept exactly as written. +- A line starting with `- ` is a list item. An indented line under an item continues the item. +- `code` spans are highlighted. + +Anything else is shown as a plain paragraph. There is no other formatting: no headings, links or emphasis. + +### Expressions + +`${{ inputs. }}` and `${{ steps..output }}`, followed by any number of `.key` and `[index]`. Nothing else: no operators and no functions. Logic belongs in a script. + +- A value that is exactly one expression keeps its JSON type, so a whole object can go to `stdin` and a number stays a number. +- An expression inside a longer string is written in as text. One that is null there is an error rather than an empty string. +- An expression may name only a declared input or an earlier step. That is checked before any step runs. + +### Command steps + +Each command step runs this `mapbox` binary again, with `--output json`, and its JSON result becomes the step's output. It resolves its token, timeouts and path encoding exactly as the same command typed by hand would, and appears in `mapbox history` on its own. + +- `args` keys are the argument names `mapbox --schema ` lists: a flag's long name, or a positional's name. Global options such as `profile`, `username` and `use-login` are accepted too. +- `output`, `quiet`, `schema` and `dry-run` belong to the runner. `token` is refused, because a token written into a workflow is a secret in a file: log in under a profile and name the `profile` instead. +- Global options given to `mapbox workflow run` (`--profile`, `--username`, `--use-login`, `--timeout`, `--yes`, `--debug`, `--token`) reach every command step that does not set its own. +- `stdin` is sent to the command, for `--data @-`. Without it, the command reads the terminal, which is where a confirmation prompt gets its answer. +- A step cannot run `mapbox workflow`. +- `save: ` keeps the command's stdout as that file in the run's working directory instead of reading it as the output, which is how a binary response, such as `styles download`'s ZIP, reaches a later step. The step's output is then `{"path": …, "bytes": …}`. The directory is readable by its owner only, and is removed when the run ends, including by Ctrl-C and, on Unix, SIGTERM or SIGHUP. A run killed outright, or a closed console window on Windows, leaves it behind. + +### Script steps + +A script runs from the installed copy of `scripts/`, in the directory `mapbox workflow run` was started from. + +- It gets `args` as its arguments and `stdin` on standard input, as JSON unless the value is a string. With no `stdin` it reads nothing. +- Whatever it writes to stdout is its output: JSON when it parses as JSON, the text otherwise. It reports to the person running it on stderr, as below. +- A non-zero exit stops the workflow. +- `MAPBOX_CLI` is the path to this `mapbox` binary, for a script that runs commands of its own. `MAPBOX_WORKFLOW_ROOT` is the installed workflow's directory. `MAPBOX_WORKFLOW_WORKDIR` is the run's working directory, where a `save`d file is; anything a script writes there is removed with it when the run ends. +- Prefer a command step where one will do, so `workflow show` and `install` can see the command. A script is for what a step cannot do: a loop, or logic. +- The interpreter must be installed on the machine. `.sh` runs under `sh`, `.py` under `python3` and `.js` under `node`, and any other extension needs `interpreter`. Windows has none of these by default, and the python.org installer adds `python` and `py` but not `python3`, so a `.py` step needs a `python3` on `PATH` there. + +### Reporting progress + +The runner shows every step the same way, so a script only says what it is doing and what it did: + +| A stderr line | Is | Shown | +| --- | --- | --- | +| `::progress uploading icons 150/561` | What the step is doing now | At a terminal, beside the spinner, until the next one replaces it. Dropped anywhere else. | +| `::warn Source uses a tileset that is not copied` | Something the person should act on | Under the step (`! …` at a terminal, `warning: …` elsewhere), and listed again after the last step. Also under `--quiet`. | +| anything else, such as `Uploaded font Yellow Banana Regular` | A detail: what the step did | Under the step. Not under `--quiet`. | + +At a terminal a step looks like this, and the last line becomes `✓