From 268e0edd10a79621c5831f66e543abd38270a17d Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Thu, 15 Oct 2026 16:56:55 -0400 Subject: [PATCH 1/2] Restore withdrawn hosted pins upstream during scan A hosted pin points the lockfile at the patch server's copy of one patch uuid. Once that patch is unpublished the server answers 410 for it, so every fresh install of the pinned lockfile fails. scan never noticed: a withdrawn patch drops out of the search results, nothing selects it, and its pin was left in place until someone ran `rollback` by hand. `scan --mode hosted` now checks every uuid the lockfiles pin, for a package this run covered, against the reference endpoint before the rewrite. A pin reported `withdrawn` or `not_found` is restored to its upstream registry entry with the same restore `rollback` runs, under the apply lock (`--dry-run` previews it). When a replacement patch is on offer, the rewrite that follows pins it, so the net effect is an update. Only those two explicit statuses unpin. A pin merely missing from search (a failed batch, the free-only proxy fallback, a lapsed entitlement) or reported forbidden / pending_build / build_failed is kept. A failed reference lookup keeps every pin with a `redirect_withdrawn_check_failed` warning, and a run whose every search batch failed exits before the check, so an outage on the patch API never unpins anything. Co-Authored-By: Claude Opus 5.5 (1M context) --- crates/socket-patch-cli/CLI_CONTRACT.md | 7 +- .../socket-patch-cli/src/commands/scan/mod.rs | 15 + .../src/commands/scan/withdrawn.rs | 274 +++++++++ .../tests/in_process_scan_withdrawn.rs | 527 ++++++++++++++++++ 4 files changed, 822 insertions(+), 1 deletion(-) create mode 100644 crates/socket-patch-cli/src/commands/scan/withdrawn.rs create mode 100644 crates/socket-patch-cli/tests/in_process_scan_withdrawn.rs diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index 70f83ee43..705cc0aa1 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -175,6 +175,8 @@ For a **9.0 root lock**, the CLI ensures `pnpm-workspace.yaml` carries `trustLoc **Takeover reconciliation (every hosted ecosystem, v5.0)**: vendoring over a hosted pin (`vendor`, `scan --mode vendored`, `get --mode vendored`) first RESTORES that purl's lock entries to their default upstream registry entry — the same restore `rollback` runs (core `patch::redirect::upstream::restore_upstream`; see "Hosted unwind coverage"), over the hosted pins lockfile discovery finds (v5 keeps no hosted ledger) — and then vendors, so the vendor ledger records the PRISTINE registry entry as its wiring `original` and `vendor --revert` lands back on upstream registry state, never on hosted. The run that takes over records a `vendor_takeover_reverted_redirect` advisory event (`skipped` action beside the purl's genuine outcome; detail ` was hosted; restored its upstream registry entry () before vendoring (mode takeover)`; the human path prints `Warning: …`), plus any advisory the restore raised (`npm_allow_remote_left`, …). **A takeover the vendored backend does not carry through keeps the hosted pin (#853, #944)**: the wet run holds the restore in its group commit, and when the backend then refuses the purl — whatever the code: a pnpm `catalog:` dependency (`vendor_lock_entry_unsupported`), a CRLF `pnpm-lock.yaml` (`vendor_lockfile_crlf_unsupported`), a conflicting `pnpm-workspace.yaml` override — a range, another version or a `>` selector (`vendor_override_conflict`), a uv inline `[tool.uv] sources` table, a prebuilt download that fails, … — or its apply fails with nothing recorded, the restore is rolled back before anything reaches disk: the purl is reported `failed ` with the backend's own code and detail, neither `vendor_takeover_reverted_redirect` nor the restore's advisories are recorded for it, and the hosted wiring stays byte-for-byte (exit 1 / `partialFailure`), so the package stays hosted-patched instead of being un-hosted and then refused. `vendor --dry-run` previews that same `failed ` by running the backend's dry run over the restored project staged in memory (nothing written). A restore that writes a file outside the group commit's captured set (`.socket/gradle/hosted-index.tsv`) is not rolled back. The `scan` / `get --mode vendored --dry-run` preview, which stages no restore (it never touches the registry), predicts the pnpm (lockfileVersion 9) part of this (#853): a hosted pin the pnpm vendored backend refuses on lock or manifest text (a `catalog:` dependency, a CRLF `pnpm-lock.yaml` / `pnpm-workspace.yaml`, a conflicting override, …) is listed `would_refuse` with the wet run's `errorCode` and the backend's own `error` detail, evaluated on the still-hosted project (the pnpm restore only rewrites the entry's `resolution:`, which none of those gates reads); a hosted pin of any other lock flavor or ecosystem — a legacy pnpm 7/8 lock (`lockfileVersion` 5.4 / 6.0) included, whose CRLF, override-conflict and entry refusals are not lock-text gated — still previews `would_vendor` even when the wet takeover refuses it (the wet refusal keeps the hosted pin), except for the gem preflight below. **Gem preflight before the takeover**: `scan` / `get --mode vendored` ask the gem vendored backend's own refusals BEFORE the upstream restore — the manifest gate (`gemfile_not_loaded`: a `gems.rb` twin or a `BUNDLE_GEMFILE`-configured manifest) and the Gemfile declaration gate evaluated on the Gemfile and Gemfile.lock text the restore would leave (`gemfile_declaration_not_editable`: a declaration inside a `group` / `platforms` / conditional block, a parenthesized or duplicate declaration, …) — so a hosted gem vendored mode cannot wire is reported `failed ` with the hosted `Gemfile` / `Gemfile.lock` byte-untouched (exit 1 / `partialFailure`), and their `--dry-run` preview reports that gem as `would_refuse` with the same `errorCode` (exit 0, like the Bun / vlt `would_refuse` rows), never `would_vendor`. Pinned against real Bundler by `tests/e2e_redirect_gem_build.rs`. `--dry-run` resolves the same restore without writing (registry lookups included): a pin that would restore reports `vendor_would_revert_redirect`, and one that would be refused surfaces in the preview with the wet run's `redirect_revert_failed` code and detail (for bun, whose hosted rewrite replaces the entry's `name@version` spec, the preview first runs the Bun vendored preflight described below and then stops at the advisory instead of reading the still-hosted lock — a lock the vendored backend would refuse is previewed as the wet run's `failed `, never as `vendor_would_revert_redirect`). A purl whose upstream entry cannot be restored — `--offline`, a registry that does not answer, a lock the restore refuses (see "Hosted unwind coverage"; a hosted binary `bun.lockb` pin IS restored for the takeover — its npm registry record is rebuilt natively — while `rollback` / `remove` refuse it) — fails `redirect_revert_failed` with the detail `cannot vendor over the live hosted pin: cannot restore to its upstream registry entry: ; restore it from version control instead (`git checkout -- `)` (for `bun.lock` / `bun.lockb` the detail also adds `, then run \`bun install --force\` (a plain \`bun install\` keeps the patched copy)`) (exit 1 / `partialFailure`, nothing vendored for it, the hosted wiring left in place). The cargo backend's `hosted_redirect_live` refusal backstops a crate whose hosted residue is still in place when it is reached; its detail names `socket-patch rollback` and `git checkout -- Cargo.toml Cargo.lock`. **Bun vendored preflight before the takeover**: `vendor` — like `scan` / `get --mode vendored`, whose pre-download preflight runs earlier — checks `bun.lock` / `bun.lockb` with the shared Bun vendored preflight BEFORE the upstream restore, so a hosted purl on a lock the vendored backend refuses (a pre-version-2 `workspace:` lock → `vendor_bun_workspace_unsupported`; a malformed or unsupported binary lock → `vendor_bun_lockb_invalid`; an unsupported text-lock version → its code) is reported `failed ` with the hosted wiring and active Bun lock byte-untouched (exit 1 / `partialFailure`): the package stays hosted-patched instead of being un-hosted and then refused. `vendor --dry-run` previews that same `failed` code (exit-code parity with the wet run, nothing written) instead of promising `vendor_would_revert_redirect`. Pinned by `tests/in_process_vendor_bun_takeover.rs` and, against real Bun, `tests/mode_migration_bun.rs`. The npm package-lock backend's lock gate gets the same placement: a hosted pin in a project whose `npm-shrinkwrap.json` / `package-lock.json` is not a v2/v3 lock (npm 6's lockfileVersion 1) is refused `failed vendor_lockfile_version_unsupported` BEFORE the restore, in `vendor`, `scan --mode vendored` and `get --mode vendored` alike, so the package stays hosted-patched; the vendored dry-run preview lists every npm purl of such a project as `would_refuse` with that code. Pinned by `tests/in_process_vendor_npm_v1_takeover.rs`. Hosted → vendored and vendored → hosted (`redirect_takeover_reverted_vendored` in `warnings[]`) both work in place on the locks the target mode accepts. **Removed in v5.0**: the run-level `vendor_supersedes_redirect` warning and its reconcile of the redirect ledger (a live lock that already proved vendored won over a stale hosted ledger record) — once the lock routes a package to `.socket/vendor/`, no hosted state is left to go stale. Which way the live lock points is decided by the same lockfile discovery rules `vex` gates attestations on (see "Manifest-less VEX (lockfile discovery)"), for `redirect_supersedes_vendored` and `hosted_wiring_retained` alike. +**Withdrawn pins (`scan --mode hosted`, additive/MINOR)**: before the rewrite, a hosted scan checks every patch uuid the lockfiles pin, for a package this run covered, against the reference endpoint. A pin the server reports `withdrawn` (unpublished) or `not_found` (no such patch) no longer installs (the patch server answers 410), so it is restored to its default upstream registry entry: the same restore `rollback` runs (see "Hosted unwind coverage"), under the apply lock, `--dry-run` previewing it without writing. If the package has a replacement patch on offer, the rewrite that follows pins the replacement. Each restored pin is one `rolledBack` event (`verified` on a dry run) with `uuid`, `errorCode: "patch_withdrawn"` and `details.mode: "hosted"`; the restore's advisories (`redirect_bun_reinstall_required`, `redirect_pypi_reinstall_required`, `redirect_vlt_reinstall_required`) join `warnings[]`. Only those two statuses unpin: a pin merely absent from the search results (a failed batch, the free-only proxy fallback, a lapsed paid entitlement), or one reported `forbidden`, `pending_build` or `build_failed`, is kept. If the reference lookup fails, every pin is kept with a `redirect_withdrawn_check_failed` warning; a pin the restore refuses (or the apply lock being held) is kept with a `redirect_withdrawn_restore_failed` warning naming the remedy. Neither changes the status or the exit code. A run whose every search batch failed exits before the check. + ### Scan modes (v5.0) **Mode resolution (`resolve_mode_flags`, MAJOR in v5.0).** `--mode`, or `--sync` (agent mode), picks the mode. With neither, `scan` and `get` keep the mode the project's patch state already records (`mode_from_project_state`, #1088; both stores are read from the project the resolved manifest belongs to, so a `--manifest-path` into another project consults that project's ledger): a non-empty vendor ledger (`.socket/vendor/state.json`; also an unreadable or malformed one, so the vendored flow reports it) means **vendored**, a manifest holding patches the ledger does not already cover (by key or base purl), or an unreadable or malformed manifest (so the agent flow reports it), means **agent** — a record the ledger covers, as `get --save-only` then `vendor` leaves, is vendored state — and a project with neither runs **hosted** mode — JSON and human alike (see the hosted paragraph below and [`scan` and `get` JSON](#scan-and-get-json-v50-major)). A project holding both agent and vendored state is a usage error (exit 2, code `mode_ambiguous`, naming the three `--mode` values) and writes nothing. So is a vendor ledger beside hosted pins in the project's lockfiles (#1437), the mixed state a partial `--mode hosted` takeover or per-package `get --mode` leaves: hosted mode keeps no ledger, so its pins are its state, and keeping vendored mode would take every pin over. A human-mode run that kept vendored or agent mode says so on stderr (`Note: using --mode vendored because … already holds vendored patches; pass --mode explicitly to switch modes`); `--json` and `--silent` print no note. **Switching modes is always explicit**: the in-place vendored → hosted takeover (see **Staged takeover**) runs only under `--mode hosted`, the hosted → vendored takeover only under `--mode vendored`, and a bare `scan`/`get` never converts a project. With hosted/vendored PATH arguments and no `--mode`, each project directory takes its mode from its own state. The one exception to the state-derived default: a `--prune` or `--global`/`--global-prefix` scan with no mode has no project lockfile to rewire, so it is **report-only** — discovery, the table, the `updates` array and the `redirectState` block below, plus the `--prune` GC — and, in human mode, ends with the hint `To apply these patches in place, run:` / ` socket-patch scan --mode agent [PATHS]` / ` socket-patch get `. A global scan's hint carries the run's scope, so it can be run verbatim: `-g` (`scan --mode agent -g` / `get -g <…>`), or `--global-prefix ` when a prefix was given (the directory shell-quoted when it needs it). An explicit `--mode hosted` or `--mode vendored` with `--global`/`--global-prefix` is a usage error (exit 2: global installs have no project lockfile to redirect, or to wire vendored artifacts into); `get` enforces the same rule with the same wording. @@ -1384,7 +1386,7 @@ command's integration tests check its output. | `removed` | `remove`, `repair`, `vendor`, `rollback`, `scan --prune` (manifest entries carry `details.manifest: true`) | A manifest entry or vendored state was removed, or (artifact-level, no `purl`) `.socket/` artifacts were swept — that GC carrier sets `bytes` and is not counted in `summary.removed`; the sweep's totals are the envelope's `gc`. | | `verified` | `apply`, `remove`, `repair`, `vendor`, `rollback`, `get`, `scan` (dry run); `vex`; `--update --dry-run` | The action *would* succeed cleanly (`files` lists previewed changes); `vex`: the patch verified and was attested. | | `rebuilt` | `repair`, `vendor` | A missing/corrupt vendored artifact was restored from its exact server download (v5.0: never a lost ledger entry — see `vendor_ledger_missing`). `summary.rebuilt` counts these. | -| `rolledBack` | `rollback`, `remove` | A patched package was restored to its original files (`files` lists them), or (`rollback`) a vendored / hosted wiring was unwound (`details.mode`). `remove`: one per restored installed copy, `details.path` naming it — v5.0, MAJOR: was the GC carrier's `details.rolledBack`. | +| `rolledBack` | `rollback`, `remove`, `scan --mode hosted` | A patched package was restored to its original files (`files` lists them), or (`rollback`) a vendored / hosted wiring was unwound (`details.mode`). `remove`: one per restored installed copy, `details.path` naming it — v5.0, MAJOR: was the GC carrier's `details.rolledBack`. `scan --mode hosted`: a pin whose patch was withdrawn was restored upstream (`errorCode: "patch_withdrawn"`). | ### Stable `errorCode` tags @@ -1574,6 +1576,9 @@ command's integration tests check its output. | `redirect_bun_reinstall_required` | rollback/remove `warnings[]` (+ human stderr), `vendor --revert` `warnings[]` | a wet hosted unwind that restored `bun.lock` / `bun.lockb` pins to the registry record while their installed copy may be kept (the `vendor_bun_reinstall_required` rule: Bun's hoisted linker keeps it through a plain `bun install`). One run-level warning naming every such `name@version` and `bun install --force` (or deleting `node_modules`). `vendor --revert` leaves it out when the vendored revert already advised `vendor_bun_reinstall_required` for every package it re-hosted. | | `vendor_pypi_reinstall_required` | `skipped` (advisory; human: `Warning: …`); rollback/remove `warnings[]`; `scan --prune` top-level `warnings[]` (human: `GC: …`) | a PyPI vendored revert (`vendor --revert`, rollback / remove of a vendored entry, the manifest reconcile, `--preserve-state` and `--dry-run` included) of a PDM, uv or Pipenv entry (#477). These tools reinstall a same-version package only when the LOCKED candidate is a URL or file that differs from the installed one, so once the lock is back on the registry release a plain `pdm sync` / `pdm install`, `uv sync` or `pipenv sync` reports nothing to do and keeps the vendored build. The detail names the purl and the reinstall that restores the upstream bytes: `pdm sync --reinstall` (or recreating the virtualenv / `__pypackages__`), `uv sync --reinstall-package `, or Pipenv's `pipenv run pip uninstall -y && pipenv sync` (the categories from `Pipfile.lock`; `pipenv --rm && pipenv sync` for a clean virtualenv). Poetry, requirements.txt, Hatch and PEP 751 entries do not get it. | | `redirect_pypi_reinstall_required` | rollback/remove `warnings[]` (+ human stderr), `vendor --revert` `warnings[]` | a hosted unwind (wet or `--dry-run`) that restored PyPI pins in `pdm.lock`, `uv.lock` or `Pipfile.lock` to their registry entry: the `vendor_pypi_reinstall_required` rule for hosted pins (the patched build's `direct_url.json` still names the patch server). One run-level warning naming each purl and its reinstall. | +| `redirect_withdrawn_check_failed` | `warnings[]` (warning; human: `Warning: …`) | scan `--mode hosted`: the reference lookup for the uuids the lockfiles pin failed, so the run could not tell whether any was withdrawn. Every pin is kept; status and exit unchanged. See "Withdrawn pins". | +| `redirect_withdrawn_restore_failed` | `warnings[]` (warning; human: `Warning: …` / `Error: …`) | scan `--mode hosted`: a pin whose patch is `withdrawn` / `not_found` could not be restored to its upstream registry entry (the restore refused it, writing the restored files failed, or the apply lock was held). The pin is kept; the detail names the remedy (`git checkout -- `, or re-run / `socket-patch rollback `). Status and exit unchanged. | +| `patch_withdrawn` | `rolledBack` / `verified` (hosted event) | scan `--mode hosted`: the lockfile pinned a patch the server reports `withdrawn` or `not_found`; the pin was restored (dry run: would be) to its upstream registry entry. `reason` names the uuid and status. See "Withdrawn pins". | | `vendor_prebuilt_stub_invalid` | `failed` | RubyGems: the server stub lacks required attributes or is otherwise invalid; no local stub fallback is permitted. | | `vendor_*` / `pypi_*` / `gemfile_*` / `lock_*` / `locked_version_mismatch` / `user_authored_*` / `native_extensions_unsupported` / `platform_gem_unsupported` | `failed`/`skipped` | vendor: per-ecosystem refusal + drift vocabulary; see the Vendor command contract section. New tags are additive (MINOR). | diff --git a/crates/socket-patch-cli/src/commands/scan/mod.rs b/crates/socket-patch-cli/src/commands/scan/mod.rs index 09b0a359c..a5eadaa04 100644 --- a/crates/socket-patch-cli/src/commands/scan/mod.rs +++ b/crates/socket-patch-cli/src/commands/scan/mod.rs @@ -58,6 +58,7 @@ pub(crate) mod rollout; pub mod rollout_args; mod socket_yml_args; pub(crate) mod vendor_flow; +mod withdrawn; use self::discovery::{ collect_vuln_ids, detect_updates, lockfile_only_contains, lockfile_supplement, @@ -2571,6 +2572,17 @@ async fn run_scan( status.finish(); } + // Hosted mode: restore the lockfile pins whose patch the server no + // longer serves, before the rewrite below re-pins any replacement. + // Runs ahead of every "nothing found" exit: a package whose only patch + // was withdrawn has no search results at all. + let withdrawn_sweep = if hosted && !args.common.is_global() && policy.writes_allowed() { + let pins = withdrawn::pins_in_scope(&hosted_pin_list, &all_purls); + Some(withdrawn::sweep(&args.common, &api_client, &pins).await) + } else { + None + }; + // Calculate patch counts let mut free_patches = 0usize; let mut paid_patches = 0usize; @@ -2683,6 +2695,9 @@ async fn run_scan( // Hosted mode: the redirect engine records its events into this // envelope and prints it. if hosted { + if let Some(sweep) = withdrawn_sweep { + sweep.record_into(&mut env); + } return run_redirect( &args, &api_client, diff --git a/crates/socket-patch-cli/src/commands/scan/withdrawn.rs b/crates/socket-patch-cli/src/commands/scan/withdrawn.rs new file mode 100644 index 000000000..c5ccc120b --- /dev/null +++ b/crates/socket-patch-cli/src/commands/scan/withdrawn.rs @@ -0,0 +1,274 @@ +//! Hosted mode's withdrawn-pin sweep: a lockfile pin whose patch the +//! server no longer serves is restored to its upstream registry entry. +//! +//! A hosted pin points the lockfile at the patch server's copy of one +//! patch uuid. Once that patch is unpublished the server stops serving it +//! (the download answers 410), so every fresh install of the pinned +//! lockfile fails. The rewrite never notices on its own: a withdrawn patch +//! drops out of the search results, so nothing selects it and its pin is +//! left as is. +//! +//! Before the rewrite, `scan --mode hosted` asks the reference endpoint +//! about every uuid the lockfiles pin for a package this run covered. A pin +//! whose uuid comes back `withdrawn` (unpublished) or `not_found` (no such +//! patch) is restored upstream, the way `rollback` restores it +//! ([`crate::commands::hosted_unwind::run_hosted_leg`]). If the package has +//! a replacement patch on offer, the rewrite that follows pins the +//! replacement, so the net effect is an update. Otherwise the package goes +//! back to the registry's unpatched copy. +//! +//! Only those two explicit statuses unpin. A pin that is merely missing +//! from the search results is kept: that is also what a failed search +//! batch, a fallback to the free-only public proxy, or a lapsed paid +//! entitlement looks like. A `forbidden`, `pending_build` or `build_failed` +//! answer keeps the pin too. If the reference lookup fails (the patch API +//! is down, say), every pin is kept and the run warns. + +use std::collections::BTreeSet; +use std::time::Duration; + +use socket_patch_core::api::client::ApiClient; +use socket_patch_core::patch::redirect::upstream::HostedPin; +use socket_patch_core::utils::purl_key::PurlKey; + +use crate::args::GlobalArgs; +use crate::commands::agent_download::tag_mode; +use crate::json_envelope::{Envelope, PatchAction, PatchEvent}; + +/// `reason` code on the `rolledBack` event of a pin the sweep restored. +pub(super) const PATCH_WITHDRAWN: &str = "patch_withdrawn"; + +/// Warning: a withdrawn pin could not be restored upstream (the restore +/// refused it, the restored files could not be written, or the apply lock +/// was held). The pin is unchanged; the detail names the remedy. +pub(super) const REDIRECT_WITHDRAWN_RESTORE_FAILED: &str = "redirect_withdrawn_restore_failed"; + +/// Warning: the reference lookup for the pinned uuids failed, so the run +/// could not tell whether any pin was withdrawn. Every pin is kept. +pub(super) const REDIRECT_WITHDRAWN_CHECK_FAILED: &str = "redirect_withdrawn_check_failed"; + +/// Whether a reference `status` means the pinned patch is gone for good. +fn is_gone(status: &str) -> bool { + matches!(status, "withdrawn" | "not_found") +} + +/// What the sweep did, for the `--json` envelope. Human output is printed +/// as the sweep runs. +#[derive(Debug, Default)] +pub(super) struct WithdrawnSweep { + /// `rolledBack` (`verified` on a dry run) per restored pin. + events: Vec, + /// `(code, detail)` run-level warnings. + warnings: Vec<(String, String)>, +} + +impl WithdrawnSweep { + /// Fold the outcome into the scan envelope. + pub(super) fn record_into(self, env: &mut Envelope) { + for event in self.events { + env.record(event); + } + for (code, detail) in self.warnings { + env.warn(code, detail); + } + } + + fn warn(&mut self, common: &GlobalArgs, code: &str, detail: String) { + // Warnings print even under --silent: a kept withdrawn pin breaks + // the next fresh install. + if !common.json { + eprintln!("Warning: {detail}"); + } + self.warnings.push((code.to_string(), detail)); + } +} + +/// The pins in `pins` whose package this run covered (`covered`: the purls +/// it queried), matched by release identity, so a qualified or +/// percent-encoded spelling still names its pin. +pub(super) fn pins_in_scope( + pins: &[HostedPin], + covered: impl IntoIterator>, +) -> Vec { + let covered: BTreeSet = covered + .into_iter() + .map(|p| PurlKey::new(p.as_ref())) + .collect(); + pins.iter() + .filter(|pin| covered.contains(&PurlKey::new(&pin.purl))) + .cloned() + .collect() +} + +/// Check `pins` against the reference endpoint and restore the withdrawn +/// ones upstream (see the module docs). A wet restore runs under the apply +/// lock, released before the hosted rewrite takes it again. +pub(super) async fn sweep( + common: &GlobalArgs, + api_client: &ApiClient, + pins: &[HostedPin], +) -> WithdrawnSweep { + let mut out = WithdrawnSweep::default(); + if pins.is_empty() { + return out; + } + let uuids: Vec = pins + .iter() + .map(|pin| pin.uuid.clone()) + .collect::>() + .into_iter() + .collect(); + let references = match api_client.fetch_registry_references(&uuids).await { + Ok(references) => references, + Err(e) => { + out.warn( + common, + REDIRECT_WITHDRAWN_CHECK_FAILED, + format!( + "could not check whether the hosted patches pinned in the lockfiles are \ + still published ({e}); every pin was kept, re-run to check again" + ), + ); + return out; + } + }; + // A uuid the response leaves out is not known to be gone: kept. + let gone: Vec<(HostedPin, String)> = pins + .iter() + .filter_map(|pin| { + let status = &references.get(&pin.uuid)?.status; + is_gone(status).then(|| (pin.clone(), status.clone())) + }) + .collect(); + if gone.is_empty() { + return out; + } + + if !common.json && !common.silent { + for (pin, status) in &gone { + eprintln!( + "Note: patch {} for {} is no longer published ({status}); restoring its \ + upstream registry entry", + pin.uuid, pin.purl + ); + } + } + + let _lock = if common.dry_run { + None + } else { + let socket_dir = common.socket_dir(); + let timeout = Duration::from_secs(common.lock_timeout.unwrap_or(0)); + match crate::commands::lock_cli::acquire_with_status(&socket_dir, timeout) { + Ok(guard) => Some(guard), + Err(err) => { + let (_, message) = crate::commands::lock_cli::lock_failure(&err, timeout); + for (pin, status) in &gone { + out.warn( + common, + REDIRECT_WITHDRAWN_RESTORE_FAILED, + format!( + "{} is pinned to patch {}, which is no longer published \ + ({status}), but it was not restored: {message}; re-run, or run \ + `socket-patch rollback {}`", + pin.purl, pin.uuid, pin.purl + ), + ); + } + return out; + } + } + }; + + let to_restore: Vec = gone.iter().map(|(pin, _)| pin.clone()).collect(); + let leg = crate::commands::hosted_unwind::run_hosted_leg(common, &to_restore).await; + + let action = if common.dry_run { + PatchAction::Verified + } else { + PatchAction::RolledBack + }; + for (pin, status) in &gone { + if !leg.reverted.iter().any(|p| p == &pin.purl) { + continue; + } + let verb = if common.dry_run { + "would be restored" + } else { + "was restored" + }; + out.events.push(tag_mode( + PatchEvent::new(action, pin.purl.as_str()) + .with_uuid(pin.uuid.as_str()) + .with_reason( + PATCH_WITHDRAWN, + format!( + "patch {} is no longer published ({status}); the pin {verb} to the \ + upstream registry entry", + pin.uuid + ), + ), + Some("hosted"), + )); + } + // `run_hosted_leg` already printed each failure to stderr. + for (purl, why) in leg.failed { + out.warnings.push(( + REDIRECT_WITHDRAWN_RESTORE_FAILED.to_string(), + format!("{purl}: {why}"), + )); + } + for (code, detail) in leg.warnings { + if !common.json && !common.silent { + eprintln!("Warning: {detail}"); + } + out.warnings.push((code, detail)); + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + + fn pin(purl: &str, uuid: &str) -> HostedPin { + HostedPin { + purl: purl.to_string(), + uuid: uuid.to_string(), + files: vec!["package-lock.json".to_string()], + } + } + + #[test] + fn only_withdrawn_and_not_found_are_gone() { + assert!(is_gone("withdrawn")); + assert!(is_gone("not_found")); + for kept in [ + "granted", + "reused", + "pending_build", + "build_failed", + "forbidden", + "", + ] { + assert!(!is_gone(kept), "{kept} must keep the pin"); + } + } + + #[test] + fn scope_matches_by_release_identity() { + let pins = vec![ + pin("pkg:npm/@scope/a@1.0.0", "u1"), + pin("pkg:npm/b@2.0.0", "u2"), + ]; + let in_scope = pins_in_scope(&pins, ["pkg:npm/%40scope/a@1.0.0"]); + assert_eq!(in_scope, vec![pins[0].clone()]); + } + + #[test] + fn uncovered_pins_are_out_of_scope() { + let pins = vec![pin("pkg:npm/b@2.0.0", "u2")]; + assert!(pins_in_scope(&pins, ["pkg:npm/b@2.0.1"]).is_empty()); + assert!(pins_in_scope(&pins, Vec::::new()).is_empty()); + } +} diff --git a/crates/socket-patch-cli/tests/in_process_scan_withdrawn.rs b/crates/socket-patch-cli/tests/in_process_scan_withdrawn.rs new file mode 100644 index 000000000..a4629eab5 --- /dev/null +++ b/crates/socket-patch-cli/tests/in_process_scan_withdrawn.rs @@ -0,0 +1,527 @@ +//! `scan --mode hosted` over a lockfile pin whose patch was withdrawn. +//! +//! Each test wires an npm package-lock project through the REAL hosted flow +//! (in-process `scan --mode hosted` against a wiremock API, the +//! `in_process_rollback_hosted.rs` fixture), then re-scans against a second +//! mock whose reference endpoint reports the pinned uuid's new state: +//! +//! - `withdrawn` / `not_found`: the pin is restored to its upstream registry +//! entry (byte-identical to the pristine lock), or, when a replacement +//! patch is offered, re-pinned to the replacement. +//! - `granted`, or a failed reference lookup: the pin is kept, even when the +//! search results no longer list the patch. +//! +//! `#[serial]`: every command's `run` mirrors env toggles into +//! process-global env vars (`apply_env_toggles`), and the upstream restore +//! reads `SOCKET_NPM_REGISTRY`. + +#[path = "common/hermetic.rs"] +mod hermetic; + +use std::path::Path; + +use serde_json::Value; +use serial_test::serial; +use socket_patch_cli::commands::scan::{run as scan_run, ScanArgs, ScanMode}; +use wiremock::matchers::{method, path, path_regex}; +use wiremock::{Mock, MockServer, ResponseTemplate}; + +const ORG: &str = "test-org"; +const NAME: &str = "in-proc-redirect"; +const VERSION: &str = "1.0.0"; +const PURL: &str = "pkg:npm/in-proc-redirect@1.0.0"; +const UUID: &str = "11111111-1111-4111-8111-111111111111"; +const HOSTED_URL: &str = "http://patch.test/patch/npm/in-proc-redirect/1.0.0/22222222-2222-4222-8222-222222222222/11111111-1111-4111-8111-111111111111/in-proc-redirect-1.0.0.tgz"; +const PATCHED_SHA512: &str = "sha512-PATCHEDpatchedPATCHEDpatched0123456789=="; +/// The replacement patch a superseding offer pins. +const UUID2: &str = "33333333-3333-4333-8333-333333333333"; +const HOSTED_URL2: &str = "http://patch.test/patch/npm/in-proc-redirect/1.0.0/22222222-2222-4222-8222-222222222222/33333333-3333-4333-8333-333333333333/in-proc-redirect-1.0.0.tgz"; +const PATCHED_SHA512_2: &str = "sha512-REPLACEMENTreplacementREPLACEMENT0123456=="; + +fn scan_args(cwd: &Path, api_url: String, dry_run: bool) -> ScanArgs { + ScanArgs { + socket_yml: Default::default(), + paths: Vec::new(), + packages: Vec::new(), + common: socket_patch_cli::args::GlobalArgs { + cwd: cwd.to_path_buf(), + org: Some(ORG.to_string()), + api_token: Some("fake".to_string()), + api_url: Some(api_url), + // The mock patch host's pins count as hosted. + patch_server_url: Some("http://patch.test".to_string()), + json: true, + yes: true, + dry_run, + ..socket_patch_cli::args::GlobalArgs::default() + }, + batch_size: Some(100), + prune: false, + sync: false, + mode: Some(ScanMode::Hosted), + all_releases: false, + vex: Default::default(), + rollout: Default::default(), + } +} + +/// Run an in-process hosted scan with the npm registry pointed at +/// `server`, scrubbing the env toggles `run` mirrors into the process. +async fn scan(cwd: &Path, server: &MockServer, dry_run: bool) -> i32 { + std::env::set_var( + "SOCKET_NPM_REGISTRY", + format!("{}/npm-registry", server.uri()), + ); + let code = scan_run(scan_args(cwd, server.uri(), dry_run)).await; + for var in ["SOCKET_NPM_REGISTRY", "SOCKET_DRY_RUN"] { + std::env::remove_var(var); + } + code +} + +fn offer(uuid: &str) -> Value { + serde_json::json!({ + "uuid": uuid, "purl": PURL, "tier": "free", + "cveIds": [], "ghsaIds": [], "severity": "high", + "title": "withdrawn fixture" + }) +} + +/// The batch search (offering `offers`) and by-package endpoints. +async fn mock_search(server: &MockServer, offers: &[&str]) { + let packages = if offers.is_empty() { + serde_json::json!([]) + } else { + serde_json::json!([{ + "purl": PURL, + "patches": offers.iter().map(|u| offer(u)).collect::>(), + }]) + }; + Mock::given(method("POST")) + .and(path(format!("/v0/orgs/{ORG}/patches/batch"))) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "packages": packages, + "canAccessPaidPatches": false, + }))) + .mount(server) + .await; + let records: Vec = offers + .iter() + .map(|u| { + serde_json::json!({ + "uuid": u, "purl": PURL, + "publishedAt": "2024-01-01T00:00:00Z", + "description": "x", "license": "MIT", "tier": "free", + "vulnerabilities": {} + }) + }) + .collect(); + Mock::given(method("GET")) + .and(path_regex(format!( + "^/v0/orgs/{ORG}/patches/by-package/.+$" + ))) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "patches": records, + "canAccessPaidPatches": false, + }))) + .mount(server) + .await; +} + +fn granted(url: &str, sha512: &str) -> Value { + serde_json::json!({ + "status": "granted", + "url": url, + "purl": PURL, + "artifacts": [{ + "kind": "tarball", + "url": url, + "integrity": { "sha512": sha512 } + }], + "registryOverride": null + }) +} + +/// The reference endpoint answering `results`. +async fn mock_references(server: &MockServer, results: Value) { + Mock::given(method("POST")) + .and(path(format!("/v0/orgs/{ORG}/patches/package"))) + .respond_with( + ResponseTemplate::new(200).set_body_json(serde_json::json!({ "results": results })), + ) + .mount(server) + .await; +} + +async fn mock_view(server: &MockServer, uuid: &str) { + Mock::given(method("GET")) + .and(path(format!("/v0/orgs/{ORG}/patches/view/{uuid}"))) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "uuid": uuid, + "purl": PURL, + "publishedAt": "2024-01-01T00:00:00Z", + "files": { + "package/index.js": { + "beforeHash": "a".repeat(64), + "afterHash": "b".repeat(64), + } + }, + "vulnerabilities": {}, + "description": "x", "license": "MIT", "tier": "free" + }))) + .mount(server) + .await; +} + +/// The npm registry's version document, for the upstream restore. +async fn mock_npm_registry(server: &MockServer) { + Mock::given(method("GET")) + .and(path(format!("/npm-registry/{NAME}/{VERSION}"))) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "name": NAME, + "version": VERSION, + "dist": { + "tarball": format!("https://registry.npmjs.org/{NAME}/-/{NAME}-{VERSION}.tgz"), + "integrity": "sha512-UPSTREAMupstream==", + } + }))) + .mount(server) + .await; +} + +/// Write the npm project and return the pristine lock bytes (normalized +/// the way the redirect writer serializes, so a restore can be compared +/// byte for byte). +fn write_npm_project(root: &Path) -> String { + std::fs::write( + root.join("package.json"), + format!( + r#"{{ "name": "consumer", "version": "0.0.0", "dependencies": {{ "{NAME}": "{VERSION}" }} }}"# + ), + ) + .unwrap(); + let pkg = root.join("node_modules").join(NAME); + std::fs::create_dir_all(&pkg).unwrap(); + std::fs::write( + pkg.join("package.json"), + format!(r#"{{ "name": "{NAME}", "version": "{VERSION}" }}"#), + ) + .unwrap(); + let raw = format!( + r#"{{ + "name": "consumer", + "version": "0.0.0", + "lockfileVersion": 3, + "requires": true, + "packages": {{ + "": {{ "name": "consumer", "version": "0.0.0", "dependencies": {{ "{NAME}": "{VERSION}" }} }}, + "node_modules/{NAME}": {{ + "version": "{VERSION}", + "resolved": "https://registry.npmjs.org/{NAME}/-/{NAME}-{VERSION}.tgz", + "integrity": "sha512-UPSTREAMupstream==" + }} + }} +}} +"# + ); + let normalized = format!( + "{}\n", + serde_json::to_string_pretty(&serde_json::from_str::(&raw).unwrap()).unwrap() + ); + std::fs::write(root.join("package-lock.json"), &normalized).unwrap(); + normalized +} + +fn lock(root: &Path) -> String { + std::fs::read_to_string(root.join("package-lock.json")).unwrap() +} + +/// Wire the project to [`UUID`] through a real hosted scan; returns the +/// pristine and the wired lock. +async fn wired_project(root: &Path) -> (String, String) { + let pristine = write_npm_project(root); + let server = MockServer::start().await; + mock_search(&server, &[UUID]).await; + mock_references( + &server, + serde_json::json!({ UUID: granted(HOSTED_URL, PATCHED_SHA512) }), + ) + .await; + mock_view(&server, UUID).await; + assert_eq!(scan(root, &server, false).await, 0, "wiring scan"); + let wired = lock(root); + assert!( + wired.contains(HOSTED_URL) && wired.contains(PATCHED_SHA512), + "the lock must be wired before the re-scan means anything:\n{wired}" + ); + (pristine, wired) +} + +/// Requests `server` received on the reference endpoint. +async fn reference_posts(server: &MockServer) -> usize { + server + .received_requests() + .await + .unwrap_or_default() + .iter() + .filter(|r| r.url.path().ends_with("/patches/package")) + .count() +} + +#[tokio::test] +#[serial] +async fn a_withdrawn_pin_is_restored_upstream() { + let tmp = tempfile::tempdir().unwrap(); + let (pristine, _) = wired_project(tmp.path()).await; + + // The patch was unpublished: search no longer lists it. + let server = MockServer::start().await; + mock_search(&server, &[]).await; + mock_references( + &server, + serde_json::json!({ UUID: { "status": "withdrawn" } }), + ) + .await; + mock_npm_registry(&server).await; + assert_eq!(scan(tmp.path(), &server, false).await, 0); + + assert_eq!( + lock(tmp.path()), + pristine, + "the withdrawn pin must be restored to the upstream registry entry" + ); + assert!( + !tmp.path().join(".socket").exists(), + "the sweep's apply lock leaves no .socket/ residue" + ); +} + +#[tokio::test] +#[serial] +async fn a_not_found_pin_is_restored_upstream() { + let tmp = tempfile::tempdir().unwrap(); + let (pristine, _) = wired_project(tmp.path()).await; + + let server = MockServer::start().await; + mock_search(&server, &[]).await; + mock_references( + &server, + serde_json::json!({ UUID: { "status": "not_found" } }), + ) + .await; + mock_npm_registry(&server).await; + assert_eq!(scan(tmp.path(), &server, false).await, 0); + + assert_eq!(lock(tmp.path()), pristine); +} + +#[tokio::test] +#[serial] +async fn a_withdrawn_pin_with_a_replacement_is_repinned() { + let tmp = tempfile::tempdir().unwrap(); + wired_project(tmp.path()).await; + + // The old patch was unpublished and a new one published for the same + // package version. + let server = MockServer::start().await; + mock_search(&server, &[UUID2]).await; + mock_references( + &server, + serde_json::json!({ + UUID: { "status": "withdrawn" }, + UUID2: granted(HOSTED_URL2, PATCHED_SHA512_2), + }), + ) + .await; + mock_view(&server, UUID2).await; + mock_npm_registry(&server).await; + assert_eq!(scan(tmp.path(), &server, false).await, 0); + + let after = lock(tmp.path()); + assert!( + after.contains(HOSTED_URL2) && after.contains(PATCHED_SHA512_2), + "the replacement patch must be pinned:\n{after}" + ); + assert!( + !after.contains(UUID), + "nothing may still name the withdrawn patch:\n{after}" + ); +} + +#[tokio::test] +#[serial] +async fn a_pin_missing_from_search_but_still_granted_is_kept() { + let tmp = tempfile::tempdir().unwrap(); + let (_, wired) = wired_project(tmp.path()).await; + + // Absent from search (a failed batch, the free-only proxy and a lapsed + // entitlement all look like this), but the server still serves it. + let server = MockServer::start().await; + mock_search(&server, &[]).await; + mock_references( + &server, + serde_json::json!({ UUID: granted(HOSTED_URL, PATCHED_SHA512) }), + ) + .await; + mock_npm_registry(&server).await; + assert_eq!(scan(tmp.path(), &server, false).await, 0); + + assert_eq!(lock(tmp.path()), wired, "a served pin is never unpinned"); + assert_eq!( + reference_posts(&server).await, + 1, + "the pin was checked against the reference endpoint" + ); +} + +#[tokio::test] +#[serial] +async fn a_forbidden_pin_is_kept() { + let tmp = tempfile::tempdir().unwrap(); + let (_, wired) = wired_project(tmp.path()).await; + + let server = MockServer::start().await; + mock_search(&server, &[]).await; + mock_references( + &server, + serde_json::json!({ UUID: { "status": "forbidden" } }), + ) + .await; + mock_npm_registry(&server).await; + assert_eq!(scan(tmp.path(), &server, false).await, 0); + + assert_eq!(lock(tmp.path()), wired); +} + +#[tokio::test] +#[serial] +async fn a_failed_reference_lookup_keeps_every_pin() { + let tmp = tempfile::tempdir().unwrap(); + let (_, wired) = wired_project(tmp.path()).await; + + // The patch API answers search but its reference endpoint is down. + let server = MockServer::start().await; + mock_search(&server, &[]).await; + Mock::given(method("POST")) + .and(path(format!("/v0/orgs/{ORG}/patches/package"))) + .respond_with(ResponseTemplate::new(500)) + .mount(&server) + .await; + mock_npm_registry(&server).await; + assert_eq!(scan(tmp.path(), &server, false).await, 0); + + assert_eq!(lock(tmp.path()), wired, "an outage must never unpin"); +} + +#[tokio::test] +#[serial] +async fn a_search_outage_never_reaches_the_sweep() { + let tmp = tempfile::tempdir().unwrap(); + let (_, wired) = wired_project(tmp.path()).await; + + // Every search batch fails: the scan fails before the sweep runs. + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path(format!("/v0/orgs/{ORG}/patches/batch"))) + .respond_with(ResponseTemplate::new(500)) + .mount(&server) + .await; + mock_references( + &server, + serde_json::json!({ UUID: { "status": "withdrawn" } }), + ) + .await; + mock_npm_registry(&server).await; + assert_eq!(scan(tmp.path(), &server, false).await, 1); + + assert_eq!(lock(tmp.path()), wired); + assert_eq!(reference_posts(&server).await, 0); +} + +#[tokio::test] +#[serial] +async fn a_dry_run_previews_the_restore_without_writing() { + let tmp = tempfile::tempdir().unwrap(); + let (_, wired) = wired_project(tmp.path()).await; + + let server = MockServer::start().await; + mock_search(&server, &[]).await; + mock_references( + &server, + serde_json::json!({ UUID: { "status": "withdrawn" } }), + ) + .await; + mock_npm_registry(&server).await; + assert_eq!(scan(tmp.path(), &server, true).await, 0); + + assert_eq!(lock(tmp.path()), wired, "a dry run writes nothing"); + assert!(!tmp.path().join(".socket").exists()); +} + +/// The `--json` envelope (through the binary: an in-process run prints to +/// the real stdout): one `rolledBack` event with `reason: patch_withdrawn`. +#[tokio::test] +#[serial] +async fn the_envelope_reports_the_restored_pin() { + let tmp = tempfile::tempdir().unwrap(); + let (pristine, _) = wired_project(tmp.path()).await; + + let server = MockServer::start().await; + mock_search(&server, &[]).await; + mock_references( + &server, + serde_json::json!({ UUID: { "status": "withdrawn" } }), + ) + .await; + mock_npm_registry(&server).await; + + let mut cmd = hermetic::binary_command(); + for key in ["NPM_CONFIG_REGISTRY", "npm_config_registry"] { + cmd.env_remove(key); + } + let out = cmd + .env( + "SOCKET_NPM_REGISTRY", + format!("{}/npm-registry", server.uri()), + ) + .args([ + "scan", + "--json", + "--yes", + "--mode", + "hosted", + "--org", + ORG, + "--api-token", + "fake", + "--api-url", + &server.uri(), + "--patch-server-url", + "http://patch.test", + "--cwd", + tmp.path().to_str().unwrap(), + ]) + .output() + .expect("run socket-patch"); + let envelope: Value = serde_json::from_slice(&out.stdout).unwrap_or_else(|e| { + panic!( + "scan --json stdout must be a JSON envelope: {e}\nstdout:\n{}\nstderr:\n{}", + String::from_utf8_lossy(&out.stdout), + String::from_utf8_lossy(&out.stderr) + ) + }); + assert_eq!(out.status.code(), Some(0), "{envelope:#}"); + assert_eq!(envelope["status"], "success", "{envelope:#}"); + let restored: Vec<&Value> = envelope["events"] + .as_array() + .unwrap() + .iter() + .filter(|e| e["action"] == "rolledBack") + .collect(); + assert_eq!(restored.len(), 1, "{envelope:#}"); + assert_eq!(restored[0]["purl"], PURL); + assert_eq!(restored[0]["uuid"], UUID); + assert_eq!(restored[0]["errorCode"], "patch_withdrawn", "{envelope:#}"); + assert_eq!(restored[0]["details"]["mode"], "hosted"); + assert_eq!(lock(tmp.path()), pristine); +} From 3d987b5550a2a187ca2c927ca06c99ce61d260cf Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 15 Oct 2026 22:15:01 +0000 Subject: [PATCH 2/2] Expect the withdrawn sweep's reference call in the hosted render test scan --mode hosted now checks every pinned uuid against the reference endpoint before the rewrite, so the hosted render test, which seeds a pin to OLD_UUID, sees two reference posts: the sweep's (OLD_UUID, forbidden, kept) and the engine's (UUID). Assert both by body instead of a bare count of one, which failed the coverage job. Co-Authored-By: Claude --- .../tests/scan/scan_render.rs | 23 ++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/crates/socket-patch-cli/tests/scan/scan_render.rs b/crates/socket-patch-cli/tests/scan/scan_render.rs index a996d2ec9..0a718b102 100644 --- a/crates/socket-patch-cli/tests/scan/scan_render.rs +++ b/crates/socket-patch-cli/tests/scan/scan_render.rs @@ -1128,10 +1128,27 @@ async fn scan_hosted_human_prints_table_updates_and_redirects() { !stderr.contains("Non-interactive mode detected"), "scan never prompts; got {stderr:?}" ); + // Two reference posts: the withdrawn-pin sweep checks the pinned + // OLD_UUID first (forbidden here, so the pin is kept), then the engine + // resolves the offered UUID. + let reqs = recorded(&mock).await; assert_eq!( - reference_posts(&recorded(&mock).await), - 1, - "the engine resolved the reference" + reference_posts(&reqs), + 2, + "the withdrawn sweep and the engine each resolved a reference" + ); + let reference_bodies: Vec = reqs + .iter() + .filter(|r| format!("{}", r.method) == "POST" && r.url.path().ends_with("/patches/package")) + .map(|r| String::from_utf8_lossy(&r.body).into_owned()) + .collect(); + assert!( + reference_bodies.iter().any(|b| b.contains(OLD_UUID)), + "the withdrawn sweep checked the pinned uuid; got {reference_bodies:?}" + ); + assert!( + reference_bodies.iter().any(|b| b.contains(UUID)), + "the engine resolved the offered uuid; got {reference_bodies:?}" ); // `--dry-run` previews without confirming (the engine honors it itself).