Skip to content

FEAT: Rebuild saved registry entries in dependency order - #3908

Open
varunj-msft wants to merge 44 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Rebuild-Saved-Entries
Open

varunj-msft wants to merge 44 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Rebuild-Saved-Entries

Conversation

@varunj-msft

@varunj-msft varunj-msft commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Description

This is one of the changes for saving Registry entries across restarts. It's the third of four stacked PRs and builds on #3892. Until #3891 and #3892 merge, the diff here also shows their changes; this PR's own code is pyrit/registry/registry_recipe_restore.py and RegistryRestoreReport, with their tests.

Problem. Nothing turns saved recipes back into registry entries.

Change. RegistryRecipeRestorer: read() reads every folder without building anything, so a caller on an event loop can run it in a worker thread; restore() then builds on the calling thread, as the target API does, because a constructor can need the running loop. restore():

  • Reads all four folders. If one can't be listed, for any reason, it builds nothing, since other entries may use the ones it couldn't read.
  • Looks classes up among the registered ones, by name and then __module__, and looks a technique's attack type up in the attack registry. It never imports a module a file names.
  • Plans one order across the four kinds from Parameter.reference and the technique fields. A name resolves to a saved entry first (even a broken one), then to a startup entry. Missing and failed dependencies and loops are reported with the entry each one waits on.
  • Builds targets, converters and scorers with create_named_instance(..., external_input=True), so a server path is refused exactly as for API input (FIX: Reject path parameters by default when building components from API input #3594): a target may set only the path parameters its class lets callers set (caller_path_parameters, such as GitHubCopilotTarget's working_directory), and converters and scorers none. Techniques go through create_factory(..., strict_json=True) and register_runtime, as the technique API does. A test checks that technique recipes and the create request accept and reject the same documents and have the same defaults.
  • Refuses a recipe that stores a value for a parameter its class marks as a credential, for targets, converters, scorers and attack parameters. It also refuses classes that take the server's upload folder (upload_directory_parameter), which restore can't set yet.
  • Reports every other problem as a failure for that entry and keeps going.
  • Keeps failure messages safe to show. A constructor error can quote a server setting, such as an endpoint with a SAS token. So the message is cut at the first ? or #, where a URL's query or fragment starts, and says the rest is hidden. A message with an @, as in a URL's user name and password, is hidden entirely. Look-alike characters such as a full-width @ count as the real ones, and the text that is shown keeps its original characters.
  • Returns a RegistryRestoreReport: restored entries, failures (kind, name, reason, message, version, dependency) and ignored files.

Not in this PR.

  • Calling it. The next PR runs it at startup and on apply.
  • Credential rules that need each class to declare its fields: credentials a class would read from server settings, and fields that can carry a credential without being marked as one (headers, raw HTTP requests, URLs with tokens). Those come with saving, and restore will apply the same rules. Until PyRIT writes recipes, a stored file is trusted the way configuration is.

Tests and Documentation

  • New tests/unit/registry/test_registry_recipe_restore.py (89 tests): order across the four kinds (including converters in either of a technique's lists), loops, missing and failed dependencies, clashes with startup entries, read() building nothing and restore() using what it returned without reading again, a startup entry found by its exact name, stored credentials, server paths refused as the API refuses them (a target's declared caller path is accepted, and only for targets), class lookup without imports, unknown classes and attack types, technique recipes matching the create request, and failure messages (queries, fragments, user info, the exact message for every form of ?, # and @, original characters kept, long messages checked one character at a time).
  • pytest tests/unit/registry tests/unit/models -n 4: 3,935 passed, 5 skipped, 4 xfailed
  • pre-commit run --from-ref origin/main --to-ref HEAD: passed
  • No doc change: nothing calls this yet. JupyText: not run (no notebooks changed).

…API input

External construction now rejects a parameter whose type includes Path unless the
calling service passes the name in allowed_path_params after making the value safe.
The converter service allows its upload parameters once they are stored, and the
target service allows the server-set upload folder and the GitHubCopilotTarget
working directory. The scorer-only check is removed, and the "Valid parameters"
list leaves out path parameters callers cannot set.
The target catalog now applies the path rule to required parameters too, so a
target that needs a disallowed path is left out instead of listed without the
field, matching the scorer catalog. allowed_path_params is typed frozenset[str]
so a bare string cannot turn the allow-list into a substring match.
The target catalog still lists a target whose required path parameter is its
server-set upload folder when the server has one, and leaves it out otherwise.
# Conflicts:
#	tests/unit/backend/test_converter_service.py

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the dependency-ordered registry restoration changes.

No open issues 🎉

What changed in this PR

Adds dependency-ordered rebuilding of saved PyRIT registry entries, building on the storage work in #3891 and #3892. Startup and apply integration remain deferred to the next PR.

Changes:

  • Restores targets, converters, scorers, and techniques after their saved dependencies, reporting missing entries, failures, and dependency loops.
  • Applies registry input checks and credential restrictions, with sanitized error messages and a structured restore report.
  • Includes the stacked recipe models, issue-aware storage listing, and tests for restoration, validation, and storage failures.
File Description
tests/​unit/​registry/​test_registry_recipe_storage.py Tests recipe parsing, versions, storage sources, and report serialization.
tests/​unit/​registry/​test_registry_recipe_restore.py Tests dependency ordering, rebuild checks, failures, and message sanitization.
tests/​unit/​registry/​test_file_document_storage.py Tests local and Blob listing issues, file replacement races, and safe errors.
pyrit/​registry/​registry_recipe_storage.py Reads and validates versioned recipes from per-registry folders.
pyrit/​registry/​registry_recipe_restore.py Plans dependencies, rebuilds entries, and reports restore outcomes.
pyrit/​registry/​file_document_storage.py Adds issue-aware listings without changing legacy listing behavior.
pyrit/​models/​request_limits.py Adds the shared text-length limit for technique descriptions.
pyrit/​models/​catalog/​registry_recipe.py Defines recipe, reference, failure, and restore-report models.
pyrit/​models/​catalog/​__init__.py Exposes the new models through lazy exports.

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants