Skip to content

feat(export): reference data, check lists, contacts, the announcement and file caps in the product export - #16680

Draft
svader0 wants to merge 37 commits into
devfrom
feat/export-reference-data
Draft

svader0 wants to merge 37 commits into
devfrom
feat/export-reference-data

Conversation

@svader0

@svader0 svader0 commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Description

The product export now carries everything a migration client needs beyond findings, and it stays small enough to stream.

Reference data in the manifest and the product header:

  • The manifest lists the finding templates (with tags and vulnerability ids), the tool types and the tool configurations. A tool configuration leaves out its password, SSH key, API key and extras.
  • The manifest's reference lists the contacts and the report types. An engagement names them in requester and report_type.
  • The manifest carries the announcement and the login_banner, each null when the instance has none.
  • Each product header lists the product's engagement presets and API scan configurations.
  • Each engagement line keeps its boolean check_list and adds check_list_detail: the engagement's check list with its issue lists as finding ids, or null.
  • The manifest's features list holds check_lists, so a client knows before it reads a product that engagement lines carry the check list.
  • The new keys leave out archived_at and archive_batch.
  • The manifest's deduplication holds the scan types whose deduplication algorithm, hash fields or null-CWE switch differ from what DefectDojo ships. settings.dist.py keeps a copy of each shipped map, taken before any override, to compare against. A scan type the instance removed goes out with its fallback.

File caps:

  • max_file_bytes accepts at most 104857600 (100 MiB). A larger value is a 400 that names the field.
  • One exported line carries at most 48 MiB of base64 file data, filled in id order. A file past the budget keeps its size and goes out with data_base64: null and error: "line_cap".
  • A file whose name the server cannot encode (uwsgi under an ASCII locale) is reported like a missing file: size: null. Before, it failed the whole export.
  • Each manifest test type row also carries dynamically_generated, so a migration client can recreate the type with the same flags. The public API's test type serializer does not change.
  • The manifest takes file_counts=false. Then it skips the storage stat for each file and returns files_over_cap: null. With the default, it makes one storage call per file instead of two.

Import and audit history

  • The manifest and the product stream take two new parameters, import_history and audit_history. Both default to false. Without them, the stream does not change, and the manifest only adds the two names to features.
  • With import_history=true, the stream sends each test's imports as test_import lines. Each import is followed by test_import_actions lines of at most 10,000 actions.
  • With audit_history=true, the stream sends the audit events of the product and its engagements, tests, findings, finding groups and risk acceptances as audit_event lines. They come by model, then object, then event id.
  • With audit_history=true, the older audit log entries come as legacy_audit_entry lines, by id.
  • The history lines come after the risk acceptances, and the end line counts them. With the parameters, the manifest's counts and totals gain test_imports, test_import_actions, audit_events and legacy_audit_entries.
  • The stream reads events and entries in pages by key, so the database never holds a copy of a whole event table.
  • The DefectDojo Pro side and the connectors service that read these lines ship in the same release as this export.

Questionnaires, benchmarks, tracking files and endpoints

  • The manifest's reference now lists questionnaires with their questions and choices, and general_surveys (the open questionnaire links).
  • It also lists open_survey_responses (answered questionnaires with no engagement) and benchmark_types with their categories and requirements. A requirement carries its references.
  • A new parameter, product_extras, defaults to false. Without it, the stream does not change. The manifest always names product_extras in features.
  • With product_extras=true, the stream sends four new kinds of product line: tracking_file, benchmark, answered_survey and product_endpoint. They come after the risk acceptances and before the history.
  • A benchmark line holds one benchmark type of the product, with its results, notes and summary.
  • A product_endpoint line holds the URI, tags and metadata of each endpoint of the product. With Locations on, it holds each location that the product references, with the status, relationship and metadata for this product.
  • The end line counts the new lines. The manifest counts tracking_files, benchmarks and answered_surveys only when the client asks.
  • The docs now say that an empty finding group crosses on its test line, and that product_meta is a key of product.
  • The DefectDojo Pro side and the connectors service that read these lines ship in the same release as this export.

Jira links

  • A new parameter, jira, defaults to false. Without it, the stream does not change. The manifest always names jira in features.
  • With jira=true, the manifest's reference gains jira_instances. An instance row carries every field except password.
  • The product header gains jira_projects, the product's own Jira projects. An engagement line gains jira_project and jira_issue, each an object or null.
  • A finding line gains jira_issue, an object or null. A finding group's issue already rides in finding_groups[].jira_issue.
  • An issue row holds id, jira_id, jira_key, jira_creation, jira_change, jira_project, finding, engagement and finding_group.
  • The manifest's product counts and the end line gain jira_projects and jira_issues, only with jira=true. They count entries inside lines.
  • Nothing in the export calls Jira.

Threat models, configuration permissions, GitHub and SonarQube links

  • An engagement line always carries threat_model_file: null, or the file with filename, size, content_type and data_base64. It follows the rules of every other file, with max_file_bytes and the line_cap error.
  • The server reads a threat model file only when its path is inside the media root. Otherwise the entry has "error": "missing".
  • The threat model counts as one engagement file in files and files_over_cap. The threat_model boolean and tmodel_path stay on the line.
  • Each user in reference.users carries configuration_permissions: a sorted list of app_label.codename strings. It holds the configuration permissions of the user and of its groups.
  • The manifest's reference.configuration_permissions lists every configuration permission of the instance, in the same form. A client uses it to give a staff user the same powers in the new instance.
  • A new parameter, external_issues, defaults to false. Without it, the stream does not change. The manifest always names external_issues in features.
  • With external_issues=true, the manifest's reference gains github_configurations, with id and configuration_name. The GitHub API key never goes out.
  • The product header gains github_projects. A finding line gains github_issue, null or an object with issue_id and issue_url.
  • A finding line's sonarqube_issue becomes an object with key, status, type and its transitions in id order. Without the parameter, it stays the integer id.
  • The manifest's product counts and the end line gain github_projects, github_issues and sonarqube_issues, only with external_issues=true.
  • Nothing in the export calls GitHub or SonarQube.

Test results

  • python3 manage.py test unittests.test_apiv2_export unittests.test_apiv2_export_performance unittests.test_configuration_default_checks --keepdb: 94 tests, OK, with unittests.test_configuration_default_checks, on this branch on the latest dev.
  • ruff check --config ruff.toml on the changed files: clean.
  • After the history work (2026-11-11): unittests.test_apiv2_export and unittests.test_apiv2_export_performance: 123 tests, OK. ruff check --config ruff.toml is clean.
  • After the product extras work (2026-11-14): unittests.test_apiv2_export and unittests.test_apiv2_export_performance: 156 tests, OK, with Locations off and again with Locations on. Each mode skips the one test for the other mode. ruff check --config ruff.toml is clean.
  • After the Jira work (2026-11-16): unittests.test_apiv2_export and unittests.test_apiv2_export_performance: 165 tests, OK, with Locations off and again with Locations on. ruff check --config ruff.toml is clean.
  • After the threat model, permission and external issue work (2026-11-17): the same two modules give 178 tests, OK, with Locations off and again with Locations on. ruff check --config ruff.toml is clean.
  • Not tested: S3 or other remote storage for the missing-file case.

Documentation

docs/content/automation/api/product_export.md describes every new key, both caps, file_counts, the check list and deduplication. It also describes the import_history, audit_history, product_extras, jira and external_issues parameters, with their lines and keys. It describes threat_model_file and the configuration permission keys too.

Checklist

  • Up to date with the latest dev (merged on 2026-11-04).
  • Submitted against the dev branch.
  • Ruff compliant.
  • Documentation updated.
  • No model change, so no migration.
  • Unit tests added.

extras holds the Cobalt.io organization token, so the export drops it
for every tool configuration through an export-only serializer.
…figurations

The archive work on dev gave Engagement_Presets and
Product_API_Scan_Configuration the archived_at and archive_batch
fields. They describe this instance's archive, not the object, so the
export does not send them.
@svader0 svader0 added this to the 3.5.100 milestone Nov 3, 2026
Split the long sentences in the export docs and replace "would". Add the
`error` row to the file entry table. Build the numbers in the
`max_file_bytes` help text from the two cap constants and split its long
sentence. Pin that an empty `file_counts` uses the default.

Remove the comment on the file sort. The files of a line sort by id so the
files that a full line drops are the same on every run.
@svader0 svader0 changed the title feat(export): finding templates, tool configurations, presets and API scan configurations in the product export feat(export): reference data and file caps in the product export Nov 3, 2026
The engagement line replaces the check_list flag with the engagement's
check list (null when it has none, the lowest id when it has several),
including the finding ids of its eight issue lists. One prefetch loads
them per chunk, so the export issues no query per engagement.
…ner in the manifest

The manifest gains announcement and login_banner (null without a row),
reference.contacts and reference.report_types, and a features list that
names check_lists. None of the new keys carry archived_at or archive_batch.
…cement and login banner

Describe the new manifest keys, the features list and the check_list key on
engagement lines, and take these items off the list of what does not cross.
…ts own key

The engagement line keeps the check_list flag as the engagement API returns
it. The check list itself moves to check_list_detail (null when the
engagement has none, the lowest id when it has several).
Split three long sentences in the export docs and note that banner_enable can
be null. Tidy the tests: drop the repeated key set, key the contact and
archive checks on rows and key sets, and put a finding of another product in
a check list issue list. Keep one comment, where the serializer relies on the
prefetch order.
@svader0 svader0 changed the title feat(export): reference data and file caps in the product export feat(export): reference data, check lists, contacts, the announcement and file caps in the product export Nov 4, 2026
The three per-scanner maps are copied right after their literal
definitions, before the environment and local settings overrides.
The export uses the copies to find the rules a customer set.
…nifest

The manifest gains a deduplication key with three maps: the algorithm per
parser, the hash fields per scanner and the null-CWE switch. Each map holds
only the scan types that differ from the shipped settings or that the shipped
settings lack.
A tuple and a list with the same hash fields now count as equal, and the
manifest sends a field list as a list. The docs row uses shorter sentences,
says what the null-CWE value means and notes that a removed scan type is not
listed. The override test folds into the in-place test.
…rule

A scan type that the shipped map has and the effective map lacks goes out
with the value DefectDojo then uses: the legacy algorithm, an empty field
list, or true for the null-CWE switch. It is skipped when that value is the
shipped one.
@github-actions github-actions Bot added the settings_changes Needs changes to settings.py based on changes in settings.dist.py included in this PR label Nov 4, 2026
@defectdojo-release-tracker defectdojo-release-tracker Bot modified the milestones: 3.5.100, 3.5.200 Nov 9, 2026
The export streams test_import and test_import_actions lines after the risk acceptances when a client passes import_history=true. The manifest and the end line count them only on request, and the manifest names import_history and audit_history in features. The audit_history parameter is accepted and does nothing yet. A client that passes neither gets the same stream as before.
The export streams audit_event and legacy_audit_entry lines after the import history when a client passes audit_history=true. The events are the django-pghistory events of the product, its engagements, tests, findings, finding groups and risk acceptances, each with its context. The legacy lines are the django-auditlog entries of the same objects. The manifest and the end line count them only on request. A client that does not pass the parameter gets the same stream as before.
The audit stream read the legacy entries of all six models in one query that OR-ed six IN subqueries. Postgres cannot use the object_id index under an OR, so each export scanned the legacy table. The stream now runs one query per model and merges them by entry id, so the lines come out in the same order as before.
Audit event lines now come by model, then by object, then by event id, so the events that one transaction made on one object arrive next to each other. The audit events and the legacy audit entries are read in pages ordered by their key, one query per page, so the source database never holds a copy of a whole result.
Each page of audit events now starts at the last object id, so Postgres uses the index instead of scanning from the start.
…equest

- The manifest reference now lists the questionnaires, with their questions and choices, and the
  open questionnaire links. It also lists the answered questionnaires with no engagement and the
  benchmark types.
- A benchmark type carries its categories and requirements, and a requirement carries its
  references.
- A new product_extras parameter adds three kinds of product line. They hold tracking files, one
  benchmark line per type, and the answered questionnaires of the product's engagements.
- The manifest lists product_extras in its features. It counts the new lines only when the client
  asks. Without the parameter the stream does not change.
- The docs say that an empty finding group crosses on its test line. It counts in the manifest and
  not in the stream.
…metadata

- With product_extras=true, the stream sends a product_endpoint line for each endpoint of the
  product. With Locations on, it sends one for each location that the product references.
- A line has the uri, tags and metadata. With Locations on, it also has the status, relationship
  and metadata of the location for this product.
- The end line counts these lines in product_endpoints.
- The docs say that product_meta is a key of product, and that product_meta and finding_meta carry
  metadata.
The jira parameter adds the Jira instances to the manifest, the product projects to the header, and the project and issue links to engagement and finding lines. The instance password never crosses. The counts gain jira_projects and jira_issues.
@defectdojo-release-tracker defectdojo-release-tracker Bot modified the milestones: 3.5.200, 3.5.300 Nov 17, 2026
…GitHub and SonarQube links

Engagement lines carry the threat model file. Users carry their configuration permissions, and the manifest lists all of them. The external_issues parameter adds GitHub projects, GitHub issues and SonarQube issues. No GitHub API key crosses.
…den threat model paths

Resolve each threat model path once, treat a ValueError as missing, build the permission set once per manifest, and match app_label.codename. Tests cover a symlink out of the media root, a .. path, and a same-codename permission of another app.
…threat_model_file

The engagement model already has a threat_model boolean. The new file payload now uses threat_model_file, so the boolean is no longer overwritten.
…izer context

The manifest no longer sets state on the user serializer from outside.
… its groups

A user's configuration_permissions now list the permissions of the user and of its Django auth groups, sorted and without duplicates. The manifest prefetches the group permissions, so its query count stays flat as users and groups grow.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs settings_changes Needs changes to settings.py based on changes in settings.dist.py included in this PR unittests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant