Repository navigation
Conversation
extras holds the Cobalt.io organization token, so the export drops it for every tool configuration through an export-only serializer.
…figurations The archive work on dev gave Engagement_Presets and Product_API_Scan_Configuration the archived_at and archive_batch fields. They describe this instance's archive, not the object, so the export does not send them.
Split the long sentences in the export docs and replace "would". Add the `error` row to the file entry table. Build the numbers in the `max_file_bytes` help text from the two cap constants and split its long sentence. Pin that an empty `file_counts` uses the default. Remove the comment on the file sort. The files of a line sort by id so the files that a full line drops are the same on every run.
6 tasks done
The engagement line replaces the check_list flag with the engagement's check list (null when it has none, the lowest id when it has several), including the finding ids of its eight issue lists. One prefetch loads them per chunk, so the export issues no query per engagement.
…ner in the manifest The manifest gains announcement and login_banner (null without a row), reference.contacts and reference.report_types, and a features list that names check_lists. None of the new keys carry archived_at or archive_batch.
…cement and login banner Describe the new manifest keys, the features list and the check_list key on engagement lines, and take these items off the list of what does not cross.
…ts own key The engagement line keeps the check_list flag as the engagement API returns it. The check list itself moves to check_list_detail (null when the engagement has none, the lowest id when it has several).
Split three long sentences in the export docs and note that banner_enable can be null. Tidy the tests: drop the repeated key set, key the contact and archive checks on rows and key sets, and put a finding of another product in a check list issue list. Keep one comment, where the serializer relies on the prefetch order.
The three per-scanner maps are copied right after their literal definitions, before the environment and local settings overrides. The export uses the copies to find the rules a customer set.
…nifest The manifest gains a deduplication key with three maps: the algorithm per parser, the hash fields per scanner and the null-CWE switch. Each map holds only the scan types that differ from the shipped settings or that the shipped settings lack.
A tuple and a list with the same hash fields now count as equal, and the manifest sends a field list as a list. The docs row uses shorter sentences, says what the null-CWE value means and notes that a removed scan type is not listed. The override test folds into the in-place test.
…rule A scan type that the shipped map has and the effective map lacks goes out with the value DefectDojo then uses: the legacy algorithm, an empty field list, or true for the null-CWE switch. It is skipped when that value is the shipped one.
The export streams test_import and test_import_actions lines after the risk acceptances when a client passes import_history=true. The manifest and the end line count them only on request, and the manifest names import_history and audit_history in features. The audit_history parameter is accepted and does nothing yet. A client that passes neither gets the same stream as before.
The export streams audit_event and legacy_audit_entry lines after the import history when a client passes audit_history=true. The events are the django-pghistory events of the product, its engagements, tests, findings, finding groups and risk acceptances, each with its context. The legacy lines are the django-auditlog entries of the same objects. The manifest and the end line count them only on request. A client that does not pass the parameter gets the same stream as before.
The audit stream read the legacy entries of all six models in one query that OR-ed six IN subqueries. Postgres cannot use the object_id index under an OR, so each export scanned the legacy table. The stream now runs one query per model and merges them by entry id, so the lines come out in the same order as before.
Audit event lines now come by model, then by object, then by event id, so the events that one transaction made on one object arrive next to each other. The audit events and the legacy audit entries are read in pages ordered by their key, one query per page, so the source database never holds a copy of a whole result.
Each page of audit events now starts at the last object id, so Postgres uses the index instead of scanning from the start.
…equest - The manifest reference now lists the questionnaires, with their questions and choices, and the open questionnaire links. It also lists the answered questionnaires with no engagement and the benchmark types. - A benchmark type carries its categories and requirements, and a requirement carries its references. - A new product_extras parameter adds three kinds of product line. They hold tracking files, one benchmark line per type, and the answered questionnaires of the product's engagements. - The manifest lists product_extras in its features. It counts the new lines only when the client asks. Without the parameter the stream does not change. - The docs say that an empty finding group crosses on its test line. It counts in the manifest and not in the stream.
…metadata - With product_extras=true, the stream sends a product_endpoint line for each endpoint of the product. With Locations on, it sends one for each location that the product references. - A line has the uri, tags and metadata. With Locations on, it also has the status, relationship and metadata of the location for this product. - The end line counts these lines in product_endpoints. - The docs say that product_meta is a key of product, and that product_meta and finding_meta carry metadata.
The jira parameter adds the Jira instances to the manifest, the product projects to the header, and the project and issue links to engagement and finding lines. The instance password never crosses. The counts gain jira_projects and jira_issues.
…GitHub and SonarQube links Engagement lines carry the threat model file. Users carry their configuration permissions, and the manifest lists all of them. The external_issues parameter adds GitHub projects, GitHub issues and SonarQube issues. No GitHub API key crosses.
…den threat model paths Resolve each threat model path once, treat a ValueError as missing, build the permission set once per manifest, and match app_label.codename. Tests cover a symlink out of the media root, a .. path, and a same-codename permission of another app.
…threat_model_file The engagement model already has a threat_model boolean. The new file payload now uses threat_model_file, so the boolean is no longer overwritten.
…izer context The manifest no longer sets state on the user serializer from outside.
… its groups A user's configuration_permissions now list the permissions of the user and of its Django auth groups, sorted and without duplicates. The manifest prefetches the group permissions, so its query count stays flat as users and groups grow.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The product export now carries everything a migration client needs beyond findings, and it stays small enough to stream.
Reference data in the manifest and the product header:
referencelists the contacts and the report types. An engagement names them inrequesterandreport_type.announcementand thelogin_banner, each null when the instance has none.check_listand addscheck_list_detail: the engagement's check list with its issue lists as finding ids, or null.featureslist holdscheck_lists, so a client knows before it reads a product that engagement lines carry the check list.archived_atandarchive_batch.deduplicationholds the scan types whose deduplication algorithm, hash fields or null-CWE switch differ from what DefectDojo ships.settings.dist.pykeeps a copy of each shipped map, taken before any override, to compare against. A scan type the instance removed goes out with its fallback.File caps:
max_file_bytesaccepts at most 104857600 (100 MiB). A larger value is a 400 that names the field.data_base64: nullanderror: "line_cap".size: null. Before, it failed the whole export.dynamically_generated, so a migration client can recreate the type with the same flags. The public API's test type serializer does not change.file_counts=false. Then it skips the storage stat for each file and returnsfiles_over_cap: null. With the default, it makes one storage call per file instead of two.Import and audit history
import_historyandaudit_history. Both default to false. Without them, the stream does not change, and the manifest only adds the two names tofeatures.import_history=true, the stream sends each test's imports astest_importlines. Each import is followed bytest_import_actionslines of at most 10,000 actions.audit_history=true, the stream sends the audit events of the product and its engagements, tests, findings, finding groups and risk acceptances asaudit_eventlines. They come by model, then object, then event id.audit_history=true, the older audit log entries come aslegacy_audit_entrylines, by id.test_imports,test_import_actions,audit_eventsandlegacy_audit_entries.Questionnaires, benchmarks, tracking files and endpoints
referencenow listsquestionnaireswith their questions and choices, andgeneral_surveys(the open questionnaire links).open_survey_responses(answered questionnaires with no engagement) andbenchmark_typeswith their categories and requirements. A requirement carries its references.product_extras, defaults to false. Without it, the stream does not change. The manifest always namesproduct_extrasinfeatures.product_extras=true, the stream sends four new kinds of product line:tracking_file,benchmark,answered_surveyandproduct_endpoint. They come after the risk acceptances and before the history.benchmarkline holds one benchmark type of the product, with its results, notes and summary.product_endpointline holds the URI, tags and metadata of each endpoint of the product. With Locations on, it holds each location that the product references, with the status, relationship and metadata for this product.tracking_files,benchmarksandanswered_surveysonly when the client asks.product_metais a key ofproduct.Jira links
jira, defaults to false. Without it, the stream does not change. The manifest always namesjirainfeatures.jira=true, the manifest'sreferencegainsjira_instances. An instance row carries every field exceptpassword.jira_projects, the product's own Jira projects. An engagement line gainsjira_projectandjira_issue, each an object or null.jira_issue, an object or null. A finding group's issue already rides infinding_groups[].jira_issue.id,jira_id,jira_key,jira_creation,jira_change,jira_project,finding,engagementandfinding_group.jira_projectsandjira_issues, only withjira=true. They count entries inside lines.Threat models, configuration permissions, GitHub and SonarQube links
threat_model_file: null, or the file withfilename,size,content_typeanddata_base64. It follows the rules of every other file, withmax_file_bytesand theline_caperror."error": "missing".filesandfiles_over_cap. Thethreat_modelboolean andtmodel_pathstay on the line.reference.userscarriesconfiguration_permissions: a sorted list ofapp_label.codenamestrings. It holds the configuration permissions of the user and of its groups.reference.configuration_permissionslists every configuration permission of the instance, in the same form. A client uses it to give a staff user the same powers in the new instance.external_issues, defaults to false. Without it, the stream does not change. The manifest always namesexternal_issuesinfeatures.external_issues=true, the manifest'sreferencegainsgithub_configurations, withidandconfiguration_name. The GitHub API key never goes out.github_projects. A finding line gainsgithub_issue, null or an object withissue_idandissue_url.sonarqube_issuebecomes an object withkey,status,typeand its transitions in id order. Without the parameter, it stays the integer id.github_projects,github_issuesandsonarqube_issues, only withexternal_issues=true.Test results
python3 manage.py test unittests.test_apiv2_export unittests.test_apiv2_export_performance unittests.test_configuration_default_checks --keepdb: 94 tests, OK, withunittests.test_configuration_default_checks, on this branch on the latestdev.ruff check --config ruff.tomlon the changed files: clean.unittests.test_apiv2_exportandunittests.test_apiv2_export_performance: 123 tests, OK.ruff check --config ruff.tomlis clean.unittests.test_apiv2_exportandunittests.test_apiv2_export_performance: 156 tests, OK, with Locations off and again with Locations on. Each mode skips the one test for the other mode.ruff check --config ruff.tomlis clean.unittests.test_apiv2_exportandunittests.test_apiv2_export_performance: 165 tests, OK, with Locations off and again with Locations on.ruff check --config ruff.tomlis clean.ruff check --config ruff.tomlis clean.Documentation
docs/content/automation/api/product_export.mddescribes every new key, both caps,file_counts, the check list anddeduplication. It also describes theimport_history,audit_history,product_extras,jiraandexternal_issuesparameters, with their lines and keys. It describesthreat_model_fileand the configuration permission keys too.Checklist
dev(merged on 2026-11-04).devbranch.