Skip to content

feat(export): cap inline file data per line and let the manifest skip file stats - #16681

Closed
svader0 wants to merge 4 commits into
devfrom
feat/export-file-caps
Closed

svader0 wants to merge 4 commits into
devfrom
feat/export-file-caps

Conversation

@svader0

@svader0 svader0 commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

Description

Two limits keep a product export small enough to stream and cheap to plan.

  • max_file_bytes now accepts at most 104857600 (100 MiB). A larger value is a 400 that names the field.
  • One exported line (an engagement, a test, a finding, or a risk acceptance's proof) carries at most 48 MiB of base64 file data. The files fill that budget in id order. A file past the budget keeps its size and goes out with data_base64: null and error: "line_cap". The manifest's files_over_cap does not count it, because it is not over the file size cap.
  • The manifest takes file_counts=false. Then it skips the storage stat for each file, still counts the files, and returns files_over_cap: null. With the default, the manifest makes one storage call per file instead of two.

Test results

  • python3 manage.py test unittests.test_apiv2_export unittests.test_apiv2_export_performance --keepdb: 59 tests, OK, on this branch on the latest dev.
  • ruff check --config ruff.toml on the changed files: clean.
  • Not tested: S3 or other remote storage. A backend that raises something other than FileNotFoundError for a missing object fails the call instead of marking the file missing.

Documentation

docs/content/automation/api/product_export.md describes both caps, the line_cap error and file_counts.

Checklist

  • Rebased against the latest dev.
  • Submitted against the dev branch.
  • Ruff compliant.
  • Documentation updated in docs/content/automation/api/product_export.md.
  • No model change, so no migration.
  • Unit tests added.

Split the long sentences in the export docs and replace "would". Add the
`error` row to the file entry table. Build the numbers in the
`max_file_bytes` help text from the two cap constants and split its long
sentence. Pin that an empty `file_counts` uses the default.

Remove the comment on the file sort. The files of a line sort by id so the
files that a full line drops are the same on every run.
@svader0

svader0 commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Folded into #16680, so both export changes review as one PR. Its commits are all in #16680. The branch stays.

@svader0 svader0 closed this Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant