Skip to content

perf(tags): look up bulk-added tags by UPPER(name) so an import stops reading the whole tag table - #16978

Merged
Maffooch merged 2 commits into
devfrom
perf/tag-mapping-upper-lookup
Nov 18, 2026
Merged

Maffooch merged 2 commits into
devfrom
perf/tag-mapping-upper-lookup

Conversation

@Maffooch

Copy link
Copy Markdown
Contributor

Description

bulk_add_tag_mapping (dojo/tags/utils.py) found the existing tags it adds with

tag_model.objects.annotate(name_lower=Lower("name")).filter(name_lower__in=[...])

which is LOWER(name) IN (...). No index serves LOWER(name): not the unique index on name, and not the UPPER(name) index dojo.0308 added (#16801). So every call read the whole tag table. Its callers are bulk_apply_parser_tags (an import whose parser sets tags, once per batch) and tag inheritance (dojo/tags/inheritance.py).

The lookup now joins the requested names to the tag table on the expression tagulous's own get_or_create(name__iexact=...) uses:

SELECT t.*, requested.name AS requested_name
FROM unnest(%s::text[]) AS requested(name)
JOIN "dojo_tagulous_finding_tags" t ON UPPER(t."name"::text) = UPPER(requested.name)
ORDER BY t.id

Each tag is keyed by the name as requested, so no Python-side case folding is involved. PostgreSQL upper-cases both sides, which matters: measured against PostgreSQL's upper() in an en_US.utf8 database over every code point, Python's str.upper() differs for 129 of them (the 102 letters it expands, like ß -> SS, and 27 letters newer than glibc's Unicode tables, like U+019B). A Python-side name.upper() key would match a different row than tagulous does for those. LOWER and UPPER also disagree on letters like the dotless i: before this change, ID-TAG against an existing ıd-tag missed the bulk lookup and needed a per-name get_or_create to find the tag. The case-sensitive branch (name__in) is unchanged.

Test results

Measured on a finding tag table of 3,000,001 rows (generate_series names, ANALYZEd, rolled back), with dojo.0308 applied:

  • bulk_apply_parser_tags for 5 findings x 3 tags (1 existing, 2 new), 7 runs after a warm-up: 369.5 ms -> 7.8 ms median wall time per call; 18 statements per call before and after.
  • The lookup itself, EXPLAIN (ANALYZE, BUFFERS):
    old: Parallel Seq Scan on dojo_tagulous_finding_tags  Filter: (lower((name)::text) = ANY (...))  Rows Removed by Filter: 1000004 (x3)  Buffers: shared hit=25116  Execution Time: 356.916 ms
    new: Nested Loop -> Function Scan on unnest requested -> Index Scan using idx_tagulous_finding_tags_upper_name  Index Cond: (upper((name)::text) = upper(requested.name))  Buffers: shared hit=12  Execution Time: 0.052 ms
    
  • By batch size (half the names existing), old -> new execution time: 10 names 345 ms -> 0.11 ms; 100 names 1,283 ms -> 0.61 ms; 1,000 names 1,210 ms -> 4.5 ms; 5,000 names 1,230 ms -> 20.6 ms. The new plan stays on the UPPER(name) index at every size.

New BulkTagMappingNameLookupTest in unittests/test_tag_utils_bulk.py, 6 tests:

  • The lookup an import issues plans an index scan on idx_tagulous_finding_tags_upper_name and no sequential scan, on a 5,000-row tag table with fresh statistics (like test_tag_name_upper_index.py).
  • A requested name resolves to the same row name__iexact resolves it to, for the dotless i, ß, an ASCII case variant, and a letter newer than the database's Unicode tables. Where tagulous finds a tag, the bulk lookup finds it without a per-name fallback query.
  • Two case variants in one batch share the existing tag, with one lookup.

Before the change, test_the_tag_lookup_uses_the_upper_name_index and the dotless-i case fail (Seq Scan ... Filter: (lower((name)::text) = ANY (...)), and 2 name lookups instead of 1). After it, all 38 tests in test_tag_utils_bulk.py pass. test_tag_inheritance.py passes with and without the change. I ran test_tag_inheritance_perf.py and test_tags.py in a container where some of their tests fail for reasons unrelated to tags (the classic UI is disabled there, and its statement baselines differ); the same tests fail with and without the change, and the statement counts differ by at most the run-to-run variation (3 statements out of about 5,400). ruff check (0.16.10, ruff.toml): clean.

Documentation

No user-visible change: the same tags are applied.

… reading the whole tag table

bulk_add_tag_mapping resolved existing tags with LOWER(name) IN (...), which no
index serves, so every import batch with parser tags and every tag-inheritance
pass read the whole tag table. It now joins the requested names to the tag
table on UPPER(name) = UPPER(requested), the expression tagulous's own
name__iexact lookup uses and dojo.0308's UPPER(name) index serves, and keys each
tag by the name as requested.

PostgreSQL upper-cases both sides, so a name resolves to the row tagulous would
pick even where Python's str.upper() differs from the database ("ß", letters
newer than the database's Unicode tables).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W9VHVg1dpeWAqT6ea3dLVb
@Maffooch Maffooch added this to the 3.5.300 milestone Nov 17, 2026
@Maffooch Maffooch added performance performance improvement or bug report unittests labels Nov 17, 2026
@Maffooch

Copy link
Copy Markdown
Contributor Author

Adversarial review: full tier, Opus

Tier: full (a backend query rewrite on the import and tag-inheritance path; performance-sensitive). Reviewed at 5dafda3, base dev (merge base e56c039). No UI change, so no harness or screenshots apply.

CI read, not re-run: every check green (ruff, Migration Graph Check, Rest Framework Unit Tests both variants, Performance Tests, all UI test shards, k8s, docker builds).

Regression test fails without the fix: yes. I ran the PR's new tests (plus BulkTagMappingTest and BulkApplyParserTagsTest) with bulk_add_tag_mapping swapped for the merge-base version, without touching the tree. 2 fail on the base: test_the_tag_lookup_uses_the_upper_name_index (Seq Scan ... Filter: (lower((name)::text) = ANY (...))) and the dotless_i case (2 name lookups instead of 1). On the head, all 36 tests that can run in my container pass (BulkTagUtilsInheritanceTest cannot load its fixture there because watson is not installed. CI ran it green). The other three parameterized name cases pass on both sides by design: they pin equivalence with name__iexact, not the bug.

Equivalence, measured rather than read: I seeded 5,106 mixed-script tag names (ASCII, CVE ids, Straße, İstanbul, ıd, Greek final sigma, Dž, ligature fi, CJK) and requested each name swapcased, upper-cased and lower-cased (11,584 names). On 1,200 sampled names, the new lookup returns the same row as tagulous's own name__iexact every time (0 differences). The old LOWER(...) IN lookup missed 53 of them (İSTANBUL-*, ID-*, σίσυφος-*), which then fell back to a per-name get_or_create. All of it ran in a transaction that was rolled back.

Break attempts (each run against the base and the head, rolled back)

  1. Legacy rows that differ only in case (Network and network both stored, which the case-sensitive unique index allows). Base: whichever row the dict comprehension saw last in ORDER BY name collation order. Head: the lowest id. With Network older, both pick Network. With network older and Network requested, the base applied Network and the head applies network. The head is deterministic, but it now resolves a different row than before, and nothing documents or tests which row wins (R1).
  2. One finding lists the same tag twice, or in two cases: ["owasp","owasp"] gives count 2 and 1 through row. ["Network","network","NETWORK"] gives count 3 and 1 row. Identical on base and head, so this is pre-existing (R2).
  3. Scale: 1,000 existing names against a 60k-row tag table, ANALYZEd. The whole call took 309 ms on the head and 327 ms on the base. At this size the through-row work dominates, and the head is not slower. The PR's own EXPLAIN gives the 3M-row lookup numbers, and its test pins the index plan.
  4. Names that stress array adaptation: a,b, {brace}, "quoted", back\slash, NULL, it's, embedded tab, trailing space, emoji. Each was stored upper-cased and requested as written. Head and base both matched all 10 existing rows and created no new ones.
  5. Another tag table: bulk_add_tag_mapping({"INHERIT-ME": [eng], "brand-new": [eng]}, tag_field_name="inherited_tags") on Engagement reuses the existing Inherit-Me and creates the new one. The table-name quoting works off the Finding table too. dojo.0308 indexes all 14 tag tables, so every caller (inheritance.py, endpoint/utils.py, migrate_endpoints_to_locations) gets the index.

Findings

  • R1 nit this-pr dojo/tags/utils.py:383: when legacy rows differ only in case, the lookup now resolves to the lowest id (ORDER BY t.id + setdefault). The base picked the last row in name-collation order, so the applied tag can change (attempt 1). Neither the comment nor a test says which row wins. Right: one sentence in the comment ("several rows matching one name resolve to the oldest") and a test with two stored case variants that asserts the oldest is applied.
  • R2 suggestion pre-existing dojo/tags/utils.py:419: a tag requested twice for one instance in one call (the same name, or two cases of it) adds one through row but increments count once per request, because pairs queued in this call are never added to existing_pairs. Issue: DefectDojo-Inc/dojo-pro#8157

Recommendation: changes required. One open this-pr finding (1 nit). R2 is pre-existing and tracked in its issue.

Checked only statically or not at all: the 3M-row timings and plans are the author's numbers. I confirmed the index plan through the test on 5k rows, not at production scale. I did not check a database built with an ICU or builtin collation provider: there upper('ß') is SS, and the parameterized test is written to accept either outcome.

Rows that differ only in case can exist (the unique index on name is case
sensitive). bulk_add_tag_mapping resolves a requested name to the oldest of
them; say so in the comment and pin it with a test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W9VHVg1dpeWAqT6ea3dLVb
@Maffooch

Copy link
Copy Markdown
Contributor Author
  • R1 fixed in ede72cb: the lookup's comment now says that a name matching several stored tags that differ only in case resolves to the oldest of them. The new test_stored_case_variants_resolve_to_the_oldest_tag stores network, Network and NETWORK, requests Network, and asserts that the oldest tag is applied with a count of 1. It fails on the base, which applies the newest row ([80642] != [80644]), and passes on the head. test_tag_utils_bulk.py 39/39 OK. ruff check clean.
  • R2 issue DefectDojo-Inc/dojo-pro#8157 (pre-existing, assigned to the author)

@Maffooch
Maffooch added this pull request to the merge queue Nov 18, 2026
Merged via the queue into dev with commit 51604fb Nov 18, 2026
49 checks passed
@Maffooch
Maffooch deleted the perf/tag-mapping-upper-lookup branch November 18, 2026 01:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance performance improvement or bug report unittests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants