Repository navigation
perf(tags): look up bulk-added tags by UPPER(name) so an import stops reading the whole tag table - #16978
Conversation
… reading the whole tag table
bulk_add_tag_mapping resolved existing tags with LOWER(name) IN (...), which no
index serves, so every import batch with parser tags and every tag-inheritance
pass read the whole tag table. It now joins the requested names to the tag
table on UPPER(name) = UPPER(requested), the expression tagulous's own
name__iexact lookup uses and dojo.0308's UPPER(name) index serves, and keys each
tag by the name as requested.
PostgreSQL upper-cases both sides, so a name resolves to the row tagulous would
pick even where Python's str.upper() differs from the database ("ß", letters
newer than the database's Unicode tables).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W9VHVg1dpeWAqT6ea3dLVb
Adversarial review: full tier, OpusTier: CI read, not re-run: every check green (ruff, Migration Graph Check, Rest Framework Unit Tests both variants, Performance Tests, all UI test shards, k8s, docker builds). Regression test fails without the fix: yes. I ran the PR's new tests (plus Equivalence, measured rather than read: I seeded 5,106 mixed-script tag names (ASCII, CVE ids, Break attempts (each run against the base and the head, rolled back)
Findings
Recommendation: changes required. One open Checked only statically or not at all: the 3M-row timings and plans are the author's numbers. I confirmed the index plan through the test on 5k rows, not at production scale. I did not check a database built with an ICU or |
Rows that differ only in case can exist (the unique index on name is case sensitive). bulk_add_tag_mapping resolves a requested name to the oldest of them; say so in the comment and pin it with a test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W9VHVg1dpeWAqT6ea3dLVb
|
Description
bulk_add_tag_mapping(dojo/tags/utils.py) found the existing tags it adds withwhich is
LOWER(name) IN (...). No index servesLOWER(name): not the unique index onname, and not theUPPER(name)indexdojo.0308added (#16801). So every call read the whole tag table. Its callers arebulk_apply_parser_tags(an import whose parser sets tags, once per batch) and tag inheritance (dojo/tags/inheritance.py).The lookup now joins the requested names to the tag table on the expression tagulous's own
get_or_create(name__iexact=...)uses:Each tag is keyed by the name as requested, so no Python-side case folding is involved. PostgreSQL upper-cases both sides, which matters: measured against PostgreSQL's
upper()in anen_US.utf8database over every code point, Python'sstr.upper()differs for 129 of them (the 102 letters it expands, likeß->SS, and 27 letters newer than glibc's Unicode tables, like U+019B). A Python-sidename.upper()key would match a different row than tagulous does for those.LOWERandUPPERalso disagree on letters like the dotless i: before this change,ID-TAGagainst an existingıd-tagmissed the bulk lookup and needed a per-nameget_or_createto find the tag. The case-sensitive branch (name__in) is unchanged.Test results
Measured on a finding tag table of 3,000,001 rows (
generate_seriesnames,ANALYZEd, rolled back), withdojo.0308applied:bulk_apply_parser_tagsfor 5 findings x 3 tags (1 existing, 2 new), 7 runs after a warm-up: 369.5 ms -> 7.8 ms median wall time per call; 18 statements per call before and after.EXPLAIN (ANALYZE, BUFFERS):UPPER(name)index at every size.New
BulkTagMappingNameLookupTestinunittests/test_tag_utils_bulk.py, 6 tests:idx_tagulous_finding_tags_upper_nameand no sequential scan, on a 5,000-row tag table with fresh statistics (liketest_tag_name_upper_index.py).name__iexactresolves it to, for the dotless i,ß, an ASCII case variant, and a letter newer than the database's Unicode tables. Where tagulous finds a tag, the bulk lookup finds it without a per-name fallback query.Before the change,
test_the_tag_lookup_uses_the_upper_name_indexand the dotless-i case fail (Seq Scan ... Filter: (lower((name)::text) = ANY (...)), and 2 name lookups instead of 1). After it, all 38 tests intest_tag_utils_bulk.pypass.test_tag_inheritance.pypasses with and without the change. I rantest_tag_inheritance_perf.pyandtest_tags.pyin a container where some of their tests fail for reasons unrelated to tags (the classic UI is disabled there, and its statement baselines differ); the same tests fail with and without the change, and the statement counts differ by at most the run-to-run variation (3 statements out of about 5,400).ruff check(0.16.10,ruff.toml): clean.Documentation
No user-visible change: the same tags are applied.