Skip to content

fix(finding): lock the finding before found_by on the v3 create and the v2/v3 updates - #16987

Open
blakeaowens wants to merge 2 commits into
devfrom
fix/finding-found-by-lock-other-writers
Open

blakeaowens wants to merge 2 commits into
devfrom
fix/finding-found-by-lock-other-writers

Conversation

@blakeaowens

Copy link
Copy Markdown
Contributor

Description

The bulk-delete found_by deadlock fixed for POST /api/v2/findings/ (#16950) is still there on the other writers that replace an existing finding's found_by: the v3 create (services.create_finding), the v3 update (services.update_finding, PATCH and PUT) and the v2 update (FindingSerializer.update, PATCH and PUT).

Each runs found_by.set() in its own transaction (ATOMIC_REQUESTS is off). set() deletes the rows that were not requested and inserts the requested ones, and both stay locked until COMMIT, where Postgres checks the new rows' deferred foreign key with FOR KEY SHARE on the finding. The chunked bulk delete (the excess-duplicate delete, a test or engagement delete) takes the same two locks the other way round: lock_findings_for_delete holds the finding FOR UPDATE, then bulk_clear_finding_m2m deletes its found_by rows. Each waits for the other and Postgres aborts one with deadlock detected (40P01).

This change sends those three writers through set_found_by_locking_finding_first, the helper the v2 create already uses, which locks the finding FOR KEY SHARE before writing found_by, the order the delete uses. The helper moves from dojo/finding/api/serializer.py to dojo/finding/helper.py (next to lock_findings_for_delete) so dojo/finding/services.py can use it without importing the API serializer; the serializer imports it from there, so its name in that module is unchanged.

clear() (a writer sent found_by: []) is left as it is. It only deletes rows, so its COMMIT checks no foreign key and never waits on the finding, and it was measured not to deadlock (below).

Measured on a Postgres-backed stack:

  • Reproduction. Two connections. Connection A runs the writer as production runs it (autocommit, no request transaction) and pauses inside its found_by write; connection B runs the delete's order (SELECT ... FOR UPDATE on the finding, then DELETE of its found_by rows). 3 rounds per writer:

    writer before after
    v2 create (control, fixed by fix(finding): lock the finding before writing found_by on create so a bulk delete cannot deadlock it #16950) 0/3 0/3
    v3 create 3/3 deadlocked 0/3
    v2 update, set() 3/3 deadlocked 0/3
    v3 update, set() 3/3 deadlocked 0/3
    v2 update, clear() 0/3 0/3
    v3 update, clear() 0/3 0/3

    After the change every write returns 200/201 (or completes, for the services), and the delete waits for the writer instead of deadlocking.

  • Cost. One extra query per found_by write: v2 update 47 to 48, v3 update 25 to 26, v3 create 39 to 40. The added statement is a primary-key index scan: 0.066 ms execution, 0.40 ms median round trip (p99 0.58 ms, n=1000) on the stack's table, and 0.042 ms execution, 0.28 ms median round trip (p99 0.50 ms) on a 2,000,000-row table of the same shape. Per-request medians before and after overlap: on the shared host the same unchanged code varied more between rounds (62 to 215 ms for the v3 update) than any before/after difference.

Test results

  • unittests/test_finding_create_found_by_lock_order.py: new TestFindingFoundByWritersLockOrder, one test per writer (v3 create, v3 update, v2 PATCH). Each pins that the finding's FOR KEY SHARE lock comes before set()'s dojo_finding_found_by delete and insert, and that found_by ends up as requested. On dev all three fail (no lock before the write); with this change all 5 tests in the file pass. The found_by-write matcher now also finds a write that pghistory prefixes with its context inside a request.
  • unittests/api_v3 (finding writes, findings, findings query counts, query report, conflict, examples): pass.
  • unittests/test_dedupe_delete_commit_race: the same tests fail with and without this change in the local container used here (environmental), identical failure set; CI runs the module in its own image.
  • ruff check (pinned 0.16.10) passes on the whole tree.

Documentation

No user-visible change; no documentation needed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HJqD5HVJcHDMyS8pibV7Tg

…he v2/v3 updates

The v3 create, the v3 update and the v2 update replaced an existing
finding's found_by with set() without locking the finding first, so each
deadlocked (40P01) against a bulk delete exactly as the v2 create did
before the found_by lock order fix. Route them through
set_found_by_locking_finding_first, moved to dojo/finding/helper.py so the
v3 services can use it without importing the API serializer.

clear() is unchanged: it only deletes rows, so its COMMIT checks no foreign
key and never waits on the finding.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJqD5HVJcHDMyS8pibV7Tg
@blakeaowens blakeaowens added this to the 3.5.300 milestone Nov 18, 2026
Review follow-up: the helper's docstring now says it covers the API
writers (the v2 and v3 create and update), not every found_by.set() caller,
and the lock-order test describes the bug in place instead of citing a
tracker readers of this repository cannot open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJqD5HVJcHDMyS8pibV7Tg

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant