Repository navigation
fix(finding): lock the finding before found_by on the v3 create and the v2/v3 updates - #16987
Open
blakeaowens wants to merge 2 commits into
Open
blakeaowens wants to merge 2 commits into
blakeaowens wants to merge 2 commits into
Conversation
…he v2/v3 updates The v3 create, the v3 update and the v2 update replaced an existing finding's found_by with set() without locking the finding first, so each deadlocked (40P01) against a bulk delete exactly as the v2 create did before the found_by lock order fix. Route them through set_found_by_locking_finding_first, moved to dojo/finding/helper.py so the v3 services can use it without importing the API serializer. clear() is unchanged: it only deletes rows, so its COMMIT checks no foreign key and never waits on the finding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJqD5HVJcHDMyS8pibV7Tg
Review follow-up: the helper's docstring now says it covers the API writers (the v2 and v3 create and update), not every found_by.set() caller, and the lock-order test describes the bug in place instead of citing a tracker readers of this repository cannot open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJqD5HVJcHDMyS8pibV7Tg
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The bulk-delete
found_bydeadlock fixed forPOST /api/v2/findings/(#16950) is still there on the other writers that replace an existing finding'sfound_by: the v3 create (services.create_finding), the v3 update (services.update_finding, PATCH and PUT) and the v2 update (FindingSerializer.update, PATCH and PUT).Each runs
found_by.set()in its own transaction (ATOMIC_REQUESTSis off).set()deletes the rows that were not requested and inserts the requested ones, and both stay locked until COMMIT, where Postgres checks the new rows' deferred foreign key withFOR KEY SHAREon the finding. The chunked bulk delete (the excess-duplicate delete, a test or engagement delete) takes the same two locks the other way round:lock_findings_for_deleteholds the findingFOR UPDATE, thenbulk_clear_finding_m2mdeletes itsfound_byrows. Each waits for the other and Postgres aborts one withdeadlock detected(40P01).This change sends those three writers through
set_found_by_locking_finding_first, the helper the v2 create already uses, which locks the findingFOR KEY SHAREbefore writingfound_by, the order the delete uses. The helper moves fromdojo/finding/api/serializer.pytodojo/finding/helper.py(next tolock_findings_for_delete) sodojo/finding/services.pycan use it without importing the API serializer; the serializer imports it from there, so its name in that module is unchanged.clear()(a writer sentfound_by: []) is left as it is. It only deletes rows, so its COMMIT checks no foreign key and never waits on the finding, and it was measured not to deadlock (below).Measured on a Postgres-backed stack:
Reproduction. Two connections. Connection A runs the writer as production runs it (autocommit, no request transaction) and pauses inside its
found_bywrite; connection B runs the delete's order (SELECT ... FOR UPDATEon the finding, thenDELETEof itsfound_byrows). 3 rounds per writer:set()set()clear()clear()After the change every write returns 200/201 (or completes, for the services), and the delete waits for the writer instead of deadlocking.
Cost. One extra query per
found_bywrite: v2 update 47 to 48, v3 update 25 to 26, v3 create 39 to 40. The added statement is a primary-key index scan: 0.066 ms execution, 0.40 ms median round trip (p99 0.58 ms, n=1000) on the stack's table, and 0.042 ms execution, 0.28 ms median round trip (p99 0.50 ms) on a 2,000,000-row table of the same shape. Per-request medians before and after overlap: on the shared host the same unchanged code varied more between rounds (62 to 215 ms for the v3 update) than any before/after difference.Test results
unittests/test_finding_create_found_by_lock_order.py: newTestFindingFoundByWritersLockOrder, one test per writer (v3 create, v3 update, v2 PATCH). Each pins that the finding'sFOR KEY SHARElock comes beforeset()'sdojo_finding_found_bydelete and insert, and thatfound_byends up as requested. Ondevall three fail (no lock before the write); with this change all 5 tests in the file pass. Thefound_by-write matcher now also finds a write that pghistory prefixes with its context inside a request.unittests/api_v3(finding writes, findings, findings query counts, query report, conflict, examples): pass.unittests/test_dedupe_delete_commit_race: the same tests fail with and without this change in the local container used here (environmental), identical failure set; CI runs the module in its own image.ruff check(pinned 0.16.10) passes on the whole tree.Documentation
No user-visible change; no documentation needed.
🤖 Generated with Claude Code
https://claude.ai/code/session_01HJqD5HVJcHDMyS8pibV7Tg