Skip to content

fix(migrations): wait for a 0305 index build to commit before judging it invalid - #16981

Merged
blakeaowens merged 1 commit into
devfrom
fix/migration-0305-wait-for-index-commit
Nov 18, 2026
Merged

blakeaowens merged 1 commit into
devfrom
fix/migration-0305-wait-for-index-commit

Conversation

@blakeaowens

@blakeaowens blakeaowens commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Description

AddIndexConcurrentlyIfMissing in migration 0305_archive_batch_indexes waits while pg_stat_progress_create_index shows an earlier build of its index, then reads pg_index.indisvalid and drops and rebuilds the index when it is not valid. Postgres ends a concurrent build's progress row just before the build's last transaction commits, and that transaction is the one that sets indisvalid. In that window the wait sees no build, indisvalid still reads false, and the migration drops an index that was about to become valid. Two outcomes were measured:

  • the in-flight index is dropped and built again (a whole extra concurrent build, which is what the wait exists to avoid, and costly on a large dojo_finding);
  • the DROP INDEX CONCURRENTLY collides with the build's uncommitted pg_index update and fails with tuple concurrently updated, so the migration errors out.

Fix: a build now counts as in progress until its transaction commits. Besides the progress row, _build_in_progress also checks whether the index's pg_index row has an uncommitted update: its xmax names a transaction that still holds the lock on its own id in pg_locks (every transaction holds that lock until it ends, and pg_locks is readable without extra privileges). The existing wait loop then covers the window. Migration state is unchanged.

Test results

New test test_build_ending_is_waited_for_until_it_commits in unittests/test_migration_0305_archive_batch_indexes.py. It holds a real CREATE INDEX CONCURRENTLY in the window with no timing: a ddl_command_end event trigger (it runs after the progress row ends and before the commit) waits on an advisory lock the test holds. The test checks that state directly (no progress row, indisvalid false), then runs the operation and releases the lock from the operation's first sleep. A DROP INDEX is refused with an assertion instead of being run, so a regression fails fast instead of hanging. Creating an event trigger needs a superuser, which the unit-test compose database is; the test skips otherwise, as test_bulk_risk_acceptance_api already does.

Measured on PostgreSQL 16.9. The adversarial review also held the build in the commit window on PostgreSQL 18.6 (the compose image) and 16.16: the old check reports no build, the new one reports it in progress.

0305 from dev this branch
new test, 3 runs 3 failed: dropped an index whose build had not committed yet: DROP INDEX CONCURRENTLY IF EXISTS "dojo_test_archive_batch" passes
the timing repro from the report (snapshot released inside the wait loop's first check, real check returned at once), 20 runs 8 OK, 8 failed with a changed index OID (dropped and rebuilt), 4 errored with tuple concurrently updated 20 OK
whole test_migration_0305_archive_batch_indexes module as a superuser, 10 runs 10 x OK (5 tests)
same module as a non-superuser, fresh test database OK (4 passed, new test skipped)
  • makemigrations dojo --check --dry-run: No changes detected.
  • A fresh test database migrated through 0305 with this change, as a superuser and as a non-superuser: OK (pg_locks needs no extra privilege).
  • ruff check (0.16.10) on the test file: clean (dojo/db_migrations is excluded from ruff).

Related: migration 0307_event_history_lookup_indexes, already on dev, has its own copy of this wait (_build_in_progress checks only pg_stat_progress_create_index), so it can drop an index whose build has not committed yet in the same way; it is tracked separately and is not changed here. #16971 also copies this class into other concurrent-index migrations; those copies carry the same wait and would need the same check.

Documentation

None needed: no user-facing change.

🤖 Generated with Claude Code

https://claude.ai/code/session_0143hYTDGiy48mntt7T9N2uT

… it invalid

Postgres ends a concurrent build's pg_stat_progress_create_index row just
before the build's last transaction (the one that sets indisvalid) commits.
In that window 0305 saw no build in progress, read indisvalid as false and
dropped the index, either rebuilding it or failing with "tuple concurrently
updated". A build now counts as in progress while its pg_index row has an
uncommitted update (xmax held by an open transaction in pg_locks).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0143hYTDGiy48mntt7T9N2uT
@blakeaowens blakeaowens added this to the 3.5.300 milestone Nov 18, 2026
@github-actions github-actions Bot added the New Migration Adding a new migration file. Take care when merging. label Nov 18, 2026
@blakeaowens

Copy link
Copy Markdown
Contributor Author

Adversarial review: full tier, Opus

Tier: full (migration logic change). Fresh session, no author context. Base dev, head ab2895a5c6.

CI read, not re-run: every check green, including Rest Framework Unit Tests (both), User Interface Tests, k8s deploy, Migration Graph Check, ruff, Unit Tests Complete. The CI unit-test database user is the compose POSTGRES_USER, so it is a superuser and the new test runs there rather than skipping.

Regression test, failing first (Django test runner in the Pro dojo container, against a private PostgreSQL 16.16 container with a superuser so the new test does not skip):

  • PR head: Ran 5 tests ... OK, 4 runs out of 4.
  • 0305_archive_batch_indexes.py taken back to origin/dev: FAILED (failures=1), test_build_ending_is_waited_for_until_it_commits fails with AssertionError: dropped an index whose build had not committed yet: DROP INDEX CONCURRENTLY IF EXISTS "dojo_test_archive_batch". The other 4 still pass. Restored, passes again.
  • On the shared non-superuser stack database (PostgreSQL 16.9) the new test skips, as described.

Read: the docstring's claim about the order of events checks out against the observed catalog state (below): the progress row is gone, indisvalid is false and pg_index.xmax names the build's still-open transaction. The test holds the window with an advisory lock in a ddl_command_end event trigger scoped by application_name, refuses the DROP INDEX rather than running it (so a regression fails instead of hanging), and its cleanups unwind in a safe LIFO order (unlock, join builder, drop trigger, drop function, close, restore index). It cannot pass on the base, as measured above.

Break attempts

  1. The commit window on both supported majors. Held a real CREATE INDEX CONCURRENTLY (200k rows) in the window on PostgreSQL 16.16 and 18.6 (the compose image). Old query: no build. New query: build in progress, indisvalid false, xmax = the builder's xid. After release: valid, xmax 0, new query false. Held.
  2. A non-superuser observer (a plain LOGIN role with no pg_read_all_stats) watching a superuser's build in the window: the new query still returns true on 16 and 18, since pg_locks and pg_index are readable by any role. Held.
  3. The build dies in the window (pg_cancel_backend on the builder while it is held): the new query turns false at once, the index stays INVALID, so the operation goes on to drop and rebuild the leftover instead of waiting forever. Held.
  4. False waits on a valid index from unrelated transactions. An idle transaction after ALTER TABLE ... CLUSTER ON <index>, and one that only read the table: neither sets the row's xmax, the new query stays false, the valid index is kept at once. A superuser holding SELECT ... FROM pg_index ... FOR KEY SHARE does make it wait (single locker), and two such lockers turn xmax into a MultiXactId the join does not match; both need superuser row locks on a catalog during a migration, so not a finding.
  5. Other copies of the same check. 0307_event_history_lookup_indexes.py on dev already carries the identical progress-row-only _build_in_progress and the same indisvalid then DROP INDEX CONCURRENTLY sequence, so it has the same window; attempt 1's "old query" result is exactly what it sees. The PR body names only fix(migrations): let every concurrent index migration re-run after an interrupted build #16971. Broke: R1, R2.

Performance: one pg_locks scan per check (once per operation, then every 5 s only while waiting); negligible at any table size.

No screenshots: no user-visible surface.

Findings

  • R1 suggestion pre-existing dojo/db_migrations/0307_event_history_lookup_indexes.py:169: 0307's _build_in_progress checks only pg_stat_progress_create_index, so it can drop an index whose concurrent build has not committed yet, the same bug this PR fixes in 0305; it needs the same pg_index.xmax/pg_locks check and a test. Issue: DefectDojo-Inc/dojo-pro#8190
  • R2 nit this-pr PR body: the "Related" line says the same check would be needed in fix(migrations): let every concurrent index migration re-run after an interrupted build #16971's copies but misses that 0307 is already on dev with it; name 0307 and its issue (DefectDojo-Inc/dojo-pro#8190) there, and say the fix was also observed on PostgreSQL 18 (the compose image) since the table only lists 16.9.

Recommendation

changes required: 1 this-pr nit open (R2, PR body only). The code change itself held under every attempt; R1 is pre-existing and tracked in its issue.

Not checked: a production-scale table (the window is independent of size, the waits were measured on 200k rows); PostgreSQL 17 (16 and 18 behave identically here).

🤖 Generated with Claude Code

@blakeaowens

Copy link
Copy Markdown
Contributor Author

Resolution of the adversarial review (verdict 1, head ab2895a):

  • R1 issue (pre-existing, filed by the review and assigned to the PR's author): migration 0307_event_history_lookup_indexes has the same progress-row-only wait. It is outside this PR's diff, so it is tracked there and not changed here; the PR description now names it.
  • R2 fixed in ab2895a (PR description only, no code change): the "Related" line now names 0307_event_history_lookup_indexes as already on dev with the same wait, alongside fix(migrations): let every concurrent index migration re-run after an interrupted build #16971's copies, and the Test results section notes the fix was also exercised on PostgreSQL 18.6 (the compose image) and 16.16 by the review. The private tracker reference is left out deliberately: this repository is public, so its PRs do not link private issues.

@blakeaowens
blakeaowens added this pull request to the merge queue Nov 18, 2026
Merged via the queue into dev with commit 5a7c2ce Nov 18, 2026
48 checks passed
@blakeaowens
blakeaowens deleted the fix/migration-0305-wait-for-index-commit branch November 18, 2026 07:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix New Migration Adding a new migration file. Take care when merging. unittests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants