Skip to content

fix(cascade): seed a delete with its keys when its restriction is another query, and keep a % in it - #1611

Open
dimitri-yatsenko wants to merge 3 commits into
fix/cascade-overdeletefrom
fix/cascade-subquery-seed
Open

dimitri-yatsenko wants to merge 3 commits into
fix/cascade-overdeletefrom
fix/cascade-subquery-seed

Conversation

@dimitri-yatsenko

@dimitri-yatsenko dimitri-yatsenko commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

A delete restricted by another query, such as
(Subject & (Session & "quality = 0")).delete(), failed. The cascade
applies the seed's compiled conditions again to a fresh copy of the table
through restrict(), which collects column names from the whole condition
text, subquery included, and rejects every name the seed table lacks. The
delete raised Attribute ... is not found in query before deleting
anything. On MySQL the backquoted schema name in every subquery tripped the
same check, so even (Subject & Session).delete() raised.

On PostgreSQL, a seed whose subquery names only the seed's columns got past
the check but still did nothing. The cascade deletes children before
parents, so by the time it reached the seed, the rows its subquery reads
were gone. (Subject & Session).delete() matched no subject, returned 0
and rolled back.

A seed whose conditions contain a subquery is now replaced by the primary
keys it selects, as a dj.Top seed already is. The traversal and the
delete then work from the rows the restriction selected when the delete
started.

The new tests delete subjects restricted by a query on their sessions, by
its negation, and by the session table itself. They check the preview
counts and the rows left afterwards, on both backends. All six cases fail
on the previous code.

This PR is stacked on #1592 (it changes the seed helper that PR adds), so its base is fix/cascade-overdelete; it moves to master when #1592 merges.

Closes #1610.

Second commit: keep a % in a delete's restriction (#1636)

A delete restricted by a condition containing %, such as (Label & "name = '50%'").delete() or a dict {"name": "50%"}, deleted nothing and returned 0. The seed's conditions are compiled SQL, in which each % is already escaped as %%; the cascade applies them again through restrict(), which escaped every % a second time. The seed's conditions now have that first escape undone before the cascade applies them again. The new test checks a string condition, a dict and a condition holding %%, on both backends, and fails before this change.

Closes #1636.

@dimitri-yatsenko dimitri-yatsenko added this to the v2.3.5 milestone Oct 17, 2026
@dimitri-yatsenko dimitri-yatsenko added the bug Indicates an unexpected problem or unintended behavior label Oct 17, 2026
@dimitri-yatsenko dimitri-yatsenko changed the title fix(cascade): seed a delete restricted by another query with its keys fix(cascade): seed a delete with its keys when its restriction is another query, and keep a % in it Oct 18, 2026

@MilagrosMarin MilagrosMarin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dimitri-yatsenko — I traced the % round trip and it holds up: restrict() appends make_condition() output, which doubles every % (condition.py), so the seed's conditions really are compiled SQL and undoing one escape before the cascade re-applies them is symmetric. _has_subquery also fails safe — a false positive falls back to keys(), which selects the same rows. I haven't run the suite locally.

Requested change

✅ test_seed_condition_with_a_percent_sign doesn't assert what its data implies — see the inline comment. The loop carries an expected label_id per case but never checks it, so the test pins the row count, not which row matched.

Minor

_has_subquery (diagram.py:163) strips single- and double-quoted text, so a PostgreSQL identifier "select" is handled, but a MySQL backtick identifier `select` isn't — the two backends differ here. The failure mode is safe (falls back to keys()), so this is just tightening:

text = re.sub(r"'[^']*'|\"[^\"]*\"|`[^`]*`", "", text)  # quoted text and identifiers

Suggestion

This widens seed materialization from dj.Top (rare) to any seed restricted by a query (common — (Subject & (Session & "quality = 0")).delete()). keys() pulls every selected primary key into Python and compiles it back to an OR list, so a large delete now carries that cost where it previously didn't. I haven't measured it; worth a line in the _seed_restriction docstring or the delete() docs so the cost is discoverable.

Before merge

  • On head e9e1d67 I see no workflow runs or check-runs and the status is pending.
  • Base is fix/cascade-overdelete, so this retargets to master when #1592 lands; on maint/2.3, #1593 goes before #1612 as you noted.

The subquery detection, the Top/subquery branch ordering, and the three parametrized seed cases all read correct to me. Let me know if anything needs adjusting!

(Label & {"name": "50%"}, 1),
(Label & "name = '5%%'", 3),
]:
assert len(seed) == 1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
assert len(seed) == 1
assert seed.to_arrays("label_id").tolist() == [label]

label is bound at line 263 but never used, so each case only asserts that exactly one row matched — not that it matched the right one. If name = '5%%' selected label 1 instead of label 3, this would still pass. The test does catch the original bug (double-escaping makes len(seed) 0), but it's weaker than the data suggests.

Ruff won't flag the unused binding either: select is ["E", "W", "F", "C90"], and pyflakes doesn't report unused loop variables — that's bugbear's B007, which isn't enabled.

A delete restricted by another query, such as
`(Subject & (Session & "quality = 0")).delete()`, failed. The cascade
applies the seed's compiled conditions again to a fresh copy of the table
through `restrict()`, which collects column names from the whole condition
text, subquery included, and rejects every name the seed table lacks. The
delete raised `Attribute ... is not found in query` before deleting
anything. On MySQL the backquoted schema name in every subquery tripped the
same check, so even `(Subject & Session).delete()` raised.

On PostgreSQL, a seed whose subquery names only the seed's columns got past
the check but still did nothing. The cascade deletes children before
parents, so by the time it reached the seed, the rows its subquery reads
were gone. `(Subject & Session).delete()` matched no subject, returned 0
and rolled back.

A seed whose conditions contain a subquery is now replaced by the primary
keys it selects, as a `dj.Top` seed already is. The traversal and the
delete then work from the rows the restriction selected when the delete
started.

The new tests delete subjects restricted by a query on their sessions, by
its negation, and by the session table itself. They check the preview
counts and the rows left afterwards, on both backends. All six cases fail
on the previous code.
A delete restricted by a condition containing %, such as `(Label & "name = '50%'").delete()` or a dict `{"name": "50%"}`, deleted nothing and returned 0. The seed's conditions are compiled SQL, in which each % is already escaped as %% for the database driver. The cascade applies them again through `restrict()`, which escaped every % a second time, so the condition looked for the literal text `50%%` and matched no row. A LIKE pattern kept working only because `%%` and `%` match the same strings there.

The seed's conditions now have that first escape undone before the cascade applies them again, so they select the same rows as the restriction on its own.

The new test checks the preview counts for a string condition, a dict and a condition holding `%%`, and deletes with a LIKE pattern, on both backends. It fails before this change.
…sert the matched row

Review follow-up. The subquery check strips quoted text before looking for SELECT; it now strips MySQL's backquoted identifiers too, as it already did PostgreSQL's double-quoted ones, so a column named `select` no longer makes a seed look like a subquery (that case was safe, since the seed's keys select the same rows). The percent-sign test now checks which row each seed selects, not only how many. The seed helper's docstring notes that replacing a seed by its keys reads them all into one condition.
@dimitri-yatsenko
dimitri-yatsenko force-pushed the fix/cascade-subquery-seed branch from e9e1d67 to 94d1105 Compare October 20, 2026 23:51
@dimitri-yatsenko dimitri-yatsenko modified the milestones: v2.3.5, v2.4 Oct 20, 2026
@dimitri-yatsenko

Copy link
Copy Markdown
Member Author

Thanks, @MilagrosMarin. Pushed 94d1105.

  • The percent-sign test now checks which row each seed selects, as you suggested.
  • _has_subquery now also strips MySQL's backquoted identifiers.
  • Cost: the _seed_restriction docstring notes that replacing a seed by its keys reads them all into a single condition.

This PR moves to 2.4. Unlike #1592, it fixes deletes that fail or do nothing, not ones that lose data, and there is a workaround: (T & (T & cond).keys()).delete(). The 2.4 traversal (Diagram.expand, datajoint/datajoint-docs#249) materializes such seeds at delete time anyway. I closed the backport, #1612. This branch is rebased onto #1592's new commit and moves to master when #1592 lands.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Indicates an unexpected problem or unintended behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants