Skip to content

Hibernate 7: reject max and offset on bulk writes and support more query shapes in them - #16822

Open
borinquenkid wants to merge 10 commits into
fix/h7-subquery-root-alias-nested-paths-8.0.xfrom
fix/h7-bulk-write-pagination-guard-8.0.x
Open

borinquenkid wants to merge 10 commits into
fix/h7-subquery-root-alias-nested-paths-8.0.xfrom
fix/h7-bulk-write-pagination-guard-8.0.x

Conversation

@borinquenkid

Copy link
Copy Markdown
Member

Fixes #16805

Stacked on #16821 (chain: #16816, #16817, #16818, #16820, #16821). no ci until the chain reaches 8.0.x.

Problem

deleteAll() and updateAll(Map) on a query run one HQL statement that has no LIMIT. A query with max or offset therefore wrote to every matching row: Person.where { age > 18 }.max(2).deleteAll() deleted all of them. The HQL builder behind these statements also had gaps:

  • a subquery with two criteria produced ... < :p2age > :p3 (no AND), which does not parse;
  • the position returned for an association criterion was dropped, so a parameter after it reused a number;
  • exists, notExists, a to-one association, an alias and sqlRestriction were refused.

Changes

  • HibernateSession.deleteAll / updateAll throw InvalidDataAccessResourceUsageException for max or offset and write nothing. QueryableCriteria gets getDefaultMax() / getDefaultOffset() (default null) for the check.
  • JpaQueryBuilder:
    • joins a subquery's criteria with AND, and drops WHERE when it has none;
    • keeps the parameter position after an association criterion;
    • writes Exists / NotExists;
    • writes a to-one association (block, dotted path, createAlias alias) as owner.id IN (SELECT ...), since HQL DML cannot join. A to-many association is still refused with the same message;
    • registerHqlRenderer lets a datastore render a criterion type it adds. Hibernate 7 registers SqlRestrictionHqlRenderer, which calls the existing grails_sql_restriction function, so {alias} and ? work.
  • Query.PropertyNameCriterion.withProperty(String) copies a criterion onto another property, used to split owner.name.
  • Two existing specs pinned the old behavior and are updated: JpaQueryBuilderFunctionSpec expected the missing AND, and HibernateCriteriaBuilderSqlRestrictionSpec expected sqlRestriction to be refused in a batch operation.

Tests

  • BulkWritePaginationGuardSpec (8): max, offset, both, derived criteria, delete by selected ids, and the unpaginated control. Red before the fix (6 failures).
  • BulkWriteQueryShapesSpec (17): subqueries, deferred subquery, exists, notExists, to-one block, path, alias, position after an association, sqlRestriction with {alias}, ? and a quoted literal. Red before the fix (9 failures).
  • JpaQueryBuilderBulkWriteSpec (13): the exact HQL and parameters, a registered renderer, and an unknown criterion.
  • SqlRestrictionHqlRendererSpec (7).
  • Full suites pass for grails-datastore-core, grails-datamapping-core, grails-datamapping-core-test, grails-data-hibernate7-core and grails-data-hibernate5-core; codeStyle, validateRepositoryConventions and rat are clean.

Not covered

  • PITest on JpaQueryBuilder (run with feat(build-logic): add opt-in PITest mutation testing support and skill #16819's mutableProjects, from grails-datamapping-core-test): no mutant survives in the methods this PR added, except dot <= 0 in splitToOnePath, which is an equivalent mutant (an empty head name matches nothing). The older methods of the class keep their uncovered mutants (for example functionOperator); they are not part of this change.
  • A dotted alias path such as createAlias('owner.address', 'a') is not resolved in a bulk write; it is refused with the property error.

Docs

8.0 upgrade guide section 26.14, the CRUD guide note, and the grails-8-upgrade skill.

… them

A bulk delete or update is one HQL statement with no LIMIT, so a criteria that carried
max or offset wrote to every matching row. deleteAll() and updateAll() in HibernateSession
now throw InvalidDataAccessResourceUsageException naming the value and write nothing.
QueryableCriteria gains getDefaultMax() and getDefaultOffset() for that check.

JpaQueryBuilder, which writes these statements:
- joins the criteria of a subquery with AND; they ran together before and no longer parsed
- keeps the parameter position after an association criterion; the position it returned
  was dropped, so a later parameter reused a number
- writes Exists and NotExists
- writes a to-one association, an association block, a dotted path or a createAlias alias as
  "owner.id IN (SELECT ...)", because HQL DML cannot join; an association that holds many
  is still refused with the same message
- accepts renderers for criterion types a datastore adds; Hibernate 7 registers one for
  SqlRestriction, so {alias} and ? values work in a bulk write

Documented in the 8.0 upgrade guide (26.14), the CRUD guide and the upgrade skill.

Fixes #16805

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@borinquenkid

Copy link
Copy Markdown
Member Author

Added integration scenarios for bulk writes to both grails-test-examples/hibernate5/grails-hibernate and hibernate7/grails-hibernate. BulkWriteScenarioSpec runs deleteAll/updateAll for comparisons, ||, null, !=, dotted and block association paths, a subquery and a whereLazy criteria, and passes on both modules with the same results.

One difference to share with the team: Hibernate 5 ignores max and offset on a bulk write and writes every matching row. On H5, where { qty > 0 }.max(1).deleteAll() deleted all 3 matching rows, and .max(1).updateAll(...) updated all 4. This PR makes H7 refuse the call and write nothing. That deviates from H5, but only where H5 writes to rows the caller excluded. BulkWriteGuardScenarioSpec pins the H7 behaviour and is H7-only for that reason. The same guard on H5 is a separate change.

@borinquenkid

Copy link
Copy Markdown
Member Author

H5 side tracked in #16832.

…write-pagination-guard-8.0.x

Takes the merge of fix/datastore-wherelazy-thread-safe-8.0.x and the review changes of #16816. buildSubQuery joined the criteria of a subquery with AND
on both branches and keeps one copy with its comment. The upgrade guide keeps 26.14 for the not-equal change of #16816, and the bulk statements of this
branch are 26.15.

Claude-Session: https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approving. A property comparison of an association and the root compares the association with itself.

if (head == null) {
return criterion;
}
return new ToOnePathCriterion(head, named.withProperty(path.substring(dot + 1)));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eqProperty('owner.age', 'age') on a pet aged 2 whose owner is aged 40. deleteAll() deletes the pet. Both sides are read as the owner's age. Keep the pet's age on the right.

…queried entity in a bulk write

deleteAll and updateAll read eqProperty('owner.age', 'age') as a comparison of the owner's age with the
owner's age, which is always true, so a pet aged 2 whose owner is aged 40 was deleted. The left side
is read on the association and the right side where it was written: a property of the queried entity
stays that of the queried entity, as pet.owner.id IN (SELECT owner1.id FROM Owner owner1 WHERE
owner1.age = pet.age), and a property of the same association is read on the association.

Claude-Session: https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL
@borinquenkid

Copy link
Copy Markdown
Member Author

@jamesfredley Reproduced, and fixed in the new commit. BulkWritePropertyComparisonSpec has your case, eqProperty('owner.age', 'age'), and all three rows fail without the change.

The DELETE and UPDATE statement cannot join, so the left side owner.age is read in a subquery on the owner, and the right side age was read there too, as the owner's age: owner1.age = owner1.age, which is always true. The right side is now read where it was written:

  • eqProperty('owner.age', 'age') is pet.owner.id IN (SELECT owner1.id FROM Owner owner1 WHERE owner1.age = pet.age), so the pet aged 2 of an owner aged 40 stays, and deleteAll deletes the rows list returns.
  • eqProperty('owner.age', 'owner.height') is read on the owner for both sides.
  • The spec covers deleteAll and updateAll, with list as the comparison for the rows.

Not changed: a comparison whose left side is a plain property and whose right side goes through an association (eqProperty('age', 'owner.age')) is written as before.

Branch policy: the new public types and throws in this PR were already in it. This commit adds one throw, for a comparison criterion of a type the builder does not know, which no criterion in the codebase is. Where the PR belongs is still open; I have not moved it.

If you find another shape that does not reproduce first time, a failing spec on a throwaway branch, or the few lines you ran, would help me a lot.

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requested changes. A comparison of a self association inside a subquery still compares the right side with itself.

}
PersistentEntity associated = association.getAssociatedEntity();
String identity = associated.getIdentity().getName();
String subqueryName = associated.getDecapitalizedName() + position;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eqProperty('manager.age', 'age') inside a subquery should compare the manager's age with the employee's age. It compares the inner age with itself, so a worker who should stay is deleted. Give the inner query its own alias.

@matrei matrei left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 65d597f (base fix/h7-subquery-root-alias-nested-paths-8.0.x, #16821's head branch). This PR's own change is the 18 files GitHub shows, measured from 8b303a2, the #16821 head it last merged. Since then #16821 has merged #16816's and #16820's heads (8142212). Merging that into this branch conflicts only in grails-8-upgrade/SKILL.md, which the base has restructured. The PR is labelled no ci, so CI has not run on it.

What I ran locally:

  • The PR's Hibernate 7 specs on this head: all pass.
  • The same specs on today's 8.0.x (9fc7dd3). Six of the eight BulkWritePaginationGuardSpec features fail because no exception is thrown: max and offset are ignored, and every matching row is written. The three BulkWritePropertyComparisonSpec features throw Cannot use [EqualsProperty] criterion on non-existent property: owner.age. All 17 BulkWriteQueryShapesSpec features fail. The subquery features fail with a SemanticException or SyntaxException; that is #16816's part, which is in the base now. Every other shape is refused with an InvalidDataAccessResourceUsageException: Joins cannot be used in a DELETE or UPDATE operation, Queries of type Exists are not supported, Queries of type SqlRestriction are not supported or non-existent property: owner.name.
  • This head merged with the current base, with the skill conflict resolved by keeping the base's version. The PR's specs pass there, and so do the full suites of grails-datastore-core, grails-datamapping-core, grails-datamapping-core-test, grails-data-hibernate5-core and grails-data-hibernate7-core (482, 1876, 738, 1219 and 5456 tests, no failures).
  • A throwaway spec on Hibernate 7 (this head and 8.0.x) and on Hibernate 5 (this head and 7.0.x), for items 1 to 4.

Needs a fix

  1. Branch: the max and offset guard belongs on 7.0.x. On 7.0.x, Hibernate 5 does what you found on 8.0.x in your comment of 11-02. new DetachedCriteria(Pet).build { gt('weight', 0) }.max(1).updateAll(weight: 7) updates all three matching rows, and .max(1).deleteAll() deletes all three. Writing rows the call left out is an old result that contradicts the call, so by the rule in the dev@ thread the fix goes to the oldest branch that has the bug, which is 7.0.x. #16833 (the Hibernate 5 guard) is approved, but it is stacked on this branch at 8.0.x. It can go to 7.0.x as one PR, together with getDefaultMax() and getDefaultOffset() on QueryableCriteria and AbstractDetachedCriteria and the Hibernate 5 BulkWriteGuardScenarioSpec, and be merged up. The Hibernate 7 guard then follows on 8.0.x, in this PR or in a small one after the merge-up. The two accessors are new public API on a patch branch, which the patch rule asks to avoid. For a fix that stops lost rows I think they are worth it, but the description should say why. The change makes a call that used to write rows throw instead, so the rule also asks for two reviewers and an upgrade note on 7.0.x (item 5). From reading the code, MongoDB's and Neo4j's deleteAll and updateAll build their statement from the criteria alone, so they ignore max and offset the same way. That is worth an issue of its own.

  2. Branch: the new query shapes belong on 8.1.x. On 8.0.x and on 7.0.x, every shape this PR adds to a bulk write is refused with a clear InvalidDataAccessResourceUsageException: a to-one path, an association block, a createAlias alias, a property comparison through an association, exists, notExists and sqlRestriction. None of them writes wrong rows today, so supporting them is a feature, not a fix. The PR also adds public API for them, all @since 8.0.1: JpaQueryBuilder.registerHqlRenderer, JpaQueryBuilder.HqlCriterionRenderer, SqlRestrictionHqlRenderer, and Query.PropertyNameCriterion.withProperty, which also makes every PropertyNameCriterion Cloneable. The patch rule allows no new public classes, methods or extension points on 7.0.x or 8.0.x, so this part goes to 8.1.x, once items 3 and 4 are fixed. That splits the PR in two: the guard from item 1, and the shapes against 8.1.x. JpaQueryBuilder is in grails-datastore-core, and Hibernate 5 builds its bulk statements with it too, so the shapes also reach Hibernate 5. At this head, an association block, a path, an alias and exists all run in a Hibernate 5 deleteAll, while the title and the description say Hibernate 7. On Hibernate 5, list() refuses eq('owner.name', 'Ned') in a DetachedCriteria (could not resolve property: owner.name), so there a bulk write would accept a path that a read does not.

  3. James's case from 11-17 still writes the wrong rows. I used an Employee class (PrbEmp in my spec) with a boss aged 50 and two workers aged 50 and 30, both reporting to the boss. With 'in'('id', new DetachedCriteria(Employee).build { eqProperty('manager.age', 'age') }.property('id')), list() returns the worker aged 50, and deleteAll() deletes both workers. Hibernate 7 writes ... pe2_0.manager_id in (select pe3_0.id from prb_emp pe3_0 where pe3_0.age=pe3_0.age). buildSubQuery names the subquery's root getDecapitalizedName() + position. handleToOneComparisonAsSubquery names the manager's subquery the same way, with the same position, because a property comparison binds no parameter. Both are employee0, the inner one hides the outer one, and the right side employee0.age is read on the manager. Hibernate 5 at this head deletes both workers too. A path alone (eq('manager.name', 'boss') inside the subquery) gives the right rows, because nothing inside refers to the outer alias. The aliases the builder makes up need names that no enclosing scope uses, for example from a counter kept for the statement instead of the parameter position, and James's case needs a feature.

  4. A to-one path inside || writes rows that list() does not return. I used three pets (PrbPet and PrbOwn in my spec): rex (owner Ned, weight 10), tom (owner Homer, weight 10) and stray (no owner, weight 50). For Pet.where { owner.name == 'Ned' || weight > 40 }, list() returns rex, because the read joins the owner (join prb_own o1_0 on o1_0.id=pp1_0.owner_id) and a pet without an owner is never returned. deleteAll() deletes rex and stray: delete from prb_pet pp1_0 where ((pp1_0.owner_id in (select ...) or pp1_0.weight>?)). or { eq('owner.name', 'Ned'); gt('weight', 40) } does the same, and so does Hibernate 5 at this head. #16816's note 26.14 makes the reads and the bulk writes of a where query agree for !=. This brings back a query that reads one set of rows and changes another, and for a delete the extra rows are lost. On 8.0.x both are refused. The statement could require each association the read joins (pp1_0.owner_id is not null), or a to-one path under || could stay refused. A negated path, not { eq('owner.name', 'Ned') }, deletes what list() returns, as it is.

  5. The upgrade note should say how to get the old result. Section 26.15 explains how to write a limited number of rows: select the ids with the limit first. The rule in the dev@ thread also asks how to keep the old result. An application that reuses a paginated criteria for "delete everything that matches this filter" got every matching row written, and it now gets an exception. A criteria cannot drop max or offset once they are set, so the note should say to call deleteAll() or updateAll() on the criteria before they are applied. With item 1, that part of the note moves to 7.0.x with the guard.

  6. Description. Please rewrite it to match this head:

    • The subquery fixes it lists (joining a subquery's criteria with AND, dropping an empty WHERE) and the JpaQueryBuilderFunctionSpec change are #16816's now, and are not in this diff. The 26.15 bullet "several criteria in one subquery, which are now joined with and" describes the same change, so it belongs with #16816 (and #17049 on 7.0.x).
    • It refers to section 26.14; the section is 26.15.
    • It says Hibernate 7, but the shapes also reach Hibernate 5 (item 2).
    • It leaves out BulkWritePropertyComparisonSpec (the 11-17 commit, for James's 11-12 case) and the example-app scenario specs on Hibernate 5 and 7.
    • What changes for applications, as the rule in the dev@ thread asks: max and offset on deleteAll() and updateAll() used to write every matching row, and now throw without writing anything.

Note

  1. offset(0) is refused as well: eq('name', 'rex') with .offset(0).deleteAll() throws, although an offset of 0 leaves no row out. A pagination helper that always sets offset(params.offset ?: 0) would trip over it. Refusing only max and an offset above 0 would be enough.

Verified as correct

  • The guard runs before the statement is built, so a refused call writes nothing. BulkWritePaginationGuardSpec checks the rows afterwards, and a criteria derived from one that set max is refused too.
  • James's case from 11-12, eqProperty('owner.age', 'age') at the top level, reads each side on its own entity now. The three BulkWritePropertyComparisonSpec features delete and update the rows that list() returns.
  • SqlRestrictionHqlRenderer doubles the quotes of the SQL it embeds and binds the ? values as parameters. {alias} is resolved through the identifier column.
  • An embedded property in a bulk write is still refused, as on 8.0.x. Only the message for the path form changes.

@borinquenkid

Copy link
Copy Markdown
Member Author

Item 1 of your review: the Hibernate 7 guard is split out as #17076 against 8.0.x, with no new public API (the criteria's fields are read by reflection) and an offset of 0 accepted (item 7). The query shapes, items 3 and 4 and the getters move to a PR against 8.1.x; this PR will be retargeted and cut down to that part.

https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

4 participants