Repository navigation
Hibernate 7: reject max and offset on bulk writes and support more query shapes in them - #16822
borinquenkid wants to merge 10 commits into
Conversation
… them
A bulk delete or update is one HQL statement with no LIMIT, so a criteria that carried
max or offset wrote to every matching row. deleteAll() and updateAll() in HibernateSession
now throw InvalidDataAccessResourceUsageException naming the value and write nothing.
QueryableCriteria gains getDefaultMax() and getDefaultOffset() for that check.
JpaQueryBuilder, which writes these statements:
- joins the criteria of a subquery with AND; they ran together before and no longer parsed
- keeps the parameter position after an association criterion; the position it returned
was dropped, so a later parameter reused a number
- writes Exists and NotExists
- writes a to-one association, an association block, a dotted path or a createAlias alias as
"owner.id IN (SELECT ...)", because HQL DML cannot join; an association that holds many
is still refused with the same message
- accepts renderers for criterion types a datastore adds; Hibernate 7 registers one for
SqlRestriction, so {alias} and ? values work in a bulk write
Documented in the 8.0 upgrade guide (26.14), the CRUD guide and the upgrade skill.
Fixes #16805
…d-paths-8.0.x' into fix/h7-bulk-write-pagination-guard-8.0.x
…/h7-bulk-write-pagination-guard-8.0.x
…/h7-bulk-write-pagination-guard-8.0.x
…/h7-bulk-write-pagination-guard-8.0.x
…/h7-bulk-write-pagination-guard-8.0.x
…nd the max and offset guard on H7 Claude-Session: https://claude.ai/code/session_01Dmki4SQGwRSj8pRJ5KCWyX
|
Added integration scenarios for bulk writes to both One difference to share with the team: Hibernate 5 ignores |
|
H5 side tracked in #16832. |
…/h7-bulk-write-pagination-guard-8.0.x
…write-pagination-guard-8.0.x Takes the merge of fix/datastore-wherelazy-thread-safe-8.0.x and the review changes of #16816. buildSubQuery joined the criteria of a subquery with AND on both branches and keeps one copy with its comment. The upgrade guide keeps 26.14 for the not-equal change of #16816, and the bulk statements of this branch are 26.15. Claude-Session: https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL
jamesfredley
left a comment
There was a problem hiding this comment.
Not approving. A property comparison of an association and the root compares the association with itself.
| if (head == null) { | ||
| return criterion; | ||
| } | ||
| return new ToOnePathCriterion(head, named.withProperty(path.substring(dot + 1))); |
There was a problem hiding this comment.
eqProperty('owner.age', 'age') on a pet aged 2 whose owner is aged 40. deleteAll() deletes the pet. Both sides are read as the owner's age. Keep the pet's age on the right.
…queried entity in a bulk write
deleteAll and updateAll read eqProperty('owner.age', 'age') as a comparison of the owner's age with the
owner's age, which is always true, so a pet aged 2 whose owner is aged 40 was deleted. The left side
is read on the association and the right side where it was written: a property of the queried entity
stays that of the queried entity, as pet.owner.id IN (SELECT owner1.id FROM Owner owner1 WHERE
owner1.age = pet.age), and a property of the same association is read on the association.
Claude-Session: https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL
|
@jamesfredley Reproduced, and fixed in the new commit. The DELETE and UPDATE statement cannot join, so the left side
Not changed: a comparison whose left side is a plain property and whose right side goes through an association ( Branch policy: the new public types and throws in this PR were already in it. This commit adds one throw, for a comparison criterion of a type the builder does not know, which no criterion in the codebase is. Where the PR belongs is still open; I have not moved it. If you find another shape that does not reproduce first time, a failing spec on a throwaway branch, or the few lines you ran, would help me a lot. |
jamesfredley
left a comment
There was a problem hiding this comment.
Requested changes. A comparison of a self association inside a subquery still compares the right side with itself.
| } | ||
| PersistentEntity associated = association.getAssociatedEntity(); | ||
| String identity = associated.getIdentity().getName(); | ||
| String subqueryName = associated.getDecapitalizedName() + position; |
There was a problem hiding this comment.
eqProperty('manager.age', 'age') inside a subquery should compare the manager's age with the employee's age. It compares the inner age with itself, so a worker who should stay is deleted. Give the inner query its own alias.
matrei
left a comment
There was a problem hiding this comment.
Reviewed at 65d597f (base fix/h7-subquery-root-alias-nested-paths-8.0.x, #16821's head branch). This PR's own change is the 18 files GitHub shows, measured from 8b303a2, the #16821 head it last merged. Since then #16821 has merged #16816's and #16820's heads (8142212). Merging that into this branch conflicts only in grails-8-upgrade/SKILL.md, which the base has restructured. The PR is labelled no ci, so CI has not run on it.
What I ran locally:
- The PR's Hibernate 7 specs on this head: all pass.
- The same specs on today's 8.0.x (9fc7dd3). Six of the eight
BulkWritePaginationGuardSpecfeatures fail because no exception is thrown:maxandoffsetare ignored, and every matching row is written. The threeBulkWritePropertyComparisonSpecfeatures throwCannot use [EqualsProperty] criterion on non-existent property: owner.age. All 17BulkWriteQueryShapesSpecfeatures fail. The subquery features fail with aSemanticExceptionorSyntaxException; that is #16816's part, which is in the base now. Every other shape is refused with anInvalidDataAccessResourceUsageException:Joins cannot be used in a DELETE or UPDATE operation,Queries of type Exists are not supported,Queries of type SqlRestriction are not supportedornon-existent property: owner.name. - This head merged with the current base, with the skill conflict resolved by keeping the base's version. The PR's specs pass there, and so do the full suites of
grails-datastore-core,grails-datamapping-core,grails-datamapping-core-test,grails-data-hibernate5-coreandgrails-data-hibernate7-core(482, 1876, 738, 1219 and 5456 tests, no failures). - A throwaway spec on Hibernate 7 (this head and 8.0.x) and on Hibernate 5 (this head and 7.0.x), for items 1 to 4.
Needs a fix
-
Branch: the
maxandoffsetguard belongs on 7.0.x. On 7.0.x, Hibernate 5 does what you found on 8.0.x in your comment of 11-02.new DetachedCriteria(Pet).build { gt('weight', 0) }.max(1).updateAll(weight: 7)updates all three matching rows, and.max(1).deleteAll()deletes all three. Writing rows the call left out is an old result that contradicts the call, so by the rule in the dev@ thread the fix goes to the oldest branch that has the bug, which is 7.0.x. #16833 (the Hibernate 5 guard) is approved, but it is stacked on this branch at 8.0.x. It can go to 7.0.x as one PR, together withgetDefaultMax()andgetDefaultOffset()onQueryableCriteriaandAbstractDetachedCriteriaand the Hibernate 5BulkWriteGuardScenarioSpec, and be merged up. The Hibernate 7 guard then follows on 8.0.x, in this PR or in a small one after the merge-up. The two accessors are new public API on a patch branch, which the patch rule asks to avoid. For a fix that stops lost rows I think they are worth it, but the description should say why. The change makes a call that used to write rows throw instead, so the rule also asks for two reviewers and an upgrade note on 7.0.x (item 5). From reading the code, MongoDB's and Neo4j'sdeleteAllandupdateAllbuild their statement from the criteria alone, so they ignoremaxandoffsetthe same way. That is worth an issue of its own. -
Branch: the new query shapes belong on 8.1.x. On 8.0.x and on 7.0.x, every shape this PR adds to a bulk write is refused with a clear
InvalidDataAccessResourceUsageException: a to-one path, an association block, acreateAliasalias, a property comparison through an association,exists,notExistsandsqlRestriction. None of them writes wrong rows today, so supporting them is a feature, not a fix. The PR also adds public API for them, all@since 8.0.1:JpaQueryBuilder.registerHqlRenderer,JpaQueryBuilder.HqlCriterionRenderer,SqlRestrictionHqlRenderer, andQuery.PropertyNameCriterion.withProperty, which also makes everyPropertyNameCriterionCloneable. The patch rule allows no new public classes, methods or extension points on 7.0.x or 8.0.x, so this part goes to 8.1.x, once items 3 and 4 are fixed. That splits the PR in two: the guard from item 1, and the shapes against 8.1.x.JpaQueryBuilderis ingrails-datastore-core, and Hibernate 5 builds its bulk statements with it too, so the shapes also reach Hibernate 5. At this head, an association block, a path, an alias andexistsall run in a Hibernate 5deleteAll, while the title and the description say Hibernate 7. On Hibernate 5,list()refuseseq('owner.name', 'Ned')in aDetachedCriteria(could not resolve property: owner.name), so there a bulk write would accept a path that a read does not. -
James's case from 11-17 still writes the wrong rows. I used an
Employeeclass (PrbEmpin my spec) with a boss aged 50 and two workers aged 50 and 30, both reporting to the boss. With'in'('id', new DetachedCriteria(Employee).build { eqProperty('manager.age', 'age') }.property('id')),list()returns the worker aged 50, anddeleteAll()deletes both workers. Hibernate 7 writes... pe2_0.manager_id in (select pe3_0.id from prb_emp pe3_0 where pe3_0.age=pe3_0.age).buildSubQuerynames the subquery's rootgetDecapitalizedName() + position.handleToOneComparisonAsSubquerynames the manager's subquery the same way, with the same position, because a property comparison binds no parameter. Both areemployee0, the inner one hides the outer one, and the right sideemployee0.ageis read on the manager. Hibernate 5 at this head deletes both workers too. A path alone (eq('manager.name', 'boss')inside the subquery) gives the right rows, because nothing inside refers to the outer alias. The aliases the builder makes up need names that no enclosing scope uses, for example from a counter kept for the statement instead of the parameter position, and James's case needs a feature. -
A to-one path inside
||writes rows thatlist()does not return. I used three pets (PrbPetandPrbOwnin my spec):rex(owner Ned, weight 10),tom(owner Homer, weight 10) andstray(no owner, weight 50). ForPet.where { owner.name == 'Ned' || weight > 40 },list()returnsrex, because the read joins the owner (join prb_own o1_0 on o1_0.id=pp1_0.owner_id) and a pet without an owner is never returned.deleteAll()deletesrexandstray:delete from prb_pet pp1_0 where ((pp1_0.owner_id in (select ...) or pp1_0.weight>?)).or { eq('owner.name', 'Ned'); gt('weight', 40) }does the same, and so does Hibernate 5 at this head. #16816's note 26.14 makes the reads and the bulk writes of a where query agree for!=. This brings back a query that reads one set of rows and changes another, and for a delete the extra rows are lost. On 8.0.x both are refused. The statement could require each association the read joins (pp1_0.owner_id is not null), or a to-one path under||could stay refused. A negated path,not { eq('owner.name', 'Ned') }, deletes whatlist()returns, as it is. -
The upgrade note should say how to get the old result. Section 26.15 explains how to write a limited number of rows: select the ids with the limit first. The rule in the dev@ thread also asks how to keep the old result. An application that reuses a paginated criteria for "delete everything that matches this filter" got every matching row written, and it now gets an exception. A criteria cannot drop
maxoroffsetonce they are set, so the note should say to calldeleteAll()orupdateAll()on the criteria before they are applied. With item 1, that part of the note moves to 7.0.x with the guard. -
Description. Please rewrite it to match this head:
- The subquery fixes it lists (joining a subquery's criteria with
AND, dropping an emptyWHERE) and theJpaQueryBuilderFunctionSpecchange are #16816's now, and are not in this diff. The 26.15 bullet "several criteria in one subquery, which are now joined withand" describes the same change, so it belongs with #16816 (and #17049 on 7.0.x). - It refers to section 26.14; the section is 26.15.
- It says Hibernate 7, but the shapes also reach Hibernate 5 (item 2).
- It leaves out
BulkWritePropertyComparisonSpec(the 11-17 commit, for James's 11-12 case) and the example-app scenario specs on Hibernate 5 and 7. - What changes for applications, as the rule in the dev@ thread asks:
maxandoffsetondeleteAll()andupdateAll()used to write every matching row, and now throw without writing anything.
- The subquery fixes it lists (joining a subquery's criteria with
Note
offset(0)is refused as well:eq('name', 'rex')with.offset(0).deleteAll()throws, although an offset of 0 leaves no row out. A pagination helper that always setsoffset(params.offset ?: 0)would trip over it. Refusing onlymaxand anoffsetabove 0 would be enough.
Verified as correct
- The guard runs before the statement is built, so a refused call writes nothing.
BulkWritePaginationGuardSpecchecks the rows afterwards, and a criteria derived from one that setmaxis refused too. - James's case from 11-12,
eqProperty('owner.age', 'age')at the top level, reads each side on its own entity now. The threeBulkWritePropertyComparisonSpecfeatures delete and update the rows thatlist()returns. SqlRestrictionHqlRendererdoubles the quotes of the SQL it embeds and binds the?values as parameters.{alias}is resolved through the identifier column.- An embedded property in a bulk write is still refused, as on 8.0.x. Only the message for the path form changes.
|
Item 1 of your review: the Hibernate 7 guard is split out as #17076 against 8.0.x, with no new public API (the criteria's fields are read by reflection) and an offset of 0 accepted (item 7). The query shapes, items 3 and 4 and the getters move to a PR against 8.1.x; this PR will be retargeted and cut down to that part. |
Fixes #16805
Stacked on #16821 (chain: #16816, #16817, #16818, #16820, #16821).
no ciuntil the chain reaches 8.0.x.Problem
deleteAll()andupdateAll(Map)on a query run one HQL statement that has noLIMIT. A query withmaxoroffsettherefore wrote to every matching row:Person.where { age > 18 }.max(2).deleteAll()deleted all of them. The HQL builder behind these statements also had gaps:... < :p2age > :p3(noAND), which does not parse;exists,notExists, a to-one association, an alias andsqlRestrictionwere refused.Changes
HibernateSession.deleteAll/updateAllthrowInvalidDataAccessResourceUsageExceptionformaxoroffsetand write nothing.QueryableCriteriagetsgetDefaultMax()/getDefaultOffset()(defaultnull) for the check.JpaQueryBuilder:AND, and dropsWHEREwhen it has none;Exists/NotExists;createAliasalias) asowner.id IN (SELECT ...), since HQL DML cannot join. A to-many association is still refused with the same message;registerHqlRendererlets a datastore render a criterion type it adds. Hibernate 7 registersSqlRestrictionHqlRenderer, which calls the existinggrails_sql_restrictionfunction, so{alias}and?work.Query.PropertyNameCriterion.withProperty(String)copies a criterion onto another property, used to splitowner.name.JpaQueryBuilderFunctionSpecexpected the missingAND, andHibernateCriteriaBuilderSqlRestrictionSpecexpectedsqlRestrictionto be refused in a batch operation.Tests
BulkWritePaginationGuardSpec(8): max, offset, both, derived criteria, delete by selected ids, and the unpaginated control. Red before the fix (6 failures).BulkWriteQueryShapesSpec(17): subqueries, deferred subquery, exists, notExists, to-one block, path, alias, position after an association,sqlRestrictionwith{alias},?and a quoted literal. Red before the fix (9 failures).JpaQueryBuilderBulkWriteSpec(13): the exact HQL and parameters, a registered renderer, and an unknown criterion.SqlRestrictionHqlRendererSpec(7).grails-datastore-core,grails-datamapping-core,grails-datamapping-core-test,grails-data-hibernate7-coreandgrails-data-hibernate5-core;codeStyle,validateRepositoryConventionsandratare clean.Not covered
JpaQueryBuilder(run with feat(build-logic): add opt-in PITest mutation testing support and skill #16819'smutableProjects, fromgrails-datamapping-core-test): no mutant survives in the methods this PR added, exceptdot <= 0insplitToOnePath, which is an equivalent mutant (an empty head name matches nothing). The older methods of the class keep their uncovered mutants (for examplefunctionOperator); they are not part of this change.createAlias('owner.address', 'a')is not resolved in a bulk write; it is refused with the property error.Docs
8.0 upgrade guide section 26.14, the CRUD guide note, and the
grails-8-upgradeskill.