Repository navigation
Sort by and join a collection without changing what a query fetches - #17072
jdaugherty wants to merge 28 commits into
Conversation
Sorting a where query by a property of an association, such as sort: 'club.name', failed with a QueryException when the query also had criteria on that association. The criteria join the association under a generated alias, and the order was added with the association path, which Hibernate cannot resolve. The order now uses the alias of the join.
A sort by a path through more than one association, such as owner.employer.name, is still not supported.
This reverts commit b494caa.
A sort by a path that crosses more than one association, such as team.club.name, failed with a QueryException whether or not the where query joined the nested association, because order() split the path at its first dot only. The sort property is now qualified with the alias of the last association on its path: an association the query has already joined is reused, and one it has not joined yet is joined for the sort and registered like any other join. A sort by two properties of the same association no longer fails with "duplicate association path" either, since the sort join is registered instead of created anew for every order.
Since order() walks the whole sort path, a sort by the identifier of an association that follows another one on the path, such as team.club.id, joined that last association too. The join is an inner join unless fetch requests another type, so the sort left out the rows where the association is null. Before, only the first association was joined and Hibernate read club.id from the foreign key of team, which kept them. The identifier is sorted by the foreign key again when the entity before the association holds it. An association without such a foreign key, a hasOne or the inverse side of a one-to-one, is still joined, and so is the first association of a path, as it always has been. The spec also covers the join type of a sort join: an inner join by default, and a left join when fetch requests one. Team.club is nullable in the fixture for these cases. The docs describe the inner join, how to request a left join, and the identifier of a nested association.
list with a map of sort properties, such as Team.list(sort: ['club.name': 'asc', name: 'desc']), sorted by the last entry only, because each entry replaced the orders of the query. The same held for a default sort that the mapping declares for more than one property. Every entry is now applied, in the order of the map. An association is joined once for all the sort properties that use it. list joined it once for each, and find and findAll with an example, findWhere and findAllWhere failed with "duplicate association path". Part 2 of the split of apache#16853.
A paged query of a detached criteria that has an alias for the queried
class, and criteria that use it, failed when totalCount was read:
new DetachedCriteria(Team, 't').eq('t.name', 'Arsenal').list(max: 2).totalCount
The count runs on a copy of the criteria, and the copy was created
without the alias, so Hibernate could not resolve t.name. The copy now
has the alias of the original.
Part 3 of the split of apache#16853.
count on a where query or a detached criteria that has a sort, such as
Team.where { ... }.sort('name').count(), kept the order by in the query
that counts the rows, which H2 rejects with "must be in the GROUP BY
list". A query that only counts its rows now has no orders.
The join that a sort by a property of an association makes is kept, so
the count is that of the rows which list returns for the same query.
Part 4 of the split of apache#16853.
… by it A sort by a property of an association joins the association with an inner join, so it leaves out the instances whose association is null: Team.list(sort: 'club.name') does not return the teams without a club, and count and the totalCount of a paged list do not count them. The new grails.gorm.leftJoinAssociationSort setting makes an association that is joined only for a sort a left join, in where queries, detached criteria, dynamic finders, list, find and findAll with an example, findWhere, findAllWhere and a criteria query that is passed pagination arguments. It is disabled by default, so the results of existing applications do not change in a patch release. Grails 8.1 enables it by default. With the setting, a sort by a nested identifier, such as team.club.id, goes through the left join. Without it, it is still read from the foreign key. A sort by a property of a collection keeps its inner join: Hibernate fetches a collection through a left join, which is handled separately. Part 5 of the split of apache#16853.
Team.where { ... }.join('club', JoinType.INNER) lists and counts the
teams with a club, but deleteAll and updateAll on it ignored the join
type and deleted or updated the teams without a club as well. They now
modify the instances that list returns.
An update or a delete cannot join, so each requested join becomes a
restriction: a subquery that selects from the entity of the first
association on the path and joins the rest of it. deleteAll selects the
identifiers first when the path starts with a collection that has a
join table, because Hibernate deletes the join rows before the
instances with the same restriction. A one-to-one whose foreign key the
other class has is matched through the property that refers back.
MySQL, and MariaDB before 10.3, do not allow an update or a delete to
select from the table that it modifies. For a join that leads back to
the queried class, these dialects get an UnsupportedOperationException
that says to restrict the query with criteria instead.
Part 8 of the split of apache#16853.
The criteria builder accepted a join type and never used it.
join('club', JoinType.INNER) has had the body of the one-argument join
since it was added, so it set a fetch mode, and a JoinType in the fetch
argument of list matched no fetch mode, so the association was not
fetched at all.
Both now join and fetch the association with the requested type, with
the rules of a where query: a path through a collection restricts the
query with a subquery, and the associations on the path are fetched.
The join types are applied once the criteria are built, so an
association that the criteria have a block for, or join through an
alias, keeps that join. A fetch mode requested afterwards, with join,
select, fetchMode or the fetch argument, replaces the join type.
The sort argument of the list of a criteria query uses a join that the
criteria already have for the association, such as the one of a join
type that the fetch argument requests for it. It joined the association
a second time, which failed with "duplicate association path".
Part 9 of the split of apache#16853.
order('club.name') in a criteria query failed with "could not resolve
property", and with a Query.Order it sorted by a property of the
queried class. A criteria query that is passed pagination arguments
and a sort by a path through more than one association failed when
totalCount was read, or with "duplicate association path" when the
criteria used the first association, and a map of sort properties
failed with a ClassCastException.
order now resolves an association path, also inside the criteria of an
association, and joins with a left join an association that the
criteria do not join. The orders are added once the criteria are
built, so an order uses the join and the join type that the criteria
or the fetch argument request, whichever comes first. An order by the
identifier of an association reads the foreign key, as before. The
sort argument reuses a join that the criteria have for a path, also
one made through an alias, and creates any other from the root
criteria, which is how the count recreates it.
The sort argument can also start with an alias that the criteria give
an association, as sort: 'c.name' with createAlias('club', 'c'), also
as an entry of a map of sort properties and followed by a further
association. Such a sort was dropped, which review 5868034911 of
apache#16853 reported for the map.
The join that an order makes is a left join only with the
grails.gorm.leftJoinAssociationSort setting. Without it the association
is joined with an inner join, as it is for the other sorts.
Part 10 of the split of apache#16853.
A where query, a detached criteria and a dynamic finder joined an association as soon as criteria, a projection or a sort needed it. The rest of the query can still change after that: criteria passed to list are added later, and the fetch argument of list, which can request a join type, is read last. The result depended on the order in which the query was put together: - a join type in the fetch argument of list was not used for the join that the criteria or a projection had already made; - a sort stored on the query with sort() was joined before a join type requested in list was known, and criteria passed to list could not join the same association through an alias; - a join type was looked up by the name of the association, so a type requested for club was used for team.club, and one requested for team.club was not; - criteria on a nested association used twice failed with "duplicate association path". Criteria, projections and orders only need the alias of a join. A join is now registered under its path from the root entity and its alias, and made when the query is executed, or when a subquery is handed to the query that embeds it, with the join type that the query requests for that path. A listener of the query events that adds criteria on an association to find and findAll with an example, findWhere or findAllWhere gets its joins made after the event. Part 6 of the split of apache#16853.
A where query, a detached criteria or a dynamic finder can request a
join type for an association, with fetch: [club: JoinType.INNER] or
join('club', JoinType.INNER). The type was only used when criteria or
a sort joined the association. Otherwise the association was fetched
with the outer join of its fetch mode, so
Team.where { ... }.list(fetch: [club: JoinType.INNER]) returned the
teams without a club, and the same query sorted by club.name did not.
A requested join type other than LEFT is now applied when the query is
executed, to list, get, find, count, the totalCount of a paged list, a
subquery and a query with a projection:
- INNER and RIGHT on an association to a single instance join and
fetch it; a right join is a right join, not an outer fetch;
- a path that ends in a collection or passes through one restricts the
query with a correlated subquery, so that the collection stays
fetched with all of its elements, each instance is counted once, and
the filters of the session, such as the tenant filter, apply;
- every association on a requested path is joined and fetched;
- a collection of a basic type is handled like any other collection;
- a fetch mode requested afterwards replaces the join type, on the
query and on the detached criteria, while the join that criteria on
the association make keeps it.
Part 7 of the split of apache#16853.
list, findWhere, findAllWhere, and find and findAll with an example ignored a JoinType in their fetch argument: they returned the instances without the association, and list did not fetch it. They now apply it, before a sort by the association, which uses that join. list fetches with the requested type, returns each instance once when it fetches a collection, and applies RIGHT like INNER, since Hibernate does not support right joins in the criteria queries that list runs. 'join' and 'eager' in the fetch argument of list fetch the association with a left join and return every instance, as a where query and a dynamic finder do. list joined it with an inner join without fetching it, so it left out the instances whose association is null or whose collection is empty. The totalCount of a paged list counts each instance once when the list fetches a collection, and keeps a sort join that hangs under a fetch. Part 11 of the split of apache#16853.
A sort by a property of a collection joined the collection with an inner join. It left out the instances whose collection is empty, and took the place of a fetch that the query requested for the collection. Criteria on a collection with a requested left join initialized the collection with only the elements that the criteria matched. - A sort joins a collection with a left join and does not fetch it through that join, unless the query asks for the collection to be fetched, with or without the sort. - A left join of a collection that criteria use does not fetch the collection. When the query also fetches another collection, that one is loaded once the query has run, because Hibernate would read it from the columns of the join of the criteria. - A sort by a collection next to a fetch of another collection repeats rows, and a bag cannot tell a repeated row from a repeated element, which is why Hibernate does not fetch two bags in one query. When either collection is a bag, the query throws an UnsupportedOperationException that names both, where it returned the bag with repeated elements. A query that projects values loads no collection and is not affected. A fetch that a mapping declares counts as fetched only where Hibernate joins it: not at or beyond hibernate.max_fetch_depth joins from the queried class, where Hibernate loads it with a query of its own (review 5868034911 of apache#16853). This applies with the grails.gorm.leftJoinAssociationSort setting. Without it a sort still joins a collection with an inner join, which leaves out the instances whose collection is empty. Part 12 of the split of apache#16853.
jamesfredley
left a comment
There was a problem hiding this comment.
Requested changes. A sort by a collection still drops a row that has no child.
| } | ||
| } | ||
| if (!leftJoin) { | ||
| return from.join(associationName, JoinType.INNER); |
There was a problem hiding this comment.
Team.list() returns a team with no players. Team.list(sort: 'players.name') does not. The sort joins the collection as an inner join. Keep the team that has no players.
There was a problem hiding this comment.
Keeping the inner join with grails.gorm.leftJoinAssociationSort disabled is intentional. The dev@ agreement keeps the old result by default on 7.0.x, 7.1.x, 7.2.x and 8.0.x, enables the setting by default in 8.1, and removes it in 9.0: https://lists.apache.org/thread/btc50c855byrw1zb6rqslyk3dlc546mm.
This part extends the opt-in sort behavior to collections while preserving their fetch behavior. Making Team.list(sort: "players.name") keep an empty collection unconditionally would change the default we agreed to preserve. If the team is dropped with leftJoinAssociationSort: true, please provide that reproducer; that would be an in-scope bug in this part.
matrei
left a comment
There was a problem hiding this comment.
Reviewed at f56d69e (base 7.0.x): 44c3a4a and the merge of #17071's head, which merges without a conflict. I probed sorts by a property of a collection, with and without a fetch of that collection or of another one, for a bag (a Collection), a set, and a bag that the mapping fetches with fetch: 'join':
- at this head, at #17071 (b42dc27), on 7.0.x (aee4c04) and at the stack top (16bd465), with the setting off and on;
- with Hibernate 7 on 8.0.x (0518f48) and at #17079 (6de9334);
- with a team that has two members and two trophies, one with a member, one with a trophy and one with neither. Members are a bag, trophies a set.
With the setting off, this head returns the instances of #17071 for every query. With the setting on, the changes that the description lists hold:
| Setting on | #17071 | This head |
|---|---|---|
Team.where { … }.sort('members.name').list(), also a projection of name |
2 teams | 4 |
Team.list(sort: 'members.name'), also order('members.name') |
3 rows | 5: the two teams without members kept, and the team with two members twice, as without the setting |
Team.findAllByNameLike('%', [sort: 'members.name', fetch: [members: 'join']]) |
2, members not fetched | 4, members fetched, as at #17079 |
Team.list(sort: 'players.name') with a set and a team without players |
the team left out | kept |
With either setting, Team.where { members { name == 'a' } }.join('members', JoinType.LEFT).list() no longer leaves the team with a fetched bag that holds only a. The bag is now loaded when it's read, with both members. That fixes a wrong collection in the session, so I agree that this one needs no setting.
Needs a fix
1. The bag exception also applies where nothing goes wrong. With the setting on, these now throw UnsupportedOperationException: Cannot sort by a property of [members] … while the query fetches [trophies]:
| Setting on | #17071 | This head | #17079 |
|---|---|---|---|
Team.list(sort: 'members.name', fetch: [trophies: 'join']), also Team.where { … }.join('trophies').list(sort: 'members.name') |
2 teams, trophies fetched complete | throws | 2, trophies complete |
Team.list(sort: 'trophies.title', fetch: [members: 'join']) |
2, the bag holds 4 members for 2 | throws | 2, the bag holds 4 for 2 |
a mapping with members fetch: 'join', and Team.where { … }.sort('trophies.title').list(), also findAllByNameLike |
2, the bag holds 4 for 2 | throws | 2, the bag holds 2 |
The second and third rows are the case that the description explains: a fetched bag can't tell a repeated row of the sort from a repeated element.
In the first row, the bag is the collection that the query sorts by. The collection that it fetches is a set of another class, and #17071 and #17079 return both collections complete. The exception there comes from the rule that a sorted collection is fetched as well when the query fetches another one. The note gives the reason for that rule: Hibernate would fill the fetched collection with the elements of the other one "when both hold elements of the same class". Members and trophies are of different classes, so that reason doesn't apply. Fetching the sorted collection only in that case, and throwing only when a fetched collection is a bag, keeps these queries working.
As the setting becomes the default in 8.1, every such query then throws for an application that upgrades to 8.1. The 8.1 notes have to name the exception as well.
2. With the setting off, a bag that list fetches still holds repeated elements. In Team.list(sort: 'trophies.title', fetch: [members: 'join']), the bag of the team with two members holds 4 members, at #17071 and here. On 7.0.x the list returned the team four times, but it didn't fetch the bag, which then loaded its 2 members. #17071 changed 'join' into a fetch, so this comes from there, and this part fixes it only with the setting on. #17079 holds 4 as well. A mapping with members fetch: 'join' and a where query sorted by trophies.title held 4 on 7.0.x already. list would need the check of point 1 without the setting too, or not fetch a bag next to a sort by another collection.
3. The collections that are loaded after the query are loaded one instance at a time. When criteria use a collection that the query requests JoinType.LEFT for, loadFetchedCollectionsAfterQuery sets every other fetched collection to FetchMode.SELECT. initializeCollections then initializes it for each instance, which is a query per instance and collection. The comment gives the reason as a collection "of the same table" as the joined one. Limiting it to that case would keep the fetch of the others in the query. Otherwise the note should say that such a query loads the other collections with a query for each instance.
4. More public API on 7.0.x, as on the earlier parts. GrailsHibernateUtil gets these new public static methods:
isFetchRequested;- two
checkSortedCollection; keepsSortedCollectionUnfetched;fetchesAnotherCollection;findFetchedCollections;initializeCollections;- two
keepCollectionUnfetched.
5. Add run ci, and resolve the conflicts with 7.0.x, as on the other parts. Locally, the tests of grails-data-hibernate5-core pass at this head (1624 tests, 159 skipped). The stack top returns what this head returns for all of these queries.
Notes
-
On James's comment, I agree with your reply. With the setting on,
Team.list(sort: 'players.name')keeps the team without players here. -
With the setting on,
Team.where { … }.sort('members.name').list()returns 4 teams andcount()5. With the setting off, it's 2 and 3. That's the count of a collection sort, which I raised on #17064. -
With Hibernate 7, at #17079,
Team.where { members { name == 'a' } }.join('members', JoinType.LEFT).join('trophies').list()holds 4 members for 2. I'll raise that on #17079. -
As a fix that changes a result, this needs a second reviewer (#17061).
Description
Part 12 of the split of #16853, as agreed there and in the dev@ thread Fixes that change results on patch and minor branches. Relates to #15589.
Stacked on #17071. Until that merges, this pull request also shows the commits below it. The change to review is the last commit, 44c3a4a.
A sort by a property of a collection joined the collection with an
inner join. It left out the instances whose collection is empty, and
took the place of a fetch that the query requested for the collection.
Criteria on a collection with a requested left join initialized the
collection with only the elements that the criteria matched.
through that join, unless the query asks for the collection to be
fetched, with or without the sort.
collection. When the query also fetches another collection, that one
is loaded once the query has run, because Hibernate would read it
from the columns of the join of the criteria.
rows, and a bag cannot tell a repeated row from a repeated element,
which is why Hibernate does not fetch two bags in one query. When
either collection is a bag, the query throws an
UnsupportedOperationException that names both, where it returned the
bag with repeated elements. A query that projects values loads no
collection and is not affected.
A fetch that a mapping declares counts as fetched only where Hibernate
joins it: not at or beyond hibernate.max_fetch_depth joins from the
queried class, where Hibernate loads it with a query of its own
(review 5868034911 of #16853).
This applies with the grails.gorm.leftJoinAssociationSort setting.
Without it a sort still joins a collection with an inner join, which
leaves out the instances whose collection is empty.
Results that change
grails.gorm.leftJoinAssociationSort, a sort by a property of a collection keeps the instances whose collection is empty and no longer keeps a query from fetching the collection; a query that sorts by one collection and fetches another throws when either is a bag. Without the setting a sort by a collection is unchanged. A requested join of a collection no longer leaves elements out of a fetched collection.The upgrade guide has a note under Grails 7.0.18, and the upgrade skill has the same entry.
Verification
The association sort and join specs of
grails-data-hibernate5-corepass on this branch, withgrails.gorm.leftJoinAssociationSortoff and on, andcodeStylepasses for the touched modules. The full build was not run locally.Contributor Checklist
Please review the following checklist before submitting your pull request. Pull requests that do not meet these requirements may be closed without review.
Issue and Scope
7.0.x): Bug fixes only. No new features or API changes.7.1.x): New features are welcome, but breaking existing APIs must be avoided.8.0.x): Reserved for major changes. Breaking API changes are permitted.Code Quality
./gradlew build --rerun-tasks../gradlew codeStyleand resolved any violations. See Code Style for details.Licensing and Attribution
Documentation