Skip to content

fix(hibernate5): the total of a paged criteria with groupProperty counts the groups - #17054

Open
borinquenkid wants to merge 8 commits into
7.0.xfrom
fix/h5-grouped-total-count-7.0.x
Open

borinquenkid wants to merge 8 commits into
7.0.xfrom
fix/h5-grouped-total-count-7.0.x

Conversation

@borinquenkid

Copy link
Copy Markdown
Member

The totalCount of a paged criteria that uses a groupProperty projection replaced the grouped projection with a row count, so it disagreed with the grouped page it belongs to: 3 groups from 6 rows reported a total of 6. It now counts the groups.

This is the Hibernate 5 half of #16843, split out for 7.0.x after the dev@ discussion on fixes that change results: the total contradicts the page it belongs to, so the fix goes to the oldest branch that has the bug. The Hibernate 7 half stays on #16843 against 8.0.x.

  • PagedResultList counts the groups of a grouped criteria; an ungrouped criteria counts rows as before.
  • GroupedCriteriaTotalCountSpec (core) and GroupedCriteriaTotalCountScenarioSpec (example app) pin 3 groups from 6 rows, with and without a restriction, and the unchanged ungrouped count. Two of the three core rows fail on the unfixed code.
  • The upgrade guide has a 7.0.18 note, because a result changes.

Fixes #14431

https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL

…nts the groups

The totalCount of a paged criteria that uses a groupProperty projection replaced the grouped projection
with a row count, so it disagreed with the grouped page it belongs to. It now counts the groups.

Fixes #14431

Claude-Session: https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

7.0.18 is staged without this change, so the note belongs to the next patch release.

Claude-Session: https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requested changes. The total of a grouped page now counts the groups by loading every group into memory.

}
totalCriteria.setProjection(impl.getProjection());
if (impl.getProjection() != null && impl.getProjection().isGrouped()) {
totalCriteria.setProjection(impl.getProjection());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A page of two groups should report how many groups there are without loading them all. This loads every group and then counts the list. Count the groups in the database.

…n the database

The total of a page of a criteria grouped by one property loaded every group to count the list. It is
now the distinct values of the property, counted by the database, plus the group of the null values,
which SQL groups but does not count as a value. A projection with more than one grouped property is
counted from its groups, as before.

Claude-Session: https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL
@borinquenkid

Copy link
Copy Markdown
Member Author

@jamesfredley: the total of a page grouped by one property is counted by the database now, as count(distinct property) plus a second query for the group of the null values, which SQL groups but count(distinct) skips. A projection with more than one grouped property is still counted from its groups, since count(distinct a, b) is not portable. GroupedTotalCountSqlSpec captures the SQL of the total: two select count(...) statements, none that returns the groups; it fails without the change. GroupedCriteriaTotalCountSpec has the null group and a two-property grouping.

https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requested changes. A page grouped by two properties still counts the groups by loading them.

}
Criteria grouped = totalCriteria(session, impl);
grouped.setProjection(projection);
return grouped.list().size();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A page of two groups, grouped by city and name, should report how many groups there are without loading them. This loads every group and then counts the list. Count those groups in the database.

@borinquenkid

Copy link
Copy Markdown
Member Author

@jamesfredley: a page grouped by city and name now reports its total without loading the groups. For several grouped properties the database counts the rows of the grouped query, select count(*) from (<the grouped query>), with the criteria's own conditions and joins; a single grouped property still uses count(distinct) plus the group of the nulls. I did not use count(distinct a, b), which not every database accepts.

Before asking for another look, I ran nine shapes (city and name, name and city, three properties, a property and an aggregate, a property of an association, a property and one of an association, two properties under a condition, no rows, all-null groups) and compared each total with the size of the same query unpaged. 12 rows; each reaches the database with one count query (two for a single property), and 10 fail without this change.

https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requested changes. A page grouped by an embedded value counts one group twice.

if (projection instanceof ProjectionList && ((ProjectionList) projection).getLength() == 1) {
only = ((ProjectionList) projection).getProjection(0);
}
if (only instanceof PropertyProjection && ((PropertyProjection) only).isGrouped()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A page grouped by an embedded value of two columns should report how many groups there are. A group whose columns are all empty is counted twice. Count that group once.

@borinquenkid

Copy link
Copy Markdown
Member Author

@jamesfredley: a page grouped by an embedded value of two columns counted the group whose columns are all empty twice. A group by one property was counted as its distinct values plus a group for "the property is null", and for an embedded value a distinct count already includes the group of rows whose columns are all null, so the extra one counted it again. Every grouped page is now counted the same way, as the rows of its grouped query (select count(*) from (<the grouped query>)), so SQL decides what a group is: all-null values are one group, whatever the number of columns, and a group by one property needs one query instead of two.

Before asking for another look, I added three shapes to the matrix: an embedded value, an embedded value with a property, and an embedded value under a condition, over rows whose street and zip are both set, only one set, and both empty. 15 rows, each total compared with the size of the same query unpaged; 4 fail without this change (the embedded ones and the two that now count in one query).

https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL

@borinquenkid

Copy link
Copy Markdown
Member Author

@matrei: the note is back under === Grails 7.0.18 in upgrading60x.adoc, as your correction asks; the 7.0.19 heading is gone. If 7.0.18 is staged again before this merges, I move it. Only the upgrade note changed.

https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requested changes. The count of a grouped page fails when query caching is on.

}

@Override
protected String preprocessSQL(String sql, QueryParameters parameters, SessionFactoryImplementor factory,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A page grouped by two properties should report how many groups there are when query caching is on. Reading the total throws, because the count is cached as if it were the grouped rows. Do not cache that count as the grouped query.

@borinquenkid

Copy link
Copy Markdown
Member Author

@jamesfredley: I could not reproduce the failure of the count of a grouped page with query caching on, at 9c4633f. GroupedTotalCountQueryCacheSpec (pushed now) reads the total of a page grouped by one property, by two, and by two with a rowCount(), twice each, on an entity whose mapping caches, with cache: true, with cache: false, and with cache(true) inside the block. Each row first checks that the page itself used the query cache, then that the total is the number of groups and that the count was not put in the cache. It also covers an unpaged grouped list before and after the total (they do not share a cached result), a group saved after the page was fetched, and a page past the last group. All of them pass, and so does the whole grails-data-hibernate5-core module. Could you give me the setting or the call that throws for you, so I can add that row? I do not want to change the count for a case I cannot see fail.

The case I did find is another route: a where query or a DetachedCriteria grouped by a property takes the base PagedResultList of grails-datamapping-core, which counts rows, on plain 7.0.x as well. I filed it as #17107 and left it out of this pull request. A distinct(...) projection and a lone rowCount() have the same row-count total on plain 7.0.x (#17108); they are not grouped, so they are out of scope here too.

https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requested changes. The count of a grouped page still fails when query caching is on.

* whose values are all null is one group, whatever the number of columns of the grouped properties.
*/
private int countGroups(Session session, CriteriaImpl impl, Projection projection) {
CriteriaImpl grouped = (CriteriaImpl) totalCriteria(session, impl);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A grouped list of cities is cached, then the same grouping is paged and the total is read. That should be 3. It throws, because the count is looked up as if it were the cached cities. Do not cache that count as the grouped query.

@borinquenkid

Copy link
Copy Markdown
Member Author

@jamesfredley: merged 7.0.x into the branch; the only conflict was the upgrade guide, where I kept both sections.

On the count that throws with query caching on: I could not reproduce it at bbe0b66, and I tried your sequence again. A grouped list of cities is cached, then the same grouping is paged and its total is read, and the total is 3. I ran it twelve ways, on an entity with cache true in its mapping and on one without: cache: true as an argument on both calls, cache(true) in the block, in one session and in two, the page first and the list after, with and without order('city'), and with a page larger than the number of groups. Each returns 3, and the count is not put in the cache. GroupedTotalCountQueryCacheSpec covers the same sequence.

What setup throws for you: the database, the cache provider (mine is H2 with Ehcache), and whether the query cache comes from grails.hibernate.cache.queries, from the argument or from the block? The exception message would show where the cached list is read as the count.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

3 participants