Skip to content

Run the subqueries of a where query before MongoDB and Neo4j compare with them - #16875

Open
borinquenkid wants to merge 6 commits into
8.0.xfrom
fix/where-subquery-mongo-neo4j-standalone-8.0.x
Open

borinquenkid wants to merge 6 commits into
8.0.xfrom
fix/where-subquery-mongo-neo4j-standalone-8.0.x

Conversation

@borinquenkid

Copy link
Copy Markdown
Member

Fixes #16809

Replaces #16824, which sat on top of a chain of PRs. This branch has only this change, on 8.0.x.

Problem

Person.where { age > avg(age) } and the other aggregate subqueries documented for where put a DetachedCriteria in the comparison. SQL datastores run it as a subquery. MongoDB and Neo4j compared the property with the criteria object and returned no rows, without an error (the issue reports a MissingMethodException for some forms). inList / notIn against a subquery, exists and the all / some forms had the same gap.

Changes

  • Query.resolveSubqueries(Junction) (new, protected, @since 8.0.1) runs each subquery and returns a copy of the criteria that holds the results. The criteria passed in are not changed, so a reused where query runs its subqueries again each time.
    • ==, !=, >, >=, <, <= take the one value the subquery returns (an empty result compares with null);
    • inList takes every row, and notIn becomes a negated inList;
    • a comparison against a subquery that returns more than one row, exists, notExists and the all / some forms throw InvalidDataAccessResourceUsageException naming the form or the property.
  • MongoQuery.executeQuery and Neo4jQuery.executeQuery call it first.
  • Docs: where reference and upgrade guide section 26.16.

Two queries instead of one $expr pipeline or a Cypher subquery: the result is a snapshot taken when the query runs, which is what a where query already is on the SQL datastores' first read. A native $expr / CALL translation can replace the substitution later without changing behavior.

Tests

  • AggregateSubquerySpec for MongoDB (12) and Neo4j (9): avg with >, >=, <, <=, the .of { } closure, max, min, sum, ==, !=, negation, ||, inList, notIn, the multi-row and exists errors, and a repeat run that sees new data. All fail without the change (10 of 10 and 7 of 7 when I checked before adding the later cases).
  • Full suites: grails-data-mongodb-core 1036 tests and grails-data-neo4j-core 623, 0 failures; grails-datastore-core 348, 0 failures. codeStyle, validateRepositoryConventions and rat pass.
  • PITest (feat(build-logic): add opt-in PITest mutation testing support and skill #16819, cross-module): 43 mutants in the new Query code, 43 killed.

Not in this PR

  • A native $expr / Cypher translation (see above).
  • count as an aggregate comparison is untested here and not documented.

…with them

A where query such as Person.where { age > avg(age) } puts a DetachedCriteria
in the comparison. MongoDB and Neo4j compared against that object and matched
no rows. Query.resolveSubqueries runs the subqueries first and returns a copy
of the criteria holding their results, so the query is correct and runs the
subquery again on every execution. Comparisons take the single value of the
subquery, in and notIn take every row, and exists, notExists and the all and
some forms throw an InvalidDataAccessResourceUsageException that names the
form.

Fixes #16809
Copilot AI balanced review requested due to automatic review settings November 5, 2026 22:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approved. A single-result MongoDB query still uses the unresolved criteria, and a subquery inside an association block is not run. An empty scalar subquery is also turned into a comparison with null.

@Override
protected List executeQuery(final PersistentEntity entity, final Junction criteria) {
protected List executeQuery(final PersistentEntity entity, final Junction queryCriteria) {
final Junction criteria = resolveSubqueries(queryCriteria);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get() still calls getMongoQuery(), which reads the original criteria. inList('firstName', Person.where { age > 40 }.property('firstName')).get() returns null, while list() returns Homer and Barney. Build the single-result filter from the resolved criteria.

…get(), and match no row for an empty subquery

A single-result MongoDB query built its filter from the original criteria, so
inList over a subquery returned null from get() while list() returned rows. A
subquery inside an association block was never run, because an association
block is not a junction. A comparison with the value of an empty subquery was
turned into a comparison with null, which matches rows whose value is missing;
in SQL the comparison is unknown and matches no row, so it now matches none.

Claude-Session: https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL
@borinquenkid

Copy link
Copy Markdown
Member Author

Pushed de7100c, by point:

  • get() on MongoDB: the single-result branch built its filter from the original criteria. It now builds it from the resolved ones, so inList('firstName', Person.where { age > 46 }.property('firstName')).get() returns Barney, and null when the subquery selects nothing.
  • Association block: resolveSubqueries now descends into an association block and returns a copy of it with the subquery results, so address { inList('city', City.where { }.property('name')) } finds the person who lives in Paris, and notIn the others. Checked on MongoDB with an embedded address and on Neo4j with a related node (owner { inList('firstName', ...) }).
  • Empty scalar subquery: a comparison with an empty subquery no longer becomes a comparison with null. It matches no row, as in SQL, where the comparison is unknown. Inside a not { } the result is every row rather than none, because the datastore negates the empty match. SQL gives none there too, so that case still differs.

Each new feature fails with the old code and passes with the change (the Neo4j one only for the association block, as n.age = null already matched nothing there). The datastore-core, MongoDB core and Neo4j core suites and codeStyle pass. The upgrade note and the where reference say the three behaviors.

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approved. Negating a comparison with an empty subquery matches every row. A subquery also runs on the default connection when the outer query uses another one. A single-result query and an association block now resolve their subqueries.

…o row

A comparison with the value of an empty subquery is unknown in SQL, and so is its negation. The
empty comparison was replaced by a criterion that matches no row, so inside not { } it matched
every row. Resolving the subqueries now tracks whether the criterion sits inside an odd number of
negations and, there, replaces the comparison with one that matches every row, so that the negation
matches none and the rows of a mixed and/or follow SQL three-valued logic.

Claude-Session: https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL
@borinquenkid

Copy link
Copy Markdown
Member Author

@jamesfredley Fixed in 4a94553, on top of de7100c, for the negation (head now 92fcdc8, which adds the documentation of the connection). not { gt('age', <empty subquery>) } now matches no row, and so do not { not { ... } } and not { <empty comparison> || x }; not { <empty comparison> && x } returns the rows where x is false, which is what SQL returns. The comparison is unknown in SQL and so is its negation, so resolving the subqueries now tracks whether the comparison sits inside an odd number of negations and, there, replaces it with a criterion that matches every row, so that the negation matches none.

The MongoDB and Neo4j AggregateSubquerySpec have a feature for it: the where form, not { }, not { not { } }, an or and an and, with a row whose age is null, which stays out. It fails without the change on MongoDB. The upgrade guide and the where reference say it.

The connection: Query in grails-datastore-core has no connection, and Hibernate 7 runs a subquery inside the same SQL, so there is nothing in this layer to fix. A subquery runs on the connection of its own criteria, and one written as avg(age).of { } is built by the compiler transform with the default connection. The two documents now say so and show withConnection for the subquery (92fcdc8). Making the subquery inherit the connection of the criteria that holds it belongs in AbstractDetachedCriteria and needs a two-connection spec for each datastore, which does not exist yet, so it is #16886.

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Negating a comparison with an empty subquery no longer matches every row, and a subquery runs on its own connection. Not approving. A subquery that returns a single null is treated as a null comparison, so it matches rows whose value is missing.

… matches no row (#16875)

A subquery that returns a single null, such as the age of a person whose age is not set, was turned into a comparison with null, which matched the rows whose value is missing. In SQL the comparison is unknown, as it is for a subquery without a row, so it now matches no row, and so does its negation. MongoDB and Neo4j specs cover the comparison, the not-equal and the negated form; the MongoDB one fails without the change.

Claude-Session: https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL
@borinquenkid

Copy link
Copy Markdown
Member Author

@jamesfredley Fixed in 6568e77. A subquery that returns one null is treated like one that returns no row: the comparison and its negation match no row. The MongoDB and Neo4j AggregateSubquerySpec have a case for it; the MongoDB one fails without the change.

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A comparison with a subquery that returns one null now matches no row. Not approving. A subquery that returns an entity is still compared as that entity, and a membership subquery that contains null still throws.

}
Object value = rows.get(0);
if (criterion instanceof Equals) {
return new Equals(property.getProperty(), value);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Neo4j, persist two pets that share one owner, then compare owner with Pet.where { name == 'Rex' }.property('owner'). The list is empty. The subquery returns the owner instance, and this comparison uses that instance instead of its identifier. Keep the empty and single-null handling. Convert the value the same way a literal comparison does.

throw unsupportedSubquery(criterion.getClass().getSimpleName());
}
if (criterion instanceof In in && in.getSubquery() != null) {
return new In(in.getProperty(), subqueryValues(in.getSubquery()));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

inList('age', Person.where { ... }.property('age')) throws when one selected age is null. The list is passed through with the null in it, and Neo4j calls getClass() on that null. Keep running the subquery on the query that holds it. Convert a null without calling getClass().

@borinquenkid
borinquenkid requested a review from matrei November 12, 2026 14:12
…ompares its identifier, and a null in a subquery's list is never a match

An association compared with a subquery that returns an association, such as
owner == Pet.where { name == 'Rex' }.property('owner'), was given the instance as the value of the
comparison, and Neo4j matched no row. The value of the subquery is converted the way a value given
in the comparison is: an entity is compared by its identifier.

A subquery that selects a null, such as the age of a node that has none, made an in list fail on
Neo4j with a NullPointerException, and made MongoDB match the documents without the field, because
its $in with a null does. A null in the list is never equal to a value, as in SQL: it adds no match,
and for the values that are not in the list the test is unknown, so its negation matches no row, as
for a comparison with one null. The values of the subquery are copied first, as a store may return a
list that loads its elements when it is read. Neo4j also leaves a null alone when it converts the
values of a list.

The Neo4j spec of the comparison with one null used AggSubPerson, a class of the MongoDB specs, and
did not compile: it uses AggSubNode, with an age that may be missing.

Claude-Session: https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL
@borinquenkid

Copy link
Copy Markdown
Member Author

@jamesfredley Both cases reproduced, and are fixed in the new commit. AggregateSubquerySpec has them for Neo4j and for MongoDB (I could run MongoDB's now); the new rows fail without the change.

  • owner compared with Pet.where { name == 'Rex' }.property('owner'). The subquery returned the owner instance, and the rewritten comparison was given the instance, not its identifier, so Neo4j got an object as the Cypher parameter (ID(m_0) = $1) and found no row. A comparison given an entity by hand goes through resolveIdIfEntity first, and the rewrite did not. It does now, for the single value of the subquery; the empty and single-null handling is as it was. The spec has the instance given by hand as well, as the reference, and MongoDB passes the row too.
  • inList('age', ...property('age')) with one null selected. On Neo4j the conversion of the list called getClass() on the null and threw. Making that null-safe was not enough: MongoDB then matched the document without an age, because its $in with a null matches a missing field, where Neo4j and SQL do not ({"age": {"$in": [null, 45]}} is what it ran). A null in the list is never equal to a value in SQL, so the rewrite drops it from the list, and for the values that are not in the list the test is unknown, so its negation matches no row, as for the comparison with one null. The spec has the list, its negation, and a double negation, on both stores. The values of the subquery are copied first: MongoDB returns a list that loads when it is read, and my first check for the null ran before it had loaded.

Something else I found in the Neo4j spec of the last commit: the comparison with one null used AggSubPerson, a class of the MongoDB specs, so the test sources of the Neo4j module did not compile at 6568e774f1. It uses AggSubNode now, with an age that may be missing.

The Neo4j module (633 tests) and the MongoDB module (1,046 tests) pass, and so does codeStyle.

Branch policy: the throws that are listed in Query.java, for a subquery comparison that returns more than one row and for a subquery form a store does not support, were already in this PR; this commit adds none. Where the PR belongs is still open; I have not moved it.

If you find another shape that does not reproduce first time, a failing spec on a throwaway branch, or the few lines you ran, would help me a lot.

@testlens-app

testlens-app Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

🔎 No tests executed 🔎

🏷️ Commit: 7798193
▶️ Tests: 0 executed
⚪️ Checks: 46/46 completed


Learn more about TestLens at testlens.app/docs.

@matrei matrei left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 7798193 (base 8.0.x). It conflicts with the current 8.0.x (0518f48) only in upgrading80x.adoc. CI has never run on it: it carries no ci.

The rounds with James have made the resolution careful. The negation and the nulls in a list follow SQL in every shape I traced, and the 20 MongoDB and 17 Neo4j features fail on the merge base (bca56da) and pass here. Three things before it can land, though.

Needs a fix

1. deleteAll and updateAll still compare with the criteria object, and deleteAll can remove every row. Only executeQuery resolves the subqueries:

  • On MongoDB, deleteAll and updateAll build their filter in buildNativeDocumentQueryFromCriteria (MongoSession and MongoCodecSession), through the no-argument getMongoQuery(), from the criteria as they are.
  • On Neo4j, deleteAll builds its Cypher from getBaseQuery(), and only the count it returns goes through singleResult().

With three people aged 10, 40 and 40:

8.0.x This head Expected
where { age != max(age) }.list(), MongoDB and Neo4j MongoDB [], Neo4j all three [Bart] [Bart]
where { age != max(age) }.deleteAll(), MongoDB deletes all three, returns 3 deletes all three, returns 3 deletes Bart
The same on Neo4j deletes all three, returns 3 deletes all three, returns 1 deletes Bart
where { age > avg(age) }.updateAll(age: 99) and where { age == max(age) }.updateAll(age: 99), both stores 0 changed 0 changed 2 changed

The deletion is as old as the subquery forms, but after this change the read and the delete of the same where query disagree, and on Neo4j the count deleteAll returns no longer says what it deleted. One way: let the public getMongoQuery() and Neo4j's getBaseQuery() build from resolveSubqueries(criteria), which covers every caller. Please add updateAll and deleteAll rows to both AggregateSubquerySpecs, with a != that must leave rows in place.

2. The MongoDB half belongs on 7.0.x. I ran this pull request's MongoDB AggregateSubquerySpec on unchanged 7.0.x (aee4c04). The only changes were registering its classes the 7.0.x way (manager.domainClasses.addAll(...)) and making address and age nullable, since 7.0.x validates an embedded null. All 20 features fail:

  • the comparisons return no rows;
  • != against an empty subquery returns every row;
  • notIn throws Queries of type NotIn are not supported;
  • exists isn't rejected by name.

By the rule in CONTRIBUTING.md (#17061), a wrong result like this goes to the oldest branch that has it. 7.0.x doesn't build Neo4j (its settings.gradle has no Neo4j module), so Neo4j stays on 8.0.x. The Query part applies to 7.0.x as it is. One MongoQuery hunk doesn't, so the MongoDB part needs a port. I'd open a 7.0.x pull request with Query and MongoDB, and leave Neo4j here, on top of its merge-up.

3. Merge 8.0.x and run CI. The upgrade note conflicts because 8.0.x now has a section per release: it goes under === Grails 8.0.1, without the number 26.14. The description calls it 26.16. On 7.0.x it goes under === Grails 7.0.18 in upgrading60x.adoc, with the same entry in the upgrade skill. As a fix that changes a result, it needs two reviewers. Please remove no ci, so that CI runs once on the merged branch.

Note

  1. Query.resolveSubqueries is a new protected method of a public class on a patch line. It has to be visible to MongoQuery and Neo4jQuery, in other packages, so that's fine. A line in its Javadoc saying it's for query implementations would keep applications from relying on it. On 7.0.x, its @since becomes 7.0.18.

Verified as correct

  • list(), count() and get() with a subquery give the expected rows on MongoDB and Neo4j at this head.
  • The criteria passed to resolveSubqueries are left as they are, so a where query that runs twice runs its subqueries again ("the subquery runs again each time the query runs").

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Datastore: Support aggregate subqueries in MongoDB and Neo4j where queries

4 participants