Repository navigation
Run the subqueries of a where query before MongoDB and Neo4j compare with them - #16875
borinquenkid wants to merge 6 commits into
Conversation
…with them
A where query such as Person.where { age > avg(age) } puts a DetachedCriteria
in the comparison. MongoDB and Neo4j compared against that object and matched
no rows. Query.resolveSubqueries runs the subqueries first and returns a copy
of the criteria holding their results, so the query is correct and runs the
subquery again on every execution. Comparisons take the single value of the
subquery, in and notIn take every row, and exists, notExists and the all and
some forms throw an InvalidDataAccessResourceUsageException that names the
form.
Fixes #16809
jamesfredley
left a comment
There was a problem hiding this comment.
Not approved. A single-result MongoDB query still uses the unresolved criteria, and a subquery inside an association block is not run. An empty scalar subquery is also turned into a comparison with null.
| @Override | ||
| protected List executeQuery(final PersistentEntity entity, final Junction criteria) { | ||
| protected List executeQuery(final PersistentEntity entity, final Junction queryCriteria) { | ||
| final Junction criteria = resolveSubqueries(queryCriteria); |
There was a problem hiding this comment.
get() still calls getMongoQuery(), which reads the original criteria. inList('firstName', Person.where { age > 40 }.property('firstName')).get() returns null, while list() returns Homer and Barney. Build the single-result filter from the resolved criteria.
…get(), and match no row for an empty subquery A single-result MongoDB query built its filter from the original criteria, so inList over a subquery returned null from get() while list() returned rows. A subquery inside an association block was never run, because an association block is not a junction. A comparison with the value of an empty subquery was turned into a comparison with null, which matches rows whose value is missing; in SQL the comparison is unknown and matches no row, so it now matches none. Claude-Session: https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL
|
Pushed de7100c, by point:
Each new feature fails with the old code and passes with the change (the Neo4j one only for the association block, as |
jamesfredley
left a comment
There was a problem hiding this comment.
Not approved. Negating a comparison with an empty subquery matches every row. A subquery also runs on the default connection when the outer query uses another one. A single-result query and an association block now resolve their subqueries.
…o row
A comparison with the value of an empty subquery is unknown in SQL, and so is its negation. The
empty comparison was replaced by a criterion that matches no row, so inside not { } it matched
every row. Resolving the subqueries now tracks whether the criterion sits inside an odd number of
negations and, there, replaces the comparison with one that matches every row, so that the negation
matches none and the rows of a mixed and/or follow SQL three-valued logic.
Claude-Session: https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL
…tion of its own criteria Claude-Session: https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL
|
@jamesfredley Fixed in 4a94553, on top of de7100c, for the negation (head now 92fcdc8, which adds the documentation of the connection). The MongoDB and Neo4j The connection: |
jamesfredley
left a comment
There was a problem hiding this comment.
Negating a comparison with an empty subquery no longer matches every row, and a subquery runs on its own connection. Not approving. A subquery that returns a single null is treated as a null comparison, so it matches rows whose value is missing.
… matches no row (#16875) A subquery that returns a single null, such as the age of a person whose age is not set, was turned into a comparison with null, which matched the rows whose value is missing. In SQL the comparison is unknown, as it is for a subquery without a row, so it now matches no row, and so does its negation. MongoDB and Neo4j specs cover the comparison, the not-equal and the negated form; the MongoDB one fails without the change. Claude-Session: https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL
|
@jamesfredley Fixed in 6568e77. A subquery that returns one null is treated like one that returns no row: the comparison and its negation match no row. The MongoDB and Neo4j |
jamesfredley
left a comment
There was a problem hiding this comment.
A comparison with a subquery that returns one null now matches no row. Not approving. A subquery that returns an entity is still compared as that entity, and a membership subquery that contains null still throws.
| } | ||
| Object value = rows.get(0); | ||
| if (criterion instanceof Equals) { | ||
| return new Equals(property.getProperty(), value); |
There was a problem hiding this comment.
On Neo4j, persist two pets that share one owner, then compare owner with Pet.where { name == 'Rex' }.property('owner'). The list is empty. The subquery returns the owner instance, and this comparison uses that instance instead of its identifier. Keep the empty and single-null handling. Convert the value the same way a literal comparison does.
| throw unsupportedSubquery(criterion.getClass().getSimpleName()); | ||
| } | ||
| if (criterion instanceof In in && in.getSubquery() != null) { | ||
| return new In(in.getProperty(), subqueryValues(in.getSubquery())); |
There was a problem hiding this comment.
inList('age', Person.where { ... }.property('age')) throws when one selected age is null. The list is passed through with the null in it, and Neo4j calls getClass() on that null. Keep running the subquery on the query that holds it. Convert a null without calling getClass().
…ompares its identifier, and a null in a subquery's list is never a match
An association compared with a subquery that returns an association, such as
owner == Pet.where { name == 'Rex' }.property('owner'), was given the instance as the value of the
comparison, and Neo4j matched no row. The value of the subquery is converted the way a value given
in the comparison is: an entity is compared by its identifier.
A subquery that selects a null, such as the age of a node that has none, made an in list fail on
Neo4j with a NullPointerException, and made MongoDB match the documents without the field, because
its $in with a null does. A null in the list is never equal to a value, as in SQL: it adds no match,
and for the values that are not in the list the test is unknown, so its negation matches no row, as
for a comparison with one null. The values of the subquery are copied first, as a store may return a
list that loads its elements when it is read. Neo4j also leaves a null alone when it converts the
values of a list.
The Neo4j spec of the comparison with one null used AggSubPerson, a class of the MongoDB specs, and
did not compile: it uses AggSubNode, with an age that may be missing.
Claude-Session: https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL
|
@jamesfredley Both cases reproduced, and are fixed in the new commit.
Something else I found in the Neo4j spec of the last commit: the comparison with one null used The Neo4j module (633 tests) and the MongoDB module (1,046 tests) pass, and so does Branch policy: the throws that are listed in If you find another shape that does not reproduce first time, a failing spec on a throwaway branch, or the few lines you ran, would help me a lot. |
🔎 No tests executed 🔎🏷️ Commit: 7798193 Learn more about TestLens at testlens.app/docs. |
matrei
left a comment
There was a problem hiding this comment.
Reviewed at 7798193 (base 8.0.x). It conflicts with the current 8.0.x (0518f48) only in upgrading80x.adoc. CI has never run on it: it carries no ci.
The rounds with James have made the resolution careful. The negation and the nulls in a list follow SQL in every shape I traced, and the 20 MongoDB and 17 Neo4j features fail on the merge base (bca56da) and pass here. Three things before it can land, though.
Needs a fix
1. deleteAll and updateAll still compare with the criteria object, and deleteAll can remove every row. Only executeQuery resolves the subqueries:
- On MongoDB,
deleteAllandupdateAllbuild their filter inbuildNativeDocumentQueryFromCriteria(MongoSessionandMongoCodecSession), through the no-argumentgetMongoQuery(), from the criteria as they are. - On Neo4j,
deleteAllbuilds its Cypher fromgetBaseQuery(), and only the count it returns goes throughsingleResult().
With three people aged 10, 40 and 40:
| 8.0.x | This head | Expected | |
|---|---|---|---|
where { age != max(age) }.list(), MongoDB and Neo4j |
MongoDB [], Neo4j all three |
[Bart] |
[Bart] |
where { age != max(age) }.deleteAll(), MongoDB |
deletes all three, returns 3 | deletes all three, returns 3 | deletes Bart |
| The same on Neo4j | deletes all three, returns 3 | deletes all three, returns 1 | deletes Bart |
where { age > avg(age) }.updateAll(age: 99) and where { age == max(age) }.updateAll(age: 99), both stores |
0 changed | 0 changed | 2 changed |
The deletion is as old as the subquery forms, but after this change the read and the delete of the same where query disagree, and on Neo4j the count deleteAll returns no longer says what it deleted. One way: let the public getMongoQuery() and Neo4j's getBaseQuery() build from resolveSubqueries(criteria), which covers every caller. Please add updateAll and deleteAll rows to both AggregateSubquerySpecs, with a != that must leave rows in place.
2. The MongoDB half belongs on 7.0.x. I ran this pull request's MongoDB AggregateSubquerySpec on unchanged 7.0.x (aee4c04). The only changes were registering its classes the 7.0.x way (manager.domainClasses.addAll(...)) and making address and age nullable, since 7.0.x validates an embedded null. All 20 features fail:
- the comparisons return no rows;
!=against an empty subquery returns every row;notInthrowsQueries of type NotIn are not supported;existsisn't rejected by name.
By the rule in CONTRIBUTING.md (#17061), a wrong result like this goes to the oldest branch that has it. 7.0.x doesn't build Neo4j (its settings.gradle has no Neo4j module), so Neo4j stays on 8.0.x. The Query part applies to 7.0.x as it is. One MongoQuery hunk doesn't, so the MongoDB part needs a port. I'd open a 7.0.x pull request with Query and MongoDB, and leave Neo4j here, on top of its merge-up.
3. Merge 8.0.x and run CI. The upgrade note conflicts because 8.0.x now has a section per release: it goes under === Grails 8.0.1, without the number 26.14. The description calls it 26.16. On 7.0.x it goes under === Grails 7.0.18 in upgrading60x.adoc, with the same entry in the upgrade skill. As a fix that changes a result, it needs two reviewers. Please remove no ci, so that CI runs once on the merged branch.
Note
Query.resolveSubqueriesis a new protected method of a public class on a patch line. It has to be visible toMongoQueryandNeo4jQuery, in other packages, so that's fine. A line in its Javadoc saying it's for query implementations would keep applications from relying on it. On 7.0.x, its@sincebecomes 7.0.18.
Verified as correct
list(),count()andget()with a subquery give the expected rows on MongoDB and Neo4j at this head.- The criteria passed to
resolveSubqueriesare left as they are, so a where query that runs twice runs its subqueries again ("the subquery runs again each time the query runs").
Fixes #16809
Replaces #16824, which sat on top of a chain of PRs. This branch has only this change, on 8.0.x.
Problem
Person.where { age > avg(age) }and the other aggregate subqueries documented forwhereput aDetachedCriteriain the comparison. SQL datastores run it as a subquery. MongoDB and Neo4j compared the property with the criteria object and returned no rows, without an error (the issue reports aMissingMethodExceptionfor some forms).inList/notInagainst a subquery,existsand theall/someforms had the same gap.Changes
Query.resolveSubqueries(Junction)(new, protected,@since 8.0.1) runs each subquery and returns a copy of the criteria that holds the results. The criteria passed in are not changed, so a reusedwherequery runs its subqueries again each time.==,!=,>,>=,<,<=take the one value the subquery returns (an empty result compares withnull);inListtakes every row, andnotInbecomes a negatedinList;exists,notExistsand theall/someforms throwInvalidDataAccessResourceUsageExceptionnaming the form or the property.MongoQuery.executeQueryandNeo4jQuery.executeQuerycall it first.wherereference and upgrade guide section 26.16.Two queries instead of one
$exprpipeline or a Cypher subquery: the result is a snapshot taken when the query runs, which is what awherequery already is on the SQL datastores' first read. A native$expr/CALLtranslation can replace the substitution later without changing behavior.Tests
AggregateSubquerySpecfor MongoDB (12) and Neo4j (9):avgwith>,>=,<,<=, the.of { }closure,max,min,sum,==,!=, negation,||,inList,notIn, the multi-row andexistserrors, and a repeat run that sees new data. All fail without the change (10 of 10 and 7 of 7 when I checked before adding the later cases).grails-data-mongodb-core1036 tests andgrails-data-neo4j-core623, 0 failures;grails-datastore-core348, 0 failures.codeStyle,validateRepositoryConventionsandratpass.Querycode, 43 killed.Not in this PR
$expr/ Cypher translation (see above).countas an aggregate comparison is untested here and not documented.