Repository navigation
fix(store): harden MultiListIterator close and next - #3445
vaidikgarg-bit wants to merge 2 commits into
Conversation
close() now closes every sub-iterator and clears the list even when one close() throws a RuntimeException or Error, then rethrows the first failure with later ones suppressed. next() moves on to the next sub-iterator itself and throws NoSuchElementException instead of NullPointerException when nothing is left. After close(), hasNext() and isValid() return false without probing any source, and a second close() does nothing. close apache#3444
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3445 +/- ##
============================================
+ Coverage 57.27% 57.67% +0.40%
+ Complexity 6577 2588 -3989
============================================
Files 776 779 +3
Lines 68881 69161 +280
Branches 9431 9506 +75
============================================
+ Hits 39449 39889 +440
+ Misses 25804 25611 -193
- Partials 3628 3661 +33
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: no. Score: 9/10. Summary: close() closes every sub-iterator and clears the list even when one close() throws, next() advances through hasNext() and throws NoSuchElementException when exhausted or closed, and the five new FilterIteratorContractTest cases cover each path and pass in pd_store / store at this head.
imbajin
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: Repeated iterator probes can drop index query results. Evidence: traced scanIndex through indexIntersection and the stateful filter iterators.
|
|
||
| @Override | ||
| public <T> T next() { | ||
| if (!hasNext()) { |
There was a problem hiding this comment.
next() probes the current child again after the caller has already called hasNext(). In multi-branch scanIndex with DEDUP, indexIntersection supplies InAccurateUnionFilterIterator or InAccurateIntersectionIterator children, whose hasNext() consumes and replaces the current match. The second probe skips the first row; a single-row child can instead be exhausted and cause NoSuchElementException. Please cache the prepared state between hasNext() and next(), and add a regression test using a child whose hasNext() advances.
There was a problem hiding this comment.
Thanks, done. hasNext() now records that it found an element, and next() and repeated hasNext()/isValid() calls use that record instead of probing the child again. testMultiListProbesAdvancingSourceOncePerRow uses a child whose hasNext() advances and replaces the pending row. It fails on the previous head, and on master too, since master's hasNext() probes the child again on every call. It passes now. This also removes the extra per-row probe I noted in the description.
On impact: the extra probe in next() came from my previous commit. Since #3424 the two InAccurate* iterators keep their pending element (if (current != null) return true;), and the other children I checked in BusinessHandlerImpl are also safe to probe twice, so I don't think it dropped rows with the current children. It did add a probe per row and relied on that idempotence, and now it doesn't.
hasNext() now records that it found an element, and next() and repeated hasNext()/isValid() calls use that record instead of probing the current sub-iterator again. A child whose hasNext() advances no longer loses rows, and next() no longer adds a probe per row.
Purpose of the PR
#3437 made
MultiListIterator.close()close every sub-iterator when it runs before the firsthasNext(). Four problems remained in the same class:close()closes the sub-iterators with notry/finally. When oneclose()throws, the remaining sub-iterators stay open and the list is never cleared.next()dereferencesinnerIterator, which is only set byhasNext()/isValid(). Callingnext()first, or afterhasNext()has returnedfalse, throwsNullPointerException. Repeatednext()calls withouthasNext()also never move past the current sub-iterator.close()leavesinnerIteratorpointing at the source it just closed. The nexthasNext()orisValid()calls into that closed source and can close it again.hasNext()andisValid()probe the current sub-iterator again on every call, so a child whosehasNext()advances (as the dedup iterators did before fix(store): keep pending rows in dedup iterators #3424) loses rows when the caller probes twice beforenext().Main Changes
closedflag, asUnionFilterIteratorandIntersectionFilterIteratorhave. Once it is set,hasNext()andisValid()returnfalsewithout touching any source, and a secondclose()does nothing.close()drops its reference to the current sub-iterator and keeps going when a sub-iterator'sclose()throws aRuntimeExceptionor anError(RocksDBScanIterator.close()rethrows both). It clears the list and then rethrows the first failure, with any later failures added as suppressed exceptions.next()callshasNext()first, so it moves on to the next sub-iterator by itself and throwsNoSuchElementExceptionwhen nothing is left, including afterclose().hasNext()records that it found an element, andnext()and repeatedhasNext()/isValid()calls use that record instead of probing the current sub-iterator again. Each sub-iterator'shasNext()runs once per row it returns, plus once when it runs out, so a child whosehasNext()advances does not lose rows.Not changed here:
position()has no closed check, so after closing mid-iteration it now throwsNullPointerExceptioninstead of reading the closed source (before iteration it already threw on master). A close failure raised whilehasNext()moves past an exhausted sub-iterator is also left as is.Verifying these changes
FilterIteratorContractTest: aRuntimeExceptionclose failure before iteration, anErrorclose failure before iteration, a close failure during iteration,next()withouthasNext(),hasNext()/isValid()/next()/close()after closing mid-iteration, and a child whosehasNext()advances on every call. All six fail on master.store-server-test(203),store-core-test(113) andstore-rocksdb-test(19) pass locally with this branch on top of8e327b8f3, andmvn editorconfig:checkis clean for the changed modules.Does this PR potentially affect the following parts?
Documentation Status
Select one option and provide the documentation location when applicable.
Doc - TODO: required documentation is pending; complete it before merging.Doc - Done: documentation is included here or linked below.Doc - No Need: no user-visible documentation is affected.Documentation files in this PR or paired hugegraph-doc PR:
Visual overview
Explicit close() marks the iterator closed, detaches the current source, attempts the current and remaining source closes despite RuntimeException/Error, clears the list, and rethrows the first failure with later failures suppressed when a failure occurred. Subsequent probes stop and repeated close() is a no-op. next() checks availability itself and throws NoSuchElementException when exhausted or closed. position() and advance-time close failures remain outside this fix; the additional hasNext() call per row has no bounded performance measurement here. This illustrates the current diff, not independent test, CI or release acceptance. Reviewed: 2026-10-31.