Skip to content

fix(store): harden MultiListIterator close and next - #3445

Open
vaidikgarg-bit wants to merge 2 commits into
apache:masterfrom
vaidikgarg-bit:fix/multilist-iterator-close-contract
Open

vaidikgarg-bit wants to merge 2 commits into
apache:masterfrom
vaidikgarg-bit:fix/multilist-iterator-close-contract

Conversation

@vaidikgarg-bit

@vaidikgarg-bit vaidikgarg-bit commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Purpose of the PR

#3437 made MultiListIterator.close() close every sub-iterator when it runs before the first hasNext(). Four problems remained in the same class:

  1. close() closes the sub-iterators with no try/finally. When one close() throws, the remaining sub-iterators stay open and the list is never cleared.
  2. next() dereferences innerIterator, which is only set by hasNext()/isValid(). Calling next() first, or after hasNext() has returned false, throws NullPointerException. Repeated next() calls without hasNext() also never move past the current sub-iterator.
  3. close() leaves innerIterator pointing at the source it just closed. The next hasNext() or isValid() calls into that closed source and can close it again.
  4. hasNext() and isValid() probe the current sub-iterator again on every call, so a child whose hasNext() advances (as the dedup iterators did before fix(store): keep pending rows in dedup iterators #3424) loses rows when the caller probes twice before next().

Main Changes

  • Add a closed flag, as UnionFilterIterator and IntersectionFilterIterator have. Once it is set, hasNext() and isValid() return false without touching any source, and a second close() does nothing.
  • close() drops its reference to the current sub-iterator and keeps going when a sub-iterator's close() throws a RuntimeException or an Error (RocksDBScanIterator.close() rethrows both). It clears the list and then rethrows the first failure, with any later failures added as suppressed exceptions.
  • next() calls hasNext() first, so it moves on to the next sub-iterator by itself and throws NoSuchElementException when nothing is left, including after close().
  • hasNext() records that it found an element, and next() and repeated hasNext()/isValid() calls use that record instead of probing the current sub-iterator again. Each sub-iterator's hasNext() runs once per row it returns, plus once when it runs out, so a child whose hasNext() advances does not lose rows.

Not changed here: position() has no closed check, so after closing mid-iteration it now throws NullPointerException instead of reading the closed source (before iteration it already threw on master). A close failure raised while hasNext() moves past an exhausted sub-iterator is also left as is.

Verifying these changes

  • Trivial rework / code cleanup without any test coverage. (No Need)
  • Already covered by existing tests, such as (please modify tests here).
  • Need tests and can be verified as follows:
    • six new tests in FilterIteratorContractTest: a RuntimeException close failure before iteration, an Error close failure before iteration, a close failure during iteration, next() without hasNext(), hasNext()/isValid()/next()/close() after closing mid-iteration, and a child whose hasNext() advances on every call. All six fail on master.
    • store-server-test (203), store-core-test (113) and store-rocksdb-test (19) pass locally with this branch on top of 8e327b8f3, and mvn editorconfig:check is clean for the changed modules.

Does this PR potentially affect the following parts?

Documentation Status

Select one option and provide the documentation location when applicable.

  • Doc - TODO: required documentation is pending; complete it before merging.
  • Doc - Done: documentation is included here or linked below.
  • Doc - No Need: no user-visible documentation is affected.

Documentation files in this PR or paired hugegraph-doc PR:

Visual overview

MultiListIterator close and next contract

Explicit close() marks the iterator closed, detaches the current source, attempts the current and remaining source closes despite RuntimeException/Error, clears the list, and rethrows the first failure with later failures suppressed when a failure occurred. Subsequent probes stop and repeated close() is a no-op. next() checks availability itself and throws NoSuchElementException when exhausted or closed. position() and advance-time close failures remain outside this fix; the additional hasNext() call per row has no bounded performance measurement here. This illustrates the current diff, not independent test, CI or release acceptance. Reviewed: 2026-10-31.

close() now closes every sub-iterator and clears the list even when one
close() throws a RuntimeException or Error, then rethrows the first
failure with later ones suppressed. next() moves on to the next
sub-iterator itself and throws NoSuchElementException instead of
NullPointerException when nothing is left. After close(), hasNext() and
isValid() return false without probing any source, and a second close()
does nothing.

close apache#3444
@codecov

codecov Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.67%. Comparing base (608cb1d) to head (4bfa917).
⚠️ Report is 27 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3445      +/-   ##
============================================
+ Coverage     57.27%   57.67%   +0.40%     
+ Complexity     6577     2588    -3989     
============================================
  Files           776      779       +3     
  Lines         68881    69161     +280     
  Branches       9431     9506      +75     
============================================
+ Hits          39449    39889     +440     
+ Misses        25804    25611     -193     
- Partials       3628     3661      +33     
Flag Coverage Δ *Carryforward flag
commons 96.37% <ø> (ø) Carriedforward from d8c2e42
hstore 36.09% <ø> (+<0.01%) ⬆️
pd 52.80% <ø> (+0.64%) ⬆️ Carriedforward from d8c2e42
server_hbase 42.18% <ø> (+<0.01%) ⬆️ Carriedforward from d8c2e42
server_memory 55.99% <ø> (+0.01%) ⬆️ Carriedforward from d8c2e42
server_rocksdb 42.66% <ø> (+<0.01%) ⬆️ Carriedforward from d8c2e42
store 35.90% <ø> (+2.75%) ⬆️

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Score: 9/10. Summary: close() closes every sub-iterator and clears the list even when one close() throws, next() advances through hasNext() and throws NoSuchElementException when exhausted or closed, and the five new FilterIteratorContractTest cases cover each path and pass in pd_store / store at this head.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: Repeated iterator probes can drop index query results. Evidence: traced scanIndex through indexIntersection and the stateful filter iterators.


@Override
public <T> T next() {
if (!hasNext()) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

‼️ next() probes the current child again after the caller has already called hasNext(). In multi-branch scanIndex with DEDUP, indexIntersection supplies InAccurateUnionFilterIterator or InAccurateIntersectionIterator children, whose hasNext() consumes and replaces the current match. The second probe skips the first row; a single-row child can instead be exhausted and cause NoSuchElementException. Please cache the prepared state between hasNext() and next(), and add a regression test using a child whose hasNext() advances.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, done. hasNext() now records that it found an element, and next() and repeated hasNext()/isValid() calls use that record instead of probing the child again. testMultiListProbesAdvancingSourceOncePerRow uses a child whose hasNext() advances and replaces the pending row. It fails on the previous head, and on master too, since master's hasNext() probes the child again on every call. It passes now. This also removes the extra per-row probe I noted in the description.

On impact: the extra probe in next() came from my previous commit. Since #3424 the two InAccurate* iterators keep their pending element (if (current != null) return true;), and the other children I checked in BusinessHandlerImpl are also safe to probe twice, so I don't think it dropped rows with the current children. It did add a probe per row and relied on that idempotence, and now it doesn't.

hasNext() now records that it found an element, and next() and repeated
hasNext()/isValid() calls use that record instead of probing the
current sub-iterator again. A child whose hasNext() advances no longer
loses rows, and next() no longer adds a probe per row.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Store MultiListIterator leaks sub-iterators on close failure, NPEs on bare next(), and probes sources after close

3 participants