Repository navigation
perf(calcite): rank string sort keys by bytes, walk reversed chunks backward - #575
Merged
Merged
Conversation
dfa1
force-pushed
the
perf/sort-string-reject
branch
from
October 20, 2026 19:03
4f79813 to
7e819dc
Compare
…ackward ORDER BY ... LIMIT built a Java String for every row's string key just to compare it with the worst row kept, and almost every row loses. Now a row whose first key is a string is compared with that worst key's bytes first (VarBinArray#compareUnsigned): FSST decodes only up to the first differing byte (Decompressor#compareUnsigned), dictionary and offset arrays compare in place. A row that loses is skipped without a String; any other row takes the existing path, so results are unchanged by construction. Byte order equals String.compareTo order when the bound is ASCII: up to the first differing byte both strings decode the same, and a non-ASCII byte of the row, valid UTF-8 or not, decodes to a character above every ASCII one (a 5M-case fuzz found no disagreement). A non-ASCII bound falls back to building every row. A column stored in the opposite order to the query (eventIdentifier DESC on ascending ids) has every row beat the worst kept, so nothing was rejected. A chunk whose five evenly spaced sample rows rank in strictly reverse order is now walked backward: its best rows come first and the rest are rejected. A sample row the filter rejects is never read, as reading it can fail (a u64 past BIGINT), so such a chunk is walked forward. Walking backward admits a row equal to the worst, since equal keys rank by scan position. Walking forward goes through RowSelection#nextSelected, keeping #580's skipping of the rows a dictionary filter rejects. 100M-row cashflow file, jdbc:vortex: driver, warm JVM (median of the last 8 of 15 runs, 3 interleaved rounds), against main 9bc1209: order by eventIdentifier desc limit 10 5616 -> 1680 ms (3.3x) order by ISIN limit 10 1273 -> 962 ms (one round) Tests: calcite SortStringKeysTest covers ties across chunk edges walked both ways, a chunk under five rows, and a filtered-out sample row holding a u64 past BIGINT (it fails without the fix). The vortex-jni cross-check sorts Rust-written FSST, dictionary and plain strings over three Rust chunks, with "é" rows and ties at the bound, against Calcite's own sort. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dfa1
force-pushed
the
perf/sort-string-reject
branch
from
October 20, 2026 19:06
7e819dc to
435b181
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ORDER BY … LIMITon a string column built a JavaStringper row just to compare it with the worst row kept, and almost every row loses that comparison.VarBinArray#compareUnsigned.Decompressor#compareUnsigned).String.String.compareTo. A 5M-case fuzz with malformed UTF-8 found no disagreement.u64pastBIGINT), so such a chunk is walked forward instead.RowSelection#nextSelected, so perf(calcite): walk only the rows a dictionary filter selects #580's skipping of rows a dictionary filter rejects is kept.Benchmark. 100M-row file through the
jdbc:vortex:driver, warm JVM, median of 15 runs, againstmain(9bc12092):order by eventIdentifier desc limit 10order by ISIN limit 10Review fixes. A subagent review raised these, all addressed:
nextSelectedfor the forward walk.SortStringKeysTestis new. It covers ties across chunk edges, a chunk under 5 rows, and the filtered-sample regression, which fails without the fix.FSST symbol lengths are deliberately not validated at load: Rust doesn't check them either.
./mvnw verifypasses (9239 tests).🤖 Generated with Claude Code