You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Expand Java/C++ cross-language compatibility matrix to full coverage - #1068
The Java/C++ compatibility matrix now covers every (type, encoding, compression)
combination both implementations can write, at four data scales, in both
directions — and building it out surfaced and fixed two real cross-language
format bugs.
Changes
Add a shared spec (compat/compatibility-matrix.csv) consumed by both the
Java and C++ tests, so the two matrices cannot drift.
Replace the table-only Java/C++ compatibility tests with a single
spec-driven EncodingCompressionCompatibilityTest on each side, covering
both models (table + tree), all six compressions, and four data scales
(10 / 1k / 100k / 1M rows) with page sizes that cross page boundaries at
every scale.
Implement the C++ codecs that were missing so every Java-writable
combination is now C++-readable: FLOAT/DOUBLE RLE, FLOAT/DOUBLE GORILLA_V1,
DATE/TIMESTAMP ZIGZAG/SPRINTZ, BOOLEAN RLE.
Fix two cross-language format bugs found by the expanded matrix:
INT64 RLE run values were read little-endian in C++ but written
big-endian by Java/IoTDB.
The SPRINTZ FIR predictor used a 128-bit intermediate in C++, diverging
from Java's 32/64-bit wrapping multiply on large values.
Narrow the compatibility workflow trigger to the binary-format surface
(main source, the spec, build files) so routine PRs don't pay the ~30 minute
matrix; the coverage/self-check tests still run in the normal unit suite.
Report the per-run coverage/gap matrix in the CI job summaries.
Validation
Java generates 3260 fixtures; C++ reads all 3260 (0 skipped).
… scales
Close the Java/C++ compatibility gaps so every Java-writable
(type, encoding, compression) combination is readable by C++, and
expand the shared fixture matrix to four data-scale tiers with
page-size coverage.
- Add C++ FLOAT/DOUBLE RLE and GORILLA_V1 encoders/decoders.
- Wire DATE/TIMESTAMP into ZIGZAG/SPRINTZ and BOOLEAN into RLE.
- Fix INT64 RLE decoder endianness: Java stores run values big-endian.
- Fix SPRINTZ FIR predictor to match Java's 32/64-bit wrapping multiply.
- Replace the table-only Java/C++ tests with a spec-driven
EncodingCompressionCompatibilityTest consumed by both sides from
compat/compatibility-matrix.csv; cover tree + table models, four
scales (10/1k/100k/1M rows), and page-boundary sizes.
- Narrow the compatibility workflow trigger to the binary-format
surface and report the coverage/gap matrix in CI.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Gorilla V1 fails to propagate read and write errors, allowing truncated streams to produce fabricated values and allocation failures to silently truncate encoded output.
Expands Apache TsFile’s Java/C++ compatibility testing toward the shared, bidirectional coverage requested in #1057, while adding missing codecs and correcting format mismatches.
Changes:
Replaces table-only tests with spec-driven table and tree fixtures, capability checks, and multiple data scales.
Adds C++ floating-point RLE and Gorilla V1 support, extends type mappings, and fixes RLE byte order and SPRINTZ prediction.
Narrows CI triggers and publishes generation and validation summaries.
…fers
DecoderFactory::free() only released the placement-new'ed object memory
without running the decoder's destructor, leaking the heap buffers held by
decoders such as Int32RleDecoder (and the FLOAT/DOUBLE RLE wrappers around
it). Move the destructor call into free(), mirroring EncoderFactory::free(),
and drop the now-redundant explicit ~Decoder() calls at the reader call
sites.
- Propagate bit-read and write errors in the Gorilla V1 encoder/decoder
instead of fabricating values or silently truncating output on failure.
- Match Java Math.round exactly in java_round (bit-exact for large doubles)
instead of floor(x + 0.5).
- Check decoder support for every cpp_read case in the C++ self-check.
- Report writer-only (not-mine) gaps in the coverage summary.
- Restore BOOLEAN/RLE null coverage in the shared matrix.
- Validate Java table fixtures inline to bound memory at 1M rows.
- Include the Java module POMs and the compatibility harness in the CI
trigger.
(double)INT64_MAX rounds up to 2^63, so a naive static_cast on a value at
that boundary is out of range. Clamp first, matching the saturation the
previous floor(x+0.5) implementation performed via its r >= 2^63 guard.
The full type-by-encoding cartesian product runs at 10/1000/100000 rows;
at 1M rows it uses a per-type representative encoding set (PLAIN/TS_2DIFF/
RLE/GORILLA/DICTIONARY). Cross-language compatibility is already fully
covered by the smaller scales, and the 1M tier only stresses page/chunk/
index machinery at scale, so this keeps the generation and validation jobs
within the 30-minute CI budget (the previous full 1M cartesian timed out).
Restore the full type-by-encoding cartesian product at all four data
scales (including 1M rows) and instead shard the generation across parallel
CI jobs so each job stays within its timeout.
- generateFixtures/GenerateFixtures take a shardIndex/shardCount and only
emit the cases whose index falls into that shard.
- validateFixtures/ValidateFixtures skip the manifest completeness check
when sharded (each shard's manifest is a partial subset).
- The CI matrix runs one generate/validate job per shard, uploading and
downloading per-shard artifacts.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The predictor change corrupts legacy C++ SPRINTZ decoding, sharded validation can accept incomplete fixture manifests, and floating-point RLE overflow formats lack test coverage.
Test RLE floating-point overflow and raw-IEEE formats
cpp/src/encoding/encoder_factory.h:110
The new FLOAT/DOUBLE RLE support is exercised only with runs and default profiles, whose values never overflow after scaling by 100. Consequently, neither the scaled-overflow nor raw-IEEE bitmap format in the new wrappers is tested, and there are no dedicated RLE floating-point codec tests. Add shared RLE fixtures using the existing ts2diff profile (large finite values, NaN and infinities), so both directions exercise these formats and their page resets.
…l spec
The previous sharding relaxation skipped the missing-fixture check entirely
when shardCount > 1, so a manifest containing only one expected fixture could
pass. Now the validator computes the expected subset using the same
shardIndex % shardCount partition the generator uses, and checks the manifest
against that subset in both directions. CI passes shardIndex to the
validation jobs.
camel_detail::java_math_round used floor(value + 0.5), which diverges from
Java's Math.round(double) for large doubles (|value| >= 2^52). Replace it
with the same bit-exact translation already used by java_round in the
TS_2DIFF/RLE encoders, so every rounding site matches Java.
Add shared table/tree fixtures for scaled overflow, raw IEEE values and page transitions, with Java and C++ codec regression tests. Clear stale overflow bitmaps when resetting the Java floating-point decoder.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #1057.
The Java/C++ compatibility matrix now covers every (type, encoding, compression)
combination both implementations can write, at four data scales, in both
directions — and building it out surfaced and fixed two real cross-language
format bugs.
Changes
compat/compatibility-matrix.csv) consumed by both theJava and C++ tests, so the two matrices cannot drift.
spec-driven
EncodingCompressionCompatibilityTeston each side, coveringboth models (table + tree), all six compressions, and four data scales
(10 / 1k / 100k / 1M rows) with page sizes that cross page boundaries at
every scale.
combination is now C++-readable: FLOAT/DOUBLE RLE, FLOAT/DOUBLE GORILLA_V1,
DATE/TIMESTAMP ZIGZAG/SPRINTZ, BOOLEAN RLE.
big-endian by Java/IoTDB.
from Java's 32/64-bit wrapping multiply on large values.
(main source, the spec, build files) so routine PRs don't pay the ~30 minute
matrix; the coverage/self-check tests still run in the normal unit suite.
Validation
spotless:checkclean.