Repository navigation
Conversation
Add a coordinated sanitizer build mode, -Dtsfile.sanitizer=<checks>, that instruments libtsfile and every Cython extension with the same checks. The C++ library gets -fsanitize on compile and on the shared-library link and keeps debug info and frame pointers; setup.py builds each extension with -O1 -g -fno-omit-frame-pointer, links the sanitizer runtime, and verifies with nm/readelf that every artifact is instrumented. Sanitizer builds never produce a wheel. The sanitizer compile flags must be re-applied after the bundled third-party build resets CMAKE_CXX_FLAGS; previously they were discarded and only the link step saw them. The C++-only ASan mode keeps its historical static-runtime setup so an instrumented libtsfile cannot mix runtimes with the test binary. Fix ownership defects the new checks expose: - free STRING/TEXT value buffers and use the length-aware binary getter for BLOB so embedded NUL bytes survive; - free the writer-side schema structs, not only their members; - free result-set metadata and schema arrays on every path; - free the property array on the C wrapper's bad_alloc path; - add tsfile_free_buffer so bindings release buffers with the allocating module's CRT. Add subprocess ownership scenarios (tree/table/dataset/Arrow reads, value types and NULLs, early termination, exception cleanup, repeated open/close, reader/result lifetime, file-object adapter), two positive controls, narrow suppressions, and a Linux CI job. Leak reports are attributed by allocation stack so only leaks reaching TsFile modules fail the run. Document build and run commands, runtime preloading (libasan before libstdc++), symbolization, and the suppression policy.
- Skip the ordinary pytest execution in the with-sanitizer Maven profile: the instrumented extensions abort on import unless the sanitizer runtime is preloaded, which Maven's test execution does not do, so `mvn verify -Dtsfile.sanitizer=address` failed before running a test. - Document the complete ASan/LSan command in python/CLAUDE.md, including TSFILE_SANITIZER (without it test_sanitizer_ownership.py skips silently) and the ASAN/LSAN options used by CI. - Add tsfile_free_table_schema_array()/tsfile_free_device_schema_array() so bindings release reader-owned schema arrays with libtsfile's allocator, and release partial conversion results when a schema conversion raises mid-loop. - Point the C API value-getter docs at tsfile_free_buffer() and document the binary getter's NULL-cell behavior. - Note the nm/binutils wheel requirement and the test/verify limitation in python/SANITIZER.md.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements #1027.
-Dtsfile.sanitizer=address) that instrumentslibtsfileand every Cython extension with the same ASan/LSan check set. C++ gets-fsanitizeon compile and on the shared-library link with debug info and frame pointers;setup.pybuilds every extension (includingpython_random_access_read_file.cc) with-O1 -g -fno-omit-frame-pointer, links the sanitizer runtime, and verifies all artifacts withnm/readelf. Sanitizer builds refuse to produce wheels.CMAKE_CXX_FLAGSand never restored, so TsFile sources were linked against the runtime but not compiled with instrumentation. The bindings mode now re-applies them afterthird_party; the C++-only ASan mode keeps its historical static-runtime setup to avoid mixing static and dynamic runtimes.bad_allocpath. Addstsfile_free_bufferso bindings release buffers with the allocating module's CRT.python/SANITIZER.md.Why
C++ sanitizer jobs did not exercise the bindings, so native ownership defects across the Python -> Cython -> C wrapper -> C++ chain were invisible. All binding leaks fixed here were present before this change.
Testing
-Dtsfile.sanitizer=address -Dbuild.type=RelWithDebInfo:BUILD SUCCESS;check_sanitizer_build.pyreports all six binaries instrumented (compile references plus linked runtime).tree_fullfail withtsfile_result_set_get_value_by_index_stringin the allocation stack; restoring the fix returns the suite to green.Reviewer notes
TSFILE_SANITIZER_SHARED_RUNTIMEkeeps the sanitizer runtime dynamic in the shared library so one preloaded runtime serves Python, the extensions, andlibtsfile. The C++-only ASan jobs do not enable it.LD_PRELOADmust listlibasanfirst andlibstdc++second; otherwise GCC's__cxa_throwinterceptor aborts the first time a C++ exception is thrown. This is documented and used in CI.nmandreadelf(binutils), installed in the CI job.Follow-up
CMAKE_CXX_FLAGSreset, intentionally out of scope here).Supersedes #1047 (same content; the source branch was renamed so repository branch names stay descriptive).