Skip to content

[C++] Instrument TsFile sources in ENABLE_ASAN builds - #1064

Open
ColinLeeo wants to merge 1 commit into
developfrom
fix/1062-cpp-asan-instrumentation
Open

ColinLeeo wants to merge 1 commit into
developfrom
fix/1062-cpp-asan-instrumentation

Conversation

@ColinLeeo

@ColinLeeo ColinLeeo commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #1062.

ENABLE_ASAN=ON builds linked the sanitizer runtime but did not compile TsFile sources with -fsanitize: the flags were appended to CMAKE_CXX_FLAGS before add_subdirectory(third_party) resets that variable, and only the Python binding mode re-applied them. Leaks and invalid frees were still visible through malloc/free interposition, but heap-buffer-overflow, use-after-free and all UBSan checks were inactive for TsFile code.

Changes

  • Store the sanitizer compile flags in TSFILE_SANITIZER_COMPILE_FLAGS and re-apply them after third_party for every ENABLE_ASAN build. Bundled third-party code stays uninstrumented.
  • Link the sanitizer runtime dynamically into libtsfile, the executables and the tools. A macOS dylib cannot carry unresolved __asan_* symbols, and two private runtimes in one process are unsupported, so the -static-libasan branch and the TSFILE_SANITIZER_SHARED_RUNTIME option are removed (cpp/build.sh, cpp/pom.xml, cpp/CLAUDE.md and python/SANITIZER.md updated).
  • Skip LTO in Release builds whenever ASan is enabled, matching the binding build.
  • Add cpp/test/asan_negative_control.cc as a CTest case when ENABLE_ASAN is ON. It performs a deliberate heap-buffer-overflow. A CMake driver script (cpp/test/asan_negative_control.cmake) runs it and passes only when the captured output contains the sanitizer report; the exit status is ignored because ASan aborts on macOS and CTest turns a subprocess abort into a test failure.

Testing

  • macOS (Apple clang, Debug, address,undefined): tsfile_cwrapper.cc.o and libtsfile carry __asan_report_* references and link libclang_rt.asan_osx_dynamic.dylib; ASan runtime initialization currently hangs on the macOS 26 host used here, so runtime checks were done on Linux.
  • Linux (Ubuntu 20.04, GCC 9.4, Debug, same checks): libtsfile.so has undefined __asan_report* symbols and NEEDED libasan.so.5.
  • ctest -R asan_negative_control: passes with ERROR: AddressSanitizer: heap-buffer-overflow, both with the default settings and with ASAN_OPTIONS=abort_on_error=1 (macOS-like abort). The same driver returns 1 with a clear message when pointed at a deliberately uninstrumented build of the control.
  • Full CTest under ASan/UBSan: 1438 of 1439 pass. The only failure is CliRequirements.VersionIncludesConcreteBuildMetadata (commit=unknown), which is caused by the out-of-tree direct CMake build used for this validation (it does not run Maven's version injection); it passes in the Maven-built suite.

The CI matrix (unit-test-cpp.yml, unit-test-cpp-msvc.yml) exercises this through the existing -Denable.asan=ON entries; no workflow change is needed.

@ColinLeeo ColinLeeo added this to the 2.5.1 milestone Nov 3, 2026
@codecov-commenter

codecov-commenter commented Nov 3, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 66.46%. Comparing base (7b96800) to head (6042764).

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #1064   +/-   ##
========================================
  Coverage    66.46%   66.46%           
========================================
  Files          773      773           
  Lines        55808    55808           
  Branches      8948     8948           
========================================
+ Hits         37092    37093    +1     
  Misses       17059    17059           
+ Partials      1657     1656    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The ASan compile flags were appended before add_subdirectory(third_party),
where CMAKE_CXX_FLAGS is reset, and were only re-applied for the Python
binding build. C++-only ASan jobs therefore linked the sanitizer runtime
without compiling any instrumentation: leaks were still visible through
malloc/free interposition, but heap-buffer-overflow, use-after-free and all
UBSan checks were inactive for TsFile code.

- Keep the sanitizer compile flags in TSFILE_SANITIZER_COMPILE_FLAGS and
  re-apply them after third_party for every ENABLE_ASAN build; third-party
  code stays uninstrumented.
- Link the sanitizer runtime dynamically into libtsfile, the executables
  and the tools. A macOS dylib cannot carry unresolved __asan_* symbols,
  and two private runtimes in one process are unsupported, so the
  static-runtime branch is removed along with TSFILE_SANITIZER_SHARED_RUNTIME.
- Skip LTO in Release builds whenever ASan is enabled, matching the binding
  build.
- Add an asan_negative_control CTest case that deliberately writes past a
  heap allocation. A CMake driver checks the captured output for the
  sanitizer report instead of relying on the exit status, because ASan
  aborts on some platforms and CTest turns a subprocess abort into a
  failure. A build that loses instrumentation fails the test.

Fixes #1062.
@ColinLeeo
ColinLeeo force-pushed the fix/1062-cpp-asan-instrumentation branch from abe0d0d to 6042764 Compare November 3, 2026 06:08
@ColinLeeo

Copy link
Copy Markdown
Contributor Author

The first CI run failed on unit-test (macos-latest, Debug, Asan): the sanitizer did report the control's out-of-bounds write (see the log), but macOS ASan aborts by default and CTest treats a subprocess abort as a failure even when PASS_REGULAR_EXPRESSION matches. The test now uses a CMake driver that inspects the captured output and ignores the exit status; validated on Linux both with and without abort_on_error=1, and it still fails when the control is built without instrumentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++] ENABLE_ASAN builds do not instrument TsFile sources

2 participants