Skip to content

Serialize IDisk::checkAccess: disks created in parallel share the probe file - #2812

Open
filimonov wants to merge 1 commit into
antalya-26.6from
fix/antalya-26.6/disk-access-check-mutex
Open

filimonov wants to merge 1 commit into
antalya-26.6from
fix/antalya-26.6/disk-access-check-mutex

Conversation

@filimonov

Copy link
Copy Markdown
Member

Two tables whose custom disks sit on one backing disk fail to attach at server start: Cannot open file …/clickhouse_access_check_<server uuid> or Can't unlink file clickhouse_access_check_… (FILE_DOESNT_EXIST) While checking access for disk …. Every disk probes its access with that one file name, and since #2573 custom disks are created outside storage_policies_mutex, so the probes of two disks over the same store overwrite and unlink each other's file. A process-wide mutex around the probe in IDisk::checkAccess; config disks are created one after another anyway, so nothing else changes.

Seen on the 26.8 port #2805 in test_filesystem_cache (tables test and test_size_in_name, cache disks over hdd_blob, server log at 12:15:44: three probes on hdd_blob within 1 ms, two fail on unlink; https://altinity-build-artifacts.s3.amazonaws.com/PRs/2805/a2147e556977e195b2d117c61b61d1274e080436/integration_tests_amd_asan_ubsan_db_disk_old_analyzer_5_8/job.log). antalya-26.6 carries 2573 since 2026-11-04; its integration suite has no test_cache_file_size_in_name, so the race has not shown in its CI.

Testing

  • New unit test DiskAccessCheck.ProbesOfDisksOverOneDirectoryDoNotOverlap (src/Disks/tests/gtest_disk_access_check.cpp): two disks over one directory, started in parallel; the probe of the second must wait for the first. Fails before the fix (two probes at once), passes after, 20/20 repeats.
  • Release unit_tests_dbms --gtest_filter='CAS*' on this branch: 2822 passed, 0 failed, 2 disabled.

Related: #2573
Related: #2805

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Tables on custom disks that share a backing disk no longer fail to load at server start with FILE_DOESNT_EXIST on clickhouse_access_check_<server uuid>: the disk access probes are serialized.

Documentation entry for user-facing changes

  • No documentation needed.

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

…be file

Every disk probes its access with the same file name, clickhouse_access_check_<server uuid>.
Since custom disks are created outside storage_policies_mutex
(#2573), two tables whose custom disks sit on one
backing disk are started in parallel at server start, and their probes overwrite and unlink
each other's file: the tables fail to attach with FILE_DOESNT_EXIST "While checking access
for disk ...". Seen in test_filesystem_cache on
#2805 (tables test and test_size_in_name, cache
disks over hdd_blob). A process-wide mutex around the probe; config disks are created one
after another anyway.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
(cherry picked from commit 086efc9)
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Workflow [PR], commit [bb9bf73]

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant