Repository navigation
Conversation
added 3 commits
October 28, 2026 12:32
The probe added in 1.7.0 iterated `rs.replicas`, a field that `vshard.router.info()` does not return (vshard 0.1.25 reports `master` and `replica` per replicaset). The loop never ran, so the container was reported ready while a replicaset master was still in the vshard handshake, and the first `crud.truncate` in spring-data integration tests failed with VHANDSHAKE_NOT_COMPLETE. Require every replicaset master to be `available`, no replica to be `unreachable`, and no unreachable or unknown buckets.
The previous probe also required no `unreachable` replica and zero unknown buckets. In CI (tests-crud-integration 3.5.0, job 122708866002) it returned false for the whole 120 s window in RepositoryViaJavaConfigTest while bootstrap had succeeded, so the container failed to start. Requiring only an `available` master is not enough either. On a stand the router state stayed `storage-002 master=unreachable replica=available` for 120 s: the router connected to storage-002-a before its storage was configured, the vshard handshake failed with "storage is not configured", and vshard closed the connection (vshard/replicaset.lua conn_vconnect_check_or_close). Only a request to that master reopens it, and discovery and failover talk to the replica. The first asynchronous crud map call reopens it but does not wait for the new handshake, which is the VHANDSHAKE_NOT_COMPLETE seen in #139. Make a synchronous `callrw('vshard.storage.buckets_count')` to the master of every replicaset: it reopens a closed connection and waits for the handshake. Require all calls to succeed and every master to be `available`, and log call errors, replica and bucket states.
checks 287c88e ("exclude nil function name from error messages",
2026-10-28) drops the " to nil" suffix from "unexpected argument"
errors. crud 1.5.0 depends on `checks >= 3.1.0-1`, so the test cluster
picks it up on every fresh build, and testSelectWrongParameters fails
on master with "... select.lua:320: unexpected argument opts.coolOption".
Accept the message with and without the suffix.
(cherry picked from commit 52700595fa36ebb3c0d0de569438bc636a93e1f0)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The vshard readiness probe added in 1.7.0 (39e9dea) iterated
rs.replicas, a field thatvshard.router.info()does not return: vshard 0.1.25 reportsmasterandreplicaper replicaset. The loop never ran, soVshardClusterContainerwas reported ready while a replicaset master was still in the vshard handshake, and the firstcrud.truncatein spring-data integration tests failed withVHANDSHAKE_NOT_COMPLETE.Seen in CI of #139: run 40517896080 (
TarantoolTemplateViaJavaConfigTest) and run 40500876277 (RepositoryViaPropertiesTest), both intests-crud-integration (3.5.0); both passed on rerun.Why it fails only sometimes: it is a race between the vshard handshake and the first test request.
vshard.storage._call('info')asynchronously and treats the connection as not ready until the reply arrives (vshard 0.1.25,vshard/replicaset.lua:120-130).crud.truncateis a map call to all replicaset masters withis_async = true(crud 1.5.0,crud/common/call.lua:130). For async calls vshard does not wait for the handshake and fails at once withVHANDSHAKE_NOT_COMPLETE(vshard/replicaset.lua:576-588); sync calls do wait.MISSING_MASTER(crud 1.5.0,crud/common/call.lua:74-90), so the error goes straight to the test.truncatedepends on how busy the storage is. On CI four storages and two routers share one container on the runner, and bootstrap is retried meanwhile (Duplicate key ... "_bucket"in the job log), so the outcome changes from run to run.storage-002-awas still incomplete tens of seconds after the container started in run 40517896080 is not established (possibly a router reconnect; router logs are not in the CI output). The fixed probe closes the window either way.The probe now requires every replicaset master to be
available, no replica to beunreachable, and no unreachable or unknown buckets.Seen in CI of #139,
tests-crud-integration (3.5.0), first attempt; both passed on rerun:TarantoolTemplateViaJavaConfigTest.testKVTemplateUpdateByIdRepositoryViaPropertiesTest.testPageablePageNumberGreaterCursorПроверка готовности vshard из 1.7.0 (39e9dea) перебирала
rs.replicas— поле, которогоvshard.router.info()не возвращает: vshard 0.1.25 отдаёт по каждому репликасетуmasterиreplica. Цикл не выполнялся, иVshardClusterContainerобъявлялся готовым, пока ведущий репликасета ещё проходил рукопожатие vshard. Первыйcrud.truncateв интеграционных тестах spring-data падал сVHANDSHAKE_NOT_COMPLETE.Падения на CI #139: run 40517896080 (
TarantoolTemplateViaJavaConfigTest) и run 40500876277 (RepositoryViaPropertiesTest), оба вtests-crud-integration (3.5.0); оба прошли при перезапуске.Почему падает не каждый раз: это гонка между рукопожатием vshard и первым запросом теста.
vshard.storage._call('info')и считает соединение неготовым, пока не придёт ответ (vshard 0.1.25,vshard/replicaset.lua:120-130).crud.truncate— рассылка на ведущие всех репликасетов сis_async = true(crud 1.5.0,crud/common/call.lua:130). Для асинхронных вызовов vshard рукопожатия не ждёт и сразу возвращаетVHANDSHAKE_NOT_COMPLETE(vshard/replicaset.lua:576-588); синхронные вызовы ждут.MISSING_MASTER(crud 1.5.0,crud/common/call.lua:74-90), поэтому ошибка сразу уходит в тест.truncate, зависит от загрузки хранилища. На CI четыре хранилища и два роутера работают в одном контейнере на общем раннере, а начальная раскладка данных (bootstrap) в это время идёт с повторами (Duplicate key ... "_bucket"в логе задания), поэтому исход от прогона к прогону разный.storage-002-aне завершилось и через десятки секунд после старта контейнера, не установлено (возможно, роутер переподключался; журналов роутера в выводе CI нет). Исправленная проверка закрывает это окно в любом случае.Теперь проверка требует: ведущий каждого репликасета —
available, ни одной репликиunreachable, нет недоступных и неизвестных бакетов.Падения на CI #139,
tests-crud-integration (3.5.0), первая попытка; оба прошли при перезапуске:TarantoolTemplateViaJavaConfigTest.testKVTemplateUpdateByIdRepositoryViaPropertiesTest.testPageablePageNumberGreaterCursorI haven't forgotten about:
Related issues:
See also #139