Repository navigation
Conversation
added 8 commits
October 27, 2026 16:25
When a server accepts the TCP connection but does not send the greeting, the greeting timer closed the channel without setting a close reason. The close handler then treated it as a close by the server: it fired CLOSE_BY_REMOTE with "Connection closed by server" next to the TimeoutException of the connect future. The pool listens to both, so every failed attempt was handled twice and the reconnect chains multiplied: in one run of the new pool test, 12 attempts to a silent server scheduled 66 reconnects instead of one per attempt. Fail the connect promise with TimeoutException first and close the channel only if that succeeded, so a connection whose greeting has just arrived is never closed by the timer. Close it with a dedicated reason and do not fire a close event for it. Name the address and the timeout in the message. The timer also stayed armed when the server closed the connection before the greeting, and then fired during the next attempt and closed its channel: in two runs of the new test the second attempt was closed 220 and 326 ms after it started, with a 600 ms timeout. Let the timer close the channel only of the attempt it was scheduled for.
When a server, or a proxy in front of it, accepted the TCP connection and closed it before the greeting, the close was reported twice: the connect future failed with ConnectionClosedException, and the close handler also fired CLOSE_BY_REMOTE. The pool listens to both and scheduled a reconnect for each, so the reconnect chains multiplied: in one run of the new pool test, 50 attempts in 3 s scheduled 1325 reconnects. A connection that has not received the greeting has no requests in flight and no registered watchers, and its failure already reaches the caller through the connect future. Fire close events only for a connection that has received the greeting.
With graceful shutdown enabled, IProtoClientImpl.connect() called
watch("box.shutdown", this::shutdownEventCallback) on every attempt.
watch() appends the callback to the existing watcher and nothing
removes it, so a client that kept reconnecting to an unavailable server
accumulated one callback per attempt, and the event after a successful
connect ran the callback once per previous attempt.
The watcher itself survives reconnects and is re-subscribed by
updateWatchers(), so add the callback only on the first connect.
The pool.connect.time sample was stopped only when the connect future succeeded, so every failed attempt left an active sample behind: in the new test, 12 attempts to a silent server left 12 active samples. Stop the sample when the connect future completes either way.
TarantoolClientImpl calls pool.setGroups() before pool.setReconnectAfter(), so the connections were created with the pool default of 1000 ms. setReconnectAfter() changed only the pool field, and PoolEntry kept its final copy, so withReconnectAfter() and the reconnect-after property of Spring Data had no effect on these connections. Pass the new value to existing connections, as setConnectTimeout() already does.
lock() and unlock() checked a plain boolean and then changed it and the unavailable counter. They are called from the connect, close and heartbeat handlers, which run on different threads, so two calls could both pass the check and change the counter twice. In the new test, 8 threads locking and unlocking one entry left the counter at 4, -8 and -2 in three runs instead of 0. A counter below the real value makes hasAvailableClients() report a free connection while all entries are locked, so the balancer loops in getNext(); one above it hides free connections, and above the pool size the pool fails with "Invalid pool state". Keep the flag in an AtomicBoolean and change the counter only in the call that won the compare-and-set.
Explain when a pool entry is locked (failed connect, close by the server or by a server shutdown, heartbeat invalidation) and unlocked (reconnect, heartbeat activation), that a long request does not lock a connection, and that a request fails with NoAvailableClientsException when all connections are locked.
The description in testcontainers/pom.xml was copied from the
connector ("Minimalistic java connector for Tarantool ...") and is
shown for the artifact on Maven Central. Describe what the module
provides.
NorthArea
force-pushed
the
northarea/connection-failure-fixes
branch
from
October 27, 2026 11:35
e1ffdd5 to
7325a00
Compare
6 tasks done
Collaborator
Author
|
Moved to #139 : opened from a repository branch so that CI can run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a server is unreachable, silent or closes the connection before the greeting, the client's reconnect loop grows instead of making one attempt per period. This PR fixes the causes and a race in the pool's locked-connection counter.
fix(core): report greeting timeout as a timeout— a silent server was reported both asTimeoutExceptionand asCLOSE_BY_REMOTE, and the pool scheduled a reconnect for each. Now onlyTimeoutException, with the address and the timeout. The greeting timer of a previous attempt no longer closes the next one.fix(core): skip close event before the greeting— the same double report when a server or proxy accepts and immediately closes. Close events are fired only after the greeting; the order from f7f34cd (stateCLOSEDfirst, then the event) is kept.fix(core): register box.shutdown callback once— with graceful shutdown, every attempt added one more callback.fix(pool): stop connect timer on failed attempt—pool.connect.timewas stopped only on success.fix(pool): pass reconnectAfter to pool entries—withReconnectAfter()and Spring Datareconnect-afterhad no effect; connections kept the 1000 ms default.fix(pool): make PoolEntry lock and unlock atomic— concurrentlock()/unlock()could corrupt the unavailable counter, sogetNext()looped or missed free connections.docs(pool): describe locked connectionsbuild(testcontainers): fix module descriptionEach fix has a unit test that fails without it.
Если сервер недоступен, молчит или закрывает соединение до приветствия, цикл переподключения клиента разрастается вместо одной попытки за период. PR устраняет причины и гонку в счётчике заблокированных соединений пула.
fix(core): report greeting timeout as a timeout— молчащий сервер сообщался и какTimeoutException, и какCLOSE_BY_REMOTE, и пул планировал переподключение на каждое. Теперь толькоTimeoutExceptionс адресом и таймаутом. Таймер приветствия прошлой попытки больше не закрывает следующую.fix(core): skip close event before the greeting— тот же двойной отчёт, когда сервер или прокси принимает соединение и сразу закрывает. События закрытия шлются только после приветствия; порядок из f7f34cd (сначалаCLOSED, потом событие) сохранён.fix(core): register box.shutdown callback once— при graceful shutdown каждая попытка добавляла ещё один колбэк.fix(pool): stop connect timer on failed attempt—pool.connect.timeостанавливался только при успехе.fix(pool): pass reconnectAfter to pool entries—withReconnectAfter()иreconnect-afterв Spring Data не действовали, соединения оставались с умолчанием 1000 мс.fix(pool): make PoolEntry lock and unlock atomic— одновременныеlock()/unlock()портили счётчик недоступных соединений, иgetNext()зацикливался или не видел свободных.docs(pool): describe locked connectionsbuild(testcontainers): fix module descriptionУ каждого исправления есть модульный тест, который без него падает.
I haven't forgotten about: