Repository navigation
to_mcp_server never deletes the ADK session of a dead MCP connection; a stateless streamable HTTP deployment leaks one session per tool call #7154
Description
Describe the bug
to_mcp_server keeps one ADK session per MCP connection in a weakref.WeakKeyDictionary (src/google/adk/tools/mcp_tool/_agent_to_mcp.py). When a connection is garbage collected, the weak map drops the entry, but nothing ever calls session_service.delete_session, so the ADK session, with its full event history, stays in the session service forever. The default runner built by to_mcp_server uses InMemorySessionService, which is a plain dict with no eviction.
Two consequences:
- Stateful serving: a long running server accumulates one dead conversation per closed connection.
- Stateless streamable HTTP, which is the mode you want behind an autoscaler such as Cloud Run: the MCP SDK builds a completely fresh transport for every request (
StreamableHTTPSessionManagerwithstateless=True), so_connection_keyreturns a per request object and every tool call creates a new ADK session that is never deleted. That is one leaked session per tool call, growing linearly with traffic, and the sessions hold full conversation content.
To Reproduce
The script below makes 10 short lived in-memory MCP connections with one tool call each, which is exactly the connection pattern a stateless streamable HTTP deployment produces, then counts the sessions left in the service.
import asyncio
from contextlib import asynccontextmanager
import gc
from typing import AsyncGenerator
import anyio
from google.adk.agents.base_agent import BaseAgent
from google.adk.agents.invocation_context import InvocationContext
from google.adk.artifacts.in_memory_artifact_service import InMemoryArtifactService
from google.adk.auth.credential_service.in_memory_credential_service import (
InMemoryCredentialService,
)
from google.adk.events.event import Event
from google.adk.memory.in_memory_memory_service import InMemoryMemoryService
from google.adk.runners import Runner
from google.adk.sessions.in_memory_session_service import InMemorySessionService
from google.adk.tools.mcp_tool import to_mcp_server
from google.genai import types
from mcp.client.session import ClientSession
from mcp.shared.memory import create_client_server_memory_streams
@asynccontextmanager
async def connected_client(lowlevel_server):
"""One in-memory MCP connection, the same shape a network client makes."""
async with create_client_server_memory_streams() as (
client_streams,
server_streams,
):
client_read, client_write = client_streams
server_read, server_write = server_streams
async with anyio.create_task_group() as tg:
tg.start_soon(
lambda: lowlevel_server.run(
server_read,
server_write,
lowlevel_server.create_initialization_options(),
)
)
try:
async with ClientSession(
read_stream=client_read, write_stream=client_write
) as session:
await session.initialize()
yield session
finally:
tg.cancel_scope.cancel()
class EchoAgent(BaseAgent):
async def _run_async_impl(
self, ctx: InvocationContext
) -> AsyncGenerator[Event, None]:
yield Event(
author=self.name,
content=types.Content(role="model", parts=[types.Part(text="ok")]),
)
async def main():
agent = EchoAgent(name="echo")
session_service = InMemorySessionService()
runner = Runner(
app_name="echo",
agent=agent,
session_service=session_service,
artifact_service=InMemoryArtifactService(),
memory_service=InMemoryMemoryService(),
credential_service=InMemoryCredentialService(),
)
server = to_mcp_server(agent, runner=runner)
lowlevel = getattr(server, "_mcp_server", None) or getattr(
server, "_lowlevel_server", server
)
for i in range(10):
async with connected_client(lowlevel) as client:
await client.call_tool("echo", {"request": f"call {i}"})
gc.collect()
remaining = await session_service.list_sessions(
app_name="echo", user_id="mcp_user"
)
print(f"sessions left in the service: {len(remaining.sessions)}")
asyncio.run(main())Output on main: sessions left in the service: 10
Also reproduced over a real transport: running the server with run_streamable_http_async(host="127.0.0.1", port=8765, stateless_http=True) and making each call over a fresh streamable_http_client connection, 9 calls leave 9 sessions in the service.
Expected behavior
Sessions whose connection is gone are deleted from the session service. With the fix below, the same runs leave 1 session (the most recent call's, reclaimed on the next call).
Environment
google-adk main (7ae1c9b), mcp 2.2.0 (also reproduces on 1.24.0 and 1.26.0), Python 3.12.
Proposed fix (PR to follow)
Record the id of every session entered into the connection map, and at the start of each tool call delete the sessions that are no longer reachable through the weak map. Reaping runs lazily from the tool call rather than from a GC finalizer, because finalizers can fire without a running event loop. This adds no TTL or retention policy of its own: connection lifetime is already governed by the MCP transport layer (client disconnects, the SDK's idle session timeout, stateless per request teardown), and the fix makes ADK sessions follow that lifetime instead of outliving it.
One design note for review: with a caller supplied Runner backed by a persistent session service, deleting orphaned sessions is a behavior change, since conversations would no longer remain readable after their connection dies. I think cleanup is the right default here, because the module creates these sessions under its internal mcp_user id and unbounded growth is the worse failure, but I am happy to add an opt-out flag if retention matters for some deployments.
Possible follow-ups, feedback welcome
- A
stateless=Trueoption onto_mcp_serverthat skips the connection map and deletes each session eagerly after its call, as explicit support for deployments that run the server withstateless_http=True. - An optional conversation id argument on the generated tool, so a client of a stateless deployment can keep multi-turn conversations by passing the id back with each call. Today stateless mode silently degrades to one single turn conversation per call, because there is no connection identity to thread a conversation on.
I have the first ready and would build the second if there is interest.
Activity
surajksharma07 commentedon Sep 18, 2026
@harshal-96 Ran your repro script against current main and got the identical count, 10 leaked sessions, so this tracks. The lazy reap-on-call approach in #7155 looks like the right shape, especially since it sidesteps the finalizer/event-loop problem. Before we push it toward merge could you confirm the behavior you're comfortable with for callers using a persistent caller-supplied session service: should that opt-out flag ship in this PR or as a fast follow? Once that's settled think we're in good shape to move forward.
harshal-96 commentedon Sep 18, 2026
Thanks for verifying the repro. I would ship the flag in this PR rather than a fast follow, and it is now in: 4311d5c adds
delete_orphaned_sessions(default True) toto_mcp_server. With False the reaping is disabled entirely and session lifecycle stays with the caller, which is the right contract for a persistent caller-supplied session service whose records are read after the fact. The default stays True so a long-running server is bounded out of the box. Added a regression test covering the opt-out; 19 tests pass on mcp 1.26.0 and 2.2.0. Happy to rename the flag if the team prefers different wording.surajksharma07 commentedon Sep 23, 2026
Thanks @harshal-96, having the flag in this PR is the better call and the name reads fine. With False, created_session_ids is None so there's no tracking set growing in the background either, which is what a persistent-backend caller would want.
Checked the latest (v2.9.2/current main): nothing in mcp_tool calls delete_session yet and _agent_to_mcp.py hasn't changed since you branched so #7155 still merges cleanly and no release covers this.
One small thing worth considering, not a blocker: the reap runs inline before the live call so with a slow DB/Vertex session service the first call after a burst of disconnects waits on every delete. Moving it to a background task would keep that latency off the caller.
Flagging this for review now.
harshal-96 commentedon Sep 23, 2026
Thanks @surajksharma07, good catch on the latency. Done in 45cbb0a: the reap now runs as a background task, so the live call never waits on deletes. At most one reap runs at a time and the task is held by reference. Testing it across the versions the pin admits turned up two details worth handling in the same change:
asyncio.create_taskandadd_done_callbackboth copy the caller's contextvars. On MCP SDK 1.x the request lives in a ContextVar that references the connection, so a stuck background reap would have kept alive the very connection whose session it should delete. The task and its callback are now created in an empty context, which also works on Python 3.10.Four new tests cover this, and each fails when its safeguard is removed.
test_agent_to_mcp.pypasses (23) on Python 3.10, 3.11 and 3.12 with each of mcp 1.24.0, 1.26.0 and 2.2.0, the mcp_tool suite also passes with the branch merged onto current main, and the stateless streamable HTTP check still leaves one session after 9 calls. I have updated the PR description to match.surajksharma07 commentedon Sep 24, 2026
Went through 45cbb0a @harshal-96: merged on current main test_agent_to_mcp.py passes 23 and the mcp_tool suite 417 (mcp 1.26.0). Your repro leaves 1 session after 300 calls even without gc.collect() against 300 on main.
One problem with the single in-flight reap and it comes from the background change I suggested. It deletes serially and skips starting a new reap while one is running so throughput is capped at 1/delete-latency. With a 50ms delete_session and calls every ~20ms the service held 63 sessions after 100 calls and 196 after 300 and they only drain when new calls come in. 4311d5c (inline) stays at 1 under the same load.
What worked for me: inside _reap_orphaned_sessions loop while created - set(sessions.values()) is non-empty, discard that batch from created and delete it with asyncio.gather(..., return_exceptions=True), putting failed ids back and stopping that pass. Under the same load it stays at 4-5 during the burst and drains to 0 without further calls and your 23 tests still pass. A semaphore would keep a big batch from hitting a DB all at once. Could you fold that in plus a regression test with a slow delete_session?
harshal-96 commentedon Sep 25, 2026
Thanks @surajksharma07, that is a real regression from the background change, and I reproduced it before touching anything: with a 50 ms delete_session and a call every 20 ms, across four runs 45cbb0a held 34 to 49 sessions after 100 calls and 99 to 140 after 300 on mcp 1.26.0, and 81 to 124 were still there after a second of idle.
Folded in your approach in da408fa, with adjustments that came out of load and failure testing it:
createdbefore awaiting, as you suggested. Instead of onegathertask per id, the batch is drained by a fixed pool of 8 workers (_MAX_CONCURRENT_DELETES) sharing one iterator. After a long outage the backlog can hold many thousands of ids, and a task per id peaked at about 15 MB per 10,000 ids (147 MB at 100,000); the pool keeps a 100,000 backlog at about 5 MB.createdand end the pass, as you suggested, but the workers only stop starting new deletes once as many have failed as there are workers. Stopping at the first failure let one session the service keeps rejecting stall the reap: its delete could fail before the other workers had started, so they skipped the rest of the batch, and in my runs the reap stopped making progress with 3 to 96 of 100 healthy sessions still undeleted after 1000 passes, depending on iteration order. With the threshold all 100 are deleted in one pass. With the service down and 200 orphans waiting, a pass makes 15 attempts and logs one summary warning, against 200 of each before (that was also true before the background change).Under your load it holds 2 to 3 sessions during the burst and drains to 0 with no further calls, across three runs. Against ADK's real Runner with InMemorySessionService and DatabaseSessionService on SQLite (also with 50 ms added per delete), on mcp 1.26.0, it holds 1 to 2 sessions and no delete failed, so the concurrent deletes do not trip SQLite locking.
Regression tests include
test_slow_deletes_run_concurrently_up_to_the_cap(slow delete_session),test_reap_drains_orphans_that_appear_while_it_runs,test_reap_stops_after_a_failure_instead_of_hammering_the_service,test_one_failing_session_does_not_hold_up_the_rest,test_hung_delete_times_out_instead_of_stopping_reaping, andtest_reap_task_count_stays_bounded_for_a_large_backlog; each fails when the safeguard it covers is removed. test_agent_to_mcp.py passes (30) on Python 3.10 to 3.12 with mcp 1.24.0, 1.26.0 and 2.2.0, and the mcp_tool suite passes merged onto current main (424). PR description updated with the numbers.surajksharma07 commentedon Sep 28, 2026
Went through da408fa @harshal-96. The shared-iterator pool with the failure threshold is a better answer than my gather-per-batch and the timeout retry is safe since InMemory, Database and VertexAi delete_session all treat a missing session as a no-op (Vertex swallows the 404) so a delete that timed out but actually landed just retries cleanly.
Merged on current main: test_agent_to_mcp.py passes 30 and the mcp_tool suite 430 (mcp 1.26.0, Python 3.12). Under the same slow-delete load (50ms delete, ~20ms calls) it held 6-9 sessions right after the burst and drained to 0 a second later with no further calls and the repro leaves 1 after 300 calls.
One small non-blocking nit: if the reap task gets cancelled mid-pass (e.g. loop shutdown) ids already claimed from created but still pending or in flight are neither deleted nor put back since CancelledError skips the except. Cancelling a 100-id backlog after ~0.1s with a 50ms delete left 16 deleted and 84 dropped from tracking. A finally that re-adds the unconsumed ids would close it and a docstring note that orphans from before a restart aren't reaped with a persistent backend would help too.
v2.10.0 doesn't touch _agent_to_mcp.py so this is still unreleased. From my side it looks ready, flagging it for maintainer review.
harshal-96 commentedon Sep 30, 2026
Thanks @surajksharma07, both points are in 1d49634.
finallyaround the gather re-adds whatever is left in the shared iterator) and the ids still in flight (each worker re-adds its current id onCancelledErrorbefore re-raising). As you noted, a delete that did land and is retried is a no-op in the built-in services. Your scenario as a test: 400 waiting, 50 ms deletes, cancelled after 0.1 s; every id ends up deleted or still tracked.asyncio.wait_forthere can swallow a cancellation that arrives as the delete finishes, so the worker carried on and the cancelled reap drained its whole backlog (about 2.7 s for 400 ids before the cancel returned). Each delete now waits withasyncio.waitinstead, and the cancel returns immediately on 3.10, 3.11 and 3.12. Switching back towait_forfails the test in 4 of 5 runs on 3.10 and 3.11.delete_orphaned_sessionsdocstring now says only sessions created by the running server process are tracked, so with a persistent session service, sessions orphaned before a restart are not deleted.test_agent_to_mcp.py passes (31) on Python 3.10 to 3.12 with mcp 1.24.0, 1.26.0 and 2.2.0, and on Linux with Python 3.10 to 3.14, stable over 20 consecutive runs on each. Merged onto current main (f44d512), the mcp_tool suite passes (443). The full unit suite gives 16573 passed there versus 16555 on main: the 18 new tests, with the same 3 failures as main, which are unrelated and fixed in #7358 and #7359. PR description updated.
surajksharma07 commentedon Oct 6, 2026
Checked 1d49634 @harshal-96. Cancelling a 100-id backlog after 0.1s with 50ms deletes now loses nothing: 16 deleted the rest still tracked, 3 of 3 runs. On da408fa the same run dropped 92. The few ids that land and also get re-added just retry as a no-op as you said. Only have 3.12 here so relying on your 3.10/3.11 runs for the wait_for race.
Merged onto current main (338 commits ahead of your base, no conflicts): test_agent_to_mcp.py passes 31 on 10 runs in a row and the mcp_tool suite gives 449 passed, 1 skipped (mcp 1.26.0). Under the same slow-delete load it holds 7-9 sessions right after 300 calls and 0 a second later against 300 on main. The repro from the issue leaves 1.
Since #7358 and #7359 were closed as fixed by 41bebdc in v2.11.0 those 3 baseline failures should be gone on main now so the full-suite numbers in the description could use a refresh. _agent_to_mcp.py is unchanged through v2.11.0 so the leak is still in every release. Nothing else from my side, back with @wukath for review.
harshal-96 commentedon Oct 6, 2026
Thanks @surajksharma07 for checking it independently, including the slow-delete load run against main.
Description refreshed. Merged onto current main (fac77be), the full unit suite now has no failures: 17780 passed against 17762 on main, the difference being the 18 new tests. The mcp_tool suite gives 450 passed on Python 3.10, 3.11, 3.13 and 3.14.
surajksharma07 commentedon Oct 7, 2026
The refreshed numbers line up @harshal-96: 17780 vs 17762 is exactly the 18 new tests and with #7358/#7359 fixed main now has no failures. Still don't see any delete_session call under mcp_tool in v2.11.0 so the leak is in every release until #7155 lands. Nothing more from my side. @wukath it's ready for your review and the two CI workflows on the PR still need maintainer approval before block-merge can report.
fix: reap ADK sessions of dead MCP connections in to_mcp_server