multiprocessing.Pool.imap with buffersize retains semaphore registrations after completion #161143

Bug report

Repeated, fully consumed Pool.imap(..., buffersize=...) and
Pool.imap_unordered(..., buffersize=...) operations leave their semaphore
objects registered in the pool. A pool that remains alive therefore retains
objects for work that has already completed.

Root cause

A buffered iterator creates a threading.Semaphore and adds it to
Pool._taskqueue_buffersize_semaphores. This registry has a legitimate
shutdown purpose: close() and termination use it to wake task submission
that may be blocked by the buffer limit.

The lifetime of that registration is longer than intended:

1. IMapIterator.__init__() registers the semaphore and retains the pool in
   self._pool.
2. Once all results have arrived, IMapIterator._set() or
   IMapUnorderedIterator._set() removes the iterator from the result cache
   and sets self._pool = None. The equivalent completion branch in
   _set_length() handles cases where the length becomes known after the
   results, including empty input.
3. When the caller exhausts the iterator, _stop_iterator() tries to discard
   its semaphore from the pool's registry, but only if self._pool is not None.
   That reference has already been cleared, so the discard is skipped.
4. The iterator can be discarded, but the live pool's set still owns a strong
   reference to the semaphore. Garbage collection therefore cannot reclaim
   that object merely because the caller has finished consuming the results.

Removing the iterator from the result cache does not remove this separate
registration. The original implementation is in
Lib/multiprocessing/pool.py at the audited base.

Consequences and observed impact

• A reused pool accumulates one stale semaphore registration per completed
  buffered iterator, rather than per individual result. This can happen even
  when the input is empty.
• Fully consuming the results does not avoid the retention. A service or batch
  worker that repeatedly maps small inputs through one long-lived pool can
  keep accumulating these Python objects after the real work has finished.
• The registry also contains stale entries when shutdown traverses it.
  Additional traversal work follows from the code path; its latency impact
  has not been benchmarked.
• The 10-iteration reproduction below leaves 10 registrations in the
  unmodified module and 0 with the patch. In a separate unpatched
  2,000-iteration experiment, 2,000 semaphores remained and tracemalloc
  measured 2,693,553 additional bytes of Python allocations
  (approximately 2.7 MB). This is a local measurement, not process RSS or a
  production growth-rate forecast.

The evidence concerns retained Python semaphore objects. It does not show
leaked worker processes, incorrect returned results, an out-of-memory failure,
or a universal shutdown hang. No official P0/P1 severity or security impact is
claimed.

Minimal reproduction

ThreadPool shares the affected iterator implementation with the process
pool and keeps this example small:

from multiprocessing.pool import ThreadPool

with ThreadPool(1) as pool:
    for _ in range(10):
        assert list(pool.imap(str, [1], buffersize=1)) == ["1"]
    print(len(pool._taskqueue_buffersize_semaphores))

Observed before the fix: 10. Expected after these completed iterations:
0. The private registry is inspected only to expose the retained
references.

The same behavior was reproduced with a spawn process pool,
imap_unordered(), and chunksizes 1 and 2. A control without buffersize
retained no registrations. The API being tested is from the 3.16 source;
running this snippet on an older, unmodified module that lacks buffersize
will not reproduce this defect.

Correction and why it addresses the cause

The fix removes the semaphore from the pool's registry before clearing
self._pool. A shared _release_pool() helper is called by each completion
path and by _stop_iterator(); using discard() makes repeated removal
harmless.

Completion means that the task iterable's length is known and all its results
have arrived. At that point, that iterator's task generator no longer needs
the registry entry to be woken during shutdown. This does not cancel pending
work, consume the caller's results, or destroy the pool. The iterator still
owns its own semaphore reference while it needs it.

The completion and exhaustion paths use the iterator's existing condition
lock. Since the result handler can now remove registry entries, termination
uses a list(set) snapshot, matching the existing close() implementation.
The snapshot rationale was reviewed against the CPython implementation and a
controlled iterator interleaving; it was not validated on a free-threaded
runtime.

The proposed correction and regression test are in PR #161144.

Validation environment and limits

Local validation used Windows CPython 3.15.0b2 loading the audited 3.16
multiprocessing.pool source, based on
4b21d154b4864531641f90e1b87108575e673d75.
It was not a native CPython 3.16 build or its native regrtest runner.
No free-threaded runtime was executed locally.

The new regression method produced 16 failing subtests across the thread and
process runs before the fix and none afterward. A selection of 64 test cases
and 68 additional bounded scenarios passed with the patched module. The
selected-method harness extracted source test methods and supplied their
parameters for the 3.15 host; these counts must not be read as a full native
3.16 test-suite run. The additional scenarios cover worker/iterable errors,
out-of-order results, and concurrent completion/shutdown.

Detailed validation and suggested native commands
provide the scope and limitations of these checks.

Related reports

#155477, #158675 and #158677 concern related buffered-iterator shutdown or
result-delivery behavior. The reported symptom here is registration retention
after completed iterations. Searches did not locate a direct duplicate; that
is not a guarantee of novelty.

AI tools assisted the investigation, implementation, test harnesses and review.

Linked PRs

• gh-161143: Release completed buffered Pool iterator registrations #161144