Skip to content

fix: start WritesRetry max_retry_time at a request's first failure - #725

Open
RishavTiwari25 wants to merge 2 commits into
influxdata:masterfrom
RishavTiwari25:fix/retry-timeout-per-request
Open

RishavTiwari25 wants to merge 2 commits into
influxdata:masterfrom
RishavTiwari25:fix/retry-timeout-per-request

Conversation

@RishavTiwari25

Copy link
Copy Markdown

Proposed Changes

WritesRetry stops retrying once max_retry_time has passed since the strategy object was created, not since the request started failing.

__init__ set self.retry_timeout = datetime.now() + timedelta(seconds=max_retry_time) and new() copied it into every object that increment() returns. urllib3 starts every request from the strategy given to the pool, so a WritesRetry passed to InfluxDBClient(retries=...), which is what examples/import_data_set_sync_batching.py and examples/write_batching_by_bytes_count.py do, has one deadline shared by all requests. With the default max_retry_time=180, any write that fails more than 3 minutes after the client was created raises MaxRetryError("max_retry_time exceeded") on its first failure, with no retry. For a long import or a long-running service, that means retries are effectively disabled after startup.

(The batching WriteApi is not affected because it builds a new WritesRetry per batch with WriteOptions.to_retry_strategy().)

Fix: retry_timeout starts as None. On a request's first failure (increment() on an object without a deadline) it is set to now + max_retry_time and given to that request's retry objects only. The base strategy is never mutated, so the next request gets its own deadline. The comparison is now <=, so max_retry_time=0 reliably disables retries: with <, two datetime.now() calls in a row can return the same value (often on Windows) and the first retry went through.

Tests in tests/test_WritesRetry.py:

  • test_max_retry_time_starts_at_first_failure: a strategy created 1.5 s before its first failure, with max_retry_time=1, still retries;
  • test_max_retry_time_is_per_request: a request past its deadline raises, and a new request from the same strategy gets a fresh max_retry_time;
  • test_max_retry_time_zero_disables_retry.

All three fail on master (the last one only on Windows timer resolution); the existing test_backoff_max_time and the other retry tests pass unchanged.

Checklist

  • CHANGELOG.md updated
  • Rebased/mergeable
  • A test has been added if appropriate
  • pytest tests completes successfully: full integration suite not run (needs a live InfluxDB). Focused: pytest tests/test_WritesRetry.py tests/test_WriteOptions.py tests/test_WriteApiBatching.py (47 passed; test_batching_data_frame needs pandas and test_WriteApiPickle needs a running InfluxDB, and both fail the same way on master locally)
  • Commit messages are conventional
  • Sign CLA (will sign)

flake8 and pydocstyle are clean on the changed file.

WritesRetry computed its retry deadline in __init__ and copied it to every
object returned by increment(). urllib3 starts each request from the
strategy passed as retries to the pool, so an instance given to
InfluxDBClient(retries=...), as the import examples do, shared one
deadline across all requests: once max_retry_time (default 180 s) had
passed since the client was created, every failed write raised
MaxRetryError("max_retry_time exceeded") without a single retry.

The deadline is now set on a request's first failure and carried by the
retry objects of that request only. The check uses <= so that
max_retry_time=0 disables retries even when datetime.now() returns the
same value twice (common on Windows).
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.50%. Comparing base (ef60f89) to head (c44688b).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #725      +/-   ##
==========================================
+ Coverage   90.49%   90.50%   +0.01%     
==========================================
  Files          40       40              
  Lines        3568     3572       +4     
==========================================
+ Hits         3229     3233       +4     
  Misses        339      339              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants