Skip to content

fastpath: deploy 20261117-27f2b28c, fall back to clickhouse2 - #546

Merged
aagbsn merged 1 commit into
mainfrom
fastpath2_clickhouse2
Nov 17, 2026
Merged

aagbsn merged 1 commit into
mainfrom
fastpath2_clickhouse2

Conversation

@aagbsn

@aagbsn aagbsn commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Update fastpath1 and fastpath2 to ooni/fastpath:20261117-27f2b28c which adds the following:

  • drains its queue into ClickHouse on docker stop (#1326)
  • retries failed inserts with a backoff, and splits a batch to drop only the rows the server rejects (#1327, #1332)
  • answers 503 at once when its queue is full (#1330)

clickhouse_url: clickhouse1, then clickhouse2 as alt_hosts.
clickhouse2's performance under oonimeasurements load isn't known and until upgrade it does not have the per-query thread limits, so it only takes writes when clickhouse1 can't be reached.

clickhouse-driver stays on the alt host once it has moved there: restart fastpath after clickhouse1 is back.

Future work: detect when alt_host is being used and periodically retry and switch back to primary.

fastpath role: set fastpath_stop_timeout to 120s so docker stop leaves time to drain

FASTPATH_QUEUE_PUT_TIMEOUT 0 instead of -1: refuse a measurement at once when the queue is full, so ooniprobe sends it to another fastpath or to S3 instead of holding its thread until the request times out.

Update fastpath1 and fastpath2 to ooni/fastpath:20261117-27f2b28c
which adds the following:
- drains its queue into ClickHouse on docker stop (#1326)
- retries failed inserts with a backoff, and splits a batch to drop only
  the rows the server rejects (#1327, #1332)
- answers 503 at once when its queue is full (#1330)

clickhouse_url: clickhouse1, then clickhouse2 as alt_hosts
clickhouse2's performance under oonimeasurements load isn't known
and until upgrade it does not have the per-query thread limits,
so it only takes writes when clickhouse1 can't be reached.

clickhouse-driver stays on the alt host once it has moved
there: restart fastpath after clickhouse1 is back.

Future work: detect when alt_host is being used and periodically retry
and switch back to primary.

fastpath role: set fastpath_stop_timeout to 120s so docker stop leaves
time to drain

FASTPATH_QUEUE_PUT_TIMEOUT 0 instead of -1: refuse a measurement at once
when the queue is full, so ooniprobe sends it to another fastpath or to
S3 instead of holding its thread until the request times out.
@aagbsn

aagbsn commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

deployed to dev; deployed to fastpath2 and shifted traffic to it, deployed to fastpath1 and returned traffic to it.

@aagbsn
aagbsn merged commit 130f4da into main Nov 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant