Repository navigation
fastpath: deploy 20261117-27f2b28c, fall back to clickhouse2 - #546
Merged
Merged
Conversation
Update fastpath1 and fastpath2 to ooni/fastpath:20261117-27f2b28c which adds the following: - drains its queue into ClickHouse on docker stop (#1326) - retries failed inserts with a backoff, and splits a batch to drop only the rows the server rejects (#1327, #1332) - answers 503 at once when its queue is full (#1330) clickhouse_url: clickhouse1, then clickhouse2 as alt_hosts clickhouse2's performance under oonimeasurements load isn't known and until upgrade it does not have the per-query thread limits, so it only takes writes when clickhouse1 can't be reached. clickhouse-driver stays on the alt host once it has moved there: restart fastpath after clickhouse1 is back. Future work: detect when alt_host is being used and periodically retry and switch back to primary. fastpath role: set fastpath_stop_timeout to 120s so docker stop leaves time to drain FASTPATH_QUEUE_PUT_TIMEOUT 0 instead of -1: refuse a measurement at once when the queue is full, so ooniprobe sends it to another fastpath or to S3 instead of holding its thread until the request times out.
Contributor
Author
|
deployed to dev; deployed to fastpath2 and shifted traffic to it, deployed to fastpath1 and returned traffic to it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update fastpath1 and fastpath2 to ooni/fastpath:20261117-27f2b28c which adds the following:
clickhouse_url: clickhouse1, then clickhouse2 as alt_hosts.
clickhouse2's performance under oonimeasurements load isn't known and until upgrade it does not have the per-query thread limits, so it only takes writes when clickhouse1 can't be reached.
clickhouse-driver stays on the alt host once it has moved there: restart fastpath after clickhouse1 is back.
Future work: detect when alt_host is being used and periodically retry and switch back to primary.
fastpath role: set fastpath_stop_timeout to 120s so docker stop leaves time to drain
FASTPATH_QUEUE_PUT_TIMEOUT 0 instead of -1: refuse a measurement at once when the queue is full, so ooniprobe sends it to another fastpath or to S3 instead of holding its thread until the request times out.