Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
fda64d3
Vendor idealista/clickhouse_role 3.5.2 as ansible/roles/clickhouse (u…
aagbsn Oct 27, 2026
a795556
clickhouse role: trim the vendored role to what devops uses
aagbsn Oct 27, 2026
e05472e
clickhouse role: pin the apt repository key and the ClickHouse version
aagbsn Oct 27, 2026
c22b6b7
clickhouse: per-user settings, quotas and grants for the role's SQL t…
aagbsn Oct 27, 2026
65ed36d
clickhouse role: converge SQL users, quotas and grants, without hidin…
aagbsn Oct 27, 2026
4fd0ee1
deploy-clickhouse: deploy one host at a time
aagbsn Oct 27, 2026
845f183
clickhouse role: test the SQL users, quotas and grants, and the apt pin
aagbsn Oct 27, 2026
8e2f8ec
oonidata_clickhouse: create the ooni database on a new replica from s…
aagbsn Oct 27, 2026
1189361
oonidata_clickhouse: test adding and removing a replica
aagbsn Oct 27, 2026
4af881a
clickhouse role: restart ClickHouse after an upgrade, and check it's …
aagbsn Oct 27, 2026
7b3fd73
clickhouse role: test rolling a version bump out one host at a time
aagbsn Oct 27, 2026
091e3e6
clickhouse role tests: create users before quotas, as the role does
aagbsn Oct 28, 2026
3ad48f2
deploy-clickhouse: leave the notebook out; no SQL quotas on it
aagbsn Oct 28, 2026
5d070fd
clickhouse: a small quota for oonitestlists
aagbsn Oct 28, 2026
201b2dc
clickhouse tests: IPv6 on the compose network
aagbsn Oct 28, 2026
875aa47
clickhouse role tests: 1.5 GB per ClickHouse container
aagbsn Oct 28, 2026
010bfa2
clickhouse: let oonitestlists read what the testlists service reads
aagbsn Nov 10, 2026
fc31a7d
clickhouse: oonimeasurements gets data2's settings
aagbsn Nov 10, 2026
9bca4b0
clickhouse role: XML-escape the secrets in its XML files
aagbsn Nov 10, 2026
4d37415
clickhouse: limit query threads per node to 2 per core, all users tog…
aagbsn Nov 10, 2026
4ee35f0
Merge pull request #539 from ooni/clickhouse_thread_soft_limit
aagbsn Nov 10, 2026
371ad7e
clickhouse: readonly = 2 for the users that only read (oonimeasuremen…
aagbsn Nov 10, 2026
33e8167
Merge pull request #540 from ooni/clickhouse_readers_readonly_2
aagbsn Nov 10, 2026
d83099f
ci: run the ClickHouse role tests on ubuntu-26.04
aagbsn Nov 16, 2026
e5b196d
oonidata_clickhouse: see the clickhouse role's admin client config
aagbsn Nov 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions .github/workflows/test_clickhouse_role.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Tests the clickhouse role's SQL users, quotas and grants and its apt pin
# (ansible/roles/clickhouse/tests), and adding and removing a replica with oonidata_clickhouse
# (ansible/roles/oonidata_clickhouse/tests), when the roles, their vars or the schema change
name: Test ClickHouse roles
on:
pull_request:
paths:
- "ansible/roles/clickhouse/**"
- "ansible/roles/oonidata_clickhouse/**"
- "ansible/group_vars/**"
- "ansible/host_vars/data*"
- "scripts/cluster-migration/schema.sql"
- ".github/workflows/test_clickhouse_role.yml"
push:
branches: [main]
paths:
- "ansible/roles/clickhouse/**"
- "ansible/roles/oonidata_clickhouse/**"
- "ansible/group_vars/**"
- "ansible/host_vars/data*"
- "scripts/cluster-migration/schema.sql"
- ".github/workflows/test_clickhouse_role.yml"

jobs:
sql-objects:
runs-on: ubuntu-26.04
timeout-minutes: 20
steps:
- uses: actions/checkout@v4
- run: pipx install ansible-core==2.17.7 && sudo apt-get update -qq && sudo apt-get install -y --no-install-recommends python3-yaml
- run: ansible/roles/clickhouse/tests/run.sh
- if: failure()
uses: actions/upload-artifact@v4
with: {name: sql-objects-out, path: ansible/roles/clickhouse/tests/out/}
replicas:
runs-on: ubuntu-26.04
timeout-minutes: 20
steps:
- uses: actions/checkout@v4
- run: pipx install ansible-core==2.17.7 && sudo apt-get update -qq && sudo apt-get install -y --no-install-recommends python3-yaml
- run: ansible/roles/oonidata_clickhouse/tests/run.sh
- if: failure()
uses: actions/upload-artifact@v4
with: {name: replicas-out, path: ansible/roles/oonidata_clickhouse/tests/out/}
70 changes: 70 additions & 0 deletions .github/workflows/test_clickhouse_upgrade.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Rolls a ClickHouse version bump out with the clickhouse role, one host at a time
# (ansible/roles/clickhouse/tests/upgrade). Slow (it installs ClickHouse twice on three hosts),
# so it only runs when it has something to test:
# - a PR that changes clickhouse_version: from the base's version to the PR's;
# - a PR that changes the role (other than its README and other tests), the schema step or
# this workflow: from production's version to tests/upgrade/next_version;
# - by hand, with any two versions.
# A change to the vars alone, e.g. a user's settings, doesn't run it.
name: Test ClickHouse upgrade
on:
pull_request:
paths:
- "ansible/group_vars/clickhouse/vars.yml"
- "ansible/roles/clickhouse/**"
- "!ansible/roles/clickhouse/README.md"
- "!ansible/roles/clickhouse/tests/**"
- "ansible/roles/clickhouse/tests/upgrade/**"
- "ansible/roles/oonidata_clickhouse/tasks/schema.yml"
- ".github/workflows/test_clickhouse_upgrade.yml"
workflow_dispatch:
inputs:
from_version: {description: "Version to install first (default: production's)", required: false}
to_version: {description: "Version to upgrade to (default: tests/upgrade/next_version)", required: false}

jobs:
upgrade:
runs-on: ubuntu-26.04
timeout-minutes: 45
steps:
- uses: actions/checkout@v4
with: {fetch-depth: 0}
- name: Choose the versions
id: versions
env:
EVENT: ${{ github.event_name }}
BASE: ${{ github.event.pull_request.base.sha }}
FROM_INPUT: ${{ inputs.from_version }}
TO_INPUT: ${{ inputs.to_version }}
run: |
# clickhouse_version's value, without quotes or a trailing comment
version() { sed -n 's/^clickhouse_version: *["'"'"']\{0,1\}\([0-9.]*\).*/\1/p'; }
run=true from=$FROM_INPUT to=$TO_INPUT
if [ "$EVENT" = pull_request ]; then
base=$(git show "$BASE:ansible/group_vars/clickhouse/vars.yml" | version)
head=$(version < ansible/group_vars/clickhouse/vars.yml)
if [ "$base" != "$head" ]; then
from=$base to=$head
else
# the files of `paths:` above, other than the vars
changed=$(git diff --name-only "$BASE" HEAD \
| grep -E '^(ansible/roles/clickhouse/|ansible/roles/oonidata_clickhouse/tasks/schema\.yml$|\.github/workflows/test_clickhouse_upgrade\.yml$)' \
| grep -v -E '^ansible/roles/clickhouse/(README\.md$|tests/)' ; \
git diff --name-only "$BASE" HEAD | grep '^ansible/roles/clickhouse/tests/upgrade/' || true)
[ -n "$changed" ] || run=false # only the vars changed, not the version
fi
fi
echo "run=$run" >> "$GITHUB_OUTPUT"
echo "from=$from" >> "$GITHUB_OUTPUT"
echo "to=$to" >> "$GITHUB_OUTPUT"
echo "run=$run from=${from:-production} to=${to:-next_version}"
- if: steps.versions.outputs.run == 'true'
run: pipx install ansible-core==2.17.7 && sudo apt-get update -qq && sudo apt-get install -y --no-install-recommends python3-yaml
- if: steps.versions.outputs.run == 'true'
env:
FROM_VERSION: ${{ steps.versions.outputs.from }}
TO_VERSION: ${{ steps.versions.outputs.to }}
run: ansible/roles/clickhouse/tests/upgrade/run.sh
- if: failure()
uses: actions/upload-artifact@v4
with: {name: upgrade-out, path: ansible/roles/clickhouse/tests/upgrade/out/}
5 changes: 4 additions & 1 deletion ansible/deploy-clickhouse.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
---
- name: Deploy oonidata clickhouse hosts
# Not the notebook: its ClickHouse isn't part of the cluster.
hosts:
- notebook.ooni.org
- data1.htz-fsn.prod.ooni.nu
- data2.htz-fsn.prod.ooni.nu
- data3.htz-fsn.prod.ooni.nu
become: true
# One host at a time: a config change restarts ClickHouse, and data1-3 are
# also the Keeper ensemble, so restarting them together loses quorum.
serial: 1
roles:
- role: prometheus_node_exporter
tags: [node_exporter]
Expand Down
Loading
Loading