[Mearman]

PR A of the decided plan in #183: a former member never starts a second cluster, and a server whose etcd member was removed never starts again by its own restart policy.

Refs #183

What changes

• Role directory. fleet-role.d in each Docker host's Compose project directory, created empty by the role, is bind-mounted read-only at /etc/github-runner into both k3s and k3s-agent. It holds role (server or agent) and join-url, both absent until something writes them; an absent role behaves exactly as before.
• Entrypoint guards (files/k3s/node-entrypoint.sh), run before Tailscale starts: a service whose role the file does not name idles, and so does a server while server/db/etcd/tombstone exists. The join URL is join-url when not empty, else K3S_JOIN_SERVER_URL. With no join URL, server/db present passes --cluster-init as before, neither present passes it (first bootstrap), and server/db.demoted without server/db exits 1 with a message and starts nothing.
• Native. fleet-role-is.sh server ROLE_FILE SERVER_DIR also exits 1 (start skipped) while the tombstone exists; the server unit passes its server directory.

Decisions where the plan and the code disagreed

1. The mount lives in its own override, docker-compose.fleet-role.yml, not in docker-compose.yml. The role keeps a fleet that does not use the controller from being recreated by an upgrade (that is why docker-compose.node-labels.yml and docker-compose.api-port.yml exist). A new bind mount in the base file would recreate every Docker node once, agents included, cutting their running jobs. The override is listed for exactly the hosts that publish the controller's labels (github_runner_cluster_publishes_node_labels: controller on, or host eligible or pinned), which are the only hosts PR C and D2 write into the directory. A host outside that set gets the directory but no mount and no recreate.
2. The idle guard does not exec sleep infinity. The entrypoint is PID 1 and the kernel delivers PID 1 only signals it handles, so an exec'd sleep would ignore docker stop's SIGTERM and be killed only after the stop timeout. The shell keeps PID 1, traps TERM and INT to exit 0, and waits on sleep infinity & in a loop, so it still never exits by itself and stops at once when asked. The mesh scenario checks the stopped idle container's exit code is 0.
3. The rejoin mesh scenario removes the member first. Moving server/db aside on a server that is still a member and starting it with a join URL is refused by k3s itself: join() in pkg/etcd/etcd.go (v1.34.1+k3s1) returns "duplicate node name found" while a member with that node name exists. The scenario therefore removes node 1's member through node 2 (etcd.k3s.cattle.io/remove=true, as a demotion does), which also lets it check the tombstone guard against real k3s: k3s writes the tombstone and keeps running (pkg/daemons/executor/etcd.go), and the restarted container idles instead of rejoining.
4. The refusal covers SQLite too. A former member with server/db.demoted, no server/db and nothing to join is refused whatever K3S_DATASTORE says, since a fresh SQLite datastore would be a second cluster as surely as --cluster-init.
5. The guards run for the self-hosted Headscale bootstrap path as well (before supervise_self_hosted), since its direct start would otherwise create a fresh datastore beside a moved-aside one.

Tests

• tests/unit/test_node_entrypoint.py (new): join URL precedence; --cluster-init with a datastore, with one moved aside, and with none; idling for a role the file does not name (both services, and an empty file); idling on a tombstone whatever the role file says, the agent unaffected; no change with no role file or with the service's own role; the guards placed before Tailscale; the override's mount, its placement and directory creation, and the mount as Compose merges it.
• tests/unit/test_native_server.py: the server unit's condition, run from the unit the role writes, refuses on a tombstone and passes once server/db is moved aside.
• Existing entrypoint harnesses set join_url the way set_join_url does; test_cluster_voters.py lists the new override for an eligible host and still none for a host that uses nothing.
• Mesh scenario rejoin (added to the workflow matrix): removal, tombstone idle, the refusal with no join URL (and no k3s start since), the join through join-url as a learner that is promoted with the same etcd cluster ID, the same node password, and a role file naming agent leaving the server container running only sh and sleep. role_controller's check-run step also deletes the new override.

Each guard was broken on purpose with a scratch edit of the production file, the test that covers it was run and went red, then the file was restored byte for byte (git diff empty) and the same test went green. Run locally against the rebased head with python3 -m unittest <module>.<class>.<test> from tests/unit.

Guard	Break	Test	Red	Green
Entrypoint idles on a role the file does not name	role comparison made false	test_node_entrypoint test_a_service_whose_role_the_role_file_does_not_name_idles	3 subtests fail: started printed for server/agent, agent/server, server/empty	OK
Entrypoint idles a server on a tombstone	tombstone test made false	test_node_entrypoint test_a_server_whose_member_was_removed_idles_whatever_its_role_file_says	2 subtests fail: started ... --server with no role file and with server	OK
--cluster-init kept with server/db present	[ ! -e server/db ] dropped from the refusal	test_node_entrypoint test_a_server_with_its_datastore_and_nothing_to_join_keeps_cluster_init	exit 1 with the refusal message instead of starting	OK
Refusal with server/db.demoted, no server/db, no join URL	refusal condition made false	test_node_entrypoint test_a_former_member_with_nothing_to_join_is_refused_and_k3s_is_not_started	0 != 1: k3s arguments built	OK
--cluster-init on a first bootstrap (no datastore)	[ -e server/db.demoted ] dropped from the refusal	test_node_entrypoint test_a_first_bootstrap_with_no_datastore_at_all_passes_cluster_init	exit 1 with the refusal message	OK
A join URL lifts the refusal	[ -z "$join_url" ] dropped from the refusal	test_node_entrypoint test_a_former_member_with_nothing_to_join_is_refused_and_k3s_is_not_started	2 subtests fail: refused with K3S_JOIN_SERVER_URL and with join-url	OK
join-url takes precedence over K3S_JOIN_SERVER_URL	file read replaced by join_url=""	test_node_entrypoint test_the_join_url_file_takes_precedence_over_the_inventorys_join_url	--server is the bootstrap URL, not node 2's	OK
The server waits for the host it will join	join_host_of given K3S_JOIN_SERVER_URL again	test_node_entrypoint test_the_guards_run_before_tailscale_starts_and_the_server_joins_the_url_they_chose	join_host_of "$join_url" not found in the server case	OK
Native fleet-role-is.sh refuses on a tombstone	tombstone test made false	test_native_server test_the_server_unit_does_not_start_while_its_datastore_holds_a_tombstone	0 != 1: the condition passed with a tombstone	OK
Native server unit passes its server directory	.../server argument removed from ExecCondition=	same test	1 != 0: the condition fails before any tombstone	OK
Override mounts the role directory	k3s service removed from docker-compose.fleet-role.yml	test_node_entrypoint RoleDirectoryTest (override shape; merged by docker compose config)	both fail: no k3s entry, no /etc/github-runner mount on k3s	OK
Override listed for hosts in the controller	docker-compose.fleet-role.yml entry removed from github_runner_cluster_compose_file_list	test_cluster_voters test_an_eligible_host_... and test_with_the_controller_on_...	both fail: override missing from the Compose files	OK

The mesh rejoin scenario needs Docker and real k3s and runs only in CI, where it passed on this head; it covers the same guards against real k3s: the mount (does not mount ... read-only at /etc/github-runner), the tombstone idle (the log line tombstone says this server's etcd member was removed, node1_idles, two started members), the idle trap (the idle container did not exit 0 on docker stop's SIGTERM), the refusal (has no server to join logged, no Starting k3s v since, restarted by policy, members and cluster ID unchanged), the join through join-url (node1_voter_started, Promoted learner, same cluster ID and node password) and the role-mismatch idle (role names 'agent', not server, only sh and sleep, no restart).

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
🔒 Security Review	⚠️ Failed 2026-11-03T22:54:27.569132Z We couldn’t complete this review. Comment @codex security review to try again.	aaca68f	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins.

[github-actions]

🎉 This PR is included in version 2.34.0 🎉

The release is available on:

• v2.34.0
• GitHub release

Your semantic-release bot 📦🚀