Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/getting-started.de.md
Original file line number Diff line number Diff line change
Expand Up @@ -494,3 +494,5 @@ Mit dem Standard `gateway_api = true` konfiguriert die SKE-DNS-Erweiterung Exter

> [!NOTE]
> Die SKE-DNS-Erweiterung installiert keine Gateway-API-CRDs. Die optionale Envoy-Gateway-Demo installiert sie automatisch. Installieren Sie für andere Gateway-API-Implementierungen kompatible CRDs separat. Sobald die CRDs verfügbar sind, wird ExternalDNS beim nächsten Cluster-Reconcile konfiguriert; ein zusätzliches OpenTofu-Apply ist nicht erforderlich.

[State-Koordination](state-locking.de.md).
2 changes: 2 additions & 0 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -460,3 +460,5 @@ With the default `gateway_api = true`, the SKE DNS extension configures External

> [!NOTE]
> The SKE DNS extension does not install Gateway API CRDs. The optional Envoy Gateway demo installs them automatically. For other Gateway API implementations, install compatible CRDs separately. Once the CRDs are available, ExternalDNS is configured during the next cluster reconcile; no additional OpenTofu apply is required.

[State coordination](state-locking.md).
63 changes: 63 additions & 0 deletions docs/state-locking.de.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# State-Locking auf STACKIT Object Storage

[English](state-locking.md) | [Deutsch](state-locking.de.md)

Der geprüfte EU01-Object-Storage-Endpunkt erzwingt keine bedingte Objekterzeugung
(`If-None-Match: *`). OpenTofus S3-Einstellung `use_lockfile = true` stellt dort
deshalb keinen gegenseitigen Ausschluss her. Ein erfolgreicher Plan oder ein
vorhandenes `.tflock`-Objekt belegt kein funktionierendes Locking. Die unabhängige
Reproduktion steht in [Issue #262](https://github.com/stackitcloud/stackit-landing-zone/issues/262).
Ein nativer Service-Feature-Request ist gestellt; ein Implementierungstermin ist
nicht angekündigt. Andere Regionen erst nach Prüfung ihres Endpunkts bewerten.

## Accelerator ohne Configurator

STACKIT S3 bleibt als State-Speicher nutzbar, wenn Zugriffe **organisatorisch
serialisiert** sind. Pro State darf genau ein CLI-/Pipeline-Verantwortlicher
gleichzeitig arbeiten, einschließlich Plan, Apply, Destroy, Import, Refresh,
Migration und Recovery. Getrennte Application-States können unabhängig laufen.
Verantwortung und gemeinsames Ausführungsverfahren festlegen; die Concurrency-
Gruppe eines Workflows koordiniert weder eine unabhängige CLI noch andere Repositories.

Das Application-Backend-Beispiel setzt ausdrücklich `use_lockfile = false`, weil
der Endpunkt diese Sperre nicht erzwingen kann. Gleichzeitiger Zugriff wird damit
nicht sicher. Wenn organisatorischer Einzelzugriff nicht garantiert werden kann,
ein Backend mit unabhängig geprüftem atomarem Locking verwenden. Keinen zweiten
leeren State anlegen und bestehende States nur mit geprüftem Verfahren und
wiederherstellbarer Sicherung migrieren.

## Vom Configurator verwaltete Ausführungen

Der Configurator serialisiert in PostgreSQL über alle API-Replikate hinweg:

- Ein physischer S3-Pfad aus Endpunkt, Bucket und Key gehört zu genau einem
logischen verwalteten Plattform- oder Application-State. Registrierungen für einen
anderen logischen State oder Arbeitsbereich werden auch bei parallelen
Anfragen abgewiesen.
- Plattform-Ausführungsreservierungen und Application-Dispatch-Reservierungen
sind atomare Datenbankoperationen. Getrennte States können parallel laufen;
derselbe State bekommt keine konkurrierenden verwalteten Ausführungen.
- Ablauf von Zugangsdaten oder Tickets belegt keinen gestoppten Prozess.
Bereits übernommene abgebrochene/fehlgeschlagene Plattform-Läufe bleiben bis
zur bestätigten Bereinigung oder explizit geprüften Recovery-Klärung gesperrt.
Application-Reconciliation sperrt auch Drift und Destroy; Ticketablauf allein
hebt die Sperre nicht auf.
- Quellen-, Eigentümer-, Mandanten-, gespeicherte Plan-, State-Versions- und
Recovery-Prüfungen gelten weiter. Historische Recovery-Nachweise bleiben unveränderlich.

Der S3-Deskriptor des Runners behält aus Kompatibilitätsgründen
`useLockfile=true`; das Objekt ist ein zusätzliches Signal, **nicht der
Synchronisationsmechanismus**. Diese Änderung migriert weder State-Objekte und
-Schlüssel noch gespeicherte Pläne. Historische Aliase eines physischen States
lassen die Migration zur Prüfung durch den Betreiber scheitern, statt einen
Schreiber auszuwählen oder die Zuordnung still zu überschreiben.

Die Koordination umfasst verwaltete Ausführungspfade einer Configurator-
Installation mit gemeinsamer Datenbank. Unabhängige CLI-Schreiber und andere
Installationen mit eigener Datenbank sind nicht eingeschlossen. Zugangsdaten und
State-Pfade einer Ausführungsverantwortung zuordnen; externer Notfallzugriff
benötigt gestoppte verwaltete Schreiber und State-Abgleich vor Wiederaufnahme.

Die Configurator-Betreiber-IaC nutzt ihre bestehende umgebungsweite Concurrency
über Workflows hinweg. Die Kunden-Ausführungsdatenbank schützt diese nicht;
direkte Betreiber-CLI-Zugriffe folgen ebenfalls dem Einzelzugriffsverfahren.
59 changes: 59 additions & 0 deletions docs/state-locking.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# State locking on STACKIT Object Storage

[English](state-locking.md) | [Deutsch](state-locking.de.md)

The tested EU01 Object Storage endpoint does not enforce conditional object
creation (`If-None-Match: *`). Therefore OpenTofu's S3 `use_lockfile = true` does
not establish mutual exclusion there. A successful Plan or an existing `.tflock`
object does not prove locking works. The independent reproduction is recorded in
[issue #262](https://github.com/stackitcloud/stackit-landing-zone/issues/262).
A native service feature request exists; implementation has no announced schedule.
Do not assume support in another region without qualifying that endpoint.

## Accelerator without Configurator

STACKIT S3 remains usable for State storage when access is **operationally
serialized**. Exactly one CLI/pipeline operator may work on a given State at a
time, including Plan, Apply, Destroy, import, refresh, migration and recovery.
Separate Application States can run independently. Agree ownership and a shared
execution procedure; one workflow's concurrency group does not coordinate an
independent CLI or another repository.

The application backend example sets `use_lockfile = false` explicitly because
this endpoint cannot enforce its lock. This does not make concurrent use safe.
If organizational single-writer operation cannot be guaranteed, use a backend
with independently qualified atomic locking. Do not create a second empty State
or migrate an existing State without a reviewed procedure and recoverable backup.

## Configurator-managed executions

The Configurator serializes executions in PostgreSQL, shared by all API replicas:

- A physical S3 location (endpoint, bucket, key) belongs to one logical managed
platform or application State. Registrations for another logical State or
workspace are rejected, including concurrent registrations.
- Platform execution reservations and application dispatch reservations are
atomic database operations. Separate States can run concurrently; the same
State cannot have competing managed executions.
- Expiry of a credential/ticket is not evidence that an issued process stopped.
Claimed cancelled/failed platform runs remain blocking until confirmed cleanup
or explicit reviewed recovery clearance. Application reconciliation blocks
drift and Destroy too; ticket expiry alone does not release the hold.
- Existing source, owner, tenant, saved-plan, State-version and recovery checks
remain in force. Historical recovery evidence stays immutable.

The Runner's S3 descriptor retains `useLockfile=true` for compatibility; that
object is an additional signal, **not the synchronization mechanism**. No State
object, State key or stored plan is migrated by this change. A historical physical
State alias causes migration to fail for operator review, rather than choosing a
writer or silently rewriting ownership.

This coordination covers the managed execution paths of one Configurator
installation using its database. Independent CLI writers or another installation
with a separate database are outside it. Keep those credentials and State paths
under one execution authority; any emergency external access requires stopping
managed writers and reconciling State before managed execution resumes.

Configurator operator IaC uses its existing cross-workflow environment concurrency.
It is not protected by the customer execution database; direct operator CLI access
must follow the same single-writer procedure.
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
-- Conditional S3 writes do not provide mutual exclusion on the qualified endpoint.
-- Bind each physical object to exactly one logical State before issuing credentials.
CREATE TABLE lzc.managed_state_targets (
target_sha256 text PRIMARY KEY CHECK(target_sha256 ~ '^[a-f0-9]{64}$'),
tenant_id uuid NOT NULL REFERENCES lzc.tenants(id),
owner_kind text NOT NULL CHECK(owner_kind IN ('platform','application')),
owner_id text NOT NULL,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
ALTER TABLE lzc.managed_state_targets ENABLE ROW LEVEL SECURITY;
ALTER TABLE lzc.managed_state_targets FORCE ROW LEVEL SECURITY;
CREATE POLICY managed_state_target_migration ON lzc.managed_state_targets TO configurator_migration USING(true) WITH CHECK(true);

CREATE FUNCTION lzc_auth.bind_managed_state_target(p_descriptor jsonb,p_tenant uuid,p_kind text,p_owner text) RETURNS void
LANGUAGE plpgsql SECURITY DEFINER SET search_path=pg_catalog AS $$
DECLARE fingerprint text; bound lzc.managed_state_targets;
BEGIN
IF p_descriptor->>'endpoint' IS NULL OR p_descriptor->>'bucket' IS NULL OR p_descriptor->>'key' IS NULL
THEN RAISE EXCEPTION 'state_backend_binding_invalid' USING ERRCODE='40001'; END IF;
fingerprint:=encode(sha256(convert_to(jsonb_build_array(p_descriptor->>'endpoint',p_descriptor->>'bucket',p_descriptor->>'key')::text,'UTF8')),'hex');
INSERT INTO lzc.managed_state_targets(target_sha256,tenant_id,owner_kind,owner_id)
VALUES(fingerprint,p_tenant,p_kind,p_owner) ON CONFLICT DO NOTHING;
SELECT * INTO bound FROM lzc.managed_state_targets WHERE target_sha256=fingerprint;
IF bound.tenant_id<>p_tenant OR bound.owner_kind<>p_kind OR bound.owner_id<>p_owner
THEN RAISE EXCEPTION 'state_backend_already_managed' USING ERRCODE='40001'; END IF;
END $$;
REVOKE ALL ON FUNCTION lzc_auth.bind_managed_state_target(jsonb,uuid,text,text) FROM PUBLIC;

CREATE FUNCTION lzc.guard_managed_state_target() RETURNS trigger
LANGUAGE plpgsql SECURITY DEFINER SET search_path=pg_catalog AS $$
DECLARE descriptor jsonb; backend uuid;
BEGIN
IF TG_TABLE_NAME='platform_states' THEN
FOREACH backend IN ARRAY ARRAY[NEW.backend_id,NEW.pending_backend_id] LOOP
IF backend IS NULL THEN CONTINUE; END IF;
SELECT b.descriptor INTO descriptor FROM lzc.state_backends b WHERE b.id=backend AND b.tenant_id=NEW.tenant_id;
PERFORM lzc_auth.bind_managed_state_target(descriptor,NEW.tenant_id,'platform',NEW.state_key);
END LOOP;
ELSE
IF NOT EXISTS(SELECT 1 FROM lzc.application_job_grants g WHERE g.job_id=NEW.job_id AND g.tenant_id=NEW.tenant_id AND g.state_key=NEW.descriptor->>'key')
THEN RAISE EXCEPTION 'state_backend_binding_invalid' USING ERRCODE='40001'; END IF;
PERFORM lzc_auth.bind_managed_state_target(NEW.descriptor,NEW.tenant_id,'application',
(SELECT instance_id::text FROM lzc.application_job_grants WHERE job_id=NEW.job_id));
END IF;
RETURN NEW;
END $$;
REVOKE ALL ON FUNCTION lzc.guard_managed_state_target() FROM PUBLIC;
CREATE TRIGGER managed_platform_state_target BEFORE INSERT OR UPDATE OF backend_id,pending_backend_id ON lzc.platform_states
FOR EACH ROW EXECUTE FUNCTION lzc.guard_managed_state_target();
CREATE TRIGGER managed_application_state_target BEFORE INSERT ON lzc.application_job_backends
FOR EACH ROW EXECUTE FUNCTION lzc.guard_managed_state_target();

-- Conflicting historical registrations require operator review; never silently
-- assign one writer's existing State to another logical platform or application.
DO $$
DECLARE binding record;
BEGIN
FOR binding IN
SELECT b.descriptor,s.tenant_id,'platform'::text AS kind,s.state_key AS owner
FROM lzc.platform_states s JOIN lzc.state_backends b ON b.id=s.backend_id OR b.id=s.pending_backend_id
UNION ALL
SELECT b.descriptor,b.tenant_id,'application',g.instance_id::text
FROM lzc.application_job_backends b JOIN lzc.application_job_grants g ON g.job_id=b.job_id
LOOP
PERFORM lzc_auth.bind_managed_state_target(binding.descriptor,binding.tenant_id,binding.kind,binding.owner);
END LOOP;
END $$;

-- Expiring a capability cannot stop a process that already received S3 credentials.
CREATE FUNCTION lzc.guard_platform_state_serialization() RETURNS trigger
LANGUAGE plpgsql SECURITY DEFINER SET search_path=pg_catalog AS $$
BEGIN
IF NEW.state_key IS NULL THEN RETURN NEW; END IF;
PERFORM pg_advisory_xact_lock(hashtextextended('managed-platform-state:'||NEW.state_key,0));
IF EXISTS(SELECT 1 FROM lzc.plan_runs r WHERE r.state_key=NEW.state_key AND r.id<>NEW.id
AND r.input_claimed AND r.status IN ('failed','cancelled') AND NOT r.cleaned
AND NOT EXISTS(SELECT 1 FROM lzc.runner_cleanup_receipts receipt WHERE receipt.runner_id=r.id AND receipt.app_id=r.runner_app_id)
AND NOT EXISTS(SELECT 1 FROM lzc.platform_reconciliations receipt WHERE receipt.run_id=r.id AND receipt.state_key=r.state_key))
THEN RAISE EXCEPTION 'state_locked' USING ERRCODE='40001'; END IF;
RETURN NEW;
END $$;
REVOKE ALL ON FUNCTION lzc.guard_platform_state_serialization() FROM PUBLIC;
CREATE TRIGGER platform_state_serialization BEFORE INSERT ON lzc.plan_runs
FOR EACH ROW EXECUTE FUNCTION lzc.guard_platform_state_serialization();

-- Reconciliation remains a durable hold for every operation, including drift
-- and destroy. Ticket expiry alone never proves that the previous writer stopped.
CREATE OR REPLACE FUNCTION lzc_auth.application_execution_blocked(p_job uuid,p_instance uuid,p_tenant uuid,p_purpose text,p_apply_only boolean DEFAULT false) RETURNS boolean
LANGUAGE sql SECURITY DEFINER SET search_path=pg_catalog AS $$
SELECT EXISTS(
SELECT 1 FROM lzc.application_dispatches d WHERE d.tenant_id=p_tenant AND d.instance_id=p_instance AND d.job_id<>p_job
AND (NOT p_apply_only OR EXISTS(SELECT 1 FROM lzc.application_jobs j WHERE j.id=d.job_id AND j.operation='apply'))
AND (d.status NOT IN ('succeeded','failed','reconciliation_required') OR
(d.status='reconciliation_required' AND NOT lzc_auth.application_recovery_replan_valid(d.job_id)))
);
$$;
Loading
Loading