diff --git a/README.md b/README.md index 82b3ec1..c277429 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Patternizer -![Version: 2.3.1](https://img.shields.io/badge/Version-2.3.1-informational?style=flat-square) +![Version: 2.13.1](https://img.shields.io/badge/Version-2.13.1-informational?style=flat-square) [![Quay Repository](https://img.shields.io/badge/Quay.io-patternizer-blue?logo=quay)](https://quay.io/repository/validatedpatterns/patternizer) [![CI Pipeline](https://github.com/validatedpatterns/patternizer/actions/workflows/build-push.yaml/badge.svg?branch=main)](https://github.com/validatedpatterns/patternizer/actions/workflows/build-push.yaml) @@ -120,7 +120,225 @@ pattern() { } ``` -Then run commands as `pattern init`, `pattern upgrade`, etc. +Then run commands as `pattern init`, `pattern upgrade`, `pattern airgap init`, etc. + +#### **Airgap / disconnected init:** + +Generate an oc-mirror v2 `imagesetconfig.yaml` plus disconnected install helpers from a local absolute path or a Git URL. + +Path-based Helm charts referenced by pattern applications (`path: charts/...`) +are scanned for container images (via `helm template` and values scrape); those +images are emitted under `additionalImages` only (the charts are not added to +`helm.local`). Remote OCI charts (`chart:` / `chartVersion:`) are resolved and +scanned as well. Use `--extra-config` for extra operators, images, and charts +not declared in values: + +```yaml +# ~/extra-images.yaml +operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.22 + packages: + - name: lvms-operator +additionalImages: + - name: registry.example.com/app:1.0 +helm: + local: + - name: my-chart + path: /abs/path/to/chart +``` + +```bash +# Absolute path to a local pattern checkout +podman run --pull=newer -v "$PWD:$PWD:z" -v "$HOME:$HOME:z" -w "$PWD" quay.io/validatedpatterns/patternizer \ + airgap init /path/to/multicloud-gitops \ + --extra-config "$HOME/extra-images.yaml" \ + --output-dir "$PWD/airgap-out" + +# Include OpenShift platform channel and tag OLM catalogs for that version +podman run --pull=newer -v "$PWD:$PWD:z" -v "$HOME:$HOME:z" -w "$PWD" quay.io/validatedpatterns/patternizer \ + airgap init /path/to/multicloud-gitops \ + --ocp-version 4.22 \ + --extra-config "$HOME/extra-images.yaml" \ + --output-dir "$PWD/airgap-out" + +# Or clone from Git into --output-dir (override with --clone-dir) +podman run --pull=newer -v "$PWD:$PWD:z" -w "$PWD" quay.io/validatedpatterns/patternizer \ + airgap init https://github.com/validatedpatterns/multicloud-gitops \ + --git-ref main \ + --download-oc-mirror \ + --output-dir "$PWD/airgap-out" +``` + +`--registry` is optional (sets `helmRepoUrl` in `values-disconnected.yaml` and the +registry placeholder in `mirror-commands.sh`). `--ocp-version` is optional: when set +it adds `mirror.platform` with channel `stable-` and tags pattern-discovered +OLM catalog indexes / CatalogSource names with that version (default catalog tag is +`4.22` when omitted). + +Git clones land under `--output-dir//`. Run `oc-mirror` yourself using +`mirror-commands.sh` (patternizer only generates the config and helpers). + +Or from a pattern repository after `Makefile-common` includes the target: + +```bash +./pattern.sh make airgap-init \ + AIRGAP_EXTRA_OPTS="--extra-config $HOME/extra-images.yaml" +# Optional: AIRGAP_OCP_VERSION=4.22 AIRGAP_REGISTRY=mirror.example.com:8443 +``` + +Outputs under `--output-dir`: + +- `imagesetconfig.yaml` +- `values-disconnected.yaml` +- `mirror-commands.sh` +- `CHECKLIST.md` (pull secrets, CatalogSources, Git, Vault/ESO) +- cloned pattern repo (when source is a Git URL) + +Parse oc-mirror logs for skipped `oci://` related images: + +```bash +pattern airgap parse-mirror-log /path/to/oc-mirror.log +``` + +#### **Airgap trust CA (Argo CD / GitOps):** + +Generate apply-ready artifacts so OpenShift GitOps trusts a private or self-signed +CA for HTTPS Git hosts (e.g. Gitea) or OCI registries (e.g. Quay mirror). Patternizer +only writes files; you apply them on a bastion with `oc` and your kubeconfig. + +```bash +# Gitea (or any Git host) CA +podman run --pull=newer -v "$PWD:$PWD:z" -v /home/fedora/gitea_container_certs:/certs:z \ + -w "$PWD" quay.io/validatedpatterns/patternizer \ + airgap trust-ca \ + --ca /certs/ca.crt \ + --host gitea-gitea.apps.airgap.rhuk.lab \ + --namespace vp-gitops \ + --output-dir "$PWD/airgap-trust" + +# Mirror registry CA (hostname only; no port) +patternizer airgap trust-ca \ + --ca /etc/pki/ca-trust/source/anchors/quay-rootCA.pem \ + --host mirror.airgap.rhuk.lab \ + --output-dir ./airgap-trust +``` + +`--ca` may be a PEM file or a directory of `.crt`/`.pem` files. `--host` is +repeatable (or comma-separated); URLs and `:port` suffixes are normalized to +hostname keys for `argocd-tls-certs-cm`. + +Outputs under `--output-dir`: + +- `argocd-tls-certs-patch.yaml` +- `trusted-ca-bundle-append.pem` +- `apply-trust-ca.sh` (run on the bastion) +- `README.md` + +```bash +# On the bastion (cluster reachable via ~/.kube/config or KUBECONFIG): +./airgap-trust/apply-trust-ca.sh +``` + +#### **Airgap registry repo-creds (Argo CD auth):** + +After TLS trust is in place, Argo CD still needs credentials to pull OCI charts from a +private mirror (otherwise sync fails with `401 unauthorized`). Generate a +`repo-creds` Secret from a docker config **or** username/password — credentials are +never hardcoded in patternizer: + +```bash +# From ~/.docker/config.json (or podman containers auth.json) +patternizer airgap repo-creds \ + --docker-config ~/.docker/config.json \ + --url mirror.airgap.rhuk.lab:8443 \ + --namespace vp-gitops \ + --output-dir ./airgap-repo-creds + +# Or explicit credentials (prefer --password-file) +patternizer airgap repo-creds \ + --url mirror.example.com:8443 \ + --username init \ + --password-file /path/to/registry.password \ + --output-dir ./airgap-repo-creds +``` + +`--url` keeps the port (e.g. `:8443`) for Argo CD. With multiple `auths` in the docker +config, `--url` selects which registry entry to use. + +Outputs under `--output-dir`: + +- `argocd-repo-creds-secret.yaml` (mode `0600` — **do not commit**) +- `apply-repo-creds.sh` +- `README.md` + +```bash +./airgap-repo-creds/apply-repo-creds.sh +``` + +#### **Airgap bootstrap (recommended one-shot):** + +Configure hostnames, CA files, and registry credentials in the pattern’s +`values-secret.yaml` / `~/values-secret-.yaml` under a top-level `airgap:` +section (same `path` / `value` field style as Vault secrets; **not** loaded into Vault). + +Then install with the airgap-aware target (leaves `make install` unchanged for connected clusters): + +```bash +# Edit airgap: in ~/values-secret-multicloud-gitops.yaml (see values-secret.yaml.template) +./pattern.sh make install-airgap +``` + +That runs: + +1. `airgap-trust-pre` — Proxy CA / `user-ca-bundle` +2. **`make install` in the background** (log: `airgap-bootstrap/install.log`) +3. `airgap-wait-gitops` — waits for `vp-gitops` while install is still running +4. `airgap-trust-post` — repo-creds + TLS (unblocks Argo CD / install) +5. Waits for the background `make install` to finish + +This avoids the deadlock where install never completes until post trust is applied. +Default wait is 20 minutes (`AIRGAP_WAIT_SECONDS=1200`): + +```bash +./pattern.sh make install-airgap AIRGAP_WAIT_SECONDS=1800 +``` + +Stop a running session with **Ctrl-C**, or from another shell: `./pattern.sh make airgap-abort`. + +Declare CatalogSources in `~/values-secret-.yaml` under `airgap:`: + +```yaml +airgap: + registry: mirror.airgap.rhuk.lab:8443 + ocpVersion: "4.22" # derives cs-*-operator-index-v4-22 + # Or: + # catalogSources: + # redhat: cs-redhat-operator-index-v4-22 + # community: cs-community-operator-index-v4-22 + # Or: discoverCatalogs: true # uses oc get catalogsource +``` + +Manual generate: + +```bash +patternizer airgap bootstrap /home/fedora/multicloud-gitops \ + --output-dir ./airgap-bootstrap +# or: --discover-catalogs # requires oc/kubeconfig +``` + +Proxy backup is written to `./airgap-bootstrap/backup/proxy-cluster.yaml`. Restore with: + +```bash +./pattern.sh make airgap-restore-proxy +``` + +Overrides: `AIRGAP_BOOTSTRAP_DIR`, `AIRGAP_SECRETS_FILE`, `AIRGAP_PATTERNIZER_BIN`, +`AIRGAP_PATTERNIZER_IMAGE`, `AIRGAP_WAIT_SECONDS`. + +Lower-level commands `airgap trust-ca` and `airgap repo-creds` remain available +for one-off use without a values-secret file. + +See the [disconnected installation guide](https://validatedpatterns.io/learn/disconnected-installation/). What upgrade does: diff --git a/src/cmd/airgap.go b/src/cmd/airgap.go new file mode 100644 index 0000000..7cf406a --- /dev/null +++ b/src/cmd/airgap.go @@ -0,0 +1,350 @@ +package cmd + +import ( + "fmt" + "path/filepath" + "strings" + + "github.com/spf13/cobra" + + "github.com/validatedpatterns/patternizer/internal/airgap" +) + +func newAirgapCommand() *cobra.Command { + opts := airgap.DefaultOptions() + var clusterGroups string + + airgapCmd := &cobra.Command{ + Use: "airgap", + Short: "Disconnected / airgap helpers for Validated Patterns", + Long: "Generate oc-mirror ImageSetConfiguration and disconnected install artifacts from pattern values files.", + } + + initCmd := &cobra.Command{ + Use: "init ", + Short: "Generate imagesetconfig.yaml and disconnected install artifacts", + Long: `Analyze a Validated Pattern repository and generate: + + - imagesetconfig.yaml (oc-mirror v2) + - values-disconnected.yaml + - mirror-commands.sh + - CHECKLIST.md + +Pattern source must be an absolute path or a Git URL. +Git clones land in --output-dir (or --clone-dir if set). + +Path-based (local) Helm charts referenced by applications are scanned for +container images only (added to additionalImages; charts themselves are not +mirrored). Provide extra operators, images, and helm.local charts via --extra-config. + +Examples: + patternizer airgap init /home/user/patterns/multicloud-gitops \ + --extra-config ~/extra-images.yaml + + patternizer airgap init /home/user/patterns/multicloud-gitops \ + --ocp-version 4.22 \ + --extra-config ~/extra-images.yaml + + patternizer airgap init https://github.com/validatedpatterns/multicloud-gitops \ + --git-ref main --download-oc-mirror +`, + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + opts.PatternSource = args[0] + if clusterGroups != "" { + for _, g := range strings.Split(clusterGroups, ",") { + g = strings.TrimSpace(g) + if g != "" { + opts.ClusterGroups = append(opts.ClusterGroups, g) + } + } + } + if opts.OutputDir != "" && !filepath.IsAbs(opts.OutputDir) { + abs, err := filepath.Abs(opts.OutputDir) + if err != nil { + return err + } + opts.OutputDir = abs + } + if opts.ExtraConfigPath != "" && !filepath.IsAbs(opts.ExtraConfigPath) { + abs, err := filepath.Abs(opts.ExtraConfigPath) + if err != nil { + return err + } + opts.ExtraConfigPath = abs + } + inv, err := airgap.Init(&opts) + if err != nil { + return err + } + fmt.Fprintf(cmd.OutOrStdout(), "Airgap init complete for pattern %q\n", inv.PatternName) + fmt.Fprintf(cmd.OutOrStdout(), " source: %s\n", opts.PatternSource) + fmt.Fprintf(cmd.OutOrStdout(), " pattern dir: %s\n", inv.ResolvedDir) + extraOps := 0 + for _, cat := range inv.ExtraOperators { + extraOps += len(cat.Packages) + } + fmt.Fprintf(cmd.OutOrStdout(), " operators: %d\n", len(inv.Operators)+extraOps) + fmt.Fprintf(cmd.OutOrStdout(), " remote charts: %d\n", len(inv.RemoteCharts)) + fmt.Fprintf(cmd.OutOrStdout(), " local charts: %d\n", len(inv.LocalCharts)) + fmt.Fprintf(cmd.OutOrStdout(), " extra images: %d\n", len(inv.AdditionalImages)) + fmt.Fprintf(cmd.OutOrStdout(), " output: %s\n", inv.OutputDir) + if opts.ExtraConfigPath != "" { + fmt.Fprintf(cmd.OutOrStdout(), " extra-config: %s (merged into imagesetconfig.yaml)\n", opts.ExtraConfigPath) + } + if len(inv.Warnings) > 0 { + fmt.Fprintf(cmd.OutOrStdout(), " warnings: %d (see CHECKLIST.md)\n", len(inv.Warnings)) + } + return nil + }, + } + + initCmd.Flags().StringVar(&opts.OCPVersion, "ocp-version", opts.OCPVersion, "When set, include OpenShift platform channel stable- and tag OLM catalogs / CatalogSources with that version (e.g. 4.22)") + initCmd.Flags().StringVar(&opts.Registry, "registry", opts.Registry, "Optional mirror registry hostname for values-disconnected.yaml and mirror-commands.sh") + initCmd.Flags().StringVar(&opts.OutputDir, "output-dir", opts.OutputDir, "Directory for generated artifacts (and Git clones by default)") + initCmd.Flags().StringVar(&opts.HelmRepo, "helm-repo", opts.HelmRepo, "OCI registry/org for Validated Patterns charts") + initCmd.Flags().BoolVar(&opts.WriteValues, "write-values", true, "Emit values-disconnected.yaml overlay") + initCmd.Flags().BoolVar(&opts.SkipOCIResolve, "skip-oci-resolve", false, "Skip live OCI chart version resolution and image discovery (wildcards will not be resolved)") + initCmd.Flags().StringVar(&clusterGroups, "cluster-groups", "", "Comma-separated cluster group names (default: all)") + initCmd.Flags().StringVar(&opts.GitRef, "git-ref", "", "Branch or tag to clone when the source is a Git URL") + initCmd.Flags().StringVar(&opts.CloneDir, "clone-dir", "", "Directory for Git clones (default: --output-dir)") + initCmd.Flags().BoolVar(&opts.DownloadOCMirror, "download-oc-mirror", false, "Download oc-mirror.tar.gz and oc-mirror.rhel9.tar.gz into --output-dir") + initCmd.Flags().StringVar(&opts.ExtraConfigPath, "extra-config", "", "External YAML with operators, additionalImages, and helm.local entries to merge into imagesetconfig") + + var mirrorLogPath string + parseLogCmd := &cobra.Command{ + Use: "parse-mirror-log [logfile]", + Short: "Extract skipped oci:// related images from an oc-mirror log", + Args: cobra.MaximumNArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + path := mirrorLogPath + if len(args) == 1 { + path = args[0] + } + if path == "" { + return fmt.Errorf("provide a log file path") + } + images, err := airgap.ParseMirrorLog(path) + if err != nil { + return err + } + fmt.Fprint(cmd.OutOrStdout(), airgap.FormatSkippedImagesYAML(images)) + if len(images) > 0 { + fmt.Fprintf(cmd.ErrOrStderr(), "Found %d skipped oci:// image(s)\n", len(images)) + } + return nil + }, + } + parseLogCmd.Flags().StringVar(&mirrorLogPath, "log", "", "Path to oc-mirror log file") + + var trustCA airgap.TrustCAOptions + var trustHosts []string + trustCmd := &cobra.Command{ + Use: "trust-ca", + Short: "Generate Argo CD TLS trust artifacts for a private CA", + Long: `Generate apply-ready artifacts so OpenShift GitOps / Argo CD trusts +a CA for one or more hostnames (Gitea, mirror registry, etc.). + +Does not contact the cluster. Copy the output directory to a bastion and run +./apply-trust-ca.sh with a working kubeconfig. + +Artifacts: + - argocd-tls-certs-patch.yaml + - trusted-ca-bundle-append.pem + - apply-trust-ca.sh + - README.md + +Examples: + patternizer airgap trust-ca \ + --ca /home/fedora/gitea_container_certs/ca.crt \ + --host gitea-gitea.apps.airgap.rhuk.lab \ + --namespace vp-gitops \ + --output-dir ./airgap-trust + + patternizer airgap trust-ca \ + --ca /etc/pki/ca-trust/source/anchors/quay-rootCA.pem \ + --host mirror.airgap.rhuk.lab \ + --output-dir ./airgap-trust +`, + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + trustCA.Hosts = trustHosts + if trustCA.OutputDir != "" && !filepath.IsAbs(trustCA.OutputDir) { + abs, err := filepath.Abs(trustCA.OutputDir) + if err != nil { + return err + } + trustCA.OutputDir = abs + } + res, err := airgap.GenerateTrustCA(&trustCA) + if err != nil { + return err + } + fmt.Fprintf(cmd.OutOrStdout(), "Airgap trust-ca artifacts written to %s\n", res.OutputDir) + fmt.Fprintf(cmd.OutOrStdout(), " namespace: %s\n", res.Namespace) + fmt.Fprintf(cmd.OutOrStdout(), " repo-server: %s\n", res.RepoServer) + fmt.Fprintf(cmd.OutOrStdout(), " hosts: %s\n", strings.Join(res.Hosts, ", ")) + fmt.Fprintf(cmd.OutOrStdout(), " ca bytes: %d\n", res.CABytes) + fmt.Fprintf(cmd.OutOrStdout(), "Apply on a host with oc/kubeconfig:\n %s/apply-trust-ca.sh\n", res.OutputDir) + return nil + }, + } + trustCmd.Flags().StringVar(&trustCA.CAPath, "ca", "", "Path to a PEM file or directory of .crt/.pem files (required)") + trustCmd.Flags().StringArrayVar(&trustHosts, "host", nil, "Hostname to trust (repeatable; comma-separated allowed; no port)") + trustCmd.Flags().StringVar(&trustCA.Namespace, "namespace", "vp-gitops", "Argo CD / OpenShift GitOps namespace") + trustCmd.Flags().StringVar(&trustCA.OutputDir, "output-dir", "./airgap-trust", "Directory for generated artifacts") + trustCmd.Flags().StringVar(&trustCA.RepoServer, "repo-server", "", "Repo-server deployment name (default: -repo-server)") + _ = trustCmd.MarkFlagRequired("ca") + _ = trustCmd.MarkFlagRequired("host") + + var repoCreds airgap.RepoCredsOptions + repoCredsCmd := &cobra.Command{ + Use: "repo-creds", + Short: "Generate Argo CD registry repo-creds Secret artifacts", + Long: `Generate apply-ready artifacts so OpenShift GitOps / Argo CD can +authenticate to a private OCI/Helm registry (Quay mirror, Harbor, etc.). + +Credentials are never hardcoded. Provide either: + + --docker-config ~/.docker/config.json (or containers auth.json) + --username USER --password PASS (or --password-file) + +Does not contact the cluster. Run ./apply-repo-creds.sh on a bastion with kubeconfig. + +Artifacts: + - argocd-repo-creds-secret.yaml (mode 0600; do not commit) + - apply-repo-creds.sh + - README.md + +Examples: + patternizer airgap repo-creds \ + --docker-config ~/.docker/config.json \ + --url mirror.airgap.rhuk.lab:8443 \ + --namespace vp-gitops \ + --output-dir ./airgap-repo-creds + + patternizer airgap repo-creds \ + --url mirror.example.com:8443 \ + --username init \ + --password-file /path/to/registry.password \ + --output-dir ./airgap-repo-creds + +Typically run after "airgap trust-ca" for the same registry so TLS and auth are both set. +`, + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + if repoCreds.OutputDir != "" && !filepath.IsAbs(repoCreds.OutputDir) { + abs, err := filepath.Abs(repoCreds.OutputDir) + if err != nil { + return err + } + repoCreds.OutputDir = abs + } + res, err := airgap.GenerateRepoCreds(&repoCreds) + if err != nil { + return err + } + fmt.Fprintf(cmd.OutOrStdout(), "Airgap repo-creds artifacts written to %s\n", res.OutputDir) + fmt.Fprintf(cmd.OutOrStdout(), " namespace: %s\n", res.Namespace) + fmt.Fprintf(cmd.OutOrStdout(), " secret: %s\n", res.SecretName) + fmt.Fprintf(cmd.OutOrStdout(), " url: %s\n", res.URL) + fmt.Fprintf(cmd.OutOrStdout(), " username: %s\n", res.Username) + fmt.Fprintf(cmd.OutOrStdout(), " repo-server: %s\n", res.RepoServer) + fmt.Fprintf(cmd.OutOrStdout(), "Apply on a host with oc/kubeconfig:\n %s/apply-repo-creds.sh\n", res.OutputDir) + fmt.Fprintf(cmd.OutOrStdout(), "Warning: %s/argocd-repo-creds-secret.yaml contains a password — do not commit it.\n", res.OutputDir) + return nil + }, + } + repoCredsCmd.Flags().StringVar(&repoCreds.DockerConfig, "docker-config", "", "Path to docker/containers config.json with auths (alternative to --username)") + repoCredsCmd.Flags().StringVar(&repoCreds.Username, "username", "", "Registry username (use with --password or --password-file)") + repoCredsCmd.Flags().StringVar(&repoCreds.Password, "password", "", "Registry password (prefer --password-file)") + repoCredsCmd.Flags().StringVar(&repoCreds.PasswordFile, "password-file", "", "File containing the registry password") + repoCredsCmd.Flags().StringVar(&repoCreds.URL, "url", "", "Registry URL for Argo CD repo-creds (host:port; required with --username, selects auth with --docker-config)") + repoCredsCmd.Flags().StringVar(&repoCreds.Namespace, "namespace", "vp-gitops", "Argo CD / OpenShift GitOps namespace") + repoCredsCmd.Flags().StringVar(&repoCreds.OutputDir, "output-dir", "./airgap-repo-creds", "Directory for generated artifacts") + repoCredsCmd.Flags().StringVar(&repoCreds.SecretName, "name", "", "Secret name (default: -repo-creds)") + repoCredsCmd.Flags().StringVar(&repoCreds.RepoServer, "repo-server", "", "Repo-server deployment name (default: -repo-server)") + + var bootstrapOpts airgap.BootstrapOptions + bootstrapCmd := &cobra.Command{ + Use: "bootstrap [pattern-dir]", + Short: "Generate unified airgap trust artifacts from values-secret airgap: config", + Long: `Read the airgap: section from values-secret.yaml (same path/value field style +as Vault secrets) and generate a single apply script for: + + - Cluster Proxy / user-ca-bundle (patterns-operator git clone TLS) + - Argo CD TLS trust (argocd-tls-certs-cm + trusted-ca-bundle) + - Argo CD registry repo-creds + - Rendered values-global.yaml and variants/*/values-*.yaml + (helmRepoUrl + CatalogSource names) + +CatalogSources (pick one approach in airgap:): + - catalogSources: { redhat: cs-..., community: cs-... } + - ocpVersion: "4.22" (derives oc-mirror style names) + - discoverCatalogs: true (or --discover-catalogs; requires oc) + +Also set airgap.registry (e.g. mirror.example.com:8443) for helmRepoUrl. +`, + Args: cobra.MaximumNArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + if len(args) == 1 { + bootstrapOpts.PatternDir = args[0] + if !filepath.IsAbs(bootstrapOpts.PatternDir) { + abs, err := filepath.Abs(bootstrapOpts.PatternDir) + if err != nil { + return err + } + bootstrapOpts.PatternDir = abs + } + } + if bootstrapOpts.OutputDir != "" && !filepath.IsAbs(bootstrapOpts.OutputDir) { + abs, err := filepath.Abs(bootstrapOpts.OutputDir) + if err != nil { + return err + } + bootstrapOpts.OutputDir = abs + } + res, err := airgap.GenerateBootstrap(&bootstrapOpts) + if err != nil { + return err + } + fmt.Fprintf(cmd.OutOrStdout(), "Airgap bootstrap artifacts written to %s\n", res.OutputDir) + fmt.Fprintf(cmd.OutOrStdout(), " secrets file: %s\n", res.SecretsFile) + fmt.Fprintf(cmd.OutOrStdout(), " argocd namespace: %s\n", res.ArgocdNamespace) + fmt.Fprintf(cmd.OutOrStdout(), " clusterProxy CAs: %d\n", res.ClusterProxyCAs) + fmt.Fprintf(cmd.OutOrStdout(), " argocd TLS hosts: %d\n", res.ArgocdTLSHosts) + fmt.Fprintf(cmd.OutOrStdout(), " registries: %d\n", res.Registries) + if len(res.RenderedFiles) > 0 { + fmt.Fprintf(cmd.OutOrStdout(), " rendered values: %d file(s)\n", len(res.RenderedFiles)) + if res.HelmRepoURL != "" { + fmt.Fprintf(cmd.OutOrStdout(), " helmRepoUrl: %s\n", res.HelmRepoURL) + } + if res.CommunityCS != "" { + fmt.Fprintf(cmd.OutOrStdout(), " patterns CS: %s\n", res.CommunityCS) + } + if res.RedHatCS != "" { + fmt.Fprintf(cmd.OutOrStdout(), " gitops/ACM CS: %s\n", res.RedHatCS) + } + } + fmt.Fprintf(cmd.OutOrStdout(), "Apply on a host with oc/kubeconfig:\n %s/apply-airgap.sh\n", res.OutputDir) + fmt.Fprintf(cmd.OutOrStdout(), "Restore Proxy backup:\n %s/restore-proxy-cluster.sh\n", res.OutputDir) + return nil + }, + } + bootstrapCmd.Flags().StringVar(&bootstrapOpts.SecretsFile, "secrets-file", "", "Path to values-secret.yaml (default: VP resolution order)") + bootstrapCmd.Flags().StringVar(&bootstrapOpts.OutputDir, "output-dir", "./airgap-bootstrap", "Directory for generated artifacts") + bootstrapCmd.Flags().StringVar(&bootstrapOpts.Namespace, "namespace", "", "Override airgap.argocdNamespace") + bootstrapCmd.Flags().StringVar(&bootstrapOpts.RepoServer, "repo-server", "", "Repo-server deployment name (default: -repo-server)") + bootstrapCmd.Flags().StringVar(&bootstrapOpts.PatternName, "pattern-name", "", "Pattern name for ~/values-secret-.yaml lookup") + bootstrapCmd.Flags().BoolVar(&bootstrapOpts.DiscoverCatalogs, "discover-catalogs", false, "Query the cluster (oc) for CatalogSource names") + bootstrapCmd.Flags().BoolVar(&bootstrapOpts.SkipRenderValues, "skip-render-values", false, "Do not rewrite values-global.yaml / variants values files") + bootstrapCmd.Flags().StringVar(&bootstrapOpts.Registry, "registry", "", "Override airgap.registry for helmRepoUrl") + bootstrapCmd.Flags().StringVar(&bootstrapOpts.OCPVersion, "ocp-version", "", "Override airgap.ocpVersion for CatalogSource name derivation") + + airgapCmd.AddCommand(initCmd) + airgapCmd.AddCommand(parseLogCmd) + airgapCmd.AddCommand(trustCmd) + airgapCmd.AddCommand(repoCredsCmd) + airgapCmd.AddCommand(bootstrapCmd) + return airgapCmd +} diff --git a/src/cmd/root.go b/src/cmd/root.go index bd9fc4f..af5faf4 100644 --- a/src/cmd/root.go +++ b/src/cmd/root.go @@ -59,6 +59,7 @@ copy updated Makefile-common and pattern.sh, and optionally replace or update th upgradeCmd.Flags().BoolVar(&replaceMakefile, "replace-makefile", false, "Replace the existing Makefile with the default") rootCmd.AddCommand(upgradeCmd) + rootCmd.AddCommand(newAirgapCommand()) // Hide the completion command from help since this is primarily used in containers rootCmd.CompletionOptions.HiddenDefaultCmd = true diff --git a/src/cmd/upgrade_test.go b/src/cmd/upgrade_test.go index 4d1d3f9..f3543b5 100644 --- a/src/cmd/upgrade_test.go +++ b/src/cmd/upgrade_test.go @@ -29,13 +29,13 @@ func cloneMCGWithCommon(dir string) { cmd.Dir = dir session, err = gexec.Start(cmd, GinkgoWriter, GinkgoWriter) Expect(err).NotTo(HaveOccurred()) - Eventually(session).Should(gexec.Exit(0), "Could not fetch SHA for last MCG commit before patternizer was run") + Eventually(session, "60s").Should(gexec.Exit(0), "Could not fetch SHA for last MCG commit before patternizer was run") cmd = exec.Command("git", "checkout", "02954705e3d58e4823cd195beb8c31418f730830") cmd.Dir = dir session, err = gexec.Start(cmd, GinkgoWriter, GinkgoWriter) Expect(err).NotTo(HaveOccurred()) - Eventually(session).Should(gexec.Exit(0), "Could not checkout SHA for last MCG commit before patternizer was run") + Eventually(session, "30s").Should(gexec.Exit(0), "Could not checkout SHA for last MCG commit before patternizer was run") } var _ = Describe("patternizer upgrade", func() { diff --git a/src/go.mod b/src/go.mod index 57e2cf7..4c05ebc 100644 --- a/src/go.mod +++ b/src/go.mod @@ -3,6 +3,7 @@ module github.com/validatedpatterns/patternizer go 1.26.3 require ( + github.com/Masterminds/semver/v3 v3.4.0 github.com/onsi/ginkgo/v2 v2.32.0 github.com/onsi/gomega v1.42.1 github.com/spf13/cobra v1.10.2 @@ -10,14 +11,12 @@ require ( ) require ( - github.com/Masterminds/semver/v3 v3.4.0 // indirect github.com/go-logr/logr v1.4.3 // indirect github.com/go-task/slim-sprig/v3 v3.0.0 // indirect github.com/google/go-cmp v0.7.0 // indirect github.com/google/pprof v0.0.0-20260402051712-545e8a4df936 // indirect github.com/inconshreveable/mousetrap v1.1.0 // indirect - github.com/rogpeppe/go-internal v1.14.1 // indirect - github.com/spf13/pflag v1.0.10 // indirect + github.com/spf13/pflag v1.0.9 // indirect go.yaml.in/yaml/v3 v3.0.4 // indirect golang.org/x/mod v0.36.0 // indirect golang.org/x/net v0.56.0 // indirect @@ -25,5 +24,4 @@ require ( golang.org/x/sys v0.46.0 // indirect golang.org/x/text v0.38.0 // indirect golang.org/x/tools v0.45.0 // indirect - gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect ) diff --git a/src/go.sum b/src/go.sum index f11141d..77efd39 100644 --- a/src/go.sum +++ b/src/go.sum @@ -23,11 +23,8 @@ github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2 github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= github.com/joshdk/go-junit v1.0.0 h1:S86cUKIdwBHWwA6xCmFlf3RTLfVXYQfvanM5Uh+K6GE= github.com/joshdk/go-junit v1.0.0/go.mod h1:TiiV0PqkaNfFXjEiyjWM3XXrhVyCa1K4Zfga6W52ung= -github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= -github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= -github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= github.com/maruel/natural v1.1.1 h1:Hja7XhhmvEFhcByqDoHz9QZbkWey+COd9xWfCfn1ioo= @@ -40,14 +37,13 @@ github.com/onsi/gomega v1.42.1 h1:iN1rCUX+44NZ1Dc97MPoeFYbFR0vh8zxoxMFwKdyZ6I= github.com/onsi/gomega v1.42.1/go.mod h1:REff/hsDsodHoKlWsP2mAPhu1+5/6hVYNf9rIEBpeSg= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ= -github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc= +github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII= +github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o= github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU= github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4= +github.com/spf13/pflag v1.0.9 h1:9exaQaMOCwffKiiiYk6/BndUBv+iRViNW+4lEMi0PvY= github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= -github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk= -github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= github.com/tidwall/gjson v1.18.0 h1:FIDeeyB800efLX89e5a8Y0BNH+LOngJyGrIWxG2FKQY= @@ -74,8 +70,7 @@ golang.org/x/tools v0.45.0 h1:18qN3FAooORvApf5XjCXgsuayZOEtXf6JK18I3+ONa8= golang.org/x/tools v0.45.0/go.mod h1:LuUGqqaXcXMEFEruIVJVm5mgDD8vww/z/SR1gQ4uE/0= google.golang.org/protobuf v1.36.7 h1:IgrO7UwFQGJdRNXH/sQux4R1Dj1WAKcLElzeeRaXV2A= google.golang.org/protobuf v1.36.7/go.mod h1:jduwjTPXsFjZGTmRluh+L6NjiWu7pchiJ2/5YcXBHnY= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= -gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/src/internal/airgap/airgap_suite_test.go b/src/internal/airgap/airgap_suite_test.go new file mode 100644 index 0000000..4f5c4f8 --- /dev/null +++ b/src/internal/airgap/airgap_suite_test.go @@ -0,0 +1,13 @@ +package airgap_test + +import ( + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func TestAirgap(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "Airgap Suite") +} diff --git a/src/internal/airgap/artifacts.go b/src/internal/airgap/artifacts.go new file mode 100644 index 0000000..7010f62 --- /dev/null +++ b/src/internal/airgap/artifacts.go @@ -0,0 +1,180 @@ +package airgap + +import ( + "fmt" + "os" + "path/filepath" + "strings" + "text/template" +) + +// WriteMirrorCommands writes mirror-commands.sh. +func WriteMirrorCommands(opts *Options, outputDir string) error { + const script = `#!/usr/bin/env bash +# Generated by patternizer airgap init +# Requires: oc-mirror plugin v2, pull credentials for source registries +set -euo pipefail + +REGISTRY="{{.Registry}}" +HERE="$(cd "$(dirname "$0")" && pwd)" +CONFIG="${HERE}/imagesetconfig.yaml" +REGISTRIES_D="${HERE}/registries.d" +CACHE="${OC_MIRROR_CACHE:-${HOME}/.cache/oc-mirror}" +WORKSPACE="${CACHE}/workspace" + +# Disable sigstore attachments for registry.connect.redhat.com (Vault / partner images). +mkdir -p "${REGISTRIES_D}" +cat > "${REGISTRIES_D}/registry.connect.redhat.com.yaml" <<'EOF' +docker: + registry.connect.redhat.com: + use-sigstore-attachments: false +EOF + +mkdir -p "${CACHE}" "${WORKSPACE}" + +echo "==> Mirror to registry (partially disconnected / bastion with registry access)" +echo oc mirror --config="${CONFIG}" --workspace "file://${WORKSPACE}" "docker://${REGISTRY}" --v2 --registries.d="${REGISTRIES_D}" +echo +echo "==> Fully disconnected: mirror to disk, then from disk to registry" +echo "# On connected host:" +echo oc mirror --config="${CONFIG}" "file://${CACHE}/mirror" --v2 --registries.d="${REGISTRIES_D}" +echo "# Transfer ${CACHE}/mirror to the disconnected host, then:" +echo oc mirror --from "file://${CACHE}/mirror" "docker://${REGISTRY}" --v2 +echo +echo "==> After mirroring, apply generated cluster resources (paths may vary by oc-mirror version):" +echo "cd ${WORKSPACE}/working-dir/cluster-resources" +echo "oc apply -f idms-oc-mirror.yaml -f itms-oc-mirror.yaml" +echo "oc apply -f cs-redhat-operator-index-v{{.DashVersion}}.yaml" +echo "oc apply -f cs-community-operator-index-v{{.DashVersion}}.yaml" +echo "# Also apply marketplace CatalogSource YAML if present" +echo +echo "==> Deploy the pattern from a host that can reach the disconnected cluster and Git:" +echo "export PATTERN_DISCONNECTED_HOME=${REGISTRY}/validatedpatterns" +echo "./pattern.sh make install" +` + + tmpl, err := template.New("mirror").Parse(script) + if err != nil { + return err + } + path := filepath.Join(outputDir, "mirror-commands.sh") + f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o755) + if err != nil { + return err + } + defer f.Close() + + registry := strings.TrimSpace(opts.Registry) + if registry == "" { + registry = "${REGISTRY}" + } + return tmpl.Execute(f, map[string]string{ + "Registry": registry, + "DashVersion": dashVersion(catalogOCPVersion(opts)), + }) +} + +// WriteChecklist writes CHECKLIST.md. +func WriteChecklist(inv *PatternInventory, opts *Options, outputDir string) error { + var b strings.Builder + b.WriteString("# Airgap init checklist\n\n") + fmt.Fprintf(&b, "Pattern: **%s** \n", inv.PatternName) + if ver := strings.TrimSpace(opts.OCPVersion); ver != "" { + fmt.Fprintf(&b, "OCP platform channel: **stable-%s** \n", ver) + } else { + b.WriteString("OCP platform channel: _(not requested; pass `--ocp-version` to include)_ \n") + } + if reg := strings.TrimSpace(opts.Registry); reg != "" { + fmt.Fprintf(&b, "Mirror registry: **%s** \n\n", reg) + } else { + b.WriteString("Mirror registry: _(not set; pass `--registry` or set `REGISTRY` when running mirror-commands.sh)_ \n\n") + } + + catVer := catalogOCPVersion(opts) + + b.WriteString("## 1. Mirror-host pull secrets (connected bastion)\n\n") + b.WriteString("Ensure `~/.docker/config.json` / `auth.json` can pull from every source registry referenced in `imagesetconfig.yaml`:\n\n") + b.WriteString("- [ ] `registry.redhat.io`\n") + b.WriteString("- [ ] `registry.connect.redhat.com` (Vault and partner images)\n") + b.WriteString("- [ ] `quay.io` (Validated Patterns charts and containers)\n") + b.WriteString("- [ ] `ghcr.io` (external-secrets images, if used)\n") + b.WriteString("- [ ] `docker.io` / `registry.access.redhat.com` as needed\n") + b.WriteString("- [ ] Review oc-mirror logs for skipped `oci://` related images and add them to `additionalImages`\n\n") + + b.WriteString("## 2. Run mirroring\n\n") + b.WriteString("- [ ] Run commands in `mirror-commands.sh`\n") + b.WriteString("- [ ] Apply IDMS/ITMS and CatalogSource manifests from the oc-mirror workspace\n") + b.WriteString("- [ ] Do **not** rename generated CatalogSource names\n\n") + + b.WriteString("## 3. Cluster registry trust\n\n") + b.WriteString("- [ ] Cluster nodes trust the mirror registry CA\n") + b.WriteString("- [ ] Global pull secret can authenticate to the internal registry if required\n\n") + + b.WriteString("## 4. Pattern values for disconnected use\n\n") + b.WriteString("- [ ] Merge `values-disconnected.yaml` into the pattern (or commit overlays)\n") + if reg := strings.TrimSpace(opts.Registry); reg != "" { + fmt.Fprintf(&b, "- [ ] `main.multiSourceConfig.helmRepoUrl` -> `%s/validatedpatterns`\n", reg) + fmt.Fprintf(&b, "- [ ] `export PATTERN_DISCONNECTED_HOME=%s/validatedpatterns` before `./pattern.sh make install`\n", reg) + } else { + b.WriteString("- [ ] Set `main.multiSourceConfig.helmRepoUrl` to `/validatedpatterns`\n") + b.WriteString("- [ ] `export PATTERN_DISCONNECTED_HOME=/validatedpatterns` before `./pattern.sh make install`\n") + } + fmt.Fprintf(&b, "- [ ] `main.patternsOperator.source` -> `%s`\n", CatalogSourceName(CatalogCommunity, catVer)) + fmt.Fprintf(&b, "- [ ] `main.gitops.operatorSource` -> `%s`\n", CatalogSourceName(CatalogRedHat, catVer)) + b.WriteString("- [ ] Per-subscription `source:` fields point at mirrored catalogs\n\n") + + b.WriteString("## 5. Git reachability\n\n") + b.WriteString("- [ ] Pattern Git repository is hosted inside the disconnected network\n") + b.WriteString("- [ ] `git remote -v` on the install host points at the disconnected Git server\n") + b.WriteString("- [ ] (oc-mirror does **not** mirror Git content)\n\n") + + b.WriteString("## 6. Secrets (Vault + External Secrets Operator)\n\n") + b.WriteString("Secrets are **not** mirrored. Keep the existing Validated Patterns secrets workflow:\n\n") + b.WriteString("- [ ] Mirror Vault / ESO operators and operand images (included when those charts/subscriptions are present)\n") + b.WriteString("- [ ] On a trusted host **inside** the airgap, create `values-secret.yaml` from `values-secret.yaml.template`\n") + b.WriteString("- [ ] Prefer `onMissingValue: generate` so fewer secrets must cross the air gap\n") + b.WriteString("- [ ] Run `./pattern.sh make load-secrets` after install (uses `rhvp.cluster_utils.load_secrets`)\n") + b.WriteString("- [ ] Never commit real `values-secret.yaml` to Git\n") + b.WriteString("- [ ] External backends (AWS SM, Azure KV, SaaS Vault) will **not** work without egress; use in-cluster Vault\n\n") + + if len(inv.ManagedClusterGroups) > 0 { + b.WriteString("## 7. Managed / spoke clusters\n\n") + b.WriteString("This pattern declares managedClusterGroups: " + strings.Join(inv.ManagedClusterGroups, ", ") + "\n\n") + b.WriteString("- [ ] Include spoke values files in init (`--cluster-groups`)\n") + b.WriteString("- [ ] Ensure spoke operators (e.g. GitOps via ACM policies) are present in the ImageSetConfig\n") + b.WriteString("- [ ] Spokes must also trust the mirror registry and use IDMS/ITMS\n\n") + } + + b.WriteString("## Discovery warnings\n\n") + if len(inv.Warnings) == 0 { + b.WriteString("_No warnings emitted during discovery._\n") + } else { + for _, w := range inv.Warnings { + fmt.Fprintf(&b, "- %s\n", w) + } + } + + b.WriteString("\n## Generated operators\n\n") + for _, op := range inv.Operators { + ch := op.Channel + if ch == "" { + ch = "(default channel)" + } + fmt.Fprintf(&b, "- `%s` channel `%s` [%s]\n", op.Name, ch, op.Catalog) + } + + b.WriteString("\n## Generated remote charts (OCI)\n\n") + for _, rc := range inv.RemoteCharts { + fmt.Fprintf(&b, "- `%s` (%s -> %s) -> `%s`\n", rc.Name, rc.VersionConstraint, rc.ResolvedVersion, rc.PullSpec) + for _, img := range rc.Images { + fmt.Fprintf(&b, " - image: `%s`\n", img) + } + } + + b.WriteString("\n## Generated local charts\n\n") + for _, lc := range inv.LocalCharts { + fmt.Fprintf(&b, "- `%s` path `%s` (%d images)\n", lc.Name, lc.Path, len(lc.Images)) + } + + return os.WriteFile(filepath.Join(outputDir, "CHECKLIST.md"), []byte(b.String()), 0o644) +} diff --git a/src/internal/airgap/bootstrap.go b/src/internal/airgap/bootstrap.go new file mode 100644 index 0000000..ee06547 --- /dev/null +++ b/src/internal/airgap/bootstrap.go @@ -0,0 +1,568 @@ +package airgap + +import ( + "fmt" + "os" + "path/filepath" + "sort" + "strings" + "text/template" +) + +// GenerateBootstrap writes a single airgap apply bundle from values-secret airgap: config. +func GenerateBootstrap(opts *BootstrapOptions) (*BootstrapResult, error) { + if opts == nil { + return nil, fmt.Errorf("options are required") + } + + patternDir := strings.TrimSpace(opts.PatternDir) + if patternDir != "" { + abs, err := filepath.Abs(patternDir) + if err != nil { + return nil, err + } + patternDir = abs + } + + patternName := strings.TrimSpace(opts.PatternName) + if patternName == "" && patternDir != "" { + patternName = PatternNameFromDir(patternDir) + } + + secretsPath, err := ResolveValuesSecretPath(patternDir, patternName, opts.SecretsFile) + if err != nil { + return nil, err + } + + spec, err := LoadAirgapSpec(secretsPath) + if err != nil { + return nil, err + } + + certs, err := ResolveCertificates(spec) + if err != nil { + return nil, err + } + regs, err := ResolveRegistries(spec) + if err != nil { + return nil, err + } + + wantRender := !opts.SkipRenderValues && (spec.WantsRenderValues() || opts.DiscoverCatalogs || opts.Registry != "" || opts.OCPVersion != "") + if len(certs) == 0 && len(regs) == 0 && !wantRender { + return nil, fmt.Errorf("%s airgap: section has no certificates, registries, or disconnected values settings", secretsPath) + } + + ns := strings.TrimSpace(opts.Namespace) + if ns == "" { + ns = strings.TrimSpace(spec.ArgocdNamespace) + } + if ns == "" { + ns = "vp-gitops" + } + poNS := strings.TrimSpace(spec.PatternsOperatorNamespace) + if poNS == "" { + poNS = "patterns-operator" + } + repoServer := strings.TrimSpace(opts.RepoServer) + if repoServer == "" { + repoServer = ns + "-repo-server" + } + + outDir := strings.TrimSpace(opts.OutputDir) + if outDir == "" { + outDir = "./airgap-bootstrap" + } + absOut, err := filepath.Abs(outDir) + if err != nil { + return nil, err + } + outDir = absOut + + if err := os.MkdirAll(filepath.Join(outDir, "backup"), 0o755); err != nil { + return nil, err + } + if err := os.MkdirAll(filepath.Join(outDir, "certs"), 0o755); err != nil { + return nil, err + } + if err := os.MkdirAll(filepath.Join(outDir, "secrets"), 0o700); err != nil { + return nil, err + } + + var proxyCAFiles []string + var tlsHosts []string + hostCA := map[string]string{} // host -> pem content for argocd + + for _, c := range certs { + pem, err := LoadCAPEM(c.CAPath) + if err != nil { + return nil, fmt.Errorf("certificate %q: %w", c.Name, err) + } + dst := filepath.Join(outDir, "certs", sanitizeFileName(c.Name)+".pem") + if err := os.WriteFile(dst, []byte(pem), 0o644); err != nil { + return nil, err + } + if c.ClusterProxy { + proxyCAFiles = append(proxyCAFiles, filepath.Base(dst)) + } + if c.ArgocdTLS { + host, err := NormalizeHost(c.Host) + if err != nil { + return nil, fmt.Errorf("certificate %q host: %w", c.Name, err) + } + tlsHosts = append(tlsHosts, host) + hostCA[host] = pem + } + } + + if len(hostCA) > 0 { + if err := writeMergedTLSCertsPatch(outDir, ns, hostCA); err != nil { + return nil, err + } + var combined strings.Builder + seen := map[string]struct{}{} + for _, host := range tlsHosts { + pem := hostCA[host] + if _, ok := seen[pem]; ok { + continue + } + seen[pem] = struct{}{} + combined.WriteString(pem) + if !strings.HasSuffix(pem, "\n") { + combined.WriteByte('\n') + } + } + if err := os.WriteFile(filepath.Join(outDir, "trusted-ca-bundle-append.pem"), []byte(combined.String()), 0o644); err != nil { + return nil, err + } + } + + var secretNames []string + for _, r := range regs { + cred, err := resolveRegistryFromResolved(&r) + if err != nil { + return nil, fmt.Errorf("registry %q: %w", r.Name, err) + } + secretName := defaultRepoCredsSecretName(cred.URL) + if r.Name != "" { + secretName = sanitizeFileName(r.Name) + "-repo-creds" + } + path := filepath.Join(outDir, "secrets", secretName+".yaml") + if err := writeRepoCredsSecret(path, ns, secretName, cred); err != nil { + return nil, err + } + secretNames = append(secretNames, secretName) + } + + result := &BootstrapResult{ + OutputDir: outDir, + SecretsFile: secretsPath, + ArgocdNamespace: ns, + ClusterProxyCAs: len(proxyCAFiles), + ArgocdTLSHosts: len(hostCA), + Registries: len(secretNames), + } + + if wantRender { + if patternDir == "" { + return nil, fmt.Errorf("pattern dir is required to render values (pass the pattern path to airgap bootstrap)") + } + dcfg, err := ResolveDisconnectedValues(spec, opts) + if err != nil { + return nil, err + } + rendered, err := RenderPatternValues(patternDir, outDir, dcfg) + if err != nil { + return nil, fmt.Errorf("render values: %w", err) + } + result.RenderedFiles = rendered + result.HelmRepoURL = dcfg.HelmRepoURL + result.RedHatCS = dcfg.RedHatCS + result.CommunityCS = dcfg.CommunityCS + } + + if err := writeApplyAirgapScript(outDir, &applyAirgapData{ + ArgocdNamespace: ns, + PatternsOperatorNamespace: poNS, + RepoServer: repoServer, + ProxyCAFiles: proxyCAFiles, + HasArgocdTLS: len(hostCA) > 0, + SecretNames: secretNames, + }); err != nil { + return nil, err + } + if err := writeRestoreProxyScript(outDir); err != nil { + return nil, err + } + if err := writeBootstrapReadme(outDir, secretsPath, ns, poNS, certs, regs, result); err != nil { + return nil, err + } + + return result, nil +} + +func resolveRegistryFromResolved(r *ResolvedRegistry) (*RegistryCredential, error) { + opts := &RepoCredsOptions{ + URL: r.URL, + DockerConfig: r.DockerConfigPath, + Username: r.Username, + Password: r.Password, + PasswordFile: r.PasswordPath, + } + return ResolveRegistryCredential(opts) +} + +func writeMergedTLSCertsPatch(outDir, namespace string, hostCA map[string]string) error { + hosts := make([]string, 0, len(hostCA)) + for h := range hostCA { + hosts = append(hosts, h) + } + sort.Strings(hosts) + var b strings.Builder + b.WriteString("# Generated by patternizer airgap bootstrap\n") + b.WriteString("apiVersion: v1\n") + b.WriteString("kind: ConfigMap\n") + b.WriteString("metadata:\n") + b.WriteString(" name: argocd-tls-certs-cm\n") + fmt.Fprintf(&b, " namespace: %s\n", namespace) + b.WriteString("data:\n") + for _, host := range hosts { + pem := hostCA[host] + fmt.Fprintf(&b, " %s: |\n", host) + for _, line := range strings.Split(strings.TrimSuffix(pem, "\n"), "\n") { + fmt.Fprintf(&b, " %s\n", line) + } + } + return os.WriteFile(filepath.Join(outDir, "argocd-tls-certs-patch.yaml"), []byte(b.String()), 0o644) +} + +type applyAirgapData struct { + ArgocdNamespace string + PatternsOperatorNamespace string + RepoServer string + ProxyCAFiles []string + HasArgocdTLS bool + SecretNames []string +} + +func writeApplyAirgapScript(outDir string, d *applyAirgapData) error { + const script = `#!/usr/bin/env bash +# Generated by patternizer airgap bootstrap +# Usage: ./apply-airgap.sh [pre|post|all] +# pre — cluster Proxy / user-ca-bundle (before make install) +# post — Argo CD TLS + repo-creds (after vp-gitops exists) +# all — both (default) +# Requires: oc on PATH and a working kubeconfig (KUBECONFIG or ~/.kube/config) +set -euo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" +BACKUP_DIR="${HERE}/backup" +ARGOCD_NS="{{.ArgocdNamespace}}" +PATTERNS_NS="{{.PatternsOperatorNamespace}}" +REPO_SERVER="{{.RepoServer}}" +PROXY_BACKUP="${BACKUP_DIR}/proxy-cluster.yaml" +USER_CA_BACKUP="${BACKUP_DIR}/user-ca-bundle.yaml" +PHASE="${1:-all}" +WAIT_SECONDS="${AIRGAP_WAIT_SECONDS:-600}" +WAIT_INTERVAL="${AIRGAP_WAIT_INTERVAL:-10}" + +PROXY_CA_FILES=( +{{range .ProxyCAFiles}} "{{.}}" +{{end}}) +HAS_ARGOCD_TLS={{if .HasArgocdTLS}}1{{else}}0{{end}} +SECRET_FILES=( +{{range .SecretNames}} "{{.}}.yaml" +{{end}}) + +RUN_PRE=0 +RUN_POST=0 +case "${PHASE}" in + pre) RUN_PRE=1 ;; + post) RUN_POST=1 ;; + all) RUN_PRE=1; RUN_POST=1 ;; + *) + echo "usage: $0 [pre|post|all]" >&2 + exit 1 + ;; +esac + +if ! command -v oc >/dev/null 2>&1; then + echo "oc is required but not found on PATH" >&2 + exit 1 +fi + +mkdir -p "${BACKUP_DIR}" + +wait_for_ns() { + local ns="$1" + local elapsed=0 + echo "==> Waiting up to ${WAIT_SECONDS}s for namespace/${ns}" + while (( elapsed < WAIT_SECONDS )); do + if oc get namespace "${ns}" >/dev/null 2>&1; then + echo " namespace/${ns} is present" + return 0 + fi + sleep "${WAIT_INTERVAL}" + elapsed=$((elapsed + WAIT_INTERVAL)) + done + echo "timed out waiting for namespace/${ns}" >&2 + return 1 +} + +wait_for_cm() { + local ns="$1" + local name="$2" + local elapsed=0 + echo "==> Waiting up to ${WAIT_SECONDS}s for configmap/${name} in ${ns}" + while (( elapsed < WAIT_SECONDS )); do + if oc get configmap "${name}" -n "${ns}" >/dev/null 2>&1; then + echo " configmap/${name} is present" + return 0 + fi + sleep "${WAIT_INTERVAL}" + elapsed=$((elapsed + WAIT_INTERVAL)) + done + echo "timed out waiting for configmap/${name} in ${ns}" >&2 + return 1 +} + +wait_for_deploy() { + local ns="$1" + local name="$2" + local elapsed=0 + echo "==> Waiting up to ${WAIT_SECONDS}s for deployment/${name} in ${ns}" + while (( elapsed < WAIT_SECONDS )); do + if oc get deployment "${name}" -n "${ns}" >/dev/null 2>&1; then + echo " deployment/${name} is present" + return 0 + fi + sleep "${WAIT_INTERVAL}" + elapsed=$((elapsed + WAIT_INTERVAL)) + done + echo " warning: deployment/${name} not found in ${ns} within ${WAIT_SECONDS}s" >&2 + return 1 +} + +if (( RUN_PRE )); then + if ((${#PROXY_CA_FILES[@]} > 0)); then + echo "==> [pre] Backing up proxy/cluster (original preserved at ${PROXY_BACKUP})" + if [[ ! -f "${PROXY_BACKUP}" ]]; then + oc get proxy cluster -o yaml > "${PROXY_BACKUP}" + echo " wrote first-time backup ${PROXY_BACKUP}" + else + echo " keeping existing first-time backup ${PROXY_BACKUP}" + fi + oc get proxy cluster -o yaml > "${BACKUP_DIR}/proxy-cluster-before-apply.yaml" + + echo "==> [pre] Backing up ConfigMap user-ca-bundle (if present)" + if oc get configmap user-ca-bundle -n openshift-config >/dev/null 2>&1; then + if [[ ! -f "${USER_CA_BACKUP}" ]]; then + oc get configmap user-ca-bundle -n openshift-config -o yaml > "${USER_CA_BACKUP}" + fi + oc get configmap user-ca-bundle -n openshift-config -o yaml > "${BACKUP_DIR}/user-ca-bundle-before-apply.yaml" + fi + + echo "==> [pre] Merging CAs into openshift-config/user-ca-bundle and enabling Proxy trustedCA" + TMP_CA="$(mktemp)" + trap 'rm -f "${TMP_CA}"' EXIT + { + oc get configmap user-ca-bundle -n openshift-config -o jsonpath='{.data.ca-bundle\.crt}' 2>/dev/null || true + echo + for f in "${PROXY_CA_FILES[@]}"; do + cat "${HERE}/certs/${f}" + echo + done + } > "${TMP_CA}" + + oc create configmap user-ca-bundle -n openshift-config \ + --from-file=ca-bundle.crt="${TMP_CA}" \ + --dry-run=client -o yaml | oc apply -f - + + oc patch proxy/cluster --type=merge \ + -p '{"spec":{"trustedCA":{"name":"user-ca-bundle"}}}' + + echo "==> [pre] Restarting patterns-operator (picks up cluster CA trust for git clone)" + if oc get deployment patterns-operator-controller-manager -n "${PATTERNS_NS}" >/dev/null 2>&1; then + oc rollout restart deployment/patterns-operator-controller-manager -n "${PATTERNS_NS}" + oc rollout status deployment/patterns-operator-controller-manager -n "${PATTERNS_NS}" + else + echo " warning: deployment/patterns-operator-controller-manager not found in ${PATTERNS_NS}" >&2 + fi + else + echo "==> [pre] No clusterProxy CAs configured; skipping Proxy trust" + fi +fi + +if (( RUN_POST )); then + if (( HAS_ARGOCD_TLS == 0 )) && ((${#SECRET_FILES[@]} == 0)); then + echo "==> [post] No Argo CD TLS or repo-creds configured; skipping" + else + # Wait for GitOps namespace (created during pattern install). + wait_for_ns "${ARGOCD_NS}" + + # Apply repo-creds as soon as the namespace exists so OCI pulls can auth + # even while waiting for Argo CD TLS ConfigMaps. + if ((${#SECRET_FILES[@]} > 0)); then + echo "==> [post] Applying Argo CD registry repo-creds Secrets" + for sf in "${SECRET_FILES[@]}"; do + oc apply -f "${HERE}/secrets/${sf}" + done + fi + + if (( HAS_ARGOCD_TLS == 1 )); then + wait_for_cm "${ARGOCD_NS}" "argocd-tls-certs-cm" + echo "==> [post] Patching Argo CD TLS trust in ${ARGOCD_NS}" + oc patch configmap argocd-tls-certs-cm -n "${ARGOCD_NS}" --type=merge --patch-file="${HERE}/argocd-tls-certs-patch.yaml" + + TMP_BUNDLE="$(mktemp)" + { + oc get configmap trusted-ca-bundle -n "${ARGOCD_NS}" -o jsonpath='{.data.ca-bundle\.crt}' 2>/dev/null || true + echo + cat "${HERE}/trusted-ca-bundle-append.pem" + } > "${TMP_BUNDLE}" + if oc get configmap trusted-ca-bundle -n "${ARGOCD_NS}" >/dev/null 2>&1; then + { + echo "data:" + echo " ca-bundle.crt: |" + sed 's/^/ /' "${TMP_BUNDLE}" + } > "${TMP_BUNDLE}.patch" + oc patch configmap trusted-ca-bundle -n "${ARGOCD_NS}" --type=merge --patch-file="${TMP_BUNDLE}.patch" + rm -f "${TMP_BUNDLE}.patch" + else + oc create configmap trusted-ca-bundle -n "${ARGOCD_NS}" --from-file=ca-bundle.crt="${TMP_BUNDLE}" + fi + rm -f "${TMP_BUNDLE}" + fi + + if wait_for_deploy "${ARGOCD_NS}" "${REPO_SERVER}"; then + echo "==> [post] Restarting deployment/${REPO_SERVER}" + oc rollout restart "deployment/${REPO_SERVER}" -n "${ARGOCD_NS}" + oc rollout status "deployment/${REPO_SERVER}" -n "${ARGOCD_NS}" --timeout="${WAIT_SECONDS}s" || \ + oc rollout status "deployment/${REPO_SERVER}" -n "${ARGOCD_NS}" + else + echo " warning: skipping repo-server restart; apply secrets/TLS anyway (restart later if needed)" >&2 + fi + fi +fi + +echo "Done (phase=${PHASE})." +echo " Restore original Proxy with: ${HERE}/restore-proxy-cluster.sh" +` + + tmpl, err := template.New("apply").Parse(script) + if err != nil { + return err + } + path := filepath.Join(outDir, "apply-airgap.sh") + f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o755) + if err != nil { + return err + } + defer f.Close() + return tmpl.Execute(f, d) +} + +func writeRestoreProxyScript(outDir string) error { + const script = `#!/usr/bin/env bash +# Generated by patternizer airgap bootstrap +# Restores proxy/cluster (and user-ca-bundle when backed up) from first-time backups. +set -euo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" +BACKUP_DIR="${HERE}/backup" +PROXY_BACKUP="${BACKUP_DIR}/proxy-cluster.yaml" +USER_CA_BACKUP="${BACKUP_DIR}/user-ca-bundle.yaml" + +if ! command -v oc >/dev/null 2>&1; then + echo "oc is required but not found on PATH" >&2 + exit 1 +fi +if [[ ! -f "${PROXY_BACKUP}" ]]; then + echo "missing ${PROXY_BACKUP}; run apply-airgap.sh first" >&2 + exit 1 +fi + +echo "==> Restoring proxy/cluster from ${PROXY_BACKUP}" +# Drop cluster-scoped metadata that blocks apply, keep spec. +oc apply -f "${PROXY_BACKUP}" + +if [[ -f "${USER_CA_BACKUP}" ]]; then + echo "==> Restoring ConfigMap user-ca-bundle from ${USER_CA_BACKUP}" + oc apply -f "${USER_CA_BACKUP}" +else + echo "==> No user-ca-bundle backup; clearing Proxy trustedCA name only if desired manually" +fi + +echo "Done. You may need to restart patterns-operator-controller-manager after restore." +` + path := filepath.Join(outDir, "restore-proxy-cluster.sh") + return os.WriteFile(path, []byte(script), 0o755) +} + +func writeBootstrapReadme(outDir, secretsPath, ns, poNS string, certs []ResolvedCertificate, regs []ResolvedRegistry, result *BootstrapResult) error { + var b strings.Builder + b.WriteString("# Airgap bootstrap (generated by patternizer airgap bootstrap)\n\n") + b.WriteString("Single apply for cluster Proxy CA trust (patterns-operator git clone),\n") + b.WriteString("Argo CD TLS trust, and registry repo-creds.\n\n") + fmt.Fprintf(&b, "Source secrets file: `%s`\n\n", secretsPath) + if result != nil && len(result.RenderedFiles) > 0 { + b.WriteString("## Rendered pattern values\n\n") + fmt.Fprintf(&b, "- helmRepoUrl: `%s`\n", result.HelmRepoURL) + fmt.Fprintf(&b, "- patternsOperator.source: `%s`\n", result.CommunityCS) + fmt.Fprintf(&b, "- gitops.operatorSource: `%s`\n", result.RedHatCS) + b.WriteString("- Updated files (backups under `backup/values/`):\n") + for _, f := range result.RenderedFiles { + fmt.Fprintf(&b, " - `%s`\n", f) + } + b.WriteString("\n") + } + b.WriteString("## Apply once\n\n") + b.WriteString("```bash\n./apply-airgap.sh # all (default)\n./apply-airgap.sh pre # Proxy CA only (before make install)\n./apply-airgap.sh post # Argo CD TLS + repo-creds (after vp-gitops)\n```\n\n") + b.WriteString("Or from the pattern repo:\n\n") + b.WriteString("```bash\n./pattern.sh make install-airgap\n```\n\n") + b.WriteString("This script:\n\n") + b.WriteString("1. Backs up `proxy/cluster` to `backup/proxy-cluster.yaml` (first run only, pre phase)\n") + b.WriteString("2. Merges CAs into `openshift-config/user-ca-bundle` and sets `Proxy.spec.trustedCA`\n") + b.WriteString("3. Restarts patterns-operator\n") + b.WriteString("4. Patches Argo CD TLS ConfigMaps (when configured, post phase)\n") + b.WriteString("5. Applies registry repo-creds Secrets (when configured, post phase)\n") + b.WriteString("6. Restarts the Argo CD repo-server\n\n") + b.WriteString("## Restore Proxy\n\n") + b.WriteString("```bash\n./restore-proxy-cluster.sh\n```\n\n") + fmt.Fprintf(&b, "Namespaces: Argo CD `%s`, patterns-operator `%s`\n\n", ns, poNS) + if len(certs) > 0 { + b.WriteString("## Certificates\n\n") + for _, c := range certs { + fmt.Fprintf(&b, "- `%s` host `%s` clusterProxy=%v argocdTLS=%v\n", c.Name, c.Host, c.ClusterProxy, c.ArgocdTLS) + } + b.WriteString("\n") + } + if len(regs) > 0 { + b.WriteString("## Registries\n\n") + for _, r := range regs { + fmt.Fprintf(&b, "- `%s` url `%s`\n", r.Name, r.URL) + } + b.WriteString("\nDo not commit `secrets/*.yaml` — they contain passwords.\n") + } + return os.WriteFile(filepath.Join(outDir, "README.md"), []byte(b.String()), 0o644) +} + +func sanitizeFileName(name string) string { + name = strings.ToLower(strings.TrimSpace(name)) + var b strings.Builder + for _, r := range name { + switch { + case r >= 'a' && r <= 'z', r >= '0' && r <= '9': + b.WriteRune(r) + case r == '-' || r == '_' || r == '.': + b.WriteRune(r) + default: + b.WriteByte('-') + } + } + out := strings.Trim(b.String(), "-.") + if out == "" { + return "item" + } + return out +} diff --git a/src/internal/airgap/bootstrap_test.go b/src/internal/airgap/bootstrap_test.go new file mode 100644 index 0000000..709b12a --- /dev/null +++ b/src/internal/airgap/bootstrap_test.go @@ -0,0 +1,148 @@ +package airgap_test + +import ( + "encoding/base64" + "os" + "path/filepath" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/validatedpatterns/patternizer/internal/airgap" +) + +const bootstrapCA = `-----BEGIN CERTIFICATE----- +MIIBkTCB+wIJAKHBfJ7eBOOTSTRAP +-----END CERTIFICATE----- +` + +var _ = Describe("Airgap values-secret parsing", func() { + It("resolves certificates and registries from airgap: section", func() { + tmp := GinkgoT().TempDir() + caPath := filepath.Join(tmp, "ca.pem") + Expect(os.WriteFile(caPath, []byte(bootstrapCA), 0o644)).To(Succeed()) + auth := base64.StdEncoding.EncodeToString([]byte("init:secret")) + dockerCfg := filepath.Join(tmp, "config.json") + Expect(os.WriteFile(dockerCfg, []byte(`{"auths":{"mirror.example.com:8443":{"auth":"`+auth+`"}}}`), 0o600)).To(Succeed()) + + secrets := filepath.Join(tmp, "values-secret.yaml") + Expect(os.WriteFile(secrets, []byte(` +version: "2.0" +secrets: [] +airgap: + argocdNamespace: vp-gitops + certificates: + - name: gitea + fields: + - name: host + value: gitea.example.com + - name: ca + path: `+caPath+` + - name: clusterProxy + value: "true" + - name: argocdTLS + value: "true" + registries: + - name: mirror + fields: + - name: url + value: mirror.example.com:8443 + - name: dockerconfig + path: `+dockerCfg+` +`), 0o600)).To(Succeed()) + + spec, err := airgap.LoadAirgapSpec(secrets) + Expect(err).NotTo(HaveOccurred()) + certs, err := airgap.ResolveCertificates(spec) + Expect(err).NotTo(HaveOccurred()) + Expect(certs).To(HaveLen(1)) + Expect(certs[0].Host).To(Equal("gitea.example.com")) + Expect(certs[0].ClusterProxy).To(BeTrue()) + Expect(certs[0].ArgocdTLS).To(BeTrue()) + + regs, err := airgap.ResolveRegistries(spec) + Expect(err).NotTo(HaveOccurred()) + Expect(regs).To(HaveLen(1)) + Expect(regs[0].URL).To(Equal("mirror.example.com:8443")) + Expect(regs[0].DockerConfigPath).To(Equal(dockerCfg)) + }) +}) + +var _ = Describe("GenerateBootstrap", func() { + It("writes apply-airgap.sh with proxy backup and combined artifacts", func() { + tmp := GinkgoT().TempDir() + caPath := filepath.Join(tmp, "gitea.pem") + Expect(os.WriteFile(caPath, []byte(bootstrapCA), 0o644)).To(Succeed()) + pwFile := filepath.Join(tmp, "pw") + Expect(os.WriteFile(pwFile, []byte("regpass"), 0o600)).To(Succeed()) + + patternDir := filepath.Join(tmp, "pattern") + Expect(os.MkdirAll(patternDir, 0o755)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(patternDir, "values-global.yaml"), []byte(` +global: + pattern: test-pattern +`), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(patternDir, "values-secret.yaml"), []byte(` +version: "2.0" +airgap: + certificates: + - name: gitea + fields: + - name: host + value: gitea.example.com + - name: ca + path: `+caPath+` + - name: clusterProxy + value: "true" + - name: argocdTLS + value: "true" + registries: + - name: mirror + fields: + - name: url + value: mirror.example.com:8443 + - name: username + value: init + - name: password + path: `+pwFile+` +`), 0o600)).To(Succeed()) + + outDir := filepath.Join(tmp, "out") + res, err := airgap.GenerateBootstrap(&airgap.BootstrapOptions{ + PatternDir: patternDir, + OutputDir: outDir, + }) + Expect(err).NotTo(HaveOccurred()) + Expect(res.ClusterProxyCAs).To(Equal(1)) + Expect(res.ArgocdTLSHosts).To(Equal(1)) + Expect(res.Registries).To(Equal(1)) + + Expect(filepath.Join(outDir, "apply-airgap.sh")).To(BeAnExistingFile()) + Expect(filepath.Join(outDir, "restore-proxy-cluster.sh")).To(BeAnExistingFile()) + Expect(filepath.Join(outDir, "argocd-tls-certs-patch.yaml")).To(BeAnExistingFile()) + Expect(filepath.Join(outDir, "certs", "gitea.pem")).To(BeAnExistingFile()) + Expect(filepath.Join(outDir, "secrets", "mirror-repo-creds.yaml")).To(BeAnExistingFile()) + Expect(filepath.Join(outDir, "backup")).To(BeADirectory()) + + script, err := os.ReadFile(filepath.Join(outDir, "apply-airgap.sh")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(script)).To(ContainSubstring("proxy-cluster.yaml")) + Expect(string(script)).To(ContainSubstring("user-ca-bundle")) + Expect(string(script)).To(ContainSubstring("patterns-operator-controller-manager")) + Expect(string(script)).To(ContainSubstring("argocd-tls-certs-cm")) + Expect(string(script)).To(ContainSubstring("mirror-repo-creds.yaml")) + Expect(string(script)).To(ContainSubstring("[pre|post|all]")) + Expect(string(script)).To(ContainSubstring("RUN_PRE")) + Expect(string(script)).To(ContainSubstring("RUN_POST")) + Expect(string(script)).To(ContainSubstring("wait_for_cm")) + Expect(string(script)).To(ContainSubstring("Applying Argo CD registry repo-creds")) + + restore, err := os.ReadFile(filepath.Join(outDir, "restore-proxy-cluster.sh")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(restore)).To(ContainSubstring("proxy-cluster.yaml")) + + info, err := os.Stat(filepath.Join(outDir, "apply-airgap.sh")) + Expect(err).NotTo(HaveOccurred()) + Expect(info.Mode().Perm() & 0o111).NotTo(BeZero()) + }) +}) diff --git a/src/internal/airgap/catalogs.go b/src/internal/airgap/catalogs.go new file mode 100644 index 0000000..fb6e9ba --- /dev/null +++ b/src/internal/airgap/catalogs.go @@ -0,0 +1,99 @@ +package airgap + +import ( + "fmt" + "strings" +) + +// catalogOCPVersion returns the OCP minor used for OLM catalog image tags and +// CatalogSource names in the values overlay. When --ocp-version is set it is +// used; otherwise defaultOLMCatalogVersion. +func catalogOCPVersion(opts *Options) string { + if opts != nil { + if ver := strings.TrimSpace(opts.OCPVersion); ver != "" { + return ver + } + } + return defaultOLMCatalogVersion +} + +// CatalogKind identifies an OLM catalog index. +type CatalogKind string + +const ( + CatalogRedHat CatalogKind = "redhat" + CatalogMarketplace CatalogKind = "marketplace" + CatalogCommunity CatalogKind = "community" +) + +// OperatorRef is a package to mirror from an OLM catalog. +type OperatorRef struct { + Name string + Channel string + Catalog CatalogKind + Source string +} + +var defaultPackageCatalog = map[string]CatalogKind{ + "patterns-operator": CatalogCommunity, + "advanced-cluster-management": CatalogRedHat, + "multicluster-engine": CatalogRedHat, + "openshift-gitops-operator": CatalogRedHat, + "openshift-external-secrets-operator": CatalogRedHat, + "lvms-operator": CatalogRedHat, + "rhacs-operator": CatalogRedHat, + "odf-operator": CatalogRedHat, + "quay-operator": CatalogRedHat, + "openshift-pipelines-operator-rh": CatalogRedHat, + "rhtas-operator": CatalogRedHat, + "rhtpa-operator": CatalogRedHat, + "openshift-cert-manager-operator": CatalogMarketplace, + "rhbk-operator": CatalogMarketplace, + "openshift-zero-trust-workload-identity-manager": CatalogMarketplace, + "compliance-operator": CatalogMarketplace, +} + +// CatalogImage returns the catalog index image for a kind and OCP version. +func CatalogImage(kind CatalogKind, ocpVersion string) string { + tag := "v" + ocpVersion + switch kind { + case CatalogMarketplace: + return fmt.Sprintf("registry.redhat.io/redhat/redhat-marketplace-index:%s", tag) + case CatalogCommunity: + return fmt.Sprintf("registry.redhat.io/redhat/community-operator-index:%s", tag) + default: + return fmt.Sprintf("registry.redhat.io/redhat/redhat-operator-index:%s", tag) + } +} + +// CatalogSourceName returns the oc-mirror v2 CatalogSource name for a catalog image tag. +func CatalogSourceName(kind CatalogKind, ocpVersion string) string { + switch kind { + case CatalogMarketplace: + return fmt.Sprintf("cs-redhat-marketplace-index-v%s", dashVersion(ocpVersion)) + case CatalogCommunity: + return fmt.Sprintf("cs-community-operator-index-v%s", dashVersion(ocpVersion)) + default: + return fmt.Sprintf("cs-redhat-operator-index-v%s", dashVersion(ocpVersion)) + } +} + +func dashVersion(ocpVersion string) string { + out := make([]rune, 0, len(ocpVersion)) + for _, r := range ocpVersion { + if r == '.' { + out = append(out, '-') + } else { + out = append(out, r) + } + } + return string(out) +} + +// LookupCatalog returns the catalog kind for a package name. +func LookupCatalog(packageName string) CatalogKind { + if kind, ok := defaultPackageCatalog[packageName]; ok { + return kind + } + return CatalogRedHat +} diff --git a/src/internal/airgap/download.go b/src/internal/airgap/download.go new file mode 100644 index 0000000..dff278c --- /dev/null +++ b/src/internal/airgap/download.go @@ -0,0 +1,76 @@ +package airgap + +import ( + "fmt" + "io" + "net/http" + "os" + "path/filepath" + "strings" +) + +const openshiftClientsBase = "https://mirror.openshift.com/pub/openshift-v4/clients/ocp" + +// DownloadOCMirrorTools downloads oc-mirror archives for the given OCP version. +func DownloadOCMirrorTools(ocpVersion, destDir string) error { + if err := os.MkdirAll(destDir, 0o755); err != nil { + return err + } + + baseURL, err := resolveClientsBaseURL(ocpVersion) + if err != nil { + return err + } + + for _, name := range []string{"oc-mirror.tar.gz", "oc-mirror.rhel9.tar.gz"} { + url := strings.TrimSuffix(baseURL, "/") + "/" + name + dest := filepath.Join(destDir, name) + if err := downloadFile(url, dest); err != nil { + return fmt.Errorf("download %s: %w", name, err) + } + } + return nil +} + +func resolveClientsBaseURL(ocpVersion string) (string, error) { + candidates := []string{} + if ocpVersion != "" { + candidates = append(candidates, openshiftClientsBase+"/stable-"+ocpVersion) + } + candidates = append(candidates, openshiftClientsBase+"/stable") + + for _, base := range candidates { + probe := strings.TrimSuffix(base, "/") + "/oc-mirror.tar.gz" + resp, err := http.Head(probe) + if err != nil { + continue + } + resp.Body.Close() + if resp.StatusCode == http.StatusOK { + return base, nil + } + } + return "", fmt.Errorf("could not find oc-mirror.tar.gz under stable-%s or stable", ocpVersion) +} + +func downloadFile(url, dest string) error { + resp, err := http.Get(url) + if err != nil { + return err + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return fmt.Errorf("%s: HTTP %s", url, resp.Status) + } + + f, err := os.Create(dest) + if err != nil { + return err + } + defer f.Close() + + if _, err := io.Copy(f, resp.Body); err != nil { + return err + } + return nil +} diff --git a/src/internal/airgap/extraconfig.go b/src/internal/airgap/extraconfig.go new file mode 100644 index 0000000..0b97ccd --- /dev/null +++ b/src/internal/airgap/extraconfig.go @@ -0,0 +1,297 @@ +package airgap + +import ( + "fmt" + "os" + "path/filepath" + "strings" + + "gopkg.in/yaml.v3" +) + +// ExtraConfig is an optional external file merged into the generated ImageSetConfiguration. +// Schema matches a subset of oc-mirror ImageSetConfiguration mirror fields: +// +// operators: +// - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.22 +// packages: +// - name: lvms-operator +// additionalImages: +// - name: registry.example.com/app:1.0 +// helm: +// local: +// - name: my-chart +// path: /abs/path/to/chart +type ExtraConfig struct { + Operators []OperatorCatalog `yaml:"operators,omitempty"` + AdditionalImages []NamedImage `yaml:"additionalImages,omitempty"` + Helm *HelmMirror `yaml:"helm,omitempty"` +} + +// LoadExtraConfig reads and validates an external extras YAML file. +// The file may use top-level operators/additionalImages/helm keys, or the same +// fields nested under mirror: (ImageSetConfiguration style). +func LoadExtraConfig(path string) (*ExtraConfig, error) { + if path == "" { + return nil, fmt.Errorf("extra config path is empty") + } + abs, err := filepath.Abs(path) + if err != nil { + return nil, fmt.Errorf("resolve extra config path: %w", err) + } + data, err := os.ReadFile(abs) + if err != nil { + return nil, fmt.Errorf("read extra config %s: %w", abs, err) + } + + cfg, err := parseExtraConfigYAML(data) + if err != nil { + return nil, fmt.Errorf("parse extra config %s: %w", abs, err) + } + if err := cfg.validate(); err != nil { + return nil, fmt.Errorf("extra config %s: %w", abs, err) + } + return cfg, nil +} + +func parseExtraConfigYAML(data []byte) (*ExtraConfig, error) { + var wrapped struct { + Mirror *ExtraConfig `yaml:"mirror"` + ExtraConfig `yaml:",inline"` + } + if err := yaml.Unmarshal(data, &wrapped); err != nil { + return nil, err + } + if wrapped.Mirror != nil { + return wrapped.Mirror, nil + } + cfg := wrapped.ExtraConfig + return &cfg, nil +} + +func (c *ExtraConfig) validate() error { + if c == nil { + return fmt.Errorf("config is nil") + } + for i, op := range c.Operators { + if strings.TrimSpace(op.Catalog) == "" { + return fmt.Errorf("operators[%d]: catalog is required", i) + } + for j, pkg := range op.Packages { + if strings.TrimSpace(pkg.Name) == "" { + return fmt.Errorf("operators[%d].packages[%d]: name is required", i, j) + } + } + } + for i, img := range c.AdditionalImages { + if strings.TrimSpace(img.Name) == "" { + return fmt.Errorf("additionalImages[%d]: name is required", i) + } + } + if c.Helm != nil { + for i, lc := range c.Helm.Local { + name := strings.TrimSpace(lc.Name) + path := strings.TrimSpace(lc.Path) + if name == "" && path == "" { + continue + } + if path == "" { + return fmt.Errorf("helm.local[%d]: path is required when name is set", i) + } + } + } + return nil +} + +// ApplyExtraConfig merges operators, additional images, and helm.local charts into the inventory. +// BuildImageSet then writes those fields into imagesetconfig.yaml under mirror: so oc-mirror +// can pull them. +func ApplyExtraConfig(inv *PatternInventory, cfg *ExtraConfig, opts *Options) error { + if inv == nil || cfg == nil { + return nil + } + + inv.ExtraConfig = cfg + inv.ExtraOperators = append(inv.ExtraOperators, cfg.Operators...) + + for _, img := range cfg.AdditionalImages { + name := strings.TrimSpace(img.Name) + if name == "" { + continue + } + inv.AdditionalImages = appendUnique(inv.AdditionalImages, name) + } + + if cfg.Helm == nil { + return nil + } + + seen := map[string]bool{} + for _, existing := range inv.LocalCharts { + seen[existing.Path] = true + } + + resolvedLocal := make([]HelmLocalChart, 0, len(cfg.Helm.Local)) + for _, lc := range cfg.Helm.Local { + name := strings.TrimSpace(lc.Name) + path := strings.TrimSpace(lc.Path) + if name == "" && path == "" { + continue + } + absPath := path + if !filepath.IsAbs(absPath) { + absPath = filepath.Join(filepath.Dir(opts.ExtraConfigPath), path) + var err error + absPath, err = filepath.Abs(absPath) + if err != nil { + return fmt.Errorf("resolve helm.local path %q: %w", path, err) + } + } + if name == "" { + name = filepath.Base(absPath) + } + resolvedLocal = append(resolvedLocal, HelmLocalChart{Name: name, Path: absPath}) + if seen[absPath] { + continue + } + seen[absPath] = true + + imgs, warnings, err := DiscoverImagesFromChartDir(absPath) + if err != nil { + inv.Warnings = append(inv.Warnings, fmt.Sprintf("extra-config local chart %s: %v", absPath, err)) + } + inv.Warnings = append(inv.Warnings, warnings...) + for _, img := range imgs { + inv.AdditionalImages = appendUnique(inv.AdditionalImages, img) + } + inv.LocalCharts = append(inv.LocalCharts, LocalChart{ + Name: name, + Path: absPath, + Images: imgs, + }) + } + // Keep ExtraConfig paths absolute so the final ImageSet merge is idempotent. + cfg.Helm.Local = resolvedLocal + return nil +} + +// MergeExtraConfigIntoImageSet merges operators, additionalImages, and helm.local from +// ExtraConfig directly into an ImageSetConfiguration (idempotent with inventory-based merge). +func MergeExtraConfigIntoImageSet(isc *ImageSetConfiguration, cfg *ExtraConfig) { + if isc == nil || cfg == nil { + return + } + + isc.Mirror.Operators = MergeOperatorCatalogs(isc.Mirror.Operators, cfg.Operators) + + seenImg := map[string]bool{} + for _, img := range isc.Mirror.AdditionalImages { + seenImg[img.Name] = true + } + for _, img := range cfg.AdditionalImages { + name := strings.TrimSpace(img.Name) + if name == "" || seenImg[name] { + continue + } + seenImg[name] = true + isc.Mirror.AdditionalImages = append(isc.Mirror.AdditionalImages, NamedImage{Name: name}) + } + + if cfg.Helm == nil { + return + } + if isc.Mirror.Helm == nil { + isc.Mirror.Helm = &HelmMirror{} + } + seenPath := map[string]bool{} + for _, lc := range isc.Mirror.Helm.Local { + seenPath[lc.Path] = true + } + for _, lc := range cfg.Helm.Local { + name := strings.TrimSpace(lc.Name) + path := strings.TrimSpace(lc.Path) + if name == "" && path == "" { + continue + } + if path == "" || seenPath[path] { + continue + } + seenPath[path] = true + if name == "" { + name = filepath.Base(path) + } + isc.Mirror.Helm.Local = append(isc.Mirror.Helm.Local, HelmLocalChart{ + Name: name, + Path: path, + }) + } +} + +// MergeOperatorCatalogs appends extra catalogs into base, combining packages when catalog strings match. +func MergeOperatorCatalogs(base, extra []OperatorCatalog) []OperatorCatalog { + if len(extra) == 0 { + return base + } + out := make([]OperatorCatalog, len(base)) + copy(out, base) + index := map[string]int{} + for i, cat := range out { + index[cat.Catalog] = i + } + for _, cat := range extra { + catalog := strings.TrimSpace(cat.Catalog) + if catalog == "" { + continue + } + if i, ok := index[catalog]; ok { + out[i].Packages = mergePackages(out[i].Packages, cat.Packages) + continue + } + index[catalog] = len(out) + out = append(out, OperatorCatalog{ + Catalog: catalog, + Packages: append([]OperatorPackage(nil), cat.Packages...), + }) + } + return out +} + +func mergePackages(base, extra []OperatorPackage) []OperatorPackage { + seen := map[string]int{} + out := append([]OperatorPackage(nil), base...) + for i, pkg := range out { + seen[pkg.Name] = i + } + for _, pkg := range extra { + name := strings.TrimSpace(pkg.Name) + if name == "" { + continue + } + if i, ok := seen[name]; ok { + if out[i].DefaultChannel == "" && pkg.DefaultChannel != "" { + out[i].DefaultChannel = pkg.DefaultChannel + } + out[i].Channels = mergeChannels(out[i].Channels, pkg.Channels) + continue + } + seen[name] = len(out) + out = append(out, pkg) + } + return out +} + +func mergeChannels(base, extra []OperatorChannel) []OperatorChannel { + seen := map[string]bool{} + out := append([]OperatorChannel(nil), base...) + for _, ch := range out { + seen[ch.Name] = true + } + for _, ch := range extra { + if ch.Name == "" || seen[ch.Name] { + continue + } + seen[ch.Name] = true + out = append(out, ch) + } + return out +} diff --git a/src/internal/airgap/extraconfig_test.go b/src/internal/airgap/extraconfig_test.go new file mode 100644 index 0000000..60154b3 --- /dev/null +++ b/src/internal/airgap/extraconfig_test.go @@ -0,0 +1,150 @@ +package airgap_test + +import ( + "os" + "path/filepath" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "gopkg.in/yaml.v3" + + "github.com/validatedpatterns/patternizer/internal/airgap" +) + +var _ = Describe("ExtraConfig", func() { + var tmpDir string + + BeforeEach(func() { + var err error + tmpDir, err = os.MkdirTemp("", "airgap-extra-") + Expect(err).NotTo(HaveOccurred()) + }) + + AfterEach(func() { + _ = os.RemoveAll(tmpDir) + }) + + It("loads and merges operators, images, and helm.local charts", func() { + chartDir := filepath.Join(tmpDir, "charts", "demo") + Expect(os.MkdirAll(filepath.Join(chartDir, "templates"), 0o755)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(chartDir, "Chart.yaml"), []byte(` +apiVersion: v2 +name: demo +version: 0.1.0 +`), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(chartDir, "values.yaml"), []byte(` +image: quay.io/example/demo:1.2.3 +`), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(chartDir, "templates", "deploy.yaml"), []byte(` +apiVersion: apps/v1 +kind: Deployment +metadata: + name: demo +spec: + template: + spec: + containers: + - name: demo + image: "{{ .Values.image }}" +`), 0o644)).To(Succeed()) + + extraPath := filepath.Join(tmpDir, "extra-images.yaml") + Expect(os.WriteFile(extraPath, []byte(` +operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.22 + packages: + - name: lvms-operator +additionalImages: + - name: registry.example.com/extra:1.0 +helm: + local: + - name: demo + path: `+chartDir+` + - name: + path: +`), 0o644)).To(Succeed()) + + cfg, err := airgap.LoadExtraConfig(extraPath) + Expect(err).NotTo(HaveOccurred()) + Expect(cfg.Operators).To(HaveLen(1)) + Expect(cfg.Operators[0].Packages[0].Name).To(Equal("lvms-operator")) + + inv := &airgap.PatternInventory{} + opts := &airgap.Options{ExtraConfigPath: extraPath} + Expect(airgap.ApplyExtraConfig(inv, cfg, opts)).To(Succeed()) + Expect(inv.AdditionalImages).To(ContainElement("registry.example.com/extra:1.0")) + Expect(inv.AdditionalImages).To(ContainElement("quay.io/example/demo:1.2.3")) + Expect(inv.LocalCharts).To(HaveLen(1)) + Expect(inv.LocalCharts[0].Path).To(Equal(chartDir)) + Expect(inv.ExtraOperators).To(HaveLen(1)) + + isc := airgap.BuildImageSet(inv, &airgap.Options{}) + Expect(isc.Mirror.Platform).To(BeNil()) + Expect(isc.Mirror.Operators).To(HaveLen(1)) + Expect(isc.Mirror.Operators[0].Catalog).To(Equal("registry.redhat.io/redhat/redhat-operator-index:v4.22")) + Expect(isc.Mirror.Helm).NotTo(BeNil()) + Expect(isc.Mirror.Helm.Local[0].Path).To(Equal(chartDir)) + + raw, err := yaml.Marshal(isc) + Expect(err).NotTo(HaveOccurred()) + Expect(string(raw)).To(ContainSubstring("lvms-operator")) + Expect(string(raw)).To(ContainSubstring("registry.example.com/extra:1.0")) + }) + + It("accepts mirror: wrapped extra config and merges into ImageSet", func() { + path := filepath.Join(tmpDir, "wrapped.yaml") + Expect(os.WriteFile(path, []byte(` +mirror: + operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.22 + packages: + - name: lvms-operator + additionalImages: + - name: registry.example.com/wrapped:1 +`), 0o644)).To(Succeed()) + + cfg, err := airgap.LoadExtraConfig(path) + Expect(err).NotTo(HaveOccurred()) + Expect(cfg.Operators).To(HaveLen(1)) + Expect(cfg.AdditionalImages[0].Name).To(Equal("registry.example.com/wrapped:1")) + + isc := &airgap.ImageSetConfiguration{ + Kind: "ImageSetConfiguration", + APIVersion: "mirror.openshift.io/v2alpha1", + } + airgap.MergeExtraConfigIntoImageSet(isc, cfg) + Expect(isc.Mirror.Operators).To(HaveLen(1)) + Expect(isc.Mirror.Operators[0].Packages[0].Name).To(Equal("lvms-operator")) + Expect(isc.Mirror.AdditionalImages).To(ContainElement(airgap.NamedImage{Name: "registry.example.com/wrapped:1"})) + }) + + It("rejects blank additional image names", func() { + path := filepath.Join(tmpDir, "bad.yaml") + Expect(os.WriteFile(path, []byte(` +additionalImages: + - name: "" +`), 0o644)).To(Succeed()) + _, err := airgap.LoadExtraConfig(path) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("additionalImages")) + }) + + It("merges packages into an existing catalog", func() { + base := []airgap.OperatorCatalog{{ + Catalog: "registry.redhat.io/redhat/redhat-operator-index:v4.22", + Packages: []airgap.OperatorPackage{ + {Name: "advanced-cluster-management"}, + }, + }} + extra := []airgap.OperatorCatalog{{ + Catalog: "registry.redhat.io/redhat/redhat-operator-index:v4.22", + Packages: []airgap.OperatorPackage{ + {Name: "lvms-operator"}, + }, + }} + merged := airgap.MergeOperatorCatalogs(base, extra) + Expect(merged).To(HaveLen(1)) + Expect(merged[0].Packages).To(HaveLen(2)) + }) +}) diff --git a/src/internal/airgap/framework.go b/src/internal/airgap/framework.go new file mode 100644 index 0000000..1cb8253 --- /dev/null +++ b/src/internal/airgap/framework.go @@ -0,0 +1,28 @@ +package airgap + +// FrameworkImages returns images always mirrored for the Validated Patterns install path. +func FrameworkImages(helmRepo string) []string { + return []string{ + fmtImage(helmRepo, "utility-container", "latest"), + fmtImage(helmRepo, "imperative-container", "v1"), + } +} + +// FrameworkOperators returns operators always included for pattern install. +func FrameworkOperators() []OperatorRef { + return []OperatorRef{ + {Name: "patterns-operator", Catalog: CatalogCommunity}, + {Name: "openshift-gitops-operator", Catalog: CatalogRedHat}, + } +} + +// ACMCompanionOperators returns operators added when ACM is subscribed. +func ACMCompanionOperators() []OperatorRef { + return []OperatorRef{ + {Name: "multicluster-engine", Catalog: CatalogRedHat}, + } +} + +func fmtImage(repo, name, tag string) string { + return repo + "/" + name + ":" + tag +} diff --git a/src/internal/airgap/framework_test.go b/src/internal/airgap/framework_test.go new file mode 100644 index 0000000..63c7bd4 --- /dev/null +++ b/src/internal/airgap/framework_test.go @@ -0,0 +1,18 @@ +package airgap_test + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/validatedpatterns/patternizer/internal/airgap" +) + +var _ = Describe("FrameworkImages", func() { + It("includes utility and imperative containers but not pattern-install", func() { + imgs := airgap.FrameworkImages("quay.io/validatedpatterns") + Expect(imgs).To(ConsistOf( + "quay.io/validatedpatterns/utility-container:latest", + "quay.io/validatedpatterns/imperative-container:v1", + )) + }) +}) diff --git a/src/internal/airgap/images.go b/src/internal/airgap/images.go new file mode 100644 index 0000000..74b5988 --- /dev/null +++ b/src/internal/airgap/images.go @@ -0,0 +1,473 @@ +package airgap + +import ( + "fmt" + "os" + "os/exec" + "path/filepath" + "sort" + "strings" + + "gopkg.in/yaml.v3" +) + +// DiscoverImagesFromChartDir discovers container images required to deploy a chart. +func DiscoverImagesFromChartDir(chartDir string) (images, warnings []string, err error) { + found := map[string]struct{}{} + + templated, warn, tmplErr := imagesFromHelmTemplate(chartDir) + if tmplErr != nil { + warnings = append(warnings, fmt.Sprintf("%s helm template: %v", chartDir, tmplErr)) + } + if warn != "" { + warnings = append(warnings, warn) + } + for _, img := range templated { + found[img] = struct{}{} + } + + fromValues, valuesErr := imagesFromValues(chartDir) + if valuesErr != nil { + warnings = append(warnings, fmt.Sprintf("%s values scrape: %v", chartDir, valuesErr)) + } + for _, img := range fromValues { + found[img] = struct{}{} + } + + return filterRequiredImages(found, warnings), warnings, nil +} + +// DiscoverOCIChartImages pulls a remote OCI Helm chart and discovers deploy-time images. +func DiscoverOCIChartImages(helmRepo, chartName, version string, skipNetwork bool) (images, warnings []string, err error) { + if skipNetwork { + return nil, []string{fmt.Sprintf("skipped OCI image discovery for %s (skip-oci-resolve)", chartName)}, nil + } + if _, err := exec.LookPath("helm"); err != nil { + return nil, nil, fmt.Errorf("helm required to discover images for %s", chartName) + } + + ref := fmt.Sprintf("oci://%s/%s", helmRepo, chartName) + + tmp, err := os.MkdirTemp("", "patternizer-chart-"+sanitizeDirName(chartName)+"-") + if err != nil { + return nil, warnings, err + } + defer os.RemoveAll(tmp) + + args := []string{"pull", ref, "--untar", "--untardir", tmp} + if version != "" && version != "*" && !strings.Contains(version, "*") { + args = append(args, "--version", version) + } + cmd := exec.Command("helm", args...) + out, err := cmd.CombinedOutput() + if err != nil { + showImgs, showErr := imagesFromHelmShowValues(ref, version) + if showErr != nil { + return nil, warnings, fmt.Errorf("helm pull %s: %w (%s); show values: %v", ref, err, truncate(string(out), 300), showErr) + } + warnings = append(warnings, fmt.Sprintf("%s helm pull failed, using show values only: %v", chartName, err)) + return filterRequiredImages(sliceToSet(showImgs), warnings), warnings, nil + } + + chartDir := findPulledChartDir(tmp, chartName) + if chartDir == "" { + return nil, warnings, fmt.Errorf("pulled chart dir not found for %s", chartName) + } + + if _, err := os.Stat(filepath.Join(chartDir, "Chart.yaml")); err == nil { + depCmd := exec.Command("helm", "dependency", "build", chartDir) + if depOut, depErr := depCmd.CombinedOutput(); depErr != nil { + warnings = append(warnings, fmt.Sprintf("%s helm dependency build: %v (%s)", chartName, depErr, truncate(string(depOut), 200))) + } + } + + pulled, pullWarns, discErr := DiscoverImagesFromChartDir(chartDir) + warnings = append(warnings, pullWarns...) + if discErr != nil { + warnings = append(warnings, fmt.Sprintf("%s discover: %v", chartName, discErr)) + } + return pulled, warnings, nil +} + +func imagesFromHelmShowValues(ref, version string) ([]string, error) { + args := []string{"show", "values", ref} + if version != "" && version != "*" && !strings.Contains(version, "*") { + args = append(args, "--version", version) + } + cmd := exec.Command("helm", args...) + out, err := cmd.CombinedOutput() + if err != nil { + return nil, fmt.Errorf("%w: %s", err, truncate(string(out), 300)) + } + var values map[string]interface{} + if err := yaml.Unmarshal(out, &values); err != nil { + return nil, err + } + var images []string + collectImageFields(values, &images) + return dropRedundantImagePrefixes(uniqueSorted(images)), nil +} + +func findPulledChartDir(parent, chartName string) string { + candidate := filepath.Join(parent, chartName) + if info, err := os.Stat(candidate); err == nil && info.IsDir() { + return candidate + } + entries, err := os.ReadDir(parent) + if err != nil { + return "" + } + for _, e := range entries { + if e.IsDir() { + return filepath.Join(parent, e.Name()) + } + } + return "" +} + +func imagesFromValues(chartDir string) ([]string, error) { + data, err := os.ReadFile(filepath.Join(chartDir, "values.yaml")) + if err != nil { + return nil, err + } + var values map[string]interface{} + if err := yaml.Unmarshal(data, &values); err != nil { + return nil, err + } + var images []string + collectImageFields(values, &images) + return images, nil +} + +func collectImageFields(v interface{}, out *[]string) { + switch t := v.(type) { + case map[string]interface{}: + if isExplicitlyDisabled(t) { + return + } + if img, ok := composeImageFromFields(t); ok { + *out = append(*out, img) + } + if img := stringFromYAML(t["image"]); img != "" && strings.Contains(img, "/") { + *out = append(*out, img) + } + for _, child := range t { + collectImageFields(child, out) + } + case []interface{}: + for _, child := range t { + collectImageFields(child, out) + } + } +} + +// composeImageFromFields builds a pullspec from common Helm image map fields. +// Charts like travelops set registry=quay.io and repository=kiali (org only) while the +// image name is hardcoded in the template (…/demo_travels_control). Those incomplete +// maps are skipped here; helm template discovery picks up the full refs. +func composeImageFromFields(t map[string]interface{}) (string, bool) { + registry := strings.TrimSuffix(stringFromYAML(t["registry"]), "/") + repo := strings.TrimPrefix(stringFromYAML(t["repository"]), "/") + name := strings.TrimPrefix(stringFromYAML(t["name"]), "/") + tag := stringFromYAML(t["tag"]) + if repo == "" && name == "" { + return "", false + } + // registry + single-segment repository (no name) is an org/project prefix, not an image. + if registry != "" && repo != "" && name == "" && !strings.Contains(repo, "/") { + return "", false + } + img := repo + if img == "" { + img = name + name = "" + } + if registry != "" { + img = registry + "/" + img + } + if name != "" { + img = img + "/" + name + } + if tag != "" { + img = img + ":" + tag + } + return img, true +} + +func isExplicitlyDisabled(m map[string]interface{}) bool { + for _, key := range []string{"enabled", "enable"} { + switch v := m[key].(type) { + case bool: + if !v { + return true + } + case string: + if strings.EqualFold(v, "false") { + return true + } + } + } + return false +} + +func stringFromYAML(v interface{}) string { + switch t := v.(type) { + case string: + return t + case int: + return fmt.Sprintf("%d", t) + case int64: + return fmt.Sprintf("%d", t) + case float64: + if t == float64(int64(t)) { + return fmt.Sprintf("%d", int64(t)) + } + return fmt.Sprintf("%v", t) + default: + return "" + } +} + +func imagesFromHelmTemplate(chartDir string) (images []string, warning string, err error) { + if _, err := exec.LookPath("helm"); err != nil { + return nil, "helm not found on PATH; skipping helm template discovery", nil + } + cmd := exec.Command("helm", "template", "airgap-discover", chartDir, "--include-crds") + out, err := cmd.CombinedOutput() + if err != nil { + return nil, "", fmt.Errorf("%w: %s", err, truncate(string(out), 400)) + } + return scrapeManifestImages(string(out)), "", nil +} + +func scrapeManifestImages(manifest string) []string { + var images []string + for _, line := range strings.Split(manifest, "\n") { + trimmed := strings.TrimSpace(line) + trimmed = strings.TrimSpace(strings.TrimPrefix(trimmed, "- ")) + if !strings.HasPrefix(trimmed, "image:") { + continue + } + val := strings.TrimSpace(strings.TrimPrefix(trimmed, "image:")) + val = strings.Trim(val, `"'`) + if val == "" || strings.HasPrefix(val, "{{") { + continue + } + images = append(images, val) + } + return images +} + +func filterRequiredImages(images map[string]struct{}, warnings []string) []string { + out := make([]string, 0, len(images)) + for img := range images { + if !isLikelyImage(img) { + continue + } + if !isVPSourceImage(img) { + continue + } + out = append(out, normalizeImagePullSpec(img)) + } + return dropRedundantImagePrefixes(uniqueSorted(out)) +} + +func sliceToSet(in []string) map[string]struct{} { + out := map[string]struct{}{} + for _, s := range in { + out[s] = struct{}{} + } + return out +} + +func isVPSourceImage(img string) bool { + img = strings.TrimSpace(img) + if img == "" { + return false + } + if strings.HasPrefix(img, "docker.io/") { + return false + } + host, _, ok := splitRegistry(img) + if !ok { + return false + } + if isKubernetesServiceHost(host) { + return false + } + switch host { + case "registry.redhat.io", + "registry.connect.redhat.com", + "registry.access.redhat.com", + "quay.io", + "ghcr.io": + return true + default: + if host == "docker.io" || host == "index.docker.io" || host == "registry-1.docker.io" { + return false + } + return strings.Contains(host, ".") + } +} + +// isKubernetesServiceHost reports whether host is a cluster-local Kubernetes +// Service DNS name (e.g. image-registry.openshift-image-registry.svc:5000). +func isKubernetesServiceHost(host string) bool { + h := host + if i := strings.LastIndex(h, ":"); i >= 0 { + h = h[:i] + } + return strings.HasSuffix(h, ".svc") || strings.Contains(h, ".svc.") +} + +func splitRegistry(img string) (host, remainder string, ok bool) { + parts := strings.SplitN(img, "/", 2) + if len(parts) < 2 { + return "", "", false + } + host = parts[0] + if strings.Contains(host, ".") || strings.Contains(host, ":") { + return host, parts[1], true + } + return "", "", false +} + +func normalizeImagePullSpec(img string) string { + return strings.TrimSpace(img) +} + +func isLikelyImage(s string) bool { + if s == "" || strings.Contains(s, "{{") || strings.Contains(s, "}}") { + return false + } + if !strings.Contains(s, "/") { + return false + } + if strings.HasPrefix(s, ".") || strings.HasPrefix(s, "/") { + return false + } + return hasCompleteImagePath(s) +} + +// hasCompleteImagePath rejects incomplete specs like quay.io/kiali:v1 where values +// only supplied an org and the image name lives in the Helm template. Multi-tenant +// hosts need registry/org/name; other hosts may use a single path segment. +func hasCompleteImagePath(img string) bool { + host, remainder, ok := splitRegistry(img) + path := img + if ok { + path = remainder + } + path = stripImageTagOrDigest(path) + n := 0 + for _, seg := range strings.Split(path, "/") { + if seg != "" { + n++ + } + } + if n < 1 { + return false + } + if !ok { + return n >= 2 + } + if n >= 2 { + return true + } + return !isMultiTenantRegistryHost(host) +} + +func isMultiTenantRegistryHost(host string) bool { + h := host + if i := strings.LastIndex(h, ":"); i >= 0 && !strings.Contains(h[i+1:], "]") { + h = h[:i] + } + switch h { + case "quay.io", + "ghcr.io", + "docker.io", + "index.docker.io", + "registry-1.docker.io", + "registry.redhat.io", + "registry.connect.redhat.com", + "registry.access.redhat.com", + "gcr.io", + "public.ecr.aws": + return true + default: + return false + } +} + +func stripImageTagOrDigest(path string) string { + if i := strings.Index(path, "@"); i >= 0 { + path = path[:i] + } + if i := strings.LastIndex(path, ":"); i >= 0 { + path = path[:i] + } + return path +} + +func imageRepoPath(img string) string { + img = strings.TrimSpace(img) + host, rem, ok := splitRegistry(img) + if !ok { + return stripImageTagOrDigest(img) + } + return host + "/" + stripImageTagOrDigest(rem) +} + +// dropRedundantImagePrefixes removes pullspecs whose repository path is a strict +// prefix of another collected image (e.g. quay.io/kiali when quay.io/kiali/app exists). +func dropRedundantImagePrefixes(images []string) []string { + if len(images) < 2 { + return images + } + paths := make([]string, len(images)) + for i, img := range images { + paths[i] = imageRepoPath(img) + } + keep := make([]bool, len(images)) + for i := range images { + keep[i] = true + for j := range images { + if i == j { + continue + } + if paths[i] != paths[j] && strings.HasPrefix(paths[j], paths[i]+"/") { + keep[i] = false + break + } + } + } + out := make([]string, 0, len(images)) + for i, img := range images { + if keep[i] { + out = append(out, img) + } + } + return out +} + +func uniqueSorted(in []string) []string { + set := map[string]struct{}{} + for _, s := range in { + if isLikelyImage(s) && isVPSourceImage(s) { + set[normalizeImagePullSpec(s)] = struct{}{} + } + } + out := make([]string, 0, len(set)) + for s := range set { + out = append(out, s) + } + sort.Strings(out) + return out +} + +func truncate(s string, n int) string { + if len(s) <= n { + return s + } + return s[:n] + "..." +} diff --git a/src/internal/airgap/images_resolve_test.go b/src/internal/airgap/images_resolve_test.go new file mode 100644 index 0000000..5735736 --- /dev/null +++ b/src/internal/airgap/images_resolve_test.go @@ -0,0 +1,62 @@ +package airgap_test + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/validatedpatterns/patternizer/internal/airgap" +) + +var _ = Describe("OCI chart version resolution", func() { + DescribeTable("IsConcreteChartVersion", + func(v string, want bool) { + Expect(airgap.IsConcreteChartVersion(v)).To(Equal(want)) + }, + Entry("semver patch", "0.9.58", true), + Entry("semver minor", "0.1.8", true), + Entry("latest", "latest", true), + Entry("wildcard star", "0.9.*", false), + Entry("legacy placeholder", "0.9.x", false), + Entry("caret constraint", "^0.9.0", false), + Entry("empty", "", false), + Entry("star only", "*", false), + ) + + DescribeTable("NormalizeChartConstraint", + func(in, want string) { + Expect(airgap.NormalizeChartConstraint(in)).To(Equal(want)) + }, + Entry("empty becomes star", "", "*"), + Entry("star unchanged", "*", "*"), + Entry("dot-star unchanged", "0.9.*", "0.9.*"), + Entry("dot-x to star", "0.9.x", "0.9.*"), + Entry("dot-X to star", "0.9.X", "0.9.*"), + Entry("concrete unchanged", "0.9.58", "0.9.58"), + ) + + It("picks the latest tag matching 0.9.*", func() { + ver, err := airgap.PickLatestMatchingTag([]string{ + "0.8.9", "0.9.41", "0.9.57", "0.9.58", "0.10.0", "latest", "0.9.x", + }, "0.9.*") + Expect(err).NotTo(HaveOccurred()) + Expect(ver).To(Equal("0.9.58")) + }) + + It("picks the latest tag matching 0.9.x placeholder form", func() { + ver, err := airgap.PickLatestMatchingTag([]string{"0.9.1", "0.9.50", "0.9.57"}, "0.9.x") + Expect(err).NotTo(HaveOccurred()) + Expect(ver).To(Equal("0.9.57")) + }) + + It("returns concrete versions as-is even when skip is set", func() { + ver, err := airgap.ResolveOCIChartVersion("quay.io/validatedpatterns", "clustergroup", "0.9.41", true) + Expect(err).NotTo(HaveOccurred()) + Expect(ver).To(Equal("0.9.41")) + }) + + It("refuses wildcards when skip-oci-resolve is set", func() { + _, err := airgap.ResolveOCIChartVersion("quay.io/validatedpatterns", "clustergroup", "0.9.*", true) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("skip-oci-resolve")) + }) +}) diff --git a/src/internal/airgap/images_test.go b/src/internal/airgap/images_test.go new file mode 100644 index 0000000..387d431 --- /dev/null +++ b/src/internal/airgap/images_test.go @@ -0,0 +1,287 @@ +package airgap_test + +import ( + "os" + "path/filepath" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/validatedpatterns/patternizer/internal/airgap" +) + +var _ = Describe("DiscoverImagesFromChartDir", func() { + It("keeps RH vault image and skips disabled injector defaults", func() { + tmp, err := os.MkdirTemp("", "vault-values-") + Expect(err).NotTo(HaveOccurred()) + defer os.RemoveAll(tmp) + + Expect(os.WriteFile(filepath.Join(tmp, "Chart.yaml"), []byte(` +apiVersion: v2 +name: hashicorp-vault +version: 0.1.8 +`), 0o644)).To(Succeed()) + Expect(os.MkdirAll(filepath.Join(tmp, "templates"), 0o755)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "templates", "dummy.yaml"), []byte(` +apiVersion: v1 +kind: ConfigMap +metadata: + name: dummy +`), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "values.yaml"), []byte(` +vault: + server: + image: + repository: "registry.connect.redhat.com/hashicorp/vault" + tag: "1.21.4-ubi" + injector: + enabled: false + image: + repository: "hashicorp/vault-k8s" + tag: "1.7.2" + csi: + enabled: false + image: + repository: "hashicorp/vault-csi-provider" + tag: "1.7.0" +`), 0o644)).To(Succeed()) + + Expect(os.MkdirAll(filepath.Join(tmp, "charts", "vault"), 0o755)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "charts", "vault", "Chart.yaml"), []byte(` +apiVersion: v2 +name: vault +version: 0.30.0 +`), 0o644)).To(Succeed()) + Expect(os.MkdirAll(filepath.Join(tmp, "charts", "vault", "templates"), 0o755)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "charts", "vault", "templates", "dummy.yaml"), []byte("{}\n"), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "charts", "vault", "values.yaml"), []byte(` +server: + image: + repository: hashicorp/vault + tag: 1.21.2 +injector: + enabled: true + image: + repository: hashicorp/vault-k8s + tag: 1.7.2 +`), 0o644)).To(Succeed()) + + imgs, _, err := airgap.DiscoverImagesFromChartDir(tmp) + Expect(err).NotTo(HaveOccurred()) + Expect(imgs).To(ContainElement("registry.connect.redhat.com/hashicorp/vault:1.21.4-ubi")) + Expect(imgs).NotTo(ContainElement("hashicorp/vault-k8s:1.7.2")) + Expect(imgs).NotTo(ContainElement("hashicorp/vault-csi-provider:1.7.0")) + Expect(imgs).NotTo(ContainElement("hashicorp/vault:1.21.2")) + }) +}) + +var _ = Describe("collectImageFields registry composition", func() { + It("joins registry/repository:tag from values when repository is complete", func() { + tmp, err := os.MkdirTemp("", "registry-compose-") + Expect(err).NotTo(HaveOccurred()) + defer os.RemoveAll(tmp) + Expect(os.WriteFile(filepath.Join(tmp, "Chart.yaml"), []byte("apiVersion: v2\nname: t\nversion: 0.1.0\n"), 0o644)).To(Succeed()) + Expect(os.MkdirAll(filepath.Join(tmp, "templates"), 0o755)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "templates", "x.yaml"), []byte("{}\n"), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "values.yaml"), []byte(` +image: + registry: quay.io + repository: kiali/demo_travels_mysqldb + tag: v1 +`), 0o644)).To(Succeed()) + + imgs, _, err := airgap.DiscoverImagesFromChartDir(tmp) + Expect(err).NotTo(HaveOccurred()) + Expect(imgs).To(ContainElement("quay.io/kiali/demo_travels_mysqldb:v1")) + }) + + It("joins registry/repository/name:tag when name is a separate field", func() { + tmp, err := os.MkdirTemp("", "registry-name-") + Expect(err).NotTo(HaveOccurred()) + defer os.RemoveAll(tmp) + Expect(os.WriteFile(filepath.Join(tmp, "Chart.yaml"), []byte("apiVersion: v2\nname: t\nversion: 0.1.0\n"), 0o644)).To(Succeed()) + Expect(os.MkdirAll(filepath.Join(tmp, "templates"), 0o755)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "templates", "x.yaml"), []byte("{}\n"), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "values.yaml"), []byte(` +image: + registry: quay.io + repository: kiali + name: demo_travels_control + tag: v1 +`), 0o644)).To(Succeed()) + + imgs, _, err := airgap.DiscoverImagesFromChartDir(tmp) + Expect(err).NotTo(HaveOccurred()) + Expect(imgs).To(ContainElement("quay.io/kiali/demo_travels_control:v1")) + }) + + It("does not invent incomplete org-only pullspecs from values", func() { + tmp, err := os.MkdirTemp("", "registry-org-only-") + Expect(err).NotTo(HaveOccurred()) + defer os.RemoveAll(tmp) + Expect(os.WriteFile(filepath.Join(tmp, "Chart.yaml"), []byte("apiVersion: v2\nname: t\nversion: 0.1.0\n"), 0o644)).To(Succeed()) + Expect(os.MkdirAll(filepath.Join(tmp, "templates"), 0o755)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "templates", "x.yaml"), []byte("{}\n"), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "values.yaml"), []byte(` +image: + registry: quay.io + repository: kiali + tag: v1 +`), 0o644)).To(Succeed()) + + imgs, _, err := airgap.DiscoverImagesFromChartDir(tmp) + Expect(err).NotTo(HaveOccurred()) + Expect(imgs).NotTo(ContainElement("quay.io/kiali:v1")) + Expect(imgs).To(BeEmpty()) + }) + + It("resolves travelops-style images where the name is hardcoded in the template", func() { + tmp, err := os.MkdirTemp("", "travel-control-") + Expect(err).NotTo(HaveOccurred()) + defer os.RemoveAll(tmp) + Expect(os.WriteFile(filepath.Join(tmp, "Chart.yaml"), []byte("apiVersion: v2\nname: travel-control\nversion: 0.1.0\n"), 0o644)).To(Succeed()) + Expect(os.MkdirAll(filepath.Join(tmp, "templates"), 0o755)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "templates", "deployment.yaml"), []byte(` +apiVersion: apps/v1 +kind: Deployment +metadata: + name: control +spec: + template: + spec: + containers: + {{- with $img := .Values.image }} + - name: control + image: {{ $img.registry }}/{{ $img.repository }}/demo_travels_control:{{ $img.tag }} + {{- end }} +`), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "values.yaml"), []byte(` +image: + registry: quay.io + repository: kiali + tag: v1 +`), 0o644)).To(Succeed()) + + imgs, _, err := airgap.DiscoverImagesFromChartDir(tmp) + Expect(err).NotTo(HaveOccurred()) + Expect(imgs).To(ContainElement("quay.io/kiali/demo_travels_control:v1")) + Expect(imgs).NotTo(ContainElement("quay.io/kiali:v1")) + }) + + It("resolves ranged app image names from helm templates", func() { + tmp, err := os.MkdirTemp("", "travel-agency-") + Expect(err).NotTo(HaveOccurred()) + defer os.RemoveAll(tmp) + Expect(os.WriteFile(filepath.Join(tmp, "Chart.yaml"), []byte("apiVersion: v2\nname: travel-agency\nversion: 0.1.0\n"), 0o644)).To(Succeed()) + Expect(os.MkdirAll(filepath.Join(tmp, "templates"), 0o755)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "templates", "deployment.yaml"), []byte(` +{{- range $apps := .Values.apps }} +{{- with $img := $.Values.image }} +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ $apps.name }} +spec: + template: + spec: + containers: + - name: {{ $apps.name }} + image: {{ $img.registry }}/{{ $img.repository }}/demo_travels_{{ $apps.name }}:{{ $img.tag }} +{{- end }} +{{- end }} +`), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "values.yaml"), []byte(` +image: + registry: quay.io + repository: kiali + tag: v1 +apps: + - name: cars + - name: hotels +`), 0o644)).To(Succeed()) + + imgs, _, err := airgap.DiscoverImagesFromChartDir(tmp) + Expect(err).NotTo(HaveOccurred()) + Expect(imgs).To(ContainElement("quay.io/kiali/demo_travels_cars:v1")) + Expect(imgs).To(ContainElement("quay.io/kiali/demo_travels_hotels:v1")) + Expect(imgs).NotTo(ContainElement("quay.io/kiali:v1")) + }) +}) + +var _ = Describe("isVPSourceImage filtering", func() { + It("accepts VP registries and rejects docker hub short names", func() { + tmp, err := os.MkdirTemp("", "filter-") + Expect(err).NotTo(HaveOccurred()) + defer os.RemoveAll(tmp) + Expect(os.WriteFile(filepath.Join(tmp, "Chart.yaml"), []byte("apiVersion: v2\nname: t\nversion: 0.1.0\n"), 0o644)).To(Succeed()) + Expect(os.MkdirAll(filepath.Join(tmp, "templates"), 0o755)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "templates", "x.yaml"), []byte("{}\n"), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "values.yaml"), []byte(` +a: + image: + repository: registry.redhat.io/ubi9/ubi-minimal + tag: latest +b: + image: + repository: docker.io/library/nginx + tag: latest +c: + image: + repository: hashicorp/vault + tag: 1.21.2 +d: + image: + repository: quay.io/validatedpatterns/imperative-container + tag: v1 +e: + image: + repository: ghcr.io/external-secrets/external-secrets + tag: v0.10.2-ubi +`), 0o644)).To(Succeed()) + + imgs, _, err := airgap.DiscoverImagesFromChartDir(tmp) + Expect(err).NotTo(HaveOccurred()) + Expect(imgs).To(ContainElement("registry.redhat.io/ubi9/ubi-minimal:latest")) + Expect(imgs).To(ContainElement("quay.io/validatedpatterns/imperative-container:v1")) + Expect(imgs).To(ContainElement("ghcr.io/external-secrets/external-secrets:v0.10.2-ubi")) + Expect(imgs).NotTo(ContainElement("docker.io/library/nginx:latest")) + Expect(imgs).NotTo(ContainElement("hashicorp/vault:1.21.2")) + }) + + It("ignores Kubernetes Service DNS image addresses", func() { + tmp, err := os.MkdirTemp("", "k8s-svc-") + Expect(err).NotTo(HaveOccurred()) + defer os.RemoveAll(tmp) + Expect(os.WriteFile(filepath.Join(tmp, "Chart.yaml"), []byte("apiVersion: v2\nname: t\nversion: 0.1.0\n"), 0o644)).To(Succeed()) + Expect(os.MkdirAll(filepath.Join(tmp, "templates"), 0o755)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "templates", "x.yaml"), []byte(` +apiVersion: v1 +kind: Pod +metadata: + name: x +spec: + containers: + - name: c + image: image-registry.openshift-image-registry.svc:5000/openshift/cli +`), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "values.yaml"), []byte(` +image: + repository: image-registry.openshift-image-registry.svc:5000/openshift/cli + tag: latest +other: + image: some-service.my-ns.svc.cluster.local:5000/ns/app:v1 +keep: + image: + repository: quay.io/validatedpatterns/imperative-container + tag: v1 +`), 0o644)).To(Succeed()) + + imgs, _, err := airgap.DiscoverImagesFromChartDir(tmp) + Expect(err).NotTo(HaveOccurred()) + Expect(imgs).To(ContainElement("quay.io/validatedpatterns/imperative-container:v1")) + Expect(imgs).NotTo(ContainElement("image-registry.openshift-image-registry.svc:5000/openshift/cli")) + Expect(imgs).NotTo(ContainElement("image-registry.openshift-image-registry.svc:5000/openshift/cli:latest")) + Expect(imgs).NotTo(ContainElement("some-service.my-ns.svc.cluster.local:5000/ns/app:v1")) + }) +}) diff --git a/src/internal/airgap/imageset.go b/src/internal/airgap/imageset.go new file mode 100644 index 0000000..7c27dc3 --- /dev/null +++ b/src/internal/airgap/imageset.go @@ -0,0 +1,276 @@ +package airgap + +import ( + "fmt" + "os" + "path/filepath" + "sort" + "strings" + + "gopkg.in/yaml.v3" +) + +// ImageSetConfiguration is the oc-mirror v2 input document. +type ImageSetConfiguration struct { + Kind string `yaml:"kind"` + APIVersion string `yaml:"apiVersion"` + Mirror Mirror `yaml:"mirror"` +} + +// Mirror is the mirror section of ImageSetConfiguration. +type Mirror struct { + Platform *Platform `yaml:"platform,omitempty"` + Operators []OperatorCatalog `yaml:"operators,omitempty"` + AdditionalImages []NamedImage `yaml:"additionalImages,omitempty"` + Helm *HelmMirror `yaml:"helm,omitempty"` +} + +// Platform mirrors OCP release content. +type Platform struct { + Graph bool `yaml:"graph,omitempty"` + Channels []PlatformChannel `yaml:"channels,omitempty"` +} + +// PlatformChannel is an OCP release channel. +type PlatformChannel struct { + Name string `yaml:"name"` + Type string `yaml:"type"` +} + +// OperatorCatalog is one catalog entry under mirror.operators. +type OperatorCatalog struct { + Catalog string `yaml:"catalog"` + Packages []OperatorPackage `yaml:"packages,omitempty"` +} + +// OperatorPackage is a package filter within a catalog. +type OperatorPackage struct { + Name string `yaml:"name"` + DefaultChannel string `yaml:"defaultChannel,omitempty"` + Channels []OperatorChannel `yaml:"channels,omitempty"` +} + +// OperatorChannel pins a channel name. +type OperatorChannel struct { + Name string `yaml:"name"` +} + +// NamedImage is an additionalImages entry. +type NamedImage struct { + Name string `yaml:"name"` +} + +// HelmMirror holds local chart paths for oc-mirror. +type HelmMirror struct { + Local []HelmLocalChart `yaml:"local,omitempty"` +} + +// HelmLocalChart is a helm.local entry. +type HelmLocalChart struct { + Name string `yaml:"name"` + Path string `yaml:"path"` +} + +// BuildImageSet constructs an ImageSetConfiguration from inventory. +func BuildImageSet(inv *PatternInventory, opts *Options) *ImageSetConfiguration { + isc := &ImageSetConfiguration{ + Kind: "ImageSetConfiguration", + APIVersion: "mirror.openshift.io/v2alpha1", + } + + if ver := strings.TrimSpace(opts.OCPVersion); ver != "" { + isc.Mirror.Platform = &Platform{ + Graph: true, + Channels: []PlatformChannel{ + {Name: "stable-" + ver, Type: "ocp"}, + }, + } + } + + catVer := catalogOCPVersion(opts) + isc.Mirror.Operators = MergeOperatorCatalogs( + buildOperatorCatalogs(inv.Operators, catVer), + inv.ExtraOperators, + ) + + imgSet := map[string]struct{}{} + for _, img := range inv.AdditionalImages { + imgSet[img] = struct{}{} + } + for _, rc := range inv.RemoteCharts { + if rc.PullSpec != "" && IsConcreteChartVersion(rc.ResolvedVersion) { + imgSet[rc.PullSpec] = struct{}{} + } + } + for _, lc := range inv.LocalCharts { + for _, img := range lc.Images { + imgSet[img] = struct{}{} + } + } + + imgs := make([]string, 0, len(imgSet)) + for img := range imgSet { + imgs = append(imgs, img) + } + sort.Strings(imgs) + for _, img := range imgs { + isc.Mirror.AdditionalImages = append(isc.Mirror.AdditionalImages, NamedImage{Name: img}) + } + + if len(inv.LocalCharts) > 0 { + isc.Mirror.Helm = &HelmMirror{} + for _, lc := range inv.LocalCharts { + isc.Mirror.Helm.Local = append(isc.Mirror.Helm.Local, HelmLocalChart{ + Name: lc.Name, + Path: lc.Path, + }) + } + } + + return isc +} + +func buildOperatorCatalogs(ops []OperatorRef, ocpVersion string) []OperatorCatalog { + type pkgKey struct { + catalog CatalogKind + name string + } + packages := map[pkgKey]*OperatorPackage{} + order := []pkgKey{} + + for _, op := range ops { + key := pkgKey{catalog: op.Catalog, name: op.Name} + pkg, ok := packages[key] + if !ok { + pkg = &OperatorPackage{Name: op.Name} + packages[key] = pkg + order = append(order, key) + } + if op.Channel != "" { + found := false + for _, ch := range pkg.Channels { + if ch.Name == op.Channel { + found = true + break + } + } + if !found { + pkg.Channels = append(pkg.Channels, OperatorChannel{Name: op.Channel}) + } + // oc-mirror requires defaultChannel when filtering channels. + if pkg.DefaultChannel == "" { + pkg.DefaultChannel = op.Channel + } + } + } + + byCatalog := map[CatalogKind][]OperatorPackage{} + catalogOrder := []CatalogKind{} + seenCat := map[CatalogKind]bool{} + for _, key := range order { + pkg := packages[key] + if !seenCat[key.catalog] { + catalogOrder = append(catalogOrder, key.catalog) + seenCat[key.catalog] = true + } + byCatalog[key.catalog] = append(byCatalog[key.catalog], *pkg) + } + + var out []OperatorCatalog + for _, kind := range catalogOrder { + out = append(out, OperatorCatalog{ + Catalog: CatalogImage(kind, ocpVersion), + Packages: byCatalog[kind], + }) + } + return out +} + +// WriteImageSet writes imagesetconfig.yaml to outputDir. +func WriteImageSet(isc *ImageSetConfiguration, outputDir string) error { + return WriteImageSetWithHeader(isc, outputDir, "") +} + +// WriteImageSetWithHeader writes imagesetconfig.yaml with an optional extra header note. +func WriteImageSetWithHeader(isc *ImageSetConfiguration, outputDir, extraNote string) error { + if err := os.MkdirAll(outputDir, 0o755); err != nil { + return err + } + data, err := yaml.Marshal(isc) + if err != nil { + return err + } + header := "# Generated by patternizer airgap init\n# apiVersion: mirror.openshift.io/v2alpha1\n" + if extraNote != "" { + header += "# " + extraNote + "\n" + } + return os.WriteFile(filepath.Join(outputDir, "imagesetconfig.yaml"), append([]byte(header), data...), 0o644) +} + +// BuildValuesOverlay creates a disconnected values overlay document. +func BuildValuesOverlay(inv *PatternInventory, opts *Options) map[string]interface{} { + catVer := catalogOCPVersion(opts) + redhatCS := CatalogSourceName(CatalogRedHat, catVer) + communityCS := CatalogSourceName(CatalogCommunity, catVer) + + multiSource := map[string]interface{}{ + "enabled": true, + "clusterGroupChartVersion": inv.ClusterGroupChartVer, + } + if reg := strings.TrimSpace(opts.Registry); reg != "" { + multiSource["helmRepoUrl"] = fmt.Sprintf("%s/validatedpatterns", reg) + } + + overlay := map[string]interface{}{ + "main": map[string]interface{}{ + "multiSourceConfig": multiSource, + "patternsOperator": map[string]interface{}{ + "source": communityCS, + }, + "gitops": map[string]interface{}{ + "operatorSource": redhatCS, + }, + }, + } + + subs := map[string]interface{}{} + for _, g := range inv.Groups { + for _, op := range g.Subscriptions { + key := sanitizeKey(op.Name) + entry := map[string]interface{}{ + "name": op.Name, + "source": CatalogSourceName(op.Catalog, catVer), + } + if op.Channel != "" { + entry["channel"] = op.Channel + } + subs[key] = entry + } + } + if len(subs) > 0 { + overlay["clusterGroup"] = map[string]interface{}{ + "subscriptions": subs, + } + } + + return overlay +} + +func sanitizeKey(name string) string { + return strings.ReplaceAll(name, "-", "_") +} + +// WriteValuesOverlay writes values-disconnected.yaml. +func WriteValuesOverlay(overlay map[string]interface{}, outputDir string) error { + if err := os.MkdirAll(outputDir, 0o755); err != nil { + return err + } + data, err := yaml.Marshal(overlay) + if err != nil { + return err + } + header := "# Generated by patternizer airgap init\n" + + "# Merge into values-global.yaml / values-hub.yaml or pass as an extra values file.\n" + + "# CatalogSource names must match those generated by oc-mirror; do not rename them.\n" + return os.WriteFile(filepath.Join(outputDir, "values-disconnected.yaml"), append([]byte(header), data...), 0o644) +} diff --git a/src/internal/airgap/imageset_test.go b/src/internal/airgap/imageset_test.go new file mode 100644 index 0000000..9f25375 --- /dev/null +++ b/src/internal/airgap/imageset_test.go @@ -0,0 +1,82 @@ +package airgap_test + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "gopkg.in/yaml.v3" + + "github.com/validatedpatterns/patternizer/internal/airgap" +) + +var _ = Describe("BuildImageSet operator channels", func() { + It("sets defaultChannel and channels from the values-defined channel", func() { + inv := &airgap.PatternInventory{ + Operators: []airgap.OperatorRef{ + { + Name: "advanced-cluster-management", + Channel: "release-2.16", + Catalog: airgap.CatalogRedHat, + }, + { + Name: "patterns-operator", + Catalog: airgap.CatalogCommunity, + }, + }, + } + opts := airgap.Options{} + isc := airgap.BuildImageSet(inv, &opts) + Expect(isc.Mirror.Platform).To(BeNil()) + Expect(isc.Mirror.Operators).NotTo(BeEmpty()) + + var acm *airgap.OperatorPackage + for i := range isc.Mirror.Operators { + for j := range isc.Mirror.Operators[i].Packages { + pkg := &isc.Mirror.Operators[i].Packages[j] + if pkg.Name == "advanced-cluster-management" { + acm = pkg + } + if pkg.Name == "patterns-operator" { + Expect(pkg.DefaultChannel).To(BeEmpty()) + Expect(pkg.Channels).To(BeEmpty()) + } + } + } + Expect(acm).NotTo(BeNil()) + Expect(acm.DefaultChannel).To(Equal("release-2.16")) + Expect(acm.Channels).To(HaveLen(1)) + Expect(acm.Channels[0].Name).To(Equal("release-2.16")) + + raw, err := yaml.Marshal(isc) + Expect(err).NotTo(HaveOccurred()) + Expect(string(raw)).To(ContainSubstring("defaultChannel: release-2.16")) + }) + + It("includes platform channel and catalog tags when OCPVersion is set", func() { + inv := &airgap.PatternInventory{ + Operators: []airgap.OperatorRef{ + {Name: "openshift-gitops-operator", Catalog: airgap.CatalogRedHat}, + }, + } + opts := airgap.Options{OCPVersion: "4.18"} + isc := airgap.BuildImageSet(inv, &opts) + Expect(isc.Mirror.Platform).NotTo(BeNil()) + Expect(isc.Mirror.Platform.Channels).To(HaveLen(1)) + Expect(isc.Mirror.Platform.Channels[0].Name).To(Equal("stable-4.18")) + Expect(isc.Mirror.Platform.Channels[0].Type).To(Equal("ocp")) + Expect(isc.Mirror.Operators).To(HaveLen(1)) + Expect(isc.Mirror.Operators[0].Catalog).To(Equal("registry.redhat.io/redhat/redhat-operator-index:v4.18")) + }) + + It("defaults OLM catalog tags when OCPVersion is omitted", func() { + inv := &airgap.PatternInventory{ + Operators: []airgap.OperatorRef{ + {Name: "openshift-gitops-operator", Catalog: airgap.CatalogRedHat}, + }, + } + isc := airgap.BuildImageSet(inv, &airgap.Options{}) + Expect(isc.Mirror.Platform).To(BeNil()) + Expect(isc.Mirror.Operators).To(HaveLen(1)) + Expect(isc.Mirror.Operators[0].Catalog).To(Equal("registry.redhat.io/redhat/redhat-operator-index:v4.22")) + }) +}) diff --git a/src/internal/airgap/init.go b/src/internal/airgap/init.go new file mode 100644 index 0000000..25bd281 --- /dev/null +++ b/src/internal/airgap/init.go @@ -0,0 +1,267 @@ +package airgap + +import ( + "fmt" + "os" + "path/filepath" + "sort" + "strings" +) + +// Init generates airgap artifacts for a pattern repository. +func Init(opts *Options) (*PatternInventory, error) { + if opts == nil { + return nil, fmt.Errorf("options are required") + } + sourceInput := opts.PatternSource + if sourceInput == "" { + sourceInput = opts.PatternDir + } + + if opts.OutputDir == "" { + opts.OutputDir = "./airgap-out" + } + outAbs, err := filepath.Abs(opts.OutputDir) + if err != nil { + return nil, err + } + opts.OutputDir = outAbs + + if opts.CloneDir == "" && IsGitURL(sourceInput) { + opts.CloneDir = opts.OutputDir + } + + src, err := ResolvePatternSource(sourceInput, opts) + if err != nil { + return nil, err + } + opts.PatternDir = src.Dir + + if opts.HelmRepo == "" { + opts.HelmRepo = "quay.io/validatedpatterns" + } + + inv, err := LoadPattern(opts) + if err != nil { + return nil, err + } + inv.ResolvedDir = src.Dir + inv.OutputDir = opts.OutputDir + inv.Warnings = append(inv.Warnings, opts.ExtraWarnings...) + if src.FromGit { + inv.Warnings = append(inv.Warnings, + fmt.Sprintf("pattern cloned from %s into %s", src.GitURL, src.Dir)) + } + + if err := enrichInventory(inv, opts); err != nil { + return inv, err + } + + if opts.ExtraConfigPath != "" { + extra, err := LoadExtraConfig(opts.ExtraConfigPath) + if err != nil { + return inv, err + } + absExtra, err := filepath.Abs(opts.ExtraConfigPath) + if err != nil { + return inv, fmt.Errorf("resolve extra config path: %w", err) + } + opts.ExtraConfigPath = absExtra + if err := ApplyExtraConfig(inv, extra, opts); err != nil { + return inv, err + } + } + + if err := os.MkdirAll(opts.OutputDir, 0o755); err != nil { + return inv, err + } + + if opts.DownloadOCMirror { + if err := DownloadOCMirrorTools(opts.OCPVersion, opts.OutputDir); err != nil { + return inv, fmt.Errorf("download oc-mirror: %w", err) + } + } + + isc := BuildImageSet(inv, opts) + // Ensure extra-config operators/images/helm.local land in the oc-mirror input. + MergeExtraConfigIntoImageSet(isc, inv.ExtraConfig) + headerNote := "" + if opts.ExtraConfigPath != "" { + headerNote = "Merged --extra-config: " + opts.ExtraConfigPath + } + if err := WriteImageSetWithHeader(isc, opts.OutputDir, headerNote); err != nil { + return inv, fmt.Errorf("write imagesetconfig: %w", err) + } + + if opts.WriteValues { + overlay := BuildValuesOverlay(inv, opts) + if err := WriteValuesOverlay(overlay, opts.OutputDir); err != nil { + return inv, fmt.Errorf("write values overlay: %w", err) + } + } + + if err := WriteMirrorCommands(opts, opts.OutputDir); err != nil { + return inv, fmt.Errorf("write mirror commands: %w", err) + } + if err := WriteChecklist(inv, opts, opts.OutputDir); err != nil { + return inv, fmt.Errorf("write checklist: %w", err) + } + + return inv, nil +} + +func enrichInventory(inv *PatternInventory, opts *Options) error { + opIndex := map[string]*OperatorRef{} + + addOp := func(op OperatorRef) { + key := op.Name + if existing, ok := opIndex[key]; ok { + if existing.Channel == "" && op.Channel != "" { + existing.Channel = op.Channel + } + if existing.Catalog == "" { + existing.Catalog = op.Catalog + } + return + } + cp := op + opIndex[key] = &cp + } + + for _, op := range FrameworkOperators() { + addOp(op) + } + + hasACM := false + for _, g := range inv.Groups { + for _, op := range g.Subscriptions { + addOp(op) + if op.Name == "advanced-cluster-management" { + hasACM = true + } + } + if g.ImperativeImage != "" { + inv.AdditionalImages = appendUnique(inv.AdditionalImages, g.ImperativeImage) + } + } + if hasACM { + for _, op := range ACMCompanionOperators() { + addOp(op) + } + } + + cgConstraint := inv.ClusterGroupChartVer + inv.RemoteCharts = append(inv.RemoteCharts, + resolveRemoteChart(inv, opts, "clustergroup", cgConstraint), + resolveRemoteChart(inv, opts, "pattern-install", "*"), + ) + + seenRemote := map[string]bool{"clustergroup": true, "pattern-install": true} + seenLocalPaths := map[string]bool{} + + for _, g := range inv.Groups { + for _, app := range g.Applications { + if app.Chart != "" { + if seenRemote[app.Chart] { + continue + } + seenRemote[app.Chart] = true + constraint := app.ChartVersion + if constraint == "" { + constraint = "*" + } + inv.RemoteCharts = append(inv.RemoteCharts, resolveRemoteChart(inv, opts, app.Chart, constraint)) + continue + } + if app.Path != "" { + // Images only — do not add pattern path charts to helm.local. + discoverLocalChartImages(inv, opts, app.Path, seenLocalPaths) + } + } + } + + for _, img := range FrameworkImages(opts.HelmRepo) { + inv.AdditionalImages = appendUnique(inv.AdditionalImages, img) + } + + keys := make([]string, 0, len(opIndex)) + for k := range opIndex { + keys = append(keys, k) + } + sort.Strings(keys) + for _, k := range keys { + inv.Operators = append(inv.Operators, *opIndex[k]) + } + + sort.Slice(inv.RemoteCharts, func(i, j int) bool { + return inv.RemoteCharts[i].Name < inv.RemoteCharts[j].Name + }) + sort.Slice(inv.LocalCharts, func(i, j int) bool { + return inv.LocalCharts[i].Path < inv.LocalCharts[j].Path + }) + sort.Strings(inv.AdditionalImages) + + return nil +} + +func resolveRemoteChart(inv *PatternInventory, opts *Options, chartName, constraint string) RemoteChart { + rc := RemoteChart{ + Name: chartName, + VersionConstraint: constraint, + } + ver, err := ResolveOCIChartVersion(opts.HelmRepo, chartName, constraint, opts.SkipOCIResolve) + if err != nil { + inv.Warnings = append(inv.Warnings, fmt.Sprintf( + "chart %s: failed to resolve %q (%v); not adding to additionalImages", + chartName, constraint, err)) + return rc + } + rc.ResolvedVersion = ver + rc.PullSpec = fmt.Sprintf("%s/%s:%s", opts.HelmRepo, chartName, ver) + + imgs, warns, discErr := DiscoverOCIChartImages(opts.HelmRepo, chartName, ver, opts.SkipOCIResolve) + inv.Warnings = append(inv.Warnings, warns...) + if discErr != nil { + inv.Warnings = append(inv.Warnings, fmt.Sprintf("chart %s image discovery: %v", chartName, discErr)) + } + rc.Images = imgs + for _, img := range imgs { + inv.AdditionalImages = appendUnique(inv.AdditionalImages, img) + } + return rc +} + +// discoverLocalChartImages finds container images in a path-based application chart. +// Images are added to additionalImages only; the chart itself is not mirrored via helm.local. +func discoverLocalChartImages(inv *PatternInventory, opts *Options, relPath string, seen map[string]bool) { + chartPath := strings.TrimSpace(relPath) + if chartPath == "" { + return + } + if !filepath.IsAbs(chartPath) { + chartPath = filepath.Join(opts.PatternDir, chartPath) + } + absPath, err := filepath.Abs(chartPath) + if err != nil { + inv.Warnings = append(inv.Warnings, fmt.Sprintf("local chart %q: resolve path: %v", relPath, err)) + return + } + if seen[absPath] { + return + } + if _, err := os.Stat(filepath.Join(absPath, "Chart.yaml")); err != nil { + inv.Warnings = append(inv.Warnings, fmt.Sprintf( + "local chart %q: missing Chart.yaml under %s (%v)", relPath, absPath, err)) + return + } + seen[absPath] = true + + imgs, warnings, discErr := DiscoverImagesFromChartDir(absPath) + inv.Warnings = append(inv.Warnings, warnings...) + if discErr != nil { + inv.Warnings = append(inv.Warnings, fmt.Sprintf("local chart %s image discovery: %v", absPath, discErr)) + } + for _, img := range imgs { + inv.AdditionalImages = appendUnique(inv.AdditionalImages, img) + } +} diff --git a/src/internal/airgap/init_test.go b/src/internal/airgap/init_test.go new file mode 100644 index 0000000..e2966c6 --- /dev/null +++ b/src/internal/airgap/init_test.go @@ -0,0 +1,259 @@ +package airgap_test + +import ( + "os" + "path/filepath" + "strings" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "gopkg.in/yaml.v3" + + "github.com/validatedpatterns/patternizer/internal/airgap" +) + +var _ = Describe("Airgap Init", func() { + var tmpDir string + var patternDir string + + BeforeEach(func() { + var err error + tmpDir, err = os.MkdirTemp("", "airgap-test-") + Expect(err).NotTo(HaveOccurred()) + patternDir = filepath.Join(tmpDir, "pattern") + Expect(os.MkdirAll(filepath.Join(patternDir, "charts", "all", "hello-world", "templates"), 0o755)).To(Succeed()) + + Expect(os.WriteFile(filepath.Join(patternDir, "values-global.yaml"), []byte(` +global: + pattern: test-pattern +main: + clusterGroupName: hub + multiSourceConfig: + enabled: true + clusterGroupChartVersion: "0.9.41" +`), 0o644)).To(Succeed()) + + Expect(os.WriteFile(filepath.Join(patternDir, "values-hub.yaml"), []byte(` +clusterGroup: + name: hub + namespaces: + hello-world: + subscriptions: + acm: + name: advanced-cluster-management + namespace: open-cluster-management + channel: release-2.14 + applications: + vault: + name: vault + namespace: vault + chart: hashicorp-vault + chartVersion: 0.1.6 + hello-world: + name: hello-world + namespace: hello-world + path: charts/all/hello-world + managedClusterGroups: + exampleRegion: + name: group-one +`), 0o644)).To(Succeed()) + + Expect(os.WriteFile(filepath.Join(patternDir, "charts", "all", "hello-world", "Chart.yaml"), []byte(` +apiVersion: v2 +name: hello-world +version: 0.1.0 +`), 0o644)).To(Succeed()) + + Expect(os.WriteFile(filepath.Join(patternDir, "charts", "all", "hello-world", "values.yaml"), []byte(` +image: + repository: registry.access.redhat.com/ubi9/httpd-24 + tag: "1.0" +`), 0o644)).To(Succeed()) + + Expect(os.WriteFile(filepath.Join(patternDir, "charts", "all", "hello-world", "templates", "deploy.yaml"), []byte(` +apiVersion: apps/v1 +kind: Deployment +metadata: + name: hello +spec: + template: + spec: + containers: + - name: httpd + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" +`), 0o644)).To(Succeed()) + }) + + AfterEach(func() { + _ = os.RemoveAll(tmpDir) + }) + + It("generates imagesetconfig and init artifacts", func() { + outDir := filepath.Join(tmpDir, "out") + opts := airgap.Options{ + PatternDir: patternDir, + OutputDir: outDir, + HelmRepo: "quay.io/validatedpatterns", + WriteValues: true, + SkipOCIResolve: true, + } + inv, err := airgap.Init(&opts) + Expect(err).NotTo(HaveOccurred()) + Expect(inv.PatternName).To(Equal("test-pattern")) + Expect(inv.ResolvedDir).To(Equal(patternDir)) + Expect(inv.Operators).NotTo(BeEmpty()) + + names := map[string]bool{} + for _, op := range inv.Operators { + names[op.Name] = true + } + Expect(names["advanced-cluster-management"]).To(BeTrue()) + Expect(names["patterns-operator"]).To(BeTrue()) + Expect(names["openshift-gitops-operator"]).To(BeTrue()) + Expect(names["multicluster-engine"]).To(BeTrue()) + + Expect(filepath.Join(outDir, "imagesetconfig.yaml")).To(BeAnExistingFile()) + Expect(filepath.Join(outDir, "values-disconnected.yaml")).To(BeAnExistingFile()) + Expect(filepath.Join(outDir, "mirror-commands.sh")).To(BeAnExistingFile()) + Expect(filepath.Join(outDir, "CHECKLIST.md")).To(BeAnExistingFile()) + + raw, err := os.ReadFile(filepath.Join(outDir, "imagesetconfig.yaml")) + Expect(err).NotTo(HaveOccurred()) + var doc map[string]interface{} + Expect(yaml.Unmarshal(raw, &doc)).To(Succeed()) + Expect(doc["kind"]).To(Equal("ImageSetConfiguration")) + Expect(doc["apiVersion"]).To(Equal("mirror.openshift.io/v2alpha1")) + + Expect(string(raw)).To(ContainSubstring("name: advanced-cluster-management")) + Expect(string(raw)).To(ContainSubstring("defaultChannel: release-2.14")) + Expect(string(raw)).To(ContainSubstring("name: release-2.14")) + Expect(string(raw)).NotTo(ContainSubstring("platform:")) + Expect(string(raw)).NotTo(ContainSubstring("stable-")) + + Expect(inv.LocalCharts).To(BeEmpty()) + Expect(inv.AdditionalImages).To(ContainElement("registry.access.redhat.com/ubi9/httpd-24:1.0")) + Expect(string(raw)).NotTo(ContainSubstring("helm:")) + Expect(string(raw)).NotTo(ContainSubstring("charts/all/hello-world")) + Expect(string(raw)).To(ContainSubstring("registry.access.redhat.com/ubi9/httpd-24:1.0")) + + checklist, err := os.ReadFile(filepath.Join(outDir, "CHECKLIST.md")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(checklist)).To(ContainSubstring("Vault + External Secrets")) + Expect(string(checklist)).To(ContainSubstring("PATTERN_DISCONNECTED_HOME")) + Expect(string(checklist)).To(ContainSubstring("group-one")) + + vals, err := os.ReadFile(filepath.Join(outDir, "values-disconnected.yaml")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(vals)).To(ContainSubstring("cs-community-operator-index-v4-22")) + Expect(string(vals)).NotTo(ContainSubstring("helmRepoUrl")) + }) + + It("adds OpenShift platform channel only when --ocp-version is set", func() { + outDir := filepath.Join(tmpDir, "out-platform") + opts := airgap.Options{ + PatternDir: patternDir, + OutputDir: outDir, + OCPVersion: "4.16", + SkipOCIResolve: true, + WriteValues: false, + } + _, err := airgap.Init(&opts) + Expect(err).NotTo(HaveOccurred()) + + raw, err := os.ReadFile(filepath.Join(outDir, "imagesetconfig.yaml")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(raw)).To(ContainSubstring("stable-4.16")) + Expect(string(raw)).To(ContainSubstring("type: ocp")) + Expect(string(raw)).To(ContainSubstring("redhat-operator-index:v4.16")) + Expect(string(raw)).NotTo(ContainSubstring("redhat-operator-index:v4.22")) + }) + + It("includes helmRepoUrl in values overlay when --registry is set", func() { + outDir := filepath.Join(tmpDir, "out-registry") + opts := airgap.Options{ + PatternDir: patternDir, + OutputDir: outDir, + Registry: "mirror.example.com:8443", + SkipOCIResolve: true, + WriteValues: true, + } + _, err := airgap.Init(&opts) + Expect(err).NotTo(HaveOccurred()) + + vals, err := os.ReadFile(filepath.Join(outDir, "values-disconnected.yaml")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(vals)).To(ContainSubstring("helmRepoUrl: mirror.example.com:8443/validatedpatterns")) + }) + + It("merges --extra-config into imagesetconfig", func() { + chartDir := filepath.Join(tmpDir, "external-chart") + Expect(os.MkdirAll(filepath.Join(chartDir, "templates"), 0o755)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(chartDir, "Chart.yaml"), []byte(` +apiVersion: v2 +name: external +version: 0.1.0 +`), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(chartDir, "values.yaml"), []byte(` +image: quay.io/example/external:9.9.9 +`), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(chartDir, "templates", "deploy.yaml"), []byte(` +apiVersion: apps/v1 +kind: Deployment +metadata: + name: external +spec: + template: + spec: + containers: + - name: app + image: "{{ .Values.image }}" +`), 0o644)).To(Succeed()) + + extraPath := filepath.Join(tmpDir, "extra-images.yaml") + Expect(os.WriteFile(extraPath, []byte(` +operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.22 + packages: + - name: lvms-operator +additionalImages: + - name: registry.example.com/addtl:1 +helm: + local: + - name: external + path: `+chartDir+` +`), 0o644)).To(Succeed()) + + outDir := filepath.Join(tmpDir, "out-extra") + opts := airgap.Options{ + PatternDir: patternDir, + OutputDir: outDir, + HelmRepo: "quay.io/validatedpatterns", + WriteValues: true, + SkipOCIResolve: true, + ExtraConfigPath: extraPath, + } + inv, err := airgap.Init(&opts) + Expect(err).NotTo(HaveOccurred()) + Expect(inv.LocalCharts).To(HaveLen(1)) + Expect(inv.LocalCharts[0].Path).To(Equal(chartDir)) + Expect(inv.AdditionalImages).To(ContainElement("registry.example.com/addtl:1")) + Expect(inv.AdditionalImages).To(ContainElement("quay.io/example/external:9.9.9")) + Expect(inv.AdditionalImages).To(ContainElement("registry.access.redhat.com/ubi9/httpd-24:1.0")) + + raw, err := os.ReadFile(filepath.Join(outDir, "imagesetconfig.yaml")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(raw)).To(ContainSubstring("lvms-operator")) + Expect(string(raw)).To(ContainSubstring("path: " + chartDir)) + Expect(string(raw)).NotTo(ContainSubstring("charts/all/hello-world")) + Expect(string(raw)).To(ContainSubstring("registry.example.com/addtl:1")) + Expect(string(raw)).To(ContainSubstring("registry.access.redhat.com/ubi9/httpd-24:1.0")) + }) + + It("maps catalogs correctly", func() { + Expect(airgap.LookupCatalog("patterns-operator")).To(Equal(airgap.CatalogCommunity)) + Expect(airgap.LookupCatalog("advanced-cluster-management")).To(Equal(airgap.CatalogRedHat)) + Expect(airgap.CatalogSourceName(airgap.CatalogRedHat, "4.22")).To(Equal("cs-redhat-operator-index-v4-22")) + Expect(strings.Contains(airgap.CatalogImage(airgap.CatalogCommunity, "4.22"), "community-operator-index")).To(BeTrue()) + }) +}) diff --git a/src/internal/airgap/mirrorlog.go b/src/internal/airgap/mirrorlog.go new file mode 100644 index 0000000..3185955 --- /dev/null +++ b/src/internal/airgap/mirrorlog.go @@ -0,0 +1,77 @@ +package airgap + +import ( + "bufio" + "fmt" + "os" + "regexp" + "sort" + "strings" +) + +var ( + ociSkipRe = regexp.MustCompile(`(?i)(?:SKIPPING|skipped).*?(oci://[^\s'"]+)`) + ociURLRe = regexp.MustCompile(`oci://[^\s'"]+`) +) + +// ParseMirrorLog returns OCI-related images skipped in an oc-mirror log. +func ParseMirrorLog(path string) ([]string, error) { + f, err := os.Open(path) + if err != nil { + return nil, err + } + defer f.Close() + + found := map[string]struct{}{} + scanner := bufio.NewScanner(f) + buf := make([]byte, 0, 64*1024) + scanner.Buffer(buf, 1024*1024) + + for scanner.Scan() { + line := scanner.Text() + if !strings.Contains(strings.ToLower(line), "oci") { + continue + } + if !strings.Contains(strings.ToLower(line), "skip") && !strings.Contains(line, "not supported") { + continue + } + for _, m := range ociURLRe.FindAllString(line, -1) { + img := strings.TrimPrefix(m, "oci://") + img = strings.TrimRight(img, ".,;:'\")") + if img != "" { + found[img] = struct{}{} + } + } + if sm := ociSkipRe.FindStringSubmatch(line); len(sm) > 1 { + img := strings.TrimPrefix(sm[1], "oci://") + img = strings.TrimRight(img, ".,;:'\")") + if img != "" { + found[img] = struct{}{} + } + } + } + if err := scanner.Err(); err != nil { + return nil, err + } + + out := make([]string, 0, len(found)) + for img := range found { + out = append(out, img) + } + sort.Strings(out) + return out, nil +} + +// FormatSkippedImagesYAML returns an additionalImages YAML snippet for skipped OCI images. +func FormatSkippedImagesYAML(images []string) string { + if len(images) == 0 { + return "# No skipped oci:// related images found in the log.\n" + } + var b strings.Builder + b.WriteString("# Add these under mirror.additionalImages in imagesetconfig.yaml\n") + b.WriteString("additionalImages:\n") + for _, img := range images { + fmt.Fprintf(&b, " - name: %s\n", img) + } + return b.String() +} diff --git a/src/internal/airgap/mirrorlog_test.go b/src/internal/airgap/mirrorlog_test.go new file mode 100644 index 0000000..8c6300e --- /dev/null +++ b/src/internal/airgap/mirrorlog_test.go @@ -0,0 +1,102 @@ +package airgap_test + +import ( + "os" + "path/filepath" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/validatedpatterns/patternizer/internal/airgap" +) + +var _ = Describe("ParseMirrorLog", func() { + It("extracts skipped oci:// related images", func() { + tmp, err := os.MkdirTemp("", "mirrorlog-") + Expect(err).NotTo(HaveOccurred()) + defer os.RemoveAll(tmp) + + logPath := filepath.Join(tmp, "oc-mirror.log") + Expect(os.WriteFile(logPath, []byte(` +INFO copying operators +WARN 'oci' is not supported in operator catalogs SKIPPING oci://quay.io/kuadrant/wasm-shim:v1.0.0 +ERROR related image skipped: oci://ghcr.io/example/sidecar:2.3.4 is not supported +INFO done +`), 0o644)).To(Succeed()) + + images, err := airgap.ParseMirrorLog(logPath) + Expect(err).NotTo(HaveOccurred()) + Expect(images).To(ContainElement("quay.io/kuadrant/wasm-shim:v1.0.0")) + Expect(images).To(ContainElement("ghcr.io/example/sidecar:2.3.4")) + + snippet := airgap.FormatSkippedImagesYAML(images) + Expect(snippet).To(ContainSubstring("additionalImages:")) + Expect(snippet).To(ContainSubstring("quay.io/kuadrant/wasm-shim:v1.0.0")) + }) +}) + +var _ = Describe("Spoke expansion", func() { + It("auto-includes managedClusterGroup values when filtering to hub", func() { + tmp, err := os.MkdirTemp("", "spoke-") + Expect(err).NotTo(HaveOccurred()) + defer os.RemoveAll(tmp) + + Expect(os.WriteFile(filepath.Join(tmp, "values-global.yaml"), []byte(` +global: + pattern: spoke-test +main: + clusterGroupName: hub + multiSourceConfig: + enabled: true + clusterGroupChartVersion: "0.9.1" +`), 0o644)).To(Succeed()) + + Expect(os.WriteFile(filepath.Join(tmp, "values-hub.yaml"), []byte(` +clusterGroup: + name: hub + namespaces: {} + subscriptions: {} + applications: {} + managedClusterGroups: + region: + name: group-one +`), 0o644)).To(Succeed()) + + Expect(os.WriteFile(filepath.Join(tmp, "values-group-one.yaml"), []byte(` +clusterGroup: + name: group-one + namespaces: + hello: + subscriptions: {} + applications: + demo: + name: demo + chart: golang-external-secrets + chartVersion: 0.1.5 +`), 0o644)).To(Succeed()) + + out := filepath.Join(tmp, "out") + opts := airgap.Options{ + PatternDir: tmp, + OutputDir: out, + ClusterGroups: []string{"hub"}, + SkipOCIResolve: true, + WriteValues: true, + } + inv, err := airgap.Init(&opts) + Expect(err).NotTo(HaveOccurred()) + + groupNames := map[string]bool{} + for _, g := range inv.Groups { + groupNames[g.Name] = true + } + Expect(groupNames["hub"]).To(BeTrue()) + Expect(groupNames["group-one"]).To(BeTrue()) + + chartNames := map[string]bool{} + for _, rc := range inv.RemoteCharts { + chartNames[rc.Name] = true + } + Expect(chartNames["golang-external-secrets"]).To(BeTrue()) + }) +}) diff --git a/src/internal/airgap/options.go b/src/internal/airgap/options.go new file mode 100644 index 0000000..46192e1 --- /dev/null +++ b/src/internal/airgap/options.go @@ -0,0 +1,35 @@ +package airgap + +// Options configures airgap init. +type Options struct { + PatternSource string + PatternDir string + OutputDir string + // OCPVersion, when set, adds mirror.platform with channel stable- + // and tags pattern-discovered OLM catalog indexes / CatalogSource names. + OCPVersion string + // Registry is optional; used in values-disconnected.yaml and mirror-commands.sh when set. + Registry string + ClusterGroups []string + WriteValues bool + HelmRepo string + SkipOCIResolve bool + GitRef string + CloneDir string + DownloadOCMirror bool + ExtraConfigPath string + ExtraWarnings []string +} + +// defaultOLMCatalogVersion tags pattern-discovered OLM catalog indexes when +// --ocp-version is omitted. +const defaultOLMCatalogVersion = "4.22" + +// DefaultOptions returns default airgap init options. +func DefaultOptions() Options { + return Options{ + OutputDir: "./airgap-out", + HelmRepo: "quay.io/validatedpatterns", + WriteValues: true, + } +} diff --git a/src/internal/airgap/rendervalues.go b/src/internal/airgap/rendervalues.go new file mode 100644 index 0000000..fd78a49 --- /dev/null +++ b/src/internal/airgap/rendervalues.go @@ -0,0 +1,373 @@ +package airgap + +import ( + "encoding/json" + "fmt" + "os" + "os/exec" + "path/filepath" + "sort" + "strings" + + "gopkg.in/yaml.v3" +) + +// DisconnectedValuesConfig drives rendering of pattern values for airgap. +type DisconnectedValuesConfig struct { + HelmRepoURL string + RedHatCS string + CommunityCS string + Marketplace string +} + +// ResolveDisconnectedValues builds helm repo + CatalogSource names from airgap spec / flags / oc. +func ResolveDisconnectedValues(spec *AirgapSpec, opts *BootstrapOptions) (*DisconnectedValuesConfig, error) { + if spec == nil { + spec = &AirgapSpec{} + } + + registry := strings.TrimSpace(opts.Registry) + if registry == "" { + registry = strings.TrimSpace(spec.Registry) + } + helmURL := strings.TrimSpace(spec.HelmRepoURL) + if helmURL == "" && registry != "" { + helmURL = strings.TrimSuffix(registry, "/") + "/validatedpatterns" + } + // Also accept registry from first registry url host if still empty. + if helmURL == "" && len(spec.Registries) > 0 { + fields, _ := resolveFieldMap(spec.Registries[0].Fields) + if u := NormalizeRegistryURL(fields["url"]); u != "" { + host := u + if i := strings.Index(u, "/"); i >= 0 { + host = u[:i] + } + helmURL = host + "/validatedpatterns" + } + } + + ocp := strings.TrimSpace(opts.OCPVersion) + if ocp == "" { + ocp = strings.TrimSpace(spec.OCPVersion) + } + + cs := map[string]string{} + for k, v := range spec.CatalogSources { + cs[strings.ToLower(strings.TrimSpace(k))] = strings.TrimSpace(v) + } + + discover := opts.DiscoverCatalogs || spec.DiscoverCatalogs + if discover { + found, err := DiscoverCatalogSources() + if err != nil { + return nil, fmt.Errorf("discover CatalogSources: %w", err) + } + for k, v := range found { + if cs[k] == "" { + cs[k] = v + } + } + } + + if cs["redhat"] == "" && ocp != "" { + cs["redhat"] = CatalogSourceName(CatalogRedHat, ocp) + } + if cs["community"] == "" && ocp != "" { + cs["community"] = CatalogSourceName(CatalogCommunity, ocp) + } + if cs["marketplace"] == "" && ocp != "" { + cs["marketplace"] = CatalogSourceName(CatalogMarketplace, ocp) + } + + cfg := &DisconnectedValuesConfig{ + HelmRepoURL: helmURL, + RedHatCS: cs["redhat"], + CommunityCS: cs["community"], + Marketplace: cs["marketplace"], + } + if cfg.HelmRepoURL == "" && cfg.RedHatCS == "" && cfg.CommunityCS == "" { + return nil, fmt.Errorf("cannot render values: set airgap.registry (or helmRepoUrl), and airgap.ocpVersion or airgap.catalogSources (or --discover-catalogs)") + } + return cfg, nil +} + +// DiscoverCatalogSources runs oc to find redhat/community/marketplace CatalogSource names. +func DiscoverCatalogSources() (map[string]string, error) { + if _, err := exec.LookPath("oc"); err != nil { + return nil, fmt.Errorf("oc not found on PATH") + } + cmd := exec.Command("oc", "get", "catalogsource", "-A", "-o", "json") + out, err := cmd.CombinedOutput() + if err != nil { + return nil, fmt.Errorf("oc get catalogsource: %w (%s)", err, truncate(string(out), 300)) + } + var list struct { + Items []struct { + Metadata struct { + Name string `json:"name"` + } `json:"metadata"` + } `json:"items"` + } + if err := json.Unmarshal(out, &list); err != nil { + return nil, fmt.Errorf("parse catalogsource JSON: %w", err) + } + + outMap := map[string]string{} + pick := func(kind, prefix string) { + if outMap[kind] != "" { + return + } + best := "" + for _, it := range list.Items { + name := it.Metadata.Name + if strings.HasPrefix(name, prefix) { + if best == "" || name > best { + best = name + } + } + } + if best != "" { + outMap[kind] = best + } + } + pick("redhat", "cs-redhat-operator-index-") + pick("community", "cs-community-operator-index-") + pick("marketplace", "cs-redhat-marketplace-index-") + if len(outMap) == 0 { + return nil, fmt.Errorf("no oc-mirror style CatalogSources found (expected cs-redhat-operator-index-* / cs-community-operator-index-*)") + } + return outMap, nil +} + +// RenderPatternValues merges disconnected settings into values-global.yaml and cluster-group values files. +// Originals are backed up under outDir/backup/values/. +func RenderPatternValues(patternDir, outDir string, cfg *DisconnectedValuesConfig) ([]string, error) { + if patternDir == "" { + return nil, fmt.Errorf("pattern dir is required to render values") + } + if cfg == nil { + return nil, fmt.Errorf("disconnected values config is required") + } + + backupRoot := filepath.Join(outDir, "backup", "values") + renderedRoot := filepath.Join(outDir, "rendered") + if err := os.MkdirAll(backupRoot, 0o755); err != nil { + return nil, err + } + if err := os.MkdirAll(renderedRoot, 0o755); err != nil { + return nil, err + } + + var written []string + + globalPath := filepath.Join(patternDir, "values-global.yaml") + if _, err := os.Stat(globalPath); err == nil { + if err := renderOneValuesFile(globalPath, backupRoot, renderedRoot, patternDir, cfg, true); err != nil { + return written, err + } + written = append(written, globalPath) + } + + files, err := listClusterGroupValuesFiles(patternDir) + if err != nil { + return written, err + } + for _, f := range files { + if err := renderOneValuesFile(f, backupRoot, renderedRoot, patternDir, cfg, false); err != nil { + return written, fmt.Errorf("%s: %w", f, err) + } + written = append(written, f) + } + return written, nil +} + +func listClusterGroupValuesFiles(patternDir string) ([]string, error) { + var files []string + seen := map[string]bool{} + + add := func(path string) { + if seen[path] { + return + } + seen[path] = true + files = append(files, path) + } + + // Classic layout at repo root. + entries, err := os.ReadDir(patternDir) + if err != nil { + return nil, err + } + for _, e := range entries { + if e.IsDir() { + continue + } + name := e.Name() + if !strings.HasPrefix(name, "values-") || !strings.HasSuffix(name, ".yaml") { + continue + } + base := strings.TrimSuffix(strings.TrimPrefix(name, "values-"), ".yaml") + switch base { + case "global", "secret", "devel": + continue + } + if strings.Contains(base, "secret") || strings.HasSuffix(base, ".template") { + continue + } + add(filepath.Join(patternDir, name)) + } + + // variants//values-*.yaml + variantsRoot := filepath.Join(patternDir, "variants") + vEntries, err := os.ReadDir(variantsRoot) + if err == nil { + for _, ve := range vEntries { + if !ve.IsDir() { + continue + } + vdir := filepath.Join(variantsRoot, ve.Name()) + filesIn, err := os.ReadDir(vdir) + if err != nil { + continue + } + for _, e := range filesIn { + if e.IsDir() { + continue + } + name := e.Name() + if !strings.HasPrefix(name, "values-") || !strings.HasSuffix(name, ".yaml") { + continue + } + base := strings.TrimSuffix(strings.TrimPrefix(name, "values-"), ".yaml") + if base == "global" || strings.Contains(base, "secret") { + continue + } + add(filepath.Join(vdir, name)) + } + } + } + + sort.Strings(files) + return files, nil +} + +func renderOneValuesFile(path, backupRoot, renderedRoot, patternDir string, cfg *DisconnectedValuesConfig, isGlobal bool) error { + raw, err := os.ReadFile(path) + if err != nil { + return err + } + + rel, err := filepath.Rel(patternDir, path) + if err != nil { + rel = filepath.Base(path) + } + bakPath := filepath.Join(backupRoot, rel) + if err := os.MkdirAll(filepath.Dir(bakPath), 0o755); err != nil { + return err + } + if _, err := os.Stat(bakPath); err != nil { + // First backup only. + if err := os.WriteFile(bakPath, raw, 0o644); err != nil { + return err + } + } + + var doc map[string]interface{} + if err := yaml.Unmarshal(raw, &doc); err != nil { + return err + } + if doc == nil { + doc = map[string]interface{}{} + } + + if isGlobal { + applyGlobalDisconnected(doc, cfg) + } else { + applyClusterGroupDisconnected(doc, cfg) + } + + out, err := yaml.Marshal(doc) + if err != nil { + return err + } + header := "# Updated by patternizer airgap bootstrap (disconnected CatalogSources / helmRepoUrl)\n" + content := append([]byte(header), out...) + + rendPath := filepath.Join(renderedRoot, rel) + if err := os.MkdirAll(filepath.Dir(rendPath), 0o755); err != nil { + return err + } + if err := os.WriteFile(rendPath, content, 0o644); err != nil { + return err + } + return os.WriteFile(path, content, 0o644) +} + +func applyGlobalDisconnected(doc map[string]interface{}, cfg *DisconnectedValuesConfig) { + main := mapFrom(doc, "main") + msc := mapFrom(main, "multiSourceConfig") + msc["enabled"] = true + if cfg.HelmRepoURL != "" { + msc["helmRepoUrl"] = cfg.HelmRepoURL + } + if cfg.CommunityCS != "" { + po := mapFrom(main, "patternsOperator") + po["source"] = cfg.CommunityCS + } + if cfg.RedHatCS != "" { + gitops := mapFrom(main, "gitops") + gitops["operatorSource"] = cfg.RedHatCS + } +} + +func applyClusterGroupDisconnected(doc map[string]interface{}, cfg *DisconnectedValuesConfig) { + if cfg.RedHatCS != "" { + acm := mapFrom(doc, "acm") + mce := mapFrom(acm, "mce_operator") + mce["source"] = cfg.RedHatCS + } + + cg := mapFrom(doc, "clusterGroup") + subs, _ := cg["subscriptions"].(map[string]interface{}) + if subs == nil { + return + } + for _, v := range subs { + entry, ok := v.(map[string]interface{}) + if !ok { + continue + } + name, _ := entry["name"].(string) + if name == "" { + continue + } + kind := LookupCatalog(name) + var src string + switch kind { + case CatalogCommunity: + src = cfg.CommunityCS + case CatalogMarketplace: + src = cfg.Marketplace + if src == "" { + src = cfg.RedHatCS + } + default: + src = cfg.RedHatCS + } + if src != "" { + entry["source"] = src + } + } +} + +func mapFrom(parent map[string]interface{}, key string) map[string]interface{} { + if parent == nil { + return map[string]interface{}{} + } + if existing, ok := parent[key].(map[string]interface{}); ok && existing != nil { + return existing + } + m := map[string]interface{}{} + parent[key] = m + return m +} diff --git a/src/internal/airgap/rendervalues_test.go b/src/internal/airgap/rendervalues_test.go new file mode 100644 index 0000000..8909df6 --- /dev/null +++ b/src/internal/airgap/rendervalues_test.go @@ -0,0 +1,101 @@ +package airgap_test + +import ( + "os" + "path/filepath" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "gopkg.in/yaml.v3" + + "github.com/validatedpatterns/patternizer/internal/airgap" +) + +var _ = Describe("ResolveDisconnectedValues", func() { + It("derives CatalogSources from ocpVersion and helmRepoUrl from registry", func() { + cfg, err := airgap.ResolveDisconnectedValues(&airgap.AirgapSpec{ + Registry: "mirror.example.com:8443", + OCPVersion: "4.22", + }, &airgap.BootstrapOptions{}) + Expect(err).NotTo(HaveOccurred()) + Expect(cfg.HelmRepoURL).To(Equal("mirror.example.com:8443/validatedpatterns")) + Expect(cfg.RedHatCS).To(Equal("cs-redhat-operator-index-v4-22")) + Expect(cfg.CommunityCS).To(Equal("cs-community-operator-index-v4-22")) + }) + + It("prefers explicit catalogSources over ocpVersion", func() { + cfg, err := airgap.ResolveDisconnectedValues(&airgap.AirgapSpec{ + Registry: "mirror.example.com:8443", + OCPVersion: "4.22", + CatalogSources: map[string]string{ + "redhat": "cs-redhat-operator-index-v4-22", + "community": "cs-community-operator-index-v4-22", + }, + }, &airgap.BootstrapOptions{}) + Expect(err).NotTo(HaveOccurred()) + Expect(cfg.RedHatCS).To(Equal("cs-redhat-operator-index-v4-22")) + Expect(cfg.CommunityCS).To(Equal("cs-community-operator-index-v4-22")) + }) +}) + +var _ = Describe("RenderPatternValues", func() { + It("updates values-global and variants hub values with disconnected fields", func() { + tmp := GinkgoT().TempDir() + patternDir := filepath.Join(tmp, "pattern") + Expect(os.MkdirAll(filepath.Join(patternDir, "variants", "hub"), 0o755)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(patternDir, "values-global.yaml"), []byte(` +global: + pattern: test-pattern +main: + variant: hub + multiSourceConfig: + enabled: true + clusterGroupChartVersion: "0.9.*" +`), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(patternDir, "variants", "hub", "values-hub.yaml"), []byte(` +clusterGroup: + name: hub + subscriptions: + acm: + name: advanced-cluster-management + channel: release-2.16 + eso: + name: openshift-external-secrets-operator + channel: stable-v1 +`), 0o644)).To(Succeed()) + + outDir := filepath.Join(tmp, "out") + cfg := &airgap.DisconnectedValuesConfig{ + HelmRepoURL: "mirror.example.com:8443/validatedpatterns", + RedHatCS: "cs-redhat-operator-index-v4-22", + CommunityCS: "cs-community-operator-index-v4-22", + } + files, err := airgap.RenderPatternValues(patternDir, outDir, cfg) + Expect(err).NotTo(HaveOccurred()) + Expect(files).To(HaveLen(2)) + + globalRaw, err := os.ReadFile(filepath.Join(patternDir, "values-global.yaml")) + Expect(err).NotTo(HaveOccurred()) + var global map[string]interface{} + Expect(yaml.Unmarshal(globalRaw, &global)).To(Succeed()) + main := global["main"].(map[string]interface{}) + msc := main["multiSourceConfig"].(map[string]interface{}) + Expect(msc["helmRepoUrl"]).To(Equal("mirror.example.com:8443/validatedpatterns")) + Expect(msc["clusterGroupChartVersion"]).To(Equal("0.9.*")) + Expect(main["patternsOperator"].(map[string]interface{})["source"]).To(Equal("cs-community-operator-index-v4-22")) + Expect(main["gitops"].(map[string]interface{})["operatorSource"]).To(Equal("cs-redhat-operator-index-v4-22")) + + hubRaw, err := os.ReadFile(filepath.Join(patternDir, "variants", "hub", "values-hub.yaml")) + Expect(err).NotTo(HaveOccurred()) + var hub map[string]interface{} + Expect(yaml.Unmarshal(hubRaw, &hub)).To(Succeed()) + subs := hub["clusterGroup"].(map[string]interface{})["subscriptions"].(map[string]interface{}) + Expect(subs["acm"].(map[string]interface{})["source"]).To(Equal("cs-redhat-operator-index-v4-22")) + Expect(subs["eso"].(map[string]interface{})["source"]).To(Equal("cs-redhat-operator-index-v4-22")) + Expect(hub["acm"].(map[string]interface{})["mce_operator"].(map[string]interface{})["source"]).To(Equal("cs-redhat-operator-index-v4-22")) + + Expect(filepath.Join(outDir, "backup", "values", "values-global.yaml")).To(BeAnExistingFile()) + Expect(filepath.Join(outDir, "rendered", "variants", "hub", "values-hub.yaml")).To(BeAnExistingFile()) + }) +}) diff --git a/src/internal/airgap/repocreds.go b/src/internal/airgap/repocreds.go new file mode 100644 index 0000000..cec6005 --- /dev/null +++ b/src/internal/airgap/repocreds.go @@ -0,0 +1,421 @@ +package airgap + +import ( + "encoding/base64" + "encoding/json" + "fmt" + "net/url" + "os" + "path/filepath" + "regexp" + "strings" + "text/template" +) + +// RepoCredsOptions configures generate-only Argo CD registry repo-creds artifacts. +type RepoCredsOptions struct { + DockerConfig string + Username string + Password string + PasswordFile string + URL string + Namespace string + OutputDir string + SecretName string + RepoServer string +} + +// RepoCredsResult summarizes generated repo-creds artifacts. +type RepoCredsResult struct { + OutputDir string + Namespace string + SecretName string + URL string + Username string + RepoServer string +} + +// RegistryCredential is a resolved registry username/password for a URL prefix. +type RegistryCredential struct { + URL string + Username string + Password string +} + +// GenerateRepoCreds writes an Argo CD repo-creds Secret and apply script. +// Credentials come from --docker-config or --username/--password (never hardcoded). +func GenerateRepoCreds(opts *RepoCredsOptions) (*RepoCredsResult, error) { + if opts == nil { + return nil, fmt.Errorf("options are required") + } + + cred, err := ResolveRegistryCredential(opts) + if err != nil { + return nil, err + } + + ns := strings.TrimSpace(opts.Namespace) + if ns == "" { + ns = "vp-gitops" + } + outDir := strings.TrimSpace(opts.OutputDir) + if outDir == "" { + outDir = "./airgap-repo-creds" + } + absOut, err := filepath.Abs(outDir) + if err != nil { + return nil, err + } + outDir = absOut + + secretName := strings.TrimSpace(opts.SecretName) + if secretName == "" { + secretName = defaultRepoCredsSecretName(cred.URL) + } + repoServer := strings.TrimSpace(opts.RepoServer) + if repoServer == "" { + repoServer = ns + "-repo-server" + } + + if err := os.MkdirAll(outDir, 0o755); err != nil { + return nil, err + } + + secretPath := filepath.Join(outDir, "argocd-repo-creds-secret.yaml") + if err := writeRepoCredsSecret(secretPath, ns, secretName, cred); err != nil { + return nil, fmt.Errorf("write argocd-repo-creds-secret.yaml: %w", err) + } + if err := writeApplyRepoCredsScript(outDir, ns, secretName, repoServer); err != nil { + return nil, fmt.Errorf("write apply-repo-creds.sh: %w", err) + } + if err := writeRepoCredsReadme(outDir, ns, secretName, repoServer, cred.URL, cred.Username); err != nil { + return nil, fmt.Errorf("write README.md: %w", err) + } + + return &RepoCredsResult{ + OutputDir: outDir, + Namespace: ns, + SecretName: secretName, + URL: cred.URL, + Username: cred.Username, + RepoServer: repoServer, + }, nil +} + +// ResolveRegistryCredential loads credentials from docker config or explicit flags. +func ResolveRegistryCredential(opts *RepoCredsOptions) (*RegistryCredential, error) { + hasDocker := strings.TrimSpace(opts.DockerConfig) != "" + hasUser := strings.TrimSpace(opts.Username) != "" + hasPass := strings.TrimSpace(opts.Password) != "" || strings.TrimSpace(opts.PasswordFile) != "" + + if hasDocker && (hasUser || hasPass) { + return nil, fmt.Errorf("use either --docker-config or --username/--password, not both") + } + if !hasDocker && !hasUser { + return nil, fmt.Errorf("provide --docker-config or --username (with --password / --password-file)") + } + + if hasDocker { + return credentialFromDockerConfig(opts.DockerConfig, opts.URL) + } + + password, err := loadPassword(opts.Password, opts.PasswordFile) + if err != nil { + return nil, err + } + if password == "" { + return nil, fmt.Errorf("--password or --password-file is required with --username") + } + regURL := NormalizeRegistryURL(opts.URL) + if regURL == "" { + return nil, fmt.Errorf("--url is required with --username/--password (e.g. mirror.example.com:8443)") + } + return &RegistryCredential{ + URL: regURL, + Username: strings.TrimSpace(opts.Username), + Password: password, + }, nil +} + +func loadPassword(password, passwordFile string) (string, error) { + if strings.TrimSpace(passwordFile) != "" { + data, err := os.ReadFile(passwordFile) + if err != nil { + return "", fmt.Errorf("read --password-file: %w", err) + } + return strings.TrimRight(string(data), "\r\n"), nil + } + return password, nil +} + +func credentialFromDockerConfig(path, wantURL string) (*RegistryCredential, error) { + data, err := os.ReadFile(path) + if err != nil { + return nil, fmt.Errorf("read --docker-config %s: %w", path, err) + } + var cfg struct { + Auths map[string]struct { + Auth string `json:"auth"` + Username string `json:"username"` + Password string `json:"password"` + } `json:"auths"` + } + if err := json.Unmarshal(data, &cfg); err != nil { + return nil, fmt.Errorf("parse docker config JSON: %w", err) + } + if len(cfg.Auths) == 0 { + return nil, fmt.Errorf("docker config %s has no auths entries", path) + } + + want := NormalizeRegistryURL(wantURL) + type entry struct { + key string + user, pass string + } + var entries []entry + for key, auth := range cfg.Auths { + user, pass, err := decodeDockerAuth(auth.Auth, auth.Username, auth.Password) + if err != nil { + return nil, fmt.Errorf("auth entry %q: %w", key, err) + } + if user == "" && pass == "" { + continue + } + entries = append(entries, entry{key: key, user: user, pass: pass}) + } + if len(entries) == 0 { + return nil, fmt.Errorf("docker config %s has no usable username/password auths", path) + } + + var matched *entry + if want != "" { + for i := range entries { + e := &entries[i] + norm := NormalizeRegistryURL(e.key) + if norm == want || strings.HasPrefix(want, norm) || strings.HasPrefix(norm, want) { + matched = e + break + } + // Match host without port against host:port keys and vice versa. + if registryHost(norm) == registryHost(want) { + matched = e + break + } + } + if matched == nil { + keys := make([]string, 0, len(entries)) + for _, e := range entries { + keys = append(keys, e.key) + } + return nil, fmt.Errorf("no docker config auth for %q (have: %s)", want, strings.Join(keys, ", ")) + } + } else { + if len(entries) != 1 { + keys := make([]string, 0, len(entries)) + for _, e := range entries { + keys = append(keys, e.key) + } + return nil, fmt.Errorf("docker config has %d auths; pass --url to select one (have: %s)", len(entries), strings.Join(keys, ", ")) + } + matched = &entries[0] + } + + regURL := want + if regURL == "" { + regURL = NormalizeRegistryURL(matched.key) + } + return &RegistryCredential{ + URL: regURL, + Username: matched.user, + Password: matched.pass, + }, nil +} + +func decodeDockerAuth(authB64, username, password string) (user, pass string, err error) { + if username != "" || password != "" { + return username, password, nil + } + authB64 = strings.TrimSpace(authB64) + if authB64 == "" { + return "", "", nil + } + raw, err := base64.StdEncoding.DecodeString(authB64) + if err != nil { + // Some configs use URL-safe or missing padding. + raw, err = base64.RawStdEncoding.DecodeString(authB64) + if err != nil { + return "", "", fmt.Errorf("decode auth field: %w", err) + } + } + parts := strings.SplitN(string(raw), ":", 2) + if len(parts) != 2 { + return "", "", fmt.Errorf("auth field is not username:password") + } + return parts[0], parts[1], nil +} + +// NormalizeRegistryURL returns host[:port][/path] suitable for Argo CD repo-creds url. +// Keeps the port (unlike TLS cert map keys). Strips scheme only. +func NormalizeRegistryURL(raw string) string { + raw = strings.TrimSpace(raw) + if raw == "" { + return "" + } + candidate := raw + if strings.Contains(candidate, "://") { + u, err := url.Parse(candidate) + if err != nil { + return strings.TrimSuffix(strings.TrimPrefix(strings.TrimPrefix(raw, "https://"), "http://"), "/") + } + candidate = u.Host + if u.Path != "" && u.Path != "/" { + candidate += strings.TrimSuffix(u.Path, "/") + } + } + return strings.TrimSuffix(candidate, "/") +} + +func registryHost(regURL string) string { + regURL = NormalizeRegistryURL(regURL) + if i := strings.Index(regURL, "/"); i >= 0 { + regURL = regURL[:i] + } + if strings.HasPrefix(regURL, "[") { + if end := strings.Index(regURL, "]"); end >= 0 { + return regURL[:end+1] + } + } + if i := strings.LastIndex(regURL, ":"); i >= 0 { + return regURL[:i] + } + return regURL +} + +var secretNameSanitizer = regexp.MustCompile(`[^a-z0-9.-]+`) + +func defaultRepoCredsSecretName(regURL string) string { + host := registryHost(regURL) + host = strings.ToLower(host) + host = secretNameSanitizer.ReplaceAllString(host, "-") + host = strings.Trim(host, "-.") + if host == "" { + host = "registry" + } + name := host + "-repo-creds" + if len(name) > 63 { + name = name[:63] + name = strings.Trim(name, "-.") + } + return name +} + +func writeRepoCredsSecret(path, namespace, secretName string, cred *RegistryCredential) error { + // Escape YAML double-quoted strings for password/username that may contain special chars. + var b strings.Builder + b.WriteString("# Generated by patternizer airgap repo-creds\n") + b.WriteString("# Contains registry credentials — do not commit to Git.\n") + b.WriteString("apiVersion: v1\n") + b.WriteString("kind: Secret\n") + b.WriteString("metadata:\n") + fmt.Fprintf(&b, " name: %s\n", secretName) + fmt.Fprintf(&b, " namespace: %s\n", namespace) + b.WriteString(" labels:\n") + b.WriteString(" argocd.argoproj.io/secret-type: repo-creds\n") + b.WriteString("stringData:\n") + fmt.Fprintf(&b, " url: %s\n", yamlDoubleQuoted(cred.URL)) + fmt.Fprintf(&b, " username: %s\n", yamlDoubleQuoted(cred.Username)) + fmt.Fprintf(&b, " password: %s\n", yamlDoubleQuoted(cred.Password)) + return os.WriteFile(path, []byte(b.String()), 0o600) +} + +func yamlDoubleQuoted(s string) string { + escaped := strings.ReplaceAll(s, `\`, `\\`) + escaped = strings.ReplaceAll(escaped, `"`, `\"`) + escaped = strings.ReplaceAll(escaped, "\n", `\n`) + return `"` + escaped + `"` +} + +func writeApplyRepoCredsScript(outDir, namespace, secretName, repoServer string) error { + const script = `#!/usr/bin/env bash +# Generated by patternizer airgap repo-creds +# Applies Argo CD registry repo-creds Secret {{.SecretName}} in {{.Namespace}} +# Requires: oc on PATH and a working kubeconfig (KUBECONFIG or ~/.kube/config) +set -euo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" +NAMESPACE="{{.Namespace}}" +SECRET_NAME="{{.SecretName}}" +REPO_SERVER="{{.RepoServer}}" +SECRET_FILE="${HERE}/argocd-repo-creds-secret.yaml" + +if ! command -v oc >/dev/null 2>&1; then + echo "oc is required but not found on PATH" >&2 + exit 1 +fi +if [[ ! -f "${SECRET_FILE}" ]]; then + echo "missing ${SECRET_FILE}" >&2 + exit 1 +fi + +echo "==> Checking namespace ${NAMESPACE}" +oc get namespace "${NAMESPACE}" >/dev/null + +echo "==> Applying Secret ${SECRET_NAME}" +oc apply -f "${SECRET_FILE}" + +echo "==> Restarting deployment/${REPO_SERVER}" +oc rollout restart "deployment/${REPO_SERVER}" -n "${NAMESPACE}" +oc rollout status "deployment/${REPO_SERVER}" -n "${NAMESPACE}" + +echo "Done. Verify with:" +echo " oc get secret ${SECRET_NAME} -n ${NAMESPACE} --show-labels" +echo " oc get application -n ${NAMESPACE}" +` + + tmpl, err := template.New("apply").Parse(script) + if err != nil { + return err + } + path := filepath.Join(outDir, "apply-repo-creds.sh") + f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o755) + if err != nil { + return err + } + defer f.Close() + + return tmpl.Execute(f, map[string]string{ + "Namespace": namespace, + "SecretName": secretName, + "RepoServer": repoServer, + }) +} + +func writeRepoCredsReadme(outDir, namespace, secretName, repoServer, regURL, username string) error { + var b strings.Builder + b.WriteString("# Argo CD registry repo-creds (generated by patternizer airgap repo-creds)\n\n") + b.WriteString("Creates an Argo CD `repo-creds` Secret so the repo-server can authenticate to a\n") + b.WriteString("private OCI/Helm registry (Quay mirror, Harbor, etc.).\n\n") + b.WriteString("**Do not commit** `argocd-repo-creds-secret.yaml` to Git — it contains a password.\n\n") + b.WriteString("## Prerequisites\n\n") + b.WriteString("- `oc` on PATH\n") + b.WriteString("- Working kubeconfig (`KUBECONFIG` or `~/.kube/config`)\n") + fmt.Fprintf(&b, "- Namespace `%s` with Argo CD installed\n", namespace) + fmt.Fprintf(&b, "- Deployment `%s`\n\n", repoServer) + b.WriteString("## Registry\n\n") + fmt.Fprintf(&b, "- URL: `%s`\n", regURL) + fmt.Fprintf(&b, "- Username: `%s`\n", username) + fmt.Fprintf(&b, "- Secret: `%s`\n\n", secretName) + b.WriteString("## Apply\n\n") + b.WriteString("```bash\n") + b.WriteString("./apply-repo-creds.sh\n") + b.WriteString("```\n\n") + b.WriteString("Typically run **after** `airgap trust-ca` for the same registry hostname so TLS\n") + b.WriteString("and auth are both configured.\n\n") + b.WriteString("## Confirm\n\n") + b.WriteString("```bash\n") + fmt.Fprintf(&b, "oc get secret %s -n %s --show-labels\n", secretName, namespace) + fmt.Fprintf(&b, "oc get application -n %s\n", namespace) + b.WriteString("# Sync errors should no longer show 401 unauthorized for the mirror\n") + b.WriteString("```\n") + return os.WriteFile(filepath.Join(outDir, "README.md"), []byte(b.String()), 0o644) +} diff --git a/src/internal/airgap/repocreds_test.go b/src/internal/airgap/repocreds_test.go new file mode 100644 index 0000000..6981eb3 --- /dev/null +++ b/src/internal/airgap/repocreds_test.go @@ -0,0 +1,157 @@ +package airgap_test + +import ( + "encoding/base64" + "os" + "path/filepath" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/validatedpatterns/patternizer/internal/airgap" +) + +var _ = Describe("NormalizeRegistryURL", func() { + DescribeTable("normalizes registry URLs", + func(in, want string) { + Expect(airgap.NormalizeRegistryURL(in)).To(Equal(want)) + }, + Entry("host:port", "mirror.example.com:8443", "mirror.example.com:8443"), + Entry("strips https", "https://mirror.example.com:8443", "mirror.example.com:8443"), + Entry("keeps path", "https://mirror.example.com:8443/validatedpatterns", "mirror.example.com:8443/validatedpatterns"), + Entry("empty", "", ""), + ) +}) + +var _ = Describe("ResolveRegistryCredential", func() { + var tmp string + + BeforeEach(func() { + tmp = GinkgoT().TempDir() + }) + + It("resolves from username and password-file", func() { + pwFile := filepath.Join(tmp, "pw") + Expect(os.WriteFile(pwFile, []byte("s3cret\n"), 0o600)).To(Succeed()) + cred, err := airgap.ResolveRegistryCredential(&airgap.RepoCredsOptions{ + Username: "init", + PasswordFile: pwFile, + URL: "https://mirror.example.com:8443/", + }) + Expect(err).NotTo(HaveOccurred()) + Expect(cred.Username).To(Equal("init")) + Expect(cred.Password).To(Equal("s3cret")) + Expect(cred.URL).To(Equal("mirror.example.com:8443")) + }) + + It("resolves from docker config with single auth", func() { + auth := base64.StdEncoding.EncodeToString([]byte("init:fromconfig")) + cfg := filepath.Join(tmp, "config.json") + Expect(os.WriteFile(cfg, []byte(`{"auths":{"mirror.example.com:8443":{"auth":"`+auth+`"}}}`), 0o600)).To(Succeed()) + cred, err := airgap.ResolveRegistryCredential(&airgap.RepoCredsOptions{ + DockerConfig: cfg, + }) + Expect(err).NotTo(HaveOccurred()) + Expect(cred.Username).To(Equal("init")) + Expect(cred.Password).To(Equal("fromconfig")) + Expect(cred.URL).To(Equal("mirror.example.com:8443")) + }) + + It("selects docker config auth by --url when multiple exist", func() { + a1 := base64.StdEncoding.EncodeToString([]byte("u1:p1")) + a2 := base64.StdEncoding.EncodeToString([]byte("u2:p2")) + cfg := filepath.Join(tmp, "config.json") + Expect(os.WriteFile(cfg, []byte(`{"auths":{ + "quay.io":{"auth":"`+a1+`"}, + "mirror.example.com:8443":{"auth":"`+a2+`"} + }}`), 0o600)).To(Succeed()) + cred, err := airgap.ResolveRegistryCredential(&airgap.RepoCredsOptions{ + DockerConfig: cfg, + URL: "mirror.example.com:8443", + }) + Expect(err).NotTo(HaveOccurred()) + Expect(cred.Username).To(Equal("u2")) + Expect(cred.Password).To(Equal("p2")) + }) + + It("rejects mixing docker-config and username", func() { + _, err := airgap.ResolveRegistryCredential(&airgap.RepoCredsOptions{ + DockerConfig: "/tmp/x", + Username: "u", + Password: "p", + URL: "r.example.com", + }) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("not both")) + }) + + It("requires url with username", func() { + _, err := airgap.ResolveRegistryCredential(&airgap.RepoCredsOptions{ + Username: "u", + Password: "p", + }) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("--url")) + }) +}) + +var _ = Describe("GenerateRepoCreds", func() { + It("writes secret YAML, apply script, and README", func() { + tmp := GinkgoT().TempDir() + outDir := filepath.Join(tmp, "out") + res, err := airgap.GenerateRepoCreds(&airgap.RepoCredsOptions{ + Username: "init", + Password: "p@ss\"word", + URL: "mirror.airgap.rhuk.lab:8443", + Namespace: "vp-gitops", + OutputDir: outDir, + }) + Expect(err).NotTo(HaveOccurred()) + Expect(res.SecretName).To(Equal("mirror.airgap.rhuk.lab-repo-creds")) + Expect(res.URL).To(Equal("mirror.airgap.rhuk.lab:8443")) + Expect(res.Username).To(Equal("init")) + Expect(res.RepoServer).To(Equal("vp-gitops-repo-server")) + + secretPath := filepath.Join(outDir, "argocd-repo-creds-secret.yaml") + Expect(secretPath).To(BeAnExistingFile()) + info, err := os.Stat(secretPath) + Expect(err).NotTo(HaveOccurred()) + Expect(info.Mode().Perm() & 0o077).To(BeZero()) + + raw, err := os.ReadFile(secretPath) + Expect(err).NotTo(HaveOccurred()) + Expect(string(raw)).To(ContainSubstring("argocd.argoproj.io/secret-type: repo-creds")) + Expect(string(raw)).To(ContainSubstring("url: \"mirror.airgap.rhuk.lab:8443\"")) + Expect(string(raw)).To(ContainSubstring("username: \"init\"")) + Expect(string(raw)).To(ContainSubstring(`password: "p@ss\"word"`)) + Expect(string(raw)).NotTo(ContainSubstring("quay.io")) + + script, err := os.ReadFile(filepath.Join(outDir, "apply-repo-creds.sh")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(script)).To(HavePrefix("#!/usr/bin/env bash")) + Expect(string(script)).To(ContainSubstring("argocd-repo-creds-secret.yaml")) + Expect(string(script)).To(ContainSubstring("rollout restart")) + + scriptInfo, err := os.Stat(filepath.Join(outDir, "apply-repo-creds.sh")) + Expect(err).NotTo(HaveOccurred()) + Expect(scriptInfo.Mode().Perm() & 0o111).NotTo(BeZero()) + + readme, err := os.ReadFile(filepath.Join(outDir, "README.md")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(readme)).To(ContainSubstring("Do not commit")) + Expect(string(readme)).To(ContainSubstring("trust-ca")) + }) + + It("respects custom secret name", func() { + tmp := GinkgoT().TempDir() + res, err := airgap.GenerateRepoCreds(&airgap.RepoCredsOptions{ + Username: "u", + Password: "p", + URL: "reg.example.com:5000", + SecretName: "my-mirror-creds", + OutputDir: filepath.Join(tmp, "out"), + }) + Expect(err).NotTo(HaveOccurred()) + Expect(res.SecretName).To(Equal("my-mirror-creds")) + }) +}) diff --git a/src/internal/airgap/resolve.go b/src/internal/airgap/resolve.go new file mode 100644 index 0000000..87ada51 --- /dev/null +++ b/src/internal/airgap/resolve.go @@ -0,0 +1,156 @@ +package airgap + +import ( + "encoding/json" + "fmt" + "os/exec" + "sort" + "strings" + + "github.com/Masterminds/semver/v3" + "gopkg.in/yaml.v3" +) + +// IsConcreteChartVersion reports whether v is a concrete OCI tag. +func IsConcreteChartVersion(v string) bool { + v = strings.TrimSpace(v) + if v == "" { + return false + } + if v == "latest" { + return true + } + if strings.ContainsAny(v, "*^~><") { + return false + } + if strings.HasSuffix(strings.ToLower(v), ".x") { + return false + } + return true +} + +// NormalizeChartConstraint converts placeholder forms like 0.9.x to 0.9.*. +func NormalizeChartConstraint(constraint string) string { + constraint = strings.TrimSpace(constraint) + if constraint == "" { + return "*" + } + lower := strings.ToLower(constraint) + if strings.HasSuffix(lower, ".x") { + return constraint[:len(constraint)-1] + "*" + } + if lower == "x" { + return "*" + } + return constraint +} + +// ResolveOCIChartVersion resolves a chartVersion constraint to a concrete OCI tag. +func ResolveOCIChartVersion(helmRepo, chartName, constraint string, skip bool) (string, error) { + constraint = NormalizeChartConstraint(constraint) + + if IsConcreteChartVersion(constraint) { + return strings.TrimPrefix(constraint, "v"), nil + } + + if skip { + return "", fmt.Errorf("cannot resolve %s constraint %q with --skip-oci-resolve; need a concrete version or omit --skip-oci-resolve", chartName, constraint) + } + + var errs []string + + ver, err := resolveChartVersionHelm(helmRepo, chartName, constraint) + if err == nil { + return ver, nil + } + errs = append(errs, "helm: "+err.Error()) + + ver, err = resolveChartVersionSkopeo(helmRepo, chartName, constraint) + if err == nil { + return ver, nil + } + errs = append(errs, "skopeo: "+err.Error()) + + return "", fmt.Errorf("resolve %s constraint %q: %s", chartName, constraint, strings.Join(errs, "; ")) +} + +func resolveChartVersionHelm(helmRepo, chartName, constraint string) (string, error) { + if _, err := exec.LookPath("helm"); err != nil { + return "", fmt.Errorf("helm not found on PATH") + } + ref := fmt.Sprintf("oci://%s/%s", helmRepo, chartName) + cmd := exec.Command("helm", "show", "chart", ref, "--version", constraint) + out, err := cmd.CombinedOutput() + if err != nil { + return "", fmt.Errorf("show chart %s --version %s: %w (%s)", ref, constraint, err, truncate(string(out), 300)) + } + var meta struct { + Version string `yaml:"version"` + } + if err := yaml.Unmarshal(out, &meta); err != nil { + return "", err + } + if !IsConcreteChartVersion(meta.Version) { + return "", fmt.Errorf("empty or non-concrete version %q", meta.Version) + } + return meta.Version, nil +} + +func resolveChartVersionSkopeo(helmRepo, chartName, constraint string) (string, error) { + tags, err := listOCITagsSkopeo(helmRepo, chartName) + if err != nil { + return "", err + } + return PickLatestMatchingTag(tags, constraint) +} + +func listOCITagsSkopeo(helmRepo, chartName string) ([]string, error) { + if _, err := exec.LookPath("skopeo"); err != nil { + return nil, fmt.Errorf("skopeo not found on PATH") + } + ref := fmt.Sprintf("docker://%s/%s", helmRepo, chartName) + cmd := exec.Command("skopeo", "list-tags", ref) + out, err := cmd.CombinedOutput() + if err != nil { + return nil, fmt.Errorf("list-tags %s: %w (%s)", ref, err, truncate(string(out), 300)) + } + var parsed struct { + Tags []string `json:"Tags"` + } + if err := json.Unmarshal(out, &parsed); err != nil { + return nil, fmt.Errorf("parse list-tags JSON: %w", err) + } + if len(parsed.Tags) == 0 { + return nil, fmt.Errorf("no tags for %s/%s", helmRepo, chartName) + } + return parsed.Tags, nil +} + +// PickLatestMatchingTag returns the highest semver tag that satisfies constraint. +func PickLatestMatchingTag(tags []string, constraint string) (string, error) { + constraint = NormalizeChartConstraint(constraint) + c, err := semver.NewConstraint(constraint) + if err != nil { + return "", fmt.Errorf("invalid version constraint %q: %w", constraint, err) + } + + var matched []*semver.Version + for _, tag := range tags { + tag = strings.TrimSpace(tag) + if tag == "" || !IsConcreteChartVersion(tag) { + continue + } + v, err := semver.NewVersion(strings.TrimPrefix(tag, "v")) + if err != nil { + continue + } + if c.Check(v) { + matched = append(matched, v) + } + } + if len(matched) == 0 { + return "", fmt.Errorf("no tags match constraint %q (%d tags scanned)", constraint, len(tags)) + } + sort.Sort(semver.Collection(matched)) + return matched[len(matched)-1].Original(), nil +} diff --git a/src/internal/airgap/secretsparse.go b/src/internal/airgap/secretsparse.go new file mode 100644 index 0000000..28d0126 --- /dev/null +++ b/src/internal/airgap/secretsparse.go @@ -0,0 +1,298 @@ +package airgap + +import ( + "fmt" + "os" + "path/filepath" + "strconv" + "strings" + + "gopkg.in/yaml.v3" +) + +// ValuesSecretFile is the subset of values-secret.yaml used by airgap bootstrap. +// The top-level airgap key is ignored by Vault load-secrets. +type ValuesSecretFile struct { + Version string `yaml:"version"` + Airgap *AirgapSpec `yaml:"airgap,omitempty"` +} + +// AirgapSpec maps hostnames, CA files, and registry credentials for disconnected install. +type AirgapSpec struct { + ArgocdNamespace string `yaml:"argocdNamespace,omitempty"` + PatternsOperatorNamespace string `yaml:"patternsOperatorNamespace,omitempty"` + Certificates []AirgapNamedBlock `yaml:"certificates,omitempty"` + Registries []AirgapNamedBlock `yaml:"registries,omitempty"` + + // Disconnected values rendering (values-global.yaml / variants/*/values-*.yaml). + Registry string `yaml:"registry,omitempty"` // e.g. mirror.example.com:8443 + HelmRepoURL string `yaml:"helmRepoUrl,omitempty"` // override; default /validatedpatterns + OCPVersion string `yaml:"ocpVersion,omitempty"` // e.g. 4.22 — derives CatalogSource names + CatalogSources map[string]string `yaml:"catalogSources,omitempty"` // keys: redhat, community, marketplace + DiscoverCatalogs bool `yaml:"discoverCatalogs,omitempty"` // query oc for CatalogSources + RenderValues *bool `yaml:"renderValues,omitempty"` // default true when pattern dir set +} + +// WantsRenderValues reports whether bootstrap should rewrite pattern values files. +func (s *AirgapSpec) WantsRenderValues() bool { + if s == nil { + return false + } + if s.RenderValues != nil { + return *s.RenderValues + } + return s.Registry != "" || s.HelmRepoURL != "" || s.OCPVersion != "" || len(s.CatalogSources) > 0 || s.DiscoverCatalogs +} + +// AirgapNamedBlock mirrors values-secret secret entries (name + fields with value/path). +type AirgapNamedBlock struct { + Name string `yaml:"name"` + Fields []AirgapField `yaml:"fields"` +} + +// AirgapField is a values-secret-style field (literal value or file path). +type AirgapField struct { + Name string `yaml:"name"` + Value string `yaml:"value,omitempty"` + Path string `yaml:"path,omitempty"` +} + +// ResolvedCertificate is a certificate entry after resolving fields. +type ResolvedCertificate struct { + Name string + Host string + CAPath string + ClusterProxy bool + ArgocdTLS bool +} + +// ResolvedRegistry is a registry credential entry after resolving fields. +type ResolvedRegistry struct { + Name string + URL string + DockerConfigPath string + Username string + PasswordPath string + Password string +} + +// BootstrapOptions configures generate-only unified airgap trust artifacts. +type BootstrapOptions struct { + PatternDir string + PatternName string + SecretsFile string + OutputDir string + Namespace string // override airgap.argocdNamespace + RepoServer string + DiscoverCatalogs bool // CLI override: query oc for CatalogSources + SkipRenderValues bool // CLI override: do not rewrite pattern values + Registry string + OCPVersion string +} + +// BootstrapResult summarizes generated bootstrap artifacts. +type BootstrapResult struct { + OutputDir string + SecretsFile string + ArgocdNamespace string + ClusterProxyCAs int + ArgocdTLSHosts int + Registries int + RenderedFiles []string + HelmRepoURL string + RedHatCS string + CommunityCS string +} + +// ResolveValuesSecretPath picks the values-secret file the same way VP install does. +// Order: explicit path, ~/values-secret-.yaml, /values-secret.yaml, +// /values-secret.yaml.template. +func ResolveValuesSecretPath(patternDir, patternName, explicit string) (string, error) { + if strings.TrimSpace(explicit) != "" { + p, err := expandPath(explicit) + if err != nil { + return "", err + } + if _, err := os.Stat(p); err != nil { + return "", fmt.Errorf("secrets file %s: %w", p, err) + } + return p, nil + } + + home, _ := os.UserHomeDir() + if patternName != "" && home != "" { + candidate := filepath.Join(home, "values-secret-"+patternName+".yaml") + if _, err := os.Stat(candidate); err == nil { + return candidate, nil + } + } + + if patternDir != "" { + for _, name := range []string{"values-secret.yaml", "values-secret.yaml.template"} { + candidate := filepath.Join(patternDir, name) + if _, err := os.Stat(candidate); err == nil { + return candidate, nil + } + } + } + + return "", fmt.Errorf("no values-secret file found (tried --secrets-file, ~/values-secret-.yaml, values-secret.yaml[.template] under pattern dir)") +} + +// LoadAirgapSpec parses the airgap section from a values-secret file. +func LoadAirgapSpec(path string) (*AirgapSpec, error) { + data, err := os.ReadFile(path) + if err != nil { + return nil, err + } + var doc ValuesSecretFile + if err := yaml.Unmarshal(data, &doc); err != nil { + return nil, fmt.Errorf("parse %s: %w", path, err) + } + if doc.Airgap == nil { + return nil, fmt.Errorf("%s has no top-level airgap: section (see values-secret.yaml.template comments)", path) + } + return doc.Airgap, nil +} + +// ResolveCertificates resolves certificate blocks from the airgap spec. +func ResolveCertificates(spec *AirgapSpec) ([]ResolvedCertificate, error) { + var out []ResolvedCertificate + for _, block := range spec.Certificates { + fields, err := resolveFieldMap(block.Fields) + if err != nil { + return nil, fmt.Errorf("certificate %q: %w", block.Name, err) + } + caPath := fields["ca"] + if caPath == "" { + return nil, fmt.Errorf("certificate %q: field ca (path or value) is required", block.Name) + } + // ca may be given as path= or as value= containing a path; LoadCAPEM needs a filesystem path. + if _, err := os.Stat(caPath); err != nil { + return nil, fmt.Errorf("certificate %q: ca file %s: %w", block.Name, caPath, err) + } + host := fields["host"] + cp := parseBoolField(fields["clusterproxy"]) + at := parseBoolField(fields["argocdtls"]) + if !cp && !at { + // Default: both, matching common airgap needs. + cp, at = true, true + } + if at && host == "" { + return nil, fmt.Errorf("certificate %q: field host is required when argocdTLS is true", block.Name) + } + out = append(out, ResolvedCertificate{ + Name: block.Name, + Host: host, + CAPath: caPath, + ClusterProxy: cp, + ArgocdTLS: at, + }) + } + return out, nil +} + +// ResolveRegistries resolves registry credential blocks from the airgap spec. +func ResolveRegistries(spec *AirgapSpec) ([]ResolvedRegistry, error) { + var out []ResolvedRegistry + for _, block := range spec.Registries { + fields, err := resolveFieldMap(block.Fields) + if err != nil { + return nil, fmt.Errorf("registry %q: %w", block.Name, err) + } + url := fields["url"] + if url == "" { + return nil, fmt.Errorf("registry %q: field url is required", block.Name) + } + r := ResolvedRegistry{ + Name: block.Name, + URL: url, + DockerConfigPath: fields["dockerconfig"], + Username: fields["username"], + } + if passField := fields["password"]; passField != "" { + if _, err := os.Stat(passField); err == nil { + r.PasswordPath = passField + } else { + r.Password = passField + } + } + if r.DockerConfigPath == "" && r.Username == "" { + return nil, fmt.Errorf("registry %q: provide dockerconfig path or username (+ password path/value)", block.Name) + } + out = append(out, r) + } + return out, nil +} + +func resolveFieldMap(fields []AirgapField) (map[string]string, error) { + out := map[string]string{} + for _, f := range fields { + key := strings.ToLower(strings.TrimSpace(f.Name)) + if key == "" { + continue + } + if strings.TrimSpace(f.Path) != "" && strings.TrimSpace(f.Value) != "" { + return nil, fmt.Errorf("field %q: set path or value, not both", f.Name) + } + if strings.TrimSpace(f.Path) != "" { + p, err := expandPath(f.Path) + if err != nil { + return nil, fmt.Errorf("field %q: %w", f.Name, err) + } + out[key] = p + continue + } + out[key] = f.Value + } + return out, nil +} + +func parseBoolField(v string) bool { + v = strings.TrimSpace(strings.ToLower(v)) + if v == "" { + return false + } + b, err := strconv.ParseBool(v) + if err == nil { + return b + } + switch v { + case "yes", "y", "on": + return true + default: + return false + } +} + +func expandPath(p string) (string, error) { + p = strings.TrimSpace(p) + if p == "" { + return "", nil + } + if strings.HasPrefix(p, "~/") { + home, err := os.UserHomeDir() + if err != nil { + return "", err + } + p = filepath.Join(home, p[2:]) + } + return filepath.Abs(p) +} + +// PatternNameFromDir reads global.pattern from values-global.yaml when present. +func PatternNameFromDir(patternDir string) string { + data, err := os.ReadFile(filepath.Join(patternDir, "values-global.yaml")) + if err != nil { + return "" + } + var doc struct { + Global struct { + Pattern string `yaml:"pattern"` + } `yaml:"global"` + } + if err := yaml.Unmarshal(data, &doc); err != nil { + return "" + } + return doc.Global.Pattern +} diff --git a/src/internal/airgap/source.go b/src/internal/airgap/source.go new file mode 100644 index 0000000..e4d241a --- /dev/null +++ b/src/internal/airgap/source.go @@ -0,0 +1,201 @@ +package airgap + +import ( + "fmt" + "net/url" + "os" + "os/exec" + "path/filepath" + "strings" +) + +// PatternSource is a resolved local pattern directory. +type PatternSource struct { + Dir string + FromGit bool + GitURL string +} + +// ResolvePatternSource resolves an absolute path or Git URL to a local pattern directory. +func ResolvePatternSource(input string, opts *Options) (*PatternSource, error) { + input = strings.TrimSpace(input) + if input == "" { + return nil, fmt.Errorf("pattern source is required: provide an absolute path or a Git repository URL") + } + + if IsGitURL(input) { + return clonePattern(input, opts) + } + + if !filepath.IsAbs(input) { + return nil, fmt.Errorf("pattern path must be absolute (got %q); pass an absolute path or a Git URL", input) + } + + info, err := os.Stat(input) + if err != nil { + return nil, fmt.Errorf("pattern path %q: %w", input, err) + } + if !info.IsDir() { + return nil, fmt.Errorf("pattern path %q is not a directory", input) + } + if err := validatePatternRoot(input); err != nil { + return nil, err + } + + return &PatternSource{Dir: input}, nil +} + +// IsGitURL reports whether s looks like a Git repository URL. +func IsGitURL(s string) bool { + s = strings.TrimSpace(s) + switch { + case strings.HasPrefix(s, "git@"): + return true + case strings.HasPrefix(s, "git://"): + return true + case strings.HasPrefix(s, "ssh://"): + return true + case strings.HasPrefix(s, "file://"): + return true + case strings.HasPrefix(s, "http://"), strings.HasPrefix(s, "https://"): + u, err := url.Parse(s) + return err == nil && u.Host != "" + case strings.HasSuffix(s, ".git"): + return strings.Contains(s, "/") && !strings.HasPrefix(s, "/") && !strings.HasPrefix(s, ".") + default: + return false + } +} + +func validatePatternRoot(dir string) error { + if _, err := os.Stat(filepath.Join(dir, "values-global.yaml")); err != nil { + return fmt.Errorf("%s does not look like a validated pattern (missing values-global.yaml)", dir) + } + return nil +} + +func clonePattern(gitURL string, opts *Options) (*PatternSource, error) { + dest, reuse, err := cloneDestination(gitURL, opts) + if err != nil { + return nil, err + } + + if !reuse { + if _, err := exec.LookPath("git"); err != nil { + return nil, fmt.Errorf("git is required to clone %s: %w", gitURL, err) + } + + args := []string{"clone", "--depth", "1"} + if opts.GitRef != "" { + args = append(args, "--branch", opts.GitRef) + } + args = append(args, gitURL, dest) + + cmd := exec.Command("git", args...) + out, err := cmd.CombinedOutput() + if err != nil { + return nil, fmt.Errorf("git clone %s: %w (%s)", gitURL, err, truncate(string(out), 400)) + } + + if err := unshallowClone(dest); err != nil { + return nil, fmt.Errorf("git fetch --unshallow %s: %w", dest, err) + } + + if err := validatePatternRoot(dest); err != nil { + return nil, err + } + } + + abs, err := filepath.Abs(dest) + if err != nil { + return nil, err + } + + return &PatternSource{Dir: abs, FromGit: true, GitURL: gitURL}, nil +} + +// unshallowClone converts a shallow clone into a full history repository. +func unshallowClone(dest string) error { + check := exec.Command("git", "-C", dest, "rev-parse", "--is-shallow-repository") + out, err := check.CombinedOutput() + if err != nil { + return fmt.Errorf("%w (%s)", err, truncate(string(out), 200)) + } + if strings.TrimSpace(string(out)) != "true" { + return nil + } + + cmd := exec.Command("git", "-C", dest, "fetch", "--unshallow") + out, err = cmd.CombinedOutput() + if err != nil { + return fmt.Errorf("%w (%s)", err, truncate(string(out), 400)) + } + return nil +} + +func cloneDestination(gitURL string, opts *Options) (dest string, reuse bool, err error) { + name := repoNameFromURL(gitURL) + base := opts.CloneDir + if base == "" { + cwd, err := os.Getwd() + if err != nil { + return "", false, err + } + base = cwd + } else { + abs, err := filepath.Abs(base) + if err != nil { + return "", false, err + } + base = abs + } + + if err := os.MkdirAll(base, 0o755); err != nil { + return "", false, err + } + + dest = filepath.Join(base, name) + if info, err := os.Stat(dest); err == nil { + if !info.IsDir() { + return "", false, fmt.Errorf("clone destination %q exists and is not a directory", dest) + } + if entries, _ := os.ReadDir(dest); len(entries) > 0 { + if err := validatePatternRoot(dest); err == nil { + return dest, true, nil + } + return "", false, fmt.Errorf("clone destination %q already exists and is not empty", dest) + } + return dest, false, nil + } + return dest, false, nil +} + +func repoNameFromURL(gitURL string) string { + u := strings.TrimSpace(gitURL) + u = strings.TrimSuffix(u, "/") + u = strings.TrimSuffix(u, ".git") + if i := strings.LastIndex(u, "/"); i >= 0 { + return sanitizeDirName(u[i+1:]) + } + if i := strings.LastIndex(u, ":"); i >= 0 { + return sanitizeDirName(u[i+1:]) + } + return "pattern" +} + +func sanitizeDirName(name string) string { + name = strings.TrimSpace(name) + if name == "" { + return "pattern" + } + var b strings.Builder + for _, r := range name { + switch { + case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '-', r == '_', r == '.': + b.WriteRune(r) + default: + b.WriteByte('-') + } + } + return b.String() +} diff --git a/src/internal/airgap/source_test.go b/src/internal/airgap/source_test.go new file mode 100644 index 0000000..eff7c28 --- /dev/null +++ b/src/internal/airgap/source_test.go @@ -0,0 +1,129 @@ +package airgap_test + +import ( + "os" + "os/exec" + "path/filepath" + "strings" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/validatedpatterns/patternizer/internal/airgap" +) + +var _ = Describe("ResolvePatternSource", func() { + It("accepts absolute paths to pattern roots", func() { + tmp, err := os.MkdirTemp("", "src-abs-") + Expect(err).NotTo(HaveOccurred()) + defer os.RemoveAll(tmp) + + Expect(os.WriteFile(filepath.Join(tmp, "values-global.yaml"), []byte("global:\n pattern: x\n"), 0o644)).To(Succeed()) + + src, err := airgap.ResolvePatternSource(tmp, &airgap.Options{}) + Expect(err).NotTo(HaveOccurred()) + Expect(src.Dir).To(Equal(tmp)) + Expect(src.FromGit).To(BeFalse()) + }) + + It("rejects relative paths", func() { + _, err := airgap.ResolvePatternSource("./relative-pattern", &airgap.Options{}) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("absolute")) + }) + + It("detects git URLs", func() { + Expect(airgap.IsGitURL("https://github.com/validatedpatterns/multicloud-gitops")).To(BeTrue()) + Expect(airgap.IsGitURL("git@github.com:validatedpatterns/multicloud-gitops.git")).To(BeTrue()) + Expect(airgap.IsGitURL("/home/user/patterns/multicloud-gitops")).To(BeFalse()) + Expect(airgap.IsGitURL(".")).To(BeFalse()) + }) + + It("clones a local git repository URL", func() { + if _, err := exec.LookPath("git"); err != nil { + Skip("git not available") + } + + bareParent, err := os.MkdirTemp("", "src-git-") + Expect(err).NotTo(HaveOccurred()) + defer os.RemoveAll(bareParent) + + repo := filepath.Join(bareParent, "repo") + Expect(os.MkdirAll(repo, 0o755)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(repo, "values-global.yaml"), []byte(` +global: + pattern: cloned-pattern +main: + clusterGroupName: hub + multiSourceConfig: + enabled: true + clusterGroupChartVersion: "0.9.1" +`), 0o644)).To(Succeed()) + Expect(os.MkdirAll(filepath.Join(repo, "charts", "all", "hello-world", "templates"), 0o755)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(repo, "values-hub.yaml"), []byte(` +clusterGroup: + name: hub + namespaces: + hello-world: + subscriptions: {} + applications: + hello-world: + name: hello-world + namespace: hello-world + path: charts/all/hello-world +`), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(repo, "charts", "all", "hello-world", "Chart.yaml"), []byte(` +apiVersion: v2 +name: hello-world +version: 0.1.0 +`), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(repo, "charts", "all", "hello-world", "values.yaml"), []byte("image: quay.io/example/hello:1.36\n"), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(repo, "charts", "all", "hello-world", "templates", "deploy.yaml"), []byte(` +apiVersion: apps/v1 +kind: Deployment +metadata: + name: hello +spec: + template: + spec: + containers: + - name: c + image: "{{ .Values.image }}" +`), 0o644)).To(Succeed()) + + cmds := [][]string{ + {"git", "init"}, + {"git", "config", "user.email", "test@example.com"}, + {"git", "config", "user.name", "test"}, + {"git", "add", "."}, + {"git", "commit", "-m", "init"}, + } + for _, c := range cmds { + cmd := exec.Command(c[0], c[1:]...) + cmd.Dir = repo + out, err := cmd.CombinedOutput() + Expect(err).NotTo(HaveOccurred(), string(out)) + } + + outDir := filepath.Join(bareParent, "out") + opts := airgap.Options{ + PatternSource: "file://" + repo, + OutputDir: outDir, + SkipOCIResolve: true, + WriteValues: true, + } + inv, err := airgap.Init(&opts) + Expect(err).NotTo(HaveOccurred()) + Expect(inv.PatternName).To(Equal("cloned-pattern")) + Expect(inv.ResolvedDir).To(Equal(filepath.Join(outDir, "repo"))) + Expect(filepath.Join(outDir, "imagesetconfig.yaml")).To(BeAnExistingFile()) + Expect(filepath.Join(outDir, "repo", "values-global.yaml")).To(BeAnExistingFile()) + Expect(inv.LocalCharts).To(BeEmpty()) + Expect(inv.AdditionalImages).To(ContainElement("quay.io/example/hello:1.36")) + + shallow := exec.Command("git", "-C", inv.ResolvedDir, "rev-parse", "--is-shallow-repository") + shallowOut, err := shallow.CombinedOutput() + Expect(err).NotTo(HaveOccurred(), string(shallowOut)) + Expect(strings.TrimSpace(string(shallowOut))).To(Equal("false")) + }) +}) diff --git a/src/internal/airgap/trustca.go b/src/internal/airgap/trustca.go new file mode 100644 index 0000000..8f2cc85 --- /dev/null +++ b/src/internal/airgap/trustca.go @@ -0,0 +1,354 @@ +package airgap + +import ( + "fmt" + "net/url" + "os" + "path/filepath" + "sort" + "strings" + "text/template" +) + +// TrustCAOptions configures generate-only Argo CD TLS trust artifacts. +type TrustCAOptions struct { + CAPath string + Hosts []string + Namespace string + OutputDir string + RepoServer string +} + +// TrustCAResult summarizes generated trust-ca artifacts. +type TrustCAResult struct { + OutputDir string + Hosts []string + Namespace string + RepoServer string + CABytes int +} + +// GenerateTrustCA writes Argo CD TLS trust artifacts for the given CA and hosts. +// It does not contact the cluster; the user applies apply-trust-ca.sh with oc. +func GenerateTrustCA(opts *TrustCAOptions) (*TrustCAResult, error) { + if opts == nil { + return nil, fmt.Errorf("options are required") + } + if strings.TrimSpace(opts.CAPath) == "" { + return nil, fmt.Errorf("--ca is required") + } + if len(opts.Hosts) == 0 { + return nil, fmt.Errorf("at least one --host is required") + } + + ns := strings.TrimSpace(opts.Namespace) + if ns == "" { + ns = "vp-gitops" + } + outDir := strings.TrimSpace(opts.OutputDir) + if outDir == "" { + outDir = "./airgap-trust" + } + absOut, err := filepath.Abs(outDir) + if err != nil { + return nil, err + } + outDir = absOut + + repoServer := strings.TrimSpace(opts.RepoServer) + if repoServer == "" { + repoServer = ns + "-repo-server" + } + + pem, err := LoadCAPEM(opts.CAPath) + if err != nil { + return nil, err + } + + hosts, err := NormalizeHosts(opts.Hosts) + if err != nil { + return nil, err + } + + if err := os.MkdirAll(outDir, 0o755); err != nil { + return nil, err + } + + if err := writeTLSCertsPatch(outDir, ns, hosts, pem); err != nil { + return nil, fmt.Errorf("write argocd-tls-certs-patch.yaml: %w", err) + } + if err := os.WriteFile(filepath.Join(outDir, "trusted-ca-bundle-append.pem"), []byte(pem), 0o644); err != nil { + return nil, fmt.Errorf("write trusted-ca-bundle-append.pem: %w", err) + } + if err := writeApplyTrustCAScript(outDir, ns, repoServer, hosts); err != nil { + return nil, fmt.Errorf("write apply-trust-ca.sh: %w", err) + } + if err := writeTrustCAReadme(outDir, ns, repoServer, hosts); err != nil { + return nil, fmt.Errorf("write README.md: %w", err) + } + + return &TrustCAResult{ + OutputDir: outDir, + Hosts: hosts, + Namespace: ns, + RepoServer: repoServer, + CABytes: len(pem), + }, nil +} + +// LoadCAPEM reads a PEM file or concatenates .crt/.pem files from a directory. +func LoadCAPEM(path string) (string, error) { + info, err := os.Stat(path) + if err != nil { + return "", fmt.Errorf("stat --ca %s: %w", path, err) + } + + var pem string + if info.IsDir() { + entries, err := os.ReadDir(path) + if err != nil { + return "", fmt.Errorf("read --ca dir %s: %w", path, err) + } + var files []string + for _, e := range entries { + if e.IsDir() { + continue + } + name := e.Name() + lower := strings.ToLower(name) + if strings.HasSuffix(lower, ".crt") || strings.HasSuffix(lower, ".pem") { + files = append(files, filepath.Join(path, name)) + } + } + sort.Strings(files) + if len(files) == 0 { + return "", fmt.Errorf("no .crt or .pem files found in %s", path) + } + var b strings.Builder + for _, f := range files { + data, err := os.ReadFile(f) + if err != nil { + return "", fmt.Errorf("read %s: %w", f, err) + } + b.Write(data) + if !strings.HasSuffix(string(data), "\n") { + b.WriteByte('\n') + } + } + pem = b.String() + } else { + data, err := os.ReadFile(path) + if err != nil { + return "", fmt.Errorf("read --ca %s: %w", path, err) + } + pem = string(data) + } + + pem = strings.TrimSpace(pem) + "\n" + if !strings.Contains(pem, "BEGIN CERTIFICATE") { + return "", fmt.Errorf("CA at %s does not contain a PEM certificate (missing BEGIN CERTIFICATE)", path) + } + return pem, nil +} + +// NormalizeHosts strips schemes, paths, and ports; returns unique hostname keys. +func NormalizeHosts(raw []string) ([]string, error) { + seen := map[string]struct{}{} + var out []string + for _, h := range raw { + for _, part := range strings.Split(h, ",") { + host, err := NormalizeHost(strings.TrimSpace(part)) + if err != nil { + return nil, err + } + if host == "" { + continue + } + if _, ok := seen[host]; ok { + continue + } + seen[host] = struct{}{} + out = append(out, host) + } + } + if len(out) == 0 { + return nil, fmt.Errorf("no valid hosts after normalization") + } + sort.Strings(out) + return out, nil +} + +// NormalizeHost returns a hostname suitable for argocd-tls-certs-cm keys. +func NormalizeHost(raw string) (string, error) { + raw = strings.TrimSpace(raw) + if raw == "" { + return "", nil + } + candidate := raw + if strings.Contains(candidate, "://") { + u, err := url.Parse(candidate) + if err != nil { + return "", fmt.Errorf("parse host %q: %w", raw, err) + } + candidate = u.Host + if candidate == "" { + candidate = u.Path + } + } else if strings.Contains(candidate, "/") { + candidate = strings.SplitN(candidate, "/", 2)[0] + } + candidate = strings.TrimSpace(candidate) + if i := strings.LastIndex(candidate, ":"); i >= 0 { + // Strip port; avoid mangling IPv6 ([::1]:443). + if !strings.HasPrefix(candidate, "[") { + candidate = candidate[:i] + } else if end := strings.Index(candidate, "]"); end >= 0 && end+1 < len(candidate) && candidate[end+1] == ':' { + candidate = candidate[:end+1] + } + } + candidate = strings.Trim(candidate, "[]") + candidate = strings.TrimSpace(candidate) + if candidate == "" { + return "", fmt.Errorf("empty hostname from %q", raw) + } + if strings.ContainsAny(candidate, "/\\ ") { + return "", fmt.Errorf("invalid hostname %q (from %q)", candidate, raw) + } + return candidate, nil +} + +func writeTLSCertsPatch(outDir, namespace string, hosts []string, pem string) error { + var b strings.Builder + b.WriteString("# Generated by patternizer airgap trust-ca\n") + b.WriteString("# Merge into argocd-tls-certs-cm (hostname keys only; no port).\n") + b.WriteString("apiVersion: v1\n") + b.WriteString("kind: ConfigMap\n") + b.WriteString("metadata:\n") + b.WriteString(" name: argocd-tls-certs-cm\n") + fmt.Fprintf(&b, " namespace: %s\n", namespace) + b.WriteString("data:\n") + for _, host := range hosts { + fmt.Fprintf(&b, " %s: |\n", host) + for _, line := range strings.Split(strings.TrimSuffix(pem, "\n"), "\n") { + fmt.Fprintf(&b, " %s\n", line) + } + } + return os.WriteFile(filepath.Join(outDir, "argocd-tls-certs-patch.yaml"), []byte(b.String()), 0o644) +} + +func writeApplyTrustCAScript(outDir, namespace, repoServer string, hosts []string) error { + const script = `#!/usr/bin/env bash +# Generated by patternizer airgap trust-ca +# Applies Argo CD TLS trust for: {{.HostsList}} +# Requires: oc on PATH and a working kubeconfig (KUBECONFIG or ~/.kube/config) +set -euo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" +NAMESPACE="{{.Namespace}}" +REPO_SERVER="{{.RepoServer}}" +TLS_PATCH="${HERE}/argocd-tls-certs-patch.yaml" +CA_PEM="${HERE}/trusted-ca-bundle-append.pem" + +if ! command -v oc >/dev/null 2>&1; then + echo "oc is required but not found on PATH" >&2 + exit 1 +fi + +if [[ ! -f "${TLS_PATCH}" ]]; then + echo "missing ${TLS_PATCH}" >&2 + exit 1 +fi +if [[ ! -f "${CA_PEM}" ]]; then + echo "missing ${CA_PEM}" >&2 + exit 1 +fi + +echo "==> Checking namespace ${NAMESPACE}" +oc get namespace "${NAMESPACE}" >/dev/null + +echo "==> Patching ConfigMap argocd-tls-certs-cm" +oc get configmap argocd-tls-certs-cm -n "${NAMESPACE}" >/dev/null +oc patch configmap argocd-tls-certs-cm -n "${NAMESPACE}" --type=merge --patch-file="${TLS_PATCH}" + +echo "==> Appending CA to ConfigMap trusted-ca-bundle (key ca-bundle.crt)" +TMP="$(mktemp)" +PATCH="$(mktemp)" +trap 'rm -f "${TMP}" "${PATCH}"' EXIT +if oc get configmap trusted-ca-bundle -n "${NAMESPACE}" >/dev/null 2>&1; then + { + oc get configmap trusted-ca-bundle -n "${NAMESPACE}" \ + -o jsonpath='{.data.ca-bundle\.crt}' 2>/dev/null || true + echo + cat "${CA_PEM}" + } > "${TMP}" + { + echo "data:" + echo " ca-bundle.crt: |" + sed 's/^/ /' "${TMP}" + } > "${PATCH}" + oc patch configmap trusted-ca-bundle -n "${NAMESPACE}" --type=merge --patch-file="${PATCH}" +else + echo "ConfigMap trusted-ca-bundle not found in ${NAMESPACE}; creating with CA only" >&2 + oc create configmap trusted-ca-bundle -n "${NAMESPACE}" --from-file=ca-bundle.crt="${CA_PEM}" +fi + +echo "==> Restarting deployment/${REPO_SERVER}" +oc rollout restart "deployment/${REPO_SERVER}" -n "${NAMESPACE}" +oc rollout status "deployment/${REPO_SERVER}" -n "${NAMESPACE}" + +echo "Done. Verify with:" +echo " oc get configmap argocd-tls-certs-cm -n ${NAMESPACE} -o yaml | head" +echo " oc rollout status deployment/${REPO_SERVER} -n ${NAMESPACE}" +` + + tmpl, err := template.New("apply").Parse(script) + if err != nil { + return err + } + path := filepath.Join(outDir, "apply-trust-ca.sh") + f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o755) + if err != nil { + return err + } + defer f.Close() + + return tmpl.Execute(f, map[string]string{ + "Namespace": namespace, + "RepoServer": repoServer, + "HostsList": strings.Join(hosts, ", "), + }) +} + +func writeTrustCAReadme(outDir, namespace, repoServer string, hosts []string) error { + var b strings.Builder + b.WriteString("# Argo CD TLS trust (generated by patternizer airgap trust-ca)\n\n") + b.WriteString("Injects a CA into OpenShift GitOps / Argo CD so HTTPS Git hosts and OCI registries\n") + b.WriteString("with private or self-signed certificates can be verified by the repo-server.\n\n") + b.WriteString("## Prerequisites\n\n") + b.WriteString("- `oc` on PATH\n") + b.WriteString("- Working kubeconfig (`KUBECONFIG` or `~/.kube/config`)\n") + fmt.Fprintf(&b, "- Namespace `%s` exists with Argo CD installed\n", namespace) + fmt.Fprintf(&b, "- Deployment `%s` exists\n\n", repoServer) + b.WriteString("## Hosts\n\n") + for _, h := range hosts { + fmt.Fprintf(&b, "- `%s`\n", h) + } + b.WriteString("\n## Apply\n\n") + b.WriteString("On a host that can reach the cluster (e.g. bastion):\n\n") + b.WriteString("```bash\n") + b.WriteString("./apply-trust-ca.sh\n") + b.WriteString("```\n\n") + b.WriteString("This will:\n\n") + b.WriteString("1. Merge `argocd-tls-certs-patch.yaml` into `argocd-tls-certs-cm`\n") + b.WriteString("2. Append `trusted-ca-bundle-append.pem` into `trusted-ca-bundle`\n") + fmt.Fprintf(&b, "3. Roll out restart `deployment/%s`\n\n", repoServer) + b.WriteString("## Confirm\n\n") + b.WriteString("```bash\n") + fmt.Fprintf(&b, "oc get configmap argocd-tls-certs-cm -n %s -o yaml\n", namespace) + b.WriteString("# From repo-server, HTTPS to the trusted host should not return x509 errors\n") + fmt.Fprintf(&b, "oc rollout status deployment/%s -n %s\n", repoServer, namespace) + b.WriteString("```\n\n") + b.WriteString("ConfigMap keys must be hostnames only (no port). Re-run `patternizer airgap trust-ca`\n") + b.WriteString("to regenerate after rotating CAs.\n") + return os.WriteFile(filepath.Join(outDir, "README.md"), []byte(b.String()), 0o644) +} diff --git a/src/internal/airgap/trustca_test.go b/src/internal/airgap/trustca_test.go new file mode 100644 index 0000000..03aade7 --- /dev/null +++ b/src/internal/airgap/trustca_test.go @@ -0,0 +1,171 @@ +package airgap_test + +import ( + "os" + "path/filepath" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/validatedpatterns/patternizer/internal/airgap" +) + +const samplePEM = `-----BEGIN CERTIFICATE----- +MIIBkTCB+wIJAKHBfJ7eEXAMPLE +-----END CERTIFICATE----- +` + +var _ = Describe("NormalizeHost", func() { + DescribeTable("normalizes host inputs", + func(in, want string) { + got, err := airgap.NormalizeHost(in) + Expect(err).NotTo(HaveOccurred()) + Expect(got).To(Equal(want)) + }, + Entry("bare hostname", "gitea.example.com", "gitea.example.com"), + Entry("strips https URL", "https://gitea.example.com/org/repo", "gitea.example.com"), + Entry("strips port from host:port", "mirror.example.com:8443", "mirror.example.com"), + Entry("strips port from URL", "https://mirror.example.com:8443/v2/", "mirror.example.com"), + Entry("empty", "", ""), + ) + + It("rejects invalid hostnames", func() { + _, err := airgap.NormalizeHost("https://") + Expect(err).To(HaveOccurred()) + }) +}) + +var _ = Describe("NormalizeHosts", func() { + It("dedupes, splits commas, and sorts", func() { + hosts, err := airgap.NormalizeHosts([]string{ + "b.example.com", + "https://a.example.com/path", + "b.example.com,c.example.com:443", + }) + Expect(err).NotTo(HaveOccurred()) + Expect(hosts).To(Equal([]string{"a.example.com", "b.example.com", "c.example.com"})) + }) +}) + +var _ = Describe("LoadCAPEM", func() { + var tmp string + + BeforeEach(func() { + tmp = GinkgoT().TempDir() + }) + + It("loads a single PEM file", func() { + path := filepath.Join(tmp, "ca.crt") + Expect(os.WriteFile(path, []byte(samplePEM), 0o644)).To(Succeed()) + pem, err := airgap.LoadCAPEM(path) + Expect(err).NotTo(HaveOccurred()) + Expect(pem).To(ContainSubstring("BEGIN CERTIFICATE")) + Expect(pem).To(HaveSuffix("\n")) + }) + + It("concatenates .crt and .pem files from a directory in sorted order", func() { + dir := filepath.Join(tmp, "certs") + Expect(os.MkdirAll(dir, 0o755)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(dir, "b.pem"), []byte("-----BEGIN CERTIFICATE-----\nB\n-----END CERTIFICATE-----\n"), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(dir, "a.crt"), []byte("-----BEGIN CERTIFICATE-----\nA\n-----END CERTIFICATE-----\n"), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(dir, "readme.txt"), []byte("ignore"), 0o644)).To(Succeed()) + + pem, err := airgap.LoadCAPEM(dir) + Expect(err).NotTo(HaveOccurred()) + Expect(pem).To(HavePrefix("-----BEGIN CERTIFICATE-----\nA\n")) + Expect(pem).To(ContainSubstring("-----BEGIN CERTIFICATE-----\nB\n")) + }) + + It("rejects non-PEM content", func() { + path := filepath.Join(tmp, "not-a-cert.txt") + Expect(os.WriteFile(path, []byte("hello"), 0o644)).To(Succeed()) + _, err := airgap.LoadCAPEM(path) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("BEGIN CERTIFICATE")) + }) + + It("rejects an empty cert directory", func() { + dir := filepath.Join(tmp, "empty") + Expect(os.MkdirAll(dir, 0o755)).To(Succeed()) + _, err := airgap.LoadCAPEM(dir) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("no .crt or .pem")) + }) +}) + +var _ = Describe("GenerateTrustCA", func() { + It("writes patch YAML, PEM copy, apply script, and README", func() { + tmp := GinkgoT().TempDir() + caPath := filepath.Join(tmp, "ca.crt") + Expect(os.WriteFile(caPath, []byte(samplePEM), 0o644)).To(Succeed()) + outDir := filepath.Join(tmp, "out") + + res, err := airgap.GenerateTrustCA(&airgap.TrustCAOptions{ + CAPath: caPath, + Hosts: []string{"https://gitea.apps.example.com/repo", "mirror.example.com:8443"}, + Namespace: "vp-gitops", + OutputDir: outDir, + }) + Expect(err).NotTo(HaveOccurred()) + Expect(res.Namespace).To(Equal("vp-gitops")) + Expect(res.RepoServer).To(Equal("vp-gitops-repo-server")) + Expect(res.Hosts).To(Equal([]string{"gitea.apps.example.com", "mirror.example.com"})) + + Expect(filepath.Join(outDir, "argocd-tls-certs-patch.yaml")).To(BeAnExistingFile()) + Expect(filepath.Join(outDir, "trusted-ca-bundle-append.pem")).To(BeAnExistingFile()) + Expect(filepath.Join(outDir, "apply-trust-ca.sh")).To(BeAnExistingFile()) + Expect(filepath.Join(outDir, "README.md")).To(BeAnExistingFile()) + + patch, err := os.ReadFile(filepath.Join(outDir, "argocd-tls-certs-patch.yaml")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(patch)).To(ContainSubstring("name: argocd-tls-certs-cm")) + Expect(string(patch)).To(ContainSubstring("namespace: vp-gitops")) + Expect(string(patch)).To(ContainSubstring("gitea.apps.example.com: |")) + Expect(string(patch)).To(ContainSubstring("mirror.example.com: |")) + Expect(string(patch)).To(ContainSubstring("BEGIN CERTIFICATE")) + Expect(string(patch)).NotTo(ContainSubstring(":8443")) + + pemCopy, err := os.ReadFile(filepath.Join(outDir, "trusted-ca-bundle-append.pem")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(pemCopy)).To(ContainSubstring("BEGIN CERTIFICATE")) + + script, err := os.ReadFile(filepath.Join(outDir, "apply-trust-ca.sh")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(script)).To(HavePrefix("#!/usr/bin/env bash")) + Expect(string(script)).To(ContainSubstring("NAMESPACE=\"vp-gitops\"")) + Expect(string(script)).To(ContainSubstring("REPO_SERVER=\"vp-gitops-repo-server\"")) + Expect(string(script)).To(ContainSubstring("argocd-tls-certs-cm")) + Expect(string(script)).To(ContainSubstring("trusted-ca-bundle")) + Expect(string(script)).To(ContainSubstring("rollout restart")) + + info, err := os.Stat(filepath.Join(outDir, "apply-trust-ca.sh")) + Expect(err).NotTo(HaveOccurred()) + Expect(info.Mode().Perm() & 0o111).NotTo(BeZero()) + + readme, err := os.ReadFile(filepath.Join(outDir, "README.md")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(readme)).To(ContainSubstring("apply-trust-ca.sh")) + Expect(string(readme)).To(ContainSubstring("gitea.apps.example.com")) + }) + + It("defaults repo-server from namespace", func() { + tmp := GinkgoT().TempDir() + caPath := filepath.Join(tmp, "ca.pem") + Expect(os.WriteFile(caPath, []byte(samplePEM), 0o644)).To(Succeed()) + res, err := airgap.GenerateTrustCA(&airgap.TrustCAOptions{ + CAPath: caPath, + Hosts: []string{"git.example.com"}, + Namespace: "openshift-gitops", + OutputDir: filepath.Join(tmp, "out"), + RepoServer: "", + }) + Expect(err).NotTo(HaveOccurred()) + Expect(res.RepoServer).To(Equal("openshift-gitops-repo-server")) + }) + + It("requires ca and host", func() { + _, err := airgap.GenerateTrustCA(&airgap.TrustCAOptions{}) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("--ca")) + }) +}) diff --git a/src/internal/airgap/valuesload.go b/src/internal/airgap/valuesload.go new file mode 100644 index 0000000..685500f --- /dev/null +++ b/src/internal/airgap/valuesload.go @@ -0,0 +1,353 @@ +package airgap + +import ( + "fmt" + "os" + "path/filepath" + "sort" + "strings" + + "gopkg.in/yaml.v3" + + "github.com/validatedpatterns/patternizer/internal/types" +) + +// PatternInventory holds discovered pattern content for airgap init. +type PatternInventory struct { + PatternName string + ClusterGroupName string + Variant string + ClusterGroupChartVer string + HelmRepoURL string + ResolvedDir string + OutputDir string + Groups []ClusterGroupInventory + Operators []OperatorRef + RemoteCharts []RemoteChart + LocalCharts []LocalChart + AdditionalImages []string + ExtraOperators []OperatorCatalog + ExtraConfig *ExtraConfig + ManagedClusterGroups []string + Warnings []string +} + +// ClusterGroupInventory is one values-.yaml parse result. +type ClusterGroupInventory struct { + Name string + File string + Subscriptions []OperatorRef + Applications []types.Application + ImperativeImage string +} + +// RemoteChart is an OCI Helm chart referenced by chart/chartVersion. +type RemoteChart struct { + Name string + VersionConstraint string + ResolvedVersion string + PullSpec string + Images []string +} + +// LocalChart is a path-based Helm chart (from pattern applications or --extra-config helm.local). +type LocalChart struct { + Name string + Path string + Images []string +} + +// LoadPattern discovers and parses values files for the pattern. +func LoadPattern(opts *Options) (*PatternInventory, error) { + root := opts.PatternDir + globalPath := filepath.Join(root, "values-global.yaml") + data, err := os.ReadFile(globalPath) + if err != nil { + return nil, fmt.Errorf("read values-global.yaml: %w", err) + } + + var vg types.ValuesGlobal + if err := yaml.Unmarshal(data, &vg); err != nil { + return nil, fmt.Errorf("parse values-global.yaml: %w", err) + } + + variant := vg.Main.Variant + if variant == "" { + if legacy, ok := vg.Main.OtherFields["clusterGroupName"].(string); ok { + variant = legacy + } + } + + inv := &PatternInventory{ + PatternName: vg.Global.Pattern, + ClusterGroupName: variant, + Variant: variant, + ClusterGroupChartVer: vg.Main.MultiSourceConfig.ClusterGroupChartVersion, + HelmRepoURL: opts.HelmRepo, + } + if inv.ClusterGroupChartVer == "" { + inv.ClusterGroupChartVer = "0.9.*" + } + + groupFiles, err := discoverClusterGroupFiles(root, variant, opts) + if err != nil { + return nil, err + } + if len(groupFiles) == 0 { + return nil, fmt.Errorf("no cluster group values files found under %s (checked classic values-*.yaml and variants/%s/)", root, variant) + } + + loaded := map[string]bool{} + spokeDirs := allValuesSearchDirs(root, variant) + for _, gf := range groupFiles { + cgi, err := loadClusterGroupFile(gf) + if err != nil { + inv.Warnings = append(inv.Warnings, fmt.Sprintf("skip %s: %v", gf, err)) + continue + } + inv.Groups = append(inv.Groups, cgi) + loaded[cgi.Name] = true + for _, mc := range extractManagedClusterGroups(gf) { + inv.ManagedClusterGroups = appendUnique(inv.ManagedClusterGroups, mc) + } + } + + for _, mc := range inv.ManagedClusterGroups { + if loaded[mc] { + continue + } + spokePath := findValuesFile(spokeDirs, "values-"+mc+".yaml") + if spokePath == "" { + inv.Warnings = append(inv.Warnings, + fmt.Sprintf("managedClusterGroup %q referenced but values-%s.yaml not found in %v", mc, mc, spokeDirs)) + continue + } + cgi, err := loadClusterGroupFile(spokePath) + if err != nil { + inv.Warnings = append(inv.Warnings, fmt.Sprintf("skip spoke %s: %v", spokePath, err)) + continue + } + inv.Groups = append(inv.Groups, cgi) + loaded[cgi.Name] = true + inv.Warnings = append(inv.Warnings, + fmt.Sprintf("auto-included spoke cluster group %q from managedClusterGroups (%s)", mc, spokePath)) + } + + return inv, nil +} + +func primaryValuesSearchDirs(root, variant string) []string { + var dirs []string + if variant != "" { + vdir := filepath.Join(root, "variants", variant) + if info, err := os.Stat(vdir); err == nil && info.IsDir() { + dirs = append(dirs, vdir) + } + } + dirs = append(dirs, root) + return dirs +} + +func allValuesSearchDirs(root, variant string) []string { + dirs := primaryValuesSearchDirs(root, variant) + variantsRoot := filepath.Join(root, "variants") + entries, err := os.ReadDir(variantsRoot) + if err != nil { + return dirs + } + for _, e := range entries { + if !e.IsDir() { + continue + } + d := filepath.Join(variantsRoot, e.Name()) + already := false + for _, existing := range dirs { + if existing == d { + already = true + break + } + } + if !already { + dirs = append(dirs, d) + } + } + return dirs +} + +func findValuesFile(dirs []string, name string) string { + for _, d := range dirs { + p := filepath.Join(d, name) + if _, err := os.Stat(p); err == nil { + return p + } + } + return "" +} + +func discoverClusterGroupFiles(root, variant string, opts *Options) ([]string, error) { + wanted := map[string]bool{} + for _, g := range opts.ClusterGroups { + wanted[g] = true + } + + dirs := primaryValuesSearchDirs(root, variant) + seenGroups := map[string]string{} + var order []string + + for _, dir := range dirs { + entries, err := os.ReadDir(dir) + if err != nil { + continue + } + for _, e := range entries { + if e.IsDir() { + continue + } + name := e.Name() + if !strings.HasPrefix(name, "values-") || !strings.HasSuffix(name, ".yaml") { + continue + } + base := strings.TrimSuffix(strings.TrimPrefix(name, "values-"), ".yaml") + switch base { + case "global", "secret", "devel": + continue + } + if base == "standalone" && dir == root { + continue + } + if strings.HasSuffix(base, ".template") || strings.Contains(base, "secret") { + continue + } + if isVersionOverlay(base) { + continue + } + groupName := base + if len(wanted) > 0 && !wanted[groupName] { + continue + } + if _, exists := seenGroups[groupName]; exists { + continue + } + seenGroups[groupName] = filepath.Join(dir, name) + order = append(order, groupName) + } + } + + files := make([]string, 0, len(order)) + for _, g := range order { + files = append(files, seenGroups[g]) + } + sort.Strings(files) + return files, nil +} + +func isVersionOverlay(base string) bool { + parts := strings.SplitN(base, "-", 2) + if len(parts) < 2 { + return false + } + return parts[0] != "" && parts[0][0] >= '0' && parts[0][0] <= '9' +} + +func loadClusterGroupFile(path string) (ClusterGroupInventory, error) { + data, err := os.ReadFile(path) + if err != nil { + return ClusterGroupInventory{}, err + } + var vcg types.ValuesClusterGroup + if err := yaml.Unmarshal(data, &vcg); err != nil { + return ClusterGroupInventory{}, err + } + + name := vcg.ClusterGroup.Name + if name == "" { + base := filepath.Base(path) + name = strings.TrimSuffix(strings.TrimPrefix(base, "values-"), ".yaml") + } + + cgi := ClusterGroupInventory{ + Name: name, + File: path, + } + + for _, sub := range vcg.ClusterGroup.Subscriptions { + if sub.Name == "" { + continue + } + cgi.Subscriptions = append(cgi.Subscriptions, OperatorRef{ + Name: sub.Name, + Channel: sub.Channel, + Catalog: LookupCatalog(sub.Name), + Source: sub.Source, + }) + } + + for _, app := range vcg.ClusterGroup.Applications { + if app.Name == "" && app.Chart == "" && app.Path == "" { + continue + } + cgi.Applications = append(cgi.Applications, app) + } + + cgi.ImperativeImage = extractImperativeImage(data) + return cgi, nil +} + +func extractImperativeImage(raw []byte) string { + var doc map[string]interface{} + if err := yaml.Unmarshal(raw, &doc); err != nil { + return "" + } + cg, _ := doc["clusterGroup"].(map[string]interface{}) + if cg == nil { + return "" + } + imp, _ := cg["imperative"].(map[string]interface{}) + if imp == nil { + return "" + } + if img, ok := imp["image"].(string); ok { + return img + } + return "" +} + +func extractManagedClusterGroups(path string) []string { + data, err := os.ReadFile(path) + if err != nil { + return nil + } + var doc map[string]interface{} + if err := yaml.Unmarshal(data, &doc); err != nil { + return nil + } + cg, _ := doc["clusterGroup"].(map[string]interface{}) + if cg == nil { + return nil + } + mcg, _ := cg["managedClusterGroups"].(map[string]interface{}) + if mcg == nil { + return nil + } + var names []string + for _, v := range mcg { + entry, _ := v.(map[string]interface{}) + if entry == nil { + continue + } + if n, ok := entry["name"].(string); ok && n != "" { + names = append(names, n) + } + } + sort.Strings(names) + return names +} + +func appendUnique(slice []string, v string) []string { + for _, s := range slice { + if s == v { + return slice + } + } + return append(slice, v) +} diff --git a/src/internal/airgap/variants_test.go b/src/internal/airgap/variants_test.go new file mode 100644 index 0000000..530d1e6 --- /dev/null +++ b/src/internal/airgap/variants_test.go @@ -0,0 +1,75 @@ +package airgap_test + +import ( + "os" + "path/filepath" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/validatedpatterns/patternizer/internal/airgap" +) + +var _ = Describe("variants layout discovery", func() { + It("loads values from variants// when main.variant is set", func() { + tmp, err := os.MkdirTemp("", "variants-layout-") + Expect(err).NotTo(HaveOccurred()) + defer os.RemoveAll(tmp) + + Expect(os.MkdirAll(filepath.Join(tmp, "variants", "hub"), 0o755)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "values-global.yaml"), []byte(` +global: + pattern: variants-pattern +main: + variant: hub + multiSourceConfig: + enabled: true + clusterGroupChartVersion: "0.9.1" +`), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "variants", "hub", "values-hub.yaml"), []byte(` +clusterGroup: + name: hub + namespaces: + vault: + subscriptions: + acm: + name: advanced-cluster-management + channel: release-2.16 + applications: {} + managedClusterGroups: + region: + name: group-one +`), 0o644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(tmp, "variants", "hub", "values-group-one.yaml"), []byte(` +clusterGroup: + name: group-one + namespaces: + hello: + subscriptions: {} + applications: + demo: + name: demo + chart: golang-external-secrets + chartVersion: 0.1.5 +`), 0o644)).To(Succeed()) + + out := filepath.Join(tmp, "out") + opts := airgap.Options{ + PatternDir: tmp, + OutputDir: out, + SkipOCIResolve: true, + WriteValues: true, + } + inv, err := airgap.Init(&opts) + Expect(err).NotTo(HaveOccurred()) + Expect(inv.Variant).To(Equal("hub")) + Expect(inv.Groups).NotTo(BeEmpty()) + + names := map[string]bool{} + for _, g := range inv.Groups { + names[g.Name] = true + } + Expect(names["hub"]).To(BeTrue()) + Expect(names["group-one"]).To(BeTrue()) + }) +}) diff --git a/src/internal/embedded/resources/Makefile-common b/src/internal/embedded/resources/Makefile-common index d68f1f0..801acc1 100644 --- a/src/internal/embedded/resources/Makefile-common +++ b/src/internal/embedded/resources/Makefile-common @@ -63,3 +63,208 @@ argo-healthcheck: ## Checks if all argo applications are synced .PHONY: run-ci-tests run-ci-tests: ## To run ci-tests, set any needed env vars @$(ANSIBLE_RUN) rhvp.cluster_utils.run_ci_tests + +##@ Disconnected / Airgap Tasks +# Optional overrides for airgap-init (passed through when set). +# AIRGAP_OCP_VERSION adds mirror.platform channel stable- when set. +# AIRGAP_REGISTRY is optional for values-disconnected / mirror-commands helpers. +AIRGAP_OCP_VERSION ?= +AIRGAP_REGISTRY ?= +AIRGAP_OUTPUT_DIR ?= ./airgap-out +AIRGAP_PATTERNIZER_IMAGE ?= quay.io/validatedpatterns/patternizer +AIRGAP_DOWNLOAD_OC_MIRROR ?= false +AIRGAP_EXTRA_OPTS ?= +# install-airgap / airgap-bootstrap (values-secret airgap: section) +AIRGAP_BOOTSTRAP_DIR ?= ./airgap-bootstrap +AIRGAP_SECRETS_FILE ?= +AIRGAP_PATTERNIZER_BIN ?= +AIRGAP_WAIT_SECONDS ?= 1200 +AIRGAP_WAIT_INTERVAL ?= 15 +AIRGAP_ARGOCD_NS ?= vp-gitops +AIRGAP_REPO_SERVER ?= $(AIRGAP_ARGOCD_NS)-repo-server + +.PHONY: airgap-init +airgap-init: ## Generate oc-mirror imagesetconfig.yaml and disconnected init artifacts + @$(ANSIBLE_RUN) rhvp.cluster_utils.airgap_init \ + -e airgap_ocp_version="$(AIRGAP_OCP_VERSION)" \ + -e airgap_registry="$(AIRGAP_REGISTRY)" \ + -e airgap_output_dir=$(AIRGAP_OUTPUT_DIR) \ + -e airgap_patternizer_image=$(AIRGAP_PATTERNIZER_IMAGE) \ + -e airgap_download_oc_mirror=$(AIRGAP_DOWNLOAD_OC_MIRROR) \ + -e airgap_extra_opts="$(AIRGAP_EXTRA_OPTS)" + +# Resolve patternizer: explicit bin, PATH, ~/patternizer, or container image. +# Usage in a recipe: $(call AIRGAP_PATTERNIZER_EXEC,airgap bootstrap "$(CURDIR)" --output-dir "$(AIRGAP_BOOTSTRAP_DIR)") +define AIRGAP_PATTERNIZER_EXEC + set -euo pipefail; \ + BIN="$(AIRGAP_PATTERNIZER_BIN)"; \ + if [ -z "$$BIN" ] && command -v patternizer >/dev/null 2>&1; then BIN="$$(command -v patternizer)"; fi; \ + if [ -z "$$BIN" ] && [ -x "$(HOME)/patternizer" ]; then BIN="$(HOME)/patternizer"; fi; \ + if [ -z "$$BIN" ] && [ -x "$(HOME)/bin/patternizer" ]; then BIN="$(HOME)/bin/patternizer"; fi; \ + if [ -n "$$BIN" ]; then \ + echo "==> Using patternizer at $$BIN"; \ + "$$BIN" $(1); \ + elif command -v podman >/dev/null 2>&1; then \ + echo "==> Using podman $(AIRGAP_PATTERNIZER_IMAGE)"; \ + podman run --rm --network=host \ + -v "$(CURDIR):$(CURDIR):z" -v "$(HOME):$(HOME):z" -w "$(CURDIR)" \ + -e HOME="$(HOME)" \ + $(AIRGAP_PATTERNIZER_IMAGE) $(1); \ + elif command -v docker >/dev/null 2>&1; then \ + echo "==> Using docker $(AIRGAP_PATTERNIZER_IMAGE)"; \ + docker run --rm --network=host \ + -v "$(CURDIR):$(CURDIR):z" -v "$(HOME):$(HOME):z" -w "$(CURDIR)" \ + -e HOME="$(HOME)" \ + $(AIRGAP_PATTERNIZER_IMAGE) $(1); \ + else \ + echo "patternizer not found. Install it, set AIRGAP_PATTERNIZER_BIN, or install podman/docker." >&2; \ + exit 1; \ + fi +endef + +.PHONY: airgap-bootstrap +airgap-bootstrap: ## Generate airgap trust artifacts from values-secret airgap: section + @SECRETS_OPT=""; \ + if [ -n "$(AIRGAP_SECRETS_FILE)" ]; then SECRETS_OPT="--secrets-file $(AIRGAP_SECRETS_FILE)"; fi; \ + $(call AIRGAP_PATTERNIZER_EXEC,airgap bootstrap "$(CURDIR)" --output-dir "$(AIRGAP_BOOTSTRAP_DIR)" $$SECRETS_OPT) + +.PHONY: airgap-trust-pre +airgap-trust-pre: airgap-bootstrap ## Apply cluster Proxy CA trust (before pattern install) + @test -x "$(AIRGAP_BOOTSTRAP_DIR)/apply-airgap.sh" || { echo "missing $(AIRGAP_BOOTSTRAP_DIR)/apply-airgap.sh; run make airgap-bootstrap" >&2; exit 1; } + @AIRGAP_WAIT_SECONDS="$(AIRGAP_WAIT_SECONDS)" AIRGAP_WAIT_INTERVAL="$(AIRGAP_WAIT_INTERVAL)" \ + "$(AIRGAP_BOOTSTRAP_DIR)/apply-airgap.sh" pre + +.PHONY: airgap-wait-gitops +airgap-wait-gitops: ## Wait for GitOps namespace / Argo CD (runs while install is in progress) + @set -euo pipefail; \ + NS="$(AIRGAP_ARGOCD_NS)"; \ + REPO="$(AIRGAP_REPO_SERVER)"; \ + WAIT="$(AIRGAP_WAIT_SECONDS)"; \ + INTERVAL="$(AIRGAP_WAIT_INTERVAL)"; \ + if ! command -v oc >/dev/null 2>&1; then echo "oc is required" >&2; exit 1; fi; \ + echo "==> Waiting up to $${WAIT}s for namespace/$${NS}"; \ + elapsed=0; \ + while [ "$$elapsed" -lt "$$WAIT" ]; do \ + if oc get namespace "$$NS" >/dev/null 2>&1; then echo " namespace/$${NS} is present"; break; fi; \ + sleep "$$INTERVAL"; \ + elapsed=$$((elapsed + INTERVAL)); \ + if [ "$$elapsed" -ge "$$WAIT" ]; then echo "timed out waiting for namespace/$${NS}" >&2; exit 1; fi; \ + done; \ + echo "==> Waiting up to $${WAIT}s for configmap/argocd-tls-certs-cm in $${NS} (optional if only repo-creds)"; \ + elapsed=0; \ + while [ "$$elapsed" -lt "$$WAIT" ]; do \ + if oc get configmap argocd-tls-certs-cm -n "$$NS" >/dev/null 2>&1; then \ + echo " configmap/argocd-tls-certs-cm is present"; \ + break; \ + fi; \ + sleep "$$INTERVAL"; \ + elapsed=$$((elapsed + INTERVAL)); \ + if [ "$$elapsed" -ge "$$WAIT" ]; then \ + echo " warning: argocd-tls-certs-cm not ready yet; airgap-trust-post will keep waiting" >&2; \ + break; \ + fi; \ + done; \ + echo "==> Waiting up to $${WAIT}s for deployment/$${REPO} in $${NS} (optional)"; \ + elapsed=0; \ + while [ "$$elapsed" -lt "$$WAIT" ]; do \ + if oc get deployment "$$REPO" -n "$$NS" >/dev/null 2>&1; then \ + echo " deployment/$${REPO} is present"; \ + break; \ + fi; \ + sleep "$$INTERVAL"; \ + elapsed=$$((elapsed + INTERVAL)); \ + if [ "$$elapsed" -ge "$$WAIT" ]; then \ + echo " warning: deployment/$${REPO} not ready yet; post will keep waiting" >&2; \ + break; \ + fi; \ + done; \ + echo "==> GitOps wait complete; running airgap-trust-post next" + +.PHONY: airgap-trust-post +airgap-trust-post: ## Apply Argo CD TLS + repo-creds (waits for vp-gitops / Argo CD resources) + @test -x "$(AIRGAP_BOOTSTRAP_DIR)/apply-airgap.sh" || { echo "missing $(AIRGAP_BOOTSTRAP_DIR)/apply-airgap.sh; run make airgap-bootstrap first" >&2; exit 1; } + @AIRGAP_WAIT_SECONDS="$(AIRGAP_WAIT_SECONDS)" AIRGAP_WAIT_INTERVAL="$(AIRGAP_WAIT_INTERVAL)" \ + "$(AIRGAP_BOOTSTRAP_DIR)/apply-airgap.sh" post + +.PHONY: airgap-restore-proxy +airgap-restore-proxy: ## Restore proxy/cluster from airgap bootstrap backup + @test -x "$(AIRGAP_BOOTSTRAP_DIR)/restore-proxy-cluster.sh" || { echo "missing $(AIRGAP_BOOTSTRAP_DIR)/restore-proxy-cluster.sh" >&2; exit 1; } + @"$(AIRGAP_BOOTSTRAP_DIR)/restore-proxy-cluster.sh" + +# Stop a background make install (and its descendants). Used by Ctrl-C cleanup and airgap-abort. +# Args: $1 = root pid. Sends TERM to the process group / tree, waits briefly, then KILL. +define AIRGAP_STOP_INSTALL + pid="$(1)"; \ + if [ -n "$$pid" ] && kill -0 "$$pid" 2>/dev/null; then \ + echo "==> Stopping background make install (pid $$pid)"; \ + kill -TERM -"$$pid" 2>/dev/null || kill -TERM "$$pid" 2>/dev/null || true; \ + if command -v pkill >/dev/null 2>&1; then pkill -TERM -P "$$pid" 2>/dev/null || true; fi; \ + i=0; \ + while [ "$$i" -lt 8 ] && kill -0 "$$pid" 2>/dev/null; do sleep 1; i=$$((i + 1)); done; \ + if kill -0 "$$pid" 2>/dev/null; then \ + echo "==> Force-killing background make install (pid $$pid)"; \ + kill -KILL -"$$pid" 2>/dev/null || kill -KILL "$$pid" 2>/dev/null || true; \ + if command -v pkill >/dev/null 2>&1; then pkill -KILL -P "$$pid" 2>/dev/null || true; fi; \ + fi; \ + wait "$$pid" 2>/dev/null || true; \ + fi +endef + +.PHONY: airgap-abort +airgap-abort: ## Stop a background install-airgap session (reads airgap-bootstrap/install.pid) + @set -euo pipefail; \ + PIDFILE="$(AIRGAP_BOOTSTRAP_DIR)/install.pid"; \ + if [ ! -f "$$PIDFILE" ]; then \ + echo "No $$PIDFILE - nothing to abort (is install-airgap running?)" >&2; \ + exit 1; \ + fi; \ + INSTALL_PID=$$(tr -d '[:space:]' < "$$PIDFILE"); \ + $(call AIRGAP_STOP_INSTALL,$$INSTALL_PID); \ + rm -f "$$PIDFILE"; \ + echo "==> Aborted. Log: $(AIRGAP_BOOTSTRAP_DIR)/install.log" + +.PHONY: install-airgap +install-airgap: ## Disconnected install: Proxy CA, background install, wait for GitOps, post TLS/creds, then await install + @$(MAKE) airgap-trust-pre + @set -euo pipefail; \ + mkdir -p "$(AIRGAP_BOOTSTRAP_DIR)"; \ + LOG="$(AIRGAP_BOOTSTRAP_DIR)/install.log"; \ + PIDFILE="$(AIRGAP_BOOTSTRAP_DIR)/install.pid"; \ + echo "==> Starting make install in background (log: $$LOG)"; \ + echo " (post trust runs as soon as vp-gitops is ready — install need not finish first)"; \ + echo " Ctrl-C or: make airgap-abort"; \ + setsid $(MAKE) install >"$$LOG" 2>&1 < /dev/null & \ + INSTALL_PID=$$!; \ + echo "$$INSTALL_PID" > "$$PIDFILE"; \ + cleanup() { \ + trap - INT TERM; \ + $(call AIRGAP_STOP_INSTALL,$$INSTALL_PID); \ + rm -f "$$PIDFILE"; \ + }; \ + trap cleanup INT TERM; \ + if ! $(MAKE) airgap-wait-gitops; then \ + cleanup; \ + echo "airgap-wait-gitops failed; see $$LOG" >&2; \ + exit 1; \ + fi; \ + if ! $(MAKE) airgap-trust-post; then \ + trap - INT TERM; \ + echo "airgap-trust-post failed; leaving install running (pid $$INSTALL_PID). See $$LOG" >&2; \ + echo " Re-run: make airgap-trust-post" >&2; \ + echo " Or abort: make airgap-abort" >&2; \ + exit 1; \ + fi; \ + echo "==> Argo CD trust applied; waiting for background make install (pid $$INSTALL_PID)..."; \ + set +e; \ + wait $$INSTALL_PID; \ + INSTALL_RC=$$?; \ + set -e; \ + trap - INT TERM; \ + rm -f "$$PIDFILE"; \ + if [ "$$INSTALL_RC" -ne 0 ]; then \ + echo "make install failed with exit $$INSTALL_RC; see $$LOG" >&2; \ + tail -n 40 "$$LOG" >&2 || true; \ + exit $$INSTALL_RC; \ + fi; \ + echo "install-airgap complete. Proxy backup: $(AIRGAP_BOOTSTRAP_DIR)/backup/proxy-cluster.yaml"; \ + echo " install log: $$LOG" diff --git a/src/internal/embedded/resources/pattern.sh b/src/internal/embedded/resources/pattern.sh index 7d8937c..e57dacc 100755 --- a/src/internal/embedded/resources/pattern.sh +++ b/src/internal/embedded/resources/pattern.sh @@ -70,6 +70,7 @@ fi if [ -n "${PATTERN_DISCONNECTED_HOME:-}" ]; then PATTERN_UTILITY_CONTAINER="${PATTERN_DISCONNECTED_HOME}/utility-container" PATTERN_INSTALL_CHART="oci://${PATTERN_DISCONNECTED_HOME}/pattern-install" + export PATTERN_UTILITY_CONTAINER PATTERN_INSTALL_CHART echo "PATTERN_DISCONNECTED_HOME is set to ${PATTERN_DISCONNECTED_HOME}" echo "Setting the following variables:" echo " PATTERN_UTILITY_CONTAINER: ${PATTERN_UTILITY_CONTAINER}" diff --git a/src/internal/embedded/resources/values-secret.yaml.template b/src/internal/embedded/resources/values-secret.yaml.template index 2104801..6f12ac1 100644 --- a/src/internal/embedded/resources/values-secret.yaml.template +++ b/src/internal/embedded/resources/values-secret.yaml.template @@ -16,3 +16,70 @@ secrets: [] # onMissingValue: generate # - name: bar # onMissingValue: generate + +# --------------------------------------------------------------------------- +# Airgap / disconnected GitOps trust (patternizer airgap bootstrap ONLY). +# Not loaded into Vault by make load-secrets. Copy this file to +# ~/values-secret-.yaml (or keep paths local) and fill in real paths. +# +# UX: +# Copy this template to ~/values-secret-.yaml and fill in airgap: paths. +# ./pattern.sh make install-airgap +# +# Or manually: +# patternizer airgap bootstrap /path/to/pattern --output-dir ./airgap-bootstrap +# ./airgap-bootstrap/apply-airgap.sh pre # before install +# ./pattern.sh make install +# ./airgap-bootstrap/apply-airgap.sh post # after vp-gitops exists +# +# clusterProxy: CA for patterns-operator git clone (Proxy/user-ca-bundle) +# argocdTLS: CA for Argo CD (argocd-tls-certs-cm + trusted-ca-bundle) +# registries: OCI/Helm mirror credentials for Argo CD repo-creds +# --------------------------------------------------------------------------- +# airgap: +# argocdNamespace: vp-gitops +# patternsOperatorNamespace: patterns-operator +# certificates: +# - name: gitea +# fields: +# - name: host +# value: gitea.airgap.rhuk.lab +# - name: ca +# path: /home/fedora/gitea_container_certs/cert.pem +# - name: clusterProxy +# value: "true" +# - name: argocdTLS +# value: "true" +# - name: mirror-registry +# fields: +# - name: host +# value: mirror.airgap.rhuk.lab +# - name: ca +# path: /etc/pki/ca-trust/source/anchors/quay-rootCA.pem +# - name: clusterProxy +# value: "true" +# - name: argocdTLS +# value: "true" +# registries: +# - name: mirror +# fields: +# - name: url +# value: mirror.airgap.rhuk.lab:8443 +# - name: dockerconfig +# path: ~/.docker/config.json +# # Or username + password file instead of dockerconfig: +# # - name: username +# # value: init +# # - name: password +# # path: /path/to/registry.password +# +# # Disconnected pattern values (rendered into values-global.yaml + variants) +# registry: mirror.airgap.rhuk.lab:8443 +# ocpVersion: "4.22" +# # Or declare CatalogSources explicitly (overrides ocpVersion derivation): +# # catalogSources: +# # redhat: cs-redhat-operator-index-v4-22 +# # community: cs-community-operator-index-v4-22 +# # Or discover from the cluster (requires oc / kubeconfig): +# # discoverCatalogs: true +# # renderValues: true # default when registry/ocpVersion/catalogSources set