Lets a namespace-scoped deploy pull private images declaratively, derived from
context (no procedural step):
- harmony: RegistryPullSecretScore — builds a kubernetes.io/dockerconfigjson
Secret in the namespace from pull-only registry creds (idempotent, namespaced).
- harmony_app: DeployConfig.image_pull_secrets renders into each pod's
spec.template.spec.imagePullSecrets (mirrors extra_env / secret_file_mounts).
- harmony_config: get_or_prompt now fills only the MISSING fields of a partial
stored value (e.g. when a Secret struct gains a field) and persists the merge,
instead of failing or re-prompting everything. Prompting is behind a
FieldPrompter seam so the flow is unit-tested (no terminal).
Storage durability is provided by the backing layer (Ceph); CNPG-level replication added ops/cost (and a stuck second replica on single-node clusters) without buying availability. Default instances 2 -> 1.
Two re-run hazards under the install/converge model:
1. Masterkey was generated fresh every run and only survived via the 409 on the
k8s Secret — a deleted Secret would mint a new key and orphan the existing
Zitadel DB. Now resolve it durably: prefer the in-cluster Secret (matches the
running DB), else the value persisted in harmony_secret, else generate; then
mirror into harmony_secret so a namespace reset reuses it. (New ZitadelMasterkey
persisted secret, mirroring ZitadelAdmin.)
2. The chart's `zitadel-setup` init Job is a pre-upgrade hook that fails when
re-run on `helm upgrade`. Set install_only=true: once the release exists, skip
the helm op entirely. Zitadel is install-once + stateful; re-deploys converge
via ZitadelSetupScore, not by re-running the chart.
Add K8sClient::ensure_namespace(ns): a GET decides — no-op if the namespace
already exists (e.g. provisioned by platform onboarding), create only when
genuinely missing. Never patches the cluster-scoped Namespace object, so a
namespace-scoped tenant deployer doesn't need cluster RBAC.
Route the tenant-deploy path through it and stop helm from patching the
namespace:
- postgresql K8sPostgreSQLScore: use ensure_namespace (was inline).
- zitadel: use ensure_namespace; HelmChartScore create_namespace=false.
- app ComposeAppScore: ensure_namespace before install; create_namespace=false.
Fixes the prod deploy 403 "cannot patch resource namespaces" under a
namespace-scoped service account.
ZitadelScore applied the (cluster-scoped) Namespace object unconditionally, so a
namespace-scoped tenant deployer 403'd on "patch namespaces" even though the
namespace was already provisioned by platform onboarding. Mirror the Postgres
module: check namespace_exists (a GET, which the deployer is allowed) and only
create when genuinely missing.
A namespace-scoped deployer (e.g. a tenant CI service account) cannot list pods
cluster-wide, so the cert-manager and CNPG operator readiness checks 403'd and
failed the whole deploy — pushing operators to grant the tenant cluster-admin,
which is a security anti-pattern.
Both checks now treat a 403 Forbidden as "this cluster-wide platform operator is
managed out-of-band": log a clean info ("Skipped … due to restricted access")
and continue, assuming the operator is present. Non-403 errors still propagate.
Hosting client CI runners + the GitHub-App auth is platform/operations, to live
in a NationTech private repo (TBD by the operator), not in harmony. Keep the
reusable module; reference the auth setup generically instead of by repo path.
NationTech-hosted model: one NationTech-owned GitHub App (org permissions
Self-hosted runners: R/W + Metadata: RO), installed once per client org — that
single authorization covers all the org's repos. Per client: a Secret with that
org's installation ID + a runner scale set whose githubConfigUrl is the org URL.
Cited from the official GitHub ARC + Apps docs. Point the module's example/docs
at the org URL.
New modules::github_runner. GithubRunnerScore yields two HelmChartScores — the
ARC controller (gha-runner-scale-set-controller) and a runner scale set
(gha-runner-scale-set) — as OCI charts. dind enabled (so docker build works),
configurable resource limits applied to both the runner and dind containers, and
the GitHub credential referenced by a pre-created Secret name (kept out of code).
The scale-set name becomes the workflow runs-on label.
ZitadelJwtBearer posted Zitadel's access_token to OpenBao's jwt/login. Zitadel
issues opaque access tokens by default, so OpenBao rejected it with "compact JWS
format must have three parts". The token endpoint already returns an id_token (a
JWT) because build_scope requests `openid`; use it. It carries the service user
in `sub` and the project id in `aud` (the latter via the project🆔aud scope) —
exactly what the OpenBao JWT role binds on. No Zitadel-side change needed (the
manual id_token exchange already proved this works).
build.args support (harmony_app):
- Parse `build.args` (advanced build form) into ComposeService.build_args,
mirroring parse_env (Simple/List/KvPair).
- build_images() forwards them as `docker build --build-arg K=V`, so an app can
bake per-environment build-time config (e.g. Quarkus profile, a Vite API URL).
Zitadel readiness (harmony):
- ZitadelSetupScore failed at 0.0s on a cold cluster ("no Running pod for
service 'zitadel'") because it checked once, before the Zitadel Deployment
pod was Running. Poll for a Running pod (300s/3s) so a one-shot `ship`
converges instead of needing a re-run.
