harmony/ROADMAP/fleet_platform/public_e2e_test_cluster.md

Public Fleet E2E environment

Goal

Run public-facing Fleet regression tests directly on the production-adjacent OKD cluster. Keep the environment definition, identities, context, and CI workflow in a standalone private repository checked out under private_repos/. The public Harmony repository contains only reusable Scores, Refs, lifecycle code, and generic test harnesses.

Build and publish run on the same CI runner. Deployment receives only digest-pinned image references published to an OCI registry.

Private repository

The private repository owns:

Cargo.toml
src/main.rs                    # compiled Fleet deployment context
.github/workflows/e2e.yml       # or the equivalent CI system
README.md                       # operator setup and recovery

It is checked out beside Harmony under private_repos/, which the public repository ignores. No customer or private project identifier belongs in a tracked Harmony file, fixture, log sample, or test name.

The private deploy binary defines its contexts as typed Rust values. Contexts are compiled into the binary; loading additional context definitions at runtime is deferred until a real requirement justifies that interface.

let context = Context {
    name: context_name!("e2e"),
    namespace: "fleet-e2e".parse()?,
    spec: ContextSpec::Remote(RemoteContext {
        registry: oci_registry!("registry.example.com"),
        repository: oci_repository!("harmony/e2e"),
        domain: domain!("e2e.example.com"),
        image_pull_secret: Some("registry-pull".parse()?),
        access: OpenBaoClusterAccess {
            namespace: openbao_namespace!("example"),
            url: http_url!("https://secrets.nationtech.io"),
            role: OpenBaoRoleName::new("example-ci")?,
            zitadel_url: http_url!("https://identity.example.com"),
            zitadel_audience: OidcAudience::new("example")?,
        },
    }),
};
deploy_fleet_with_context(context).await

Concrete names and identity details stay private.

Context type design

ContextSpec encodes the access-mode choice instead of representing it with unrelated optional fields:

pub enum ContextSpec {
    Local(LocalContext),
    Remote(RemoteContext),
}

RemoteContext carries required registry, domain, and OpenBao values directly; its image-pull Secret is optional for public registries. LocalContext carries only its k3d settings. This makes impossible combinations unrepresentable and removes checks such as "exactly one of autoprovision, k3d, or OpenBao must be set."

Use existing domain types where they are already correct:

• Profile remains an enum derived from the local or remote context variant.
• The context's namespace places every tenant Fleet component and custom resource in one Kubernetes namespace.
• Kubernetes namespaces, Secret names, and other DNS-label resource names use K8sName.
• IP and MAC literals continue using the existing Harmony macros.

Add types only for distinct grammars or values that are easy to transpose:

• ContextName: CLI selection key. It is not implicitly a Kubernetes name.
• DomainName: normalized DNS name without scheme, path, port, or wildcard.
• HttpUrl: HTTP(S)-only remote URL. The existing Url also permits local folders and is too broad for OpenBao and Zitadel endpoints.
• OciRegistry: registry authority with optional port, but no repository path.
• OciRepository: validated OCI repository path.
• OpenBaoNamespace: a safe OpenBao path namespace, distinct from K8sName.
• OpenBaoRoleName: auth-role identifier, distinct from the namespace.
• OidcAudience: non-empty OIDC audience, kept semantically distinct even though the protocol permits several string shapes.

Do not add a macro for every newtype. Literal macros are justified for the grammars where compile-time diagnostics materially help: DNS names, HTTP URLs, OCI registries/repositories, Kubernetes names, and OpenBao path namespaces. Opaque identifiers can use typed constructors.

The runtime parser and literal macro must share one validator in harmony_types; the macro must not copy the grammar. harmony_macros already depends on harmony_types, so it can validate the literal during expansion and emit the typed value. Each type needs accepted/rejected unit cases plus compile-fail macro tests.

Application integration

The lifecycle entry point receives a compiled catalog:

harmony_cli::app::app_main(FleetApp, contexts).await

--context e2e and HARMONY_CONTEXT=e2e select only from contexts compiled into that deploy binary. The public Fleet binary can compile its local context; the private binary compiles the remote E2E context. Both use the same exported FleetApp and Scores.

CI trust bootstrap

The CI runner receives one secret:

HARMONY_ZITADEL_KEY_JSON

It is the JSON key for a dedicated Zitadel machine identity. Harmony uses that identity to obtain an OpenBao token from secrets.nationtech.io. ConfigClient then loads the namespace-scoped kubeconfig and deployment secrets. The runner holds no standing kubeconfig, OpenBao token, registry password, or application secret.

Required server-side setup:

• Create a dedicated Zitadel machine identity for E2E CI.
• Grant only the role accepted by the E2E OpenBao auth role.
• Create an OpenBao policy restricted to the E2E config and secret paths.
• Store ClusterAccess with the namespace-scoped kubeconfig.
• Store registry credentials and deploy inputs under the E2E namespace.
• Add the machine key JSON as the CI repository's only secret.
• Compile the typed e2e context into the private deploy binary.
• Select it with HARMONY_CONTEXT=e2e.

Known implementation gap

Cluster access already follows this flow. Image publication currently reads REGISTRY_USER and REGISTRY_TOKEN directly from the environment. Before the single-secret workflow is complete, registry authentication must be loaded through the resolved AppContext and ConfigClient like the other secrets.

The context should resolve once at the start of the combined build-and-publish job. Build itself remains credential-free; publish consumes the registry credential fetched from OpenBao.

Phase 1: cluster and identity

• Add the context domain types and their shared validators to harmony_types.
• Add only the justified literal macros to harmony_macros, including compile-fail tests.
• Replace TOML discovery with a caller-supplied ContextCatalog.
• Export the canonical FleetApp composition so the private deploy binary does not duplicate it.
• Separate cluster-owned Fleet CRDs from tenant operator releases.
• Create a dedicated E2E namespace or namespace prefix on the public OKD cluster.
• Configure namespace-scoped RBAC for the OpenBao-brokered kubeconfig.
• Add public DNS and exact-host OKD Routes for E2E services.
• Verify the CI machine identity can read ClusterAccess and no unrelated OpenBao path.
• Run a read-only context check from CI with only HARMONY_ZITADEL_KEY_JSON present.

Phase 2: OCI publication

• Move registry credentials behind ConfigClient.
• Create a disposable registry project or repository scope for E2E images.
• Run build and publish in one CI job.
• Verify the pushed manifest digest matches the digest returned by build.
• Pull and run that exact digest in a fresh namespace.
• Apply retention rules so E2E images expire automatically.

The production publication path remains the same operation against the production registry; only the context and credential policy differ.

Phase 3: NATS regression

• Deploy the existing certificate and mTLS-capable NATS supercluster Scores unchanged before modifying their APIs.
• Exercise public HTTPS/WebSocket and native TLS/mTLS endpoints through OKD Routes.
• Assert cluster formation, authenticated traffic, SNI routing, reconnect after pod restart, and certificate validation from outside the cluster.
• Record failures as regressions against the existing module before changing Fleet composition.

Phase 4: Fleet lifecycle

• Build and publish the operator and callout images in one job.
• Pass their digest-pinned references to deploy.
• Deploy Fleet through the private e2e context using production Scores and typed Refs.
• Enroll an external test agent through the public NATS endpoint.
• Apply, update, and delete a Podman deployment and verify reflected state.
• Exercise Zitadel group changes and OpenBao secret access.
• Verify failed reconciliation recovers after the dependency is restored.
• Remove per-run resources and verify no desired-state or grant residue remains.

Guardrails

• Use a dedicated machine identity, OpenBao role, registry scope, and Kubernetes RBAC grant for E2E.
• Keep every concrete private identifier in the private repository or OpenBao.
• Put a TTL label on per-run namespaces and clean abandoned runs automatically.
• Never print the machine key, OpenBao token, kubeconfig, or registry credential.
• Split CI into identity, registry, NATS, and Fleet stages so failures identify one boundary.