harmony/adr/018-Template-Hydration-For-Workload-Deployment.md

Architecture Decision Record: Template Hydration for Kubernetes Manifest Generation

Initial Author: Jean-Gabriel Gill-Couture & Sylvain Tremblay

Initial Date: 2025-01-23

Last Updated Date: 2025-01-23

Status

Implemented

Context

Harmony's philosophy is built on three guiding principles: Infrastructure as Resilient Code, Prove It Works — Before You Deploy, and One Unified Model. Our goal is to shift validation and verification as left as possible—ideally to compile time—rather than discovering errors at deploy time.

After investigating a few approaches such as compile-checked Askama templates to generate Kubernetes manifests for Helm charts, we found again that this approach suffered from several fundamental limitations:

• Late Validation: Typos in template syntax or field names are only discovered at deployment time, not during compilation. A mistyped metadata.name won't surface until Helm attempts to render the template.
• Brittle Maintenance: Templates are string-based with limited IDE support. Refactoring requires grep-and-replace across YAML-like template files, risking subtle breakage.
• Hard-to-Test Logic: Testing template output requires mocking the template engine and comparing serialized strings rather than asserting against typed data structures.
• No Type Safety: There is no guarantee that the generated YAML will be valid Kubernetes resources without runtime validation.

We also faced a strategic choice around Helm: use it as both templating engine and packaging mechanism, or decouple these concerns. While Helm's ecosystem integration (Harbor, ArgoCD, OCI registry support) is valuable, the Jinja-like templating is at odds with Harmony's "code-first" ethos.

Decision

We will adopt the Template Hydration Pattern—constructing Kubernetes manifests programmatically using strongly-typed kube-rs objects, then serializing them to YAML files for packaging into Helm charts.

Specifically:

• Write strongly typed k8s_openapi Structs: All Kubernetes resources (Deployment, Service, ConfigMap, etc.) will be constructed using the typed structs generated by k8s_openapi.
• Direct Serialization to YAML: Rather than rendering templates, we use serde_yaml::to_string() to serialize typed objects directly into YAML manifests. This way, YAML is only used as a data-transfer format and not a templating/programming language - which it is not.
• Helm as Packaging-Only: Helm's role is reduced to packaging pre-rendered templates into a tarball and pushing to OCI registries. No template rendering logic resides within Helm.
• Ecosystem Preservation: The generated Helm charts remain fully compatible with Harbor, ArgoCD, and any Helm-compatible tool—the only difference is that the templates/ directory contains static YAML files.

The implementation in backend_app.rs demonstrates this pattern:

let deployment = Deployment {
    metadata: ObjectMeta {
        name: Some(self.name.clone()),
        labels: Some([("app.kubernetes.io/name".to_string(), self.name.clone())].into()),
        ..Default::default()
    },
    spec: Some(DeploymentSpec { /* ... */ }),
    ..Default::default()
};

let deployment_yaml = serde_yaml::to_string(&deployment)?;
fs::write(templates_dir.join("deployment.yaml"), deployment_yaml)?;

Rationale

Aligns with "Infrastructure as Resilient Code"

Harmony's first principle states that infrastructure should be treated like application code. By expressing Kubernetes manifests as Rust structs, we gain:

• Refactorability: Rename a label and the compiler catches all usages.
• IDE Support: Autocomplete for all Kubernetes API fields; documentation inline.
• Code Navigation: Jump to definition shows exactly where a value comes from.

Achieves "Prove It Works — Before You Deploy"

The compiler now validates that:

• All required fields are populated (Rust's Option type prevents missing fields).
• Field types match expectations (ports are integers, not strings).
• Enums contain valid values (e.g., ServiceType::ClusterIP).

This moves what was runtime validation into compile-time checks, fulfilling the "shift left" promise.

Enables True Unit Testing

Developers can now write unit tests that assert directly against typed objects:

let deployment = create_deployment(&app);
assert_eq!(deployment.spec.unwrap().replicas.unwrap(), 3);
assert_eq!(deployment.metadata.name.unwrap(), "my-app");

No string parsing, no YAML serialization, no fragile assertions against rendered output.

Preserves Ecosystem Benefits

By generating standard Helm chart structures, Harmony retains compatibility with:

• OCI Registries (Harbor, GHCR): helm push works exactly as before.
• ArgoCD: Syncs and manages releases using the generated charts.
• Existing Workflows: Teams already consuming Helm charts see no change.

The Helm tarball becomes a "dumb pipe" for transport, which is arguably its ideal role.

Consequences

Positive

• Compile-Time Safety: A broad class of errors (typos, missing fields, type mismatches) is now caught at build time.
• Better Developer Experience: IDE autocomplete, inline documentation, and refactor support significantly reduce the learning curve for Kubernetes manifests.
• Testability: Unit tests can validate manifest structure without integration or runtime checks.
• Auditability: The source-of-truth for manifests is now pure Rust—easier to review in pull requests than template logic scattered across files.
• Future-Extensibility: CustomResources (CRDs) can be supported via kopium-generated Rust types, maintaining the same strong typing.

Negative

• API Schema Drift: Kubernetes API changes require regenerating k8s_openapi types and updating code. A change in a struct field will cause the build to fail—intentionally, but still requiring the pipeline to be updated.
• Verbosity: Typed construction is more verbose than the equivalent template. Builder patterns or helper functions will be needed to keep code readable.
• Learning Curve: Contributors must understand both the Kubernetes resource spec and the Rust type system, rather than just YAML.
• Debugging Shift: When debugging generated YAML, you now trace through Rust code rather than template files—more precise but different mental model.

Alternatives Considered

1. Enhance Askama with Compile-Time Validation

Pros: Stay within familiar templating paradigm; minimal code changes. Cons: Rust's type system cannot fully express Kubernetes schema validation without significant macro boilerplate. Errors would still surface at template evaluation time, not compilation.

2. Use Helm SDK Programmatically (Go)

Pros: Direct access to Helm's template engine; no YAML serialization step. Cons: Would introduce a second language (Go) into a Rust codebase, increasing cognitive load and compilation complexity. No improvement in compile-time safety.

3. Raw YAML String Templating (Manual)

Pros: Maximum control; no external dependencies. Cons: Even more error-prone than Askama; no structure validation; string concatenation errors abound.

4. Use Kustomize for All Manifests

Pros: Declarative overlays; standard tool. Cons: Kustomize is itself a layer over YAML templates with its own DSL. It does not provide compile-time type safety and would require externalizing manifest management outside Harmony's codebase.

Note that this template hydration architecture still allows to override templates with tools like kustomize when required

Additional Notes

Scalability to Future Topologies

The Template Hydration pattern enables future Harmony architectures to generate manifests dynamically based on topology context. For example, a CostTopology might adjust resource requests based on cluster pricing, manipulating the typed Deployment::spec directly before serialization.

Implementation Status

As of this writing, the pattern is implemented for BackendApp deployments (backend_app.rs). The next phase is to extend this pattern across all application modules (webapp.rs, etc.) and to standardize on this approach for any new implementations.