harmony/docs/adr/drafts/012-1-release-architecture.md

ADR-012 follow-up: Release Architecture Mechanism (Capability-Driven Build, Package, Push)

Initial Author: Jean-Gabriel Gill-Couture (drafted with Claude)

Initial Date: 2026-05-27

Last Updated Date: 2026-05-27

Status

Draft. Clarification + mechanism for ADR-012, not a separate decision. ADR-012 (Project Delivery Automation, 2025-06-04) locked the intent: application-level Scores (LAMPScore-style) drive an opinionated pipeline — empty-check → package&publish → deploy staging → sanity-check → deploy production → sanity-check — with CD tools (Argo / Flux) activated by default and called via API; the same harmony binary runs the project's module locally and in CI.

This document supplements that intent with the concrete mechanism: how steps 2, 3, 5 (package&publish, deploy staging, deploy production) actually compile and execute against today's Score-Topology-Capability primitives, and what to delete from the two interim attempts (modules/application + the per-component harmony-fleet-release binary) when the mechanism lands.

Companion references:

• ADR-023 (Deploy Architecture, Accepted) — the DeployScore proposed here plugs into the harmony_cli::run flow and *-deploy crate pattern ADR-023 mandates; the smoke-test contract (ADR-023 principle 4) is how steps 4 and 6 of ADR-012 block.
• ADR-024 draft (Fleet Platform Capability Decomposition) — same decomposition shape, applied to release: framework-level capability traits, not per-app methods.
• ADR-003 — capability traits represent industry concepts, not tools. Every new capability here passes the swap-out test.
• ADR-018 — template hydration. ChartSource::Builder keeps the chart fully typed up to package time.

Open questions remain in the Open questions section below; they are deliberately left open for follow-up clarifications to bolt on the same way this one bolts on to ADR-012.

Context

ADR-012's intent never got a concrete mechanism beyond LAMPScore and the (later-abandoned) modules/application parallel hierarchy. As a result, today every app harmony hosts re-implements its own release pipeline:

• A ~270-line per-component release binary (fleet_release.rs).
• A per-component release.sh wrapper script.
• A per-component .gitea/workflows/<app>-<component>.yaml.
• Hand-written chart hydration (fleet/harmony-fleet-deploy/src/operator/chart.rs).

PR review (#301) flagged this directly: "we should not have to rebuild a new cli for every component of every app using harmony" and "the burden of designing the process and cli should not be on the user. Fleet is the equivalent of a user." The framework owns deploy (ADR-023); release is the missing half of ADR-012.

A naive fix — "add a define_release! macro so each component is one declarative invocation" — solves the duplication but leaves the mechanism wrong. Building, packaging, and pushing are not app concerns. They are capabilities that an environment provides (a docker daemon, an OCI registry, a CI runner), the same way DNS, LoadBalancer, or PostgreSQL are.

This ADR pulls release into the Score-Topology-Capability pattern that already governs deploy.

Two prior attempts and what each got right

Attempt 1 — modules/application (the RustWebapp example, ApplicationScore, ApplicationFeature)

examples/rust/src/main.rs shows what good DX looks like:

let application = Arc::new(RustWebapp {
    name: "harmony-example-rust-webapp".to_string(),
    project_root: PathBuf::from("./webapp"),
    framework: Some(RustWebFramework::Leptos),
    service_port: 3000,
    ..
});

let app = ApplicationScore {
    features: vec![
        Box::new(PackagingDeployment { application: application.clone() }),
        Box::new(Monitoring { application: application.clone(), alert_receiver: vec![...] }),
    ],
    application,
};

What it got right:

Property	Why it's right
Declarative app spec at the call site	One struct, ~15 lines, describes the whole thing
Per-app type implements capability traits (OCICompliant, HelmPackage, Webapp)	Compiler rejects nonsensical combos — features state their bounds (impl<A: OCICompliant + HelmPackage> ApplicationFeature for PackagingDeployment<A>)
Features compose orthogonally	vec![PackagingDeployment, Monitoring, …] extensible without touching the app
One feature does one thing	Small, testable, replaceable

What it got wrong:

Anti-pattern	Symptom
Parallel hierarchy	Application / ApplicationFeature / ApplicationInterpret / ApplicationScore live alongside Score / Interpret — two ways to express the same thing
App-owned capabilities	OCICompliant::build_push_oci_image() is a method on the app — the app does the build. Should be: topology provides a builder, score uses it
MultiTargetTopology + DeploymentTarget::LocalDev / Production	Inline match topology.current_target() { LocalDev => …, _ => Argo… } in PackagingDeployment::ensure_installed. The author's own comment: "It still does not feel right though."
ArgoCD hardcoded as the production mechanism	Should be a ContinuousDelivery capability; OPNsense vs CoreDNS distinction (ADR-003) applies — Argo today, Flux or something else tomorrow

Attempt 2 — harmony-fleet-release binary + release.sh (the current branch)

After abandoning modules/application's parallel hierarchy, fleet went the other direction: hand-rolled per-component binaries, no framework-level abstraction. Solved the parallel-hierarchy problem by removing the hierarchy entirely; created a 270-line per-component duplication problem in its place.

Net diagnosis: attempt 1 had the right shape (declarative app + composable features + capability bounds), wrong mechanism (parallel layer, app-side capabilities). Attempt 2 had the right mechanism (use the existing Score/Topology primitives) but no abstraction. The fix is to keep attempt 1's shape and express it in attempt 2's mechanism.

Mechanism (proposed)

A release is a Score driven by Topology capabilities, exactly like a deploy. No parallel hierarchy. No per-component CLI binaries. No app-side build/push methods. ADR-012's opinionated lifecycle pipeline becomes a composed Vec<Box<dyn Score<T>>> — each pipeline step is one Score, harmony executes them in order, each blocks on the smoke-test contract (ADR-023, step 4).

Capabilities (topology-side)

New capability traits, alongside DnsServer, HelmCommand, etc.:

pub trait ContainerBuilder {
    async fn build(&self, ctx: &BuildContext, image_ref: &str) -> Result<(), Error>;
}

pub trait OciRegistry {
    async fn push_image(&self, image_ref: &str) -> Result<(), Error>;
    fn registry_url(&self) -> &str;
}

pub trait HelmRegistry {
    async fn push_chart(&self, tgz: &Path, project: &str) -> Result<(), Error>;
}

pub trait ContinuousDelivery {
    async fn sync_to(&self, chart_ref: &ChartRef, target: &DeployTarget) -> Result<(), Error>;
}

Each capability is an industry concept (per ADR-003 rule). Adapters:

Capability	LocalDev provider	CI provider	Production provider
ContainerBuilder	local docker daemon	gitea runner docker	remote buildkit
OciRegistry	local registry, or noop (k3d image-import)	hub.nationtech.io	hub.nationtech.io
HelmRegistry	noop (local helm)	hub.nationtech.io	hub.nationtech.io
ContinuousDelivery	direct helm install	direct helm install	ArgoCD ArgoHelmScore

K8sAnywhereTopology composes these capabilities the way it already composes K8sclient + HelmCommand + Ingress + …. A CI-runner topology variant composes the same caps with CI-flavored adapters.

Application spec (data only)

pub struct AppSpec {
    pub name: &'static str,
    pub project_root: PathBuf,
    pub image: ImageSource,    // Dockerfile path + build args, OR a `Buildable` impl that emits one
    pub chart: ChartSource,    // PathBuf to a directory, OR a `ChartBuilder` closure / impl returning HelmChart
}

No methods that do work. The spec is pure data. Methods (build, push, package) live on the topology capabilities. This is the shift that fixes attempt 1: RustWebapp no longer implements OCICompliant::build_push_oci_image(); instead, AppSpec describes what to build, and ContainerBuilder (on the topology) does it.

Scores

pub struct ReleaseScore { app: Arc<AppSpec>, version: String }
pub struct DeployScore  { app: Arc<AppSpec>, version: String, target: DeployTarget }

impl<T: Topology + ContainerBuilder + OciRegistry + HelmRegistry> Score<T> for ReleaseScore { ... }
impl<T: Topology + ContinuousDelivery>                            Score<T> for DeployScore { ... }

The trait bounds are the spec of what a topology must provide. Compile-time guarantee: you cannot construct ReleaseScore against a topology that can't release.

The opinionated pipeline (ADR-012 realized)

ADR-012's pipeline maps directly onto a Score sequence:

ADR-012 step	Mechanism
1. Empty check	Out of scope for harmony (matches ADR-012). Project's own CI step runs ahead of harmony-<app> invocation.
2. Package & publish	ReleaseScore — builds image, hydrates chart, pushes both.
3. Deploy to staging	DeployScore { target: DeployTarget::Staging } — talks to the topology's ContinuousDelivery (Argo by default per ADR-012).
4. Sanity check staging	Smoke-test companion on the DeployScore (ADR-023 principle 4).
5. Deploy to production	Same DeployScore with DeployTarget::Production. CI gates approval; harmony just runs the score.
6. Sanity check production	Same smoke-test companion against production.

Call site (after the migration)

let app = Arc::new(AppSpec {
    name: "harmony-fleet-operator",
    project_root: ".".into(),
    image: ImageSource::Dockerfile("fleet/harmony-fleet-operator/Dockerfile".into()),
    chart: ChartSource::Builder(Box::new(operator::chart::build)),
});

let scores: Vec<Box<dyn Score<_>>> = vec![
    Box::new(ReleaseScore { app: app.clone(), version: from_tag(ref_name)? }),
    Box::new(DeployScore  { app: app.clone(), version, target: DeployTarget::Staging }),
    Box::new(DeployScore  { app: app.clone(), version, target: DeployTarget::Production }),
];

harmony_cli::run(Inventory::autoload(), CITopology::from_env(), scores, None).await?

That's the user-facing surface for any app. It matches ADR-012's "run the same command anywhere" promise — same binary, local laptop or CI runner; topology adapters differ, Scores don't. No per-component release binary. No release.sh per component. Adding a component is one new AppSpec value.

CI integration

Workflow yaml shrinks to:

- name: Release
  run: |
    harmony-fleet --release --from-tag "$GITHUB_REF_NAME"

The --from-tag flag is implemented inside harmony_release (Rust, not bash). It parses <app>-<component>-v<semver> against the registered AppSpecs and constructs ReleaseScore { app, version } for the matching one. Tag malformed → fail at CI start, not three minutes into the docker build.

.gitea/scripts/resolve-release-version.sh (just shipped) is the interim form; the script disappears once --from-tag lands.

What replaces the modules/application layer

Application / ApplicationFeature / ApplicationInterpret / ApplicationScore are deleted. Migration path:

Old	New
RustWebapp (struct implementing capabilities)	AppSpec { name, project_root, image: ImageSource::Buildable(Box::new(RustLeptosBuilder)), chart: … }
OCICompliant::build_push_oci_image	ContainerBuilder::build on the topology (Score calls it)
HelmPackage::build_push_helm_package	HelmRegistry::push_chart on the topology (Score calls it)
PackagingDeployment (ApplicationFeature)	ReleaseScore + DeployScore (regular Scores)
Monitoring (ApplicationFeature)	MonitoringScore (regular Score)
ApplicationScore wrapping features	Vec<Box<dyn Score<T>>> passed to harmony_cli::run

The RustWebapp Leptos Dockerfile generator survives — it becomes a Buildable impl that emits a Dockerfile for ImageSource::Buildable to use. Same code, different home.

Alternatives considered

Macro per component — define_release!(Component::Operator { … })

Solves the duplication. Still leaves build/push as app-side work, still gives every app its own CLI surface. Doesn't address the capability gap. Rejected.

Keep modules/application as-is, add a release feature

Adds a third release variant to the existing parallel hierarchy. Doubles down on the wrong mechanism. Rejected.

Plugin-discovery harmony top-level binary (ADR-023 "tomorrow C")

Orthogonal to this clarification. Plugin discovery is how the binaries are invoked; this is what the binaries do. Compatible: each app ships a harmony-<app> plugin binary that registers its AppSpecs with harmony_release::cli.

Promote to a fresh ADR-025

The shape proposed here doesn't disagree with ADR-012 — it implements it. A new ADR would invite reading them as competing decisions and split the conversation in two. Keeping this as ADR-012's first clarification (012-1) keeps the lineage explicit. Rejected for now; revisit if a future decision genuinely diverges from ADR-012's intent.

Consequences

Positive

• ADR-012 intent finally has a mechanism. The opinionated pipeline becomes a composed Vec<Box<dyn Score<T>>> — concrete, testable, identical local and in CI.
• One framework, one mechanism. Release joins deploy in the Score-Topology-Capability pattern. No second concept to learn.
• Capability-driven swap-out. Move from docker→podman, Harbor→ECR, helm→ko, Argo→Flux without touching any app's AppSpec. ADR-003 rule held.
• ~250 lines deleted from each app (per-component bin + release.sh
  • workflow yaml duplication).
• Adding a component is one AppSpec value, not a new binary.
• LocalDev vs CI vs Production is topology selection at runtime (ADR-023 principle 6), not branching inside a Score.

Negative / costs

• Migration of modules/application and the RustWebapp example to the new shape is a non-trivial PR (preserve the Leptos Dockerfile generation as a Buildable impl).
• New crate harmony_release (or new module inside harmony core). Sizing TBD during implementation; target the same ~ADR-023 scope.
• Adapter coverage: at least three ContainerBuilder adapters (local docker, CI docker, k3d-image-import) and two OciRegistry / HelmRegistry adapters (hub.nationtech.io, noop) need to land for the first migration to be useful.

Risks to watch

• Capability surface creep. Resist adding a fourth or fifth capability for every minor variant (e.g. MultiArchContainerBuilder, SignedOciRegistry). Extend existing capabilities first; create new ones only when the swap-out test (ADR-003) holds.
• Build context size. BuildContext could absorb every flag ever needed (build_args, secrets, platforms, targets, cache_from, …). Start small; grow with documented justification.
• Buildable trait explosion. Per-language Dockerfile generators (RustLeptos, Python, Node, …) belong as Buildable impls, not as variants on AppSpec. Resist adding them to core until at least two callers need each one.

Implementation order (separate PRs, each shippable)

1. harmony_release crate — AppSpec, ImageSource, ChartSource, ReleaseScore, capability trait stubs, --from-tag parsing. No adapters yet.
2. Local + CI capability adapters — LocalDockerBuilder, CiDockerBuilder, HarborOciRegistry, HarborHelmRegistry. Compose into K8sAnywhereTopology and a new CiTopology.
3. Migrate fleet operator release to AppSpec + ReleaseScore. Delete harmony-fleet-release binary, release.sh, .gitea/scripts/resolve-release-version.sh. Workflow yaml goes to the ~15-line form. Validates the framework on a real app.
4. Add fleet agent + callout as AppSpec entries when their pipelines land. One-line additions, no scaffolding.
5. Migrate modules/application RustWebapp example to the new shape. Preserves the Leptos Dockerfile generator. Deletes the parallel Application / ApplicationFeature hierarchy.
6. (Stretch) ContinuousDelivery capability as a real abstraction above the current ArgoHelmScore. Enables Flux / direct-helm as peer providers. Out of v1 scope; ADR remains compatible.

Open questions

• Chart hydration genericism. Should ChartSource::FromScore exist? The deploy Score's resources are a near-superset of what the chart contains. If we get this right, ChartSource::Builder becomes the exception, not the rule. Worth scoping in #5 or later.
• DeployTarget shape. Today fleet uses (namespace, image_tag). A typed enum (InCluster { namespace, … }, MultiCluster { … }, EdgeDevice { device_id, … }) may capture more. Defer to the agent-CD work (manual upgrades per device) where this gets exercised.
• Image identity. The image's full ref (<registry>/<project>/<name>:<version>) is currently built by string concatenation in three places. Should there be an ImageRef type with parse/display impls? Probably; add in #1 if cost is small.
• Buildable for non-Rust languages. Out of scope until a non-Rust app needs it, then add by demand.