harmony/docs/adr/drafts/012-1-release-architecture.md

# ADR-012 follow-up: Release Architecture Mechanism (Capability-Driven Build, Package, Push)

Initial Author: Jean-Gabriel Gill-Couture (drafted with Claude)

Initial Date: 2026-05-27

Last Updated Date: 2026-05-27

## Status

**Draft. Clarification + mechanism for ADR-012, not a separate
decision.** ADR-012 (Project Delivery Automation, 2025-06-04)
locked the *intent*: application-level Scores (LAMPScore-style)
drive an opinionated pipeline — *empty-check → package&publish →
deploy staging → sanity-check → deploy production → sanity-check* —
with CD tools (Argo / Flux) activated by default and called via
API; the same `harmony` binary runs the project's module locally
and in CI.

This document supplements that intent with the concrete *mechanism*:
how steps 2, 3, 5 (package&publish, deploy staging, deploy
production) actually compile and execute against today's
Score-Topology-Capability primitives, and what to delete from the
two interim attempts (`modules/application` + the per-component
`harmony-fleet-release` binary) when the mechanism lands.

Companion references:
- **ADR-023 (Deploy Architecture, Accepted)** — the `DeployScore`
  proposed here plugs into the `harmony_cli::run` flow and
  `*-deploy` crate pattern ADR-023 mandates; the smoke-test
  contract (ADR-023 principle 4) is how steps 4 and 6 of ADR-012
  block.
- **ADR-024 draft (Fleet Platform Capability Decomposition)** —
  same decomposition shape, applied to release: framework-level
  capability traits, not per-app methods.
- **ADR-003** — capability traits represent industry concepts, not
  tools. Every new capability here passes the swap-out test.
- **ADR-018** — template hydration. `ChartSource::Builder` keeps
  the chart fully typed up to package time.

Open questions remain in the *Open questions* section below; they
are deliberately left open for follow-up clarifications to bolt on
the same way this one bolts on to ADR-012.

## Context

ADR-012's intent never got a concrete mechanism beyond `LAMPScore`
and the (later-abandoned) `modules/application` parallel hierarchy.
As a result, today every app harmony hosts re-implements its own
release pipeline:

- A ~270-line per-component release binary (`fleet_release.rs`).
- A per-component `release.sh` wrapper script.
- A per-component `.gitea/workflows/<app>-<component>.yaml`.
- Hand-written chart hydration (`fleet/harmony-fleet-deploy/src/operator/chart.rs`).

PR review (#301) flagged this directly: *"we should not have to
rebuild a new cli for every component of every app using harmony"*
and *"the burden of designing the process and cli should not be on
the user. Fleet is the equivalent of a user."* The framework owns
deploy (ADR-023); release is the missing half of ADR-012.

A naive fix — "add a `define_release!` macro so each component is
one declarative invocation" — solves the duplication but leaves the
mechanism wrong. Building, packaging, and pushing are not app
concerns. They are *capabilities* that an environment provides (a
docker daemon, an OCI registry, a CI runner), the same way DNS,
LoadBalancer, or PostgreSQL are.

This ADR pulls release into the Score-Topology-Capability pattern
that already governs deploy.

## Two prior attempts and what each got right

### Attempt 1 — `modules/application` (the `RustWebapp` example, `ApplicationScore`, `ApplicationFeature`)

`examples/rust/src/main.rs` shows what good DX looks like:

```rust
let application = Arc::new(RustWebapp {
    name: "harmony-example-rust-webapp".to_string(),
    project_root: PathBuf::from("./webapp"),
    framework: Some(RustWebFramework::Leptos),
    service_port: 3000,
    ..
});

let app = ApplicationScore {
    features: vec![
        Box::new(PackagingDeployment { application: application.clone() }),
        Box::new(Monitoring { application: application.clone(), alert_receiver: vec![...] }),
    ],
    application,
};
```

**What it got right:**

| Property | Why it's right |
|---|---|
| Declarative app spec at the call site | One struct, ~15 lines, describes the whole thing |
| Per-app type implements capability traits (`OCICompliant`, `HelmPackage`, `Webapp`) | Compiler rejects nonsensical combos — features state their bounds (`impl<A: OCICompliant + HelmPackage> ApplicationFeature for PackagingDeployment<A>`) |
| Features compose orthogonally | `vec![PackagingDeployment, Monitoring, …]` extensible without touching the app |
| One feature does one thing | Small, testable, replaceable |

**What it got wrong:**

| Anti-pattern | Symptom |
|---|---|
| Parallel hierarchy | `Application` / `ApplicationFeature` / `ApplicationInterpret` / `ApplicationScore` live alongside `Score` / `Interpret` — two ways to express the same thing |
| App-owned capabilities | `OCICompliant::build_push_oci_image()` is a method on the app — the app *does* the build. Should be: topology *provides* a builder, score *uses* it |
| `MultiTargetTopology` + `DeploymentTarget::LocalDev / Production` | Inline `match topology.current_target() { LocalDev => …, _ => Argo… }` in `PackagingDeployment::ensure_installed`. The author's own comment: *"It still does not feel right though."* |
| ArgoCD hardcoded as the production mechanism | Should be a `ContinuousDelivery` capability; OPNsense vs CoreDNS distinction (ADR-003) applies — Argo today, Flux or something else tomorrow |

### Attempt 2 — `harmony-fleet-release` binary + `release.sh` (the current branch)

After abandoning `modules/application`'s parallel hierarchy, fleet
went the other direction: hand-rolled per-component binaries, no
framework-level abstraction. Solved the parallel-hierarchy problem
by removing the hierarchy entirely; created a 270-line per-component
duplication problem in its place.

**Net diagnosis:** attempt 1 had the right *shape* (declarative
app + composable features + capability bounds), wrong *mechanism*
(parallel layer, app-side capabilities). Attempt 2 had the right
mechanism (use the existing Score/Topology primitives) but no
abstraction. The fix is to keep attempt 1's shape and express it
in attempt 2's mechanism.

## Mechanism (proposed)

**A release is a Score driven by Topology capabilities, exactly like
a deploy.** No parallel hierarchy. No per-component CLI binaries. No
app-side build/push methods. ADR-012's opinionated lifecycle pipeline
becomes a composed `Vec<Box<dyn Score<T>>>` — each pipeline step is
one Score, harmony executes them in order, each blocks on the
smoke-test contract (ADR-023, step 4).

### Capabilities (topology-side)

New capability traits, alongside `DnsServer`, `HelmCommand`, etc.:

```rust
pub trait ContainerBuilder {
    async fn build(&self, ctx: &BuildContext, image_ref: &str) -> Result<(), Error>;
}

pub trait OciRegistry {
    async fn push_image(&self, image_ref: &str) -> Result<(), Error>;
    fn registry_url(&self) -> &str;
}

pub trait HelmRegistry {
    async fn push_chart(&self, tgz: &Path, project: &str) -> Result<(), Error>;
}

pub trait ContinuousDelivery {
    async fn sync_to(&self, chart_ref: &ChartRef, target: &DeployTarget) -> Result<(), Error>;
}
```

Each capability is an industry concept (per ADR-003 rule). Adapters:

| Capability | LocalDev provider | CI provider | Production provider |
|---|---|---|---|
| `ContainerBuilder` | local docker daemon | gitea runner docker | remote buildkit |
| `OciRegistry` | local registry, or noop (k3d image-import) | hub.nationtech.io | hub.nationtech.io |
| `HelmRegistry` | noop (local helm) | hub.nationtech.io | hub.nationtech.io |
| `ContinuousDelivery` | direct helm install | direct helm install | ArgoCD `ArgoHelmScore` |

`K8sAnywhereTopology` composes these capabilities the way it already
composes `K8sclient + HelmCommand + Ingress + …`. A CI-runner
topology variant composes the same caps with CI-flavored adapters.

### Application spec (data only)

```rust
pub struct AppSpec {
    pub name: &'static str,
    pub project_root: PathBuf,
    pub image: ImageSource,    // Dockerfile path + build args, OR a `Buildable` impl that emits one
    pub chart: ChartSource,    // PathBuf to a directory, OR a `ChartBuilder` closure / impl returning HelmChart
}
```

No methods that *do* work. The spec is pure data. Methods (`build`,
`push`, `package`) live on the topology capabilities. This is the
shift that fixes attempt 1: `RustWebapp` no longer implements
`OCICompliant::build_push_oci_image()`; instead, `AppSpec` describes
*what* to build, and `ContainerBuilder` (on the topology) does it.

### Scores

```rust
pub struct ReleaseScore { app: Arc<AppSpec>, version: String }
pub struct DeployScore  { app: Arc<AppSpec>, version: String, target: DeployTarget }

impl<T: Topology + ContainerBuilder + OciRegistry + HelmRegistry> Score<T> for ReleaseScore { ... }
impl<T: Topology + ContinuousDelivery>                            Score<T> for DeployScore { ... }
```

The trait bounds *are* the spec of what a topology must provide.
Compile-time guarantee: you cannot construct `ReleaseScore` against
a topology that can't release.

### The opinionated pipeline (ADR-012 realized)

ADR-012's pipeline maps directly onto a Score sequence:

| ADR-012 step | Mechanism |
|---|---|
| 1. Empty check | Out of scope for harmony (matches ADR-012). Project's own CI step runs ahead of `harmony-<app>` invocation. |
| 2. Package & publish | `ReleaseScore` — builds image, hydrates chart, pushes both. |
| 3. Deploy to staging | `DeployScore { target: DeployTarget::Staging }` — talks to the topology's `ContinuousDelivery` (Argo by default per ADR-012). |
| 4. Sanity check staging | Smoke-test companion on the `DeployScore` (ADR-023 principle 4). |
| 5. Deploy to production | Same `DeployScore` with `DeployTarget::Production`. CI gates approval; harmony just runs the score. |
| 6. Sanity check production | Same smoke-test companion against production. |

### Call site (after the migration)

```rust
let app = Arc::new(AppSpec {
    name: "harmony-fleet-operator",
    project_root: ".".into(),
    image: ImageSource::Dockerfile("fleet/harmony-fleet-operator/Dockerfile".into()),
    chart: ChartSource::Builder(Box::new(operator::chart::build)),
});

let scores: Vec<Box<dyn Score<_>>> = vec![
    Box::new(ReleaseScore { app: app.clone(), version: from_tag(ref_name)? }),
    Box::new(DeployScore  { app: app.clone(), version, target: DeployTarget::Staging }),
    Box::new(DeployScore  { app: app.clone(), version, target: DeployTarget::Production }),
];

harmony_cli::run(Inventory::autoload(), CITopology::from_env(), scores, None).await?
```

**That's the user-facing surface for any app.** It matches ADR-012's
"run the same command anywhere" promise — same binary, local laptop
or CI runner; topology adapters differ, Scores don't. No per-component
release binary. No `release.sh` per component. Adding a component is
one new `AppSpec` value.

### CI integration

Workflow yaml shrinks to:

```yaml
- name: Release
  run: |
    harmony-fleet --release --from-tag "$GITHUB_REF_NAME"
```

The `--from-tag` flag is implemented inside `harmony_release` (Rust,
not bash). It parses `<app>-<component>-v<semver>` against the
registered `AppSpec`s and constructs `ReleaseScore { app, version }`
for the matching one. Tag malformed → fail at CI start, not three
minutes into the docker build.

`.gitea/scripts/resolve-release-version.sh` (just shipped) is the
interim form; the script disappears once `--from-tag` lands.

### What replaces the `modules/application` layer

`Application` / `ApplicationFeature` / `ApplicationInterpret` /
`ApplicationScore` are deleted. Migration path:

| Old | New |
|---|---|
| `RustWebapp` (struct implementing capabilities) | `AppSpec { name, project_root, image: ImageSource::Buildable(Box::new(RustLeptosBuilder)), chart: … }` |
| `OCICompliant::build_push_oci_image` | `ContainerBuilder::build` on the topology (Score calls it) |
| `HelmPackage::build_push_helm_package` | `HelmRegistry::push_chart` on the topology (Score calls it) |
| `PackagingDeployment` (ApplicationFeature) | `ReleaseScore + DeployScore` (regular Scores) |
| `Monitoring` (ApplicationFeature) | `MonitoringScore` (regular Score) |
| `ApplicationScore` wrapping features | `Vec<Box<dyn Score<T>>>` passed to `harmony_cli::run` |

The `RustWebapp` Leptos Dockerfile generator survives — it becomes a
`Buildable` impl that emits a `Dockerfile` for `ImageSource::Buildable`
to use. Same code, different home.

## Alternatives considered

### Macro per component — `define_release!(Component::Operator { … })`

Solves the duplication. Still leaves build/push as app-side work,
still gives every app its own CLI surface. Doesn't address the
capability gap. **Rejected.**

### Keep `modules/application` as-is, add a release feature

Adds a third release variant to the existing parallel hierarchy.
Doubles down on the wrong mechanism. **Rejected.**

### Plugin-discovery `harmony` top-level binary (ADR-023 "tomorrow C")

Orthogonal to this clarification. Plugin discovery is *how the
binaries are invoked*; this is *what the binaries do*. Compatible:
each app ships a `harmony-<app>` plugin binary that registers its
`AppSpec`s with `harmony_release::cli`.

### Promote to a fresh ADR-025

The shape proposed here doesn't disagree with ADR-012 — it
*implements* it. A new ADR would invite reading them as competing
decisions and split the conversation in two. Keeping this as
ADR-012's first clarification (012-1) keeps the lineage explicit.
**Rejected for now;** revisit if a future decision genuinely
diverges from ADR-012's intent.

## Consequences

### Positive

- ADR-012 intent finally has a mechanism. The opinionated pipeline
  becomes a composed `Vec<Box<dyn Score<T>>>` — concrete, testable,
  identical local and in CI.
- One framework, one mechanism. Release joins deploy in the
  Score-Topology-Capability pattern. No second concept to learn.
- Capability-driven swap-out. Move from docker→podman, Harbor→ECR,
  helm→ko, Argo→Flux without touching any app's `AppSpec`. ADR-003
  rule held.
- ~250 lines deleted from each app (per-component bin + release.sh
  + workflow yaml duplication).
- Adding a component is one `AppSpec` value, not a new binary.
- LocalDev vs CI vs Production is *topology selection at runtime*
  (ADR-023 principle 6), not branching inside a Score.

### Negative / costs

- Migration of `modules/application` and the `RustWebapp` example to
  the new shape is a non-trivial PR (preserve the Leptos Dockerfile
  generation as a `Buildable` impl).
- New crate `harmony_release` (or new module inside `harmony` core).
  Sizing TBD during implementation; target the same ~ADR-023 scope.
- Adapter coverage: at least three `ContainerBuilder` adapters
  (local docker, CI docker, k3d-image-import) and two `OciRegistry`
  / `HelmRegistry` adapters (hub.nationtech.io, noop) need to land
  for the first migration to be useful.

### Risks to watch

- **Capability surface creep.** Resist adding a fourth or fifth
  capability for every minor variant (e.g. `MultiArchContainerBuilder`,
  `SignedOciRegistry`). Extend existing capabilities first; create
  new ones only when the swap-out test (ADR-003) holds.
- **Build context size.** `BuildContext` could absorb every flag
  ever needed (`build_args`, `secrets`, `platforms`, `targets`,
  `cache_from`, …). Start small; grow with documented justification.
- **`Buildable` trait explosion.** Per-language Dockerfile generators
  (`RustLeptos`, `Python`, `Node`, …) belong as `Buildable` impls,
  not as variants on `AppSpec`. Resist adding them to core until at
  least two callers need each one.

## Implementation order (separate PRs, each shippable)

1. **`harmony_release` crate** — `AppSpec`, `ImageSource`,
   `ChartSource`, `ReleaseScore`, capability trait stubs, `--from-tag`
   parsing. No adapters yet.
2. **Local + CI capability adapters** — `LocalDockerBuilder`,
   `CiDockerBuilder`, `HarborOciRegistry`, `HarborHelmRegistry`.
   Compose into `K8sAnywhereTopology` and a new `CiTopology`.
3. **Migrate fleet operator release** to `AppSpec` + `ReleaseScore`.
   Delete `harmony-fleet-release` binary, `release.sh`,
   `.gitea/scripts/resolve-release-version.sh`. Workflow yaml goes to
   the ~15-line form. Validates the framework on a real app.
4. **Add fleet agent + callout** as `AppSpec` entries when their
   pipelines land. One-line additions, no scaffolding.
5. **Migrate `modules/application` `RustWebapp` example** to the new
   shape. Preserves the Leptos Dockerfile generator. Deletes the
   parallel `Application` / `ApplicationFeature` hierarchy.
6. **(Stretch) `ContinuousDelivery` capability** as a real abstraction
   above the current `ArgoHelmScore`. Enables Flux / direct-helm as
   peer providers. Out of v1 scope; ADR remains compatible.

## Open questions

- **Chart hydration genericism.** Should `ChartSource::FromScore` exist?
  The deploy Score's resources are a near-superset of what the chart
  contains. If we get this right, `ChartSource::Builder` becomes the
  exception, not the rule. Worth scoping in #5 or later.
- **`DeployTarget` shape.** Today fleet uses `(namespace, image_tag)`.
  A typed enum (`InCluster { namespace, … }`, `MultiCluster { … }`,
  `EdgeDevice { device_id, … }`) may capture more. Defer to the
  agent-CD work (manual upgrades per device) where this gets exercised.
- **Image identity.** The image's full ref (`<registry>/<project>/<name>:<version>`)
  is currently built by string concatenation in three places. Should
  there be an `ImageRef` type with parse/display impls? Probably; add
  in #1 if cost is small.
- **Buildable for non-Rust languages.** Out of scope until a non-Rust
  app needs it, then add by demand.