harmony/docs/adr/025-application-lifecycle-cli.md

# Architecture Decision Record: Application Lifecycle CLI — Dev-Facing Verbs, Contexts, and the Declarative Boundary

Initial Author: Sylvain Tremblay

Initial Date: 2026-06-10

Last Updated Date: 2026-06-10

## Status

Proposed (draft)

Extends ADR-023 (deploy architecture) — whose CLI principle (8) only
covers *how binaries are discovered* — with the CLI's **experience
contract**: verb grammar, the context/identity model, the
declarative/operational boundary, and where "config" lives. Revises
ADR-012 (project delivery automation) on three points: no mandatory
staging, GitOps is not the deploy interface, and environment targeting
is an explicit context rather than an ambient kubeconfig.

## Context

We need one CLI that lets a person *or* a machine drive the full
lifecycle of a client application running on a Harmony-managed tenant —
build, deploy, inspect, operate. The first concrete case — a small team
shipping a timesheet-style app — sets the shape of the problem: they
develop locally and ship **straight to production, roll-forward only** —
no staging, no rollback.

The audience is three-headed and **shares a single surface**:
developers (the priority for this phase), CI/automation, and autonomous
agents. The design is informed by a survey of kubectl, argocd, flux,
fly.io, heroku, pulumi, dagger, terraform, ansible, and gcloud, plus
current practice on CLIs built for AI agents. Two through-lines drove
every decision below:

1. **Harmony already owns the vocabulary the best tools bolt on.** A
   *computed* idempotent result (`Outcome`: NOOP/RUNNING/…), declarative
   desired state (Scores), a reconciling engine (Maestro), and named
   targets (Topologies). The CLI's job is to *surface* these, not invent
   machinery.
2. **The agent/CI contract is a strict-mode projection of good human
   DX.** Non-interactive, JSON, idempotent, predictable exit codes — the
   same features a careful human wants. Build the human CLI right and the
   machine CLI falls out.

Constraints that bound the design:

- **Runs locally (k3d) and on any K8s** (`K8sAnywhereTopology`) — nothing
  may be OKD- or vendor-specific.
- **Harmony's anti-"YAML mud pit" / compile-time-safety ethos** (ADR-005)
  forbids reintroducing untyped, runtime-validated configuration in any
  form, including a TOML or YAML file of deployment knobs.

## Decision

Eleven principles, grouped.

### Targeting & safety

1. **No default context, ever.** Every command that touches a cluster
   requires an explicit target: `--context <name>` flag, else the
   `HARMONY_CONTEXT` env var, else a **hard error** that lists known
   contexts. There is no "current context" and no implicit fallback —
   *even local k3d* must be selected (`--context local`). This makes
   "deployed to the wrong cluster" structurally impossible.

2. **Deploy is CLI-direct and context-targeted; GitOps is not the
   interface.** A developer runs `harmony app deploy --context <env>`
   explicitly. The *same verb and the same Scores* deploy to local k3d
   (to test) or to a production tenant — only the Topology selected by
   the context changes (ADR-023 §2). CI runs the identical command;
   git-push-triggers-deploy is optional sugar a team may add, never a
   requirement. A continuous in-cluster reconciler (GitOps-style) is a
   possible future Topology, not the primary path. This supersedes
   ADR-012's Argo/Flux-first stance.

### Project & identity

3. **Implicit app, explicit context.** The app identity comes from the
   project (a tiny, identity-only `Harmony.toml` at the repo root, or
   `[package.metadata.harmony]` in the deploy crate) — the app *name*,
   and little else. Everything else derives by convention from
   *name × context*: namespace = `{tenant-from-context}-{app}`, image =
   `{registry}/{project}/{app}`, workload selector = by app label. **A
   behavioral knob (replicas, env, resources) in this file is a defect**
   — it belongs in a Score (see §6). Apps that break the naming
   convention fall back to the deploy crate; that escape hatch is not
   built until a real non-conventional app exists (Rule of Three).

### The declarative boundary

4. **Two verb classes.** This is the load-bearing distinction. It
   honors ADR-023's "no handrolled manifests" while still giving
   developers day-2 ergonomics.

   | Class | Verbs | Path | Mutates desired state? | Needs compile? |
   |---|---|---|---|---|
   | **Declarative** | `build` · `publish` · `deploy` · `ship` (= build+publish+deploy) · `check` · `run` (one-off Job) | through the project's **Scores**, re-converge | yes | yes (runs deploy crate) |
   | **Operational** | `logs` · `status` · `exec` · `forward` · `restart` · `describe` · `history` | **direct kube client**, read-only / ephemeral | no | no (metadata only) |

   `build`, `publish`, and `deploy` are independently invocable and
   compose into `ship`. **`deploy` takes an explicit image
   (`--image <digest>`) and only converges — it does not build.** `build`
   produces a digest-pinned image, `publish` pushes it, and `ship` =
   build + publish + deploy. This is what makes roll-forward recovery —
   the only recovery a no-rollback team has — a plain
   `harmony app deploy --image <prior-good-digest>` with no rebuild, and
   it lets CI run the stages separately (digest handoff, debuggable
   failures). **`publish` is Topology-specific:** against a remote context
   it pushes the digest-pinned image to the registry; against a local
   context it is coded to import the image directly into k3d
   (`k3d image import`) instead of pushing. The verb is the same; the
   Topology supplies the behavior (ADR-023 §2).

   The local inner loop is simply `harmony app ship --context local`
   (build + deploy to local k3d) — there is no separate `serve` verb,
   because an explicit local context plus the standard verbs already cover
   it (principle 11; a second verb would be ceremony). `check` is the only pre-deploy validation we have today: it
   compiles and type-checks the Scores (there is **no** computed diff
   against live state — see Out of scope).

5. **No imperative state mutation.** There is no `scale 3` or
   `set-env`. Operational verbs are precisely the ones that *don't* touch
   desired state — that is the basis of the split. Changing replicas,
   env, resources, or routes means **editing the Score and
   redeploying**. An imperative shortcut would only create drift that the
   next deploy clobbers.

6. **Three homes for "config" — none of them a config file.**
   - *Desired state / behavior* (replicas, env, resources, routes,
     dependencies) → **typed Scores**, git-versioned, compile-checked.
   - *Target + credentials* (which cluster/tenant, how to auth) →
     **context / Topology**, selected explicitly.
   - *Secrets* → **OpenBao** (`SecretVault`), referenced by Scores,
     fetched at deploy. Never in code, never in a file.

7. **Per-environment values are a Score computed as a function of the
   context.** Differences (prod = 3 replicas + managed Postgres; local =
   1 replica + sqlite) are expressed in typed code that branches on a
   **profile tag carried by the context** — the Pulumi-stack idea in pure
   Rust, validated by the compiler. The profile is a **structured field on
   the context, not encoded in its name**: a Score branches on
   `ctx.profile()`, never on the context name (a free human handle —
   `myapp-prod` and `otherapp-prod` may share `profile = Prod`). (A dedicated
   typed `Profile` input is deferred until divergence justifies it — Rule
   of Three.)

8. **Reconciliation is invocation-driven (today).** Convergence happens
   when someone runs `harmony app deploy`, not continuously. Out-of-band
   changes survive until the next deploy, then are overwritten. A
   continuous controller that reverts drift in real time is a deliberate
   future, not v1.

### One surface for humans, CI, and agents

9. **Machine contract, always on.** Human-readable output by default
   (TTY-detected); `--json` opts into a **frozen, versioned** schema —
   JSON to stdout, all logs/progress to stderr. No hidden prompts when
   stdin is not a TTY. Verbs are idempotent. `Outcome` maps onto exit
   codes (e.g. success/noop = 0, failure/blocked = non-zero) so CI and
   agents branch programmatically without scraping text. An agent
   skill/MCP surface is *derived from* this CLI later — never hand-built
   in parallel.

### Credentials

10. **The credential source is part of the context, and degrades to
    local.** A context is `{ cluster (endpoint + CA), tenant, credential
    source / identity, profile }` — and carries **no `role`**:
    authorization is a server-side property of the identity (Zitadel token
    claims → OpenBao policy → RBAC), never a client-trusted field.
    Locally, the context yields the ambient k3d/kubeconfig — no
    identity infrastructure required. Remotely, the context carries a
    brokered source: authenticate a Zitadel **service user** (machine
    identity, role-scoped), exchange for an **OpenBao**-brokered,
    short-lived, namespace-scoped Kubernetes token, and present that. The
    CLI holds **nothing standing**; authorization is enforced
    server-side (OpenBao policy + cluster RBAC). The broker is K8s-
    agnostic by construction (standard `TokenRequest` + RBAC). Detailed
    broker mechanics and tenant/identity provisioning may warrant a
    follow-up ADR; provisioning is manual for now.

### Grammar

11. **Grammar: `harmony <scope> <verb>`.** A small, fixed set of scope
    nouns organizes the surface by persona and object: **`app`** (the
    developer lifecycle — `harmony app deploy`, `app logs`, `app ship`,
    …), **`tenant`** (tenant-admin), **`cluster`** (cluster-admin), and
    **`context`** (everyone). The *app instance stays implicit* — it is
    the project's app (principle 3); `app` is the scope group, not an
    instance argument, so it is `harmony app deploy`, never
    `harmony app deploy myapp`. This uniform tree is predictable for humans
    and breadth-first-discoverable for agents (`harmony --help` → scopes,
    `harmony app --help` → verbs). We grow by adding verbs / resource
    types under an existing scope, never by proliferating top-level
    commands (argocd's sprawl is the anti-pattern), and add **no magic
    top-level verb aliases** — there is no flat `harmony deploy` shortcut;
    one way to do it.

## Rationale

- **No default context** trades a keystroke for the elimination of an
  entire class of catastrophic mistake. For a tool whose first job is
  shipping to production, that trade is obviously correct.
- **Declarative-only** is the single highest-leverage choice: every
  surveyed tool that offers both imperative and declarative mutation
  (kubectl `edit`/`scale` vs `apply`) suffers drift and clobbering. By
  making operational verbs structurally incapable of touching desired
  state, the conflict cannot occur.
- **Convention over config** keeps the project file at one line, which is
  the only way to keep it from rotting into the mud pit ADR-005 rejects —
  and it makes operational verbs instant and toolchain-free, since they
  need only metadata + a kube client.
- **One surface** is cheaper and more correct than maintaining separate
  human and machine tools, and it means an agent and a developer learn
  the same verbs.

## Consequences

- `harmony_cli` grows from a flag-only score-runner into a verb-noun CLI;
  deploy crates expose the standard verbs rather than ad-hoc clap.
- Developers never hand-edit live state; every change is code +
  redeploy → reproducible, reviewable, diffable in git, with no drift
  surprises (until the next deploy, per §8).
- The same command works local → prod; CI is just another caller, and
  agents drive the JSON contract.
- A behavioral knob added to `Harmony.toml` is a review failure, not a
  feature.
- Cost: no live "what will change" preview until a diff capability is
  built (Out of scope); the v1 substitute is "deploy to local and
  observe."

## Alternatives considered

- **GitOps-first (Argo/Flux), per ADR-012.** Rejected as the primary
  interface: it forces a git round-trip for every change, hides the build
  behind a controller, and answers poorly to "deploy locally to test."
  Retained only as a possible future Topology.
- **A config manifest with deployment knobs** (`Harmony.toml` holding
  replicas/env/resources). Rejected: it is the YAML mud pit in TOML
  clothing — untyped, runtime-validated, the exact anti-pattern of
  ADR-005.
- **Ambient / current context** (kubectl/fly style). Rejected: a default
  target is a standing invitation to deploy to the wrong cluster.
- **A standing service-account token as a CI secret.** Rejected: a
  long-lived credential where Zitadel + OpenBao can mint short-lived,
  identity-bound ones.
- **No project file — pure deploy-crate discovery.** Rejected: it forces
  a `cargo` compile just to learn an app's own namespace for `logs`.

## Out of scope (deferred, not rejected)

- **`plan` / `diff` / delta.** No capability computes desired-vs-live
  state today; `Outcome` is a per-Score *result*, not a preview. This is
  the eventual differentiator and deserves its own design.
- **Continuous in-cluster reconciler** (real-time drift correction).
- **Step-0 provisioning via CLI** (tenant, app, identity triple) — manual
  for now.
- **Tenant dashboard** (web UI for the tenant-admin persona).
- **MCP server / agent skill manifest** (derived from the CLI later).
- **CI-platform OIDC federation** replacing the handed-off deploy key.
- **Typed `Profile` input** for per-environment values (§7).
- **Non-convention app fallback** (multi-namespace / custom layout, §3).

## References

- `docs/adr/012-project-delivery-automation.md` — predecessor; this ADR
  revises its staging/GitOps/kubeconfig assumptions.
- `docs/adr/023-deploy-architecture.md` — deploy crates, Scores, the E2E
  contract; this ADR fills in its CLI principle (8).
- `docs/adr/005-interactive-project.md` — Rust DSL over YAML/HCL
  ("no mud pit").
- `docs/adr/020-1-zitadel-openbao-secure-config-store.md` — identity +
  secret backends the credential model (§10) composes.
- `docs/adr/016-Harmony-Agent-And-Global-Mesh-…md` — the mesh that a
  future remote-deploy control plane would ride on.
- `CLAUDE.md` — Score-Topology-Interpret, capability rules.