harmony/docs/adr/025-application-lifecycle-cli.md

Architecture Decision Record: Application Lifecycle CLI — Dev-Facing Verbs, Contexts, and the Declarative Boundary

Initial Author: Sylvain Tremblay

Initial Date: 2026-06-10

Last Updated Date: 2026-06-10

Status

Proposed (draft)

Extends ADR-023 (deploy architecture) — whose CLI principle (8) only covers how binaries are discovered — with the CLI's experience contract: verb grammar, the context/identity model, the declarative/operational boundary, and where "config" lives. Revises ADR-012 (project delivery automation) on three points: no mandatory staging, GitOps is not the deploy interface, and environment targeting is an explicit context rather than an ambient kubeconfig.

Context

We need one CLI that lets a person or a machine drive the full lifecycle of a client application running on a Harmony-managed tenant — build, deploy, inspect, operate. The first concrete case — a small team shipping a timesheet-style app — sets the shape of the problem: they develop locally and ship straight to production, roll-forward only — no staging, no rollback.

The audience is three-headed and shares a single surface: developers (the priority for this phase), CI/automation, and autonomous agents. The design is informed by a survey of kubectl, argocd, flux, fly.io, heroku, pulumi, dagger, terraform, ansible, and gcloud, plus current practice on CLIs built for AI agents. Two through-lines drove every decision below:

1. Harmony already owns the vocabulary the best tools bolt on. A computed idempotent result (Outcome: NOOP/RUNNING/…), declarative desired state (Scores), a reconciling engine (Maestro), and named targets (Topologies). The CLI's job is to surface these, not invent machinery.
2. The agent/CI contract is a strict-mode projection of good human DX. Non-interactive, JSON, idempotent, predictable exit codes — the same features a careful human wants. Build the human CLI right and the machine CLI falls out.

Constraints that bound the design:

• Runs locally (k3d) and on any K8s (K8sAnywhereTopology) — nothing may be OKD- or vendor-specific.
• Harmony's anti-"YAML mud pit" / compile-time-safety ethos (ADR-005) forbids reintroducing untyped, runtime-validated configuration in any form, including a TOML or YAML file of deployment knobs.

Decision

Eleven principles, grouped.

Targeting & safety

1. No default context, ever. Every command that touches a cluster requires an explicit target: --context <name> flag, else the HARMONY_CONTEXT env var, else a hard error that lists known contexts. There is no "current context" and no implicit fallback — even local k3d must be selected (--context local). This makes "deployed to the wrong cluster" structurally impossible.

2. Deploy is CLI-direct and context-targeted; GitOps is not the interface. A developer runs harmony app deploy --context <env> explicitly. The same verb and the same Scores deploy to local k3d (to test) or to a production tenant — only the Topology selected by the context changes (ADR-023 §2). CI runs the identical command; git-push-triggers-deploy is optional sugar a team may add, never a requirement. A continuous in-cluster reconciler (GitOps-style) is a possible future Topology, not the primary path. This supersedes ADR-012's Argo/Flux-first stance.

Project & identity

3. Implicit app, explicit context. The app identity comes from the project (a tiny, identity-only Harmony.toml at the repo root, or [package.metadata.harmony] in the deploy crate) — the app name, and little else. Everything else derives by convention from name × context: namespace = {tenant-from-context}-{app}, image = {registry}/{project}/{app}, workload selector = by app label. A behavioral knob (replicas, env, resources) in this file is a defect — it belongs in a Score (see §6). Apps that break the naming convention fall back to the deploy crate; that escape hatch is not built until a real non-conventional app exists (Rule of Three).

The declarative boundary

4. Two verb classes. This is the load-bearing distinction. It honors ADR-023's "no handrolled manifests" while still giving developers day-2 ergonomics.

   Class	Verbs	Path	Mutates desired state?	Needs compile?
   Declarative	build · publish · deploy · ship (= build+publish+deploy) · check · run (one-off Job)	through the project's Scores, re-converge	yes	yes (runs deploy crate)
   Operational	logs · status · exec · forward · restart · describe · history	direct kube client, read-only / ephemeral	no	no (metadata only)

   build, publish, and deploy are independently invocable and compose into ship. deploy takes an explicit image (--image <digest>) and only converges — it does not build. build produces a digest-pinned image, publish pushes it, and ship = build + publish + deploy. This is what makes roll-forward recovery — the only recovery a no-rollback team has — a plain harmony app deploy --image <prior-good-digest> with no rebuild, and it lets CI run the stages separately (digest handoff, debuggable failures). publish is Topology-specific: against a remote context it pushes the digest-pinned image to the registry; against a local context it is coded to import the image directly into k3d (k3d image import) instead of pushing. The verb is the same; the Topology supplies the behavior (ADR-023 §2).

   The local inner loop is simply harmony app ship --context local (build + deploy to local k3d) — there is no separate serve verb, because an explicit local context plus the standard verbs already cover it (principle 11; a second verb would be ceremony). check is the only pre-deploy validation we have today: it compiles and type-checks the Scores (there is no computed diff against live state — see Out of scope).

5. No imperative state mutation. There is no scale 3 or set-env. Operational verbs are precisely the ones that don't touch desired state — that is the basis of the split. Changing replicas, env, resources, or routes means editing the Score and redeploying. An imperative shortcut would only create drift that the next deploy clobbers.

6. Three homes for "config" — none of them a config file.

   • Desired state / behavior (replicas, env, resources, routes, dependencies) → typed Scores, git-versioned, compile-checked.
   • Target + credentials (which cluster/tenant, how to auth) → context / Topology, selected explicitly.
   • Secrets → OpenBao (SecretVault), referenced by Scores, fetched at deploy. Never in code, never in a file.

7. Per-environment values are a Score computed as a function of the context. Differences (prod = 3 replicas + managed Postgres; local = 1 replica + sqlite) are expressed in typed code that branches on a profile tag carried by the context — the Pulumi-stack idea in pure Rust, validated by the compiler. The profile is a structured field on the context, not encoded in its name: a Score branches on ctx.profile(), never on the context name (a free human handle — myapp-prod and otherapp-prod may share profile = Prod). (A dedicated typed Profile input is deferred until divergence justifies it — Rule of Three.)

8. Reconciliation is invocation-driven (today). Convergence happens when someone runs harmony app deploy, not continuously. Out-of-band changes survive until the next deploy, then are overwritten. A continuous controller that reverts drift in real time is a deliberate future, not v1.

One surface for humans, CI, and agents

9. Machine contract, always on. Human-readable output by default (TTY-detected); --json opts into a frozen, versioned schema — JSON to stdout, all logs/progress to stderr. No hidden prompts when stdin is not a TTY. Verbs are idempotent. Outcome maps onto exit codes (e.g. success/noop = 0, failure/blocked = non-zero) so CI and agents branch programmatically without scraping text. An agent skill/MCP surface is derived from this CLI later — never hand-built in parallel.

Credentials

10. The credential source is part of the context, and degrades to local. A context is { cluster (endpoint + CA), tenant, credential source / identity, profile } — and carries no role: authorization is a server-side property of the identity (Zitadel token claims → OpenBao policy → RBAC), never a client-trusted field. Locally, the context yields the ambient k3d/kubeconfig — no identity infrastructure required. Remotely, the context carries a brokered source: authenticate a Zitadel service user (machine identity, role-scoped), exchange for an OpenBao-brokered, short-lived, namespace-scoped Kubernetes token, and present that. The CLI holds nothing standing; authorization is enforced server-side (OpenBao policy + cluster RBAC). The broker is K8s- agnostic by construction (standard TokenRequest + RBAC). Detailed broker mechanics and tenant/identity provisioning may warrant a follow-up ADR; provisioning is manual for now.

Grammar

11. Grammar: harmony <scope> <verb>. A small, fixed set of scope nouns organizes the surface by persona and object: app (the developer lifecycle — harmony app deploy, app logs, app ship, …), tenant (tenant-admin), cluster (cluster-admin), and context (everyone). The app instance stays implicit — it is the project's app (principle 3); app is the scope group, not an instance argument, so it is harmony app deploy, never harmony app deploy myapp. This uniform tree is predictable for humans and breadth-first-discoverable for agents (harmony --help → scopes, harmony app --help → verbs). We grow by adding verbs / resource types under an existing scope, never by proliferating top-level commands (argocd's sprawl is the anti-pattern), and add no magic top-level verb aliases — there is no flat harmony deploy shortcut; one way to do it.

Rationale

• No default context trades a keystroke for the elimination of an entire class of catastrophic mistake. For a tool whose first job is shipping to production, that trade is obviously correct.
• Declarative-only is the single highest-leverage choice: every surveyed tool that offers both imperative and declarative mutation (kubectl edit/scale vs apply) suffers drift and clobbering. By making operational verbs structurally incapable of touching desired state, the conflict cannot occur.
• Convention over config keeps the project file at one line, which is the only way to keep it from rotting into the mud pit ADR-005 rejects — and it makes operational verbs instant and toolchain-free, since they need only metadata + a kube client.
• One surface is cheaper and more correct than maintaining separate human and machine tools, and it means an agent and a developer learn the same verbs.

Consequences

• harmony_cli grows from a flag-only score-runner into a verb-noun CLI; deploy crates expose the standard verbs rather than ad-hoc clap.
• Developers never hand-edit live state; every change is code + redeploy → reproducible, reviewable, diffable in git, with no drift surprises (until the next deploy, per §8).
• The same command works local → prod; CI is just another caller, and agents drive the JSON contract.
• A behavioral knob added to Harmony.toml is a review failure, not a feature.
• Cost: no live "what will change" preview until a diff capability is built (Out of scope); the v1 substitute is "deploy to local and observe."

Alternatives considered

• GitOps-first (Argo/Flux), per ADR-012. Rejected as the primary interface: it forces a git round-trip for every change, hides the build behind a controller, and answers poorly to "deploy locally to test." Retained only as a possible future Topology.
• A config manifest with deployment knobs (Harmony.toml holding replicas/env/resources). Rejected: it is the YAML mud pit in TOML clothing — untyped, runtime-validated, the exact anti-pattern of ADR-005.
• Ambient / current context (kubectl/fly style). Rejected: a default target is a standing invitation to deploy to the wrong cluster.
• A standing service-account token as a CI secret. Rejected: a long-lived credential where Zitadel + OpenBao can mint short-lived, identity-bound ones.
• No project file — pure deploy-crate discovery. Rejected: it forces a cargo compile just to learn an app's own namespace for logs.

Out of scope (deferred, not rejected)

• plan / diff / delta. No capability computes desired-vs-live state today; Outcome is a per-Score result, not a preview. This is the eventual differentiator and deserves its own design.
• Continuous in-cluster reconciler (real-time drift correction).
• Step-0 provisioning via CLI (tenant, app, identity triple) — manual for now.
• Tenant dashboard (web UI for the tenant-admin persona).
• MCP server / agent skill manifest (derived from the CLI later).
• CI-platform OIDC federation replacing the handed-off deploy key.
• Typed Profile input for per-environment values (§7).
• Non-convention app fallback (multi-namespace / custom layout, §3).

References

• docs/adr/012-project-delivery-automation.md — predecessor; this ADR revises its staging/GitOps/kubeconfig assumptions.
• docs/adr/023-deploy-architecture.md — deploy crates, Scores, the E2E contract; this ADR fills in its CLI principle (8).
• docs/adr/005-interactive-project.md — Rust DSL over YAML/HCL ("no mud pit").
• docs/adr/020-1-zitadel-openbao-secure-config-store.md — identity + secret backends the credential model (§10) composes.
• docs/adr/016-Harmony-Agent-And-Global-Mesh-…md — the mesh that a future remote-deploy control plane would ride on.
• CLAUDE.md — Score-Topology-Interpret, capability rules.