harmony/docs/guides/fleet-application-cd.md

Fleet application continuous delivery

An application repository owns a Rust deployment crate whose binary is named harmony. It declares image builds, runtime services, Fleet placement, and compiled deployment contexts. Harmony supplies the release tag at runtime and passes registry-returned digests into one Fleet Deployment.

The reference two-service manifest is examples/harmony_apply_deployment. It builds frontend and backend images and advances them together as one rollout revision.

Production command

After checks pass, the production pipeline runs:

harmony ship \
  --context production \
  --tag "$CI_COMMIT_SHA" \
  --wait

ship performs these steps in order:

1. Build every image declared by the Rust manifest.
2. Push each image to the context's registry repository.
3. Record the digest returned by the registry.
4. Apply one namespaced Fleet Deployment containing those digests.
5. Wait for the operator to report the exact applied rollout revision.
6. Exit successfully when all frozen targets converge, or fail with rollout counts and the latest device error.

The command prints the supplied tag and every deployed digest. The pipeline is roll-forward only. A failed rollout blocks that run. Shipping a repaired image or deployment spec changes the Fleet Deployment and creates the next generation; rerunning an unchanged commit does not. Harmony does not rewrite a Git manifest or roll the application back to an old tag.

One CI secret

The runner receives exactly one secret variable, HARMONY_ZITADEL_KEY_JSON.

This is the tenant-scoped Zitadel machine key. Harmony exchanges it for the short-lived identity used to read the rest of the deployment configuration from the tenant's OpenBao instance.

The compiled context contains non-secret coordinates such as the registry hostname, repository, OpenBao URL, Zitadel URL, and OpenBao role. OpenBao holds:

• RegistryCredentials in the context namespace, using a push-scoped registry robot account;
• Kubernetes cluster access for the tenant namespace;
• application secrets and pull-only registry credentials under <device-prefix>/<deployment>/<key> (same folder; the agent reads image_pull_secret as JSON and secret_env as UTF-8 — it never injects the pull robot into a container).

Each device pull secret is JSON scoped to one registry authority. The authority includes the port when the registry uses one:

{
  "registry": "hub.nationtech.io:5000",
  "username": "tenant-device-pull",
  "password": "..."
}

The credential schema requires all three string fields: registry, username, and password.

The agent normalizes the host name and requires the image registry authority, including its port, to match before forwarding credentials to Podman.

Do not inject registry passwords, kubeconfigs, OpenBao tokens, or device pull credentials into CI variables. publish loads RegistryCredentials through the resolved context and uses a temporary Docker configuration directory.

Separate push and pull identities

CI and devices never share registry credentials.

Identity	Registry access	OpenBao reader
CI publisher	Push to the application's repositories	Tenant CI machine identity
Device puller	Pull only from the application's repositories	Device groups authorized for the Fleet Deployment

The Rust manifest places only an image_pull_secret key name in each private service (no slashes). That key lives in the deployment subtree the operator already grants (<device-prefix>/<deployment>/*). The agent reads the credential only when Podman needs to pull a missing image and sends it through Podman's registry-auth header. It does not run podman login, write an auth file, or place credentials in desired state, container environment, labels, or logs.

The registry should provide separate repository-scoped robot accounts for push and pull. For the first hosted deployments, those repositories can live under hub.nationtech.io/<tenant>.

Rust manifest

The application crate compiles to harmony:

[[bin]]
name = "harmony"
path = "src/main.rs"

Its HarmonyApp implementation declares stable build and deployment policy:

• frontend and backend build contexts and Dockerfiles;
• one production context and tenant namespace;
• one Fleet Deployment name;
• allowed device groups and placement labels;
• ports, environment, application-secret references, and pull-secret references.

Those values change only when application architecture or policy changes. A normal release supplies a new commit tag to the same compiled manifest.

Deploy an existing image

deploy skips build and publication:

harmony deploy \
  --context production \
  --tag "$CI_COMMIT_SHA" \
  --image frontend=hub.nationtech.io/example/frontend@sha256:... \
  --image backend=hub.nationtech.io/example/backend@sha256:... \
  --wait

This application accepts registry digests and explicit dev-* tags. It rejects latest and ordinary mutable tags.

Watching and debugging

While ship --wait is running, the dashboard deployment page polls the current summary every three seconds. It shows succeeded, pending, and failed targets, plus the latest failing device, error, and timestamp. The CLI waits for the status revision matching the object UID and generation returned by its apply, so success from an older rollout cannot complete a newer pipeline run.

The initial dashboard view remains aggregate. Detailed per-device rollout state and explicit canary-stage presentation are later work.