feat(fleet): harmony apply — deploy the published operator chart (minimal) #306
94
docs/adr/drafts/012-2-mvp-harmony-apply.md
Normal file
94
docs/adr/drafts/012-2-mvp-harmony-apply.md
Normal file
@@ -0,0 +1,94 @@
|
|||||||
|
# ADR: Continuous Delivery via `harmony apply`
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Proposal
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
NationTech needs a continuous-delivery mechanism for tenant applications on multi-tenant Kubernetes clusters, including production OKD and edge Pico deployments. Existing Harmony tooling already builds container images and publishes hydrated Helm charts. The missing step is applying those charts to target clusters.
|
||||||
|
|
||||||
|
Constraints shaping the decision:
|
||||||
|
|
||||||
|
- Tenants are isolated by namespace or namespace-prefix group (e.g., `tenanta-*`), each mapped 1:1 to a Zitadel organization.
|
||||||
|
- Apps may span multiple clusters per tenant.
|
||||||
|
- Tenants deploy via Harmony CLI and CI/CD, not interactively.
|
||||||
|
- Pico deployments run on resource-constrained hardware (three M920q nodes) where per-tenant CD overhead is prohibitive.
|
||||||
|
- Harmony is the platform product; the customer experience should be coherent and Harmony-owned, not bolted together from foreign components.
|
||||||
|
- A tenant-facing UI is desirable but not blocking. The fleet operator dashboard will eventually fill this gap.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Extend Harmony with a `harmony apply` capability that performs `helm install` / `helm upgrade` on already-hydrated Helm charts, in the same spirit as the existing `helm publish` step.
|
||||||
|
|
||||||
|
Reconciliation, credential handling, and tenant scoping live inside Harmony. No external GitOps controller is introduced.
|
||||||
|
|
||||||
|
## Alternatives Considered
|
||||||
|
|
||||||
|
- **Argo CD, instance-per-tenant via argocd-operator.** Strong credential-layer isolation and a usable UI, but per-tenant resource overhead is incompatible with Pico, and it introduces a foreign component inside Harmony's product surface. Rejected.
|
||||||
|
- **Argo CD, single shared instance with AppProjects.** Lower overhead, but isolation is policy-based rather than credential-based, and the operational burden of running Argo across all environments does not pay for itself given tenants don't deploy through the UI. Rejected.
|
||||||
|
- **Flux CD, single instance, SA-per-tenant.** Strong credential-layer isolation, low overhead, good fit for CLI/CI-CD-driven deploys. Rejected because it still introduces a foreign component to operate, monitor, and version-manage across the fleet, and provides no tenant-facing UI either — so the only advantage over `harmony apply` is feature maturity, which is not worth the integration cost given the scope we actually need.
|
||||||
|
- **Argo CD hub + argocd-agent across clusters.** Solves multi-cluster credential sprawl elegantly but does not solve the foreign-component or Pico-overhead problems. Rejected.
|
||||||
|
|
||||||
|
## Rationale
|
||||||
|
|
||||||
|
- Harmony already owns adjacent concerns: chart hydration, image publishing, cluster provisioning, networking, storage. Extending it to apply the charts it produces closes a natural seam.
|
||||||
|
- `helm install/upgrade` against a hydrated chart is a bounded problem with a mature library (Helm SDK). The scope we need is small relative to Argo's or Flux's full feature surface.
|
||||||
|
- Resource cost is effectively zero on top of Harmony's existing footprint. Pico becomes viable without compromise.
|
||||||
|
- Credential isolation is handled by impersonating per-tenant ServiceAccounts when applying, identical in mechanism to Flux's model.
|
||||||
|
- Avoids committing to a third-party CD tool whose lifecycle, support window, and upgrade cadence we would otherwise need to track across every cluster.
|
||||||
|
- Keeps Harmony as the single product surface customers interact with.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
**In scope**
|
||||||
|
|
||||||
|
- `harmony apply` performing `helm install` / `helm upgrade` on hydrated charts produced by the existing pipeline.
|
||||||
|
- Per-tenant ServiceAccount impersonation, with RBAC scoped to the tenant's namespace(s).
|
||||||
|
- Multi-cluster targeting per tenant via Harmony's existing cluster registry.
|
||||||
|
- Status reporting (success, failure, diff summary) surfaced through Harmony's existing output channels.
|
||||||
|
- Idempotent re-apply and rollback via Helm's native release history.
|
||||||
|
|
||||||
|
**Out of scope**
|
||||||
|
|
||||||
|
- A tenant-facing web UI for application status, log viewing, scaling, or restart. Deferred to the fleet operator dashboard.
|
||||||
|
- Continuous reconciliation / drift detection. Apply is invoked explicitly by CLI or CI/CD; we are not running a controller loop.
|
||||||
|
- Sync waves, pre/post hooks beyond what Helm natively provides, and other Argo-specific CD semantics.
|
||||||
|
- Image-based auto-update workflows.
|
||||||
|
- GitOps-style pull from a tenant Git repository. Tenants push through Harmony; Harmony applies.
|
||||||
|
|
||||||
|
## Demo Strategy
|
||||||
|
|
||||||
|
Until the fleet operator dashboard ships, demos use the OKD web console for real-time pod status, logs, scaling, and restart visualization. `harmony apply` output in the terminal demonstrates the deploy itself. No production Argo instance is provisioned for demo aesthetics.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
**Positive**
|
||||||
|
|
||||||
|
- Single coherent product surface; no foreign CD component to operate.
|
||||||
|
- Works identically on production OKD and on Pico without per-environment compromise.
|
||||||
|
- Lowest possible resource overhead.
|
||||||
|
- Credential-layer tenant isolation via ServiceAccount impersonation.
|
||||||
|
- Composes cleanly with existing `helm publish` step — same chart, same hydration, next logical action.
|
||||||
|
- Removes external version-management and lifecycle dependencies (Argo support window, operator upgrades, agent compatibility).
|
||||||
|
|
||||||
|
**Negative / Tradeoffs**
|
||||||
|
|
||||||
|
- No tenant-facing UI in the near term. Mitigated by OKD console for technical users and by the future fleet operator dashboard.
|
||||||
|
- No continuous drift detection. Acceptable because deploys are explicit and tenants do not have direct cluster write access outside Harmony.
|
||||||
|
- We carry the maintenance burden of our own apply logic, including error handling, status surfacing, and edge cases that Argo/Flux have already encountered. Bounded by the small feature surface we need.
|
||||||
|
- If requirements later expand to drift detection, sync waves, or rich UI workflows, we may need to revisit. This is acceptable; the decision is reversible.
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
- Define the precise tenant model in Harmony that generates per-tenant ServiceAccount, RBAC bindings across `tenanta-*` namespaces, and target-cluster credentials.
|
||||||
|
- Decide where Helm release history lives and how rollback is exposed through the CLI.
|
||||||
|
- Determine status-reporting format for CI/CD consumption.
|
||||||
|
|
||||||
|
## Revisit Criteria
|
||||||
|
|
||||||
|
Reconsider this decision if any of the following become true:
|
||||||
|
|
||||||
|
- Tenants demand a self-service UI before the fleet operator dashboard is available.
|
||||||
|
- Drift detection becomes a contractual or compliance requirement.
|
||||||
|
- Harmony's apply logic accumulates enough complexity that adopting Flux or Argo would meaningfully reduce maintenance burden.
|
||||||
@@ -675,10 +675,10 @@ key_json = """
|
|||||||
output_dir: PathBuf::new(), // unused on this code path
|
output_dir: PathBuf::new(), // unused on this code path
|
||||||
image: OPERATOR_IMAGE_TAG.to_string(),
|
image: OPERATOR_IMAGE_TAG.to_string(),
|
||||||
image_pull_policy: "IfNotPresent".to_string(),
|
image_pull_policy: "IfNotPresent".to_string(),
|
||||||
namespace: OPERATOR_NAMESPACE.to_string(),
|
|
||||||
nats_url: format!("nats://{NATS_RELEASE}.{NATS_NAMESPACE}.svc.cluster.local:4222"),
|
nats_url: format!("nats://{NATS_RELEASE}.{NATS_NAMESPACE}.svc.cluster.local:4222"),
|
||||||
log_level: "info,kube_runtime=warn".to_string(),
|
log_level: "info,kube_runtime=warn".to_string(),
|
||||||
credentials: Some(OperatorCredentials { credentials_toml }),
|
credentials: Some(OperatorCredentials { credentials_toml }),
|
||||||
|
chart_version: None,
|
||||||
};
|
};
|
||||||
|
|
||||||
// CRDs first — the operator watches them on startup.
|
// CRDs first — the operator watches them on startup.
|
||||||
@@ -693,7 +693,7 @@ key_json = """
|
|||||||
|
|
||||||
// RBAC.
|
// RBAC.
|
||||||
K8sResourceScore::single(
|
K8sResourceScore::single(
|
||||||
build_service_account(&opts),
|
build_service_account(),
|
||||||
Some(OPERATOR_NAMESPACE.to_string()),
|
Some(OPERATOR_NAMESPACE.to_string()),
|
||||||
)
|
)
|
||||||
.interpret(&Inventory::autoload(), topology)
|
.interpret(&Inventory::autoload(), topology)
|
||||||
@@ -705,7 +705,7 @@ key_json = """
|
|||||||
.await
|
.await
|
||||||
.context("operator ClusterRole apply")?;
|
.context("operator ClusterRole apply")?;
|
||||||
|
|
||||||
K8sResourceScore::single(build_cluster_role_binding(&opts), None)
|
K8sResourceScore::single(build_cluster_role_binding(OPERATOR_NAMESPACE), None)
|
||||||
.interpret(&Inventory::autoload(), topology)
|
.interpret(&Inventory::autoload(), topology)
|
||||||
.await
|
.await
|
||||||
.context("operator ClusterRoleBinding apply")?;
|
.context("operator ClusterRoleBinding apply")?;
|
||||||
|
|||||||
@@ -36,5 +36,5 @@ pub mod server;
|
|||||||
pub use agent::{FleetAgentScore, PodTarget};
|
pub use agent::{FleetAgentScore, PodTarget};
|
||||||
pub use companion::AgentObservation;
|
pub use companion::AgentObservation;
|
||||||
pub use nats::{FleetNatsScore, UserPassCredentials};
|
pub use nats::{FleetNatsScore, UserPassCredentials};
|
||||||
pub use operator::{FleetOperatorScore, OperatorCredentials};
|
pub use operator::{FleetOperatorScore, OperatorCredentials, PublishedChart};
|
||||||
pub use server::FleetServerScore;
|
pub use server::FleetServerScore;
|
||||||
|
|||||||
@@ -22,6 +22,7 @@ use clap::Parser;
|
|||||||
use harmony::inventory::Inventory;
|
use harmony::inventory::Inventory;
|
||||||
use harmony::score::Score;
|
use harmony::score::Score;
|
||||||
use harmony::topology::K8sAnywhereTopology;
|
use harmony::topology::K8sAnywhereTopology;
|
||||||
|
use harmony_cli::Args as HarmonyCliArgs;
|
||||||
use harmony_fleet_deploy::nats::UserPassCredentials;
|
use harmony_fleet_deploy::nats::UserPassCredentials;
|
||||||
use harmony_fleet_deploy::{FleetAgentScore, FleetNatsScore, FleetOperatorScore, agent::PodTarget};
|
use harmony_fleet_deploy::{FleetAgentScore, FleetNatsScore, FleetOperatorScore, agent::PodTarget};
|
||||||
|
|
||||||
@@ -88,6 +89,31 @@ struct CliConfig {
|
|||||||
/// NATS device password. Required.
|
/// NATS device password. Required.
|
||||||
#[arg(long, env = "HARMONY_FLEET_NATS_DEVICE_PASS")]
|
#[arg(long, env = "HARMONY_FLEET_NATS_DEVICE_PASS")]
|
||||||
nats_device_pass: Option<String>,
|
nats_device_pass: Option<String>,
|
||||||
|
|
||||||
|
/// Deploy the published operator chart at this version
|
||||||
|
/// (`oci://<registry>/<project>/harmony-fleet-operator:<version>`)
|
||||||
|
/// instead of rendering one from source — the CD `harmony apply`
|
||||||
|
/// path. Re-run with a newer version to roll forward.
|
||||||
|
#[arg(long, env = "HARMONY_FLEET_OPERATOR_CHART_VERSION")]
|
||||||
|
operator_chart_version: Option<String>,
|
||||||
|
|
||||||
|
#[arg(
|
||||||
|
long,
|
||||||
|
env = "HARMONY_FLEET_OPERATOR_CHART_REGISTRY",
|
||||||
|
default_value = "hub.nationtech.io"
|
||||||
|
)]
|
||||||
|
operator_chart_registry: String,
|
||||||
|
|
||||||
|
#[arg(
|
||||||
|
long,
|
||||||
|
env = "HARMONY_FLEET_OPERATOR_CHART_PROJECT",
|
||||||
|
default_value = "harmony"
|
||||||
|
)]
|
||||||
|
operator_chart_project: String,
|
||||||
|
|
||||||
|
// Flattened so CI can pass `--yes` / `--filter` etc.
|
||||||
|
#[command(flatten)]
|
||||||
|
harmony_cli: HarmonyCliArgs,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl CliConfig {
|
impl CliConfig {
|
||||||
@@ -124,10 +150,17 @@ async fn main() -> Result<()> {
|
|||||||
let device_user = creds.device_user.clone();
|
let device_user = creds.device_user.clone();
|
||||||
let device_pass = creds.device_pass.clone();
|
let device_pass = creds.device_pass.clone();
|
||||||
let nats = FleetNatsScore::user_pass(cli.namespace.clone(), cli.nats_node_port, creds);
|
let nats = FleetNatsScore::user_pass(cli.namespace.clone(), cli.nats_node_port, creds);
|
||||||
let operator = FleetOperatorScore::new()
|
let mut operator = FleetOperatorScore::new()
|
||||||
.namespace(cli.namespace.clone())
|
.namespace(cli.namespace.clone())
|
||||||
.image(cli.operator_image.clone())
|
.image(cli.operator_image.clone())
|
||||||
.nats_url(nats.in_cluster_url());
|
.nats_url(nats.in_cluster_url());
|
||||||
|
if let Some(version) = cli.operator_chart_version.clone() {
|
||||||
|
operator = operator.published_chart(
|
||||||
|
cli.operator_chart_registry.clone(),
|
||||||
|
cli.operator_chart_project.clone(),
|
||||||
|
version,
|
||||||
|
);
|
||||||
|
}
|
||||||
let agent = FleetAgentScore::pod(
|
let agent = FleetAgentScore::pod(
|
||||||
cli.namespace.clone(),
|
cli.namespace.clone(),
|
||||||
PodTarget::user_pass(
|
PodTarget::user_pass(
|
||||||
@@ -146,7 +179,7 @@ async fn main() -> Result<()> {
|
|||||||
Inventory::autoload(),
|
Inventory::autoload(),
|
||||||
K8sAnywhereTopology::from_env(),
|
K8sAnywhereTopology::from_env(),
|
||||||
scores,
|
scores,
|
||||||
None,
|
Some(cli.harmony_cli),
|
||||||
)
|
)
|
||||||
.await
|
.await
|
||||||
.map_err(|e| anyhow::anyhow!("{e}"))
|
.map_err(|e| anyhow::anyhow!("{e}"))
|
||||||
|
|||||||
@@ -51,11 +51,6 @@ pub struct ChartOptions {
|
|||||||
/// sideloaded k3d images, `Never` if the image must already be
|
/// sideloaded k3d images, `Never` if the image must already be
|
||||||
/// present.
|
/// present.
|
||||||
pub image_pull_policy: String,
|
pub image_pull_policy: String,
|
||||||
/// Namespace the operator Deployment runs in. `helm install
|
|
||||||
/// --create-namespace` creates it if absent; the chart itself
|
|
||||||
/// doesn't include a Namespace resource so the chart stays
|
|
||||||
/// reusable across namespaces.
|
|
||||||
pub namespace: String,
|
|
||||||
/// NATS URL the operator connects to. For in-cluster NATS at
|
/// NATS URL the operator connects to. For in-cluster NATS at
|
||||||
/// `fleet-nats.fleet-system` the default `nats://fleet-nats.fleet-system:4222`
|
/// `fleet-nats.fleet-system` the default `nats://fleet-nats.fleet-system:4222`
|
||||||
/// works with no config.
|
/// works with no config.
|
||||||
@@ -67,6 +62,12 @@ pub struct ChartOptions {
|
|||||||
/// Secret entirely and lets the operator connect to NATS without
|
/// Secret entirely and lets the operator connect to NATS without
|
||||||
/// auth — only sensible when there's no callout in front of NATS.
|
/// auth — only sensible when there's no callout in front of NATS.
|
||||||
pub credentials: Option<OperatorCredentials>,
|
pub credentials: Option<OperatorCredentials>,
|
||||||
|
/// Chart-level version written into `Chart.yaml`. `None` falls back
|
||||||
|
/// to the deploy crate's `CARGO_PKG_VERSION` — fine for in-process
|
||||||
|
/// uses (e2e harness, runtime operator Score). The release binary
|
||||||
|
/// sets this to the released tag so the OCI chart artifact lands
|
||||||
|
/// at `…/harmony-fleet-operator:<tag>` matching the image tag.
|
||||||
|
pub chart_version: Option<String>,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// What the operator pod needs to authenticate to NATS via the auth
|
/// What the operator pod needs to authenticate to NATS via the auth
|
||||||
@@ -107,10 +108,14 @@ impl Default for ChartOptions {
|
|||||||
output_dir: PathBuf::from("/tmp/fleet-load-test/chart"),
|
output_dir: PathBuf::from("/tmp/fleet-load-test/chart"),
|
||||||
image: "localhost/harmony-fleet-operator:latest".to_string(),
|
image: "localhost/harmony-fleet-operator:latest".to_string(),
|
||||||
image_pull_policy: "IfNotPresent".to_string(),
|
image_pull_policy: "IfNotPresent".to_string(),
|
||||||
namespace: "fleet-system".to_string(),
|
// Deliberately uses a non-fleet-specific in-cluster DNS
|
||||||
nats_url: "nats://fleet-nats.fleet-system:4222".to_string(),
|
// assuming NATS sits in the same namespace as the operator;
|
||||||
|
// the e2e harness and production overrides set this
|
||||||
|
// explicitly when their NATS lives elsewhere.
|
||||||
|
nats_url: "nats://fleet-nats:4222".to_string(),
|
||||||
log_level: "info,kube_runtime=warn".to_string(),
|
log_level: "info,kube_runtime=warn".to_string(),
|
||||||
credentials: None,
|
credentials: None,
|
||||||
|
chart_version: None,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -131,10 +136,17 @@ pub fn build_chart(opts: &ChartOptions) -> Result<PathBuf> {
|
|||||||
std::fs::create_dir_all(&opts.output_dir)
|
std::fs::create_dir_all(&opts.output_dir)
|
||||||
.with_context(|| format!("creating {:?}", opts.output_dir))?;
|
.with_context(|| format!("creating {:?}", opts.output_dir))?;
|
||||||
|
|
||||||
let mut chart = HelmChart::new(
|
// `HelmChart::new(name, app_version)` only sets appVersion — the
|
||||||
RELEASE_NAME.to_string(),
|
// chart-level `version` field defaults to `"0.1.0"` and has to be
|
||||||
env!("CARGO_PKG_VERSION").to_string(),
|
// assigned directly. For a release artifact we want both to track
|
||||||
);
|
// the released tag (one tag → one image + chart at the same
|
||||||
|
// version), so set both.
|
||||||
|
let chart_version = opts
|
||||||
|
.chart_version
|
||||||
|
.clone()
|
||||||
|
.unwrap_or_else(|| env!("CARGO_PKG_VERSION").to_string());
|
||||||
|
let mut chart = HelmChart::new(RELEASE_NAME.to_string(), chart_version.clone());
|
||||||
|
chart.version = chart_version;
|
||||||
chart.description = "IoT operator — Deployment CRD → NATS KV".to_string();
|
chart.description = "IoT operator — Deployment CRD → NATS KV".to_string();
|
||||||
|
|
||||||
chart.add_resource(HelmResourceKind::Crd(crd_with_keep_annotation(
|
chart.add_resource(HelmResourceKind::Crd(crd_with_keep_annotation(
|
||||||
@@ -144,12 +156,18 @@ pub fn build_chart(opts: &ChartOptions) -> Result<PathBuf> {
|
|||||||
Device::crd(),
|
Device::crd(),
|
||||||
)));
|
)));
|
||||||
|
|
||||||
chart.add_resource(HelmResourceKind::ServiceAccount(service_account(
|
chart.add_resource(HelmResourceKind::ServiceAccount(service_account()));
|
||||||
&opts.namespace,
|
|
||||||
)));
|
|
||||||
chart.add_resource(HelmResourceKind::ClusterRole(cluster_role()));
|
chart.add_resource(HelmResourceKind::ClusterRole(cluster_role()));
|
||||||
|
// The CRB's subject must reference the ServiceAccount's namespace.
|
||||||
|
// Since the chart itself is namespace-neutral (helm assigns the
|
||||||
|
// release namespace to the SA + Deployment at install time), we
|
||||||
|
// emit a literal helm template token so helm substitutes the
|
||||||
|
// release namespace at the same moment. This is the one chart
|
||||||
|
// resource that can't be made namespace-neutral by simply omitting
|
||||||
|
// the field — `subjects[].namespace` is part of the resource
|
||||||
|
// identity and must point somewhere concrete after rendering.
|
||||||
chart.add_resource(HelmResourceKind::ClusterRoleBinding(cluster_role_binding(
|
chart.add_resource(HelmResourceKind::ClusterRoleBinding(cluster_role_binding(
|
||||||
&opts.namespace,
|
"{{ .Release.Namespace }}",
|
||||||
)));
|
)));
|
||||||
// Secret intentionally NOT included in the on-disk helm chart —
|
// Secret intentionally NOT included in the on-disk helm chart —
|
||||||
// credentials are operator-environment-specific and out of scope
|
// credentials are operator-environment-specific and out of scope
|
||||||
@@ -175,10 +193,13 @@ pub fn operator_secret(opts: &ChartOptions) -> Option<Secret> {
|
|||||||
SECRET_KEY_CREDENTIALS_TOML.to_string(),
|
SECRET_KEY_CREDENTIALS_TOML.to_string(),
|
||||||
ByteString(creds.credentials_toml.as_bytes().to_vec()),
|
ByteString(creds.credentials_toml.as_bytes().to_vec()),
|
||||||
);
|
);
|
||||||
|
// Namespace deliberately omitted — the caller passes the target
|
||||||
|
// namespace to `K8sResourceScore::single`, which injects it at
|
||||||
|
// apply time. Keeps the Secret manifest reusable across
|
||||||
|
// environments without baking a namespace into source.
|
||||||
Some(Secret {
|
Some(Secret {
|
||||||
metadata: ObjectMeta {
|
metadata: ObjectMeta {
|
||||||
name: Some(SECRET_NAME.to_string()),
|
name: Some(SECRET_NAME.to_string()),
|
||||||
namespace: Some(opts.namespace.clone()),
|
|
||||||
..Default::default()
|
..Default::default()
|
||||||
},
|
},
|
||||||
data: Some(data),
|
data: Some(data),
|
||||||
@@ -201,11 +222,13 @@ fn crd_with_keep_annotation(mut crd: CustomResourceDefinition) -> CustomResource
|
|||||||
crd
|
crd
|
||||||
}
|
}
|
||||||
|
|
||||||
fn service_account(namespace: &str) -> ServiceAccount {
|
// Namespace-neutral: helm fills in the release namespace at install
|
||||||
|
// time, and the direct-apply path (`K8sResourceScore::single(sa,
|
||||||
|
// Some(ns))`) injects the namespace through its second argument.
|
||||||
|
fn service_account() -> ServiceAccount {
|
||||||
ServiceAccount {
|
ServiceAccount {
|
||||||
metadata: ObjectMeta {
|
metadata: ObjectMeta {
|
||||||
name: Some(SERVICE_ACCOUNT.to_string()),
|
name: Some(SERVICE_ACCOUNT.to_string()),
|
||||||
namespace: Some(namespace.to_string()),
|
|
||||||
..Default::default()
|
..Default::default()
|
||||||
},
|
},
|
||||||
..Default::default()
|
..Default::default()
|
||||||
@@ -325,10 +348,14 @@ fn operator_deployment(opts: &ChartOptions) -> K8sDeployment {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Namespace deliberately omitted — same rationale as the
|
||||||
|
// ServiceAccount: helm fills in the release namespace at install
|
||||||
|
// time, and the direct-apply path injects it via
|
||||||
|
// `K8sResourceScore::single(.., Some(ns))`. Keeps the chart
|
||||||
|
// reusable without baking a namespace into source.
|
||||||
K8sDeployment {
|
K8sDeployment {
|
||||||
metadata: ObjectMeta {
|
metadata: ObjectMeta {
|
||||||
name: Some(RELEASE_NAME.to_string()),
|
name: Some(RELEASE_NAME.to_string()),
|
||||||
namespace: Some(opts.namespace.clone()),
|
|
||||||
labels: Some(match_labels.clone()),
|
labels: Some(match_labels.clone()),
|
||||||
..Default::default()
|
..Default::default()
|
||||||
},
|
},
|
||||||
@@ -364,14 +391,20 @@ fn operator_deployment(opts: &ChartOptions) -> K8sDeployment {
|
|||||||
|
|
||||||
// Re-export the manifest builders so the e2e bring-up can apply the
|
// Re-export the manifest builders so the e2e bring-up can apply the
|
||||||
// operator inline (Score-style) without re-implementing the manifests.
|
// operator inline (Score-style) without re-implementing the manifests.
|
||||||
pub fn build_service_account(opts: &ChartOptions) -> ServiceAccount {
|
//
|
||||||
service_account(&opts.namespace)
|
// The SA + Deployment helpers return namespace-neutral manifests;
|
||||||
|
// callers inject the target namespace through `K8sResourceScore::single`.
|
||||||
|
// The CRB takes the SA's namespace as an explicit argument because the
|
||||||
|
// CRB subject must reference a concrete namespace — there's no
|
||||||
|
// kube-side "current namespace" for cluster-scoped resources.
|
||||||
|
pub fn build_service_account() -> ServiceAccount {
|
||||||
|
service_account()
|
||||||
}
|
}
|
||||||
pub fn build_cluster_role() -> ClusterRole {
|
pub fn build_cluster_role() -> ClusterRole {
|
||||||
cluster_role()
|
cluster_role()
|
||||||
}
|
}
|
||||||
pub fn build_cluster_role_binding(opts: &ChartOptions) -> ClusterRoleBinding {
|
pub fn build_cluster_role_binding(subject_namespace: &str) -> ClusterRoleBinding {
|
||||||
cluster_role_binding(&opts.namespace)
|
cluster_role_binding(subject_namespace)
|
||||||
}
|
}
|
||||||
pub fn build_operator_deployment(opts: &ChartOptions) -> K8sDeployment {
|
pub fn build_operator_deployment(opts: &ChartOptions) -> K8sDeployment {
|
||||||
operator_deployment(opts)
|
operator_deployment(opts)
|
||||||
|
|||||||
@@ -16,4 +16,4 @@ pub use chart::{
|
|||||||
build_chart, operator_secret,
|
build_chart, operator_secret,
|
||||||
};
|
};
|
||||||
pub use install::install_crds;
|
pub use install::install_crds;
|
||||||
pub use score::FleetOperatorScore;
|
pub use score::{FleetOperatorScore, PublishedChart};
|
||||||
|
|||||||
@@ -41,6 +41,17 @@ use harmony::topology::{HelmCommand, K8sclient, Topology};
|
|||||||
use crate::operator::chart;
|
use crate::operator::chart;
|
||||||
use crate::operator::chart::{ChartOptions, OperatorCredentials, build_chart, operator_secret};
|
use crate::operator::chart::{ChartOptions, OperatorCredentials, build_chart, operator_secret};
|
||||||
|
|
||||||
|
/// The already-published OCI chart to install (the CD `harmony apply`
|
||||||
|
/// path). When set, the operator installs
|
||||||
|
/// `oci://{registry}/{project}/harmony-fleet-operator:{version}` and the
|
||||||
|
/// score's `image` is ignored (the image is baked into the chart).
|
||||||
|
#[derive(Debug, Clone, Serialize)]
|
||||||
|
pub struct PublishedChart {
|
||||||
|
pub registry: String,
|
||||||
|
pub project: String,
|
||||||
|
pub version: String,
|
||||||
|
}
|
||||||
|
|
||||||
/// Declarative install of the harmony fleet operator. Construct via
|
/// Declarative install of the harmony fleet operator. Construct via
|
||||||
/// [`new`](Self::new), tune with the builder-style methods, hand to
|
/// [`new`](Self::new), tune with the builder-style methods, hand to
|
||||||
/// a topology that implements [`HelmCommand`].
|
/// a topology that implements [`HelmCommand`].
|
||||||
@@ -57,6 +68,9 @@ pub struct FleetOperatorScore {
|
|||||||
pub nats_url: String,
|
pub nats_url: String,
|
||||||
pub log_level: String,
|
pub log_level: String,
|
||||||
pub credentials: Option<OperatorCredentials>,
|
pub credentials: Option<OperatorCredentials>,
|
||||||
|
/// `None` renders + installs the chart from local source (dev/e2e);
|
||||||
|
/// `Some` installs the published OCI chart at a pinned version (CD).
|
||||||
|
pub published_chart: Option<PublishedChart>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl FleetOperatorScore {
|
impl FleetOperatorScore {
|
||||||
@@ -65,16 +79,35 @@ impl FleetOperatorScore {
|
|||||||
pub fn new() -> Self {
|
pub fn new() -> Self {
|
||||||
let defaults = ChartOptions::default();
|
let defaults = ChartOptions::default();
|
||||||
Self {
|
Self {
|
||||||
namespace: defaults.namespace,
|
// FleetOperatorScore's own default; the chart itself is
|
||||||
|
// namespace-neutral. Callers override via `.namespace(..)`.
|
||||||
|
namespace: "fleet-system".to_string(),
|
||||||
release_name: "harmony-fleet-operator".to_string(),
|
release_name: "harmony-fleet-operator".to_string(),
|
||||||
image: defaults.image,
|
image: defaults.image,
|
||||||
image_pull_policy: defaults.image_pull_policy,
|
image_pull_policy: defaults.image_pull_policy,
|
||||||
nats_url: defaults.nats_url,
|
nats_url: defaults.nats_url,
|
||||||
log_level: defaults.log_level,
|
log_level: defaults.log_level,
|
||||||
credentials: defaults.credentials,
|
credentials: defaults.credentials,
|
||||||
|
published_chart: None,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Install the published OCI chart at `version` instead of rendering
|
||||||
|
/// one from local source (the CD `harmony apply` path).
|
||||||
|
pub fn published_chart(
|
||||||
|
mut self,
|
||||||
|
registry: impl Into<String>,
|
||||||
|
project: impl Into<String>,
|
||||||
|
version: impl Into<String>,
|
||||||
|
) -> Self {
|
||||||
|
self.published_chart = Some(PublishedChart {
|
||||||
|
registry: registry.into(),
|
||||||
|
project: project.into(),
|
||||||
|
version: version.into(),
|
||||||
|
});
|
||||||
|
self
|
||||||
|
}
|
||||||
|
|
||||||
pub fn namespace(mut self, ns: impl Into<String>) -> Self {
|
pub fn namespace(mut self, ns: impl Into<String>) -> Self {
|
||||||
self.namespace = ns.into();
|
self.namespace = ns.into();
|
||||||
self
|
self
|
||||||
@@ -136,30 +169,17 @@ impl<T: Topology + HelmCommand + K8sclient> Interpret<T> for FleetOperatorInterp
|
|||||||
inventory: &Inventory,
|
inventory: &Inventory,
|
||||||
topology: &T,
|
topology: &T,
|
||||||
) -> Result<Outcome, InterpretError> {
|
) -> Result<Outcome, InterpretError> {
|
||||||
// Tempdir lives for the duration of this function; helm reads
|
// Apply the credentials Secret BEFORE the helm install (the
|
||||||
// the chart files synchronously inside HelmChartScore::execute,
|
// chart's Deployment references it via secretKeyRef). Applied
|
||||||
// so dropping after the await is safe.
|
// directly, not via the chart — it's environment-specific. The
|
||||||
let tmp = tempfile::tempdir()
|
// published-chart CD path runs without credentials today, so
|
||||||
.map_err(|e| InterpretError::new(format!("operator chart tempdir: {e}")))?;
|
// this is a no-op there.
|
||||||
|
if let Some(creds) = &self.score.credentials
|
||||||
let chart_options = ChartOptions {
|
&& let Some(secret) = operator_secret(&ChartOptions {
|
||||||
output_dir: tmp.path().to_path_buf(),
|
credentials: Some(creds.clone()),
|
||||||
image: self.score.image.clone(),
|
..ChartOptions::default()
|
||||||
image_pull_policy: self.score.image_pull_policy.clone(),
|
})
|
||||||
namespace: self.score.namespace.clone(),
|
{
|
||||||
nats_url: self.score.nats_url.clone(),
|
|
||||||
log_level: self.score.log_level.clone(),
|
|
||||||
credentials: self.score.credentials.clone(),
|
|
||||||
};
|
|
||||||
|
|
||||||
// Apply the credentials Secret BEFORE the helm install. The
|
|
||||||
// chart's Deployment references it via `envFrom` /
|
|
||||||
// `secretKeyRef`; missing it stalls the pod in
|
|
||||||
// `CreateContainerConfigError` indefinitely. The Secret is
|
|
||||||
// applied directly (not part of the chart) because it's
|
|
||||||
// operator-environment-specific and out of scope for a
|
|
||||||
// redistributable chart — see the comment in `chart::build_chart`.
|
|
||||||
if let Some(secret) = operator_secret(&chart_options) {
|
|
||||||
info!(
|
info!(
|
||||||
"Applying operator credentials Secret '{}' in {}",
|
"Applying operator credentials Secret '{}' in {}",
|
||||||
chart::SECRET_NAME,
|
chart::SECRET_NAME,
|
||||||
@@ -170,55 +190,69 @@ impl<T: Topology + HelmCommand + K8sclient> Interpret<T> for FleetOperatorInterp
|
|||||||
.await?;
|
.await?;
|
||||||
}
|
}
|
||||||
|
|
||||||
info!(
|
// Install through HelmChartScore from either the published OCI
|
||||||
"Rendering operator chart (image={}, namespace={}, nats_url={}) into {}",
|
// chart (CD) or a freshly rendered local chart (dev/e2e). Each
|
||||||
self.score.image,
|
// branch runs its own install so the local tempdir stays alive
|
||||||
self.score.namespace,
|
// across it.
|
||||||
self.score.nats_url,
|
let helm_outcome = if let Some(p) = &self.score.published_chart {
|
||||||
tmp.path().display()
|
let chart_ref = format!("oci://{}/{}/{}", p.registry, p.project, chart::RELEASE_NAME);
|
||||||
);
|
info!(
|
||||||
let chart_path = build_chart(&chart_options)
|
"Installing helm release '{}' from published chart {chart_ref} version {}",
|
||||||
|
self.score.release_name, p.version
|
||||||
|
);
|
||||||
|
HelmChartScore {
|
||||||
|
namespace: Some(non_blank(&self.score.namespace, "namespace")?),
|
||||||
|
release_name: non_blank(&self.score.release_name, "release_name")?,
|
||||||
|
chart_name: non_blank(&chart_ref, "chart_name")?,
|
||||||
|
chart_version: Some(non_blank(&p.version, "chart_version")?),
|
||||||
|
values_overrides: None,
|
||||||
|
values_yaml: None,
|
||||||
|
create_namespace: true,
|
||||||
|
install_only: false,
|
||||||
|
repository: None,
|
||||||
|
}
|
||||||
|
.interpret(inventory, topology)
|
||||||
|
.await?
|
||||||
|
} else {
|
||||||
|
let tmp = tempfile::tempdir()
|
||||||
|
.map_err(|e| InterpretError::new(format!("operator chart tempdir: {e}")))?;
|
||||||
|
let chart_path = build_chart(&ChartOptions {
|
||||||
|
output_dir: tmp.path().to_path_buf(),
|
||||||
|
image: self.score.image.clone(),
|
||||||
|
image_pull_policy: self.score.image_pull_policy.clone(),
|
||||||
|
nats_url: self.score.nats_url.clone(),
|
||||||
|
log_level: self.score.log_level.clone(),
|
||||||
|
credentials: self.score.credentials.clone(),
|
||||||
|
chart_version: None,
|
||||||
|
})
|
||||||
.map_err(|e| InterpretError::new(format!("build operator chart: {e}")))?;
|
.map_err(|e| InterpretError::new(format!("build operator chart: {e}")))?;
|
||||||
|
let chart_path_str = chart_path.to_str().ok_or_else(|| {
|
||||||
let chart_path_str = chart_path
|
InterpretError::new("operator chart path is not utf-8".to_string())
|
||||||
.to_str()
|
})?;
|
||||||
.ok_or_else(|| InterpretError::new("operator chart path is not utf-8".to_string()))?;
|
info!(
|
||||||
|
"Installing helm release '{}' from rendered chart {}",
|
||||||
info!(
|
self.score.release_name, chart_path_str
|
||||||
"Installing helm release '{}' from rendered chart {}",
|
);
|
||||||
self.score.release_name, chart_path_str
|
HelmChartScore {
|
||||||
);
|
namespace: Some(non_blank(&self.score.namespace, "namespace")?),
|
||||||
|
release_name: non_blank(&self.score.release_name, "release_name")?,
|
||||||
// chart_version is what HelmChartScore::find_installed_release
|
chart_name: non_blank(chart_path_str, "chart_name")?,
|
||||||
// matches against on re-runs; pinning it to harmony's package
|
chart_version: Some(non_blank(env!("CARGO_PKG_VERSION"), "chart_version")?),
|
||||||
// version means a second run on an unchanged release short-
|
values_overrides: None,
|
||||||
// circuits cleanly with "already at desired version".
|
values_yaml: None,
|
||||||
let helm = HelmChartScore {
|
create_namespace: true,
|
||||||
namespace: Some(non_blank(&self.score.namespace, "namespace")?),
|
install_only: false,
|
||||||
release_name: non_blank(&self.score.release_name, "release_name")?,
|
repository: None,
|
||||||
chart_name: non_blank(chart_path_str, "chart_name")?,
|
}
|
||||||
chart_version: Some(non_blank(env!("CARGO_PKG_VERSION"), "chart_version")?),
|
.interpret(inventory, topology)
|
||||||
values_overrides: None,
|
.await?
|
||||||
values_yaml: None,
|
|
||||||
create_namespace: true,
|
|
||||||
install_only: false,
|
|
||||||
repository: None,
|
|
||||||
};
|
};
|
||||||
|
|
||||||
// `Score::interpret` (rather than `create_interpret().execute`)
|
|
||||||
// makes the inner helm install fire its own InterpretExecution
|
|
||||||
// events — gives the cli_logger a per-step "Helm Chart …"
|
|
||||||
// progress line with timing.
|
|
||||||
let helm_outcome = helm.interpret(inventory, topology).await?;
|
|
||||||
|
|
||||||
// Wrap the helm message with operator-specific facts the
|
|
||||||
// cli_reporter will surface in the end-of-run summary.
|
|
||||||
Ok(Outcome::success_with_details(
|
Ok(Outcome::success_with_details(
|
||||||
helm_outcome.message,
|
helm_outcome.message,
|
||||||
vec![
|
vec![
|
||||||
format!("operator namespace: {}", self.score.namespace),
|
format!("operator namespace: {}", self.score.namespace),
|
||||||
format!("operator release: {}", self.score.release_name),
|
format!("operator release: {}", self.score.release_name),
|
||||||
format!("operator image: {}", self.score.image),
|
|
||||||
format!("operator NATS URL: {}", self.score.nats_url),
|
format!("operator NATS URL: {}", self.score.nats_url),
|
||||||
],
|
],
|
||||||
))
|
))
|
||||||
|
|||||||
Reference in New Issue
Block a user