harmony/ROADMAP/14-tenant-application-monitoring.md

Phase 14: Tenant Application Monitoring & Alerting (deferred — design first)

Status

Deferred (2026-06-22). Captured while baking Harmony deploys into a client app (compose app → tenant + Postgres + live Zitadel on K8sAnywhere). Monitoring/alerting is the next brick for that deploy, but the existing modules don't yet fit; decide the model before implementing. No code deprecated yet — that waits on the decision below.

Goal

One lightweight application monitoring + alerting capability a deploy crate adds with .with(Monitoring::…), that:

• can be installed or not in dev (cheap to omit; no heavy stack for a local k3d inner loop),
• is installed correctly in prod as a function of the cluster's configuration — not a fixed backend. It must answer, per target cluster:
  • is there already a prometheus-operator / cluster monitoring stack to attach to (OKD user-workload-monitoring, kube-prometheus-stack, COO)?
  • are in-cluster Prometheus metrics already scraped at the cluster level (kube-state-metrics, node/cadvisor) so the tenant needs only rules, not a scraper?
  • how are Prometheus permissions scoped for a tenant (namespaced PrometheusRule/ServiceMonitor + RBAC vs. a tenant-owned Prometheus)?

The tension to resolve (the actual reason this is deferred)

• Anti-pattern risk: if every tenant rolls its own Prometheus, we duplicate data already monitored at the cluster level — N Prometheis re-scraping the same kube-state-metrics, N Alertmanagers, N Grafanas. Wasteful and operationally noisy.
• Counter-value: a self-contained tenant is high customer value — a client can look at their namespace and understand their app's health without cluster-admin context. Easy to reason about, easy to hand off.
• The resolution is probably not "one backend" but a capability that picks: attach-to-cluster-stack (rules + a scoped Grafana view) when one exists, vs. provision a minimal tenant-scoped stack when it doesn't — driven by a topology capability/probe, the same way K8sAnywhereTopology already detects distro and TenantManager already owns namespace isolation (ADR-011) and identity (ADR-027). Cluster-level data stays shared; the tenant view + app alert rules are what's per-tenant.

Current state (audited 2026-06-22)

Four overlapping, none-satisfying backends (all AlertSender impls):

Module	Sender	Author / created	Note
monitoring/kube_prometheus/helm_prometheus_alert_score.rs + helm/	KubePrometheus	Willem, 2025-06	Installs full kube-prometheus-stack via Helm. Works on k3d. Used by harmony_app::capabilities::Monitoring. Heaviest; the "tenant rolls its own" path.
monitoring/application_monitoring/application_monitoring_score.rs	CRDPrometheus	Willem, 2025-08	Prometheus-operator CRDs directly.
monitoring/application_monitoring/rhobs_application_monitoring_score.rs	RHOBObservability	Willem, 2025-09	Cluster Observability Operator MonitoringStack. OKD-oriented.
monitoring/prometheus/	Prometheus	Willem, 2025-07	Standalone Prometheus.

Adjacent (keep — different concern, the cluster/ops view, not per-app alerting):

• monitoring/cluster_dashboards/ — Grafana-operator dashboards + datasource + route (Sylvain Tremblay, 2026-03). Cluster-level dashboards on an existing Prometheus; OKD-route oriented.
• monitoring/ceph_alerts.rs, monitoring/okd/cluster_alert_rules.rs — alerts on OKD's native stack (Sylvain Tremblay, 2026-04).

harmony_app::capabilities::Monitoring (the .with(Monitoring::new().alert(r)) DX) currently emits HelmPrometheusAlertingScore (the Helm/KubePrometheus path)

• a namespace-scoped "app has no available replicas for 5m" rule via kube-state-metrics. The DX shape is right; the backend choice underneath it is the open question.

Decision criteria (to settle next)

1. Capability detects cluster monitoring posture (operator present? cluster metrics scraped?) and attaches vs. provisions accordingly — ideally a Topology/TenantManager capability, not per-app branching.
2. Default to rules + tenant-scoped view on shared cluster data; provision a tenant-local stack only when no cluster stack exists (e.g. bare k3d dev).
3. Keep the deploy-crate DX a one-liner (.with(Monitoring::…)); the environment, not the app, decides install vs. attach.
4. Once chosen: collapse the four backends to the canonical one and #[deprecated] (not delete) the overlaps; keep the workspace compiling.

First consumer waiting on this

The first client app deploy crate using this paradigm (frontend + backend + Zitadel; its external SaaS dependency is not deployed). It deliberately ships without a monitoring Score today — add it here once the model lands.