Files
harmony/adr/020-monitoring-alerting-architecture/monitoring_v2
Jean-Gabriel Gill-Couture af6145afe3
All checks were successful
Run Check Script / check (pull_request) Successful in 1m23s
doc: monitoring module documentation
2026-03-09 18:33:35 -04:00
..
2026-03-09 18:33:35 -04:00
2026-03-09 18:33:35 -04:00

Monitoring v2 - Improved Architecture

This example demonstrates the improved monitoring architecture that addresses the "WTF/minute" issues in the original design.

Key Improvements

1. Single AlertChannel Trait with Generic Sender

The original design required 9-12 implementations for each alert channel (Discord, Webhook, etc.) - one for each sender type. The new design uses a single trait with generic sender parameterization:

pub trait AlertChannel<Sender: AlertSender> { async fn install_config(&self, sender: &Sender) -> Result<Outcome, InterpretError>; fn name(&self) -> String; fn as_any(&self) -> &dyn std::any::Any; }

Benefits:

  • One Discord implementation works with all sender types
  • Type safety at compile time
  • No runtime dispatch overhead

2. MonitoringStack Abstraction

Instead of manually selecting CRDPrometheus vs KubePrometheus vs RHOBObservability, you now have a unified MonitoringStack that handles versioning:

let monitoring_stack = MonitoringStack::new(MonitoringApiVersion::V2CRD) .set_namespace("monitoring") .add_alert_channel(discord_receiver) .set_scrape_targets(vec![...]);

Benefits:

  • Single source of truth for monitoring configuration
  • Easy to switch between monitoring versions
  • Automatic version-specific configuration

3. TenantMonitoringScore - True Composition

The original monitoring_with_tenant example just put tenant and monitoring as separate items in a vec. The new design truly composes them:

let tenant_score = TenantMonitoringScore::new("test-tenant", monitoring_stack);

This creates a single score that:

  • Has tenant context
  • Has monitoring configuration
  • Automatically installs monitoring scoped to tenant namespace

Benefits:

  • No more "two separate things" confusion
  • Automatic tenant namespace scoping
  • Clear ownership: tenant owns its monitoring

4. Versioned Monitoring APIs

Clear versioning makes it obvious which monitoring stack you're using:

pub enum MonitoringApiVersion { V1Helm, // Old Helm charts V2CRD, // Current CRDs V3RHOB, // RHOB (future) }

Benefits:

  • No guessing which API version you're using
  • Easy to migrate between versions
  • Backward compatibility path

Comparison

Original Design (monitoring_with_tenant)

  • Manual selection of each component
  • Manual installation of both components
  • Need to remember to pass both to harmony_cli::run
  • Monitoring not scoped to tenant automatically

New Design (monitoring_v2)

  • Single composed score
  • One score does it all

Usage

cd examples/monitoring_v2 cargo run

Migration Path

To migrate from the old design to the new:

  1. Replace individual alert channel implementations with AlertChannel
  2. Use MonitoringStack instead of manual *Prometheus selection
  3. Use TenantMonitoringScore instead of separate TenantScore + monitoring scores
  4. Select monitoring version via MonitoringApiVersion