harmony/adr/013-monitoring-notifications.md

Architecture Decision Record: Monitoring Notifications

Initial Author: Taha Hawa

Initial Date: 2025-06-26

Last Updated Date: 2025-06-26

Status

Proposed

Context

We need to send notifications (typically from AlertManager/Prometheus) and we need to receive said notifications on mobile devices for sure in some way, whether it's push messages, SMS, phone call, email, etc or all of the above.

Decision

We should go with https://ntfy.sh except host it ourselves.

ntfy is an open source solution written in Go that has the features we need.

Rationale

ntfy has pretty much everything we need (push notifications, email forwarding, receives via webhook), and nothing/not much we don't. Good fit, lightweight.

Consequences

Pros:

• topics, with ACLs
• lightweight
• reliable
• easy to configure
• mobile app
  • the mobile app can listen via websocket, poll, or receive via Firebase/GCM on Android, or similar on iOS.
• Forward to email
• Text-to-Speech phone call messages using Twilio integration
• Operates based on simple HTTP requests/Webhooks, easily usable via AlertManager

Cons:

• No SMS pushes
• SQLite DB, makes it harder to HA/scale

Alternatives considered

AWS SNS: Pros:

• highly reliable
• no hosting needed

Cons:

• no control, not self hosted
• costs (per usage)

Apprise: Pros:

• Way more ways of sending notifications
• Can use ntfy as one of the backends/ways of sending

Cons:

• Way too overkill for what we need in terms of features

Gotify: Pros:

• simple, lightweight, golang, etc

Cons:

• Pushes topics are per-user

Additional Notes