From Metrics to Phone Alerts: The Full Observability Pipeline

Prometheus, Loki, Alloy, Alertmanager, ntfy, and Uptime Kuma: how I built an end-to-end observability pipeline on a single-node Kubernetes cluster that pushes alerts to my phone for 1.5 GB of RAM.

My homelab runs 60 pods on a single laptop with 15.6 GB of RAM. When something breaks at 2 AM, I want to know about it from my phone, not from my browser the next morning. So I built an observability pipeline that goes from raw metrics and container logs all the way to push notifications: Prometheus for metrics, Loki for logs, Alertmanager for routing, and ntfy for delivery.

Here's the entire pipeline: what collects the data, what evaluates the rules, what routes the alerts, and what wakes me up. Every component is real, running in production, and constrained to work within 16 GB of total node memory.

The Architecture

The observability stack has four layers, and data flows in one direction through them:

Metrics Sources ──→ Prometheus ──→ Alert Rules ──→ Alertmanager ──→ ntfy-alertmanager ──→ ntfy ──→ Phone
                                                       ↑
Pod Logs ──→ Alloy ──→ Loki ──→ LogQL Rules ───────────┘

Uptime Kuma ──→ HTTP probes (27 monitors) ──→ Status dashboard

Prometheus and Loki both evaluate rules and fire alerts to the same Alertmanager instance. Alertmanager deduplicates, groups, and forwards to the ntfy-alertmanager bridge, which formats them and publishes to an ntfy topic. My phone subscribes to that topic. Total latency from event to notification: under 2 minutes.

Metrics: Prometheus + kube-prometheus-stack

The kube-prometheus-stack Helm chart deploys Prometheus, Grafana, Alertmanager, node-exporter, and kube-state-metrics in one shot. It's the standard for Kubernetes monitoring, and for good reason: the default recording rules and dashboards cover 90% of what you need out of the box.

My tuning for a single-node cluster:

prometheus:
  prometheusSpec:
    retention: 30d
    retentionSize: 5GB
    scrapeInterval: 60s
    evaluationInterval: 60s
    resources:
      requests:
        memory: 512Mi
      limits:
        memory: 768Mi
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: longhorn
          resources:
            requests:
              storage: 10Gi

The 60-second scrape interval is a deliberate choice. The default 15 seconds produces 4x the time-series data and 4x the memory pressure. On a single node where Prometheus shares RAM with everything else, 60 seconds gives me enough resolution to debug issues without starving other workloads. For a production cluster I'd want 15 or 30 seconds. For a homelab, 60 is fine.

Retention is 30 days or 5 GB, whichever comes first. After tuning, Prometheus sits at about 560 MB of memory: tight, but stable. The initial deployment with default settings was hitting 1 GB before I right-sized it.

What Gets Scraped

Prometheus discovers targets through ServiceMonitors. The kube-prometheus-stack deploys monitors for all the standard Kubernetes components, and I add explicit ServiceMonitors for anything else that exposes metrics:

Node exporter: CPU, memory, disk, network per node (64 MB, runs as DaemonSet)
kube-state-metrics: Pod status, deployment health, PVC capacity (128 MB)
Longhorn: Volume robustness, replica health, storage usage
Speedtest exporter: ISP bandwidth measured every 60 minutes
cert-manager: Certificate expiry timelines
Alloy: Log collector health and throughput

One thing Talos Linux changes: the standard Prometheus targets for kubeControllerManager, kubeScheduler, kubeProxy, and kubeEtcd don't work because Talos runs these as static pods with non-standard configurations. I disable all four in the Helm values and rely on the API server metrics endpoint instead.

Logs: Loki + Alloy

Metrics tell you that something is wrong. Logs tell you why. I added Loki and Alloy to the stack specifically because I got tired of running kubectl logs -f after every alert.

Alloy runs as a DaemonSet on every node. It discovers all pods via the Kubernetes API, reads their log files from /var/log/pods/ on the host filesystem, and ships them to Loki with labels for namespace, pod, container, and app name.

discovery.kubernetes "pods" {
  role = "pod"
}

discovery.relabel "pod_logs" {
  targets = discovery.kubernetes.pods.targets

  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label  = "pod"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label  = "container"
  }
}

loki.source.kubernetes "pod_logs" {
  targets    = discovery.relabel.pod_logs.output
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki.loki.svc.cluster.local:3100/loki/api/v1/push"
  }
}

Loki runs in single-binary mode: one pod doing ingestion, storage, querying, and rule evaluation. No read/write separation, no microservices deployment. On a single node, the distributed architecture is pure overhead.

The key settings:

7-day retention: Logs older than a week are compacted away. I don't need months of logs. If I haven't investigated an issue within 7 days, the logs won't help.
Filesystem storage: No S3, no MinIO. Logs sit on a 5 GB Longhorn PVC. For my log volume (~60 pods, mostly low-traffic), this is plenty.
Auth disabled: Single-tenant mode with a hardcoded tenant ID of fake. There's one user (me) and one Loki instance. Multi-tenancy adds complexity for zero benefit.
256 MB memory limit: Loki's actual usage runs around 120-180 MB. The initial 192 MB limit was too tight. Alloy can spike Loki's ingestion during bulk pod restarts.

Alloy itself uses 100-170 MB depending on how many pods it's tailing. Both together consume about 350 MB for full cluster log coverage. That's the cost of answering "what happened?" without SSHing into anything.

Alert Rules: Two Engines, One Alertmanager

Prometheus evaluates metric-based rules. Loki's ruler evaluates log-based rules. Both fire to the same Alertmanager instance, which means all alerts, regardless of source, follow the same routing, grouping, and notification path.

Prometheus Alert Rules

I run four rule groups covering the things that actually break:

Flux GitOps health: Catches stuck Kustomizations and HelmReleases. If a Flux resource hasn't reconciled in 30 minutes, something is wrong: maybe a bad manifest, maybe a Helm chart that won't render, maybe a network issue pulling from a registry.

- alert: FluxReconciliationStalled
  expr: max(gotk_reconcile_duration_seconds) by (kind, name, namespace) > 1800
  for: 30m
  labels:
    severity: warning
  annotations:
    summary: "Flux {{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} stalled"

Longhorn storage: Volume degraded (lost a replica), volume faulted (data at risk), storage usage above 85% (warning) or 95% (critical). On a single node with no replica redundancy, a faulted volume is an emergency.

Certificate lifecycle: Fires when a TLS certificate expires in less than 14 days or when a renewal attempt fails. cert-manager handles renewals automatically, but I want to know if it stops working.

ISP speed: The speedtest exporter runs every 60 minutes. If download speed drops below 50 Mbps for 2 hours straight, I get a warning. If the test fails entirely for 2 hours, it's critical: usually means the ISP is down or DNS is broken.

Loki Log-Based Alert

Loki's ruler evaluates LogQL queries on a schedule and fires alerts when they match. I have one rule that catches crash signatures across all pods:

- alert: ContainerLogPanic
  expr: |
    sum by (namespace, pod, container) (
      count_over_time(
        {namespace=~".+"} |~ `(panic:|runtime error:|OOMKilled|Segmentation fault|SIGABRT)` [5m]
      )
    ) > 0
  for: 1m
  labels:
    severity: warning

This catches Go panics, runtime errors, OOM kills, segfaults, and SIGABRT signals. The regex is deliberately specific. An earlier version used (?i)fatal which matched PostgreSQL's routine FATAL: role "root" does not exist log lines every 5 seconds from failed probe connections. Noisy alerts are worse than no alerts.

Alert Routing: Alertmanager to Phone

Alertmanager receives alerts from both Prometheus and Loki. The routing is simple because my needs are simple (one cluster, one user, one notification channel):

route:
  groupBy: ["alertname", "namespace"]
  groupWait: 30s
  groupInterval: 5m
  repeatInterval: 4h
  receiver: ntfy-webhook
  matchers:
    - matchType: "!="
      name: alertname
      value: Watchdog
    - matchType: "!="
      name: alertname
      value: InfoInhibitor

All alerts (except the synthetic Watchdog heartbeat and InfoInhibitor meta-alerts) route to the ntfy webhook receiver. groupWait: 30s gives time for related alerts to fire before sending a batch. repeatInterval: 4h means I won't get the same alert again for 4 hours unless it resolves and re-fires.

The ntfy Bridge

Alertmanager can't talk to ntfy directly: it speaks webhooks, ntfy speaks its own topic-based API. The ntfy-alertmanager bridge sits between them:

Alertmanager → POST http://ntfy-alertmanager:80 → ntfy topic "alerts" → Phone

The bridge maps Prometheus severity labels to ntfy priority levels:

severity: critical → priority 5 (max urgency, overrides Do Not Disturb)
severity: warning → priority 3 (default, shows notification)
Resolved alerts → priority 1 (low, silent confirmation)

The ntfy server itself is self-hosted, accessible at ntfy.naviauxlab.com, and relays through ntfy.sh's Firebase Cloud Messaging integration for reliable mobile delivery. The entire notification path, from Alertmanager webhook to phone buzz, uses 160 MB total (128 MB ntfy + 32 MB bridge).

External Health: Uptime Kuma + AutoKuma

Prometheus and Loki monitor from inside the cluster. Uptime Kuma monitors from a user's perspective: can I actually reach this service over HTTP?

I run 27 monitors covering every service in the cluster:

Infrastructure (11): Grafana, Prometheus, Alertmanager, Traefik, Authentik, and key apps
Deep health (7): Longhorn frontend, Cloudflare Tunnel readiness endpoint, Flux controllers, cert-manager
Applications (7): Every user-facing app
External (1): steven.naviauxlab.com from inside the cluster (validates the Cloudflare Tunnel path)

What makes this maintainable is AutoKuma, a controller that watches custom Kubernetes resources (KumaEntity) and syncs them to Uptime Kuma's API. Monitors are defined in YAML and managed via GitOps, not clicked together in a web UI:

apiVersion: kuma.bigboot.dev/v1
kind: KumaEntity
metadata:
  name: ghost
  namespace: uptime-kuma
spec:
  type: monitor
  data:
    type: http
    name: Ghost
    url: http://ghost.ghost.svc.cluster.local:80
    interval: 60
    maxretries: 3

Add a monitor: commit YAML. Remove a monitor: delete the resource. AutoKuma reconciles every 30 seconds. No manual Uptime Kuma configuration survives outside of Git.

Dashboards

Grafana serves five custom dashboards, all provisioned via ConfigMaps so they survive pod restarts:

Homelab Overview: CPU, memory, storage, and network at a glance. This is my default home dashboard: one tab tells me if the cluster is healthy.
Flux GitOps: Kustomization and HelmRelease reconciliation status. Green means synced, red means investigate.
Longhorn Storage: Volume robustness, replica health, and capacity usage per volume.
cert-manager: Certificate expiry timeline. Shows which certs renew next and whether renewals are succeeding.
Speedtest: ISP bandwidth trends over time. Useful for spotting degradation patterns.

Grafana also has Loki configured as a datasource, so I can run LogQL queries in the Explore view. This is where I go when Prometheus tells me a pod is crash-looping and I need to see the actual error message.

Grafana authenticates through Authentik via OIDC. One SSO login covers Grafana and every other service behind forward-auth.

What It Costs

On a 15.6 GB node, every megabyte counts. Here's the real memory cost of full observability:

Prometheus          560 MB
Grafana             266 MB
Loki                183 MB
Alloy               ~120 MB (varies with pod count)
Uptime Kuma         150 MB
ntfy                ~40 MB
ntfy-alertmanager    ~10 MB
Alertmanager        ~60 MB
node-exporter       ~30 MB
kube-state-metrics  ~60 MB
AutoKuma            ~20 MB
Speedtest exporter  ~30 MB
──────────────────────────
Total              ~1.5 GB

That's about 10% of node memory for complete metrics collection, log aggregation, 27 health monitors, 5 dashboards, and push notifications to my phone. I'd argue it's the most valuable 1.5 GB in the cluster.

Lessons Learned

A few things that bit me during setup:

Loki's first startup takes 10 minutes. The compactor waits for ring stability before becoming ready. Set your readiness probe initialDelaySeconds high enough or you'll watch Kubernetes restart Loki in a loop.
Loki ruler + Helm chart = pain. The Helm chart's ruler.directories value doesn't work in SingleBinary mode. It silently does nothing. You have to create an explicit ConfigMap and mount it with extraVolumes/extraVolumeMounts at /var/loki/rules/fake/ (where fake is the tenant ID for single-tenant mode).
Alloy's memory usage scales with pod count. My initial 192 MB limit worked fine until I restarted 15 pods at once. Alloy opens a file handle per container log file, and bulk restarts spike memory. 256 MB gives enough headroom.
Broad log regex = alert fatigue. (?i)fatal matches PostgreSQL's routine authentication failures. (?i)error matches every HTTP 404. Be specific about what crash patterns you care about.
Cilium egress rules match the pod port, not the Service port. If Grafana's Service is on port 80 but the container listens on 3000, your egress policy needs to allow 3000. This caused silent monitoring failures until I traced the DNAT behavior.

The stack took about a week to get fully integrated: Prometheus first, then Grafana, then the alert pipeline, then Loki and Alloy as a follow-on project. Each piece works independently, but the real value is in the integration: a single alert path from any source to my phone, and a single pane of glass in Grafana for investigation.

For a homelab, this is probably overkill. I built it anyway, because the best way to learn observability tooling is to operate it. And because I like knowing what my cluster is doing at 2 AM without getting out of bed to check.