My cluster had metrics. Dashboards. Alerting rules that would fire if a pod started crash-looping or memory spiked past its limit. What it didn't have was logs.

When something went wrong, my debugging workflow was kubectl logs -n whatever pod-name --previous, hoping the pod hadn't been evicted yet. If it had? Gone. If I needed to correlate log events across multiple pods? Terminal tabs and guesswork. If I wanted to know when something first started failing? Scroll and squint.

Prometheus tells you what happened. Logs tell you why. Without centralized logging, I was flying half-blind. And on a single-node cluster with 16GB of RAM, adding a log aggregation stack isn't as simple as helm install loki.

Why Loki

The log aggregation landscape has a clear heavyweight: the ELK stack (Elasticsearch, Logstash, Kibana). It's powerful, battle-tested, and absolutely enormous. Elasticsearch alone wants multiple gigabytes of RAM for a production deployment. On a node where the entire cluster budget is 16GB, that's not an option.

Loki takes a fundamentally different approach. Where Elasticsearch indexes the full content of every log line (enabling fast full-text search), Loki only indexes labels: metadata like namespace, pod name, and container. The actual log content is stored compressed and only parsed at query time. This makes Loki dramatically cheaper to run: less RAM, less CPU, less storage.

The trade-off is query speed on large datasets. But for a homelab with 20-something pods generating modest log volume, that trade-off is invisible. And since Loki is a Grafana Labs project, it plugs directly into the Grafana instance I already have running.

The Architecture

The stack has three components:

• Loki: the log database. Receives log streams via HTTP, stores them on disk, serves queries via LogQL.
• Alloy: the log collector. Runs as a DaemonSet (one pod per node), tails container log files from /var/log/pods/, and ships them to Loki.
• Grafana: the query interface. Already deployed as part of kube-prometheus-stack. Just needed a new datasource pointing at Loki.

On a multi-node cluster, you'd run Loki in microservices mode with separate read, write, and backend components for scalability. On a single node, that's pointless overhead. Loki supports a SingleBinary mode that runs everything in one process: one pod, one PVC, minimal resource footprint.

Deploying Loki

Loki ships as a Helm chart from Grafana's repo. The chart defaults to microservices mode, so the first job is overriding it to SingleBinary and zeroing out the components you don't need:

deploymentMode: SingleBinary

singleBinary:
  replicas: 1
  persistence:
    enabled: true
    size: 5Gi
    storageClass: longhorn
  resources:
    requests:
      cpu: 50m
      memory: 128Mi
    limits:
      cpu: "1"
      memory: 256Mi

# Disable microservices components
read:
  replicas: 0
write:
  replicas: 0
backend:
  replicas: 0
gateway:
  enabled: false

The 5Gi Longhorn PVC stores the TSDB index and compressed log chunks. With 7-day retention and modest log volume, this is more than enough. Resource limits are set at 256Mi. Loki idles around 176Mi in practice.

Retention needs explicit configuration in Loki 6.x. Enable the compactor, set the retention period, and (this is the gotcha that cost me a startup failure) set delete_request_store:

loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  storage:
    type: filesystem
  limits_config:
    retention_period: 168h
  compactor:
    retention_enabled: true
    working_directory: /var/loki/compactor
    compaction_interval: 10m
    delete_request_store: filesystem  # Required in 6.x — omit this and Loki panics

Without delete_request_store: filesystem, Loki fails config validation at startup with a cryptic error. It's not in the getting-started docs. It's a quiet requirement that the chart's default values don't set for you.

The Canary Trap

The Loki Helm chart 6.x has two separate toggles for the canary (a synthetic log generator for self-monitoring). You need to disable both:

lokiCanary:
  enabled: false
monitoring:
  selfMonitoring:
    enabled: false
  lokiCanary:
    enabled: false

Setting only one still deploys a canary pod. On a memory-constrained node, every unnecessary pod matters. Same logic for disabling the gateway, chunks cache, and results cache: all reasonable defaults for a production cluster, all unnecessary overhead on a single node.

Deploying Alloy

Alloy is Grafana's newer log/metrics/traces collector (successor to Promtail and Grafana Agent). It runs as a DaemonSet, which on a single-node cluster means one pod:

controller:
  type: daemonset

alloy:
  mounts:
    varlog: true
  resources:
    requests:
      cpu: 10m
      memory: 64Mi
    limits:
      cpu: 200m
      memory: 256Mi

The mounts.varlog: true setting mounts the host's /var/log directory into the pod, giving Alloy access to /var/log/pods/ where Kubernetes writes container logs. This requires the namespace to have the privileged Pod Security Standard. Without it, the pod fails admission:

apiVersion: v1
kind: Namespace
metadata:
  name: alloy
  labels:
    pod-security.kubernetes.io/enforce: privileged

Alloy's pipeline configuration uses a flow-based syntax. The pipeline discovers pods via the Kubernetes API, extracts labels, tails the logs, and pushes them to Loki:

discovery.kubernetes "pods" {
  role = "pod"
}

discovery.relabel "pod_logs" {
  targets = discovery.kubernetes.pods.targets
  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label  = "pod"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label  = "container"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
    target_label  = "app"
  }
}

loki.source.kubernetes "pod_logs" {
  targets    = discovery.relabel.pod_logs.output
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki.loki.svc.cluster.local:3100/loki/api/v1/push"
  }
}

Four labels per log stream: namespace, pod, container, and app. That's enough for any query I'd realistically run. More labels mean more index entries and more memory. On a constrained node, you want the minimum that's useful.

Network Policies

Both namespaces get the standard default-deny-all treatment, with explicit allow rules for exactly the traffic they need:

Every arrow in the architecture diagram has a corresponding network policy rule. Nothing more, nothing less. The Architect doesn't leave implicit trust between namespaces.

Alerting on Crash Patterns

Metrics-based alerting already catches crash-looping pods via kube_pod_container_status_waiting_reason. But metrics tell you a pod is crashing. They don't tell you why. Log-based alerting fills that gap.

Loki has a built-in ruler that evaluates LogQL expressions on a schedule and pushes alerts to Alertmanager, just like Prometheus rules. The alert I deployed watches for crash signatures in log output:

groups:
  - name: container-log-alerts
    rules:
      - alert: ContainerLogPanic
        expr: |
          sum by (namespace, pod, container) (
            count_over_time(
              {namespace=~".+"} |~ `(panic:|runtime error:|OOMKilled|Segmentation fault|SIGABRT)` [5m]
            )
          ) > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Crash pattern detected in {{ $labels.namespace }}/{{ $labels.pod }}"

The pattern selection was deliberate. My first attempt used (?i)fatal, which seemed reasonable: catch any log line containing "fatal." Except Postgres emits FATAL: role "root" does not exist every five seconds from health probe connections that don't authenticate. That's not a crash; it's expected behavior. The alert would have been permanently firing.

The final patterns (panic:, runtime error:, OOMKilled, Segmentation fault, SIGABRT) are specific enough to mean something is genuinely wrong, not just noisy.

The Ruler Mount Gotcha

Loki's Helm chart has a ruler.directories value that looks like the obvious way to provide alert rules. It even takes a ConfigMap reference. In microservices mode, it works. In SingleBinary mode, it's silently ignored: no ConfigMap is created, no volume is mounted, no error is logged.

The fix is explicit: create the ConfigMap yourself and mount it via singleBinary.extraVolumes and extraVolumeMounts. The mount path matters too: rules must live at <storage_dir>/<tenant_id>/rules.yaml. With auth disabled, the tenant ID defaults to fake, so the path is /var/loki/rules/fake/.

Wiring It Into Grafana

The last piece is telling Grafana where Loki lives. Since Grafana is deployed via kube-prometheus-stack, the datasource is added to the HelmRelease values:

grafana:
  additionalDataSources:
    - name: Loki
      type: loki
      access: proxy
      url: http://loki.loki.svc.cluster.local:3100
      jsonData:
        maxLines: 1000

One commit, one Flux reconciliation, and Loki appears in Grafana's Explore view. LogQL queries work immediately: filter by namespace, pod, container, or app label, then search log content with regex or substring matches.

The Numbers

251Mi of actual memory for centralized logging with 7-day retention and log-based alerting. That's less than what Mealie uses to serve recipes. The entire stack (collection, storage, querying, and alerting) runs in two pods with a combined footprint smaller than Grafana alone.

Alloy's memory does fluctuate with log volume. During heavy output (a deployment rollout, a burst of errors), it can spike to 168Mi. The 256Mi limit gives it headroom without risking OOM kills. I learned this the hard way: the original 192Mi limit was too tight, and Alloy got killed during exactly the kind of event I most wanted logs for.

What It Looks Like

The before/after is stark. Before: kubectl logs in a terminal, hoping the pod still exists. After: Grafana Explore with a query like:

{namespace="ghost"} |= "error" | logfmt

Filter by namespace, search for patterns, parse structured fields, see timestamps, correlate across pods, all in the same Grafana instance where the metrics dashboards live. One tool, one interface, metrics and logs side by side.

And when something crashes at 3 AM, the Loki ruler catches the panic pattern in the logs, pushes an alert to Alertmanager, which routes it to ntfy, which sends a push notification to my phone. By the time I look at it, the logs are already preserved, even if the pod has been evicted and restarted.

Lessons From the Gotcha List

This deployment landed in a single commit, but that commit was the result of iterating through a surprising number of silent failures. Loki's documentation assumes microservices mode. The Helm chart's values have subtle differences between modes that aren't always documented. Features that "should work" in SingleBinary mode sometimes don't.

The gotchas worth remembering:

• delete_request_store: filesystem is mandatory when retention is enabled. Without it, Loki panics at startup.
• Two canary toggles: lokiCanary.enabled and monitoring.lokiCanary.enabled are independent. Disable both.
• Ruler directories don't work in SingleBinary mode: use explicit ConfigMap + extraVolumes instead.
• Tenant ID is fake when auth is disabled. Rules must be mounted at that path.
• First boot takes 10 minutes: the compactor waits for ring stability. Readiness probe returns 503 the entire time. Don't kill it.
• Alloy needs privileged PSS: it mounts hostPath for /var/log. No way around it.
• Regex breadth matters: (?i)fatal catches health probe noise. Be specific about what constitutes a crash.

Every one of these cost me a restart cycle and a round of debugging. None of them are difficult once you know them. That's the nature of infrastructure work: the hard part isn't the architecture, it's the papercuts.

What's Next

The logging stack is running. The next step is building useful dashboards: a log volume panel showing which namespaces generate the most output, an error rate timeline, maybe a "last N errors" table for each namespace. The data is there; it just needs visualization.

There's also a case for structured logging. Right now, most of my apps output plain text logs. Parsing them with LogQL's pattern matching works but is brittle. Apps that emit JSON logs would let me extract fields natively (response times, error codes, user actions) without regex gymnastics. That's a longer-term investment, but the infrastructure is ready for it.

Prometheus tells you the house is on fire. Loki tells you which room, and what sparked it. On a cluster this small, every debugging minute matters. And the 251Mi cost of having real answers is one of the best trades I've made.

This post builds on Observability on 16GB: Monitoring a Single-Node Kubernetes Cluster, which covers the Prometheus + Grafana + Alertmanager stack that Loki now complements.