Subscribe

Zero-Trust Networking in a Homelab: NetworkPolicies with Cilium

Zero-Trust Networking in a Homelab: NetworkPolicies with Cilium

By default, every pod in a Kubernetes cluster can talk to every other pod. There are no firewalls between namespaces, no access controls between services, no segmentation whatsoever. Your password manager can reach your RSS reader. Your recipe app can query your monitoring stack. Everything trusts everything.

This is fine for a lab you're experimenting with. It's not fine for a cluster you want to operate like production infrastructure.

This post covers how I implemented zero-trust networking across my entire homelab cluster using Kubernetes NetworkPolicies and Cilium-specific CiliumNetworkPolicies. It includes the 2-hour incident where I broke the entire monitoring stack, the root cause (Cilium's eBPF DNAT behavior), and the patterns that survived.

The Model: Default-Deny + Explicit Allow

The approach is simple in concept: deny all traffic by default, then explicitly allow only what's needed. Every namespace in my cluster starts with this policy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: miniflux
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

An empty podSelector matches all pods in the namespace. An empty Ingress and Egress list means no traffic is allowed: not in, not out, not even DNS. From the moment this policy is applied, every pod in the namespace is completely isolated.

Then you add allow rules, one at a time, for each legitimate traffic flow.

A Real Example: Miniflux RSS Reader

Miniflux is my RSS feed reader. It has a web UI, a PostgreSQL database, a backup CronJob, and it needs to reach the internet to fetch feeds. Here's the complete set of policies, 12 resources in one file:

Ingress from Traefik (web traffic):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-miniflux-ingress-from-traefik
  namespace: miniflux
spec:
  podSelector:
    matchLabels:
      app: miniflux
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: traefik
      ports:
        - protocol: TCP
          port: 8080

Egress from Miniflux to PostgreSQL:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-miniflux-to-postgres
  namespace: miniflux
spec:
  podSelector:
    matchLabels:
      app: miniflux
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: miniflux-postgres
      ports:
        - protocol: TCP
          port: 5432

Ingress to PostgreSQL (only from Miniflux and the backup job):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-postgres-ingress
  namespace: miniflux
spec:
  podSelector:
    matchLabels:
      app: miniflux-postgres
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: miniflux
        - podSelector:
            matchLabels:
              app: miniflux-postgres-backup
      ports:
        - protocol: TCP
          port: 5432

Then there are additional policies for Prometheus scraping, Uptime Kuma health checks, Homepage dashboard queries, and the feed-seed job. Each traffic flow gets its own policy with explicit source, destination, and port.

The total for Miniflux: 12 policy resources. For a single app with a database. That's the cost of zero-trust: you pay for it in YAML.

The Two-Layer Strategy

Early on, I discovered that standard Kubernetes NetworkPolicies don't cover every use case. Cilium, the CNI I use, has its own CRD, CiliumNetworkPolicy, that adds capabilities standard policies lack. I ended up using both:

  • Standard K8s NetworkPolicy for all ingress rules. Pod-to-pod ingress works perfectly with namespace selectors and pod selectors.
  • CiliumNetworkPolicy for egress rules. Cilium's toEntities selector is the only reliable way to allow traffic to the Kubernetes API, external internet, and host-network pods.

Here's why the distinction matters. To allow a pod to resolve DNS, you might try this with a standard policy:

# This works but is fragile
egress:
  - to:
      - namespaceSelector: {}
    ports:
      - protocol: UDP
        port: 53
      - protocol: TCP
        port: 53

The Cilium equivalent is more precise:

# CiliumNetworkPolicy — targets kube-dns specifically
egress:
  - toEndpoints:
      - matchLabels:
          "k8s:io.kubernetes.pod.namespace": kube-system
          k8s-app: kube-dns
    toPorts:
      - ports:
          - port: "53"
            protocol: UDP
          - port: "53"
            protocol: TCP

The Cilium version only allows DNS traffic to the actual kube-dns pods in kube-system, not to any pod on port 53 in any namespace.

The 2-Hour Incident: How I Broke Monitoring

The monitoring namespace was the most complex to lock down. Prometheus scrapes metrics from every namespace. Grafana queries Prometheus and authenticates via OIDC (Authentik, external URL). Alertmanager sends webhooks to ntfy in a different namespace. kube-state-metrics and the Prometheus Operator both need Kubernetes API access.

I wrote all the allow rules, tested them individually, and then applied the default-deny policy. Everything looked fine for about 10 minutes.

Then Grafana dashboards went blank. Prometheus targets showed as down. Alertmanager stopped sending notifications. The entire monitoring stack was deaf and blind.

The Root Cause: Cilium's eBPF DNAT Behavior

The problem was subtle. I had written egress rules using ipBlock to allow Prometheus to reach the Kubernetes API server:

# This does NOT work with Cilium
egress:
  - to:
      - ipBlock:
          cidr: 10.96.0.1/32  # kubernetes.default Service ClusterIP
    ports:
      - protocol: TCP
        port: 443

In a traditional CNI, this would work. The Service ClusterIP is a stable virtual IP, and traffic to it gets DNAT'd to the actual API server pod.

But Cilium's eBPF datapath evaluates network policies after DNAT. By the time the policy is checked, the destination IP is no longer 10.96.0.1. It's the node IP where kube-apiserver is running. The ipBlock rule never matches because the packet's destination has already been rewritten.

The fix is to use Cilium's toEntities selector, which understands Kubernetes-native concepts:

# This works — Cilium resolves the entity correctly
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-prometheus-egress
  namespace: monitoring
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: prometheus
  egress:
    - toEntities:
        - kube-apiserver

toEntities: kube-apiserver matches traffic to the Kubernetes API server regardless of DNAT. Cilium knows which IPs belong to the API server and evaluates the policy correctly.

More Entities I Had to Learn

The Prometheus egress policy ended up using four different entity types:

egress:
  # DNS resolution
  - toEndpoints:
      - matchLabels:
          "k8s:io.kubernetes.pod.namespace": kube-system
          k8s-app: kube-dns
    toPorts:
      - ports:
          - port: "53"
            protocol: UDP
  # Kubernetes API (service discovery)
  - toEntities:
      - kube-apiserver
  # Scrape targets in any namespace
  - toEntities:
      - cluster
    toPorts:
      - ports:
          - port: "9090"
            protocol: TCP
          - port: "8080"
            protocol: TCP
          # ... all scrape target ports
  # Node-exporter runs on the host network
  - toEntities:
      - host
      - remote-node
    toPorts:
      - ports:
          - port: "9100"
            protocol: TCP

Each entity serves a specific purpose:

  • kube-apiserver: The Kubernetes API server (required for service discovery, CRD access)
  • cluster: Any pod endpoint in the cluster (required for cross-namespace scraping)
  • host and remote-node: Pods using hostNetwork: true like node-exporter (the pod shares the node's IP, so normal pod selectors don't match)
  • world: Any IP outside the cluster (used for apps that need internet access)

The Monitoring Namespace: 14 Policies

After resolving the DNAT issue, the monitoring namespace ended up with the most complex policy set in the cluster. Here's the traffic map:

  • Grafana: ingress from Traefik (web UI) + Prometheus (metrics scrape). Egress to Prometheus (data queries), kube-apiserver, DNS, and the external internet (Authentik OIDC)
  • Prometheus: ingress from Grafana + Operator. Egress to DNS, kube-apiserver, cluster (all scrape targets), host/remote-node (node-exporter), Alertmanager
  • Alertmanager: ingress from Prometheus. Egress to DNS, kube-apiserver, ntfy-alertmanager bridge (cross-namespace)
  • kube-state-metrics: ingress from Prometheus. Egress to DNS, kube-apiserver
  • Prometheus Operator: ingress from Prometheus. Egress to DNS, kube-apiserver, cluster (manages Prometheus/Alertmanager pods)
  • Speedtest exporter: ingress from Prometheus. Egress to DNS, internet (Ookla servers)
  • Helm hooks + CRD upgrade jobs: Egress to DNS, kube-apiserver

Plus cross-namespace ingress for Homepage widgets and Uptime Kuma health checks. Every single traffic flow is explicitly declared.

A particularly tricky case was the CRD upgrade job. When kube-prometheus-stack upgrades, it runs a Kubernetes Job to update CRDs. This Job gets a job-name label, not the app.kubernetes.io/instance label that other Helm-managed resources get. So it doesn't match the general Helm hooks policy and needs its own:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-crds-upgrade-egress
  namespace: monitoring
spec:
  endpointSelector:
    matchLabels:
      job-name: kube-prometheus-stack-crds-upgrade
  egress:
    - toEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
    - toEntities:
        - kube-apiserver

You only discover this when the CRD upgrade job hangs during a Helm upgrade and you spend 30 minutes figuring out why.

What We Deliberately Left Open

Not every namespace gets a default-deny policy. Some system namespaces are intentionally left open:

  • kube-system: CoreDNS, kube-proxy, Cilium agent. Locking this down risks breaking cluster-level DNS and networking
  • flux-system: Flux controllers need to reach the Kubernetes API and Git repositories. Restricting this would break GitOps reconciliation
  • longhorn-system: Longhorn's storage controller and instance managers communicate extensively. The traffic patterns are complex and change dynamically

These are conscious decisions, not oversights. Each one is documented with the reasoning. The principle is: if you can't fully map the traffic patterns and the consequences of blocking them, don't apply default-deny until you can.

The Deployment Strategy That Saved Me

After the monitoring incident, I adopted a deployment strategy for network policies:

  1. Write all allow rules first: deploy them without default-deny
  2. Verify everything works: check that all traffic flows still function
  3. Apply default-deny last: the deny policy activates enforcement
  4. If anything breaks: delete the default-deny policy (instant rollback) and investigate

You can see this reflected in my monitoring policies file, where the default-deny has a comment:

# =============================================================================
# DEFAULT DENY — Phase 2 (added after all allow rules verified healthy)
# =============================================================================
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: monitoring

The "Phase 2" comment is the deployment strategy encoded in the manifest. Allow rules first, deny last.

The Numbers

Across the entire cluster: 19 network policy files covering 19 namespaces. The simplest app (IT-Tools, a static web app) has 5 policy resources. The most complex (monitoring) has 14. The total YAML for network policies alone is over 800 lines.

Is it worth it? On a single-node homelab, the security benefit is modest: there's only one user and one node. But the operational discipline is immense. Writing network policies forces you to map every traffic flow in your cluster. You can't write a policy for "allow Miniflux to reach Postgres" without understanding that Miniflux talks to Postgres on TCP 5432 with a specific pod label. That understanding is the real value.

When I look at network policies at work now, they're not abstract YAML. They're traffic flow maps I can read fluently. That fluency was built one homelab policy at a time.

This post is part of a series documenting my homelab build on Talos Linux. Next up: push notifications with ntfy as the homelab alert bus. You can find the full series on the blog index.