The Authentik SSO Marathon: 11 Commits to ForwardAuth

Before Authentik, every service's login page was exposed to the internet. TT-RSS, Uptime Kuma, Grafana. Anyone who found the URL could sit there and guess credentials. The apps had their own authentication, but there was nothing stopping the world from reaching those login forms. Authentik adds a gatekeeper: unauthenticated users never see the application at all.

How ForwardAuth Works

The pattern is elegant. Traefik intercepts every request to a protected service and asks Authentik: "does this user have a valid session?" If yes, the request passes through untouched. If no, the user gets redirected to Authentik's login page. After authenticating, they're redirected back to where they were going.

Browser → Cloudflare Tunnel → Traefik → ForwardAuth check
  → Authentik: valid session?
    → Yes: pass through to service
    → No: 302 redirect to auth.naviauxlab.com/login
      → User logs in → cookie set for *.naviauxlab.com
      → Redirect back → ForwardAuth passes

One login covers every service because the session cookie is set on the *.naviauxlab.com domain. Log in once at auth.naviauxlab.com, and TT-RSS, Uptime Kuma, and Grafana all recognize the session.

The Stack

Authentik runs as a Flux HelmRelease in the auth namespace with four components:

• Authentik Server: handles authentication flows, user management, and the admin UI
• Authentik Worker: background tasks like email, cleanup, and outpost coordination
• PostgreSQL: stores users, sessions, and application configuration
• Redis: session caching

The server pod includes an embedded outpost that handles ForwardAuth requests directly. No separate outpost deployment needed. A Proxy Provider in forward_domain mode with cookie domain naviauxlab.com ties everything together.

The ForwardAuth Middleware

This is the bridge between Traefik and Authentik:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: authentik-forwardauth
  namespace: auth
spec:
  forwardAuth:
    address: http://authentik-server.auth.svc.cluster.local/outpost.goauthentik.io/auth/traefik
    trustForwardHeader: true
    authResponseHeaders:
      - X-authentik-username
      - X-authentik-uid
      - X-authentik-name
      - X-authentik-email
      - X-authentik-groups

Services opt in via an Ingress annotation:

annotations:
  traefik.ingress.kubernetes.io/router.middlewares: auth-authentik-forwardauth@kubernetescrd

For services that also need the HTTPS scheme fix from the Cloudflare Tunnel setup, middlewares chain with commas:

traefik.ingress.kubernetes.io/router.middlewares: auth-authentik-forwardauth@kubernetescrd,rss-https-scheme@kubernetescrd

The 11-Commit Debugging Marathon

This was the hardest stage of the entire build. What should have been "deploy a Helm chart and configure a middleware" turned into 11 commits spanning chart migrations, SOPS issues, image tag deprecation, and password injection bugs.

Commit 1-2: macOS vs Linux tooling

Deployment scripts used grep -oP (Perl regex), which doesn't exist on macOS. Switched to grep -oE. Then sed -i '' (macOS BSD sed) mangled a kustomization file by appending a new entry onto the end of an existing line instead of adding a new line. Replaced with Python string manipulation.

Commit 3-4: SOPS path mismatches

Secrets created in /tmp/ didn't match the .sops.yaml creation rules (path_regex: .*\.sops\.yaml$). SOPS refused to encrypt them with "no matching creation rules found." Had to create the secret files directly in the repo path. Then the HelmRepository was created in the wrong directory: the pattern is one HelmRepository per component directory, not a shared infrastructure/sources/ folder.

Commit 5-6: Helm values schema evolution

First attempt used valueFrom.secretKeyRef in the HelmRelease values. That's Deployment syntax, not Flux syntax. Flux uses valuesFrom with targetPath at the HelmRelease spec level. Then the chart's env key was deprecated in version 2024.2+ in favor of global.env.

Commit 7-8: Bitnami image tag graveyard

The Authentik Helm chart uses Bitnami subcharts for PostgreSQL and Redis. Bitnami regularly removes old image tags from Docker Hub. The specific tag in the chart's defaults was gone. Tried overriding tags, but Bitnami subcharts mount paths as read-only that the official PostgreSQL image expects to be writable. The /var/run/postgresql directory threw Read-only file system errors. The fix: upgrade to chart version 2025.*, which uses official Docker Hub images instead of Bitnami.

Commit 9: The dual password path

This one was subtle. PostgreSQL was running fine. Authentik server kept failing with fe_sendauth: no password supplied. The chart needs the PostgreSQL password injected at two paths:

valuesFrom:
  # Path 1: Configures the PostgreSQL subchart
  - kind: Secret
    name: authentik-secrets
    valuesKey: postgres_password
    targetPath: postgresql.auth.password
  # Path 2: Tells Authentik what password to use when connecting
  - kind: Secret
    name: authentik-secrets
    valuesKey: postgres_password
    targetPath: authentik.postgresql.password

postgresql.auth.password sets the password on the database. authentik.postgresql.password tells the application what password to use. Miss the second one and the database works fine but the app can't connect.

Commit 10-11: YAML and annotation cleanup

Grafana's HelmRelease had duplicate annotations: blocks at the same YAML level, causing an unmarshal error that blocked all of infrastructure reconciliation, not just Grafana. Then wiring ForwardAuth to the protected services and adding outpost callback routes for the forward_single auth mode.

What Gets Protected

Defense in Depth: Three More Layers

Authentik SSO was the trigger for hardening the entire stack. Three additional security layers went in at the same time.

Cloudflare Access (Edge Authentication)

Cloudflare Access adds an email OTP gate at the Cloudflare edge, before traffic even reaches the tunnel. Only [email protected] can pass. Protected services (rss, uptime, grafana) require the OTP. The public portfolio and Authentik itself are bypassed (the portfolio is public by design, and Authentik must be reachable for the login flow to work).

Security Headers Middleware

A Traefik Middleware that injects hardened response headers on every HTTP response:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: security-headers
  namespace: auth
spec:
  headers:
    contentTypeNosniff: true
    browserXssFilter: true
    frameDeny: true
    customFrameOptionsValue: "SAMEORIGIN"
    stsIncludeSubdomains: true
    stsPreload: true
    stsSeconds: 31536000
    referrerPolicy: "strict-origin-when-cross-origin"
    permissionsPolicy: "camera=(), microphone=(), geolocation=(), payment=()"

HSTS with preload, X-Frame-Options, content type sniffing protection, and a restrictive permissions policy. Applied to all services, including the public portfolio.

Cloudflare Dashboard Hardening

SSL/TLS set to Full (Strict), minimum TLS 1.2, Always Use HTTPS enabled, Automatic HTTPS Rewrites on, Browser Integrity Check active. Belt and suspenders.

The Full Security Stack

Seven layers. Each independent. Losing any one doesn't compromise the others.

What I Learned

Helm charts with database subcharts often need the password at two paths. One configures the database, one tells the application how to connect. The subchart docs don't always make this obvious. When the app can't connect but the database is running, check for a missing application-side password path.

Bitnami image tags are ephemeral. Bitnami removes old tags from Docker Hub regularly. If your Helm chart pins a Bitnami image tag and that tag gets deleted, your next pod restart pulls nothing. Use charts that reference official images when possible, or pin to tags that are actively maintained.

Duplicate YAML keys break everything, not just the affected service. A duplicate annotations: block in one HelmRelease caused a kustomize build failure that blocked the entire infrastructure Kustomization. Fourteen services went unreconciled because of one YAML typo in one file.

Defense in depth isn't overkill. It's the minimum. Edge authentication, cluster SSO, security headers, app-level auth, network policies, and zero open ports. Each layer catches what the others miss. The OTP at Cloudflare stops bots before they ever reach the cluster. ForwardAuth stops humans without accounts. Network policies stop lateral movement if something inside gets compromised. No single layer is sufficient.

Eleven commits to get a login page working. But now every service is behind seven layers of security, and I log in once for all of them.