Adopting Flux: GitOps Migration for a Running Cluster

My cluster was running. Services were up. But everything had been deployed with kubectl apply and helm install: imperative commands that live only in terminal history. If the node dies, I'm rebuilding from memory. Flux turns the Git repo into the single source of truth: every manifest, every Helm release, every config change is versioned, auditable, and automatically applied.

What GitOps Actually Means

GitOps is a pattern where your Git repository defines the desired state of your cluster. A controller running inside the cluster continuously watches the repo and reconciles any drift. If someone manually changes something, Flux reverts it. If you push a new manifest, Flux applies it. The repo is the cluster.

This sounds great on paper. In practice, migrating a running cluster to GitOps has some sharp edges.

Bootstrapping Flux

Flux needs a CLI tool on your admin workstation and a GitHub repo to store manifests. The bootstrap command installs Flux controllers and connects them to the repo:

export GITHUB_TOKEN=<your-pat>

flux bootstrap github \
  --owner=snaviaux \
  --repository=home-ops \
  --branch=main \
  --path=clusters/homelab \
  --personal

This does five things in sequence:

1. Verifies the GitHub repo exists (or creates it)
2. Installs four controllers: source-controller (fetches Git repos), kustomize-controller (applies manifests), helm-controller (manages Helm releases), notification-controller (sends alerts)
3. Creates a clusters/homelab/flux-system/ directory with Flux's own manifests
4. Sets up an ECDSA deploy key on the repo
5. Verifies everything reconciles

One gotcha: your GitHub PAT needs Administration (Read & Write) permission, not just Contents. Without it, Flux can't create the deploy key and you get a cryptic 403 Resource not accessible by personal access token.

The Repository Structure

The structure that worked best is a two-layer approach with dependency ordering:

home-ops/
├── clusters/homelab/
│   ├── flux-system/            # Flux's own manifests (auto-generated)
│   ├── infrastructure.yaml     # Flux Kustomization → watches ./infrastructure
│   └── apps.yaml               # Flux Kustomization → watches ./apps
├── infrastructure/
│   ├── kustomization.yaml      # Lists all subdirectories
│   ├── cilium/
│   ├── traefik/
│   ├── longhorn/
│   └── cert-manager/
└── apps/
    ├── kustomization.yaml
    ├── uptime-kuma/
    └── ttrss/

The key design decision: apps.yaml includes dependsOn: infrastructure. Flux will always fully reconcile infrastructure (Cilium, Traefik, Longhorn, cert-manager) before attempting apps. No more deploying an app before its storage driver or ingress controller is ready.

Converting Helm Installs to HelmReleases

Instead of helm install commands, each Helm-managed component gets declarative manifests. Here's what Traefik looks like as a Flux HelmRelease:

# helmrepository.yaml — where to find the chart
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: traefik
  namespace: flux-system
spec:
  interval: 24h
  url: https://traefik.github.io/charts
---
# helmrelease.yaml — the declarative equivalent of helm install
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: traefik
  namespace: traefik
spec:
  interval: 30m
  chart:
    spec:
      chart: traefik
      version: ">=34.0.0"
      sourceRef:
        kind: HelmRepository
        name: traefik
        namespace: flux-system
  values:
    service:
      type: LoadBalancer
    providers:
      kubernetesCRD:
        enabled: true
        allowCrossNamespace: true
      kubernetesIngress:
        enabled: true

Every value that was a --set flag in the Helm install command becomes a key in the values: section. Same configuration, but version-controlled and auditable.

The Adoption Problem

Here's where the migration gets painful. If you've already installed something with helm install, Flux can't directly adopt it. It sees a release it didn't create and refuses to touch it. The error is unhelpful:

"traefik" has no deployed releases

The fix is to delete the Helm release tracking secret, not the actual pods or resources, just the internal bookkeeping that Helm uses to track releases:

# Find the existing release secret
kubectl get secrets -n traefik -l owner=helm

# Delete just the tracking metadata
kubectl delete secret -n traefik -l owner=helm,name=traefik

# Force Flux to retry
flux reconcile helmrelease traefik -n traefik

The actual workloads keep running. You're only removing Helm's internal state so Flux can take over and create its own tracking. This pattern applies to every Helm release you need to adopt.

The Imperative Trap

The hardest habit to break is kubectl apply. After setting up Flux, I patched a Kustomization with kubectl patch to add SOPS decryption config. It worked. For about 60 seconds, until Flux reconciled from Git and reverted my change.

This is GitOps working as designed. If it's not in Git, it doesn't exist. The decryption config had to go in the Git manifest (clusters/homelab/apps.yaml), not applied imperatively:

# clusters/homelab/apps.yaml
spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-age

The same lesson applies everywhere. Manual kubectl set env changes get reverted. Manual ConfigMap updates get overwritten. Manual anything gets undone within minutes. Git is the only interface.

How Changes Work Now

This is the payoff. To change anything in the cluster:

1. Edit the YAML in the home-ops repo
2. git add, git commit, git push
3. Wait ~60 seconds (or flux reconcile source git flux-system to trigger immediately)
4. Flux applies the change automatically

To add a new app: create a directory under apps/, add manifests, reference it in apps/kustomization.yaml, push. Done.

To upgrade a Helm chart: bump the version in the HelmRelease, push. Flux handles the rest.

No more kubectl apply -f. No more helm upgrade. Every change is a Git commit with a message, a diff, and a revert path.

Verification

flux get kustomizations
# NAME            REVISION            READY  MESSAGE
# apps            main@sha1:4964a8af  True   Applied revision: main@sha1:4964a8af
# flux-system     main@sha1:4964a8af  True   Applied revision: main@sha1:4964a8af
# infrastructure  main@sha1:4964a8af  True   Applied revision: main@sha1:4964a8af

All three Ready: True. Zero non-running pods. Both services responding. If the cluster were destroyed tomorrow, git clone + flux bootstrap rebuilds everything except persistent data.

What I Learned

Start with Flux from day one. Migrating a running cluster to GitOps means deleting Helm release secrets and letting Flux reinstall things. It works, but it's easier to never have the adoption problem in the first place.

Every manual change gets reverted. This isn't a bug. It's the entire point. If a manual fix "stops working after a few minutes," Flux reverted it because it wasn't in Git. Commit everything.

dependsOn prevents deployment races. Without explicit ordering, Flux will try to create an Ingress before Traefik is ready, or a PVC before Longhorn is installed. Always make apps depend on infrastructure.

Git becomes your audit log. Every change to the cluster is a commit with a timestamp, a message, and a diff. Six months from now, when you wonder "why did I configure Traefik this way," the answer is in git log.