homelab

The homelab is where I practice the job without a change-review board. It runs about forty workloads (AI inference, home automation, media, file sync, CI and a handful of game servers) on a single k3s node, next to a Proxmox server it also monitors. The node is rebuilt from one NixOS flake, and Git is the only way anything changes.

The repository is private because a homelab accumulates personal data. This page covers the architecture and the decisions behind it.

Goals

• Rebuildable from scratch. If the disk dies, reinstalling means booting the installer, applying the flake, and letting GitOps put every workload back.
• Git is the control plane. Cluster changes go through commits that ArgoCD reconciles, and host changes go through the flake.
• Real practice. The same patterns I’d use at work (GitOps, private registry, image automation, metrics) at a scale where I can break things safely.

Architecture

┌────────────────────────── NixOS host (one flake) ──────────────────────────┐
│ hardware · networking · storage (NFS/SMB) · nvidia · tailscale             │
│ backup · agenix secrets · node-exporter · custom k3s 1.34 build            │
│                                                                            │
│   ┌──────────────────────────────── k3s ─────────────────────────────────┐ │
│   │ MetalLB (L2) ──► Traefik ──► ~45 hostnames, private-CA TLS           │ │
│   │                                                                      │ │
│   │ ArgoCD (app of apps) ◄── Git                                         │ │
│   │  ├─ platform:  metallb, traefik, cert-manager CA,                    │ │
│   │  │             sealed-secrets                                        │ │
│   │  ├─ delivery:  Argo Events + Workflows (Kaniko), Harbor              │ │
│   │  └─ workloads: AI, automation, media, storage, games                 │ │
│   │                                                                      │ │
│   │ Prometheus + Grafana ◄── exporters (node, apps, Proxmox)             │ │
│   └──────────────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────────────┘

By the numbers:

• 32 LoadBalancer services
• 45 ingress hostnames
• 35+ namespaces with external access
• 15 NixOS modules
• ~650 commits in the first year

The host: one flake, fifteen modules

The machine is a nixosConfigurations entry in a flake that pins nixos-unstable, with nixos-hardware, nixvim and agenix as inputs. Each concern is its own module with its own options (modules.monitoring.enable, modules.backup.enable, …), so the host file reads like an inventory instead of a script. Host-level secrets are age-encrypted with agenix and decrypted at activation; they never touch the Nix store in plaintext.

Patching Kubernetes with Nix

k3s 1.33 had a startup race: CNI symlinks were created before the data directory was in place, so pods on a fresh boot could fail networking. The fix landed upstream in 1.34.1 before nixpkgs packaged it. Instead of waiting, I wrote a custom derivation for k3s 1.34.1+k3s1, pinning the versions, chart versions, image list and hashes, and I override the system k3s with a NixOS module. Going back to the stock package is a one-line change. This is the reason to run Kubernetes on Nix: the cluster runtime is just another pinned, reviewable input.

GitOps: ArgoCD, after trying to run two tools

Everything in the cluster is an ArgoCD Application, generated from an app-of-apps. Plain manifests are organised with Kustomize, and third-party software is deployed as ArgoCD Helm applications.

It didn’t start that way. Early on, some workloads were Flux HelmReleases and others were ArgoCD Applications, and only four apps were actually registered in ArgoCD. The fix was a migration:

• every workload became an ArgoCD Application;
• the Helm releases became ArgoCD Helm sources;
• storage and namespace setup moved into separate “prereqs” applications, so ordering is explicit;
• every workload folder was normalised (namespace declared, one resource per file).

Lesson: pick one reconciler per cluster. Two controllers that each think they own the desired state is how you get drift you can’t explain.

Delivery: from git push to a running pod

GitHub push ──► Argo Events (webhook) ──► Argo Workflows
                                                    │
                                        test ─► Kaniko build (no Docker daemon)
                                                    │
                                                    ▼
                         Harbor (private registry, robot accounts, own CA)
                                                    │
                          workflow step: kustomize edit set image, commit to Git
                                                    │
                                                    ▼
                                       ArgoCD syncs ──► rollout

The pipeline never touches the cluster directly. Its last step writes the new image tag into the Kustomize overlay and commits it, so every deploy is a Git commit you can read, revert or bisect. Adding CI for a new repository is a template copy: an EventSource, a Sensor and a WorkflowTemplate. Before settling on Argo Workflows I ran Tekton, and I also prototyped Woodpecker with ArgoCD Image Updater. Argo won because it shares a UI, RBAC model and GitOps flow with the rest of the platform, so there’s one less system to operate.

Networking and TLS

MetalLB hands out LAN addresses in L2 mode for anything that needs a real IP (DNS, game servers, the registry). Everything HTTP goes through Traefik. A cluster-internal CA, managed by cert-manager, issues certificates for internal hostnames, and Tailscale provides remote access without exposing ports to the internet.

Observability

Prometheus scrapes node-exporter on the host, the cluster, and exporters for individual apps and game servers. Dashboards and alerts live in Git like everything else.

For example, the Proxmox hardware monitoring app is a Kustomize package that ArgoCD deploys. It contains:

• an Endpoints/Service pair pointing at the Proxmox box;
• an additional Prometheus scrape config for its node-exporter hwmon sensors (15 temperature sensors, fan speeds, voltage rails, PWM duty cycles);
• a Grafana dashboard shipped as a ConfigMap;
• seven alert rules, such as CPU temperature warning and critical, fan failure, NVMe temperature and exporter down.

One piece I’m proud of is a small Rust service (tokio, reqwest, twilight) that runs PromQL queries against Prometheus and publishes live server stats as Discord channel names. It is built for Discord’s rate limits:

• it only updates on meaningful changes (per-metric thresholds);
• it persists its last state to a volume, so restarts don’t spam the API;
• it ships with health checks and resource limits like any other production service.

What I’d do differently

• Enforce secret hygiene from commit one. Secrets management grew from ad-hoc manifests into sealed-secrets over time. A gitleaks pre-commit hook and a “no plaintext Secret kind” policy check in CI on day one would have made that migration unnecessary.
• One of everything. I tried two GitOps controllers and three CI systems. Trying them was worth it, but running more than one at a time wasn’t.
• Real backups, not copies. Today the backup job rsyncs the repo and host config to a second disk. I’d replace that with restic to an off-site target plus scheduled restore tests, because a backup you haven’t restored isn’t a backup.
• Plan for a second node. A single node keeps it cheap and simple, but upgrades mean downtime. A small second node would make replicated storage and drain-and-upgrade maintenance possible.