Khaled ArsalaneDevOps & Platform Engineer
← Projects

homelab

How my Kubernetes homelab is built

Ten machines, one Kubernetes (k3s) cluster and about thirty self-hosted apps, run the way I’d run production. Everything, from the OS install to each app’s deployment, is declared in git.

From the apps down to the hardware

Apps & AIOps

About thirty apps, each deployed from git and behind single sign-on. The AI work runs on local hardware.

AIOps

On-demand GPU
  1. Request to the LLM
  2. Wake proxy in the cluster
  3. Wake-on-LAN
  4. Ollama answers

When idle, the workstation suspends itself: GPU capacity with near-zero idle power.

LLM alert digest
  1. Alertmanager
  2. LLM groups and summarises
  3. chat

Related alerts are grouped and summarised before they reach my phone: one incident, one message.

LLM first triage
  1. Alert or failed Argo CD sync
  2. Matching Loki logs
  3. LLM drafts a first triage

Alerts, failed syncs and degraded apps each come with a first read of what went wrong, drafted from the logs around them.

Nightly new error patterns
  1. Today’s Loki errors
  2. Compared with yesterday’s
  3. Only new patterns posted

Errors that were already there yesterday stay quiet; only new patterns reach the chat.

Weekly alert-noise report
  1. A week of alert history
  2. LLM reads it
  3. Rules to tune or drop

The model reads the week’s alert history from VictoriaMetrics and suggests which rules to tune or drop.

LLM PR reviewer

A local model reviews Renovate’s dependency PRs and comments on them.

Agent workspace

An AI agent workspace running in the cluster, with its own Vault access.

Self-hosted apps

Personal data
  • Immich
  • Nextcloud
  • Paperless-ngx
  • Vaultwarden
Home
  • Home Assistant
  • Homepage
Media
  • Servarr
Tools
  • Stirling-PDF
  • IT-Tools
  • draw.io
Built by me

My Kubernetes operator for freqtrade trading bots: bots, strategies and backtests are custom resources. Paper trading only.

Delivery & lifecycle

Every layer is declared in git, so changing the cluster means making a commit.

Declared in git, layer by layer

git
  1. OS install
    Fedora CoreOS · Butane / Ignition

    An immutable OS, installed from a config file.

  2. Host setup
    Ansible

    Network and storage roles, run against every machine.

  3. Kubernetes version
    System Upgrade Controller

    Upgrade plans: the version lives in a file.

  4. Platform & apps
    Argo CD · app-of-apps

    About thirty apps, split into infrastructure and app projects.

  5. Secrets
    Vault

    Only references in git; the values are synced from Vault.

Upgrades at every level

Node OS
  1. New CoreOS release
  2. Node updates itself
  3. Waits its turn: one node at a time
  4. Drained, rebooted, back in service
Kubernetes
  1. Version bumped in a plan
  2. Commit
  3. Nodes upgrade themselves
Apps
  1. Renovate opens a PR
  2. Local LLM reviews it
  3. Merge
  4. Argo CD rolls it out

Two pipelines

In-cluster apps
  1. Push
  2. Gitea Actions builds the image
  3. Argo CD deploys from git

Gitea and its CI runners, amd64 and arm64, run in the cluster itself.

This website
  1. dev: lint, type-check, build
  2. staging: served by Apache, smoke-tested
  3. main: deployed, smoke-tested again

The page you’re reading shipped through it, on the cluster’s own runners.

Edge & security

Every app has a single name. From the internet it leads through Cloudflare and a web application firewall; at home, the same name leads straight to a private gateway. No inbound port is open.

One name, two paths

<app>.<domain>Split-horizon DNS: the answer depends on where the question comes from.
From the internet
  1. Cloudflare DNS

    Public records, created from each app’s manifest. Only public apps get one.

  2. Cloudflare Tunnel

    Outbound-only: no open port and no exposed home IP.

  3. BunkerWeb

    Web application firewall and public gateway, with rules and rate limits per app.

From the home network
  1. Technitium

    Answers for the same zone at home, from three replicas.

  2. MetalLB

    Gives the private gateway its address on the home network.

  3. NGINX Gateway Fabric

    The private gateway: traffic from home never leaves the house.

Both paths end in the same place

Cluster, default-deny network
  1. Authentik

    Sign-in on both paths: OIDC where the app supports it, a proxy where it doesn’t. MFA required.

  2. App

    Private-only apps have no public record at all.

Two external-dns instances write both views from the same manifests, so no DNS record is edited by hand.

Around the paths

Cilium

Default-deny across the cluster; each app opens only the flows it needs. From the home network, the nodes answer only SSH, the Kubernetes API and ping.

WireGuard

Traffic between nodes is encrypted, even inside the home network.

Vault

Three-node raft cluster with auto-unseal. Secrets are synced into apps, never stored in git.

cert-manager

TLS certificates issued and renewed automatically, wildcards included.

Pod Security Admission

Every namespace has a security level: baseline, privileged only where host access is needed.

Argo CD

The built-in admin account is disabled: access goes through SSO with MFA.

Hardware, power & data

Ten machines serve the cluster: seven run k3s, three support it.

The machines

Home network
UPS

Keeps storage, the control plane and the network up through outages. Alerts over SNMP when on battery.

Routeron UPS

No inbound port forwarded.

Switchon UPS
k3s cluster
4 amd64 nodes

Fedora CoreOS. Three form a highly available control plane; one is a worker.

3 arm64 Raspberry Pis

Fedora CoreOS too. Cheap extra capacity; heavy workloads are kept off them.

NASon UPS

Holds backups outside the cluster they protect.

GPU workstation

Serves local LLMs. Sleeps when idle, woken on demand.

Off-site
Cloud VPS

Provisioned and firewalled as code, linked over WireGuard.

Off-site backup

A second copy of the NAS backups, outside the house.

On the UPS: the NAS, 1 control-plane node, the router and the switch.

Where data goes

  1. Longhorn volumes

    Replicated across nodes, with a snapshot every hour.

  2. Database dumps

    Daily, consistent at the application level. etcd snapshots are copied off the nodes too.

  3. NAS

    Nightly backups of every volume.

  4. Off-site backup

    The last line if the house itself is the problem.

Restores are rehearsed on a scratch database, not just planned.

Watching all of it

Metrics
VictoriaMetrics, Prometheus
Logs
Loki, Promtail
Traces
Tempo
Probes
Blackbox exporter
Hardware
SNMP
Dashboards
Grafana
Alerts
Alertmanager → chat

SNMP covers the UPS and the NAS, so a power cut raises an alert like any other failure.