Homelab Runbook: legacy (control plane) + gpu (data plane)

A step-by-step guide to build a two-node Kubernetes homelab with public ingress, Headscale VPN, replicated Postgres, Kafka, and VPN-only Ollama.

| Host | IP | Role | |---|---|---| | legacy | 124.123.33.30 (public) / LEGACY_LAN_IP (find with ip a) | k8s control plane, ingress for domain, Headscale server, Postgres primary | | gpu | 192.168.0.44 (LAN only) | k8s worker, Kafka, Postgres read replica, Ollama (NVIDIA GPU) |

Placeholders used throughout — replace before running:

- LEGACY_LAN_IP — legacy's 192.168.0.x address

- legacytoai.example.com — your real domain

- 100.64.0.X — Tailscale/Headscale IPs assigned after Phase 5

Assumes Ubuntu 22.04/24.04 on both machines. Run commands over ssh legacy / ssh gpu.

---

Phase 0 — Prerequisites (BOTH machines)

sudo apt-get update && sudo apt-get upgrade -y
sudo hostnamectl set-hostname legacy   # or: gpu

# Mutual name resolution
echo "LEGACY_LAN_IP legacy" | sudo tee -a /etc/hosts
echo "192.168.0.44 gpu"     | sudo tee -a /etc/hosts

# Disable swap (kubeadm requirement)
sudo swapoff -a
sudo sed -i '/\sswap\s/s/^/#/' /etc/fstab

# Kernel modules + sysctl for Kubernetes
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay && sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sudo sysctl --system

Firewall

legacy:

sudo ufw allow from 192.168.0.0/24 to any port 22 proto tcp
sudo ufw allow 80,443/tcp                       # public ingress + headscale
sudo ufw allow 3478/udp                         # DERP STUN
sudo ufw allow 41641/udp                        # tailscale
sudo ufw allow from 192.168.0.0/24 to any port 6443 proto tcp   # k8s API (LAN)
sudo ufw allow from 192.168.0.0/24              # cluster traffic (CNI, kubelet)
sudo ufw enable

gpu — accept traffic only from the LAN; nothing public, no router forwarding:

sudo ufw default deny incoming
sudo ufw allow from 192.168.0.0/24
sudo ufw enable

After Phase 5 (VPN), also allow 100.64.0.0/10 on both and remove any public SSH exposure.

---

Phase 1 — Docker + containerd (BOTH machines)

# Docker apt repo
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
  https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER   # re-login to take effect

# containerd config for Kubernetes (SystemdCgroup)
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd && sudo systemctl enable containerd

---

Phase 2 — NVIDIA GPU support (gpu only)

# Driver
sudo ubuntu-drivers install
sudo reboot
nvidia-smi   # must show the GPU

# nvidia-container-toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit

# Wire into containerd
sudo nvidia-ctk runtime configure --runtime=containerd
sudo systemctl restart containerd

# Sanity test via docker
sudo docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

The k8s device plugin is deployed after the cluster exists (Phase 3, last step).

---

Phase 3 — Kubernetes with kubeadm

3.1 Install kubeadm/kubelet/kubectl (BOTH)

sudo apt-get install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

3.2 Init control plane (legacy)

sudo kubeadm init \
  --pod-network-cidr=192.168.128.0/17 \
  --apiserver-advertise-address=LEGACY_LAN_IP \
  --apiserver-cert-extra-sans=LEGACY_LAN_IP,124.123.33.30,legacy

mkdir -p $HOME/.kube
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Pod CIDR 192.168.128.0/17 avoids clashing with the home LAN 192.168.0.0/24.

Save the printed kubeadm join ... command. Regenerate later with

kubeadm token create --print-join-command.

After Headscale is up (Phase 5), you may re-issue certs adding the 100.64.x IP as a SAN

if you want kubectl over VPN, or simply use kubectl via SSH.

3.3 CNI — Calico (chosen for NetworkPolicy support)

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/tigera-operator.yaml
# Installation CR with our pod CIDR
cat <<EOF | kubectl apply -f -
apiVersion: operator.tigera.io/v1
kind: Installation
metadata: { name: default }
spec:
  calicoNetwork:
    ipPools:
    - cidr: 192.168.128.0/17
      encapsulation: VXLAN
EOF

3.4 Join worker (gpu)

Run the saved sudo kubeadm join ... on gpu, then on legacy:

kubectl get nodes -o wide          # both Ready
kubectl label node gpu node-role.kubernetes.io/worker= tier=data
kubectl label node legacy tier=control

Keep the control-plane taint on legacy so general workloads land on gpu; things that must run on legacy (ingress, Postgres primary) get an explicit toleration.

3.5 NVIDIA device plugin

kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.16.2/deployments/static/nvidia-device-plugin.yml
kubectl describe node gpu | grep nvidia.com/gpu   # capacity: 1

3.6 Helm + local storage (on legacy)

curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.30/deploy/local-path-storage.yaml
kubectl patch storageclass local-path -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

---

Phase 4 — Public ingress, domain, TLS

1. Router: forward TCP 80 and 443 → LEGACY_LAN_IP. Nothing to gpu, ever.
2. DNS at your registrar:

- A legacytoai.example.com → 124.123.33.30 - A *.legacytoai.example.com → 124.123.33.30 - If the ISP IP is dynamic: set up Cloudflare DDNS or ddclient.

1. ingress-nginx, pinned to legacy with hostNetwork (simplest single-entry-node setup):

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx && helm repo update
helm install ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx --create-namespace \
  --set controller.hostNetwork=true \
  --set controller.kind=DaemonSet \
  --set controller.nodeSelector.tier=control \
  --set controller.tolerations[0].key=node-role.kubernetes.io/control-plane \
  --set controller.tolerations[0].operator=Exists \
  --set controller.service.type=ClusterIP

1. cert-manager + Let's Encrypt:

helm repo add jetstack https://charts.jetstack.io && helm repo update
helm install cert-manager jetstack/cert-manager -n cert-manager --create-namespace --set crds.enabled=true

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata: { name: letsencrypt }
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef: { name: letsencrypt-key }
    solvers:
    - http01: { ingress: { ingressClassName: nginx } }
EOF

Any Ingress annotated cert-manager.io/cluster-issuer: letsencrypt now gets auto-TLS.

Verify: point a test Ingress at any service, browse https://test.legacytoai.example.com from mobile data — valid certificate.

---

Phase 5 — Headscale VPN (legacy)

Run Headscale as a systemd service outside k8s so the VPN survives cluster problems.

# On legacy — install (check latest release)
HS_VER=0.23.0
wget https://github.com/juanfont/headscale/releases/download/v${HS_VER}/headscale_${HS_VER}_linux_amd64.deb
sudo dpkg -i headscale_${HS_VER}_linux_amd64.deb

Edit /etc/headscale/config.yaml:

server_url: https://headscale.legacytoai.example.com
listen_addr: 127.0.0.1:8080
prefixes:
  v4: 100.64.0.0/10
derp:
  server:
    enabled: true
    stun_listen_addr: "0.0.0.0:3478"
dns:
  magic_dns: true
  base_domain: ts.legacytoai.example.com

Expose it through the cluster ingress via an ExternalName-style path, or simpler: a dedicated Caddy/nginx on legacy for headscale.legacytoai.example.com → 127.0.0.1:8080 (Headscale needs WebSocket/long-poll passthrough). Since ingress-nginx owns :443 via hostNetwork, create a k8s Service+Endpoints pointing at the host:

apiVersion: v1
kind: Service
metadata: { name: headscale, namespace: default }
spec: { ports: [ { port: 8080 } ] }
---
apiVersion: v1
kind: Endpoints
metadata: { name: headscale, namespace: default }
subsets: [ { addresses: [ { ip: LEGACY_LAN_IP } ], ports: [ { port: 8080 } ] } ]
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: headscale
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
  ingressClassName: nginx
  tls: [ { hosts: [headscale.legacytoai.example.com], secretName: headscale-tls } ]
  rules:
  - host: headscale.legacytoai.example.com
    http: { paths: [ { path: /, pathType: Prefix, backend: { service: { name: headscale, port: { number: 8080 } } } } ] }

(Change listen_addr to 0.0.0.0:8080 for this to be reachable from pods; ufw already limits it to LAN.)

Start and enroll devices:

sudo systemctl enable --now headscale
headscale users create arun
headscale preauthkeys create --user arun --expiration 24h

# On legacy, gpu, laptop, phone (Tailscale app for phone):
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --login-server https://headscale.legacytoai.example.com --authkey <KEY>

# Optional: reach whole home LAN via legacy
sudo tailscale up --login-server ... --advertise-routes=192.168.0.0/24   # on legacy
headscale routes list && headscale routes enable -r <route-id>

Update laptop ~/.ssh/config to the 100.64.x IPs (or MagicDNS names legacy.ts.legacytoai.example.com), verify SSH from mobile data, then allow `100.64.0.0/10` in ufw on both machines and remove any public SSH forwarding.

---

Phase 6 — Postgres: primary on legacy, read replica on gpu (CloudNativePG)

helm repo add cnpg https://cloudnative-pg.github.io/charts && helm repo update
helm install cnpg cnpg/cloudnative-pg -n cnpg-system --create-namespace
kubectl create namespace db

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata: { name: pg, namespace: db }
spec:
  instances: 2                      # 1 primary + 1 streaming replica
  primaryUpdateStrategy: unsupervised
  storage: { size: 50Gi, storageClass: local-path }
  affinity:
    tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
    podAntiAffinityType: required   # one instance per node
    topologyKey: kubernetes.io/hostname

Anti-affinity guarantees one instance on each node. If the primary lands on gpu instead of legacy, swap roles once:

kubectl get pods -n db -o wide     # see which pod/node is primary
kubectl cnpg status pg -n db       # requires the cnpg kubectl plugin
kubectl cnpg promote pg <replica-on-legacy> -n db

Connection endpoints (auto-created):

• pg-rw.db.svc:5432 — primary (writes) on legacy
• pg-ro.db.svc:5432 — read replica on gpu
• Credentials: kubectl get secret pg-app -n db -o jsonpath='{.data.password}' | base64 -d

Verify replication: insert via pg-rw, select via pg-ro; kubectl cnpg status pg -n db shows a streaming replica; delete the primary pod and confirm the operator recovers.

Backups (replica ≠ backup): enable CNPG ScheduledBackup with Barman to an S3-compatible target (e.g. a MinIO pod on gpu, or external object storage) once the cluster is stable.

---

Phase 7 — Kafka on gpu (Strimzi, KRaft)

helm repo add strimzi https://strimzi.io/charts && helm repo update
helm install strimzi strimzi/strimzi-kafka-operator -n kafka --create-namespace

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: dual
  namespace: kafka
  labels: { strimzi.io/cluster: my-kafka }
spec:
  replicas: 1
  roles: [controller, broker]
  storage: { type: persistent-claim, size: 30Gi, class: local-path }
  template:
    pod:
      metadata: {}
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions: [ { key: tier, operator: In, values: [data] } ]
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-kafka
  namespace: kafka
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 3.7.0
    listeners:
    - { name: plain, port: 9092, type: internal, tls: false }
  entityOperator: { topicOperator: {}, userOperator: {} }

Bootstrap from inside the cluster: my-kafka-kafka-bootstrap.kafka.svc:9092.

Verify:

kubectl -n kafka run producer -ti --image=quay.io/strimzi/kafka:latest-kafka-3.7.0 --rm \
  -- bin/kafka-console-producer.sh --bootstrap-server my-kafka-kafka-bootstrap:9092 --topic test
kubectl -n kafka run consumer -ti --image=quay.io/strimzi/kafka:latest-kafka-3.7.0 --rm \
  -- bin/kafka-console-consumer.sh --bootstrap-server my-kafka-kafka-bootstrap:9092 --topic test --from-beginning

---

Phase 8 — Ollama on gpu, VPN-only

8.1 Migrate existing models

On gpu, find the current docker volume and copy models so they aren't re-downloaded:

docker volume inspect <ollama-volume>          # note Mountpoint
# after the PVC exists, copy Mountpoint contents into the local-path PVC dir
# (local-path PVCs live under /opt/local-path-provisioner/ on the node)

8.2 Deploy

apiVersion: v1
kind: Namespace
metadata: { name: ai }
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: ollama-models, namespace: ai }
spec: { accessModes: [ReadWriteOnce], resources: { requests: { storage: 100Gi } }, storageClassName: local-path }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: ollama, namespace: ai }
spec:
  replicas: 1
  selector: { matchLabels: { app: ollama } }
  template:
    metadata: { labels: { app: ollama } }
    spec:
      nodeSelector: { tier: data }
      containers:
      - name: ollama
        image: ollama/ollama:latest
        ports: [ { containerPort: 11434 } ]
        resources: { limits: { nvidia.com/gpu: 1 } }
        volumeMounts: [ { name: models, mountPath: /root/.ollama } ]
      volumes:
      - name: models
        persistentVolumeClaim: { claimName: ollama-models }
---
apiVersion: v1
kind: Service
metadata: { name: ollama, namespace: ai }
spec:
  type: NodePort
  selector: { app: ollama }
  ports: [ { port: 11434, nodePort: 31434 } ]

8.3 VPN-only exposure

The NodePort (31434) is open on both nodes' interfaces; ufw is what restricts it. On legacy, allow it only from the tailnet, and keep gpu's ufw LAN+VPN-only:

sudo ufw allow in on tailscale0 to any port 31434 proto tcp
# ensure NO general 'allow 31434' rule and no router forward for it

Clients on the VPN use legacy's tailnet address: http://100.64.0.X:31434 (or http://legacy.ts.legacytoai.example.com:31434).

8.4 Continue / coding-agent config (laptop on VPN)

{
  "models": [{
    "title": "Ollama homelab",
    "provider": "openai",
    "apiBase": "http://legacy.ts.legacytoai.example.com:31434/v1",
    "apiKey": "ollama",
    "model": "qwen2.5-coder:14b"
  }]
}

Verify from mobile data with VPN on: curl http://legacy.ts.legacytoai.example.com:31434/v1/models → model list; and without VPN the same URL/IP is unreachable.

---

Phase 9 — Lockdown checklist

• [ ] Router: only 80, 443 (TCP) and 3478 (UDP) forwarded, all to legacy. Zero forwards to gpu.
• [ ] ufw status on gpu: only 192.168.0.0/24 and 100.64.0.0/10 allowed inbound.
• [ ] k8s API 6443 on legacy: LAN + tailnet only, not public.
• [ ] SSH: key-only auth, no public exposure — VPN/LAN only.
• [ ] Calico NetworkPolicies: default-deny in db, kafka, ai; allow only from app namespaces and ingress-nginx.
• [ ] CNPG scheduled backups configured (the replica is HA, not a backup).
• [ ] External scan (from mobile data, no VPN): nmap 124.123.33.30 shows only 80/443.

Operational notes

• New worker join command: kubeadm token create --print-join-command (on legacy).
• New VPN device: headscale preauthkeys create --user arun.
• TLS renewals: automatic via cert-manager; check kubectl get certificate -A.
• Upgrades: kubeadm upgrade plan (unhold packages first); Headscale via new .deb.
• If public IP changes: update DNS A records (or rely on DDNS); VPN keeps working via DERP once clients reconnect.

Execution order

Phases 0 → 1 → 2 → 3 sequential. 4 and 5 in parallel after 3. 6/7/8 after 3.6 (storage). 9 last.