Ceald's Blog

Homelab

Background

After messing with K3S for a bit I was kind of sick of dealing with the lack of resources on a PI 4 so I brought it to my laptop.

So We Begin!

alt text

First and most important step is to pick the Kubernetes distribution for the right job, I went with K3S for because it comes with traefik but I’d use something like RKE2 or Kubeadm instead because of the customizability next time.

Second is to pick a CNI/Network plugin for Kubernetes, depending on the distribution it might already come with one or maybe you’d want to switch it out. I personally used Cilium because of its observability and Hubble UI. Without a CNI your cluster won’t be able to communicate with anything inside or outside it.

Third is to pick a storage class for persistence, I went with longhorn because of the frontend for it and flexability, longhorn is also distributed/replicated rather than just being local on a node.

Fourth it’d be nice to have a frontend for managing everything. Something like either rancher or headlamp would be best. I went with headlamp because of how lightweight it is. The rest after this is optional but if you want the most Kuber Kubernetes cluster it’d be a good idea to set these up too.

Side Note

The next services that are going to be setup is overkill for a basic homelab but this is an elitist homelab so nothing is overkill!

Other services

Getting metrics with grafana is almost always a must so installing the kubestack is a good idea and loki. These were my values.yaml files:

yaml
# kube-stack-values.yaml
USER-SUPPLIED VALUES:
grafana:
  additionalDataSources: []
  persistence:
    accessModes:
      - ReadWriteOnce
    enabled: true
    size: 10Gi
    storageClassName: longhorn
    type: pvc
  resources:
    # Main Grafana container
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 512Mi
persistence:
  storageClass: longhorn

prometheus:
  prometheusSpec:
    scrapeInterval: 30s

    serviceMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelector: {}
    serviceMonitorNamespaceSelector: {}

    podMonitorSelectorNilUsesHelmValues: false
    podMonitorSelector: {}
    podMonitorNamespaceSelector: {}

    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 50Gi
          storageClassName: longhorn

    resources:
      requests:
        cpu: 500m
        memory: 512Mi
      limits:
        cpu: 1000m
        memory: 2Gi

and loki:

yaml
# loki-values.yaml
loki:
  commonConfig:
    replication_factor: 1

  auth_enabled: false

  isDefault: false

  schemaConfig:
    configs:
      - from: "2024-01-01"
        store: tsdb
        object_store: filesystem
        schema: v13
        index:
          prefix: loki_index_
          period: 24h

  storage:
    type: filesystem
    bucketNames:
      chunks: chunks
      ruler: ruler
      admin: admin

  limits_config:
    allow_structured_metadata: true

deploymentMode: SingleBinary

singleBinary:
  replicas: 1

gateway:
  enabled: false

read:
  replicas: 0

write:
  replicas: 0

backend:
  replicas: 0

querier:
  replicas: 0

queryFrontend:
  replicas: 0

queryScheduler:
  replicas: 0

ingester:
  replicas: 0
distributor:
  replicas: 0
compactor:
  replicas: 0
lokiCanary:
  enabled: false
chunksCache:
  enabled: false
resultsCache:
  enabled: false
test:
  enabled: false
monitoring:
  dashboards:
    enabled: false
  rules:
    enabled: false
  serviceMonitor:
    enabled: false

another thing that would be great for observability is falco! Here’s my values.yaml file:

yaml
# falco-values.yaml
json_output: true
json_include_output_property: true
http_output:
  enabled: true
  url: "http://falco-falcosidekick:2801/"
falcosidekick:
  enabled: true

  config:
    loki:
      hostport: http://loki.monitoring.svc.cluster.local:3100
      format: json
falcoctl:
  config:
    artifact:
      install:
        refs:
          - falco-rules:5
          - falco-incubating-rules:2
          - falco-sandbox-rules:2
      follow:
        refs:
          - falco-rules:5
          - falco-incubating-rules:2
          - falco-sandbox-rules:2

falco is a runtime security tool that detects policy violations like for example reading /etc/shadow and can return an error early using eBPFs.

you’ll get metrics and logs exported so you can view them in grafana.

It’s probably apparent that you won’t be able to access anything because you don’t have any ingresses setup in the cluster yet, that’s where kyverno and cert manager will come into play.

Install kyverno and cert manager then create a cluster issuer and everything else for cert manager like:

yaml
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
  namespace: cert-manager
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: orbit-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: orbit-ca
  secretName: orbit-ca-secret
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-issuer
    kind: Issuer
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: orbit-ca-issuer
spec:
  ca:
    secretName: orbit-ca-secret

this issuer is cluster wide btw so you can get certs signed from this issuer anywhere.

For kyverno policy:

yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: force-external-dns-target
spec:
  admission: true
  background: true
  emitWarning: false
  mutateExistingOnPolicyUpdate: true
  rules:
  - match:
      any:
      - resources:
          kinds:
          - Ingress
    mutate:
      patchStrategicMerge:
        metadata:
          annotations:
            cert-manager.io/cluster-issuer: orbit-ca-issuer
            external-dns.alpha.kubernetes.io/target: 127.0.0.1
      targets:
      - apiVersion: networking.k8s.io/v1
        kind: Ingress
    name: set-target-annotation
    skipBackgroundRequests: true
  validationFailureAction: Audit

You don’t need the external dns annotation in the policy if you’re not going to use external dns but if you do it’d be a good idea to also get etcd or a source for external dns to write to.

The policy will mutate all your ingresses to have the orbit-ca-issuer added so you don’t need to manually add it everytime you make an ingress.

Now for your first ingress on the cluster, you’d write something like this:

yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
  name: prometheus-grafana-ingress
  namespace: monitoring
spec:
  ingressClassName: traefik
  rules:
  - host: dashboard.orbit.orbit
    http:
      paths:
      - backend:
          service:
            name: prometheus-grafana
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - dashboard.orbit.orbit
    secretName: grafana-tls

If you get that ingress like: kubectl get ingress -n monitoring -o yaml you’ll notice that it’s been changed! If not then add this cluster role:

yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kyverno-background-controller-namespace
  labels:
    rbac.kyverno.io/aggregate-to-background-controller: "true"
rules:
  - apiGroups:
      - ""
    resources:
      - namespaces
    verbs:
      - get
      - list
      - watch
      - update
      - patch

Nice, not even halfway done yet! I won’t post ALL my ingresses because that’d be insane.

alt text

Time to get some services for security going for enforcement, security posture, and observability.

Falco

here’s my falco dashboard if you chose to use flaco: dashboard file

Kubescape

Kubescape is a tool for security posture and allows for vulnerability scans on images, here’s my helm installation command for it:

bash
helm upgrade --install kubescape kubescape/kubescape-operator \
                               -n kubescape \
                               --create-namespace \
                               --set capabilities.continuousScan=enable \
                               --set capabilities.prometheusExporter=enable \
                               --set kubescape.serviceMonitor.enabled=false \
                               --set clusterName=default

you’ll notice that the service monitor is disabled that’s because it’s actually broken and does not work with newer versions of the kubestack, here’s the manifest I used to create one:

yaml
# kubescape-monitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kubescape-prometheus-exporter
  namespace: kubescape
spec:
  namespaceSelector:
    matchNames:
      - kubescape
  selector:
    matchLabels:
      app.kubernetes.io/component: prometheus-exporter
      app.kubernetes.io/instance: kubescape
      app.kubernetes.io/name: kubescape-operator
  endpoints:
    - targetPort: 8080
      path: /metrics
      interval: 30s
      scrapeTimeout: 10s

Istio

A service mesh like istio is always nice to have for encrypted communication between pods and services, here’s how I set it up:

bash
istioctl install --set components.ingressGateways[0].name=istio-ingressgateway --set components.ingressGateways[0].enabled=false

then for the grafana dashboards:

bash
istioctl dashboard grafana -n monitoring

create a policy:

yaml
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: PERMISSIVE # normally STRICT is preferred but PERMISSIVE won't make any breakages if any occur like excluded namespaces communicating to ones that have mtls.

finally make a cluster policy for applying it to selected namespaces:

yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-istio-injection-label-all
spec:
  mutateExistingOnPolicyUpdate: true
  rules:
    - name: label-all-namespaces
      match:
        any:
          - resources:
              kinds:
                - Namespace
      exclude:
        any:
          - resources:
              kinds:
                - Namespace
              names:
                - kube-system
                - kube-public
                - kube-node-lease
                - istio-system
                - longhorn-system
                - cilium-secrets
                - cilium-monitoring
                - cattle-system
                - cert-manager
                - kyverno
                - kubevirt
                - cdi
                - homepage
      mutate:
        targets:
          - apiVersion: v1
            kind: Namespace
            preconditions:
              all:
                - key: "{{ target.metadata.name }}"
                  operator: AnyNotIn
                  value:
                    - kube-system
                    - kube-public
                    - kube-node-lease
                    - istio-system
                    - longhorn-system
                    - cilium-secrets
                    - cilium-monitoring
                    - cattle-system
                    - cert-manager
                    - kyverno
                    - kubevirt
                    - cdi
                    - homepage
        patchStrategicMerge:
          metadata:
            labels:
              istio-injection: enabled
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: remove-istio-injection-label-from-excluded
spec:
  mutateExistingOnPolicyUpdate: true
  rules:
    - name: unlabel-excluded-namespaces
      match:
        any:
          - resources:
              kinds:
                - Namespace
              names:
                - kube-system
                - kube-public
                - kube-node-lease
                - istio-system
                - longhorn-system
                - cilium-secrets
                - cilium-monitoring
                - cattle-system
                - cert-manager
                - kyverno
                - kubevirt
                - cdi
                - homepage
      mutate:
        targets:
          - apiVersion: v1
            kind: Namespace
            preconditions:
              all:
                - key: "{{ target.metadata.name }}"
                  operator: AnyIn
                  value:
                    - kube-system
                    - kube-public
                    - kube-node-lease
                    - istio-system
                    - longhorn-system
                    - cilium-secrets
                    - cilium-monitoring
                    - cattle-system
                    - cert-manager
                    - kyverno
                    - kubevirt
                    - cdi
                    - homepage
        patchStrategicMerge:
          metadata:
            labels:
              istio-injection: null

There needs to be one that reverts the mutation because for some reason kyverno doesn’t fully exclude namespaces properly

Kubevirt

The last service I’ll be going over is setting up kubevirt with an Azure Linux vm. Azure Linux is essentially trashy Fedora with half the packages.

bash
export RELEASE=$(curl https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt)
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-cr.yaml
kubectl -n kubevirt wait kv kubevirt --for condition=Available
export TAG=$(curl -s -w %{redirect_url} https://github.com/kubevirt/containerized-data-importer/releases/latest)
export VERSION=$(echo ${TAG##*/})
kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml
kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml

This script will install kubevirt and now to make a VM!

yaml
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:

  finalizers:
    - kubevirt.io/virtualMachineControllerFinalize
  name: test
  namespace: default

spec:
  dataVolumeTemplates:
    - metadata:
        name: test-boot-volume
      spec:
        source:
          http:
            url: https://aka.ms/azurelinux-4.0-x86_64.iso
        storage:
          resources:
            requests:
              storage: 1Gi
    - metadata:
        name: test-target-disk
      spec:
        source:
          blank: {}
        storage:
          resources:
            requests:
              storage: 50Gi
          storageClassName: longhorn
  runStrategy: Halted
  template:
    metadata:
      annotations:
        kubevirt.io/pci-topology-version: v3
    spec:
      architecture: amd64
      domain:
        cpu:
          cores: 2
        devices:
          autoattachGraphicsDevice: true
          disks:
            - bootOrder: 300
              disk:
                bus: sata
              name: test-boot-volume
            - disk:
                bus: virtio
              name: cloudinitdisk
            - bootOrder: 1
              disk:
                bus: virtio
              name: disk-1
          interfaces:
            - masquerade: {}
              name: default
        firmware:
          bootloader:
            efi:
              persistent: true
              secureBoot: false
          serial: 774cc0a1-2a83-488a-b17a-a6c3e8cc251f
          uuid: 21e8b2bb-7b44-4b47-8e5e-5915aa02c562
        machine:
          type: q35
        resources:
          requests:
            memory: 2Gi
      networks:
        - name: default
          pod: {}
      volumes:
        - dataVolume:
            name: test-boot-volume
          name: test-boot-volume
        - cloudInitNoCloud:
            userData: |
              #cloud-config
          name: cloudinitdisk
        - dataVolume:
            name: test-target-disk
          name: disk-1

Kiali

Kiali is a ui for istio’s service mesh so you can have observability into pods that have the proxy attached for mtls, here’s how to set up:

bash
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.30/samples/addons/prometheus.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.30/samples/addons/grafana.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.30/samples/addons/kiali.yaml

Final Product!

Here’s a diagram of the final product:

  %%{init: {
  "flowchart": {
    "useMaxWidth": false,
    "nodeSpacing": 100,
    "rankSpacing": 120,
    "padding": 20,
    "htmlLabel": true,
    "defaultRenderer": "dagre"
  }
}}%%
flowchart LR

    cil["Cilium eBPF"]

    subgraph kube_system["kube-system namespace"]
        h_ui["Hubble UI"]
        cdns["Core DNS"]
        ed["etcd DNS"]
        cagent["Cilium Agent"]
        hl["headlamp"]

        h_ui -->|"reads from"| cagent
        cdns -->|"reads from"| ed
    end

    subgraph scape["Kubescape namespace"]
        kscape["Kubescape"]
        scapeStorage["Kubescape storage"]
        scapeEx["Kubescape exporter"]

        kscape -->|"logs data to"| scapeStorage
        scapeEx -->|"reads from"| scapeStorage
    end

    subgraph monitoring["monitoring namespace"]
        graf["grafana"]
        lki["loki"]
        promo["prometheus"]

        graf -->|"reads from"| promo
        graf -->|"reads from"| lki
        promo -->|"scrapes metrics"| cdns
        promo -->|"scrapes metrics"| cagent
        promo -->|"scrapes data"| scapeEx
    end

    subgraph istio_system["istio-system namespace"]
        istio["istio"]
        isproxy["istio proxy"]
        kiali["kiali"]

        istio -->|"manages injections"| isproxy
        kiali -->|"visualizations of"| istio
    end

    subgraph external_dns["external-dns namespace"]
        eDNS["external dns"]

        eDNS -->|"sends new entries to"| ed
    end

    subgraph falco_ns["falco namespace"]
        f["falco"]
        fside["falco sidekick"]

        f -->|"outputs to"| fside
        promo -->|"scrapes from"| fside
    end

    subgraph longhorn_ns["longhorn-system namespace"]
        lmanage["longhorn manager"]
        csiPlug["longhorn csi plugin"]
        lui["longhorn ui"]

        csiPlug -->|"longhorn api"| lmanage  
        lui -->|"longhorn api"| lmanage
    end

    subgraph kyverno_ns["kyverno namespace"]
        kyv["kyverno"]
    end

    subgraph default_ns["default namespace"]
        def_workloads["Default Workloads"]
    end

    %% Cross-namespace / Global connections
    cil -->|"observes kernel"| cagent & kscape & promo & kyv & def_workloads & eDNS & f & lmanage
    isproxy -->|"mTLS sidecar"| kscape & promo & kyv & def_workloads & eDNS & f & lmanage
    f -->|"observes syscalls"| cagent & kscape & promo & kyv & def_workloads & eDNS & lmanage
    kyv -->|"enforces policies"| cagent & kscape & promo & def_workloads & eDNS & f & lmanage

Now you have a homelab better than any Proxmox monolithic + NAS setup and can run everything in the based Kubernetes way with better security!

Tags: