Skip to content

0003. IPv6 phased rollout — ULA in-cluster first, global via BGP later

  • Status: proposed
  • Date: 2026-04-15
  • Deciders: lusoris
  • Blocked by: Cilium 1.19 → 1.20 upgrade (ADR-0002 IPv6 prerequisites); UDM Pro BGP peering config

Context

The cluster runs IPv4-only today (10.244.0.0/16 pods, 10.96.0.0/12 services, 10.{1,2}.20.0/24 nodes). Public DNS still resolves A-only; AAAA records would be reachable only if workloads listened on a v6 address, which they don't.

Drivers to start the IPv6 lift:

  • Several upstream sources (Docker Hub/GHCR edge caches, Cloudflare, some k8s.io mirrors) increasingly resolve to v6 first; pods fall back to v4 after a negative cache timeout, adding latency.
  • Happy-eyeballs clients hitting *.cauda.dev currently only see A records — a future dual-stack Cloudflare tunnel can't advertise AAAA without our backend supporting it.
  • Cilium 1.20 improved v6 support + shipped a production-grade BGP Control Plane usable with UDM Pro.
  • Office LAN gateway already has a /56 delegation from ISP; home WAN (Starlink) has dynamic /56.

Cilium L2 announce does not support IPv6 (tracked upstream). External v6 LB IPs therefore must route via BGP — not via ARP/NDP on the LAN — which means the cluster must peer with the site gateway router.

Decision (proposed, phased)

Three phases, each independently deployable. Stop after any phase if the next is too costly.

Phase A — ULA in-cluster only (no external reachability)

  • Enable dual-stack on the cluster CIDRs. Pod v6 CIDR from ULA fd00:cafe::/48 (cluster-scoped, globally unreachable by design).
  • No BGP yet. Services reach each other over v6 inside the cluster; nothing on the LAN or internet can address pods on v6.
  • Unblocks: CoreDNS v6 upstream (2606:4700:4700::1111 etc.), pod→pod v6 (pairs with network-policy experiments), v6-happy-eyeballs pod egress still falls back to v4 masquerade.

Changes (Cilium Helm values):

ipv4:
  enabled: true
ipv6:
  enabled: true
ipam:
  operator:
    clusterPoolIPv4PodCIDRList: ["10.244.0.0/16"]
    clusterPoolIPv6PodCIDRList: ["fd00:cafe:0:1::/64"]
tunnel: "geneve"        # unchanged
routingMode: "tunnel"   # unchanged

Node IPv6: assign ULA on each node's VLAN 20 interface (fd00:cafe::/64 on VLAN 20; ::home1, ::home2, ::office1, …). Ansible handles netplan.

Phase B — BGP peering between each CP and the site's UDM Pro

  • Deploy CiliumBGPClusterConfig peering each CP with its site gateway (home UDM Pro, office UDM Pro). AS numbers: 65001 (cluster), 65002 (home UDM), 65003 (office UDM).
  • Gateway advertises the cluster's LB pools (IPv4 today; adds IPv6 in phase C). Cluster advertises nothing outbound.
  • Verifies BGP plumbing works before we depend on it for v6.

Changes (new manifest cluster/cilium/bgp.yaml):

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: cluster-bgp
spec:
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/control-plane: ""
  bgpInstances:
    - name: instance-home
      localASN: 65001
      peers:
        - name: udm-home
          peerASN: 65002
          peerAddress: 10.1.30.1
          peerConfigRef:
            name: udm-generic
    - name: instance-office
      localASN: 65001
      peers:
        - name: udm-office
          peerASN: 65003
          peerAddress: 10.2.30.1
          peerConfigRef:
            name: udm-generic

UDM Pro config: per-VLAN neighbor with password auth, soft-reconfiguration inbound. Captured in terraform (Unifi provider) once the topology is confirmed.

Phase C — Global IPv6 LB IPs via BGP, AAAA records published

  • Allocate a /64 from each site's ISP /56 as the LB pool (home <starlink-prefix>:dead::/64, office <isp-prefix>:dead::/64).
  • Add CiliumLoadBalancerIPPool (v6) per site. Assign IPs to Gateways via infrastructure labels matching the lbipam.cilium.io/pool already in place.
  • External-dns publishes AAAA alongside A for Gateways (cloudflared tunnels don't forward UDP, so public hostnames still use the tunnel CNAME; AAAA applies only to LAN-facing Gateways main-home / main-office).
  • Cloudflare public records stay A-only on cloudflared paths until tunnel v6 ingress lands upstream.

Consequences

Positive:

  • Pod egress can use v6 first where upstream is v6-native; fewer happy-eyeballs round-trips.
  • Removes a standing blocker on several follow-ups (v6 for Authentik outposts, home Assistant, etc.).
  • Forces explicit BGP design, which is also needed for IPv4 LB route advertisement long-term (currently only works because each LB IP is in the same VLAN as the L2 announce lease holder).

Negative:

  • Starlink prefix rotates; home LB prefix needs a renumbering runbook (or stays ULA + NAT66 for now).
  • UDM Pro BGP is less battle-tested than enterprise routers; the CP-side timer tuning may need iteration.
  • Dual-stack doubles the surface for network-policy bugs — Cilium policies default-allow v6 unless explicit.

Open questions

  1. Home WAN prefix stability — Starlink /56 is dynamic. Do we accept v6 external LB only for office (static ISP) and stay v4-only for home public? Or renumber on prefix change (scripted)?
  2. UDM Pro BGP tuning — Unifi BGP implementation is Quagga/FRR-based; default hold time 180 s may mask Cilium agent restarts. Tune to 30/10?
  3. Cloudflare AAAA — Tunnel v4-only means any *.cauda.dev hostname fronted by cloudflared stays A-only; only LAN-exposed hostnames get AAAA. Split-DNS already models this.
  4. Service CIDR — dual-stack Service CIDR requires restart of kube-apiserver with --service-cluster-ip-range=10.96.0.0/12,fd00:cafe:1::/112; add to kubeadm-config patch runbook.

Alternatives considered

  • Enable IPv6 globally in one shot: rejected — too many moving parts (ULA choice + BGP peering + Cilium upgrade + cloudflared) to debug concurrently.
  • v6 pods only, no external routing ever: rejected — loses the external-reachability benefit; half of the motivation was dual-stack Gateway listeners.
  • NAT66 instead of BGP: rejected — conflates prefix scope; NAT66 on the UDM is Starlink-workable but doesn't solve the office-LAN peer story.

Implementation order (once unblocked)

  1. Upgrade Cilium to 1.20 (ADR-0002 pre-req; runbook: docs/upgrades/cilium-1.20.md).
  2. Phase A: enable dual-stack + ULA (Cilium values + kubeadm patch + node v6 via Ansible).
  3. Phase B: deploy CiliumBGPClusterConfig + configure UDM BGP. Validate v4 advertisement first.
  4. Phase C: allocate ISP /64 pool(s), add v6 CiliumLoadBalancerIPPool, extend ExternalDNS to emit AAAA.