Skip to content

0002. Dual cloudflared tunnels with per-zone ExternalDNS

  • Status: accepted (partial — LB path dropped)
  • Date: 2026-04-15
  • Deciders: lusoris
  • Blocked by: Cloudflare tunnel creation (Terraform) — office-side tunnel not yet provisioned

Decision update 2026-04-15 (after review)

No Cloudflare Load Balancer. The free-tier DNS round-robin path is the chosen fallover mechanism: each shared hostname gets two CNAMEs (home + office tunnel), Cloudflare returns both, clients happy-eyeballs across them. Cost: zero. Downside accepted: fallover is TTL-bound (~30–60 s) instead of health-probe driven.

This changes the implementation steps below: step 6 ("Create Cloudflare LB pool") is replaced with "emit dual CNAMEs per shared hostname from both ExternalDNS instances." The rest of the design stands.

Context

External traffic to *.cauda.dev currently flows through a single cloudflared tunnel pinned to the home zone (Gateway main, 10.1.30.12). That same tunnel is the --default-targets for a single external-dns instance, which writes CNAME records for every HTTPRoute in the cluster.

Consequences today:

  • Every external request for office-only apps (*.office.cauda.dev, Nextcloud, Paperless, etc.) traverses home → office site-to-site at the backend level (Service resolves office-pinned pod, reply crosses 150–200 ms RTT link twice).
  • If home site loses WAN, the entire public surface goes dark even though office has working internet.
  • No way to pin a given hostname to a specific site at the DNS layer.

The post-1.19.2 → 1.20 follow-up plan called out three split-DNS deltas:

  1. site: label on each Gateway (metadata — trivial).
  2. A second external-dns instance per site, each scoped to its zone's Gateway via label selector, pointing at that site's cloudflared tunnel target.
  3. trafficDistribution: PreferClose on Services with multi-zone backends.

(1) landed in cluster/gateway/gateway.yaml and cluster/gateway/gateway-dual.yaml. (2) and (3) are this ADR.

Decision (proposed)

Two separate cloudflared tunnels (one per site) feeding two separate ExternalDNS instances, each scoped by a Gateway selector. A Cloudflare Load Balancer fronts both tunnel CNAMEs for hostnames that should be reachable from either site.

Tunnel topology

Tunnel Runs on LB pool Fed by Gateways
cloudflared-home home zone nodes home pool main (10.1.30.12), main-home (10.1.30.10)
cloudflared-office office zone nodes office pool main-office (10.2.30.10)

Tunnel credentials sealed as SealedSecret in sealed-secrets/sealed/cloudflared-*.yaml. Created via Terraform (cloudflare_tunnel resource), credential JSON pulled down once and sealed.

ExternalDNS wiring

Two Helm releases under argocd-apps/networking/:

# external-dns-home
--source=gateway-httproute
--label-filter=site=home
--txt-owner-id=k8s-cluster-home
--default-targets=<home-tunnel-uuid>.cfargotunnel.com
--force-default-targets

# external-dns-office
--source=gateway-httproute
--label-filter=site=office
--txt-owner-id=k8s-cluster-office
--default-targets=<office-tunnel-uuid>.cfargotunnel.com
--force-default-targets

--label-filter=site=home|office is the pivot. Because each HTTPRoute attaches to exactly one Gateway, and each Gateway carries exactly one site: label, ExternalDNS inherits the label via parentRefs → Gateway → labels in v0.20+ (verify behaviour with single-app pilot before full rollout).

DNS record shape

Hostname pattern Strategy
*.media.cauda.dev, *.office.cauda.dev Pinned to whichever site owns the backend (HTTPRoute attaches to one Gateway). No LB needed — one CNAME.
*.infra.cauda.dev, *.security.cauda.dev, *.dev.cauda.dev Published by both ExternalDNS instances (HTTPRoute duplicated on both Gateways). Each instance emits its own CNAME → CF returns both records on A/AAAA query → client happy-eyeballs picks. TTL 60 s keeps fallover tight.
k8s.infra.cauda.dev (API) Dual CNAMEs to both tunnel hostnames; tunnels each proxy to their site's 10.{1,2}.20.10:6443.

trafficDistribution: PreferClose

Applied to cross-site Services with multi-zone pod topology. Discovery:

kubectl get svc -A -o json \
  | jq -r '.items[] | select(.spec.selector != null) | "\(.metadata.namespace)/\(.metadata.name)"' \
  | while read s; do
      ns=${s%%/*}; name=${s##*/}
      zones=$(kubectl -n "$ns" get endpointslice -l kubernetes.io/service-name="$name" \
                -o json | jq -r '[.items[].endpoints[].zone // "-"] | unique | length')
      [[ "$zones" -gt 1 ]] && echo "$s → $zones zones"
    done

Apply spec.trafficDistribution: PreferClose only where the discovery returns >1 zone. Single-replica stateful services (Longhorn-backed) don't benefit.

Consequences

Positive:

  • Office apps reach users without crossing site-to-site at the proxy layer.
  • Home WAN outage stops affecting office-hosted apps (and vice-versa) once CF LB is in play.
  • Explicit site pinning at DNS means trafficDistribution is belt-and-suspenders, not the primary mechanism.

Negative:

  • Two tunnels + two ExternalDNS instances = ~2× the moving parts to monitor.
  • CF Load Balancer costs ~$5/mo per LB host + $0.50/500K queries. A single LB covering ~10 pooled hostnames is trivial; must confirm plan tier carries this.
  • Dual TXT-registry means running both ExternalDNS instances forever — if one gets deleted without cleanup, stale TXT ownership records block the other.

Open questions (block implementation)

  1. CF account plan tier — does current plan include Load Balancers? If not, the fallover story downgrades to DNS round-robin (slower fallover, no health probes).
  2. Tunnel-side high availability — run cloudflared as a 2-replica Deployment per site? Or rely on tunnel-native reconnect? Current single-replica setup has had zero outages in 6 months; 2 replicas is paranoia.
  3. Authentik + OIDC — once k8s.infra.cauda.dev is LB-fronted, the OIDC login redirect loop needs to resolve back to the same tunnel that served the initial request (sticky session at LB layer or explicit per-site hostname).

Alternatives considered

  • Stay single-tunnel, rely on trafficDistribution: PreferClose only: rejected — PreferClose can't cross-site-fail-over traffic already inside the cluster, and the single-tunnel WAN failure mode is unaddressed.
  • Two tunnels but one ExternalDNS with --target-net-filter: rejected — that flag filters endpoint IPs, not target CNAMEs. The clean split is per-instance --default-targets.
  • Skip tunnels entirely, expose Gateways via public IP: rejected — home WAN is on Starlink (no static IPv4); office WAN has CGNAT. Cloudflare tunnel is the authentic solution.

Implementation order (once unblocked)

  1. Provision cloudflared-office tunnel via Terraform; record tunnel ID.
  2. Seal credentials; add argocd-apps/infra/cloudflared-office.yaml.
  3. Deploy office cloudflared; smoke-test with a single HTTPRoute pinned to main-office pointing at the new tunnel CNAME by hand.
  4. Split ExternalDNS into external-dns-home + external-dns-office with --label-filter=site=.... Bump txt-owner-id to preserve ownership segregation.
  5. Duplicate shared HTTPRoutes (*.infra.cauda.dev, etc.) onto both Gateways.
  6. Create Cloudflare LB pool with both tunnel CNAMEs; flip shared hostnames from direct CNAME → LB alias.
  7. Pass: trafficDistribution: PreferClose on multi-zone Services (run discovery script above).