chrony PTB/NTS alerts

Source rule: cluster/monitoring/chrony-ptb-alerts.yaml (PrometheusRule chrony-ptb-alerts, namespace monitoring, group chrony-ptb-conformance).

Scope and remediation path (read first)

Node clock policy is not an ArgoCD workload. It is node-level configuration owned by the Ansible role ansible/roles/chrony_ptb and baked into the Packer node image. Every alert in this group is fixed by changing that role in Git and re-running ansible/playbooks/tune-cluster-ntp-ptb.yml — not by syncing an Application and not by editing files on a node. See docs/runbooks/chrony-ptb-nts.md for the rollout procedure.

Desired state: exactly four hostname-based PTB sources (ptbtime1.ptb.de … ptbtime4.ptb.de), NTS-only (authselectmode require), minsources 2, zero Ubuntu/DHCP/fallback sources.

Relevant on-node paths:

Thing	Path
Sources	/etc/chrony/sources.d/ptb.sources
Policy	/etc/chrony/conf.d/ptb-policy.conf
Main config	/etc/chrony/chrony.conf
Exporter script	/usr/local/sbin/chrony-ptb-textfile-exporter
Textfile output	/var/lib/node_exporter/textfile/chrony-ptb.prom
Units	chrony.service, chrony-ptb-textfile-exporter.{service,timer}

All commands below are read-only. This repo is strictly GitOps for cluster state and Ansible-managed for node state: do not mutate either from a shell.

---

ChronyPTBExporterMissing

• Severity: warning · for: 10m
• Fires when: an instance present in node_uname_info{job="node-exporter"} has no chrony_ptb_exporter_success{job="node-exporter"} series at all (unless on(instance)), for 10 minutes. This is series absence, not a value comparison.

Likely causes

1. chrony-ptb-textfile-exporter.timer is not enabled or its service failed, so no textfile is produced.
2. The node was built from an image predating the chrony_ptb role, or the playbook has not run against it.
3. node-exporter is not reading /var/lib/node_exporter/textfile, or the file is unreadable.

Diagnose (read-only)

ssh <node> systemctl status chrony-ptb-textfile-exporter.timer chrony-ptb-textfile-exporter.service
ssh <node> ls -la /var/lib/node_exporter/textfile/
ssh <node> journalctl -u chrony-ptb-textfile-exporter.service --no-pager -n 50

node_uname_info{job="node-exporter"} unless on(instance) chrony_ptb_exporter_success{job="node-exporter"}

Resolution — Git change. Add the node to the inventory and/or re-run tune-cluster-ntp-ptb.yml; if the unit template is wrong, fix ansible/roles/chrony_ptb/templates/ and re-run the playbook. Do not hand-place the unit on the node.

---

ChronyPTBExporterFailed

• Severity: warning · for: 5m
• Fires when: chrony_ptb_exporter_success{job="node-exporter"} != 1 for 5 minutes. Note this is != 1, so a missing/garbled value also fires, not only a literal 0.

Likely causes

1. chronyc -n -c sources failed — chrony.service is down, wedged, or its socket is unreachable.
2. The exporter's systemd sandbox (ProtectSystem=strict, PrivateDevices) is blocking something it needs.
3. Malformed chronyc output the parser could not read.

Diagnose (read-only)

ssh <node> systemctl status chrony.service
ssh <node> journalctl -u chrony-ptb-textfile-exporter.service --no-pager -n 50
ssh <node> chronyc -n -c sources
ssh <node> cat /var/lib/node_exporter/textfile/chrony-ptb.prom

Resolution — If chrony.service itself is unhealthy, that is the real incident; investigate its logs first. Any durable fix (unit hardening, exporter script logic) is a Git change to ansible/roles/chrony_ptb followed by a playbook run.

---

ChronyPTBExporterStale

• Severity: warning · for: 5m
• Fires when: abs(time() - chrony_ptb_exporter_timestamp_seconds{job="node-exporter"}) > 300 for 5 minutes. The abs() matters: this fires when the node timestamp is more than 300s behind or ahead of Prometheus — a node clock running fast trips it just as readily as a stalled timer.

Likely causes

1. The exporter timer is not firing (interval default is 60s) or the oneshot service hangs.
2. Severe node clock skew — which is itself the thing this group exists to catch. Cross-check ChronyPTBNotSynchronised.
3. Clock skew on the Prometheus side, if many nodes go stale simultaneously.

Diagnose (read-only)

ssh <node> systemctl list-timers chrony-ptb-textfile-exporter.timer --all
ssh <node> stat /var/lib/node_exporter/textfile/chrony-ptb.prom
ssh <node> chronyc tracking
kubectl top nodes

abs(time() - chrony_ptb_exporter_timestamp_seconds{job="node-exporter"})

Resolution — If only the timer stalled, resolving the underlying node issue restores it. Interval changes (chrony_ptb_exporter_interval) are a Git change in ansible/roles/chrony_ptb/defaults/main.yml. If the node clock is genuinely skewed, treat ChronyPTBNotSynchronised as the primary alert.

---

ChronyPTBConfigurationDrift

• Severity: critical · for: 5m
• Fires when: any of the following holds for 5 minutes:
• chrony_ptb_policy_configured{job="node-exporter"} != 1, or
• chrony_ptb_configured_sources{job="node-exporter"} != 4, or
• chrony_ptb_non_ptb_sources{job="node-exporter"} != 0

Note configured_sources must be exactly 4 — five configured sources fires this just as three does.

Likely causes

1. Node config edited outside Ansible, or a package upgrade re-added a distro default.
2. A DHCP/cloud-init hook injected an NTP source, raising non_ptb_sources.
3. The node booted an image predating the current chrony_ptb role.

Diagnose (read-only)

ssh <node> cat /etc/chrony/sources.d/ptb.sources
ssh <node> cat /etc/chrony/conf.d/ptb-policy.conf
ssh <node> grep -nE '^(server|pool|peer|sourcedir|include)' /etc/chrony/chrony.conf
ssh <node> chronyc -n sources

chrony_ptb_policy_configured != 1 or chrony_ptb_configured_sources != 4 or chrony_ptb_non_ptb_sources != 0

Resolution — Git change. Re-run tune-cluster-ntp-ptb.yml (use --check --diff first to see the drift) to restore the managed files. If the drift came from something outside the role — a distro package, cloud-init — fix the source of the injection in the role or the Packer template, otherwise it returns on the next boot.

---

ChronyPTBDesiredSourceMissing

• Severity: critical · for: 5m
• Fires when: chrony_ptb_source_configured{job="node-exporter"} != 1 for 5 minutes. This is per-hostname — the series carries a source label (ptbtime1.ptb.de … ptbtime4.ptb.de); check {{ $labels.source }} to see which one is absent. The metric asserts the hostname appears exactly once with NTS.

Likely causes

1. A hostname was removed, misspelled, or duplicated in chrony_ptb_sources.
2. The source line lost its nts option.
3. The playbook has not been re-run since the source list changed.

Diagnose (read-only)

ssh <node> cat /etc/chrony/sources.d/ptb.sources
ssh <node> chronyc -n sources

chrony_ptb_source_configured{job="node-exporter"} != 1

Desired list lives in ansible/roles/chrony_ptb/defaults/main.yml (chrony_ptb_sources).

Resolution — Git change. Correct chrony_ptb_sources or the sources template in the role, then re-run the playbook. There is no ConfigMap and no Application for this.

---

ChronyPTBActiveSourceCountUnexpected

• Severity: warning · for: 30m
• Fires when: chrony_ptb_observed_sources{job="node-exporter"} != 4 for 30 minutes. Exactly four — this fires on five active sources as well as on three. The long window tolerates transient PTB maintenance.

Likely causes

1. DNS failure for one PTB hostname, so chrony never instantiates that source.
2. NTS-KE (TCP/4460) blocked or failing for one endpoint.
3. An untracked runtime source appeared (count above four).

Diagnose (read-only)

ssh <node> chronyc -n sources
ssh <node> chronyc ntpdata
ssh <node> getent hosts ptbtime1.ptb.de ptbtime2.ptb.de ptbtime3.ptb.de ptbtime4.ptb.de
ssh <node> journalctl -u chrony.service --no-pager -n 100

Resolution — Transient PTB-side or network issues self-recover; the 30m window already absorbs short blips. A count above four means drift — see ChronyPTBConfigurationDrift and fix via the role in Git. Persistent DNS or firewall problems are infrastructure fixes, not node edits.

---

ChronyPTBInsufficientAgreement

• Severity: critical · for: 5m
• Fires when: chrony_ptb_selectable_sources{job="node-exporter"} < 2 for 5 minutes.

This is the policy boundary: minsources 2 means chronyd deliberately stops updating the clock below two agreeing sources. The node clock is now free- running and will drift.

Likely causes

1. Fewer than two sources reachable (NTP/UDP 123) or authenticated (NTS-KE/TCP 4460).
2. Sources reachable but rejected by selection as falsetickers.
3. Local offset already so large that sources are discarded.

Diagnose (read-only)

ssh <node> chronyc tracking
ssh <node> chronyc -n sources -v
ssh <node> chronyc selectdata
ssh <node> chronyc ntpdata

chrony_ptb_selectable_sources{job="node-exporter"} < 2

Resolution — Restore reachability/authentication for at least two PTB sources; the alert clears on its own once selection recovers. Do not lower minsources to silence this — that setting is the conformance guarantee, and changing it is a Git change to ansible/roles/chrony_ptb/files/ptb-policy.conf that needs an explicit decision, not an incident-time workaround.

---

ChronyPTBNTSAuthenticationInsufficient

• Severity: critical · for: 5m
• Fires when: chrony_ptb_authenticated_sources{job="node-exporter"} < 2 for 5 minutes.

Likely causes

1. NTS-KE (TCP/4460) blocked by a firewall or upstream path.
2. CA trust store on the node cannot validate the PTB certificate chain.
3. DNS returns addresses whose certificate does not match the hostname (this is why sources are pinned by hostname, never by IP).
4. Node clock skew large enough to fail TLS validity checks — a feedback loop with ChronyPTBNotSynchronised.

Diagnose (read-only)

ssh <node> chronyc ntpdata
ssh <node> chronyc authdata
ssh <node> getent hosts ptbtime1.ptb.de
ssh <node> openssl s_client -connect ptbtime1.ptb.de:4460 -servername ptbtime1.ptb.de </dev/null

Resolution — Under authselectmode require, unauthenticated sources are never eligible, so this alert overlaps ChronyPTBInsufficientAgreement. Fix the path (firewall, DNS, CA bundle). CA-bundle or source-list changes are Git changes to the role plus a playbook run.

---

ChronyPTBSourceUnreachable

• Severity: warning · for: 30m
• Fires when: chrony_ptb_active_source_reachable{job="node-exporter"} == 0 for 30 minutes. Per-address, labelled address — the metric tracks chrony's reach register for one of the four active NTS-assigned addresses.

Redundancy is intact as long as two other sources still agree; this is a warning, not a clock-loss condition.

Diagnose (read-only)

ssh <node> chronyc -n sources -v
ssh <node> chronyc ntpdata
ssh <node> ping -c 3 <address>
kubectl get node <node-name> -o wide

chrony_ptb_active_source_reachable{job="node-exporter"} == 0

Resolution — A single PTB endpoint under maintenance recovers on its own; confirm the other three are healthy and let it clear. If all four are unreachable, escalate to egress/firewall — do not change node network config by hand.

---

ChronyPTBSourceAuthenticationFailed

• Severity: warning · for: 10m
• Fires when: for the same (instance, address) pair, both: chrony_ptb_active_source_reachable{job="node-exporter"} == 1 and chrony_ptb_active_source_authenticated{job="node-exporter"} == 0, for 10 minutes. Packets arrive but are not NTS-authenticated.

Likely causes

1. NTS cookies exhausted or the NTS-KE re-handshake is failing while NTP/123 still answers.
2. Certificate chain no longer validates for that address.
3. Node clock skew breaking TLS validity windows.

Diagnose (read-only)

ssh <node> chronyc ntpdata
ssh <node> chronyc authdata
ssh <node> journalctl -u chrony.service --no-pager | grep -i -E 'nts|cert|auth'

Resolution — Under authselectmode require this address is not eligible for selection, so it silently reduces redundancy. Fix the NTS-KE path or CA trust. Trust-store and source changes are Git changes to the role. If the node clock is badly off, resolve ChronyPTBNotSynchronised first — it can be the cause here, not the effect.

---

ChronyPTBNotSynchronised

• Severity: critical · for: 5m
• Fires when: chrony_ptb_synchronised{job="node-exporter"} != 1 for 5 minutes — no PTB source is selected.

The node clock is free-running. Expect downstream damage: TLS validation failures, etcd lease churn, Kubernetes token/certificate rejections.

Diagnose (read-only)

ssh <node> chronyc tracking
ssh <node> chronyc -n sources -v
ssh <node> chronyc selectdata
ssh <node> chronyc ntpdata
kubectl get nodes
kubectl -n kube-system get pods --field-selector spec.nodeName=<node-name>

Resolution — Almost always a downstream symptom: work ChronyPTBInsufficientAgreement / ChronyPTBNTSAuthenticationInsufficient first, since chronyd will resynchronise by itself once two authenticated sources agree. Do not force a step with chronyc makestep or a manual date set — that bypasses the conformance policy this whole group enforces and can step control-plane clocks under running etcd. If the node cannot recover, escalate rather than stepping the clock. Config fixes go through the role in Git.

---

ChronyPTBClockOffsetHigh

• Severity: warning · for: 5m
• Fires when: abs(chrony_ptb_last_offset_seconds{job="node-exporter"}) > 0.05 for 5 minutes — last measured offset above 50 ms in either direction.

Likely causes

1. Network latency or asymmetric routing to PTB, inflating measured offset.
2. Source disagreement causing the selected root to oscillate.
3. A recent source switch, after which chrony is still converging.
4. Host-level clock instability (virtualised TSC, CPU frequency effects).

Diagnose (read-only)

ssh <node> chronyc tracking
ssh <node> chronyc -n sources -v
ssh <node> chronyc sourcestats

abs(chrony_ptb_last_offset_seconds{job="node-exporter"}) > 0.05
chrony_ptb_rms_offset_seconds{job="node-exporter"}

chrony_ptb_rms_offset_seconds distinguishes a transient spike from sustained error — it is exported but has no alert of its own.

Resolution — Short excursions after a source change are normal and clear themselves. Sustained offset points at network path quality or host clock instability (check the hypervisor for virtualised nodes). Chrony tuning changes are Git changes to ansible/roles/chrony_ptb plus a playbook run.

---

Coverage

All 12 alerts in the source group are documented above: ChronyPTBExporterMissing, ChronyPTBExporterFailed, ChronyPTBExporterStale, ChronyPTBConfigurationDrift, ChronyPTBDesiredSourceMissing, ChronyPTBActiveSourceCountUnexpected, ChronyPTBInsufficientAgreement, ChronyPTBNTSAuthenticationInsufficient, ChronyPTBSourceUnreachable, ChronyPTBSourceAuthenticationFailed, ChronyPTBNotSynchronised, ChronyPTBClockOffsetHigh.

Exported but unalerted: chrony_ptb_reachable_sources, chrony_ptb_active_source_selectable, chrony_ptb_tracking_stratum, chrony_ptb_rms_offset_seconds — useful as supporting signal during triage.