Change alerts & safe deletion

The tracker tells you when something changes and stops you breaking things by accident. Two things to understand here: the notifications it raises (which come from two separate systems), and the reference checks that guard deletions.

It’s worth being precise, because they carry different events through different channels, and the split is deliberate: a subscription carries config changed and nothing else, while drift, failed collections and offline collectors travel through the alert center exclusively. Nothing you do to a subscription will make it mail you about drift.

When a tracked config changes (a new version is cut), Taranac emails the people who subscribed to that config. Subscriptions are by recipient group: you attach one or more user groups to a config (on its detail page, under Change notifications), and every member gets an email — in their own language. A group with no active member holding an email address is refused when you attach it, rather than accepted as a subscription that would mail nobody; if the last such member later leaves, the subscription is flagged No email instead. The same group can be subscribed to any number of configs, but only once per config.

The run’s trigger decides how it’s delivered — nothing on the subscription does:

• A scheduled sweep that changes a config is coalesced into a digest, so a big sweep sends one email per group, not a storm. The window is ncm.notify_digest_window_minutes (default 15 minutes).
• An on-demand Collect now that changes a config is delivered immediately — a single-config email, in practice within the drainer’s 30-second tick.
• A preview never notifies: it cuts no version.

Delivery is email/SMTP, so it needs SMTP configured. Both kinds are queued in a durable outbox rather than sent from the collection itself, so a slow mail server can never hold up a device collection: a row is stamped sent only once the mail actually went out, a failed send is retried on the next tick (up to 10 attempts, after which the row is left in place for diagnosis), and with SMTP switched off nothing is drained and nothing is lost. The queue survives restarts and HA failover.

The email itself carries the config name, its new version number and a link to the Configs list (the link only when email.base_url is set) — never the config body or the diff.

Four tracker conditions are raised as problems in Taranac’s alert center, each de-duplicated so a persistent condition is one open problem rather than a repeating one, and each cluster-wide — one problem per cluster, raised by whichever node holds the leader role, never one per node:

Problem	Fires when	Scope
Drift detected	A new capture’s hash diverges from the config’s baseline.	Per tracked config. Turned off globally with ncm.drift_system_alert_enabled (default on); turning it off also resolves whatever was already firing.
Collection failed	The last ncm.retry.escalate_after (default 3) scheduled collections of a config all failed.	Per tracked config. On-demand failures neither escalate nor count towards the run — but an on-demand success does clear a standing one. The newest failure class (auth / transient / partial / too_large) is in the title.
Collector offline	A standalone collector’s last heartbeat is older than 15 minutes, scanned every 5 (ncm.collector_offline_scan_interval_seconds).	One problem per collector, carrying the number of enabled configs it serves rather than one problem per config. Never raised for the embedded collector, which has no dial-in heartbeat to go stale.
Collection falling behind	A scheduled sweep is still running when the next is due, so the tick is deferred.	System-wide, embedded collector only — the cue to move those configs to a standalone collector or lengthen the interval.

These resolve themselves when the condition clears — the baseline matches again (moving or clearing the baseline label re-checks it on the spot, without waiting for a collection), the collector reports back, or a collection succeeds. Severities, and how the alert center then delivers them, are configured there, independently of the per-config change emails above.

A collector that went quiet after an upgrade. Since 1.2.6 a standalone collector whose snapshot contract doesn’t match its core is refused before its poll counts as a heartbeat, so it stops refreshing its last-seen stamp and turns up here as Collector offline within 15 minutes. The alert names a silent collector, not the version skew underneath it — so after any core upgrade that’s the first thing to check. The cure is to move the collector to the version its core runs (./collector-join.sh --update), covered in Deploy a standalone collector.

Every reusable object shows a Used by count, and the tracker won’t let you delete something still in use in a way that would break a dependent object. Try to delete one and Taranac runs a reference check: if anything blocks, the delete is refused and you’re shown exactly what’s referencing it.

Deleting a…	Is blocked while…	Otherwise
Source	Any tracked config uses it	—
Collector	Any source or tracked-config address uses it	The default (embedded) collector can never be deleted
Network object	Any source targets it	—
Scrub set	Any recipe references it	—
Local account (used by a JIT credential)	A JIT credential or a live lease binds it	—
Recipe	(not blocked)	Configs using it revert to the default recipe; the change is recorded
Credential	(not blocked)	Sources using it are simply left credential-less; any live JIT lease is rotated first
User group subscribed to a config	(not blocked by the subscription)	The subscriptions clear themselves and appear in the drawer as doing so. The change emails queued for that group hold it more firmly, and the queue rows are kept after delivery — so once a config the group watched has actually changed, the delete is refused with a “still used by” that names nothing, until that config and its versions are gone

The “Used by” drawer lists the blocking references (and any that would auto-resolve) so you can detach or reassign them first, then delete.

Tell the network team when a firewall config changes. Add their user group as a recipient on that config’s Change notifications. They’ll get an immediate email on a Collect now change and a digest for scheduled changes.

Get paged when a config drifts. Leave drift alerts on (they’re on by default) and wire the alert center to your channel of choice. Drift raises one problem per config until you re-baseline or the config matches again.

Retire a recipe safely. Check its Used by count first; deleting it reverts dependent configs to the default recipe (recorded in the audit log) rather than breaking them.

Concern	Behaviour
Config-changed email	Per-config recipient groups; config_changed only. Digest for a scheduled sweep (ncm.notify_digest_window_minutes, default 15), immediate for Collect now. Durable outbox, 10 delivery attempts, per-member language. A group with no e-mail member is refused.
Drift alert	Alert-center problem per config; ncm.drift_system_alert_enabled, default on. Re-checked when the baseline label moves.
Collector-offline alert	Alert-center problem per collector, carrying its affected-config count; 15-minute staleness, scanned every 5 minutes (ncm.collector_offline_scan_interval_seconds). Standalone collectors only.
Collection-failed alert	Alert-center problem per config; ncm.retry.escalate_after (default 3) consecutive scheduled failures. Carries the newest failure class: auth, transient, partial or too_large.
Sweep-behind alert	One system-wide alert when the embedded collector can’t finish a sweep inside the interval.
Config status vs alert	The Failing badge follows the newest scheduled result; the alert follows a run of them. The badge moves first.
Reference block	source, collector, network object, scrub set, and a JIT local account block deletion while referenced; recipe and credential don’t (they detach cleanly). A subscribed user group detaches too, but its queued change emails still hold it.

• Alerts — the alert center that carries drift / offline / failed problems.
• Snapshots, diff & drift — what “changed” and “drifted” mean.
• Collectors — the offline condition behind the collector alert.
• Deploy a standalone collector — keeping a collector in version lockstep, the usual cause of a post-upgrade offline alert.
• Recipes & scrub sets — what a subscriber can end up reading when automatic masking is off.
• Users & groups — the groups you subscribe to change notifications.
• Audit & syslog — where deletes and reverts are recorded.