Troubleshooting & diagnostics
When something is wrong on a running deployment, resist the urge to restart first and ask questions later. Taranac is a stack of cooperating services, and a blind restart both loses the evidence and often just resets the clock on the real problem. This page is a diagnostic-first runbook: gather a small, specific set of read-only facts, read them, then act. Every command in the bundle below is safe to run on production — none of them change state.
It is also a symptom index. Most of what goes wrong in the field has a name and a page that owns it, so the index routes rather than restates: find the symptom, read the one-line cause, and follow the link.
First-response bundle
Section titled “First-response bundle”Run these read-only commands and keep the output — this is the baseline every other section builds on.
./taranac version # installed version (+ whether a newer one exists)./taranac ps # per-service status: is anything down or restarting?top -b -n1 | head -20 # host CPU + memory snapshot, busiest processes firstdocker stats --no-stream # per-container CPU / memory / IOdf -h # disk usage per mount (a full disk breaks everything)./taranac ps is the single most useful line: it tells you whether this is a service
problem (something is Exited or Restarting) or a host problem (every service is up
but the box is starved). Read it before anything else. The names every command below takes
are the compose service names — tacacs, radius, nac, api, postgres, edge,
frontend, captive-portal and taranac-mfa among them.
On a witness or a standalone collector host, ./taranac ps and ./taranac logs
drive that host’s own stack. ./taranac logs <service> for a service that has no
container on the host says so — no <service> container on this host. — and lists the
ones that exist, so an empty answer means the container itself logged nothing. The commands that need the product’s database or API
(cluster …, unlock, reset-password, create-admin, installation-id, reset) say
so on such a host and do nothing.
Symptom index
Section titled “Symptom index”Device administration — TACACS+ and RADIUS
Section titled “Device administration — TACACS+ and RADIUS”| Symptom | What it is, and what to do |
|---|---|
| Login succeeds and reaches the right privilege level, but every command is denied — the authorization log shows a refusal with an empty profile | Some devices omit the RFC 8907 = / * separator and send a bare <cr> as an argument of its own (Edgecore ECS4100 is the reported platform). Vendor Compatibility on AAA → Settings → TACACS+ tolerates it and is on by default; check it has not been switched off. |
| The device refuses the whole authorization over an attribute it does not implement | The attribute is marked Mandatory (=) in the template. A device must refuse an authorization carrying a mandatory attribute it does not know; mark it Optional (*) instead. See TACACS+. |
A MikroTik administrator’s successful login is preceded by a reject with reason empty_password_probe | Expected. RouterOS proves the empty password before prompting the human. The row stays in the log and counts towards no failure metric. See MikroTik RouterOS. |
| A TACACS+ rule restricted to certain devices never matches | A device restriction that resolves to nothing — an emptied device group, or a device on an FQDN network object — fails closed. Give the rule a group with members, and the device a host, subnet or range object. |
| A device is missing from the RADIUS clients and is not authenticated at all | Its network object is an FQDN, which carries no address. Use a host, subnet or range object. |
| The Password prompt you configured never appears on the device | It applies only while MFA is on, and only an interactive (ASCII) login can display it — a device that sends PAP carries the password in its first packet and asks nothing. A login whose matching rule requires MFA is shown the MFA password prompt instead (Password+MFA: by default), again only while TACACS+ MFA Mode is on and only on an ASCII login. Leave that prompt empty to show the default one to everybody. See Password prompts and MFA. |
| A RADIUS device-admin login times out and nothing appears in the AAA log | The device sends from an address that no RADIUS-enabled device record covers. RADIUS AAA identifies the device by the packet’s source address; NAS-IP-Address does not count. Pin the device’s RADIUS source interface, or register the address it sends from. See How a request finds its device. |
| A RADIUS device-admin login matches the wrong device, or the wrong WHERE rule | A subnet device record, or a NAT address several devices share, covers the source address — devices behind one NAT address are one device. Register the source as a more specific record, or remove the NAT. See How a request finds its device. |
| A TACACS+ over TLS device is rejected at the handshake, or no TLS client connects at all | The device trusts neither the server certificate nor the address it names, or its client certificate comes from a CA the listener does not trust. TLS Client Trust on the TACACS+ settings trusts every enabled internal CA by default; an imported CA counts only under Selected CAs, and Selected with nothing usable admits no client. See TACACS+ over TLS → Troubleshooting. |
| A TACACS+ device is refused with Not in the inventory and an Unknown TACACS+ device alert | TACACS+ answers only inventoried devices, whatever key it uses; the global key is the fallback of inventoried devices only. Add the device (or enable its entry) with TACACS+ on. See Only inventoried devices log in. |
IPv6 devices are logged, matched or refused with the address 10.255.255.225 | Docker’s ip6tables is off, so Docker’s proxy forwards IPv6 from the gateway’s address. Turn it back on, restart Docker and ./taranac up -d; never add the gateway to the inventory. See IPv6. |
| An IPv6 device gets no answer or is refused as unknown, while its IPv4 address works | Each IPv6 device is matched by its own IPv6 address and needs an entry covering it, with its key. See IPv6 devices. |
| ufw rules do not stop traffic to a service port (IPv4 or IPv6) | Docker forwards published ports into the containers; INPUT never sees them. Filter in the DOCKER-USER chain. See IPv6 → The host firewall. |
802.1X, NAC and sessions
Section titled “802.1X, NAC and sessions”| Symptom | What it is, and what to do |
|---|---|
| Wi-Fi 802.1X clients are accepted in the NAC Auth Log and then dropped by the access point right after EAP-Success | The access point derives the WPA key from the MS-MPPE-* keys Taranac returns in the Access-Accept, decrypting them with the shared secret. Check that the access point’s RADIUS secret equals the device’s secret in Taranac and that nothing between them strips vendor-specific attributes. See Wireless 802.1X. |
| A MikroTik access point authorizes the station, which then never gets an address | RouterOS wireless reads the VLAN from Mikrotik-Wireless-VLANID, not from the RFC 3580 tunnel attributes. Check that your mikrotik vendor dictionary entry sends it — see MikroTik RouterOS. |
| A CoA (disconnect, re-authenticate, VLAN change, a portal grant) times out | Taranac sent it to a port the device does not listen on, or with a CoA secret it cannot verify. Read CoA Port on the device’s Details card and the level it came from: the order is device → device group → vendor dictionary → global nac.coa_port. Cisco IOS listens on 1700 — pin Cisco IOS / IOS-XE as the device’s Vendor Dictionary or type the port. Then press CoA test on the same card, which proves port, secret and the device’s dynamic-authorization configuration in one datagram. See Network devices. |
| A CoA is acknowledged, the session ends, and the client never comes back | The switch tore the session down and never sent another Access-Request, so the client never received its new authorization. ./taranac diagnose --protocol nac --device <IP> raises it as a problem — a session stop with no re-authentication from the same station within 60 seconds. See Two CoA problems that look like success. |
| A Disconnect is answered with a NAK (Error-Cause 503) and the user is still on the network | 503 means “the session you named is not the one I have” — usually the port re-authenticated in place. Nothing is archived on a NAK; the accounting stream stays the authority on whether a session is alive. |
| Every session of one switch closed at the same moment, with reason NAS restart | Expected: the switch sent Accounting-On or Accounting-Off, which means it restarted and its sessions are gone. The NAC Auth Log has one NAS restart row for the event. See When a NAS restarts. |
The reject reason reads Client requested … — not offered by Taranac (eap_method_unsupported) | The supplicant asks for an EAP method this server does not offer (LEAP, EAP-MD5 from a real supplicant, …). Set the supplicant to PEAP, EAP-TTLS or EAP-TLS; for EAP-TLS (13), check that a client CA bundle exists. Not counted toward brute-force detection. See EAP rejects with a reason. |
MAB on a TP-Link switch or a Cisco port with mab eap is rejected with mab_md5_mismatch | The switch carries MAB as EAP-MD5, and its MAB password or MAC spelling differs from the User-Name it sends — the password must be the MAC exactly as the switch spells it there. See MAB carried as EAP-MD5. |
The same switches are rejected with eap_md5_mab_disabled | Accept MAB over EAP-MD5 is off (NAC → Settings → NAC, MAC Authentication Bypass). |
EAP-MD5 is rejected with eap_md5_wireless | EAP-MD5 derives no keys, so it cannot work on Wi-Fi. Use plain MAB on that SSID. |
The reject reason is tls_server_cert_rejected | The endpoint does not trust Taranac’s server certificate and aborted the handshake — a certificate problem, not a password one. Import the issuing CA into the supplicant’s trust store (the Local Machine store on Windows). See PKI & trust. |
EAP-TLS is rejected with tls_identity_domain_mismatch or tls_identity_no_domain | The certificate names a directory account, but its UPN or CN is not in that LDAP configuration’s domain suffix (an exact, case-insensitive match; sub-domains and alternative UPN suffixes do not match) and it is not the account’s own mail. A bare CN has no domain at all. Set the domain suffix — required for OpenLDAP and FreeIPA — or fix the certificate template. See EAP-TLS for domain users → Troubleshooting. |
| A device’s EST renewal is refused, while enrollment by password works | Renewal with the current certificate works only through the EST gateway on port 8445 — sent to 443 it is always refused. The certificate must also be one this Taranac issued and still active. Open 8445 on the firewall and point the device’s renewal at it. See EST enrollment → Troubleshooting. |
A local account fails over 802.1X when the supplicant sends DOMAIN\user or user@domain | A qualified name may only ever resolve to a directory account. Send the bare name for a local account — on Windows, switch off Automatically use Windows logon name. See 802.1X authentication. |
| A Huawei attribute you added to the vendor dictionary by hand never reaches the switch | NAC reads Huawei campus switches with Huawei’s campus attribute list. A name that exists only in the carrier list cannot be resolved, and an unresolvable name is absent from the reply rather than an error. Use the campus name. |
A classification rule matches more than you wrote — PC-01 also catches PC-011, OLD-PC-01 | By design. A host-name, certificate-identity or manufacturer value without a wildcard is a substring match — required for manufacturer names, where you type Cisco and the database holds “Cisco Systems, Inc”. Use a character class to make it a glob: PC-0[1] matches PC-01 and not PC-011. See Endpoints. |
| After a portal sign-in, the phone’s sign-in window does not close | Where the portal opens access itself, the window closes through the RFC 8908 captive-portal API, published only when the Portal URL is https:// with a host name (not an address) that has a certificate a phone can verify. Without that the window closes when the page reloads itself, or on the phone’s own timer. Where the network opens access, the guest is moved by a CoA — check the CoA row above. See Standalone captive portal. |
Discovery and profiling
Section titled “Discovery and profiling”| Symptom | What it is, and what to do |
|---|---|
| DHCP Probe hears nothing: Taken stays at 0 on its listening point | Work from the relay inward. The relay must name the listening point as an additional ip helper-address on the client VLAN’s gateway interface; UDP 67 from the relay’s address must be allowed to it; on a collector the probe must be switched on (COLLECTOR_DHCP_PROBE_ENABLED=true) with host networking; and Accept relays from on the Settings tab must include the relay if it is not empty. Where the switch serves DHCP itself there is nothing to relay — put a collector in the segment. See DHCP Probe. |
| A scanner zone shows nothing in NAC Logs → Scanner | A scanner is always a site collector. Check on the Scanner page that the zone is Enabled, that its interface is not flagged Down or Address lost, that the interface Can Listen and scan if you chose an active method, and that the collector runs with host networking (COLLECTOR_NETWORK_MODE=host) — in bridge mode it sees none of the site’s wires. Listening alone only hears what devices announce. See Scanner. |
| The SNMP Test fails | The drawer names the failure: Silence (SNMP off, an ACL, or a wrong v1/v2c community, in that order), Unknown user, Credential refused, Clocks disagree (fix NTP). On v3 open Show the whole conversation — nearly every v3 failure is in the engine-ID exchange at the top. For a poll that answers but comes back short — an empty forwarding table — run ./taranac diagnose --protocol snmp --device "<name>". See SNMP. |
| SNMP fills Model and Serial number but not OS version or Location, or does not change them | A poll fills these fields only while they are empty or were last written by SNMP; a value someone typed (or imported) stays, and the field offers take from SNMP instead. The OS version comes from the template, else from ENTITY-MIB’s entPhysicalSoftwareRev; a device that reports neither leaves it empty. Location comes from sysLocation. See SNMP. |
| A device’s profile stops at a shallow node, or the card says no rule recognises it | The card says why the chain stopped, and each reason points somewhere different: nothing reported anything, the next level needs a signal no source provides, a gap in the rules, a dispute, a threshold not reached. See Why the chain stopped; a gap in the rules is already grouped on the Not identified tab. |
| The rule-set Exchange fails | Check connection on the taranac.pro account card walks DNS, TCP, TLS (through your proxy) and the site’s answer, and names the step that failed. An installation that reaches nothing imports the rule set as a file. See Rule-set updates. |
Directories and identity
Section titled “Directories and identity”| Symptom | What it is, and what to do |
|---|---|
LDAPS fails with certificate verify failed | The Directory server CA certificate on the configuration’s Connection tab is empty or is not the CA that issued the directory’s certificate. The EAP-TLS CA on the other tab is never used for this. An encrypted connection with validation on and an empty CA is legitimate only behind a publicly-trusted directory certificate. See LDAP directories. |
| A group brings in nobody and shows Members: 0 | The picker, the sync preview and the run’s log say why: members under an attribute Taranac does not read, login names (memberUid) that match no account, or a primary group with Import primary group members switched off. See When a group brings in nobody. |
| A member you added to an imported group by hand disappears at the next sync | Membership of an imported group is whatever the directory says. The removal is audited; the account itself is never removed. |
| No email arrives when a directory sync fails | A failed sync is an alert, one per configuration and phase (users, computers): an error when the run fails, a warning when it finishes with failed records, resolved by the next clean run. Only the success report is an email, and it is a per-configuration setting. See What a run reports. |
A silent directory makes logins fail after about ten seconds with domain_unreachable | The directory did not answer within the configuration’s Connection timeout (10 seconds by default). The request fails closed and counts against nobody’s lockout. |
| MS-CHAPv2 against FreeIPA is refused | ipaNTHash must be readable by the bind account, and FreeIPA’s own ACIs exclude it. The MS-CHAPv2 tab generates the ACI’s LDIF, and Check the directory reports whether the hash is readable. See LDAP directories. |
| You cannot change the user ’…’: they hold permissions you do not, or You cannot add users to the group ’…’ | The account you are changing, or the group you are adding someone to, holds permissions your role does not; the message lists them. An administrator, or a role that holds them, must make the change. See Who may change whom. |
| An existing user’s logins are refused as if the account did not exist, and the daemon log says the username is held by N accounts | Several accounts — a local one and a directory one, say — share the name, and a shared name is refused everywhere. The alert Usernames shared by several accounts lists them; delete or rename the extra accounts. See One username, one account. |
| Another LDAP synchronization is running | The same sync of that directory configuration is already running, and a manual one started meanwhile is refused (a scheduled one is skipped). Syncs of different configurations run side by side. Try again when it has finished. |
A directory sync fails with sizeLimitExceeded, timeLimitExceeded or a referral | The directory cut the answer short — OpenLDAP caps a search at 500 entries by default. A partial answer is treated as a failure, so nothing is removed on the strength of it. Raise the limit for the bind account. See When the answer cannot be trusted. |
| System MFA Mode set to disabled and people are still challenged | See Behaviour that looks like a fault. |
Logs, audit and alerts
Section titled “Logs, audit and alerts”| Symptom | What it is, and what to do |
|---|---|
| A configuration change that saved cleanly is not reflected on the device | Check for a *_config:validation alert first — a failed validation means the last change did not load. See Alerts. |
| Audit archiving is enabled and there are no JSONL files anywhere | See below. |
Configuration tracker
Section titled “Configuration tracker”| Symptom | What it is, and what to do |
|---|---|
| Bulk onboarding reports SSH negotiation failed (did not finish within 8 s) or did not answer the login within … for devices that are up | The devices are reachable but slow to complete the SSH key exchange or the login; the credential was not rejected. Raise Probe timeout, s (5–30) in the wizard’s first step and probe again. No SSH algorithm in common is different: the device and the collector cannot agree on key exchange, cipher or MAC. See Bulk onboarding. |
| A configuration collection fails with Recipe step N (…) did not return to the device prompt | That recipe step changes the CLI context (config vdom, edit, configure terminal), which collection does not support, because it only reads. Fetch the configuration from the start prompt instead. If the command does not change the prompt, it did not finish within the timeout. See Recipes. |
| A collection fails with Internal error while … (…). Details are in the api log. | The failure is inside Taranac, not on the device; the message names the stage (preparing, talking to the device, storing the version, recording the result) and the error class. Read the api log around that time (./taranac logs api) and include it in a ticket. See Collection runs. |
Cluster, upgrades and backups
Section titled “Cluster, upgrades and backups”High availability documents the mechanics of every row here; this table only indexes the symptom.
| Symptom | What it is, and what to do |
|---|---|
member count is unequal on a joining machine, or a rebuilt node whose etcd refuses to start | A stale declared etcd member list in that node’s .env. Each node’s ./taranac update rewrites it from the roster and the live ring before it restarts anything, and add-node refreshes its own first; ./taranac cluster sync, run on the node concerned, does it at once without recreating anything. |
./taranac update refuses: this is a cluster, and … is not the next member to update | Expected — the order is every replica, then the leader, then the witness. Nothing was changed. Run the command the refusal prints (Start here: <node> → ./taranac update --version <target>) on the machine it names. See Upgrades under HA. |
./taranac update refuses: … is being updated right now | Another member holds the cluster’s single update lock (an update, or a topology command such as add-node). Wait for that run to end, then run the printed command. A run that died frees the lock by itself within two minutes. |
| A replica’s update ends with AAA (tacacs/radius/nac) and the UI on this node are down until the leader is updated | Expected on a release that changes the schema: the replica’s api waits for the leader to migrate it. Update the leader next, right away. A finisher container (taranac-update-finisher) starts the rest of the replica by itself; nothing to re-run there. |
A plain ./taranac update refuses: this cluster is in the middle of a rollout to … | Some members already run a newer version (a release candidate, say) than the latest stable release a bare update would aim at. Continue with the printed ./taranac update --version <that version>. |
cluster:connection_ceiling warning right after an update | The cluster was built when the database’s max_connections default was lower, and the value is bootstrap-only — updates never raise it. Not a new fault: raise it once, replicas first and the leader last. See Raise the connection ceiling. |
cluster:etcd_member_down during a planned reboot | Expected on two database nodes with a witness, or three database nodes: any one machine down for more than a minute or two leaves the ring one loss from stopping writes, which is an error. It clears when the machine is back. |
add-witness, add-node or diagnose --ha says this node’s own 2379 or 8008 is dropped from the Docker network ([DOCKER PATH]) | The host firewall drops traffic from the stack’s containers to the node’s own address. Admit 10.255.255.240/28 to that address on tcp 2379 and 8008; every check prints the exact rule. See The node’s own Docker network. |
A container is listed as <12 hex>_taranac-db (or another <12 hex>_taranac-… name) | docker compose was interrupted while recreating it. Any ./taranac command renames it back without restarting it; if the real name is still held by the container being replaced, remove the leftover as printed and never start it. ./taranac diagnose --ha reports it as [RENAMED]. |
./taranac up, down -v, install.sh or a cluster command refuses on a cluster node | Each refusal is a guard against a command that would break a member — a copied .env or cloned VM, a wipe, a re-install outside the update order, a decommission of a live node. The message names what to run instead. See Commands that refuse, and why. |
| The database container refuses to start with a message about the base compose file | Something started docker-compose.yml without the Patroni overlay on a cluster node — a writable PostgreSQL on Patroni’s data directory. The refusal is the guard working; bring the node up with ./taranac up -d. |
| An update stops and names a phase — replacing files, writing the version pin, or pulling images | Nothing after that phase ran. Fix what it names and run the update again; an installation that cannot fetch the bundle updates from a file with ./taranac update --from taranac-bundle-<version>.tar.gz. |
install.sh, bootstrap.sh or ./taranac update refuses: Docker … is too old, The docker command here is Podman, kernel has IPv6 turned off, or Docker refused to create a network with IPv6 beside IPv4 | A host requirement is not met; nothing was changed. Follow the fix printed under the refusal, then run the same command again. See IPv6 → What the host needs. |
./taranac update or install.sh stops naming inventory entries that cover the Docker gateway | IPv6 devices matched those entries only through the gateway; with their own address they need IPv6 entries. Add an IPv6 entry beside each with the same key (or ::/0) and run it again, or go on with CONTINUE / --ipv6-own-address-ok / TARANAC_IPV6_OWN_ADDRESS_OK=1. See IPv6. |
./taranac update lists TACACS+ addresses that logged in without an inventory entry | A warning, not a stop: TACACS+ answers only inventoried devices, so those addresses are refused. Add them to the inventory; the list is also in .taranac-update.log. |
| A cluster rollout stops half-way: one database node refuses the update over its Docker version | That node’s Docker is older than 28.0; the members updated before it already run the new release. Update Docker on it, then run the printed command there. Update Docker on every database node before the first. See Upgrades under HA. |
An update or a cluster command says an image is damaged on this host; or cluster add-witness failed with exec format error | An image’s files differ from its own layers. The command replaces it when it can and otherwise stops with nothing changed, naming the file and the cure (docker save / docker load, or copying the image from a host whose copy passes). See Damaged images. |
| The host loses its IPv6 default route hours after installing Taranac | Docker’s IPv6 network turned on IPv6 forwarding, and the interface ignores router advertisements with accept_ra=1. install.sh and the update set accept_ra=2 where they detect this; set it on the interface they missed and persist it in /etc/sysctl.d/99-taranac-accept-ra.conf. See IPv6. |
| Disk space after updates | An update removes superseded images and keeps two versions, so the previous one is there for a rollback. ./taranac update --keep-images keeps them all. |
A backup is marked incomplete (backup:incomplete alert) | A part of the archive could not be collected — an unreadable file degrades the archive instead of aborting it. The alert names what is missing. See Backup & upgrades. |
| Rotating the master key | Only on the host: ./taranac rotate-key. The console does not offer it, because a rotation must re-encrypt every stored secret — device keys, enable passwords, CA keys, NCM credentials, TOTP secrets and the licence — at once. |
A collector shows Requires update (and a collector:update:<id> alert); configurations are not collected, SNMP and the Web Probe are fine | The core pauses only the job whose contract the collector’s release does not match; the collector stays Online. Run ./collector-join.sh --update on its host. See Keep the collector in step with its core. |
./collector-join.sh --update says “already in step with the core — nothing to do” while nothing is being collected | The version it compares against is the last one the core reported, not a measurement. Read the core’s version on the core (./taranac version) and name it: ./collector-join.sh --update --version <ver>. See Collectors. |
Behaviour that looks like a fault
Section titled “Behaviour that looks like a fault”Each item here is current behaviour. It presents as a fault, and none of it is one you can fix by upgrading or by changing a setting.
“I disabled MFA system-wide and people are still challenged.” mfa.mode — the
System MFA Mode control on Integrations → MFA — is read by no login path.
Setting it to disabled greys the MFA settings page out and stops a setup link being
auto-mailed to a newly created directory account, and that is the whole of its effect. The
AAA policy tester does read it, which is why the tester can report MFA as not enforced on
a rule the daemons will still enforce. The switches that actually stop enforcement are per
path — tacacs.mfa_mode and radius.mfa_mode for device administration, and the rule’s
own mfa field or mfa_required everywhere else. See
Multi-factor (MFA).
Audit archiving writes to a path no volume mounts. With audit_log.archive_enabled on,
the nightly retention job writes audit_archive_<timestamp>.jsonl under
audit_log.archive_path, whose default /data/audit-archive is not a volume in the
shipped compose — standalone included. The files therefore live on the api container’s
writable layer and go the moment the container is recreated, which every ./taranac update
does. Three details make it worse than it looks: the retention purge deletes the rows
whether the archive succeeded or not, the archive step swallows its own errors, and the
meta-audit row records the path as archived regardless. Treat the JSONL archive as
non-durable: keep the history you need in the database
by raising audit_log.retention_days, and use syslog forwarding to a
SIEM — or the audit log’s own export — as the durable copy.
A licence that cannot be decrypted falls back to Community in silence. The stored .lic
is encrypted with the install’s MASTER_KEY. If that key no longer matches the database — a
hand-rolled rotation, or a database moved between installs without its key — the file simply
cannot be read, and the install reports plain Community with no problem code and no
alert, exactly as if none had ever been uploaded. (A stored licence that fails its
signature or its installation binding does raise license:validation; this one does
not.) If Pro disappears after key or database surgery, re-upload the .lic — the file is
not secret and re-uploading is always safe. See Licensing.
High CPU usage
Section titled “High CPU usage”Taranac’s own alert engine watches host CPU and opens a monitoring alert (cpu:load)
once usage stays above 90% for alerts.sustained_minutes (default 2 minutes) — a brief
spike during a config apply or an upgrade is normal and is deliberately debounced. Sustained
high CPU is what’s worth chasing.
1. Find what is hot. docker stats --no-stream attributes the load to a container.
Map the busy container to a service name from ./taranac ps. Common culprits:
tacacs/radius/nachot → you’re under real authentication load, or a misconfigured supplicant is retrying aggressively. Cross-check with the NAC/AAA alerts (a brute-force or unknown-device pattern shows up there) and the daemon logs below.apihot → heavy admin-UI/API use, or a background job. Usually transient.postgreshot → a slow or missing-index query pattern, most often driven by an unusually high AAA or logging rate. Correlate with the traffic you’re seeing.
2. Confirm it’s Taranac and not a neighbour. On a shared host, top -b -n1 | head -20
tells you whether the CPU is actually going to Taranac’s containers or to something else on
the box (a backup job, another application, antivirus). Taranac can only be the cause if its
processes are the ones consuming the CPU.
3. Read the hot service’s logs for the window when load climbed:
./taranac logs --since 30m <service> # e.g. tacacs, radius, nac, apiLook for a tight repeating line — the same client, identity, or query over and over is the usual signature of a runaway load, and it points straight at the fix (a looping supplicant, a monitoring probe hammering the API, a device misconfiguration).
Gather for a ticket: ./taranac version, ./taranac ps, docker stats --no-stream,
top -b -n1 | head -20, and ./taranac logs --since 30m <hot-service>.
Memory pressure
Section titled “Memory pressure”The alert engine opens a memory:usage alert at 85% (warning) / 95% (critical), and a
separate swap:usage condition (disabled by default). Heavy swapping presents as high CPU
and slowness, so rule it out early with the top / docker stats snapshots from the bundle.
If one container’s memory climbs steadily over time rather than tracking load, capture two
docker stats --no-stream snapshots a few minutes apart and attach both — the trend is what
support needs, not a single reading.
Slow or failing authentication
Section titled “Slow or failing authentication”If TACACS+ or RADIUS is timing out or rejecting, there are three states to tell apart, not two, and they have completely different fixes.
-
Service down.
./taranac psshows the daemonExited/Restarting, and Taranac raises a no-debouncecriticalreachability alert (service:tacacs/service:radius). Go to Crash-looping or exited services. -
Service up, reachable, and refusing everything. The container is healthy, the port answers, and on a cluster
cluster statusmay still print Healthy — but the daemon has lost the database and is answering ERROR to nearly every request. This is what theservice:tacacs_backend/service:radius_backendalert exists for:critical, on that node only, after a 30-second stall. The daemon keeps retrying the database, says so in its log, and requests refused during the window are replayed into the AAA log once the database is reachable — so the gap is visible afterwards rather than lost. Check the database first, not the policy. -
Service up, denying specific requests. The daemon is running and most things work. This is almost always policy, device, or identity configuration, not a platform fault. Reproduce one failure and read that request in the logs:
Terminal window ./taranac logs --since 15m radius # or the tacacs / nac serviceA single failed request usually shows the reason inline (unknown NAS/device, wrong shared secret, no matching policy, identity-source unreachable). Recent AAA config changes are a frequent trigger — a failed validation opens a
*_config:validationalert, which is your hint that the last change didn’t load cleanly. If the reason is a pattern rather than a one-off, the symptom index above is where to look next: many of the rows there present exactly as “the daemon is up and says no”.
Crash-looping or exited services
Section titled “Crash-looping or exited services”When ./taranac ps shows a service stuck Restarting or Exited, the logs from just before
it died hold the reason:
./taranac logs --since 15m <service>Read the last lines before each restart — a bad config, a missing secret/certificate, or
a dependency it can’t reach will name itself there. If the failure started right after an
upgrade, check ./taranac version against the versioning & compatibility
notes. Only once you know why it’s failing should you ./taranac restart <service> — a
restart on an unfixed cause just resumes the loop.
One loop has a known cause rather than a log line to read:
- The database container refusing to start with a message about the base compose file.
Something started
docker-compose.ymlwithout the Patroni overlay on a cluster node. That refusal is the guard working; bring the node up with./taranac up -dinstead.
Clustered (HA) deployments
Section titled “Clustered (HA) deployments”On an HA cluster, add the cluster’s own view before drilling into one node — a problem that looks local is often a replication or failover symptom:
./taranac cluster status # verdict, every machine (witness included), roles, lag./taranac cluster verify # can this cluster survive losing a machine?./taranac ps # local container health on THIS node./taranac diagnose --ha # this node's whole view of the cluster, for a ticketThe cluster_sync and cluster_license alert sources surface HA-only conditions (replica
lag, a node down, a machine of the etcd ring not answering, a connection ceiling too low,
an orphaned replication slot, the config-apply readiness gate). Run cluster status and
cluster verify on a database node — on a witness they explain why that host has
neither the database nor the API they drive; ./taranac diagnose --ha works there and
opens with the same verdict. See
High availability for the full failover and recovery
model, and the cluster rows of the index above for the
symptoms operators meet most.
When to escalate
Section titled “When to escalate”If the first-response bundle and the relevant section above don’t point at a cause — or the
fix would be destructive (a restore, a downgrade, anything touching the master key) — open a
ticket with the gathered output rather than guessing. ./taranac diagnose packages a whole
answer in one file — one device’s traffic decoded, why a configuration collection fails, why
an SNMP poll comes back short, or what each node believes about the cluster — and carries no
secret; see
Diagnostic bundles. Include what changed most recently
(an upgrade, a config edit, a traffic change); that single detail resolves a large share of
cases on the first reply.