Перейти к содержимому

Troubleshooting & diagnostics

When something is wrong on a running deployment, resist the urge to restart first and ask questions later. Taranac is a stack of cooperating services, and a blind restart both loses the evidence and often just resets the clock on the real problem. This page is a diagnostic-first runbook: gather a small, specific set of read-only facts, read them, then act. Every command in the bundle below is safe to run on production — none of them change state.

It is also a symptom index. Most of what goes wrong in the field has a name and a page that owns it, so the index routes rather than restates: find the symptom, read the one-line cause, and follow the link.

Run these read-only commands and keep the output — this is the baseline every other section builds on.

Terminal window
./taranac version # installed version (+ whether a newer one exists)
./taranac ps # per-service status: is anything down or restarting?
top -b -n1 | head -20 # host CPU + memory snapshot, busiest processes first
docker stats --no-stream # per-container CPU / memory / IO
df -h # disk usage per mount (a full disk breaks everything)

./taranac ps is the single most useful line: it tells you whether this is a service problem (something is Exited or Restarting) or a host problem (every service is up but the box is starved). Read it before anything else. The names every command below takes are the compose service names — tacacs, radius, nac, api, postgres, edge, frontend, captive-portal and taranac-mfa among them.

On a witness or a standalone collector host, ./taranac ps and ./taranac logs drive that host’s own stack. ./taranac logs <service> for a service that has no container on the host says so — no <service> container on this host. — and lists the ones that exist, so an empty answer means the container itself logged nothing. The commands that need the product’s database or API (cluster …, unlock, reset-password, create-admin, installation-id, reset) say so on such a host and do nothing.

Device administration — TACACS+ and RADIUS

Section titled “Device administration — TACACS+ and RADIUS”
SymptomWhat it is, and what to do
Login succeeds and reaches the right privilege level, but every command is denied — the authorization log shows a refusal with an empty profileSome devices omit the RFC 8907 = / * separator and send a bare <cr> as an argument of its own (Edgecore ECS4100 is the reported platform). Vendor Compatibility on AAA → Settings → TACACS+ tolerates it and is on by default; check it has not been switched off.
The device refuses the whole authorization over an attribute it does not implementThe attribute is marked Mandatory (=) in the template. A device must refuse an authorization carrying a mandatory attribute it does not know; mark it Optional (*) instead. See TACACS+.
A MikroTik administrator’s successful login is preceded by a reject with reason empty_password_probeExpected. RouterOS proves the empty password before prompting the human. The row stays in the log and counts towards no failure metric. See MikroTik RouterOS.
A TACACS+ rule restricted to certain devices never matchesA device restriction that resolves to nothing — an emptied device group, or a device on an FQDN network object — fails closed. Give the rule a group with members, and the device a host, subnet or range object.
A device is missing from the RADIUS clients and is not authenticated at allIts network object is an FQDN, which carries no address. Use a host, subnet or range object.
The Password prompt you configured never appears on the deviceIt applies only while MFA is on, and only an interactive (ASCII) login can display it — a device that sends PAP carries the password in its first packet and asks nothing. A login whose matching rule requires MFA is shown the MFA password prompt instead (Password+MFA: by default), again only while TACACS+ MFA Mode is on and only on an ASCII login. Leave that prompt empty to show the default one to everybody. See Password prompts and MFA.
A RADIUS device-admin login times out and nothing appears in the AAA logThe device sends from an address that no RADIUS-enabled device record covers. RADIUS AAA identifies the device by the packet’s source address; NAS-IP-Address does not count. Pin the device’s RADIUS source interface, or register the address it sends from. See How a request finds its device.
A RADIUS device-admin login matches the wrong device, or the wrong WHERE ruleA subnet device record, or a NAT address several devices share, covers the source address — devices behind one NAT address are one device. Register the source as a more specific record, or remove the NAT. See How a request finds its device.
A TACACS+ over TLS device is rejected at the handshake, or no TLS client connects at allThe device trusts neither the server certificate nor the address it names, or its client certificate comes from a CA the listener does not trust. TLS Client Trust on the TACACS+ settings trusts every enabled internal CA by default; an imported CA counts only under Selected CAs, and Selected with nothing usable admits no client. See TACACS+ over TLS → Troubleshooting.
A TACACS+ device is refused with Not in the inventory and an Unknown TACACS+ device alertTACACS+ answers only inventoried devices, whatever key it uses; the global key is the fallback of inventoried devices only. Add the device (or enable its entry) with TACACS+ on. See Only inventoried devices log in.
IPv6 devices are logged, matched or refused with the address 10.255.255.225Docker’s ip6tables is off, so Docker’s proxy forwards IPv6 from the gateway’s address. Turn it back on, restart Docker and ./taranac up -d; never add the gateway to the inventory. See IPv6.
An IPv6 device gets no answer or is refused as unknown, while its IPv4 address worksEach IPv6 device is matched by its own IPv6 address and needs an entry covering it, with its key. See IPv6 devices.
ufw rules do not stop traffic to a service port (IPv4 or IPv6)Docker forwards published ports into the containers; INPUT never sees them. Filter in the DOCKER-USER chain. See IPv6 → The host firewall.
SymptomWhat it is, and what to do
Wi-Fi 802.1X clients are accepted in the NAC Auth Log and then dropped by the access point right after EAP-SuccessThe access point derives the WPA key from the MS-MPPE-* keys Taranac returns in the Access-Accept, decrypting them with the shared secret. Check that the access point’s RADIUS secret equals the device’s secret in Taranac and that nothing between them strips vendor-specific attributes. See Wireless 802.1X.
A MikroTik access point authorizes the station, which then never gets an addressRouterOS wireless reads the VLAN from Mikrotik-Wireless-VLANID, not from the RFC 3580 tunnel attributes. Check that your mikrotik vendor dictionary entry sends it — see MikroTik RouterOS.
A CoA (disconnect, re-authenticate, VLAN change, a portal grant) times outTaranac sent it to a port the device does not listen on, or with a CoA secret it cannot verify. Read CoA Port on the device’s Details card and the level it came from: the order is device → device group → vendor dictionary → global nac.coa_port. Cisco IOS listens on 1700 — pin Cisco IOS / IOS-XE as the device’s Vendor Dictionary or type the port. Then press CoA test on the same card, which proves port, secret and the device’s dynamic-authorization configuration in one datagram. See Network devices.
A CoA is acknowledged, the session ends, and the client never comes backThe switch tore the session down and never sent another Access-Request, so the client never received its new authorization. ./taranac diagnose --protocol nac --device <IP> raises it as a problem — a session stop with no re-authentication from the same station within 60 seconds. See Two CoA problems that look like success.
A Disconnect is answered with a NAK (Error-Cause 503) and the user is still on the network503 means “the session you named is not the one I have” — usually the port re-authenticated in place. Nothing is archived on a NAK; the accounting stream stays the authority on whether a session is alive.
Every session of one switch closed at the same moment, with reason NAS restartExpected: the switch sent Accounting-On or Accounting-Off, which means it restarted and its sessions are gone. The NAC Auth Log has one NAS restart row for the event. See When a NAS restarts.
The reject reason reads Client requested … — not offered by Taranac (eap_method_unsupported)The supplicant asks for an EAP method this server does not offer (LEAP, EAP-MD5 from a real supplicant, …). Set the supplicant to PEAP, EAP-TTLS or EAP-TLS; for EAP-TLS (13), check that a client CA bundle exists. Not counted toward brute-force detection. See EAP rejects with a reason.
MAB on a TP-Link switch or a Cisco port with mab eap is rejected with mab_md5_mismatchThe switch carries MAB as EAP-MD5, and its MAB password or MAC spelling differs from the User-Name it sends — the password must be the MAC exactly as the switch spells it there. See MAB carried as EAP-MD5.
The same switches are rejected with eap_md5_mab_disabledAccept MAB over EAP-MD5 is off (NAC → Settings → NAC, MAC Authentication Bypass).
EAP-MD5 is rejected with eap_md5_wirelessEAP-MD5 derives no keys, so it cannot work on Wi-Fi. Use plain MAB on that SSID.
The reject reason is tls_server_cert_rejectedThe endpoint does not trust Taranac’s server certificate and aborted the handshake — a certificate problem, not a password one. Import the issuing CA into the supplicant’s trust store (the Local Machine store on Windows). See PKI & trust.
EAP-TLS is rejected with tls_identity_domain_mismatch or tls_identity_no_domainThe certificate names a directory account, but its UPN or CN is not in that LDAP configuration’s domain suffix (an exact, case-insensitive match; sub-domains and alternative UPN suffixes do not match) and it is not the account’s own mail. A bare CN has no domain at all. Set the domain suffix — required for OpenLDAP and FreeIPA — or fix the certificate template. See EAP-TLS for domain users → Troubleshooting.
A device’s EST renewal is refused, while enrollment by password worksRenewal with the current certificate works only through the EST gateway on port 8445 — sent to 443 it is always refused. The certificate must also be one this Taranac issued and still active. Open 8445 on the firewall and point the device’s renewal at it. See EST enrollment → Troubleshooting.
A local account fails over 802.1X when the supplicant sends DOMAIN\user or user@domainA qualified name may only ever resolve to a directory account. Send the bare name for a local account — on Windows, switch off Automatically use Windows logon name. See 802.1X authentication.
A Huawei attribute you added to the vendor dictionary by hand never reaches the switchNAC reads Huawei campus switches with Huawei’s campus attribute list. A name that exists only in the carrier list cannot be resolved, and an unresolvable name is absent from the reply rather than an error. Use the campus name.
A classification rule matches more than you wrote — PC-01 also catches PC-011, OLD-PC-01By design. A host-name, certificate-identity or manufacturer value without a wildcard is a substring match — required for manufacturer names, where you type Cisco and the database holds “Cisco Systems, Inc”. Use a character class to make it a glob: PC-0[1] matches PC-01 and not PC-011. See Endpoints.
After a portal sign-in, the phone’s sign-in window does not closeWhere the portal opens access itself, the window closes through the RFC 8908 captive-portal API, published only when the Portal URL is https:// with a host name (not an address) that has a certificate a phone can verify. Without that the window closes when the page reloads itself, or on the phone’s own timer. Where the network opens access, the guest is moved by a CoA — check the CoA row above. See Standalone captive portal.
SymptomWhat it is, and what to do
DHCP Probe hears nothing: Taken stays at 0 on its listening pointWork from the relay inward. The relay must name the listening point as an additional ip helper-address on the client VLAN’s gateway interface; UDP 67 from the relay’s address must be allowed to it; on a collector the probe must be switched on (COLLECTOR_DHCP_PROBE_ENABLED=true) with host networking; and Accept relays from on the Settings tab must include the relay if it is not empty. Where the switch serves DHCP itself there is nothing to relay — put a collector in the segment. See DHCP Probe.
A scanner zone shows nothing in NAC Logs → ScannerA scanner is always a site collector. Check on the Scanner page that the zone is Enabled, that its interface is not flagged Down or Address lost, that the interface Can Listen and scan if you chose an active method, and that the collector runs with host networking (COLLECTOR_NETWORK_MODE=host) — in bridge mode it sees none of the site’s wires. Listening alone only hears what devices announce. See Scanner.
The SNMP Test failsThe drawer names the failure: Silence (SNMP off, an ACL, or a wrong v1/v2c community, in that order), Unknown user, Credential refused, Clocks disagree (fix NTP). On v3 open Show the whole conversation — nearly every v3 failure is in the engine-ID exchange at the top. For a poll that answers but comes back short — an empty forwarding table — run ./taranac diagnose --protocol snmp --device "<name>". See SNMP.
SNMP fills Model and Serial number but not OS version or Location, or does not change themA poll fills these fields only while they are empty or were last written by SNMP; a value someone typed (or imported) stays, and the field offers take from SNMP instead. The OS version comes from the template, else from ENTITY-MIB’s entPhysicalSoftwareRev; a device that reports neither leaves it empty. Location comes from sysLocation. See SNMP.
A device’s profile stops at a shallow node, or the card says no rule recognises itThe card says why the chain stopped, and each reason points somewhere different: nothing reported anything, the next level needs a signal no source provides, a gap in the rules, a dispute, a threshold not reached. See Why the chain stopped; a gap in the rules is already grouped on the Not identified tab.
The rule-set Exchange failsCheck connection on the taranac.pro account card walks DNS, TCP, TLS (through your proxy) and the site’s answer, and names the step that failed. An installation that reaches nothing imports the rule set as a file. See Rule-set updates.
SymptomWhat it is, and what to do
LDAPS fails with certificate verify failedThe Directory server CA certificate on the configuration’s Connection tab is empty or is not the CA that issued the directory’s certificate. The EAP-TLS CA on the other tab is never used for this. An encrypted connection with validation on and an empty CA is legitimate only behind a publicly-trusted directory certificate. See LDAP directories.
A group brings in nobody and shows Members: 0The picker, the sync preview and the run’s log say why: members under an attribute Taranac does not read, login names (memberUid) that match no account, or a primary group with Import primary group members switched off. See When a group brings in nobody.
A member you added to an imported group by hand disappears at the next syncMembership of an imported group is whatever the directory says. The removal is audited; the account itself is never removed.
No email arrives when a directory sync failsA failed sync is an alert, one per configuration and phase (users, computers): an error when the run fails, a warning when it finishes with failed records, resolved by the next clean run. Only the success report is an email, and it is a per-configuration setting. See What a run reports.
A silent directory makes logins fail after about ten seconds with domain_unreachableThe directory did not answer within the configuration’s Connection timeout (10 seconds by default). The request fails closed and counts against nobody’s lockout.
MS-CHAPv2 against FreeIPA is refusedipaNTHash must be readable by the bind account, and FreeIPA’s own ACIs exclude it. The MS-CHAPv2 tab generates the ACI’s LDIF, and Check the directory reports whether the hash is readable. See LDAP directories.
You cannot change the user ’…’: they hold permissions you do not, or You cannot add users to the group ’…’The account you are changing, or the group you are adding someone to, holds permissions your role does not; the message lists them. An administrator, or a role that holds them, must make the change. See Who may change whom.
An existing user’s logins are refused as if the account did not exist, and the daemon log says the username is held by N accountsSeveral accounts — a local one and a directory one, say — share the name, and a shared name is refused everywhere. The alert Usernames shared by several accounts lists them; delete or rename the extra accounts. See One username, one account.
Another LDAP synchronization is runningThe same sync of that directory configuration is already running, and a manual one started meanwhile is refused (a scheduled one is skipped). Syncs of different configurations run side by side. Try again when it has finished.
A directory sync fails with sizeLimitExceeded, timeLimitExceeded or a referralThe directory cut the answer short — OpenLDAP caps a search at 500 entries by default. A partial answer is treated as a failure, so nothing is removed on the strength of it. Raise the limit for the bind account. See When the answer cannot be trusted.
System MFA Mode set to disabled and people are still challengedSee Behaviour that looks like a fault.
SymptomWhat it is, and what to do
A configuration change that saved cleanly is not reflected on the deviceCheck for a *_config:validation alert first — a failed validation means the last change did not load. See Alerts.
Audit archiving is enabled and there are no JSONL files anywhereSee below.
SymptomWhat it is, and what to do
Bulk onboarding reports SSH negotiation failed (did not finish within 8 s) or did not answer the login within … for devices that are upThe devices are reachable but slow to complete the SSH key exchange or the login; the credential was not rejected. Raise Probe timeout, s (5–30) in the wizard’s first step and probe again. No SSH algorithm in common is different: the device and the collector cannot agree on key exchange, cipher or MAC. See Bulk onboarding.
A configuration collection fails with Recipe step N (…) did not return to the device promptThat recipe step changes the CLI context (config vdom, edit, configure terminal), which collection does not support, because it only reads. Fetch the configuration from the start prompt instead. If the command does not change the prompt, it did not finish within the timeout. See Recipes.
A collection fails with Internal error while … (…). Details are in the api log.The failure is inside Taranac, not on the device; the message names the stage (preparing, talking to the device, storing the version, recording the result) and the error class. Read the api log around that time (./taranac logs api) and include it in a ticket. See Collection runs.

High availability documents the mechanics of every row here; this table only indexes the symptom.

SymptomWhat it is, and what to do
member count is unequal on a joining machine, or a rebuilt node whose etcd refuses to startA stale declared etcd member list in that node’s .env. Each node’s ./taranac update rewrites it from the roster and the live ring before it restarts anything, and add-node refreshes its own first; ./taranac cluster sync, run on the node concerned, does it at once without recreating anything.
./taranac update refuses: this is a cluster, and … is not the next member to updateExpected — the order is every replica, then the leader, then the witness. Nothing was changed. Run the command the refusal prints (Start here: <node> → ./taranac update --version <target>) on the machine it names. See Upgrades under HA.
./taranac update refuses: … is being updated right nowAnother member holds the cluster’s single update lock (an update, or a topology command such as add-node). Wait for that run to end, then run the printed command. A run that died frees the lock by itself within two minutes.
A replica’s update ends with AAA (tacacs/radius/nac) and the UI on this node are down until the leader is updatedExpected on a release that changes the schema: the replica’s api waits for the leader to migrate it. Update the leader next, right away. A finisher container (taranac-update-finisher) starts the rest of the replica by itself; nothing to re-run there.
A plain ./taranac update refuses: this cluster is in the middle of a rollout to …Some members already run a newer version (a release candidate, say) than the latest stable release a bare update would aim at. Continue with the printed ./taranac update --version <that version>.
cluster:connection_ceiling warning right after an updateThe cluster was built when the database’s max_connections default was lower, and the value is bootstrap-only — updates never raise it. Not a new fault: raise it once, replicas first and the leader last. See Raise the connection ceiling.
cluster:etcd_member_down during a planned rebootExpected on two database nodes with a witness, or three database nodes: any one machine down for more than a minute or two leaves the ring one loss from stopping writes, which is an error. It clears when the machine is back.
add-witness, add-node or diagnose --ha says this node’s own 2379 or 8008 is dropped from the Docker network ([DOCKER PATH])The host firewall drops traffic from the stack’s containers to the node’s own address. Admit 10.255.255.240/28 to that address on tcp 2379 and 8008; every check prints the exact rule. See The node’s own Docker network.
A container is listed as <12 hex>_taranac-db (or another <12 hex>_taranac-… name)docker compose was interrupted while recreating it. Any ./taranac command renames it back without restarting it; if the real name is still held by the container being replaced, remove the leftover as printed and never start it. ./taranac diagnose --ha reports it as [RENAMED].
./taranac up, down -v, install.sh or a cluster command refuses on a cluster nodeEach refusal is a guard against a command that would break a member — a copied .env or cloned VM, a wipe, a re-install outside the update order, a decommission of a live node. The message names what to run instead. See Commands that refuse, and why.
The database container refuses to start with a message about the base compose fileSomething started docker-compose.yml without the Patroni overlay on a cluster node — a writable PostgreSQL on Patroni’s data directory. The refusal is the guard working; bring the node up with ./taranac up -d.
An update stops and names a phase — replacing files, writing the version pin, or pulling imagesNothing after that phase ran. Fix what it names and run the update again; an installation that cannot fetch the bundle updates from a file with ./taranac update --from taranac-bundle-<version>.tar.gz.
install.sh, bootstrap.sh or ./taranac update refuses: Docker … is too old, The docker command here is Podman, kernel has IPv6 turned off, or Docker refused to create a network with IPv6 beside IPv4A host requirement is not met; nothing was changed. Follow the fix printed under the refusal, then run the same command again. See IPv6 → What the host needs.
./taranac update or install.sh stops naming inventory entries that cover the Docker gatewayIPv6 devices matched those entries only through the gateway; with their own address they need IPv6 entries. Add an IPv6 entry beside each with the same key (or ::/0) and run it again, or go on with CONTINUE / --ipv6-own-address-ok / TARANAC_IPV6_OWN_ADDRESS_OK=1. See IPv6.
./taranac update lists TACACS+ addresses that logged in without an inventory entryA warning, not a stop: TACACS+ answers only inventoried devices, so those addresses are refused. Add them to the inventory; the list is also in .taranac-update.log.
A cluster rollout stops half-way: one database node refuses the update over its Docker versionThat node’s Docker is older than 28.0; the members updated before it already run the new release. Update Docker on it, then run the printed command there. Update Docker on every database node before the first. See Upgrades under HA.
An update or a cluster command says an image is damaged on this host; or cluster add-witness failed with exec format errorAn image’s files differ from its own layers. The command replaces it when it can and otherwise stops with nothing changed, naming the file and the cure (docker save / docker load, or copying the image from a host whose copy passes). See Damaged images.
The host loses its IPv6 default route hours after installing TaranacDocker’s IPv6 network turned on IPv6 forwarding, and the interface ignores router advertisements with accept_ra=1. install.sh and the update set accept_ra=2 where they detect this; set it on the interface they missed and persist it in /etc/sysctl.d/99-taranac-accept-ra.conf. See IPv6.
Disk space after updatesAn update removes superseded images and keeps two versions, so the previous one is there for a rollback. ./taranac update --keep-images keeps them all.
A backup is marked incomplete (backup:incomplete alert)A part of the archive could not be collected — an unreadable file degrades the archive instead of aborting it. The alert names what is missing. See Backup & upgrades.
Rotating the master keyOnly on the host: ./taranac rotate-key. The console does not offer it, because a rotation must re-encrypt every stored secret — device keys, enable passwords, CA keys, NCM credentials, TOTP secrets and the licence — at once.
A collector shows Requires update (and a collector:update:<id> alert); configurations are not collected, SNMP and the Web Probe are fineThe core pauses only the job whose contract the collector’s release does not match; the collector stays Online. Run ./collector-join.sh --update on its host. See Keep the collector in step with its core.
./collector-join.sh --update says “already in step with the core — nothing to do” while nothing is being collectedThe version it compares against is the last one the core reported, not a measurement. Read the core’s version on the core (./taranac version) and name it: ./collector-join.sh --update --version <ver>. See Collectors.

Each item here is current behaviour. It presents as a fault, and none of it is one you can fix by upgrading or by changing a setting.

“I disabled MFA system-wide and people are still challenged.” mfa.mode — the System MFA Mode control on Integrations → MFA — is read by no login path. Setting it to disabled greys the MFA settings page out and stops a setup link being auto-mailed to a newly created directory account, and that is the whole of its effect. The AAA policy tester does read it, which is why the tester can report MFA as not enforced on a rule the daemons will still enforce. The switches that actually stop enforcement are per path — tacacs.mfa_mode and radius.mfa_mode for device administration, and the rule’s own mfa field or mfa_required everywhere else. See Multi-factor (MFA).

Audit archiving writes to a path no volume mounts. With audit_log.archive_enabled on, the nightly retention job writes audit_archive_<timestamp>.jsonl under audit_log.archive_path, whose default /data/audit-archive is not a volume in the shipped compose — standalone included. The files therefore live on the api container’s writable layer and go the moment the container is recreated, which every ./taranac update does. Three details make it worse than it looks: the retention purge deletes the rows whether the archive succeeded or not, the archive step swallows its own errors, and the meta-audit row records the path as archived regardless. Treat the JSONL archive as non-durable: keep the history you need in the database by raising audit_log.retention_days, and use syslog forwarding to a SIEM — or the audit log’s own export — as the durable copy.

A licence that cannot be decrypted falls back to Community in silence. The stored .lic is encrypted with the install’s MASTER_KEY. If that key no longer matches the database — a hand-rolled rotation, or a database moved between installs without its key — the file simply cannot be read, and the install reports plain Community with no problem code and no alert, exactly as if none had ever been uploaded. (A stored licence that fails its signature or its installation binding does raise license:validation; this one does not.) If Pro disappears after key or database surgery, re-upload the .lic — the file is not secret and re-uploading is always safe. See Licensing.

Taranac’s own alert engine watches host CPU and opens a monitoring alert (cpu:load) once usage stays above 90% for alerts.sustained_minutes (default 2 minutes) — a brief spike during a config apply or an upgrade is normal and is deliberately debounced. Sustained high CPU is what’s worth chasing.

1. Find what is hot. docker stats --no-stream attributes the load to a container. Map the busy container to a service name from ./taranac ps. Common culprits:

  • tacacs / radius / nac hot → you’re under real authentication load, or a misconfigured supplicant is retrying aggressively. Cross-check with the NAC/AAA alerts (a brute-force or unknown-device pattern shows up there) and the daemon logs below.
  • api hot → heavy admin-UI/API use, or a background job. Usually transient.
  • postgres hot → a slow or missing-index query pattern, most often driven by an unusually high AAA or logging rate. Correlate with the traffic you’re seeing.

2. Confirm it’s Taranac and not a neighbour. On a shared host, top -b -n1 | head -20 tells you whether the CPU is actually going to Taranac’s containers or to something else on the box (a backup job, another application, antivirus). Taranac can only be the cause if its processes are the ones consuming the CPU.

3. Read the hot service’s logs for the window when load climbed:

Terminal window
./taranac logs --since 30m <service> # e.g. tacacs, radius, nac, api

Look for a tight repeating line — the same client, identity, or query over and over is the usual signature of a runaway load, and it points straight at the fix (a looping supplicant, a monitoring probe hammering the API, a device misconfiguration).

Gather for a ticket: ./taranac version, ./taranac ps, docker stats --no-stream, top -b -n1 | head -20, and ./taranac logs --since 30m <hot-service>.

The alert engine opens a memory:usage alert at 85% (warning) / 95% (critical), and a separate swap:usage condition (disabled by default). Heavy swapping presents as high CPU and slowness, so rule it out early with the top / docker stats snapshots from the bundle. If one container’s memory climbs steadily over time rather than tracking load, capture two docker stats --no-stream snapshots a few minutes apart and attach both — the trend is what support needs, not a single reading.

If TACACS+ or RADIUS is timing out or rejecting, there are three states to tell apart, not two, and they have completely different fixes.

  • Service down. ./taranac ps shows the daemon Exited/Restarting, and Taranac raises a no-debounce critical reachability alert (service:tacacs / service:radius). Go to Crash-looping or exited services.

  • Service up, reachable, and refusing everything. The container is healthy, the port answers, and on a cluster cluster status may still print Healthy — but the daemon has lost the database and is answering ERROR to nearly every request. This is what the service:tacacs_backend / service:radius_backend alert exists for: critical, on that node only, after a 30-second stall. The daemon keeps retrying the database, says so in its log, and requests refused during the window are replayed into the AAA log once the database is reachable — so the gap is visible afterwards rather than lost. Check the database first, not the policy.

  • Service up, denying specific requests. The daemon is running and most things work. This is almost always policy, device, or identity configuration, not a platform fault. Reproduce one failure and read that request in the logs:

    Terminal window
    ./taranac logs --since 15m radius # or the tacacs / nac service

    A single failed request usually shows the reason inline (unknown NAS/device, wrong shared secret, no matching policy, identity-source unreachable). Recent AAA config changes are a frequent trigger — a failed validation opens a *_config:validation alert, which is your hint that the last change didn’t load cleanly. If the reason is a pattern rather than a one-off, the symptom index above is where to look next: many of the rows there present exactly as “the daemon is up and says no”.

When ./taranac ps shows a service stuck Restarting or Exited, the logs from just before it died hold the reason:

Terminal window
./taranac logs --since 15m <service>

Read the last lines before each restart — a bad config, a missing secret/certificate, or a dependency it can’t reach will name itself there. If the failure started right after an upgrade, check ./taranac version against the versioning & compatibility notes. Only once you know why it’s failing should you ./taranac restart <service> — a restart on an unfixed cause just resumes the loop.

One loop has a known cause rather than a log line to read:

  • The database container refusing to start with a message about the base compose file. Something started docker-compose.yml without the Patroni overlay on a cluster node. That refusal is the guard working; bring the node up with ./taranac up -d instead.

On an HA cluster, add the cluster’s own view before drilling into one node — a problem that looks local is often a replication or failover symptom:

Terminal window
./taranac cluster status # verdict, every machine (witness included), roles, lag
./taranac cluster verify # can this cluster survive losing a machine?
./taranac ps # local container health on THIS node
./taranac diagnose --ha # this node's whole view of the cluster, for a ticket

The cluster_sync and cluster_license alert sources surface HA-only conditions (replica lag, a node down, a machine of the etcd ring not answering, a connection ceiling too low, an orphaned replication slot, the config-apply readiness gate). Run cluster status and cluster verify on a database node — on a witness they explain why that host has neither the database nor the API they drive; ./taranac diagnose --ha works there and opens with the same verdict. See High availability for the full failover and recovery model, and the cluster rows of the index above for the symptoms operators meet most.

If the first-response bundle and the relevant section above don’t point at a cause — or the fix would be destructive (a restore, a downgrade, anything touching the master key) — open a ticket with the gathered output rather than guessing. ./taranac diagnose packages a whole answer in one file — one device’s traffic decoded, why a configuration collection fails, why an SNMP poll comes back short, or what each node believes about the cluster — and carries no secret; see Diagnostic bundles. Include what changed most recently (an upgrade, a config edit, a traffic change); that single detail resolves a large share of cases on the first reply.