Troubleshooting & diagnostics

When something is wrong on a running deployment, resist the urge to restart first and ask questions later. Taranac is a stack of cooperating services, and a blind restart both loses the evidence and often just resets the clock on the real problem. This page is a diagnostic-first runbook: gather a small, specific set of read-only facts, read them, then act. Every command here is safe to run on production — none of them change state.

Run these read-only commands and keep the output — this is the baseline every other section builds on.

./taranac version            # installed version (+ whether a newer one exists)
./taranac ps                 # per-service status: is anything down or restarting?
top -b -n1 | head -20        # host CPU + memory snapshot, busiest processes first
docker stats --no-stream     # per-container CPU / memory / IO
df -h                        # disk usage per mount (a full disk breaks everything)

./taranac ps is the single most useful line: it tells you whether this is a service problem (something is Exited or Restarting) or a host problem (every service is up but the box is starved). Read it before anything else.

Taranac’s own alert engine watches host CPU and opens a monitoring alert (cpu:load) once usage stays above 90% for alerts.sustained_minutes (default 2 minutes) — a brief spike during a config apply or an upgrade is normal and is deliberately debounced. Sustained high CPU is what’s worth chasing.

1. Find what is hot. docker stats --no-stream attributes the load to a container. Map the busy container to a service name from ./taranac ps. Common culprits:

• radius / nac / the TACACS+ daemon hot → you’re under real authentication load, or a misconfigured supplicant is retrying aggressively. Cross-check with the NAC/AAA alerts (a brute-force or unknown-device pattern shows up there) and the daemon logs below.
• api hot → heavy admin-UI/API use, or a background job. Usually transient.
• The database container hot → a slow or missing-index query pattern, most often driven by an unusually high AAA or logging rate. Correlate with the traffic you’re seeing.

2. Confirm it’s Taranac and not a neighbour. On a shared host, top -b -n1 | head -20 tells you whether the CPU is actually going to Taranac’s containers or to something else on the box (a backup job, another application, antivirus). Taranac can only be the cause if its processes are the ones consuming the CPU.

3. Read the hot service’s logs for the window when load climbed:

./taranac logs --since 30m <service>     # e.g. radius, nac, api

Look for a tight repeating line — the same client, identity, or query over and over is the usual signature of a runaway load, and it points straight at the fix (a looping supplicant, a monitoring probe hammering the API, a device misconfiguration).

Gather for a ticket: ./taranac version, ./taranac ps, docker stats --no-stream, top -b -n1 | head -20, and ./taranac logs --since 30m <hot-service>.

The alert engine opens a memory:usage alert at 85% (warning) / 95% (critical), and a separate swap:usage condition (disabled by default). Heavy swapping presents as high CPU and slowness, so rule it out early with the top / docker stats snapshots from the bundle. If one container’s memory climbs steadily over time rather than tracking load, capture two docker stats --no-stream snapshots a few minutes apart and attach both — the trend is what support needs, not a single reading.

If TACACS+ or RADIUS is timing out or rejecting, separate “the service is down” from “the service is up but saying no” — they have completely different fixes.

• Service down. ./taranac ps shows the daemon Exited/Restarting, and Taranac raises a no-debounce critical reachability alert (service:tacacs / service:radius). Go to Crash-looping or exited services.

• Service up, denying. The daemon is running but auth fails. This is almost always policy, device, or identity configuration, not a platform fault. Reproduce one failure and read that request in the logs:

  ./taranac logs --since 15m radius     # or the tacacs / nac service

  A single failed request usually shows the reason inline (unknown NAS/device, wrong shared secret, no matching policy, identity-source unreachable). Recent AAA config changes are a frequent trigger — a failed validation opens a *_config:validation alert, which is your hint that the last change didn’t load cleanly.

When ./taranac ps shows a service stuck Restarting or Exited, the logs from just before it died hold the reason:

./taranac logs --since 15m <service>

Read the last lines before each restart — a bad config, a missing secret/certificate, or a dependency it can’t reach will name itself there. If the failure started right after an upgrade, check ./taranac version against the versioning & compatibility notes. Only once you know why it’s failing should you ./taranac restart <service> — a restart on an unfixed cause just resumes the loop.

On an HA cluster, add the cluster’s own view before drilling into one node — a problem that looks local is often a replication or failover symptom:

./taranac cluster status     # nodes, roles, replication lag / health
./taranac ps                 # local container health on THIS node

The cluster_sync and cluster_license alert sources surface HA-only conditions (replica lag, a node down, an orphaned replication slot, the config-apply readiness gate). See High availability for the full failover and recovery model.

If the first-response bundle and the relevant section above don’t point at a cause — or the fix would be destructive (a restore, a downgrade, anything touching the master key) — open a ticket with the gathered output rather than guessing. Include what changed most recently (an upgrade, a config edit, a traffic change); that single detail resolves a large share of cases on the first reply.