Diagnostic bundles

./taranac diagnose is one command with three collectors behind it. Each answers a different question, and each hands you one thing you can attach to a ticket whole, without gathering anything else by hand.

The first is a capture — it records a window while you reproduce a fault. The other two are not: an NCM collection is a session Taranac itself opens, and HA faults are state rather than events. That difference is why they are three collectors and not three flags on one.

For host-level problems — high CPU, memory pressure, crash-looping services — start at Troubleshooting & diagnostics instead.

Everything from here to Reading decoded.txt is the traffic collector — --protocol tacacs, radius or nac. The other two collectors have their own sections below.

When device administration or 802.1X stops working for a device, the explanation is almost never in one place. It is spread across four: the packets on the wire, the daemon’s own log for those few seconds, the records Taranac wrote at the time, and how the device is configured. Nobody gathers four things by hand while the problem is happening. This collector gathers all four for one device and one protocol, decodes the traffic so the archive is readable, prints in plain words what it noticed, and packs everything into a single .tar.gz. For 802.1X there is a fifth place — the backend’s own log, the only record of a CoA — and it is collected as well.

Four things must be true. The collector checks each of them and stops with a specific message rather than producing an empty archive:

One thing the tool cannot check for you: the address must be the one the device sends from. That is its source IP — often a loopback or management interface, not the address you SSH into. Getting this wrong is the single commonest cause of an empty capture.

./taranac diagnose --protocol tacacs --device 10.0.0.5

Start it, reproduce the problem on the device, press Enter. That is the whole workflow.

Use --duration when reproducing the fault and watching the terminal are not the same person’s job — start the collector, tell the other end to try, walk back.

The archive is named for what it contains: taranac-diag-<protocol>-<device>-<UTC timestamp>.tar.gz.

The order matters: the two checks come before the capture, because the answer is frequently already there.

1. How is this address configured in Taranac? Every network object covering the address is listed — one IP can sit inside a host entry and two subnets at once, each with its own key or inheriting a different one — most specific first, since that is the one the daemon applies. For each: whether the device record is enabled, whether the protocol is switched on for it, where its shared secret comes from and that secret’s fingerprint.

Three conditions are reported as findings here, because each one makes the configuration generator skip the device entirely — so the daemon has no client entry for it and drops its requests before any policy is evaluated:

• the device record is disabled;
• the protocol is off for that device, even though the device exists;
• no shared secret resolves — device, groups and the global setting are all empty.

If nothing in Taranac covers the address at all, that is said outright. For RADIUS and 802.1X the dynamic-client networks are consulted too, so a device accepted with the global secret and no device record of its own is reported as such rather than as “not found”.

2. Which ports is the daemon actually listening on? Read from the running process, never from the documented defaults, and combined with the published host ports — with a remapped port the packet carries one number on the physical interface and another on the Docker bridge after NAT, and filtering on only one set silently loses half the story. Loopback-only helper sockets are dropped from the filter.

On a nac run the filter is widened once more, with the CoA ports. Change of Authorization (RFC 5176) is the one direction the daemon never sees: a disconnect, a VLAN change, a port bounce and the stale-session probe all leave the api container for a port on the device, and nothing in the daemon container listens on it. Filtering on the daemon’s ports alone therefore hides every one of them — and hides the switch’s answer, which is usually the whole question. The ports come from the configuration check above, which resolves them per matching device; when they are not already in the daemon’s own set, the run says → Capturing CoA as well, on port(s): … on its own line.

This check exists because a device whose protocol flag is off and a daemon on a custom port produce identical symptoms and identical empty captures.

3. Capture. tcpdump runs on the host, on all interfaces, filtered to that device and those ports, with full-length packets. Capturing on any sees both the physical interface and the Docker bridge — so a packet appearing twice means it was forwarded, and a packet appearing only on the physical interface means it never reached the container, which is a firewall or NAT problem rather than a Taranac one.

The filter also takes ICMP from that address. A port-unreachable is how a switch says nothing was listening where Taranac sent — and for a CoA that is the difference between the switch ignored our request and the request never had a listener, two opposite diagnoses that a ports-only filter cannot tell apart. ICMP is not decoded; it sits in capture.pcap, where Wireshark names it in one line.

4. Collect. The daemon’s log is sliced to the capture window by byte offset taken before and after — not by timestamp, because the daemons log in three different time formats and a date parser that misreads one returns nothing silently. Then: the records Taranac itself wrote during the window, the generated daemon configuration with secrets masked, short context tails from the neighbouring services, and the backend’s own log sliced to the same window — the only place a CoA outcome is written down.

5. Decode. The capture is decoded inside the api container — capture in, readable text out, the key never leaves.

6. Report. A summary is printed to your terminal and written into the archive as summary.txt.

This is a real collection from the demo platform — a TACACS+ login and a few commands on a Cisco switch, reproduced while the collector was running:

$ sudo ./taranac diagnose --protocol tacacs --device 10.30.0.14
Taranac diagnostic collector
  protocol : tacacs
  device   : 10.30.0.14
  daemon   : taranac-tacacs

→ Checking how 10.30.0.14 is configured in Taranac…
  device: SW6-DEMO [10.30.0.14/32] - enabled, protocol on
          secret from device, fingerprint 709ad291ecf8
          groups: Cisco Devices, Demo Devices
  device: demo-network [10.30.0.0/24] - enabled, protocol on
          secret from device, fingerprint 6aabedfde070
          groups: Demo Devices
  ! 2 network objects contain 10.30.0.14; the daemon uses the most specific one (10.30.0.14/32). Differing keys between them is a known misconfiguration class — compare the fingerprints above.

→ taranac-tacacs is listening on: 49,6049

→ Capturing on all interfaces, filter: host 10.30.0.14 and ((port 49 or port 6049) or icmp)

════════════════════════════════════════════════════════════════
  Capturing. Reproduce the problem on 10.30.0.14 now.
  Press Enter when you are done.
════════════════════════════════════════════════════════════════


→ Capture stopped.

Taranac diagnostic bundle
=========================

protocol : tacacs
device   : 10.30.0.14
window   : 2026-08-13T21:16:51Z .. 2026-08-13T21:17:45Z
packets  : 387

How this address is configured
------------------------------
  2 object(s) in Taranac cover this address:
    SW6-DEMO [10.30.0.14/32] host  <- most specific, used by the daemon
        enabled=True  protocol=on  key from device (fp 709ad291ecf8)
        groups: Cisco Devices, Demo Devices
    demo-network [10.30.0.0/24] subnet
        enabled=True  protocol=on  key from device (fp 6aabedfde070)
        groups: Demo Devices

Findings
--------
  [note]   2 network objects contain 10.30.0.14; the daemon uses the most specific one (10.30.0.14/32). Differing keys between them is a known misconfiguration class — compare the fingerprints above.
  [traffic] 387 packets captured.
  [ok]      Taranac reacted: 13 database record(s), 12769 bytes of daemon log.
  [decode]  Traffic decoded into readable text — see decoded.txt.

Contents
--------
  capture.pcap  — raw traffic (387 packets; open in Wireshark)
  config/daemon-config.masked.txt  — generated daemon config, secrets masked
  db-logs.json  — what Taranac recorded during the window (13 rows)
  decoded.txt  — the exchange decoded into readable text (no secret included)
  environment.txt  — versions, containers, capture settings
  logs/accounting_202608_2026-08-13.log  — daemon log output during the window
  logs/authentication_202608_2026-08-13.log  — daemon log output during the window
  logs/authorization_202608_2026-08-13.log  — daemon log output during the window
  logs/context/taranac-nac_radius_nac.log  — recent tail from a neighbouring service (context, not the window)
  logs/mavis_stderr.log  — daemon log output during the window
  logs/syslog.log  — daemon log output during the window
  preflight.json  — how the device is configured in Taranac
  tcpdump.stderr  — capture tool output (empty unless something went wrong)

Note on secrets: no shared secret is included. Secrets appear only as a
12-hex fingerprint. To decode TACACS+ payloads in Wireshark yourself, enter
the device's key locally — it is not, and must not be, in this archive.

Archive: /home/taranac-demo/taranac/diagnostics/taranac-diag-tacacs-10_30_0_14-20260813-211649.tar.gz  (24K)
Send this file to support.

Two things in that output are worth pausing on, because they are the tool doing its job before any packet was read.

The address is covered by two network objects with different keys. SW6-DEMO as a /32 host entry and demo-network as the surrounding /24, each carrying its own shared secret — fingerprints 709ad291ecf8 and 6aabedfde070. The daemon applies the most specific one, and the collector says which that is. If the switch had been configured with the other key, every request would have gone unanswered with nothing in any log to explain it; here the two fingerprints sitting side by side are the whole diagnosis.

The daemon is listening on 49,6049, not just the documented 49. That second port is read from the running process — which is the point of the check. Ports differ per protocol and per installation; the same three protocols on this platform report:

→ taranac-tacacs is listening on: 49,6049
→ taranac-radius is listening on: 1812,1813
→ taranac-nac is listening on: 1814,1815,18140,3799

A capture filtered on a port the daemon is not actually using comes back empty, and an empty capture is indistinguishable from “the device sent nothing”.

Capturing packets needs root. The collector checks before it does any work and tells you how to re-run rather than failing later inside tcpdump:

$ ./taranac diagnose --protocol tacacs --device 10.30.0.14
…
→ taranac-tacacs is listening on: 49,6049
✗ Capturing packets needs root, and this account cannot sudo without a password.
  Re-run as root:   sudo /home/taranac-demo/taranac/diagnose.sh --protocol tacacs --device 10.30.0.14

Note that the configuration check and the port check still ran — so even a refused collection tells you how the device is set up and where the daemon is listening. Simplest practice: run the collector with sudo from the start.

The summary opens with the capture parameters, then how the address is configured, then Findings. Each finding carries a label, and the vocabulary is small enough to learn once:

[EMPTY] — nothing arrived. The collector refuses to end an empty run with “send this to support”. Instead it lists what to check: that the problem really was reproduced while the collector was running, that the address is the one the device sends from, that the device points at this server on those ports, and that nothing on the path drops it. An archive with no traffic in it cannot show anything, and mailing one costs a round-trip.

[!] — packets arrived, Taranac recorded nothing. This is a different problem from “policy said no”, and the two look identical to whoever reports that it does not work. The collector separates two cases:

• No reply was sent either. The requests are being discarded before any policy runs. Usual causes: the device is not a configured client, the shared secret does not match, or it is sending a different protocol to that port.
• The daemon did answer — its replies are in the decode — but there is no log line and no database row. The exchange started and never completed: the device stopped mid-conversation, or gave up before sending credentials.

[KEY] — the secret does not match. A key mismatch is invisible from the outside, because the service does not raise an error; it simply stops answering. The decode is the only place it can be stated outright, and it is stated by trying every key that could apply to the address and reporting that none of them opens the body.

It is also the verdict that must not be reached carelessly, because it sends someone off to re-check keys that are fine. Two guards keep it honest. A header that cannot be TACACS+ — the version nibble, the packet type and the sequence number are each constrained by the standard — now stops the stream splitter instead of being handed to the decoder and diagnosed: a capture that started mid-connection, or a segment boundary landing inside a packet, is misframing, and misframing is reported as the end of the readable stream. And once any packet of a session has decoded with a key, a later failure in that same session is reported as a truncated or misassembled capture, not as a different secret — the key is demonstrably right, and saying otherwise would contradict evidence printed a few lines above.

The Contents section of the summary is generated from what is actually on disk, not from a fixed list — a bundle that promises a file it does not contain sends the reader looking for something that was never written.

Empty log slices are deleted rather than shipped, so a directory of zero-byte files never masquerades as collected data.

Three of them, for three specific reasons.

winbind is what verifies an Active Directory password: MS-CHAPv2 and PEAP for domain accounts go out through ntlm_auth. Taranac’s own log does name the failure class now — each NT_STATUS maps to a reason and a hint — but the machine account, the trust and which controller answered are winbind’s own business, and only its log holds them. It is in every bundle because it is the log nobody thinks to ask for. A directory that publishes the NT hash instead — FreeIPA — takes a branch that never touches winbind, and there the evidence is in the daemon’s own log.

The other two daemons are included because the wrong protocol is easy to collect. A device sending RADIUS while --protocol tacacs is running produces an empty capture and a bundle that says nothing arrived — when the truth, traffic on the other port, is one file away.

The backend is included because half of NAC never passes through a daemon at all. A disconnect, a VLAN change, a port bounce and the stale-session probe are all sent from the api container, and their outcome — CoA-ACK received, timeout waiting for a CoA response, no CoA secret — is recorded only there. The daemon’s log cannot mention a packet it never saw. This tail is sliced to the capture window; if nothing was logged in it, the last 200 lines are kept instead, so “was the worker even running?” still has an answer.

Decoding is not a convenience; for TACACS+ it is the difference between a usable bundle and an opaque one. A TACACS+ body is obfuscated on the wire, so a raw capture of your own traffic is unreadable to you. Taranac holds both the traffic and the key, so the decode happens inside the api container and the key stays there.

RADIUS and 802.1X are decoded for a different reason. Three things exist only in the capture and in no log anywhere:

• the attributes Taranac sent back;
• everything the device sent beyond the handful of values that get stored;
• requests discarded before they were processed — which is exactly the “traffic is clearly flowing and Taranac says nothing” case.

Attributes are named using the same dictionary the service itself generates, so entries from your own attribute dictionary are named correctly here too, rather than appearing as numbers in the one document where you most need to recognise them.

decoded.txt opens with a header naming the key actually in use, then prints the traffic grouped by session — not in stream order. Stream order interleaves the two directions, every request first and then every reply, which is unreadable for the one question being asked: what did the server answer to this request.

These four sessions are an excerpt from a longer capture — a successful login on a Cisco switch, the shell authorization that follows it, one command, and the accounting record:

TACACS+ decode for 10.30.0.14
packets: 35 seen, 14 session(s)
key in use: SW6-DEMO [10.30.0.14/32], key from device

----------------------------------------------------------------------

═══ session 0x4a87cff5 ═══

  [key: SW6-DEMO [10.30.0.14/32], key from device]

  21:17:23.399  seq 1  device  -> Taranac  AUTHEN
    AUTHEN START  action=LOGIN priv_lvl=1 type=ASCII service=LOGIN
      user='taranac' port='tty2' rem_addr='10.6.0.110'

  21:17:23.400  seq 2  Taranac -> device   AUTHEN
    AUTHEN REPLY  status=GETPASS flags=0x01
      server_msg='User Access Verification\n\nPassword: '

  21:17:23.406  seq 3  device  -> Taranac  AUTHEN   (+6ms)
    AUTHEN CONTINUE  user_msg=<REDACTED 21 bytes>

  21:17:23.482  seq 4  Taranac -> device   AUTHEN   (+77ms)
    AUTHEN REPLY  status=PASS flags=0x00

═══ session 0xa6113343 ═══

  21:17:23.520  seq 1  device  -> Taranac  AUTHOR
    AUTHOR REQUEST  method=TACACSPLUS priv_lvl=1 type=ASCII service=LOGIN
      user='taranac' port='tty2' rem_addr='10.6.0.110'
      arg: 'service=shell'
      arg: 'cmd*'

  21:17:23.521  seq 2  Taranac -> device   AUTHOR   (+2ms)
    AUTHOR RESPONSE  status=PASS_ADD
      arg: 'priv-lvl=15'

═══ session 0x748eea97 ═══

  21:17:26.893  seq 1  device  -> Taranac  AUTHOR
    AUTHOR REQUEST  method=NONE priv_lvl=1 type=ASCII service=NONE
      user='taranac' port='tty2' rem_addr='10.6.0.110'
      arg: 'service=shell'
      arg: 'cmd=show'
      arg: 'cmd-arg=version'
      arg: 'cmd-arg=<cr>'

  21:17:27.003  seq 2  Taranac -> device   AUTHOR   (+110ms)
    AUTHOR RESPONSE  status=PASS_ADD

═══ session 0xbadc07f2 ═══

  21:17:27.106  seq 1  device  -> Taranac  ACCT
    ACCT REQUEST  flags=STOP method=TACACSPLUS priv_lvl=15 service=LOGIN
      user='taranac' port='tty2' rem_addr='10.6.0.110'
      arg: 'task_id=100443'
      arg: 'timezone=UTC'
      arg: 'service=shell'
      arg: 'priv-lvl=1'
      arg: 'cmd=show version <cr>'

  21:17:27.108  seq 2  Taranac -> device   ACCT   (+2ms)
    ACCT REPLY  status=SUCCESS

Four things in that output are worth knowing how to read:

• The password is never printed — user_msg=<REDACTED 21 bytes>. You get its presence and its length, which is all a diagnosis needs.
• [key: …] names which candidate opened the session. It is printed only when there was more than one candidate — that is, only when there was a choice to get wrong. One session is decoded with one key throughout, so a coincidence cannot split a conversation across two of them.
• Sides are named by role, not by address. DNAT means a request is addressed to the host while the reply comes from the container, so printing raw addresses showed one exchange with two different “server” addresses and made replies read backwards. The addresses are still in capture.pcap if you need them.
• The gap annotations are diagnostic. (+77ms) on the reply is the whole authentication round trip. A reply that takes seconds is the signature of a backend timing out — a directory behind MAVIS, most often — and that wait is invisible in the daemon log, which records the outcome rather than the time spent waiting for it.

Also visible here: the trailing cmd-arg=<cr> that the standard requires to carry a separator. A device that sends a bare <cr> instead is what the Vendor Compatibility setting on the TACACS+ settings page exists for — and this decode is where you can see which of the two your device does.

A NAC capture opens with the list of attempts before printing any of them, so you can find the interesting one without reading the whole file. This is an excerpt from a capture that contained thirteen:

NAC decode for 10.30.0.12
packets: 40 seen, 40 decoded
key in use: SW2 [10.30.0.12/32], key from device

----------------------------------------------------------------------

Attempts in this capture:
  …
  7. '500000080000'  ·  PAP  ·  Access-Accept  ·  1 round(s), 0.0s
  8. 'lab\\anonymous'  ·  EAP-TTLS  ·  Access-Accept  ·  8 round(s), 0.4s
  9. 'lab\\anonymous'  ·  accounting Start  ·  Accounting-Response
  10. 'lab\\anonymous'  ·  accounting Interim-Update  ·  Accounting-Response

Attempt 7 is MAB — a MAC address as the user name, accepted into VLAN 30. Attempt 8 is a real 802.1X supplicant doing EAP-TTLS in eight rounds. Printed in full, with the rounds in the middle elided:

═══ attempt 8 — 'lab\\anonymous'  ·  EAP-TTLS  ·  Access-Accept  ·  8 round(s), 0.4s ═══

  21:26:33.434  id 85  device  -> Taranac  Access-Request
    User-Name = 'lab\\anonymous'
    Service-Type = Framed-User (2)
    Cisco-AVPair = 'service-type=Framed'
    Framed-MTU = 1500
    Called-Station-Id = '50-00-00-03-00-05'
    Calling-Station-Id = '50-00-00-08-00-00'
    Message-Authenticator = present (0x1449a1a3937532e4…)
    Cisco-AVPair = 'audit-session-id=0A1E000C000000EB3AE33919'
    Cisco-AVPair = 'method=dot1x'
    NAS-IP-Address = 10.30.0.12
    NAS-Port-Id = 'GigabitEthernet1/1'
    NAS-Port-Type = Ethernet (15)
    NAS-Port = 50101
    EAP-Message = Response/Identity 'lab\\anonymous'

  21:26:33.437  id 85  Taranac -> device   Access-Challenge   (+3ms)
    [key verified: SW2 [10.30.0.12/32], key from device]
    Message-Authenticator = present (0x31d4ed2aabe70fc5…)
    State = 0x1f5691e31f549c1c99b4938d922146c6
    EAP-Message = Request/EAP-TLS

  … six more challenge/response rounds …

  21:26:33.866  id 92  Taranac -> device   Access-Accept   (+215ms)
    [key verified: SW2 [10.30.0.12/32], key from device]
    Message-Authenticator = present (0x6dd57406341b8d22…)
    User-Name = 'lab\\anonymous'
    Tunnel-Type = 13
    Tunnel-Medium-Type = 6
    Tunnel-Private-Group-Id = '10'
    Class = '3bbb68c676524464ab5c319ad252d559'
    EAP-Message = Success

This is the part that exists nowhere else. The three Tunnel-* attributes in the Access-Accept are what actually put the endpoint into VLAN 10 — they are what Taranac sent, and no log records them. If a supplicant authenticates successfully and still lands in the wrong VLAN, this is the only place the answer is written down. The same applies to everything the switch sent beyond the few values that get stored: the port, the audit-session-id, the method, the MTU.

[key verified: …] on each reply is the RADIUS twin of the TACACS+ key check — the Message-Authenticator proves which shared secret was actually in use.

An 802.1X capture is not printed as one flat stream. It opens with a list of the attempts it found, then prints each one separately with a headline saying who, by what method, how it ended, how many rounds it took and how long.

Two of those endings cannot appear in any log, which is the whole argument for reading the wire:

• An attempt abandoned by the client. This is the shape of a supplicant refusing an untrusted server certificate: the user declines the warning, the supplicant goes quiet, and the server never learns the exchange ended — so there is nothing to log, because from the server’s side it is still waiting.
• A TLS alert, spelled out rather than left in hex — for example unknown_ca — the client does not trust the server certificate's CA, or certificate_expired.

Both are a certificate problem being reported to the operator as an authentication problem, which is where the hours go. See 802.1X with PEAP and PKI for the fix once the decode has named it.

Change of Authorization reverses the roles. Everywhere else the switch asks and Taranac answers; in a CoA Taranac is the client and the switch is the server, listening on its own CoA port. The decoder singles the six RFC 5176 codes out — Disconnect-Request, CoA-Request and their ACK/NAK replies — and takes the direction from the code itself, because both halves travel on the same port pair and no port heuristic can tell them apart. That is not cosmetic: the device’s address is what the key candidates and the pre/post-NAT deduplication are keyed on, so a CoA read the wrong way round came out labelled backwards, listed twice, and with the switch’s answer unverifiable.

The reply is verified against the CoA secret, not the authentication one. The two are frequently different — CoA resolves the device’s own CoA secret, then the oldest group carrying one, then the global nac.coa_secret, independently of the shared secret the same device authenticates with. Both are handed to the decoder, so a CoA-ACK verifies as [key verified: … CoA secret] instead of being reported as a key mismatch that does not exist.

Where the port comes from. The collector does not restate the inheritance chain: it asks the CoA sender itself to resolve port and secret, so the port in the bundle is the port a real CoA would have gone to. That is deliberately not the same code path as the effective CoA port the device form displays, and the two can disagree in one narrow case — a device with no CoA secret of its own, whose CoA port is left at the 3799 default, in a group that supplies both a secret and a port, follows the group’s port. Where the bundle and the form differ, the bundle’s number is the one the switch would actually have seen.

And when nothing answers at all, the ICMP in capture.pcap is the answer. A port-unreachable from the switch means nothing was listening where Taranac sent, which is a different fault from a switch that received the request and ignored it — and neither leaves a trace anywhere else. [note] findings cover the cases that never reached the wire: no CoA secret resolves for the device, its vendor entry declares the liveness probe unsupported, or it is not pinned to a Vendor Dictionary entry at all.

./taranac diagnose --protocol ncm --config "SW1 - show runn"

Config Tracker is the mirror image of AAA. There Taranac is the server and the packets are the evidence; here Taranac is the client, the session is encrypted, and a capture would prove nothing. The evidence lives inside the session: which driver was resolved, whether the port answered, what the SSH handshake agreed on, which prompt the device showed, and where exactly the attempt died.

So this collector is not a capture but a session recorder. It runs one real collection for one tracked configuration through the same engine a scheduled run uses, and writes a single readable report — one file, meant to be attached to a ticket whole.

Run it on the core, where the api container is: that container is the only place holding both the database and the master key, and the collector says so and stops if it is not running. The connection to the device is made from the core, by the same embedded collector Collection Preview uses — not from the standalone collector that may normally serve that source. Plan, driver and credential resolution are identical either way; reachability and the handshake describe the path from the core.

Nothing is persisted: no run, no result, no snapshot, no version, no report row. The file it hands you is the only artefact.

Exactly one device login is attempted — never the retry loop a scheduled run would use, because retrying a rejected credential is how an account gets locked out. The reachability and handshake layers are credential-free for the same reason: they open their own connection, offer nothing, and cannot contribute to a lockout.

The one thing it does change is outside Taranac: on a source using a just-in-time credential, the mint rotates the managed account’s password exactly as a real collection would, because a live login needs a committed password. If the mint does not succeed, the report says so, prints what it was about to do, and no device is contacted — a real run would have failed in the same place.

The order is the point rather than a formality. Each layer stands on its own and is reported even when the next one cannot run — the failure that prompted this tool, a driver name the library does not have, happens before a socket is ever opened, so a tool that only recorded sessions would have handed back an empty file and a shrug.

Layer 0 is where the unglamorous failures are. The driver is resolved the same way the collection resolves it — a device_type override on the source wins, then the recipe’s CLI driver, then the default, with _telnet appended for a telnet transport — and the result is checked against the drivers actually installed. When it is not one of them the report says so outright and lists the drivers that exist for that vendor, including the common case of a platform that ships an SSH driver and no telnet one.

Layer 2 earns its place on old equipment. When a modern client and an old switch share no key exchange or host-key algorithm, the failure text plus the list of what Taranac offered is the entire diagnosis — it names the algorithm to enable rather than leaving you with “the handshake failed”.

The terminal gets the verdict and the standing warnings; the report goes to a file, because a transcript scrolled past is a transcript nobody sends:

$ ./taranac diagnose --protocol ncm --config "SW1 - show runn"
Taranac diagnostic collector
  protocol : ncm
  config   : SW1 - show runn
  api      : taranac-api-dev

Connecting to the device once. Nothing is written to the configuration history.

verdict   : collection succeeded
  - credential values (passwords, keys, passphrases, tokens) — masked at capture
  - the device configuration itself — the transcript stops before the first command

Report: /opt/taranac/diagnostics/ncm-diagnostics-20260814-193529.txt  (214 lines)

The verdict is the first line of the report as well, and there are only a few of them: collection succeeded; failed at stage '<stage>' (<class>), naming the first stage that failed and how the engine classified it; failed (no stage reported it — see the transcript); this tracked config has no recipe — it can never collect, which is answered before any network; and the just-in-time mint failure above.

The file itself is fixed in shape, so two reports from different devices compare line by line: the header with the verdict and the standing “what this file does not contain” block, then the four layers in order, then the session transcript, then — when the run failed — the raw failure text from the transport library, which the engine’s own trace deliberately withholds. It ends with a machine-readable JSON summary of the same facts, for support to read without parsing prose.

In the transcript, < is what the device sent, > is what Taranac wrote, and # is the library’s own commentary — In disable_paging, Pattern found and friends, which are often the whole answer.

By default the recording is cut at the moment the session issues the first recipe command. That cut is exactly where the diagnostic value ends and the running-config — full of password hashes and community strings — would begin. The report says the cut happened and how to opt out.

--full-transcript opts out, and then the report contains the device’s configuration. The flag says so, the header of the report says so, and the finished run says so again on the terminal. Use it when the fault is in what the device answered to a command, and treat the resulting file like the configuration it contains.

Credential values never survive either way: every secret the plan resolved is masked before the transcript is buffered — the login password genuinely passes through this recording, and is genuinely removed, including the escaped spelling a password with punctuation produces. The username is deliberately kept: seeing which account was offered is half of reading a login dialogue, and it is not a secret. A very long session is truncated at 256 KB, and the truncation is announced in the file rather than left to look like the session simply ended.

./taranac diagnose --ha

One command, no roles, no flags to get wrong, run on every machine in the cluster — each database node and the witness — and send all the archives together.

On a witness this is one of the few things the wrapper will do at all. ./taranac refuses most commands there — a witness runs etcd and nothing else, so anything that reads the cluster’s records or drives its containers has nothing to work with — and diagnose --ha is on the short list it offers instead. The witness is a third of the cluster’s evidence.

That is the design rather than a formality. Almost every HA fault is a disagreement between hosts: a timeline that forked, a secret that differs, a ring one node knows and another does not, a version upgraded on one machine only. A disagreement cannot be seen from one side of it. Part of the picture is cluster-wide from any single host — etcd and Patroni’s REST are cluster APIs, and both are collected — but the rest of it (this node’s .env, its containers, its clock, its disk, what it can reach) exists only locally, which is exactly why the command runs everywhere.

This is a snapshot, not a capture. There is no window and nothing to reproduce: HA faults are state. It reads that state and changes nothing.

High availability is a licensed feature — see Licensing. The collector itself is not gated and is safe on any node at any time, including one that is not clustered at all, which it will say.

Every bundle prints one cluster-fingerprint line — a hash over the fingerprints of every value that must be identical cluster-wide: the cluster name, the database and replication passwords, the application secret key, the master key, the MFA API key and the Patroni REST password. Identical on every node means every cluster-wide secret agrees, and that question is closed at a glance instead of by diffing seven values. If it differs, the per-key block in config.txt names the one that drifted.

The bundle deliberately splits configuration into two classes: values that must be identical everywhere, and values that must differ (node name, node address, etcd name). Topology — addresses, member lists, node names — is not secret and is recorded verbatim; it is the entire point of the bundle.

That is when it gets run, so no probe is allowed to take the others down with it. Every check fails softly and records why it could not answer; one dead subsystem never costs the bundle the other eight. Specifically:

• .env is read from the host, never through a container;
• the database timeline is read straight off the data volume with a throwaway container, so a node whose Postgres will not start still reports the number that explains why;
• Patroni is asked locally first and then at every declared member, so a node with a dead stack still gets the cluster’s real state — reads need no credential, which is what makes this possible without handling the REST password;
• a witness has no .env in some layouts, so its identity is read from the etcd container instead. The same command works on the third machine, which is a third of the cluster.

The role is decided by construction before any daemon is asked anything: a name in the etcd ring with no Postgres behind it is a witness, and every database probe being empty there is correct rather than a fault.

Three database nodes and a witness; taranac-node-1 is stopped. This is the whole terminal output from taranac-node-3:

$ ./taranac diagnose --ha
Taranac diagnostic collector
  mode     : ha (cluster-state snapshot)
  node     : taranac-node-3 (10.20.0.43)
  cluster  : taranac
  looks like: database node

→ reading this node's containers
→ reading the declared configuration
→ reading config/cluster.yml
→ asking Patroni for the cluster's real state
→ asking etcd about the ring
→ reading this node's database state
→ probing reachability to every declared member
→ checking the etcd certificates
→ collecting component logs (last 400 lines each)

Taranac HA diagnostic bundle
============================

node       : taranac-node-3 (10.20.0.43)
hostname   : taranac-replica2
cluster    : taranac
role       : database node
version    : 1.2.6
collected  : 2026-08-14 19:33:44 UTC

cluster-fingerprint : 639906aca2bf
  ^ Compare this ONE string across every node's bundle. Identical on all of
    them means every cluster-wide secret agrees. If it differs, the per-key
    block in config.txt names the value that drifted.

declared members    : 4
  taranac-node-1         10.20.0.40       db
  witness                10.20.0.41       witness
  taranac-node-2         10.20.0.42       db
  taranac-node-3         10.20.0.43       db

taranac-db  : running=true  image=ghcr.io/taranaclabs/taranac/postgres-ha:1.2.6
              compose=/opt/taranac/docker-compose.yml,/opt/taranac/docker-compose.ha.yml
taranac-etcd: running=true
patroni view: taken from 127.0.0.1
control file: timeline 6, last checkpoint 0/18ECD230
              (a streaming replica's checkpoint timeline legitimately lags
               the cluster's until its next restartpoint — see patroni-cluster.json
               for the timeline Patroni actually has this node on)

Findings
--------
  [LAG]      Patroni reports member(s) that are not keeping up:
             taranac-node-1 is not streaming (state: stopped)
           A replica that is behind is not a spare: a failover to it loses whatever
           it has not replayed, and past 45s it also freezes daemon-config reloads.

  [QUORUM]   An EVEN number of etcd voting members (4) is declared. That
           tolerates no more failures than 4 minus 1 while adding one more
           thing that can fail. Prefer an odd count.

  [SLOT]     An INACTIVE replication slot is present (see postgres.txt, active = f).
           An inactive physical slot pins WAL on the primary indefinitely — the disk
           fills and the cluster stops. If the node it belonged to is gone for good,
           drop the slot; the eviction commands do this for you.

  [UNREACHABLE] From this node, these member ports do not answer:
             taranac-node-1:5432 taranac-node-1:8008 taranac-node-1:443
           Collect this bundle on the OTHER nodes too: if the reverse direction
           works, this is asymmetric filtering on the path (a firewall rule or a
           route), which is a different fault from a host being down. Remember that
           BOTH etcd ports matter — the ring forms over 2380, and a ring that cannot
           form makes every 2379 check look perfectly healthy.

Contents
--------
  certs.txt         — etcd CA and this node's leaf: fingerprint, dates, SANs
  config.txt        — declared topology verbatim; secrets as fingerprints only
  etcd-members.json — the ring as data: what every etcd verdict is computed from
  etcd.txt          — ring membership, health, raft term/index
  logs/api.log          — last 400 lines
  logs/db.log           — last 400 lines
  logs/etcd.log         — last 400 lines
  manifest.txt      — config/cluster.yml: what this node declares the cluster to be
  node.txt          — this host: containers, images, compose files, clock, disk
  patroni-cluster.json — the ring as Patroni sees it (roles, timelines, lag)
  patroni-config.json  — the cluster's Patroni configuration
  patroni-history.json — timeline history: where each timeline forked
  patroni-local.json   — what THIS node thinks it is
  patronictl-list.txt  — the same view from patronictl inside the container
  postgres.txt      — timeline, checkpoint, replication, slots, WAL size
  reachability.txt  — this node → every member, per port (run everywhere = the full matrix)

Note on secrets: no password, key or token is in this archive. Values that
must match across nodes appear only as 12-hex fingerprints, which prove
agreement or disagreement without carrying the secret.

This bundle describes ONE node. Send one from every node — each database
node and the witness — or the comparison that finds the fault cannot be made.

Archive: /opt/taranac/diagnostics/taranac-diag-ha-taranac-node-3-20260814-193231.tar.gz  (28K)

Run the SAME command on every other node — including the witness — and send
all the archives together. Most HA faults are disagreements between nodes,
and a disagreement cannot be seen from one side of it.

Four findings, and reading them together is the skill worth having.

Three of them are one fact seen from three angles. taranac-node-1 is stopped, so Patroni reports it not streaming ([LAG]), its replication slot on the primary is therefore inactive and pinning WAL ([SLOT]), and from this node its database, Patroni and api ports do not answer ([UNREACHABLE]). Note which ports are missing from that list: 2379 and 2380 answered, so node-1’s etcd is alive and voting while everything else on that host is down. That is a machine that is up with a stopped stack, not a machine that is gone — a distinction that decides whether you restart a service or re-clone a node.

One is not about the incident at all. [QUORUM] is a standing comment on the design: three database nodes plus a witness is four voting etcd members, and an even ring buys no extra failure tolerance while adding one more thing that can fail. It would appear on a perfectly healthy day, which is the point — the collector reports what it sees rather than only what is on fire.

And the [SLOT] finding is the one that becomes an outage if node-1 stays down and nobody looks: an inactive physical slot pins WAL on the primary indefinitely, the disk fills, and the cluster stops for a reason that looks nothing like the original fault.

That last distinction is repeated by the collector itself, and it is the reason the command is run everywhere: a clean bundle from one node proves only that this node sees nothing wrong.

Each bundle probes this node → every declared member, on five ports: 2379 etcd client, 2380 etcd peer, 5432 Postgres, 8008 Patroni REST and 443 the api. Run on every node, N bundles are the full N×N matrix, asymmetry included — and asymmetric reachability (A sees B, B does not see A) is a different fault from a host being down, with a different cure.

2380 is probed because it is the port the join tooling never checked: a ring whose members cannot reach each other’s peer port forms no quorum while every client-port check looks perfectly healthy. A witness legitimately has only the two etcd ports open, and the collector expects that rather than reporting it.

Beyond the summary, each archive carries the node’s containers and which compose files created them, the declared configuration, the cluster manifest, Patroni’s cluster view / history / config / local opinion, the etcd ring as a table and as data, the database’s timeline and replication and slots, the reachability matrix, the etcd certificates, the measured clock skew against each reachable peer, and the last 400 lines of the database, etcd and api container logs.

Those log tails are the services’ own words, included verbatim because a node that will not start usually explains itself only there. Everything Taranac itself collects is fingerprinted, but the logs are not ours to rewrite — give them a glance before sending the archive on if your policy requires it.

The rule is the same in all three collectors: a value that must only be compared is carried as a fingerprint, and a value that must not travel at all does not travel.

• No shared secret, ever — only a 12-character fingerprint of it, the same one Taranac uses for key-anomaly reports. Enough to prove a mismatch, useless to an interceptor. The CoA secret is treated identically: a NAC preflight prints the CoA port and the fingerprint of the secret that would be used, never the secret. In an HA bundle the same applies to every cluster-wide value: database and replication passwords, the application secret key, the master key, the MFA API key and the Patroni REST password.
• No passwords. The decoder reports that a password was present and how long it was, and redacts the value. In an NCM report every credential the plan resolved is masked before the transcript is buffered; the username is kept on purpose.
• The generated daemon config is masked before it is stored. It is the one place a plaintext key is legitimately written, so it is also the one file that must never be handed over as-is.
• The device’s configuration is not collected by an NCM run — the transcript stops before the first command. --full-transcript is the one deliberate exception, and it announces itself three times over.

Topology is not secret and is deliberately recorded verbatim in an HA bundle: addresses, member lists and node names are the entire point of it. Component log tails are the one place worth a glance before forwarding if your policy requires it — the etcd, database and api tails in an HA bundle, and the winbind, neighbouring-daemon and backend tails in a traffic bundle. They are the services’ own output rather than ours, and are included verbatim for exactly that reason.

If you want to decode TACACS+ payloads in Wireshark yourself, enter the device’s key in your own Wireshark — it is not, and must not be, in the archive.

• The capture is capped at --max-size (100 MB by default) and wraps at the cap: the tail survives, the beginning does not. The summary says so when it happens, because a silently half-missing trace is worse than a short one — the gap looks like the device went quiet. Re-run with a shorter window or a larger cap if the start of the exchange matters.
• A log may rotate mid-window. If the daemon trims or rotates a file during the capture, the byte offset no longer points at the right content; the bundle takes a bounded tail instead and the summary says which file it happened to.
• Two clocks, two zones. The capture window is UTC and tac_plus logs in UTC, while the FreeRADIUS log is in the container’s local time. Both clocks are recorded in environment.txt — a three-hour “gap” between the capture and the log is usually just this.
• Packets can appear twice. That is deliberate: capturing on any covers the physical interface and the Docker bridge, and the duplication is what proves the packet was forwarded into the container.
• One device, one protocol, per run. If you are not sure which protocol the device is using, the context logs in the first bundle will tell you where the traffic actually went.
• CoA is collected on a nac run only. The CoA port and secret are resolved by the 802.1X configuration check, so a tacacs or radius capture is filtered to the daemon’s own ports plus ICMP. Taranac’s CoA sender belongs to NAC sessions, so there is nothing on the other two runs for it to widen the filter for.

And for the other two:

• An NCM report describes one tracked configuration, collected from the core. A second tracked config on the same device is a second run — cheap, since each one is a single login.
• An HA bundle describes one node. Nothing in it can prove the cluster is healthy; that conclusion needs a bundle from every node, the witness included. The one cross-node measurement a single bundle makes honestly is the clock skew against each reachable peer.

• Troubleshooting & diagnostics — host-level problems: CPU, memory, failing or crash-looping services.
• Devices — device records, shared secrets and which protocols are enabled for each.
• TACACS+ · RADIUS · NAC authentication — what the decoded exchange should look like when it works.
• Logging — the records the bundle collects from the database side.
• High availability — what the cluster is supposed to look like, and the operations an --ha bundle is usually collected around.
• Config Tracker — tracked configurations, recipes and scrub sets and credentials: the settings an NCM report reads back to you in layer 0.