High availability

High availability (HA) runs Taranac as a cluster of nodes — as many as your HA license’s max_nodes allows — so that authentication keeps serving even when a node fails. Every node runs the full Taranac stack — UI, API, and the TACACS+/RADIUS/NAC daemons — on top of a replicated PostgreSQL database: one node is the read-write primary, the others are read-only replicas that continuously stream the primary’s data. The cluster is managed for you by Patroni (which supervises Postgres on each node) and etcd (the agreement layer that decides who is primary); you never operate those directly — ./taranac cluster init, add-witness and add-node are the whole interface, and each prints the single command to run on the machine being added. HA is a Pro feature, unlocked by an HA license that both allows ha and sets how many nodes you may run (max_nodes). The single most important property: because each daemon answers AAA/NAC requests from its own node-local database replica and in-memory cache, the authentication data plane never depends on the primary being up — point your network devices at every node as AAA servers and they ride straight through a failover, with only configuration writes pausing for a few seconds during promotion. This page is the operator guide: what HA gives you and when to use it, the architecture in plain terms, prerequisites, how to enable it, how to prove it survives a node loss with cluster verify --rehearse, the security you must apply, day-2 operations, rolling upgrades, and diagnostics. The shipped distribution bundle also carries the full operator runbook deploy/dist/HA.md with the exact long CLI transcripts; everything you need to decide and drive an HA deployment is on this page.

Use HA when an authentication outage is not acceptable — when a switch, VPN or firewall that authenticates against Taranac must keep working even if one Taranac host goes down for a reboot, a failure, or maintenance. A single-node install is perfectly fine for many deployments; HA is the answer when you need the AAA/NAC service to survive the loss of a node.

• Two or more nodes (up to your license’s max_nodes), each running the complete stack (UI, API, daemons, database).
• One primary (read-write), the rest read-only replicas streaming the primary’s data continuously.
• Authentication keeps serving on every node, including during a failover. The daemons answer from node-local data plus their caches, so the AAA/NAC data plane does not depend on the primary. Configure your NAS devices with every node as an AAA server.
• Automatic failover. If the primary dies, the most up-to-date replica is promoted and the cluster keeps running. Only configuration writes (UI edits, new accounting rows) pause for a few seconds during promotion, then resume on the new primary.

There are two kinds of host in a cluster:

• DB nodes each run the full Taranac stack: the api, the three daemons (radius / tacacs / nac), one etcd voting member, and the database container — Patroni supervising PostgreSQL 16. Exactly one DB node is the primary at any time.
• A witness host (for 2-node clusters — see below) runs only an etcd member. No Postgres, no Taranac stack.

Roles. Exactly one node is the primary (read-write); the rest are read-only replicas streaming the primary’s write-ahead log. The agreement on “who is primary” lives in etcd; Patroni reads it to elect a leader and to fail over. In Taranac the rule is simply leader == Postgres primary.

Read/write routing. The api and all three daemons read a cluster-wide DB_HOSTS list (every node’s Postgres endpoint). Writes are tagged target_session_attrs=read-write, so they always land on whichever node is currently primary, wherever it is. Daemon reads stay node-local — each daemon reads its own node’s replica and serves AAA from that plus in-memory caches. That is precisely why authentication never depends on the primary: a replica keeps answering TACACS+/RADIUS/NAC from local data even mid-failover, and only writes briefly pause.

Quorum and split-brain. etcd needs a majority of its members to act. Two DB nodes are an even split, so a network partition between them would deadlock — that is what the witness (a third voting member) is for. A partitioned old-primary that can no longer reach a majority self-demotes to read-only on its own, so two writers can never coexist. Three or four DB nodes already have a natural majority and do not need a dedicated witness.

The failover window. Patroni’s DCS timings are the stock production-sane set: ttl 30 s, loop_wait 10 s, retry_timeout 10 s. A hard loss of the primary therefore takes roughly the TTL to be noticed and acted on; a planned switchover is near-instant, because the leader hands over rather than expiring. use_pg_rewind is on, so a returning old primary re-attaches without a full re-clone.

A 2-node cluster cannot safely fail over on its own. If the network splits the two nodes, neither can tell whether the other is dead or merely unreachable, so neither may safely become primary — this is split-brain. The tiebreaker is a third etcd voting member, the witness, running on a separate failure domain — a different host or VM, ideally a different rack or site.

• The witness runs only etcd (docker-compose.witness.yml) — no Postgres, no Taranac stack. It is tiny; a management VM with ~100 MB and Docker is fine.
• It must be a separate host. witness-join.sh refuses to run on any machine that already has a taranac-db or taranac-api container: an arbiter co-located with a node it arbitrates is not a smaller deployment, it is a broken one.
• The tooling refuses a deploy whose declared etcd ring lists fewer than 3 members — both cluster init’s preflight and ha-convert.sh check ETCD_INITIAL_CLUSTER and stop. This is not optional.
• For three or more DB nodes a dedicated witness is not required — you already have a majority — but you still list every node’s etcd member.

See Living with the witness for what the witness host can and cannot do day to day.

Before you convert:

1. A running single node, installed normally, that already holds your data. It becomes node-1, the seed/primary. init refuses a host with no taranac_pg_data volume — the conversion adopts an existing, populated data directory; a fresh machine should be a join, not a convert.
2. The HA license uploaded on node-1 while it is still single-node (Settings → System → Licensing). Nodes inherit the license when they join; you never upload it per node.
3. All nodes on the same image version. Check with ./taranac version on each host — they must match before you convert or join.
4. The full topology decided up front — every node’s address, plus a witness host for a 2-node cluster. cluster init asks for all of it once and declares it for the whole cluster: the etcd ring must be complete before the conversion, because a member missing from it gets no certificate and can never join.
5. A private network between the nodes. Replication, etcd and Patroni traffic must travel a trusted, firewalled interconnect — never the public internet.
6. SSH to each machine you are adding, to paste one command per node. Nothing else travels by hand — no TLS files, no master key (see Cluster secrets).

Three commands, run on the node that already holds your data. The order is enforced by the mechanism, not by this page:

./taranac cluster init                          # declare the cluster, prepare the conversion
./taranac cluster add-witness --address <addr>  # the etcd arbiter — REQUIRED before nodes
./taranac cluster add-node    --address <addr>  # once per additional node

Each command says what it is about to do, checks that it actually happened, and names the step when it does not. Take a backup from the in-app Backup & Recovery tools first anyway: the conversion adopts your existing database in place.

init asks for the whole cluster at once, because the etcd ring has to be declared complete before anything converts:

Cluster name (identical on every node): taranac
This node's address, as the OTHER nodes route to it: 10.20.0.40
Witness address (the 3rd etcd arbiter, a SEPARATE host): 10.20.0.41
Another node — <address>, or <address>=<name> to choose its name (blank when done): 10.20.0.42
     PG_ALLOW_CIDR derived from the declared addresses: 10.20.0.0/24

It then shows the plan and waits for a y before writing anything. What it does: writes config/cluster.yml, writes the cluster block into .env — you no longer hand-edit either — generates the etcd CA and a certificate for every declared member, and stops.

Names are chosen here or not at all. Omit =<name> and one is derived (<cluster>-node-N, from the first free number, not from position). There is no rename afterwards: the name is written into the etcd ring, the certificates, the roster row and every node’s .env, so changing it means rebuilding that machine. The wizard says so before you accept a name it invented, and it refuses two nodes that would answer to the same name.

The conversion is deliberately not done at this point, and nothing has failed. Securing etcd is a chicken-and-egg: the witness must be a live etcd member before the ring can be pruned, but its certificate can only be signed by the CA that was created a moment ago. init does the half that can be done alone. Your database is still standalone and still serving.

./taranac cluster add-witness --address 10.20.0.41

This opens a one-shot join channel on port 9443 and prints the exact command to run on the witness host — with the broker’s own certificate fingerprint, so the witness verifies what it is talking to rather than trusting it on first sight:

./witness-join.sh     --broker 10.20.0.40:9443     --token 6b9ef07396…      --pin sha256:2fac71aa7c…

The address in that command is visible and editable — correct it if the witness reaches the primary by a different name or port. On the witness host the script checks the host is suitable, verifies the pin, takes the ring topology from the broker, generates its own private key and sends only a request for a certificate, then starts etcd and waits for the rest of the ring.

Back on the primary, the conversion now runs by itself: it forms quorum with the witness, drops the members that have not started yet, swaps the database to the Patroni image — which adopts your data in place, no re-initialisation — and comes up as the primary. Then it verifies both halves: that Postgres reports itself primary, and that the witness is a live voting member.

./taranac cluster add-node --address 10.20.0.42

Before anything grows, add-node runs a preflight against every existing member and prints what each one says about itself:

  Every member of this cluster, as IT reported itself (replicated heartbeat):

    NODE                     REPORTED     TOPOLOGY    PG_ALLOW_CIDR covers 10.20.0.42
    taranac-node-1           6s ago       reference   yes (10.20.0.0/24)

  etcd: 2 voting member(s) now, 3 after this add; 2 of 2 endpoint(s) healthy.

A member that has not reported inside the freshness window stops the add rather than being assumed healthy, and a member that does not yet know a node which already joined is named along with the command that fixes it. Nothing is rewritten on a live member during someone else’s join.

The preflight also compares every member’s declared etcd member list against the ring that is actually running, and stops the add when they differ. That drift used to be reported as harmless and is not: etcd compares the declared list against the live ring whenever a member starts on an empty data directory — a rebuilt disk, a --reinstall, a restored node — and refuses to start on any difference; and add-node hands its own list to the machine being added, so running it from a node that has fallen behind produces a join that ends in member count is unequal on the joining machine. The cure the preflight prints is ./taranac cluster sync, run on the node it named.

Then it prints the command for the machine being added, the same shape as the witness one:

./ha-join.sh     --broker 10.20.0.40:9443     --token KHzBlVCWRZ…      --pin sha256:e581f5e57c…

On the joining node, ha-join.sh is explicit about the one destructive thing it does. It counts what that node currently holds and says so before touching it:

  This node already holds a database of its own:

      devices                0
      administrators         0   (beyond the default account)
      AAA / NAC log records  0

  Joining DISCARDS all of it. A replica is cloned from the cluster's primary by
  Patroni; two separately seeded databases cannot be merged, so no variant of
  this operation keeps the local copy.

It then takes a backup of that node first and tells you where it is, unwraps the cluster’s key material, discards the local data directory so Patroni can clone, and brings the stack up on the HA overlay. Only when Patroni reports the node actually streaming does it redeem the token and report success to the primary.

./taranac cluster status

Healthy — one primary and 2 replicas close enough to be promoted. Losing any one
database node keeps writes.

Cluster: 3 node(s) / max 8 — Pro (HA)
Leader:  taranac-node-1
Worst replica lag: 0.0s
Promotion bar:     1.0 MB of WAL (read from Patroni)

NAME                    ROLE      SYNC         LAG       BEHIND     ADDRESS
taranac-node-1 *        primary   in_sync      -         -          10.20.0.40
taranac-node-2          replica   in_sync      0.0s      0 B        10.20.0.42
taranac-node-3          replica   in_sync      -         0 B        10.20.0.43

The verdict comes first, in words — “if a machine dies, do I keep working?” is the question people arrive with, and the table has always contained the answer without stating it. Everything below the headline is the evidence.

The same picture is in the UI under Settings → System → Cluster, with the edition, the leader, worst replica lag, and any pending join tokens (including their expiry):

The UI page shows the roster, roles, sync state and lag. The promotion bar, the per-node connection ceiling and the failover rehearsal are CLI-only — they are in cluster status and cluster verify.

status tells you what the cluster believes. Two commands go further, and both are worth running on a cluster you have just built:

• ./taranac cluster connectivity — proves every node can actually log in to every other.
• ./taranac cluster verify --rehearse — makes a failover actually happen, and puts the cluster back.

The HA control plane carries the cluster’s control state and its database. As of 1.0.7 it is encrypted by default (the convert secures etcd), and since 1.2.3 Patroni’s control API is authenticated unconditionally — but the firewall is the always-on baseline; TLS is not a substitute. Keep the cluster on a private, firewalled interconnect and apply these:

• Firewall the cluster ports to the private interconnect only — never expose them publicly:

  Port	What it carries
  etcd 2379 / 2380 (client + peer)	Cluster control state
  Postgres 5432	The database itself
  Patroni REST 8008	Cluster control — leadership, restart, reload
  9443	The join channel, open only while a join is running

  9443 is opened on the primary by add-node / add-witness, must be reachable from the machine being added, and closes when that join finishes or times out. Nothing listens on it between joins.

• Narrow PG_ALLOW_CIDR from any wide default to the cluster’s private subnet (e.g. 10.0.0.0/24). The wizard derives it from the addresses you declare and refuses a topology where a declared node falls outside it — check the derived value and tighten it if your addresses are spread wider than you want pg_hba to reach. An unset PG_ALLOW_CIDR means 0.0.0.0/0.

• The only ports that should face users and devices are the same as a single-node install: 443 (UI) and the AAA/NAC service ports you use.

TLS is on by default and provisioned by the tooling — you do not hand-edit it. The default convert encrypts both etcd channels: the peer channel (2380, cross-host Raft) via auto-TLS, and the client channel (2379, Patroni↔etcd) via a shared CA — one-way, meaning etcd presents a CA-signed cert that clients verify, with no client certs. Since 1.2.3 the CA is held by every node (delivered wrapped over the join channel), so losing the machine that created it can no longer leave a cluster that is unable to admit or replace a node.

Patroni’s REST API is authenticated, unconditionally. Port 8008 accepts POST /switchover, /failover, /restart and /reload — full control of leadership — and is published to the host. Every node therefore refuses to start without PATRONI_REST_PASSWORD; there is no flag and no fallback. The value is cluster-wide (a joining node adopts the cluster’s over its own), and PATRONI_REST_USERNAME defaults to patroni. Reads (GET /cluster, /patroni, /health) stay open on purpose — the backend’s topology polling, the slot reaper and the container healthcheck depend on them.

Several secrets are cluster-wide and must be identical on every node. A wrong or missing one fails silently — undecryptable stored secrets, MFA 401s, or invalid login sessions on that node. Since 1.2.3 none of them is carried by hand:

How the join channel is protected. The join token authenticates and encrypts under two separately derived keys, so the token itself never crosses the wire and the material it protects is never in plaintext at any layer. The joining machine pins the channel by a fingerprint printed on the primary’s console — you compare it there before pasting. That is not trust-on-first-use: an attacker who could answer on the broker’s address still cannot present the right certificate.

The one thing that never travels at all is a private key. Each machine generates its own and sends a certificate request; the answer is a signed certificate. That is what removed the scp of TLS material for database nodes and the witness alike.

status describes how the cluster is built. connectivity proves the legs that would matter after a failover. Neither has ever made one happen. verify is the command that does:

./taranac cluster verify              # a read — never disruptive
./taranac cluster verify --rehearse   # …and actually hand leadership over, and back

Without a flag it is a read. It prints the survivability verdict, the roles, the lag, the connectivity matrix and the max_connections each node is actually running with, and it exits non-zero if anything is not ready. On a standalone installation it exits 0 and says there is no cluster here to verify — a standalone install is not a broken cluster, and reporting a failure would imply a repair exists.

With --rehearse it hands leadership to a healthy replica, writes a real audited row on the new leader and times how long that took, waits for the old leader to come back as a streaming replica, and hands leadership back — then says in words what happened and where it left the cluster:

The cluster survived losing its leader.
  Writes resumed 6.9s after the handover began.
  taranac-node-1 left, came back, and took the lead again.
  You can lose any one DATABASE node and keep working.

It is a planned handover, never a kill. Patroni waits for the target to catch up, so nothing is lost. A kill proves less and costs more. The rehearsal’s own row lands in the audit trail (CLUSTER_FAILOVER_REHEARSED) on purpose — “we rehearsed a failover at 14:02 and writes came back in 7 s” is exactly what an audit trail is for.

When it refuses, and what gets past it.

• Not healthy → refused outright. There is nothing to learn from removing the leader of a cluster that has nothing to take over, and no flag makes a missing replica promotable.
• A ring older than 24 hours → refused, with --even-if-serving as the way past. Age is a proxy for “somebody may be depending on this”, never a traffic measurement, and the refusal says so. The ring’s formation moment is stamped into .env by cluster init and adopted by every node that joins, so a cluster rebuilt on an existing database is dated from the rebuild — that is 1.2.8’s fix for a gate that refused on a cluster built that morning because the surviving node keeps its roster row through a rebuild. A cluster built before that stamp existed has none, and the refusal then says plainly that what it measured was the oldest roster row instead.
• Nothing ever answers with a Python traceback. A refusal is a sentence, a non-zero exit and, where one exists, the command that fixes it. Because the rehearsal moves leadership, an interruption also says which machine it left leading and how to put it back.

Run it once, on the cluster you have just built, before it carries traffic. Two leader changes are a few seconds without writes each.

Until 1.2.8, cluster status opened with “losing any one database node keeps writes” judged from replay lag in seconds against a threshold of Taranac’s own — while Patroni promotes on WAL bytes against maximum_lag_on_failover. On an idle cluster both are zero, so they never disagreed on any stand. Under real writing they disagree constantly: across 150 samples on a loaded 1.9 GB stand the line claimed survivability every time while 46 of them had no replica inside Patroni’s bar — and within one second, status promised survival while a switchover answered “2.5 GB of WAL behind”.

The judgement is now made in bytes, against the threshold read from Patroni’s live configuration rather than copied — so raising Patroni’s setting is respected instead of being fought by a second, hidden number. status prints the bar it used:

Promotion bar:     1.0 MB of WAL (read from Patroni)

If Patroni’s threshold cannot be read, the verdict is unknown and says which question it failed to answer. A survivability line computed against an assumed number looks exactly like a real one, which is worse than none.

The LAG column still shows seconds, because that is what a person reads; nothing is concluded from it. The BEHIND column is the byte figure — the one that says what a forced promotion would actually throw away.

./taranac cluster verify tells you whether this applies to you. If it does not, skip it — a cluster built on 1.2.8 or later already has what it needs.

Why it is not simply raised for you. Under HA every node’s writes route to the one primary, so the primary carries the write side of the whole cluster on top of its own local reads. That needs max_connections = 300 (three API engines alone ask for 90 of Postgres’ default 100, before a single daemon connects). New clusters get it from the bundled Patroni configuration — but that is bootstrap config, which Patroni applies only when a cluster is first built, and max_connections is one of the parameters Patroni accepts only from the DCS, so a local override is ignored. An existing cluster therefore keeps 100 until somebody raises it deliberately, and upgrading does not do it.

What running out looks like: the daemons answer authentications with an error, a plain local psql is refused with FATAL: sorry, too many clients already — and cluster status goes on printing Healthy throughout, because nothing about replication is wrong.

verify asks each node what it is running with — over that node’s own connection, not read from any configuration file, because after edit-config the DCS already says 300 while every node keeps serving on 100 until it is restarted:

NAME                    ROLE      max_connections
taranac-node-1          primary   100
taranac-node-2          replica   100
taranac-node-3          replica   100

A node that is short makes verify exit non-zero and prints the procedure. It does not block a rehearsal: a cluster on the old ceiling can still lose a node and keep working, and that is a different question from whether the ceiling is the one this version budgets for. When the rehearsal passes and the ceiling is short, verify says so explicitly — “Exit 1 is for the connection ceiling above, not for the rehearsal — that passed.”

The procedure. Replicas first, the leader last — a standby whose max_connections is below the primary’s cannot start hot standby and stops replaying, so raising the leader first breaks replication until you catch up.

# 1 — raise it once, cluster-wide. On ANY DB node:
docker exec taranac-db patronictl -c /tmp/patroni.yml edit-config --force -p max_connections=300

# 2 — see who is waiting for a restart. Patroni marks the REPLICAS "Pending restart"
#     and leaves the leader unflagged until they are done, so this gives you the order:
docker exec taranac-db patronictl -c /tmp/patroni.yml list

# 3 — restart every REPLICA, one at a time:
docker exec taranac-db patronictl -c /tmp/patroni.yml restart <cluster> <replica> --force

# 4 — and only then the leader:
docker exec taranac-db patronictl -c /tmp/patroni.yml restart <cluster> <leader> --force

Each restart is a few seconds of that node’s Postgres; authentication keeps serving throughout from the daemons’ caches. Re-run ./taranac cluster verify when you are done — it is useful during the procedure too, because it names the nodes still to go and repeats the order.

etcd keeps every revision it has ever written unless told not to, and nothing tells it by default. Patroni rewrites the leader key every ten seconds or so, so on a default configuration the store grows forever — about 13 MB a day on an otherwise idle cluster, holding history nothing ever reads, until fsync latency starts costing leader elections. Measured on a customer’s cluster at 127 MB, at which point their leader was demoting itself and re-promoting every few minutes; and on our own lab at 27 MB two days after a clean rebuild. It is universal rather than a symptom of scale, which is why the default is not good enough to ship.

New clusters need nothing here. Both compose files that start an etcd — docker-compose.ha.yml on a DB node and docker-compose.witness.yml on the witness — now ask it to compact (ETCD_AUTO_COMPACTION_MODE=revision, ETCD_AUTO_COMPACTION_RETENTION=1000, roughly half an hour of history), which holds a genuinely busy cluster’s store around half a megabyte.

This is for three other cases: a cluster older than that setting, a cluster whose leader keeps flapping, and a store that has already grown.

# Check what each machine is actually carrying — ask each one separately:
docker exec taranac-etcd         etcdctl endpoint status -w table   # on each DB node
docker exec taranac-witness-etcd etcdctl endpoint status -w table   # on the witness

1. Make sure the setting is present. ./taranac update refreshes the compose files, so it arrives with any update — but it only takes effect when that member’s etcd is recreated, which the update does not do on its own: ./taranac up -d --force-recreate --no-deps etcd. Recreating an etcd member is a quorum event: one machine at a time, waiting for etcdctl endpoint health --cluster to be healthy in between.
2. Throw the old history away. Compaction bounds only new growth. Discard the rest once, from any one member — it is cluster-wide: etcdctl endpoint status --write-out=fields | grep Revision, then etcdctl compact <that revision>.
3. Give the disk back, on every machine separately. Compaction frees space inside etcd’s file; defragmentation shrinks the file. Run docker exec taranac-etcd etcdctl defrag on every DB node one at a time, and docker exec taranac-witness-etcd etcdctl defrag on the witness.

The tooling is HA-aware: once a node is converted (its .env carries TARANAC_HA=1, or DB_HOSTS is set), ./taranac and ./taranac update auto-detect HA and merge the Patroni overlay for you — so ps, logs, restart, down, up and update all act on the Patroni-managed database correctly. You do not pass compose files by hand. If the overlay file is ever missing on an HA node, a container-starting command refuses loudly rather than silently start a second writable Postgres — restore it with ./taranac update.

./taranac cluster status     # verdict, nodes, roles, replication lag/health
./taranac ps                 # local container health on this node

cluster status opens with the survivability verdict in words, then shows the edition (Pro (HA) vs Community), node count against max_nodes, the leader, the worst replica lag, the promotion bar, and a per-node table — NAME / ROLE / SYNC / LAG / BEHIND / ADDRESS. Read the SYNC column:

• in_sync — streaming and within Patroni’s promotion bar (maximum_lag_on_failover, 1 MB by default). Healthy; this node could take over.
• behind — streaming or converging, but past the bar in bytes, so Patroni would not promote it. That much redundancy is already gone.
• unreachable — an active roster node that is neither streaming here nor tracked by Patroni: broken replication, or a down node.

A replica that fell off entirely has no pg_stat_replication row, so it cannot be the worst in a MAX over one. That used to print Worst replica lag: 0.0s while a replica sat 1.5 GB behind and disconnected. The line now names those replicas instead of skipping them:

Worst replica lag: UNKNOWN — taranac-node-3 is not streaming, so it has no lag to read
                   (worst among the replicas that could be measured: 0.2s)

The cluster also raises alerts in the UI (and via syslog) when a role changes or a node falls out of sync — you do not have to poll.

• With a witness (or 3+ nodes): failover is automatic. When the primary stops renewing its etcd lease (the DCS TTL, 30 s), the most up-to-date replica is promoted, the other replicas re-point to it, and the api reconnects to the new primary.
• Authentication keeps serving on every node throughout the window. Only writes pause briefly (config edits, accounting) and then resume on the new primary — at Taranac’s volume that is a handful of buffered records, retried automatically, not lost.
• The old node rejoins automatically as a replica when it comes back (Patroni uses pg_rewind, so it is fast — no full re-clone). You do nothing.
• A 2-node cluster without a witness has no automatic failover — promotion is manual. This is exactly why the witness is mandatory.

status tells you what each node believes. connectivity proves it:

./taranac cluster connectivity
./taranac cluster connectivity --format tsv    # one line per ordered pair, for scripts

SOURCE \ TARGET   taranac-node-1  taranac-node-2  taranac-node-3
taranac-node-1 *  —               ok              ok
taranac-node-2    ok              —               ok
taranac-node-3    ok              ok              —

All 6 ordered pair(s) proven by a fresh replicator login.

Every node attempts a real replication login to every other and publishes the result. A port that answers is not proof — a refusal arrives only after the handshake, so only a completed login counts. A verdict older than the freshness window shows as stale and never counts as proof; unknown means that node never published one. The command exits non-zero unless every ordered pair is proven by a fresh login, so ./taranac cluster connectivity || echo "NOT fully verified" is a usable gate.

Run it after any firewall change, and before you trust a cluster you have just built. The join itself does not wait on this matrix — it proves the new node is replicating from the current primary, which is the leg that must work today; the matrix proves the legs that matter after a failover.

One command on any node currently holding the primary role, within your license’s max_nodes — the same add-node you used to build the cluster:

./taranac cluster add-node --address <the new node's address>

If that node was declared at init, nothing needs changing. If it was not, the command extends .env to name it and tells you plainly that this node’s containers still carry the previous value until they are recreated — preflight will keep reporting the difference until you run ./taranac up -d at your convenience.

For an address that is already a member — it died, it was rebuilt, its join never finished — --reinstall evicts and re-adds it from scratch:

./taranac cluster add-node --address <addr> --reinstall

It is destructive on that node: it is evicted, its database is discarded, and Patroni clones it again, so the cluster runs one replica short until that finishes. It is refused for the current leader (switch over first). On a member that is not streaming — the case the flag exists for — you are asked to confirm, and --reinstall is what skips the question. add-witness accepts it too.

A node that has fallen behind what the cluster reports about itself is fixed on that node, deliberately:

./taranac cluster sync

Run it on the peer the preflight named as stale — not on the primary. It reconciles two things: this node’s config/cluster.yml against the copy the cluster publishes (offering to take the cluster’s copy, after backing yours up), and then DB_HOSTS / ETCD_HOSTS / ETCD_INITIAL_CLUSTER in .env. That last one matters more than it looks: an out-of-date etcd member list stops a rebuilt member from starting at all and produces joins that cannot succeed.

The wizard deliberately never rewrites a live member’s configuration from the primary — a surprise write during someone else’s join is the worst possible moment for it — which is why sync is the one command meant to be run somewhere else.

When the two manifest copies disagree the wizard stops and names both; it never picks one for you, because that file carries your hand-written skip: section. The two cures are:

./taranac cluster manifest-publish < config/cluster.yml   # this node's copy is right
./taranac cluster sync                                    # the cluster's copy is right

./taranac cluster switchover                                # planned; Patroni picks the target
./taranac cluster switchover --to <name>                    # planned, to a named node
./taranac cluster switchover --to <name> --force --confirm FAILOVER   # when the leader is gone

A plain switchover is a planned handover — Patroni waits for the target to catch up, so nothing is lost. It is refused if the leader is unreachable (use --force then) or if the named candidate is not streaming — and the refusal names the candidate’s actual state, because “it is behind” would send you off waiting for a catch-up that is not happening.

--force is a Patroni failover: it promotes a node that may not have replayed everything, and it discards whatever that node is missing. Patroni’s /failover needs a named candidate, and so does Taranac for a second reason — a promotion that throws writes away must name the node it is throwing them away for — so --force requires --to and the typed --confirm FAILOVER. It states the cost concretely first: how many bytes of WAL the target is behind, and that those writes will be lost. If the leader is alive and the target is caught up it is refused outright, because a plain switchover would do the job at no cost.

The candidate check uses the same bar Patroni does (maximum_lag_on_failover, 1 MB by default), so Taranac’s refusal predicts Patroni’s rather than arguing with it — and raising Patroni’s setting is not fought by a second, hidden threshold.

Removal is two decisions, and they are separate commands on separate machines.

Step 1 — evict it from the cluster. On a surviving DB node (conventionally the primary):

./taranac cluster remove-node --node taranac-node-2      # or --address <addr>

It removes the etcd member, decommissions the roster row, cleans up the replication slot, updates the manifest, and prunes the node from this machine’s .env. Both safety questions are answered from the running cluster — the live etcd member list and Patroni’s own view of who leads — never from this node’s .env, which is exactly the thing that can be out of date.

You do not repeat this on the other survivors. They read cluster membership from etcd, which this command changed once, cluster-wide. (That is the one thing the older ha-deconvert.sh --decommission-dead path could not do, and why it asked you to re-run it everywhere.)

It refuses only one thing outright: evicting the current leader — hand leadership over first. Dropping below three etcd voters is not refused; it is described and you are asked, because the two shapes are different hazards:

• one DB node left, plus the witness — not split-brain (there is nothing to split). It is a 2-member ring that needs both for quorum, so if the witness then goes down the surviving database node is demoted to read-only — worse than not running HA at all. If the goal is a single server, finish with ./ha-deconvert.sh.
• two DB nodes left with no arbiter — the classic split-brain shape: on a partition neither side holds a majority, so neither may promote and failover stops working.

Step 2 — decide what that machine becomes. It is still running, and it still holds a full copy of everything. Two honest options: wipe it, or keep it as an independent install — on that machine:

./taranac cluster detach --confirm FORK-MY-DATA

detach refuses while that machine is still a cluster member — Step 1 has to happen first, or the detached node’s etcd keeps trying to rejoin the ring and its Postgres stays a follower of a primary that still counts on it. It takes a backup, stops the HA stack, clears the HA markers from .env, brings the node up on the base compose so the standalone Postgres adopts the data directory in place, and waits until the database actually accepts writes before reporting success.

Until a removed node is wiped or detached it may still show as a cluster:phantom_node alert — that is the cue the physical half is still pending. Once its stack is stopped, Patroni expires its membership and the leader auto-drops its replication slot; never drop a replication slot by hand.

Removing a node that is dead (won’t come back). Run Step 1 on a survivor — ./taranac cluster remove-node --node <name> does the etcd removal and the roster soft-delete, and no other machine needs anything. You simply cannot do Step 2 on a machine that is gone. If that dead node ever returns, wipe its pg_data and etcd_data volumes before joining it again — or just run ./taranac cluster add-node --address <addr> --reinstall against it, which handles both.

remove-node is the command to use. The pieces underneath are still first-class subcommands, because they are cures the tooling prints and a cure you cannot run is not a cure:

./taranac cluster decommission --id <node-id>       # soft-delete the roster row by id
./taranac cluster decommission --name <node-name>   # or by name (active nodes only)

This marks the node decommissioned (its id is retained forever so old records still resolve). It does not stop the node — that replica is still streaming and still holds a full copy of the database and MASTER_KEY. Finish it physically on that machine, both compose files so etcd stops too, and -v to remove volumes:

docker compose --env-file .env -f docker-compose.yml -f docker-compose.ha.yml down -v
# then securely wipe anything left of its Postgres data and its MASTER_KEY (.env)

To go all the way back to a plain standalone install, on the surviving primary, after the other nodes are removed and torn down:

./ha-deconvert.sh          # back up first; add --yes to skip the prompt

It stops the HA overlay, strips the HA markers from .env, and brings the stack up on the base compose so the standalone Postgres image adopts the data dir in place (the reverse of the conversion), drops the orphan replication slots, and removes the orphan etcd volume. It refuses to run on a replica or while replicas are still streaming — remove and tear those down first. Afterwards the witness host is no longer needed (docker compose --env-file .env -f docker-compose.witness.yml down -v).

Taranac’s scheduled jobs each declare where they may run: leader-only (once per cluster) or per-node (on every node). The scheduler evaluates leadership live at each tick, so a leader-only job follows a failover without a restart, and the declaration is inert on a single node. Settings → System → Workers names the node you are looking at and its role, and marks a leader-only job “Leader-only — idle” when you open the page on a replica.

Since 1.2.8 the LDAP directory sync runs on the leader only. On an N-node cluster it was previously N concurrent binds against your directory every interval, N audit rows, racing upserts on the same users and groups, and N copies of any failure email. The NCM collection sweep, the update feed, the OUI refresh and the cluster reconciles are leader-only for the same reason.

Two jobs are not, and it is worth knowing what that means for you:

• Audit-log retention (cleanup_audit_logs, daily 03:00) runs on every node. The delete itself is harmless — writes route to the leader, so the second node finds nothing left to remove — but each node writes its own AUDIT_RETENTION_PURGE row, so an N-node cluster records N purge events a night, of which typically one carries a non-zero count. If you have enabled archive before delete, the JSONL file is written to a node-local path, so the nightly archive lands on whichever node won the race and the others write nothing — the archive set ends up split across hosts with nothing recording which host holds which night. If you archive, collect from every node, or turn archiving off and rely on syslog forwarding for long-term retention.
• The syslog forwarder (syslog_forward, every 20 s) runs on every node, and the watermark it advances (syslog_forward_cursor) is a single shared row on the primary. Each node independently reads the same cursor, selects the same window of AAA rows and ships it, so a SIEM bound to an N-node cluster can receive up to N copies of every TACACS+/RADIUS/NAC record, and each destination’s last_delivered_at / dropped_record_count counters are inflated by the same factor. Nothing is lost — the cursor can only end up at a position some node has actually forwarded past — but de-duplicate at the SIEM, or bind syslog destinations expecting the multiplier.

Keep every node on the same image version, and upgrade the cluster one node at a time: update the replicas first, the primary last. Schema migrations run only once, automatically, on the primary. Take a backup from the in-app Backup & Recovery tools before upgrading.

./taranac update is HA-aware: on a converted node it brings the database back up under the Patroni overlay automatically, and it refreshes the HA overlay, tooling and runbook files so they stay in step with the new images.

On the primary it now refuses, before anything is written, until every other member already reports the target version. The old guard printed “update the replicas first” and carried on — a warning about order is powerless the moment an earlier step lies about having succeeded, which is exactly how one cluster ended up split across versions on 15 August. The question it asks is about versions, not about the order you happened to take:

• A replica is never blocked — replicas first is the correct order.
• A member that is merely slow to report is waited for (a node publishes every 30 s; the wait is bounded, and a member whose report is older than 90 s counts as unconfirmed).
• A member that cannot be confirmed at all counts as not ready. An out-of-date answer is not an answer.
• The refusal names each node and the version it is running.

If you are certain — for instance because you are repairing exactly that state — the override is ./taranac update --allow-out-of-order.

Why it matters: recreating the primary’s container restarts its Postgres and triggers a failover. If a node with the old code takes the lead, the upgraded machine becomes a replica running new code — and a replica does not migrate, while the old leader has no migration to make. Both then wait forever, and switching leadership back does not help, because the schema check is exact in both directions.

During a rolling upgrade a replica’s API waits for a schema head only the leader can create. It now says which way the two revisions differ, that only the leader can create the schema, and that the order is replicas first and the primary last — rather than restating two revision hashes at you.

If an upgrade stops half-way it names the phase it stopped in — replacing files, writing the version pin, or pulling images — and reports the bundle’s version, so ”✓ Bundle files updated” is no longer the last thing you see before a node that did not actually move.

The witness unpacks the same bundle as a Taranac node — that is how it gets witness-join.sh — so the wrapper and the updater are both present on it. Since 1.2.7 both know what kind of host they are on.

./taranac update on the witness is safe, and is how it stays in step. It refreshes the bundle’s files only — witness-join.sh, the witness compose file, diagnose.sh, the HA runbook — and stops there: it neither reads nor edits the witness .env, and it starts nothing. The witness runs one pinned upstream etcd image that a Taranac release never changes, so there is nothing to pull, and it deliberately does not restart etcd on its own — recreating the arbiter is a quorum event, not housekeeping.

If a release does change docker-compose.witness.yml, the update says so and prints the single command that applies it:

docker compose --env-file .env -f docker-compose.witness.yml up -d

Apply it deliberately, while the cluster is healthy, and never at the same time as a DB node.

What works on a witness host:

./taranac ps                                     # the arbiter's etcd, not a product stack
./taranac logs -f etcd
./taranac up -d                                  # bring the arbiter back
./taranac diagnose --ha                          # snapshot for support
docker exec taranac-witness-etcd etcdctl endpoint health
./witness-join.sh --broker … --token … --pin …   # re-point it at a cluster

What does not, and why: every cluster … subcommand, plus unlock, reset-password, create-admin, installation-id and reset. A witness has no database and no API, so those commands have nothing to act on — it now says exactly that and lists what you can run there, instead of failing with a missing-container message that describes the symptom and hides the reason. ./taranac down, stop and kill do work, and warn you first that stopping the arbiter costs the cluster a vote.

Witnesses joined before 1.2.7 are recognised from the .env that witness-join.sh writes and need no change; new ones also carry an explicit TARANAC_ROLE=witness line — keep it if you ever hand-edit that file.

Start at ./taranac cluster status; most issues are visible there and in ./taranac ps. When status looks wrong, Patroni’s own view is authoritative:

docker exec taranac-db patronictl -c /tmp/patroni.yml list
# role (Leader/Replica), state (running/streaming), and lag per member

A few quick checks:

• Which node is primary: docker exec taranac-db psql -U taranac -d taranac -tAc "select not pg_is_in_recovery()" — t = this node is the primary, f = replica.
• Replication health (run on the primary): select application_name, state, sync_state, replay_lag from pg_stat_replication; — one row per streaming replica.
• etcd / quorum: docker exec taranac-etcd etcdctl endpoint health and etcdctl member list.
• “Config won’t apply on a node”: grep the api log for Readiness gate BLOCKED — that node is out of sync and its daemon-config reloads are frozen on the last-good version while it catches up (auth keeps serving from caches throughout).

For anything you cannot resolve from one machine, collect the HA bundle on every node, the witness included, and send all of them:

./taranac diagnose --ha

Most HA faults are disagreements between nodes, and a disagreement cannot be seen from one side of it. It is safe on a node that will not start. See Diagnostic bundles.

Taranac raises these stateful alerts in the UI (and via syslog); they auto-resolve when the condition clears. See Alerts for how the alert lifecycle works.

Two 1.2.8 fixes worth knowing behind that last daemon alert. A daemon that lost its database used to latch: under load on the stand, TACACS+ refused 163 of 180 logins while the container stayed healthy, port 49 answered and cluster status printed Healthy — and the node stayed poisoned twenty minutes after the load stopped. It retries now, and says so. And an ordinary restart no longer pages anyone: a daemon that has just come up refuses while its caches load (7–20 s on a loaded node), so the alert waits out a 30 s grace. A real outage lasts longer than the grace and still alerts at once.

Scheduled-backup alerts are per node, too — a backup fails because this node ran out of space or lost a mount, and a shared identity used to let a failover launder that: a node with a full disk had its alert cleared by a successful backup on the node that took leadership, while nothing on it had changed.

Everything ./taranac cluster accepts, and where it runs. ./taranac cluster with no arguments prints the same list; each subcommand takes --help. On a witness or a standalone collector host every one of them is refused, with the reason.

Building the cluster — host-side, on a node holding the primary role (they rewrite that host’s .env and recreate containers):

Operating it — inside the node’s api container, so RBAC and the audit log apply; any DB node will do:

Two scripts sit outside the cluster family and are still the right tool for their jobs: ./ha-deconvert.sh (shrink this node back to standalone), and ./witness-join.sh / ./ha-join.sh, which you never type yourself — the primary prints them, complete with token and pin, for the machine being added.

You do not operate Patroni and etcd directly, but knowing the moving parts makes diagnostics obvious. The full architecture — failover internals, quorum proofs, the readiness gate, replication-slot lifecycle and the reasoning behind every default — is in the design spec docs/guide/ha.md in the project repository, and the self-contained operator runbook (with the exact long CLI transcripts, the TLS-scheme migrations and the durable gotchas) ships in the distribution bundle as deploy/dist/HA.md.

• Licensing — the Pro license that unlocks HA and sets max_nodes
• Backup & upgrades — take a backup before converting or upgrading; backups complement HA, they do not replace it
• Alerts — how the cluster:*, service:* and license:* alerts fire and resolve
• Audit & syslog — where cluster events surface for your SIEM
• Diagnostic bundles — ./taranac diagnose --ha, the other half of an HA investigation
• Installation — deploy the single node that becomes node-1
• Taranac on GitHub