High availability

High availability (HA) runs Taranac as a cluster of 2 to 4 nodes so that authentication keeps serving even when a node fails. Every node runs the full Taranac stack — UI, API, and the TACACS+/RADIUS/NAC daemons — on top of a replicated PostgreSQL database: one node is the read-write primary, the others are read-only replicas that continuously stream the primary’s data. The cluster is managed for you by Patroni (which supervises Postgres on each node) and etcd (the agreement layer that decides who is primary); you never operate those directly — the bundled ha-convert.sh / ha-join.sh scripts and the ./taranac cluster commands are the whole interface. HA is a Pro feature, unlocked by an HA license that both allows ha and sets how many nodes you may run (max_nodes). The single most important property: because each daemon answers AAA/NAC requests from its own node-local database replica and in-memory cache, the authentication data plane never depends on the primary being up — point your network devices at every node as AAA servers and they ride straight through a failover, with only configuration writes pausing for a few seconds during promotion. This page is the operator guide: what HA gives you and when to use it, the architecture in plain terms, prerequisites, how to enable it, the security you must apply, day-2 operations (status, failover, adding and removing nodes, shrinking back to standalone), rolling upgrades, and diagnostics. The shipped distribution bundle also carries the full operator runbook deploy/dist/HA.md with the exact long CLI transcripts; everything you need to decide and drive an HA deployment is on this page.

Use HA when an authentication outage is not acceptable — when a switch, VPN or firewall that authenticates against Taranac must keep working even if one Taranac host goes down for a reboot, a failure, or maintenance. A single-node install is perfectly fine for many deployments; HA is the answer when you need the AAA/NAC service to survive the loss of a node.

• 2–4 nodes, each running the complete stack (UI, API, daemons, database).
• One primary (read-write), the rest read-only replicas streaming the primary’s data continuously.
• Authentication keeps serving on every node, including during a failover. The daemons answer from node-local data plus their caches, so the AAA/NAC data plane does not depend on the primary. Configure your NAS devices with every node as an AAA server.
• Automatic failover. If the primary dies, the most up-to-date replica is promoted and the cluster keeps running. Only configuration writes (UI edits, new accounting rows) pause for a few seconds during promotion, then resume on the new primary.

There are two kinds of host in a cluster:

• DB nodes (2–4 of them) each run the full Taranac stack: the api, the three daemons (radius / tacacs / nac), one etcd voting member, and the database container — Patroni supervising PostgreSQL 16. Exactly one DB node is the primary at any time.
• A witness host (for 2-node clusters — see below) runs only an etcd member. No Postgres, no Taranac stack.

Roles. Exactly one node is the primary (read-write); the rest are read-only replicas streaming the primary’s write-ahead log. The agreement on “who is primary” lives in etcd; Patroni reads it to elect a leader and to fail over. In Taranac the rule is simply leader == Postgres primary.

Read/write routing. The api and all three daemons read a cluster-wide DB_HOSTS list (every node’s Postgres endpoint). Writes are tagged target_session_attrs=read-write, so they always land on whichever node is currently primary, wherever it is. Daemon reads stay node-local — each daemon reads its own node’s replica and serves AAA from that plus in-memory caches. That is precisely why authentication never depends on the primary: a replica keeps answering TACACS+/RADIUS/NAC from local data even mid-failover, and only writes briefly pause.

Quorum and split-brain. etcd needs a majority of its members to act. Two DB nodes are an even split, so a network partition between them would deadlock — that is what the witness (a third voting member) is for. A partitioned old-primary that can no longer reach a majority self-demotes to read-only on its own, so two writers can never coexist. Three or four DB nodes already have a natural majority and do not need a dedicated witness.

A 2-node cluster cannot safely fail over on its own. If the network splits the two nodes, neither can tell whether the other is dead or merely unreachable, so neither may safely become primary — this is split-brain. The tiebreaker is a third etcd voting member, the witness, running on a separate failure domain — a different host or VM, ideally a different rack or site.

• The witness runs only etcd (docker-compose.witness.yml) — no Postgres, no Taranac stack. It is tiny; a management VM is fine.
• It must be a separate host. Co-locating it with a DB node is useless: losing that host loses a DB node and the tiebreaker together.
• The tooling refuses a 2-node deploy that does not list at least 3 etcd members. This is not optional.
• For 3 or 4 DB nodes a dedicated witness is not required — you already have a majority — but you still list every node’s etcd member.

Before you convert:

1. A running single node, installed normally, that already holds your data. It becomes node-1, the seed/primary.
2. The HA license uploaded on node-1 while it is still single-node (Settings → System → Licensing). Nodes inherit the license when they join; you never upload it per node.
3. All nodes on the same image version. Check with ./taranac version on each host — they must match before you convert or join.
4. The full topology decided up front — every node’s name and address, plus a witness host for a 2-node cluster. You write the same cluster-wide block into every node’s .env, so plan it once.
5. A private network between the nodes. Replication, etcd and Patroni traffic must travel a trusted, firewalled interconnect — never the public internet.
6. The out-of-band secret set in hand. Several secrets are cluster-wide and must be delivered out-of-band, identically on every node (see Cluster secrets).

Conversion is sequential and the order matters: node-1 must be the live primary before anything joins, or an empty joining node could win the bootstrap race and clone over your real data. The tooling guards against this, but follow the order anyway. Secure etcd is the default (peer auto-TLS + one-way client TLS), which makes the convert two-phase — the witness must be a live etcd member before the real convert, but its TLS cert can only be signed by the CA the convert generates, a chicken-and-egg the two phases resolve. Take a backup from the in-app Backup & Recovery tools first; the adoption is in-place.

Put the same cluster-wide block in .env on every DB node and the witness. Set the shared values identically everywhere:

# Identical on every node + the witness
TARANAC_CLUSTER_NAME=taranac
DB_HOSTS=<node-1-addr>:5432,<node-2-addr>:5432
PG_ALLOW_CIDR=<your private cluster subnet, e.g. 10.0.0.0/24>
# https:// member URLs are the secure default; ha-convert --prepare rewrites these
# for you on the DB nodes, and the witness .env.witness.example ships them https.
ETCD_INITIAL_CLUSTER=taranac-node-1=https://<node-1-addr>:2380,taranac-node-2=https://<node-2-addr>:2380,witness=https://<witness-addr>:2380
ETCD_HOSTS=<node-1-addr>:2379,<node-2-addr>:2379,<witness-addr>:2379
ETCD_INITIAL_CLUSTER_STATE=new
POSTGRES_REPLICATION_PASSWORD=<the same value node-1 already generated>
POSTGRES_PASSWORD=<the same value node-1 already has>

POSTGRES_PASSWORD and POSTGRES_REPLICATION_PASSWORD on a joining node must match node-1’s — they are the cluster’s shared app and replication credentials, generated once on node-1 at install time. DB_HOSTS lists every DB node’s Postgres endpoint (never the witness — it runs no Postgres); it is the one knob the api and all three daemons read so writes always reach whichever node is currently primary.

Then set the per-node values, different on each host:

# Unique per node
TARANAC_NODE_NAME=taranac-node-1     # this node's unique cluster name
NODE_ADDRESS=<this node's address the others route to>
ETCD_NAME=taranac-node-1             # this node's etcd member name (= its key in ETCD_INITIAL_CLUSTER)

The witness .env needs only the etcd block and ships secure by default in .env.witness.example. On the witness host, cp .env.witness.example .env and set TARANAC_CLUSTER_NAME, the same ETCD_INITIAL_CLUSTER (https:// member URLs) with ETCD_INITIAL_CLUSTER_STATE=new, ETCD_NAME=witness, and NODE_ADDRESS=<the witness host's address>.

Step 1 — Prepare, on node-1 (your existing single node). --prepare generates the etcd CA and node-1’s cert, enables peer auto-TLS, rewrites the member URLs to https://, pre-issues a cert for every other member into config/cluster-secrets/etcd/<member>/, writes the https knobs, and then stops (node-1 stays standalone; your data is untouched):

./ha-convert.sh --prepare --node-name taranac-node-1 --node-address <node-1-addr>

Step 2 — Bring up the witness, on the witness host. With its .env filled in, copy its pre-issued cert out-of-band and start etcd only:

# From node-1, out-of-band (scp/USB): copy config/cluster-secrets/etcd/witness/
#   into the witness host's config/etcd-tls/
docker compose --env-file .env -f docker-compose.witness.yml up -d

Step 3 — Continue, back on node-1. Now that the witness is a live etcd member, --continue does the real convert: it swaps node-1’s database to the Patroni image, which adopts your existing data in place (no re-initialisation, no data loss), forms quorum with the witness, and brings node-1 up as the primary that initialises the cluster:

./ha-convert.sh --continue

The script waits until node-1 reports itself as primary, then prints the next commands. This must finish before you join any node. If it times out, etcd most likely has no quorum — confirm the witness from Step 2 is up and reachable.

Step 4 — Join each additional node. For each new node, do three things. On node-1, issue a join token:

./taranac cluster join-token --name taranac-node-2 --address <node-2-addr>

On the new node, first write its .env without starting a standalone stack — a started standalone would initialise the data volume and block the clone — and copy its pre-issued etcd cert out-of-band:

./install.sh --no-start
# From node-1, out-of-band: copy config/cluster-secrets/etcd/taranac-node-2/
#   into this node's config/etcd-tls/

Then, with its .env filled per the block above, supply MASTER_KEY out-of-band and run the join with --primary (it fetches the cluster config from the seed and auto-follows the https scheme):

MASTER_KEY=<the cluster master key> \
  ./ha-join.sh --primary <node-1-addr> \
    --node-name taranac-node-2 --node-address <node-2-addr> --join-token <secret>

Patroni clones the new node from the primary (its own basebackup + streaming — there is no manual pg_basebackup), the node comes up as a read-only replica, and the script redeems the token so the node registers in the roster. Repeat for any further nodes, up to your license’s max_nodes.

Step 5 — Verify. Every node should appear, with exactly one primary and the rest streaming:

./taranac cluster status

The HA control plane carries the cluster’s control state and its database. As of 1.0.7 it is encrypted by default (the convert secures etcd), but the firewall is the always-on baseline — TLS is not a substitute. Keep the cluster on a private, firewalled interconnect and apply these:

• Firewall the cluster ports to the private interconnect only — never expose them publicly:

  Port	What it carries
  etcd 2379 / 2380 (client + peer)	Cluster control state
  Postgres 5432	The database itself
  Patroni REST 8008	Cluster control

• Narrow PG_ALLOW_CIDR from any wide default to the cluster’s private subnet (e.g. 10.0.0.0/24). Do not leave it open.

• The only ports that should face users and devices are the same as a single-node install: 443 (UI) and the AAA/NAC service ports you use.

TLS is on by default and provisioned by the tooling — you do not hand-edit it. The default convert encrypts both etcd channels: the peer channel (2380, cross-host Raft) via auto-TLS, and the client channel (2379, Patroni↔etcd) via a shared CA — one-way, meaning etcd presents a CA-signed cert that clients verify, with no client certs, and the CA private key never leaves the seed node. The witness ships secure by default too; you copy each member’s pre-issued cert into place out-of-band as shown in the conversion steps.

Five secrets are cluster-wide and must be identical on every node. A wrong or missing one fails silently (undecryptable stored secrets, MFA 401s, or invalid login sessions on that node), so provision them deliberately — never as a replicated DB row:

The seed’s ha-convert assembles the out-of-band set into config/cluster-secrets/; carry it to each node exactly as you carry MASTER_KEY. The join API deliberately returns DB creds, SECRET_KEY and TARANAC_MFA_API_KEY, but never MASTER_KEY, the mfa enckey, or the etcd CA key.

The tooling is HA-aware: once a node is converted (its .env has DB_HOSTS set), ./taranac and ./taranac update auto-detect HA and merge the Patroni overlay for you — so ps, logs, restart, down, up and update all act on the Patroni-managed database correctly. You do not pass compose files by hand. If the overlay file is ever missing on an HA node, a container-starting command refuses loudly rather than silently start a second writable Postgres — restore it with ./taranac update.

./taranac cluster status     # nodes, roles, replication lag/health
./taranac ps                 # local container health on this node

cluster status shows the edition (Pro (HA) vs Community), node count against max_nodes, the leader, and a per-node table — NAME / ROLE / SYNC / LAG / ADDRESS. Read the SYNC column:

• in_sync — streaming, lag < 10 s. Healthy.
• behind — streaming but lag ≥ 10 s. Watch it; a node that stays behind past 45 s trips the readiness gate and raises a cluster:readiness alert.
• unreachable — a roster node not streaming at all: broken replication or a down node.

The cluster also raises alerts in the UI (and via syslog) when a role changes or a node falls out of sync — you do not have to poll.

• With a witness (or 3+ nodes): failover is automatic. When the primary stops renewing its etcd lease (≈ the DCS TTL, ~30 s), the most up-to-date replica is promoted, the other replicas re-point to it, and the api reconnects to the new primary.
• Authentication keeps serving on every node throughout the window. Only writes pause briefly (config edits, accounting) and then resume on the new primary — at Taranac’s volume that is a handful of buffered records, retried automatically, not lost.
• The old node rejoins automatically as a replica when it comes back (Patroni uses pg_rewind, so it is fast — no full re-clone). You do nothing.
• A 2-node cluster without a witness has no automatic failover — promotion is manual. This is exactly why the witness is mandatory.

Same as the join step above, within your license’s max_nodes: fill the new node’s .env, issue a token on the current primary (./taranac cluster join-token --name … --address …), copy the pre-issued etcd cert out-of-band, and run ha-join.sh --primary on the new node.

Removing a node is a logical removal followed by a physical teardown — both are required.

Step 1 — soft-delete from the roster (run from any node):

./taranac cluster decommission --id <node-id>       # node-id from `./taranac cluster status`
./taranac cluster decommission --name <node-name>   # or by name (active nodes only)

This marks the node decommissioned (its id is retained forever so old records still resolve). It does not stop the node — that replica is still streaming and still holds a full copy of the database and MASTER_KEY.

Step 2 — physically tear it down on that node (both compose files so etcd stops too, and -v to remove volumes):

docker compose --env-file .env -f docker-compose.yml -f docker-compose.ha.yml down -v
# then securely wipe anything left of its Postgres data and its MASTER_KEY (.env)

Until Step 2 completes, the still-streaming node shows up as a cluster:phantom_node alert — that is your cue physical teardown is pending. Once the node’s stack is stopped, Patroni expires its membership and the leader auto-drops its replication slot; never drop a replication slot by hand.

Removing a node that is dead (won’t come back). If the node is permanently gone you can’t run Step 2 on it, and its etcd voting membership must be removed or it drags quorum math down forever. Run ha-deconvert.sh on a surviving node:

./ha-deconvert.sh --decommission-dead --node-name node-2 --node-address <node-2-addr>

It soft-deletes the roster row, removes the dead etcd member, and prunes it from this node’s DB_HOSTS / ETCD_HOSTS / ETCD_INITIAL_CLUSTER. The roster and etcd changes are cluster-wide, but the .env prune is per-node — re-run the same command on every other surviving node, then restart them so etcd picks up the shrunk membership.

To go all the way back to a plain standalone install, on the surviving primary, after the other nodes are decommissioned and torn down:

./ha-deconvert.sh          # back up first; add --yes to skip the prompt

It stops the HA overlay, strips the HA markers from .env, and brings the stack up on the base compose so the standalone Postgres image adopts the data dir in place (the reverse of the conversion), drops the orphan replication slots, and removes the orphan etcd volume. It refuses to run on a replica or while replicas are still streaming — decommission and tear those down first. Afterwards the witness host is no longer needed.

Keep every node on the same image version, and upgrade the cluster one node at a time: update the replicas first, the primary last. Schema migrations run only once, automatically, on the primary. Take a backup from the in-app Backup & Recovery tools before upgrading.

./taranac update is HA-aware: on a converted node it brings the database back up under the Patroni overlay automatically, and it detects whether this node is the primary — if so, it warns you to update the replicas first and this node last, so you can abort if you started on the primary by mistake. Recreating a node’s container restarts its Postgres, and on the primary that triggers a failover — which is exactly why you do the primary last. The update also refreshes the HA overlay, tooling and runbook files so they stay in step with the new images.

Start at ./taranac cluster status; most issues are visible there and in ./taranac ps. When status looks wrong, Patroni’s own view is authoritative:

docker exec taranac-db patronictl -c /tmp/patroni.yml list
# role (Leader/Replica), state (running/streaming), and lag per member

A few quick checks:

• Which node is primary: docker exec taranac-db psql -U taranac -d taranac -tAc "select not pg_is_in_recovery()" — t = this node is the primary, f = replica.
• Replication health (run on the primary): select application_name, state, sync_state, replay_lag from pg_stat_replication; — one row per streaming replica.
• etcd / quorum: docker exec taranac-etcd etcdctl endpoint health and etcdctl member list.
• “Config won’t apply on a node”: grep the api log for Readiness gate BLOCKED — that node is out of sync and its daemon-config reloads are frozen on the last-good version while it catches up (auth keeps serving from caches throughout).

Taranac raises these stateful alerts in the UI (and via syslog); they auto-resolve when the condition clears. See Alerts for how the alert lifecycle works.

You do not operate Patroni and etcd directly, but knowing the moving parts makes diagnostics obvious. The full architecture — failover internals, quorum proofs, the readiness gate, replication-slot lifecycle and the reasoning behind every default — is in the design spec docs/guide/ha.md in the project repository, and the self-contained operator runbook (with the exact long CLI transcripts) ships in the distribution bundle as deploy/dist/HA.md.

• Licensing — the Pro license that unlocks HA and sets max_nodes
• Backup & upgrades — take a backup before converting or upgrading; backups complement HA, they do not replace it
• Alerts — how the cluster:* and license:* alerts fire and resolve
• Audit & syslog — where cluster events surface for your SIEM
• Installation — deploy the single node that becomes node-1
• Taranac on GitHub