Last time nearly everything came from somebody else: four public issues, a customer’s Calix switch, the first HA cluster built outside our lab. That did not turn out to be a one-off week. It is now simply how the product finds out what is wrong with it.

Two more public issues are closed here, and they have the same shape: in both, the product held a belief about itself that had never been examined. It believed only Active Directory could verify an MS-CHAPv2 password. It believed a synchronized directory user was a person who should be able to log in. Neither was true, and nobody inside the project noticed, because you do not test a belief you have not articulated.

The third piece — the login banner — is the same shape again, and the second instance in two releases: a field that saved, rendered in a preview, and reached no device.

MS-CHAPv2 without a domain, and without winbind

This is public issue #13, and answering it meant separating two facts the product had been treating as one.

What Active Directory alone can do is hold a machine account: one winbindd, one join, one domain, and everything that follows from being a member of it. What MS-CHAPv2 actually needs is much smaller — the account’s NT hash. FreeIPA publishes one. ipaNTHash is created for every user, byte for byte MD4(UTF-16LE(password)), even on a realm where ipa-adtrust-install was never run.

So a directory that hands out the hash now gets a mechanism of its own. Device administration over RADIUS and 802.1X PEAP-MSCHAPv2 both take that branch, and which branch a login takes is decided by the user’s own directory — a node may hold a domain join and any number of hash-readable directories at the same time, and neither borrows the other’s verifier. Unlike the domain, this second mechanism is not exclusive: there can be as many of them as you configure.

The hash is read on each login and stored nowhere. Caching it would put a password-equivalent in a daemon’s memory and go stale the moment the person changes their password.

One operator action is always required, and the product hands it over rather than describing it. ipaNTHash sits in an explicit exclusion list even in the administrator’s own ACI, so an ACI granting your bind account read access has to be added. The MS-CHAPv2 tab generates that LDIF with your bind DN and search base already filled in, and Check directory reports whether the hash is readable — asked with a filter that establishes the fact without pulling a single hash out of the directory.

The part worth dwelling on is what happens when the ACI is missing. FreeIPA does not refuse the request; it answers without the attribute. A product that counted that as a wrong password would feed the brute-force guard and send you hunting a credential problem that does not exist — so a missing ACI is reported as exactly that, and never as a bad password.

The console is a decision you make once, for the whole installation

Public issue #14, and the reporter was describing a hole rather than an inconvenience.

A directory synchronized for the sake of NAC and AAA policy brings in policy subjects, not console operators. A Domain Users mirror is thousands of accounts, and until this release every one of them was a valid credential pair against the management interface. The only lever that existed was is_active = false, which is not a console switch at all: the TACACS+ user cache reads the same column, so switching it off removes the person from AAA — from precisely the thing the directory was synchronized for.

A group now carries a three-state console mode — inherit, allow, deny — resolved against a new installation-wide default in Settings → Authentication. inherit is the state that makes the default retroactive, and that is the whole design: a group nobody has touched is already sitting in it, so one setting closes the hundred groups already synced and every group the directory adds tomorrow. A per-group flag could never have done that. Sync assigns policy only to groups it creates, so a default set today would never have reached the ninety already there.

Administrators are never shut out — is_admins wins over everything — and the shipped default is allow, so upgrading changes nothing about who can log in today. What the control closes is the console, not the account: password reset, the certificate download link and MFA enrolment keep working, and the BYOD captive portal is deliberately untouched, because taking somebody’s console away must not take their network with it.

Two details that only matter once you are living with it. The gate is re-checked at the second factor, not only at the password — a deny that arrives between the two steps has to land, or the window between them is the hole. And the group table reports the resolved answer rather than the stored one, marking which groups inherited it rather than chose it; assigning an RBAC role to a group that cannot reach the console is meaningless, so the picker goes read-only there while keeping the existing assignment visible. “We did not delete it” is something you have to be able to see.

A banner, at the scope a banner can actually live at

TACACS+ now carries a pre-login banner and a post-login message of the day, set at the realm, at a device group, or at a single device. A banner resolves device → oldest device group → global, which is the same chain and the same ordering the TACACS+ key already uses, and the two banners inherit independently of each other.

They are two settings rather than one because they ride in different packets and therefore reach different people: the pre-login banner is seen by anyone who can open a session, the MOTD only by somebody who authenticated. Fold them together and you have promised one audience something you are showing the other.

This replaces the MOTD field that used to sit on a TACACS+ profile, which could never have worked. A profile is chosen during authorization; a login banner is carried in the authentication reply, and the daemon accepts a banner at realm and device scope only. The field promised a per-role banner the protocol cannot express — and promised it convincingly, because the profile’s config preview rendered it while the deployed configuration carried it nowhere.

That is the second field of exactly this kind in two releases. 1.2.7 fixed the TACACS+ password prompt that was saved, rendered in the preview and never once put on the wire. Neither was found by a test, because both behaved correctly everywhere a test looks — the value was stored, read back, and displayed. What was missing was the last step, and the only instrument that sees it is somebody’s device.

The probe that changed what it was meant to observe

The stale-session worker used to ask a NAS whether a silent session still existed. First with a Disconnect-Request, which disconnected the users it was checking. Then with an identification-only CoA-Request, on the theory that a CoA carrying no changing attribute is a question.

It is not a question. Measured on a customer’s Huawei CloudEngine S5735, with debugging on the switch and captures at both ends: that packet, sent to a session that is definitely alive, is answered CoA-ACK in 30–90 ms and the RADIUS-assigned VLAN is removed from the session. The user stays authenticated and drops to the port PVID. RFC 5176 defines no read-only packet — a CoA-Request carries the session’s authorization, and a NAS with replace semantics applies an empty one literally. The probe was changing the state it existed to observe, on exactly the users who were still online.

So it is gone, and nothing in Taranac now closes a session on the strength of a NAS’s answer. Liveness comes from accounting, in two windows, the way ISE, ClearPass and PacketFence read it. A session that sends interim updates ends after silence past the stale threshold. A session where only the Start ever arrived cannot be judged by silence at all — there was never anything to fall silent — so it is held by age, and that default moves from 72 to 120 hours, with a warning that names the cure rather than the symptom: periodic interim accounting on the switch.

Disabling that job also persists now. It used to be a pause held in the scheduler’s memory and lost on the next container restart, which is a trap laid specifically for the person who had paused it to stop a switch from being probed.

Two VLANs, and which one you are looking at

Public issue #11 started with a MAB’d phone that tags its own voice VLAN — nothing is assigned to it, and the VLAN appears only inside NAS-Port-Id. Pulling that thread found that sessions and authentication records carried a single VLAN field that meant one thing or the other by turns, because accounting overwrote the assignment. Nobody could tell which of the two a row was showing, and the one case genuinely worth seeing — the two disagreeing, from a CoA that never landed, a hybrid port, or a port somebody moved by hand — was invisible by construction.

They are two columns with two meanings now. On a session, a disagreement is flagged, because a session is where the port settled after authorization. On an authentication record it is not, because that is the VLAN the frame arrived in before any assignment was applied, and flagging it there would light an icon on every ordinary dynamic-VLAN authentication in the log. Filtering and sorting cover either half, so searching for a VLAN finds the sessions in it whichever way the number got there.

AAA, in the places your devices found

Most of this section exists because somebody’s configuration did not survive contact with ours.

• A TACACS+ AV-pair name may contain # again. The directive-name whitelist shipped in 1.2.2 banned it outright, which locks an operator out of Cisco’s numbered families — inacl#1, outacl#1, route#1, interface-config#1. Templates saved before the ban could still be read but no longer re-saved: a 422 on a profile nobody had edited. # is the daemon’s comment character, but only at the front of a token, so that is where it stays banned.
• A raw AV-pair sharing a service with the template now reaches the device. Each raw pair was given a block of its own, and the daemon stops evaluating at the first permit it reaches — so whenever the profile’s template already defined that service, the raw pair sat behind a block that always terminated. Login worked, authorization succeeded, and the attribute simply was not on the wire, with nothing logged, because nothing had failed.
• A RADIUS template created after the daemon started never reached it. The handler read templates once at process start into a cache nothing refreshed. The Access-Accept went out carrying nothing at all while the UI and the AAA log both showed the profile as applied. Raw attributes survived that path, which is why they looked like the only part that worked.
• A seeded TACACS+ profile spells out its command default. NULL was read as permit by the renderer and as deny by the form, and a form submits what it shows — so a seeded profile authorized every command until somebody opened it to change something unrelated and pressed Save, from which moment the device refused every command while login still succeeded.
• A new profile permits commands by default, which is a behaviour change and a deliberate one. A profile that works the moment it is attached and is narrowed afterwards beats one that authenticates and then refuses everything until you find the field — that failure reads as “TACACS+ is broken”, not as “this profile is strict”.
• A cleared attribute stays cleared, and a mandatory one may not be left empty. Clearing a field is how a defaulted attribute is switched off, so the form no longer discards the empty value on the way to the server; and an attribute the template calls mandatory is refused on save instead of quietly shipping a profile that does not do what its template says.
• A single-valued RADIUS attribute can no longer be saved with every alternative at once. A Palo Alto role could go out as superuser, superreader and deviceadmin in one Access-Accept, with nothing to tell the device which to believe. The template had already declared the answer; nothing read it.
• The capture decoder stops blaming your shared secret for its own misframing. A capture that starts mid-connection, or a segment boundary inside a packet, left the stream splitter misaligned, and the bogus packet it then produced was reported as the device is using a different shared secret — the most common real fault, asserted to somebody whose keys are fine. On a lab capture, 18 of 19 packets decoded with the device’s key and the 19th produced that verdict.

LDAP, where a sync reported nothing you could use

• Sync filters are visible without opening Advanced. A customer reported that AD user filters “do not work in any configuration”. They did — both sync filters were rendered inside the Advanced branch, so in the ordinary view they were off screen entirely while the login-time filter was not, and the preview answered about something other than what the form was showing.
• A group that brings in nobody now says why. “Members: 0” meant three different things — an empty group, a membership attribute we do not read, or an entry that could not be read at all — and looked like one. A customer spent an exchange hunting a fault in their own tree over the second case.
• The directory sync runs on the leader, not on every node at once. On an N-node cluster it was N concurrent binds against the customer’s directory every interval, N audit rows, racing upserts on the same users and groups, and N copies of any failure email.

High availability: a rehearsal instead of a description

cluster status and cluster connectivity describe how a cluster is built — connectivity says as much itself, proving the legs that would matter after a failover. Nobody had ever made one happen.

cluster verify does. Without a flag it is a read: the survivability verdict, roles, lag, the connectivity matrix, and a non-zero exit if anything is not ready. With --rehearse it hands leadership to a healthy replica, writes a real audited row on the new leader and times it, waits for the old leader to come back as a streaming replica, and hands leadership back — then says in words what happened and where it left the cluster. It is a planned handover, never a kill: Patroni waits for the target to catch up, so nothing is lost. On a standalone installation it exits 0 rather than complaining about a cluster that was never meant to exist.

The verdict it opens with had to be rebuilt first. cluster status used to say “losing any one database node keeps writes” on the strength of replay lag in seconds, against a threshold of ours, while Patroni decides promotion on WAL bytes, against its own. On an idle cluster both are zero, so the two never disagreed on any stand we had. On a 1.9 GB stand under real writing they disagree constantly: across 150 samples the line claimed survivability every time while 46 of them had no replica inside Patroni’s bar — and live, within one second, status promised survival while a switchover answered “2.5 GB of WAL behind”. The threshold is read from Patroni’s live configuration now rather than copied into ours, so raising it is respected.

The rest of the cluster work is a list of things a real deployment does and a lab does not:

• Every database node now runs with a budgeted connection ceiling. The product set max_connections nowhere, so Postgres’ own hundred was in force — and under HA every node’s writes route to one leader, so the primary carries the whole cluster’s write side on top of its own reads. Three API engines alone ask for 90 of those 100 before a single daemon connects. New clusters get the budget on their own; existing ones cannot, because the value lives in a section Patroni applies only when a cluster is first built. So verify reports the ceiling each node is actually running with — asked over its own connection, not read out of a file — and exits non-zero while a node is short, naming the procedure.
• etcd compacts its history. Patroni rewrites the leader key every ten seconds or so, and etcd’s default keeps every revision of it: about 13 MB a day on an otherwise idle cluster, holding history nothing ever reads, until fsync latency starts costing leader elections. Measured on a customer’s cluster at 127 MB, and on our own lab at 27 MB two days after a clean rebuild — universal rather than a symptom of scale, which is exactly why the default was not good enough to ship.
• The primary refuses to update while any member is behind, before anything is written. The old guard printed “update the replicas first” and carried on. One cluster met every hole in that at once on 15 August: a replica upgrade had died silently, so the operator read the warning and concluded, from everything he could see, that he had already done it. A warning about order is powerless the moment an earlier step lies about having succeeded.
• An upgrade that stops half-way says so, names the phase it stopped in — replacing files, writing the version pin, or pulling images — and reports the bundle’s version rather than a guess. Two defects that hid each other, and together they are why a customer was told to upgrade, watched it appear to work, and stayed where they were.
• A replica waiting for the leader’s schema says the node is fine. It used to restate two revision hashes, which is the one fact the operator was already looking at. It now says which way the two differ, that only the leader can create that schema, and that the order is replicas first and the primary last.
• The backup archive carried less than it claimed and said nothing about it. config/etcd-ca is the only protection against losing a machine and never being able to admit or replace a node again — and it has been missing from every archive since the feature shipped. The keys are written readable only by the operator’s own account, and the component that collects them is opportunistic, so there was neither a missing entry nor an alert. Both halves are fixed: the files are made readable, and an incomplete archive now reports itself.
• A scheduled-backup alert is per node. It fails because this node ran out of space or lost a mount, so a shared identity let a failover launder a fault: a node with a full disk had its alert cleared by a successful backup on the node that took leadership, and nothing on it had changed.
• A daemon that loses the database retries, and stops being silent. Under load on the stand, TACACS+ refused 163 of 180 logins while the container stayed healthy, port 49 answered and cluster status printed Healthy — with 17 rows in the AAA journal for 180 attempts. A child that failed to get a pool within ten seconds latched for the life of the process and answered ERROR from then on, and the node stayed poisoned twenty minutes after the load stopped.
• An ordinary daemon restart no longer pages anyone. A daemon that has just come up refuses while its caches load — 7 to 20 seconds on a loaded node — so a routine recreate landed a critical alert and an email, then resolved a minute later. A real outage lasts longer than the grace and still alerts at once.

Smaller things

• The dashboard’s backup widget says whether the archive is usable, and what is inside it. It used to say only that a backup exists. What it reports now is read out of the manifest that lives inside the tar, beside the payload, in plaintext — so the answer travels with the archive, survives being copied, and can be read without the master key.
• A device names its groups from its own form. Membership could only be written from the group side: linking a device meant leaving the device, opening the group and adding it there. Devices were the last entity to work that way, and the first customer migrating from TACACSGUI hit it. An absent field leaves membership alone, so editing an unrelated setting cannot silently unlink a device.
• The diagnostic bundle can explain a CoA exchange. Everything that reaches a switch outside the request/response flow is a CoA — an admin disconnect, a VLAN change, a port bounce, a guest revoke — and the bundle could explain none of it. It now captures on the CoA port resolved the same way the sender resolves it, captures the ICMP that distinguishes “the switch ignored us” from “nothing was listening there”, decodes replies against the right secret, and collects the backend log where a CoA outcome is recorded. See Diagnostics.
• Ubuntu 26.04 LTS is a verified installation target (issue #15). Installed on a clean 26.04 by the documented route — bootstrap.sh → install.sh — without a single workaround, and verified by more than containers starting: a live TACACS+ authentication with the right verdict for both a correct and a wrong password, and all eleven containers surviving a host reboot.
• A sorted column header no longer rides over the resize line it shares with the next column.

---

1.2.8 ran 5067 passing, 0 failing on the dedicated runner: 4820 unit, service and integration at 81% backend line coverage, 233 end-to-end across six shards, and 14 load. Eight end-to-end cases are expected-fail — the documented known gaps — and nothing was skipped.

Every headline in this release has a name attached to it that is not ours: an issue, a switch, a directory, a cluster in somebody else’s rack. That is a much faster way to find out what a product actually does than any amount of testing our own assumptions, and it only works because people take the trouble to report things. Thank you — please keep doing it.

The bundle and the appliance images are on the download page, and what is coming next is on the roadmap.