Last week was three releases in three days, and every one of them came from the product telling on itself — we shipped a tool whose job is to find out what is really happening, and then sat with what it said.
This one is the opposite direction. Nearly everything below was found by somebody else: four public issues, a customer’s Calix switch, and the first high-availability cluster built outside our lab, which found five separate defects in its own lifecycle simply by being built by someone who had not watched us build one.
That last one is worth saying plainly. HA has been tested here for months. It took one cluster in somebody else’s rack, assembled by somebody following the runbook rather than remembering it, to surface a command we document and never implemented, an upgrade path that leaves the database unable to start, and a way to reattach a machine to a cluster it should have been a stranger to.
An endpoint, classified by where its machine sits in the directory
Public issue #5. A machine bound to a directory computer object already carried that
object’s DN, and no rule could reach it — so “every machine under OU=Workstations
gets this profile” had to be maintained by hand, one endpoint at a time.
Two new classification match types close that: LDAP DN and LDAP group.
A DN is matched component-wise, never as a substring, so a rule for OU=Computers
does not also capture OU=Computers-Test. One semantic covers both uses: a container
matches its whole subtree, a full object DN matches that one object.
The rule also carries the directory it speaks about. With several forests joined, the same DN — and certainly the same group name — exists in more than one, and an unscoped rule is exactly how a machine from one domain lands in a group written for another with nobody finding out.
Matching on group membership needed data nobody was importing: a computer’s memberOf
was never read, for any directory. Collecting it is a new switch on the LDAP
configuration, off by default — an upgrade must not quietly start recording which
groups a customer’s machines are in without being asked. Turning it back off stops the
collection and keeps what was already collected. The endpoint’s card shows every group
the directory reports, with the time each was last seen, which is what answers why did
my rule not fire.
And machines are imported from directories that are not Active Directory
Computer sync now runs against FreeIPA, OpenLDAP and generic LDAP as well. The
difference between directories turns out to be one line of filter —
(objectClass=computer) on AD, (objectClass=ipaHost) on FreeIPA,
(objectClass=ieee802Device) on an OpenLDAP keeping machines the RFC 2307 way — and
every input was already a field on the form.
On FreeIPA and OpenLDAP it is in fact simpler than on AD, because both publish a MAC address: a machine becomes an endpoint on the first sync, before it has ever authenticated. The staging list of computers awaiting an address exists because Active Directory has no standard attribute for one; there it stays the normal path.
Google Workspace still declines, and says so: Secure LDAP serves users and groups, and devices live behind an API no filter reaches.
The password prompt that was never sent
Public issue #8, and its author was right.
The TACACS+ Password prompt setting was seeded, saved, and rendered as a field on
the settings page. No code ever put it into the generated daemon configuration. The
compiled-in Password: went out no matter what you typed, on every device, in every
release.
It reaches the device now. Leaving it empty restores the daemon’s own default rather than blanking the prompt.
Two things worth knowing about where it applies. The value is used only while MFA is on — announcing an OTP is what the prompt is for, and a value left behind by a since-disabled MFA setup must not keep taking effect. And only an interactive login can display it at all: a device that logs in over PAP carries the password inside the first packet and asks nothing, which Extreme SLX-OS does exactly. The hint beside the field now says both.
An attribute declared optional was sent as mandatory
RFC 8907 separates attr=value from attr*value. The first is mandatory — a device
that does not implement the attribute must refuse the whole authorization. The
second may be ignored.
Templates have always carried the distinction, and the profile form has always shown
it. The configuration renderer wrote = in both cases, so the declaration never
reached the wire.
The shipped example is our own built-in Cisco IOS-XR template, whose task
attribute is declared optional and has been going out mandatory in every deployment
there has ever been. A device that rejects it fails the exec authorization outright —
and on the device the symptom is indistinguishable from a policy problem, which is
where the time goes. Templates that declare nothing are unaffected, and the Config
Preview renders exactly what is deployed.
The interface, in the places you kept hitting
Two more public issues, and a set of things they dragged out with them.
#9 — list columns can be resized, and they remember their widths. On wide screens the columns did not fill the space and there was no way to give one more of it. Every list now measures its columns once real rows are on screen, freezes them, and puts a divider on the right of each header that drags that border. Narrowing a column can pull an overflowing table back inside its container and take the horizontal scrollbar with it; the last column is elastic, so the table always reaches the right edge. Double-click a divider to drop the layout and start over; arrow keys nudge a focused border.
Widths are stored per table in the browser, in pixels — a share of the container would follow the window, and a table that overflowed by 30px would keep overflowing on a wider monitor.
#10 — a parent breadcrumb led to “404 — Route not found”. Clicking NAC, AAA or Settings in the trail landed on the error page, because those crumbs are sidebar groupings rather than pages and were being linked anyway. A crumb is a link only when its path is a real page you have permission to open; otherwise it is plain text, which is what a section is. The same rule turns the last crumb of a detail page into a working link back to its list.
And the trail now names the page you have open. /nac/endpoints/<id> read “NAC /
Endpoints” — naming an ancestor and going silent about the object on screen. It closes
with the object’s own name now, the same label the page header already shows, so no
page had to opt in.
Three things that came out with them:
- Tables reserved width for text they would never show. Every list cut its cell text at a width written into that page’s own file — 200px here, 300 there — and the browser has to reserve those pixels, so the cap acted as the column’s floor. TACACS+ Profiles demanded 975px inside a 941px container and scrolled horizontally with nothing on screen explaining why. The caps are gone from all sixteen shared tables.
- Whatever a column clips is readable on hover — in every list, rather than in the ones whose author remembered to add a tooltip.
- The LDAP browser draws a tree. Five defects in the one screen you open to pick a container or a group: levels had no indentation at all, the row highlight was dead, the group-members preview answered “50 of 1000” forever, and a level holding four thousand machines drew four thousand rows. Leaves are capped at 200 now with the remainder counted and said out loud — while containers are never cut, because cutting one makes everything below it unreachable while looking exactly like a container that is empty.
The first cluster in the field found five things
All five are lifecycle defects: not “HA does not work”, but “building, rebuilding, upgrading or repairing a cluster does not work the way the runbook says”. They are the kind that only appear when somebody else drives.
Upgrading a cluster converted before 1.2.3 left its database unable to start.
Patroni’s control API became authenticated unconditionally in 1.2.3, so the database
image refuses to start without PATRONI_REST_PASSWORD. Every path that creates a node
has provisioned that key since — a fresh install, a conversion, a join. The path every
existing cluster takes did not. So ./taranac update handed such a cluster a database
image that would not run, and the failure surfaced three steps later as a container in a
restart loop rather than at the moment it could have been fixed calmly.
The upgrade provisions it now, and provisions it identically on every node — which
is the whole difficulty. This is a cluster-wide credential, and two nodes that each
generated their own would authenticate nothing to each other: every switchover, failover
and restart the cluster sent that node would come back 401, quietly, right up to the
failover that needed it. So the value is derived from the master key the cluster
already shares, which gives the same answer on every node with no coordination and
nothing new on the wire. A cluster that already has a password keeps it untouched.
--reinstall was printed as the cure by three different refusals, and existed in no
parser. Re-adding a machine that is already a member — it died, it was rebuilt, its
join never finished — is ./taranac cluster add-node --address <addr> --reinstall. The
refusals that send you there printed that exact command. The runbook documented it. The
wizard’s own --help listed it. Typing it produced unknown option '--reinstall'.
The behaviour behind the flag had been implemented in full; only the two lines that
accept it on the command line were missing, which is why reading the code did not reveal
it either. Both add-node and add-witness take it now.
docker compose -f docker-compose.yml up -d on an HA node started a second writable
database. On a cluster node the database belongs to Patroni, through the HA overlay.
The base compose file on its own put an ordinary, writable PostgreSQL on Patroni’s data
directory — two write histories on one data set, which no failover can reconcile.
./taranac and install.sh both refused to do that, and both are bypassed the moment
someone runs docker compose directly, which is a reasonable thing to do and is how it
happened. The refusal now lives in the compose file itself, so the guard is where the
damage would be.
A stale etcd member list was called harmless, and it stopped joins and rebuilds. Each
node keeps a declared list of the cluster’s etcd members; adding a node updates it on the
node you ran the command from and deliberately not on the others, so peers fall behind —
and the preflight reported that as drift that “breaks nothing right now”. It breaks two
things. etcd compares the list against the live ring whenever a member starts on an empty
data directory — a rebuilt disk, a --reinstall, a restored node — and refuses to start
on any difference. And cluster add-node hands its list to the node being added, so
running it from a node that has fallen behind produces a join that cannot succeed. The
preflight now compares every member’s declared list against the ring that is actually
running and stops the add, naming the members that differ — and ./taranac cluster sync,
the command the preflight prints as the cure, actually reconciles that list now. It
previously left the one key it was prescribed for untouched and reported “already matches
what the cluster reports”.
A cluster rebuilt under the same name was, to etcd, the same cluster. etcd derives a ring’s identity from a token plus its first members, and the token was the cluster’s name — so tearing a cluster down and building it again with the same name and the same nodes produced a byte-identical cluster identity. A machine still holding the old cluster’s etcd data, most often one whose join had failed and which nobody thought to wipe, was not a stranger to the new ring. It reattached in silence. A cluster draws its own identity when it is created now and discards it when it is dismantled, so a rebuild is genuinely a different cluster. Existing clusters keep the identity they have; there is nothing to migrate.
The witness that tried to run the whole product
A witness unpacks the same bundle as a Taranac node — that is how it gets
witness-join.sh — so ./taranac update has always been reachable on it, and it ran the
node’s update: pulled the product images, brought docker-compose.yml up on the
arbiter, and edited a .env belonging to a different stack.
What an operator saw was a wall of “variable is not set” warnings ending in unable to get image '/radius:'. That visible failure was the lucky outcome. A witness whose
.env happened to carry an image prefix and version would instead have started
PostgreSQL, the API and the AAA daemons on the host whose entire purpose is to be cheap
to lose.
The updater recognises what kind of host it is on now. On a witness it refreshes the
bundle’s files and stops there: it neither reads nor edits the witness .env, and it
starts nothing. It deliberately does not restart etcd either, because recreating the
arbiter is a quorum event and not housekeeping — if a release does change the witness
compose file, the update says so and prints the single command that applies it, for you
to run while the cluster is healthy.
The same rule now covers a host running a standalone NCM collector, and ./taranac
itself: on a witness or collector host it drives that host’s stack, and the commands
needing the database or API explain that they belong on a Taranac node instead of failing
with the same wall of warnings.
A collector was told it was up to date while nothing was being collected
When a collector’s snapshot contract does not match its core, the core fail-closes: it
dispatches nothing, and the collector logs that it must be upgraded. The way out is
./collector-join.sh --update, which moves the collector to the version its core runs.
It answered “already in step with the core — nothing to do”. For hours. While nothing was being collected.
The core reports its version on every authenticated answer and the collector records it — but that header was attached only when the request succeeded. So the one answer that mattered, the refusal, carried no version, and after the core was upgraded the collector kept reporting the version it had heard before the upgrade, which of course matched the version it was still running. The single door out of the fail-closed state was held shut by the failure it exists for.
The core reports its version on refusals too now. Unauthenticated callers still learn
nothing: the version is attached only after a caller’s signature has verified. And the
collector’s own tool no longer claims to be in step while its log shows the core refusing
it — it explains the contradiction and what to do — with
./collector-join.sh --update --version <ver> available to move it to a version you name
without consulting the recorded one.
Smaller things
- Calix AXOS ships as a built-in template. AXOS authorizes an exec session under the
service name
Loginrather than the RFC 8907shell, and takes the role from aROLESattribute — it does not implement priv-lvl at all. A stock shell profile is therefore refused, and what the operator sees is a successful SSH login into a CLI that accepts onlyexit, which reads like a policy problem and is not one. Verified against a customer’s E7 running AXOS-R24.1.0. Command authorization stays off deliberately: AXOS documents accounting for commands but no per-command authorization, and offering command sets the device never asks for would only invite a profile that silently does nothing. - A sync mode nobody knows ran a user sync and reported success.
POST /ldap/{config_id}/synctakes an optional per-run mode, and anything that was not one of the two it knows fell through to a user sync answering{"status": "completed"}—computer_sync, which is a real operation on another route entirely, and every typo alike. An unknown mode is refused now, naming the modes that exist. The interface was never affected; this cost scripts and integrations calling the API directly. - A setting with a fixed set of values could be changed to nonsense through the API. Four LDAP settings accepted any string on update. The tolerance was deliberate and its reason was sound — every edit form posts the whole object, so a stored value this release does not know has to come back unchanged on saves that have nothing to do with it — but it was implemented as “accept anything”. The rule asks the right question now: re-sending what is stored always passes, changing it must land inside the known set.
- A classification pattern without a wildcard matches any part of the name, and now
says so. A hostname, certificate-identity or vendor rule written as
PC-01also catchesPC-011,OLD-PC-01andPC-01-TEST. That behaviour is deliberate and stays — changing it would rewrite rules customers have already written, and for vendor names the substring is the required behaviour, since an operator typesCiscowhere the database holds “Cisco Systems, Inc”. What was missing was the sentence. The field hints and match-type descriptions carry it now, along with the way out: a character class turns the value into a glob, soPC-0[1]matchesPC-01and notPC-011.
1.2.7 ran 4706 passing, 0 failing on the dedicated runner — unit through sustained-rate load under HA failover — at 81% backend line coverage, dated 2026-08-18 and commit-pinned. One number in there went down and is worth a sentence: the end-to-end tier reads 220 against 1.2.6’s 229, and none of those nine went missing. They are time-window cases that build a cron window around the current hour, and this run crossed the single 23:00–24:00 UTC hour in which the shape they need cannot be expressed at all — so they skip, with that reason attached, rather than assert a window that cannot hold. Eight further end-to-end cases are tracked openly as expected-fail, as they have been every release.
Four of the things above have issue numbers, which means four people took the time to write them down instead of working around them. That is worth more to this product than it probably feels like from the outside: the password prompt had been inert since the setting existed, the IOS-XR template had been sending a mandatory attribute in every deployment there has ever been, and neither would have been found from in here, because from in here they both look like they work.
If something is behaving strangely, the issue tracker
is the fastest route, and ./taranac diagnose will
usually tell you which half of the problem is ours. Existing installations update in
place with ./taranac update; new ones start from the appliance. Where the
platform goes next is on the roadmap.
Self-hosted, artifact-only, Elastic License 2.0.