Cluster
A cluster runs OrbisID on two or more servers at once, so users keep working when one of them stops. Every server serves the full web interface and API, and users can sign in to any of them, usually through a load balancer. This is an Enterprise feature, and the Administrator role manages it under Administration > Infrastructure > Cluster. For installing and operating a cluster, see High availability.
How a cluster behaves
- Everyone sees the same data. All servers read and write the same database, so there is nothing to wait for or reconcile when you move between them.
- Scheduled work runs on one server. One server at a time is the scheduler primary. It runs scheduled scans, finding refreshes, SIEM exports, ticket and webhook delivery and every other background task, so nothing runs twice. If it stops, another server takes over within about half a minute.
- A scan can be started anywhere. Click Scan on any server: the request is queued and the scheduler primary runs it. A scan that was running on a server that stopped is marked failed with a clear reason. It is not run again by itself, so that nothing is applied twice. Start it again, or wait for its next schedule.
- You can see which server did what. The footer of every page shows the name of the server you are signed in to. Scanning > Scan History shows, for each scan, the server it was started on and the server that ran it, and the scan's log shows the server for each system.
- Settings and caches follow. A setting you change on one server applies on the others within a moment.
- Sign-in works on any server. A sign-in with your identity provider can finish on a different server from the one it started on.
There are two roles that are easy to confuse:
| Role | What it is |
|---|---|
| Scheduler primary | The OrbisID server that runs scheduled work and scans. Changes automatically if it stops. |
| Database primary | The PostgreSQL server that accepts writes. The others hold a copy and take over if it fails. |
The Cluster page
The top of the page shows the cluster's health:
| Health | Meaning |
|---|---|
| Healthy | Every server is responding, one server is the scheduler primary and the database can take writes. |
| Degraded | Something needs attention, such as a server that has stopped responding or is draining, a mix of OrbisID versions, or a database that was made read-only. The reasons are listed. |
| Replication paused | Replication is paused on a standby database. |
| Read-only | No server has a writable database, so OrbisID cannot save changes. |
Below that, each server is a row with its status, roles, how recently it reported in, its OrbisID version (flagged if it differs from the others) and, when OrbisID manages the database replication, the state of its standby copy, how far it is behind and how much of the primary's log it is holding back. Recent events lists joins, removals, handovers, failovers and replication changes.
While replication is paused or the cluster is degraded, a banner on every page tells administrators and links here. You also get an email when a server stops responding, a standby falls behind or its replication slot nears its limit, replication stays paused for a long time, or the scheduler primary changes.
Cluster settings
The Cluster Settings tab (next to Cluster Health, which shows the servers, events and join tokens) sets automatic scheduler failover, how long a join string stays valid, the longest a replication pause may last, and the standby lag and replication slot alert thresholds. A change applies on every server within a moment.
Turning clustering on
On a server where clustering is off, the Cluster page shows Turn on clustering. Enter a name for the server and the host name or IP address the other servers will use to reach it, then click Turn on clustering and type confirm in the box that opens. Under Advanced settings you can change the port used for cluster traffic (8444 by default) or give a full address if it differs from the host name and port. OrbisID writes the cluster settings file and asks you to restart; clustering starts when it restarts.
If this server uses the bundled PostgreSQL, the form also offers Let OrbisID replicate this server's database to new servers (on by default). It prepares the database as described under Database replication below. Leave it off if your database platform replicates the database. If OrbisID cannot manage the database (it is on another machine, in a container, or its login is not a PostgreSQL superuser), the form says why and new servers must use a database your platform replicates.
Confirming an action
Removing a node, pausing replication and promoting a database ask you to confirm. If you sign in with a local account, enter your password. If you sign in with SSO you have no OrbisID password to type: click Confirm with SSO, sign in again at your identity provider, and you are brought back to the page. The confirmation lasts five minutes, during which you can carry out those actions without a password. Evicting a node, like turning clustering on, only asks you to type confirm.
Turning clustering off
Turn off clustering…, at the top of the Cluster Health tab, ends the cluster. It is available only for the last server of a cluster: with other servers still in it, remove them first. Type confirm in the box that opens. OrbisID clears this server's cluster membership, join tokens and certificates, moves its cluster settings file and certificates into a backup folder next to them, and asks you to restart. After the restart it runs as an ordinary single server. Until the restart this server no longer counts as a cluster member, so work that runs only on the scheduler primary waits. Turning clustering on again later starts a new cluster. If clustering was switched on with the ORBISID_CLUSTER_ENABLED environment variable, remove the variable and restart instead.
Adding a server
If the cluster shows Replicated by OrbisID under Cluster Settings:
- Click Add node here and optionally enter the new server's host name so the token only works for it. Copy the join string.
- Run the OrbisID installer on the new server (the same version). On the Encryption Key page choose Use an existing key and enter this cluster's key. The key is never sent over the network, and a server with a different key refuses to join. Keep the bundled database. On the High Availability page choose Join an existing cluster, paste the join string, and enter the address other servers use to reach the new one.
- The installer copies this cluster's database to the new server and starts OrbisID. The server appears on Cluster Health as joining, then active. Nothing else needs to be done on the new server.
If your database platform replicates the database:
- Install the same OrbisID version on the new server, choosing Use an existing key on the installer's Encryption Key page and entering this cluster's key, and point it at this cluster's database (the installer's external database option).
- Click Add node here and copy the join string.
- Sign in to the new server as an administrator and open Administration > Infrastructure > Cluster. Choose Join an existing cluster, paste the join string, enter a name for the server and the host name the other servers use to reach it, click Join cluster… and type
confirm. - Restart OrbisID on the new server (the page has a Restart OrbisID now button where OrbisID runs as the Windows service). It appears on Cluster Health as joining, then active.
Until the restart the new server still runs as a single server, so its scheduled tasks may run alongside the cluster's; join and restart it straight away. A join string used against the wrong kind of cluster is refused before it is used up, and the message says what to do.
The join string is shown once, works for one server and expires after 30 minutes by default. It also carries a fingerprint of the cluster's certificate authority, so the new server only talks to the real cluster. Unused tokens can be revoked from the list on the page.
Actions on a server
Open a server's menu (the three dots):
| Action | What it does |
|---|---|
| Make scheduler primary | Asks the current primary to hand over. It stops starting new work and hands over when what it is running finishes. |
| Set as preferred primary | The cluster hands the role to this server whenever it has been healthy for 30 seconds. Only one server can be preferred. |
| Drain | Takes the server out of service: its health check at /actuator/health/readiness starts answering 503, so a load balancer stops sending it users, and it hands over the scheduler primary role if it holds it. Return to service reverses it. The last active server cannot be drained. |
| Remove | For a drained server. Revokes its certificate and its database login, and ends its database connections. It can only return with a new join token. |
| Evict | The same for a server that has stopped responding and is not coming back. Type confirm to continue. |
| Pause replication / Resume replication | See below. |
| Promote its database | See below. |
Removing, and pausing replication and promoting a database, ask you to confirm again (see below); evicting only asks you to type confirm. Removing also asks you to type the server's name. The page will not remove the server you are signed in through, the server that hosts the database primary, or the last server.
Database replication
How the database is replicated depends on how the cluster was installed:
- Replicated by OrbisID. The bundled PostgreSQL, once database replication is on (chosen when clustering is turned on, in the installer's Create a new cluster, or later under Cluster Settings). OrbisID sets up streaming replication from the database primary to a standby on each other server and shows its state on this page.
- Replicated by your database platform. An Amazon RDS or Aurora Multi-AZ database, Azure Database for PostgreSQL with high availability, Cloud SQL, or a Patroni cluster you run. The platform fails over on its own, OrbisID reconnects to the new primary, and the page shows servers and roles but not replication details.
Turning on database replication
Cluster Settings > Database replication shows who replicates the cluster's database. When it says your database platform does and the database is the bundled PostgreSQL on this server, Turn on database replication… lets OrbisID do it instead, so that new servers can be installed with their own bundled database. Type confirm in the box that opens. OrbisID then:
- makes PostgreSQL accept connections from the other servers of the cluster, over TLS and with passwords only;
- opens PostgreSQL's port in this server's firewall (on other systems, the page tells you to);
- restarts PostgreSQL once, which makes OrbisID unavailable for a few seconds (where it cannot restart PostgreSQL, the page tells you to restart it);
- asks you to restart OrbisID, with a Restart OrbisID now button where OrbisID runs as the Windows service.
It is offered only while this is the only server running on the database, and only when OrbisID can reach and configure the database: on this machine, with a PostgreSQL superuser login. Otherwise the card says why.
Pausing replication
Pause replication freezes a standby's data at that moment while it keeps receiving changes, so nothing is lost and the primary is never held back. Use it to take a consistent snapshot of the standby or to investigate a change before it spreads. A paused standby is not ready to take over until it has caught up, so it resumes by itself after the maximum pause time (4 hours unless an administrator changes it). Replication cannot be paused on the last running standby while the primary waits for a synchronous standby, because every write would stop.
Switching the database primary
Promote its database moves the database primary to this server's standby while both are running: the current primary is made read-only, the standby catches up and takes over, and OrbisID reconnects. Users may see errors for a few seconds. Afterwards, run orbisid-cluster rejoin on the old primary's server to make it a standby of the new one. Until then it stays read-only, and OrbisID makes any former primary that comes back read-only on its own, so two servers can never both accept writes.
If the database primary has failed, OrbisID cannot save the change from the page. Run orbisid-cluster promote on the server whose standby should take over (see High availability). OrbisID does not promote a database by itself.
Limits
- All servers must run the same OrbisID version. A server will not start while a running peer has a different version.
- Auditors have no access to the Cluster page.
- A cluster needs at least two servers. A two-server cluster can replicate asynchronously only.