High availability
Run OrbisID on two or more servers so that users keep working, and scheduled scans keep running, when a server stops. It needs an Enterprise licence. For what a cluster does and the Cluster page, see Cluster.
How it fits together
users, agents, sensors
│
load balancer (yours)
/ \
┌────────────┐ ┌────────────┐
│ server 1 │ │ server 2 │ every server runs the full web UI and API
│ OrbisID │ │ OrbisID │
└─────┬──────┘ └─────┬──────┘
│ all read and write the database primary
▼
PostgreSQL primary ───▶ PostgreSQL standby (a copy that takes over if the primary fails)
- Every OrbisID server is active. Any of them can serve any user.
- One server at a time, the scheduler primary, runs scheduled work and scans. It is chosen through the database, so servers do not need to reach each other for that.
- The database is one PostgreSQL primary with one or more standbys. OrbisID reads and writes through the primary.
- Servers coordinate through the database and, for a few things such as reading another server's log files, over a small encrypted node API on its own port.
What you need
- OrbisID Enterprise, the same version on every server, and the same
ENCRYPTION_KEYon every server. The key encrypts the credentials OrbisID stores, so a server with a different key cannot read them and refuses to start. - At least two servers. Each needs the node API port (default
8444) reachable from the other servers. Do not expose that port through the public load balancer. - A load balancer, or DNS round-robin, in front of the servers' web ports. No sticky sessions are needed.
- PostgreSQL 15 or later with one of the two replication options below.
| From | To | Port | Use |
|---|---|---|---|
| Users, agents, sensors | Load balancer, then each server | 8080 (or your HTTPS port) | Web UI and API |
| Each server | Each other server | 8444 (node API, mutual TLS) | Joining, log files, database switchover |
| Each server | The PostgreSQL servers | 5432 | Database. With OrbisID-managed replication also standby to primary. |
Choose how the database is replicated
| Replicated by your database platform | Replicated by OrbisID | |
|---|---|---|
| For | Amazon RDS or Aurora Multi-AZ, Azure Database for PostgreSQL with high availability, Cloud SQL, a Patroni cluster | The Windows installer's bundled PostgreSQL |
| Replication and failover | The platform, automatic | PostgreSQL streaming replication that OrbisID sets up; you promote a standby (the page, or orbisid-cluster promote) |
| Cluster page | Servers and roles | Servers, roles, replication state, lag, pause and promote |
| Setting | orbisid.cluster.managed-replication=false (default) | true |
OrbisID does not fail a database over by itself in the second mode: doing that safely needs a third voting member. If you need automatic database failover, use your platform's.
A cluster on a database platform
-
Create the database with high availability on your platform, and note its writer endpoint (or, for a self-managed pair, both servers).
-
On the first server, install OrbisID as usual, pointing it at the database. For several PostgreSQL servers, give a full JDBC URL so the driver finds whichever is the writable primary, including after a failover:
DB_URL=jdbc:postgresql://db1:5432,db2:5432/orbisid?targetServerType=primaryWith a managed service, its single writer endpoint is enough.
-
Make it the first node:
java -jar orbisid-backend.jar cluster init --node-name server1 --node-api-url https://server1.example.com:8444This writes
config/cluster.properties. Instead of the command, an administrator can open Administration > Infrastructure > Cluster on a server where clustering is off, fill in Turn on clustering (server name and the host name other servers use to reach it) and typeconfirmin the confirmation box; it writes the same file. Either way, restart OrbisID afterwards. With the bundled PostgreSQL the same form can also set up OrbisID-managed replication (see below); on a database your platform replicates, leave that off. Start OrbisID withORBISID_CLUSTER_CONFIG_FILEpointing at it (the Windows service and the container images readconfig/cluster.propertiesby default). The first start creates the cluster, its certificate authority and the server's own certificate. -
Sign in as an administrator, open Administration > Infrastructure > Cluster > Add node, and create a join string.
-
On the second server, install the same version. In the installer choose Use an existing key on the Encryption Key page and enter the first server's key, and use the cluster's database as the external database. Sign in to the second server, open Administration > Infrastructure > Cluster, choose Join an existing cluster, paste the join string, enter a name and the host name the first server can reach it at, and type
confirm. Restart OrbisID on the second server. It appears on the Cluster page as active within a few seconds.The command line does the same, with
ENCRYPTION_KEYset to the cluster's key:java -jar orbisid-backend.jar cluster join "<join string>" --node-name server2 --node-api-url https://server2.example.com:8444 --db-url "<the same DB_URL>"
If the database login OrbisID uses is allowed to create roles (always true for the bundled database), each server gets its own database login, which is dropped again when you remove the server. Otherwise the second server uses the same login as the first, and the join says so.
A cluster with the Windows installer (OrbisID-managed replication)
The Commercial installer's High Availability page does the steps above, and for the bundled PostgreSQL it also sets up replication.
First server. Choose Create a new cluster. The installer prepares PostgreSQL to accept standbys (SSL, wal_level=replica, SSL-only pg_hba.conf rules, a limit on the log a lagging standby may hold back), restarts it, and starts OrbisID, which creates the cluster. Enter this server's address and the node API port.
Each further server. Install the same version, choose Use an existing key on the Encryption Key page and enter the first server's key, then choose Join an existing cluster and enter the join string from the Cluster page. This works only for a cluster whose database OrbisID replicates: one created by the installer with the bundled database and Create a new cluster, or one where Turn on database replication was used under Administration > Infrastructure > Cluster > Cluster Settings (or ticked when clustering was turned on there). A cluster whose database your platform replicates is joined with an external database pointing at the cluster's database, from Administration > Infrastructure > Cluster; a join string used against the wrong kind of cluster is refused before it is used up, with a message that says what to do. The installer then replaces the new server's empty bundled database with a copy of the primary's (a streaming standby), and starts OrbisID.
Open the firewall for the node API port and PostgreSQL's port between the servers; the installer adds Windows Firewall rules for both.
Without the installer's first-server step. On a server where clustering was turned on from the Cluster page, Cluster Settings > Database replication > Turn on database replication does what Create a new cluster does for the database: it prepares PostgreSQL, opens its port in the Windows Firewall, restarts PostgreSQL once and records the choice, and a Restart OrbisID now button finishes it. No command line is needed. The command line equivalents are orbisid-cluster configure-primary on the first server and orbisid-cluster join ... --seed-standby --pg-bin ... --pg-data ... on the next (run java -jar orbisid-backend.jar cluster help for every option).
Load balancer
- Point health checks at
/actuator/health/readinesson each server. It answers200when the server should receive users and503when it is draining, still joining, or its database accepts no writes./actuator/health/livenesssays only that the process is up. - No session affinity is needed: sessions are in the database. Single sign-on works through the balancer, and a sign-in can finish on a different server from the one it started on.
- Terminate TLS on the balancer or on each server, as you do for a single server.
- Keep the node API port (
8444) off the balancer. Requests to its paths on the public port are refused.
Agents and Endpoint Sensors
The agent's and sensor's server address may be a load balancer, or several addresses separated by commas. They use one until it can't be reached (or answers 502, 503 or 504), then move to the next:
orbisid.agent.instance-url=https://orbisid1.example.com,https://orbisid2.example.com
Upgrading
All servers must run the same version, and a server will not start while a running peer has a different one. Upgrade like this:
- In the load balancer, or with Drain, take the servers you are not upgrading out of service, and stop them. Wait 30 seconds so they no longer count as running.
- Upgrade one server and start it. It applies any database changes.
- Upgrade and start each of the others.
Users see a short interruption while you move from the first server to the last.
When things go wrong
A server stops
Nothing to do for the cluster to keep working. If it was the scheduler primary, another server takes over within about 25 seconds, and a scan that was running on it is marked failed (start it again). When the server returns it rejoins as an ordinary node. If it is not coming back, open Cluster → Evict.
The database primary fails
Database replicated by your platform: it fails over; OrbisID reconnects by itself.
Database replicated by OrbisID: OrbisID cannot save changes until a standby is promoted. The Cluster page needs a writable database to sign in, so promote from the command line on the server whose standby should take over:
java -jar orbisid-backend.jar cluster promote
It refuses while the old primary still accepts writes (add --force-fence to make it read-only first) and, when the old primary cannot be reached, asks you to confirm it is switched off or cut off from the network, because changes it accepts after that would be lost. The other servers reconnect on their own. Any former primary that comes back is made read-only automatically.
To make the old primary a standby again: stop its PostgreSQL, then
java -jar orbisid-backend.jar cluster rejoin --pg-bin <path to PostgreSQL bin> --pg-data <its data directory>
and start PostgreSQL. It rewinds onto the new primary's history, or copies it afresh with --reseed if it cannot.
A standby falls too far behind
If a standby is stopped or paused for long, the primary keeps the log it still needs, up to a limit (10 GB by default, max_slot_wal_keep_size) so its disk can never fill. You get an alert when the standby's slot nears the limit. Past it PostgreSQL drops the slot and the standby must be copied again: stop it and run cluster rejoin --reseed ....
Replication was paused too long
A paused standby resumes by itself after the maximum pause time. Resume it earlier from the Cluster page. A paused standby is not ready to take over.
A database shows as fenced
A server whose database accepted writes after a standby was promoted over it is made read-only by OrbisID (its role shows as fenced). It stays that way until you run cluster rejoin on it.
Settings
| Setting | Where | Default | Meaning |
|---|---|---|---|
cluster.auto_failover.enabled | Administration > Infrastructure > Cluster > Cluster Settings tab | on | When the scheduler primary stops, another server takes over. If off, scheduled work stops until you choose a primary. |
cluster.join_token.ttl_minutes | Cluster settings | 30 | How long a join string stays valid (5 to 1440). |
cluster.replication.max_pause_minutes | Cluster settings | 240 | Resume a paused standby after this long (5 to 1440). |
cluster.replication.lag_alert_seconds | Cluster settings | 60 | Alert when a standby is this far behind. |
cluster.replication.slot_alert_percent | Cluster settings | 80 | Alert when a slot holds back this share of the allowed log. |
orbisid.cluster.node-api-port | server config | 8444 | The node API port. 0 turns it off (then servers cannot be added through this one). |
orbisid.cluster.managed-replication | server config | false | OrbisID replicates the database (see above). |
Server configuration keys can also be set as environment variables: orbisid.cluster.node-api-port is ORBISID_CLUSTER_NODE_API_PORT.
Trying a cluster on one machine
The repository includes docker-compose.cluster.yml, a two-server cluster with a load balancer and a PostgreSQL primary with a streaming standby, for evaluation and development. It is not a production template. See docker/cluster/README.md.