DB-유

Handling Galera Cluster Split-Brain and Recovery

· mariadb, galera, high-availability

The first time a Galera cluster went non-Primary on me, I thought the database was corrupted. It wasn't. Galera (MariaDB Galera, Percona XtraDB Cluster or the Codership build) only accepts writes on a node that is part of the Primary Component, the group holding more than half the votes. When a partition or a run of crashes leaves no group with a majority, every node stops serving. That is Galera avoiding split-brain, not suffering one, though the application sees the same thing: the database is down.

Commands here are for MariaDB 10.1/10.2 Galera on CentOS 7; other builds differ mostly in service names. I assume a three-node cluster (any odd size; two nodes need an arbitrator, garbd) and SSH access to every node plus the MySQL root credentials.

Read the cluster state on each node

SHOW STATUS WHERE Variable_name IN
 ('wsrep_cluster_status','wsrep_cluster_size','wsrep_local_state_comment',
  'wsrep_ready','wsrep_connected','wsrep_last_committed','wsrep_cluster_state_uuid');
  • wsrep_cluster_status: Primary means the node can serve; non-Primary means it lost quorum; Disconnected means it is not talking to anybody.
  • wsrep_cluster_size: how many nodes this node sees.
  • wsrep_local_state_comment: Synced is normal; Donor/Desynced, Joiner and Initialized are transitional or broken.
  • wsrep_last_committed: the sequence number of the last applied transaction, used to decide which node is most advanced.

An application trying to use a non-Primary node gets:

ERROR 1047 (08S01): WSREP has not yet prepared node for application use

A network partition

If the partition heals on its own, Galera re-merges automatically and the minority side re-syncs via IST (incremental state transfer). Nothing to do except watch wsrep_cluster_size return to 3.

If the partition is permanent, for example one data center is gone, and the surviving nodes are a minority (one out of three), you have to tell them to become Primary manually. Do this on exactly one node and only once you are certain the other side is not still serving writes:

SET GLOBAL wsrep_provider_options = 'pc.bootstrap=true';

The node forms a new Primary Component; other reachable nodes rejoin it. When the lost nodes come back, they will see a different cluster history and perform SST (full state snapshot) from the survivors. If they had accepted writes in the meantime, those writes are discarded, which is the actual split-brain you want to avoid. Verify twice before bootstrapping; I check from a second terminal that the other side really is down.

After a full shutdown or crash

After a power failure, all nodes are down and none has a Primary Component to join. Galera refuses to start any node normally; the error log shows:

[ERROR] WSREP: It may not be safe to bootstrap the cluster from this node. It was not the last one to leave the cluster and may not contain all the updates.

First find the most advanced node. Check grastate.dat on each:

cat /var/lib/mysql/grastate.dat

# GALERA saved state
version: 2.1
uuid:    5c6f2e3a-4b1d-11e9-8f0c-0800271e1a2b
seqno:   184271
safe_to_bootstrap: 1

On a clean shutdown, the last node to stop has safe_to_bootstrap: 1 and a real seqno. After a crash, seqno is -1 on every node; recover it with:

sudo -u mysql mysqld --wsrep-recover

or, on MariaDB, galera_recovery. The log prints WSREP: Recovered position: <uuid>:<seqno>. Pick the node with the highest seqno, set safe_to_bootstrap: 1 in its grastate.dat, and bootstrap it:

# MariaDB 10.1+
sudo galera_new_cluster
# Percona XtraDB Cluster 5.7
sudo systemctl start mysql@bootstrap.service

Then start the other nodes normally with systemctl start mariadb. They join with IST if their gcache still covers the gap, or with SST otherwise.

Making the next one less painful

  • Use three or five nodes, or two nodes plus garbd on a third machine.
  • Set pc.weight in wsrep_provider_options so a primary site keeps quorum if a secondary site disappears (for example weight 2 on one node and 1 on the others).
  • Size gcache.size (default 128M) so a node that is down for the length of a typical maintenance can rejoin with IST instead of a full SST.
  • Use xtrabackup-v2 or mariabackup as wsrep_sst_method, not rsync, so a donor stays available during SST.
[galera]
wsrep_provider_options = "gcache.size=2G; pc.weight=1; gmcast.segment=0"
wsrep_sst_method       = mariabackup
wsrep_sst_auth         = sstuser:Sst-P@ss

Did it work?

On every node after recovery:

SHOW STATUS LIKE 'wsrep_cluster_size';
SHOW STATUS LIKE 'wsrep_cluster_status';
SHOW STATUS LIKE 'wsrep_local_state_comment';

All three should report the same wsrep_cluster_state_uuid, size 3, Primary and Synced. Do a write on one node and read it on another to confirm replication flows in both directions.

Log lines I have seen

  • [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out) on a joining node: it cannot reach any address in wsrep_cluster_address, usually a firewall. Galera needs TCP 4567 (group communication), 4568 (IST) and 4444 (SST) between nodes.
  • [ERROR] WSREP: SST failed: 1 (Operation not permitted) or WSREP_SST: [ERROR] xtrabackup_checkpoints missing: the SST user or password in wsrep_sst_auth is wrong on the donor, or mariabackup/xtrabackup is not installed on both nodes. Check /var/lib/mysql/innobackup.backup.log on the donor.
  • A node stays in Joiner for a long time and the donor shows Donor/Desynced: SST is in progress. With rsync the donor is read-only; with mariabackup it keeps serving. Do not restart either node.
  • wsrep_cluster_size is 3 but wsrep_ready is OFF on one node: the node is a member but is not synced yet. Wait for wsrep_local_state_comment to reach Synced.

Galera would rather stop than diverge. Find the node with the newest data, bootstrap only that one, and let the rest catch up. Bootstrapping two is how real split-brain happens.