In this blog post, I’ll look at how to make Percona XtraDB Cluster and SELinux work when used together.
Recently, I encountered an issue with Percona XtraDB Cluster startup. We tried to setup a three-node cluster using Percona XtraDB Cluster with a Vagrant CentOS box, but somehow node2 was not starting. I did not get enough information to debug the issue in the donor/joiner error log. I got only the following error message:
2018-02-08 16:58:48 7910 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.100.20' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '7910' --binlog 'mysql-bin' ' 2018-02-08 16:58:48 7910 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.100.20' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '7910' --binlog 'mysql-bin' Read: '(null)' 2018-02-08 16:58:48 7910 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.100.20' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '7910' --binlog 'mysql-bin' : 2 (No such file or directory) 2018-02-08 16:58:48 7910 [ERROR] WSREP: Failed to prepare for 'xtrabackup-v2' SST. Unrecoverable. 2018-02-08 16:58:48 7910 [ERROR] Aborting 2018-02-08 16:58:50 7910 [Note] WSREP: Closing send monitor...
The donor node error log also failed to give any information to debug the issue. After spending a few hours on the problem, one of our developers (Krunal) found that the error is due to SELinux. By default, SELinux is enabled in Vagrant CentOS boxes.
We have already documented how to disable SELinux when installing Percona XtraDB Cluster. Since we did not find any SELinux related error in the error log, we had to spend few hours finding out the root cause
You should also disable SELinux on the donor node to start the joiner node. Otherwise, the SST script starts but startup will fail with this error:
2018-02-09T06:55:06.099021Z 0 [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.100.20:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' '' --gtid '0dc70996-0d60-11e8-b008-074abdb3291a:1') 2018-02-09T06:55:06.099556Z 2 [Note] WSREP: DONOR thread signaled with 0 2018-02-09T06:55:06.099722Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.100.20:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' '' --gtid '0dc70996-0d60-11e8-b008-074abdb3291a:1': 2 (No such file or directory) 2018-02-09T06:55:06.099781Z 0 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.100.20:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' '' --gtid '0dc70996-0d60-11e8-b008-074abdb3291a:1'
Disable SELinux on all nodes to start Percona XtraDB Cluster.
The Percona XtraDB Cluster development team is working on providing the proper error message for SELinux issues.
via Planet MySQL
Percona XtraDB Cluster and SELinux: Getting It To Work