pt-stalk [OPTIONS] [-- MYSQL OPTIONS]

Tweet

When I work with High Availability software, I’m reminded of the maze in the original computer adventure game “You are Lost in a maze of twisty-turny passages all alike…”.

If you search the web for HA programs you will find many well maintained projects all related that refer each other. The goal of this document is to give you with a step by step guide to a production worthy MySQL system. It should provide at least 99.999% access to your data and be able to scale read requests as you grow.

I have chosen these programs and utilities because they are free (as in beer) and each has enterprise support available. (When you make the money to pay for it.) If you start with this MySQL platform you will avoid many common problems. Just write your application to read and write data from different servers.

Here is what we’ll be using:

• CentOS (Redhat) 6.2
• MySQL 5.5.xx (Percona) or the MySQL of your choice
• Pacemaker - to monitor MySQL’s health
• MySQL HA (MHA) – to fail over when something happens to the master
• DRDB Cluster Management Console
• Percona Toolkit – to make live easy
• OpenArk Kit – a set of utilities for MySQL
• xtrabackup for database backups
• mytop and innotop – for real time monitoring

If you want to see all this work check out my Youtube Video at : http://tinyurl.com/7yqj5gz

I’ve worked hard to make these instructions cut and past. The GREEN stuff gets cut and pasted into the linux command line. The BLUE is copied into an application (vi, crm etc), but the RED needs to be edited to fit your environment (passwords, IP). I enjoyed the work. I hope you do to.

Getting started

Every good system starts with good hardware. Two things database servers hunger for are disk space and memory. You should supply yours with as much of each as you can afford. My rule of thumb is three times size of one copy of the data. A production system with a RAID-10 disk system is good and two network ports or more is recommended.

To test this installation I’m building on a VMware server. If you’d like to know more about my hardware read my “Building a Home VMware server” story.

The Hardware

The operating system I’m using is CentOS 6.2 64 bit. Don’t use the 32bit versions.

How you define disk space is important part. Choose the type, size and partitions carefully. You don’t want daily activity filling the root file system (/) and taking down the database. The common places this can happen are, the MySQL data directory, the /tmp directory and log space. Keep these in separate partitions. I’ve found the OS system logs in /var/log take care of them selves. MySQL logs should be keep in the MySQL /data directory.

I’m test hardware (VMware ESXi 4) has four SAS hard disks. For each MySQL server (db1 and db2) I create four (4) 15Gig virtual hard disks, one on each physical SAS disk. I split each virtual disk into four partitions. The /boot partition is RAID-1 and the / (root), /data and /tmp partitions are RAID-5. The sizes of these partitions depend you your needs but your /boot needs to be about 1G and the / needs to be at least 12G.

This table show my for virtual disk and how they are raid-ed and partitioned.

Install the OS

Start with a minimum installation so you’ll have as few applications installed as possible. No desktop or server applications are needed.

After the install you should update the installed packages and include any packages you know you will be wanting or needed.

 yum -y update
 yum -y install openssh-clients rsync wget perl-DBI perl-TermReadKey

Remove any supplied MySQL.

Even with a minimum install there is a little clean up. I remove packages that are un-needed.  For better security you should remove  anything that connect to remote systems like bluetooth and printing.

 rpm -e mysql
 rpm -e mysql-libs --nodeps

Security

For the install process I turn off the firewall. In secure environments I recommend you leave it in place and deal with restrictions as they come up.

 service iptables stop
 chkconfig iptables off
 service ip6tables stop
 chkconfig ip6tables off

Because we are moving the MySQL data directory, you will need to disable SELinux or update it. To disable it, edit /etc/selinux/config and change the SELINUX line to SELINUX=disabled. Then, so you don’t have to reboot Linux for this to take effect, just write a 0 to the SELinux control file.

 vi /etc/selinux/config

SELINUX=disabled

 echo 0 >/selinux/enforce

If you don’t want to disable To update SELinux for the new data directory you will need to have the the SELinux tools installed.

 yum -y install policycoreutils-python
 semanage fcontext -a -t mysqld_db_t "/data/mysql(/.*)?"
 restorecon -Rv /data/mysql

Swap System

Serving queries from memory is much better then feeding it from disk. MySQL works hard to store data in a way to improve disk assess.  Feeding data from swap-ed out memory seems like a bad thing.  You not only have to go extra system code and the data is not stored in a way optimized for data.

Turning swap off altogether is a bad idea.  Some system utilities and applications expect there to be some swap available.  Turning down how much swap is used is the way to go.

Edit system control file and turn ‘swappness’ to zero.

vi /etc/sysctl.conf

and add the line

sys.swappiness=0

Syncing Time

Time sync is very important to maintaining accurate data. You may want to edit the /etc/ntp.conf file to point to your primary NTP time server. CentOS and Redhat provide time servers for your use. I recommend using pool.ntp.org.

 yum -y install ntp
 chkconfig ntpd on
 ntpdate 0.pool.ntp.org
 service ntpd start

Setup SSH

Now is a good time to make sure DNS works on all servers and each knows their names.

MHA need ssh access to each server. You need to create ssh keys and copy to the mysql servers.

Skip this if you already have ssh keys installed.

 ssh-keygen -t dsa -f ~/.ssh/id_dsa -N ""
 cp ~/.ssh/id_dsa.pub ~/.ssh/authorized_keys
 scp -r ~/.ssh db2:.

Network configuration

You can edit all the network setting with the network system configuration utility.

 system-config-network

Except you need to insure the hosts can ALWAY resolve each other.

 vi /etc/hosts

192.168.2.160 db.grennan.com db
192.168.2.161 db1.grennan.com db1
192.168.2.162 db2.grennan.com db2
192.168.2.163 db3.grennan.com db3

 scp /etc/hosts db2:/etc

Now is a good time to reboot each system and check the install.

 init 6

MySQL Setup

Install the MySQL of your choise. I’m installing Percona’s version 5.5.

 rpm -e mysql-libs-5.1.52 --nodeps
 rpm -Uhv http://www.percona.com/downloads/percona-release/percona-release-0.0-1.x86_64.rpm
 yum -y install Percona-Server-shared-compat
 yum -y install percona-toolkit
 yum -y install Percona-Server-server-55.x86_64
 yum -y install Percona-Server-client-55

The default location for MySQL data is /var/lib/mysql. MySQL benefits from being in it’s own partition. I’ll be putting the MySQL data in /data/mysql.

 service mysql stop
 mkdir /data
 mkdir /data/mysql
 cp -prv /var/lib/mysql/* /data/mysql
 mv /var/lib/mysql /var/lib/mysql-empty
 ln -s /data/mysql /var/lib/mysql
 chown -R mysql.mysql /data/mysql

Configure MySQL

You need to change the server-id number on each server. The minimum setting you will need are:

 vi /etc/my.cnf

[mysqld]
log-bin=mysql-bin
server-id=1                         # Each system needs a unique id number
innodb_flush_log_at_trx_commit=2
sync_binlog=1
# relay_log_purge=0 # uncomment on slaves
# read_only=1                 # uncomment on slaves

Percona has a service to help you configure MySQL for your hardware.   MySQL Configuration Wizard

Now we need to add a couple of user before we install the replicator. We also need to set a password for the root user. The root password is blank so just it return.

If you have skip-name-resolve set you will need to substitute the host names for IP addresses.

 service mysql start
 mysql -h localhost -u root -p

The next set of command are entered into the mysql client. You should have a ‘mysql >’ prompt.

 DROP USER ``@`localhost`;
 DROP USER ``@`db1.grennan.com` ;
 GRANT ALL on *.* TO `root`@`192.168.2.%` IDENTIFIED by `P@ssw0rd` with GRANT option;
 GRANT ALL on *.* TO `root`@`localhost` IDENTIFIED by `P@ssw0rd` with GRANT option;
 CREATE USER `repl`@`192.168.2.%` IDENTIFIED BY `RepP@ssw0rd`;
 GRANT REPLICATION SLAVE ON *.* TO `repl`@`192.168.2.%`;
 grant all on *.* to `hauser`@`db1.grennan.com` identified by `P@ssw0rd`;
 grant all on *.* to `hauser`@`db2.grennan.com` identified by `P@ssw0rd`;
 flush privileges;
 quit ;

Now, stop MySQL on all servers and copy all the MySQL data files to the slaves.

From the master (db1) at a system prompt ‘#’ :

 service mysql stop
 ssh db2 `service mysql stop`
 rsync -rog --delete /data/mysql root@db2:/data

Rather than type the password for MySQL each type we connect. We can also setup a usr .my.cnf to prevent this. This should connect you to the master (RW) server from each host.

 vi ~/.my.cnf

[Client]
user=root
password=P@ssw0rd
host=localhost
socket=/data/mysql/mysql.sock

 scp .my.cnf db2:.

Start Replication

On each system start MySQL and connect the two slaves to the master.

On the master we need to now the bin-file and the position. Note the numbers in purple.

 service mysql start
 mysql -e `reset master; reset slave;`
 mysql -e `show master status\G`

*************************** 1. row ***************************
File: mysql-bin.000001
Position: 107
Binlog_Do_DB:
Binlog_Ignore_DB:
1 row in set (0.00 sec)

On all slaves, use the master’s bin log position to setup replication. Start the slave and note that the IO and SQL are running. (Yes).
service mysql start

 mysql

 stop slave;
 CHANGE MASTER TO MASTER_HOST='192.168.2.201', MASTER_USER='repl',
 MASTER_PASSWORD='RepP@ssw0rd', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=107 ;
 start slave ;
 show slave status\G

Slave_IO_Running: Yes
Slave_SQL_Running: Yes

Installing MHA

Every MySQL server needs to have a copy of the MHA Node and Manager installed. Download the latest version of MHA from the Google Code server.

http://code.google.com/p/mysql-master-ha/downloads/list

There are a number of required Perl modules. I chose to use the Fedora evaluation packages to resolve these.

• DBD::mysql
• Config::Tiny
• Log::Dispatch
• Parallel::ForkManager
• Time::HiRes

Here is my install with the fedora packages.

 wget http://download.fedora.redhat.com/pub/epel/6/x86_64/epel-release-6-5.noarch.rpm
 rpm -i epel-release-6-5.noarch.rpm
 yum -y install perl-DBD-MySQL
 yum -y install perl-Config-Tiny
 yum -y install perl-Log-Dispatch
 yum -y install perl-Parallel-ForkManager
 yum -y install perl-Time-HiRes
 wget http://mysql-master-ha.googlecode.com/files/mha4mysql-node-0.53-0.el6.noarch.rpm
 rpm -i mha4mysql-node-0.53-0.el6.noarch.rpm
 wget http://mysql-master-ha.googlecode.com/files/mha4mysql-manager-0.53-0.el6.noarch.rpm
 rpm -i mha4mysql-manager-0.53-0.noarch.rpm

MHA only need one configuration file on each server. You can name this file anything you want. Change the setting and make sure the directories exist and are writable.

 vi /etc/MHA.cnf

Insert this configuration data into the MHA.cnf file.

[server default]
user=root
password=P@ssw0rd
manager_workdir=/var/log/masterha
manager_log=/var/log/masterha/MHA.log
remote_workdir=/var/log/masterha

[server1]
hostname=db1

[server2]
hostname=db2

You should have already installed SSH keys (see above) and the ‘hauser’ installed in MySQL.

masterha_check_ssh is used to check if you have SSH working. Look for the OK after the connect test.

 masterha_check_ssh --conf=/etc/MHA.cnf

Thu Dec 1 10:10:52 2011 – [info] Starting SSH connection tests..
Thu Dec 1 10:10:53 2011 – [debug]
Thu Dec 1 10:10:52 2011 – [debug] Connecting via SSH from root@db1(192.168.0.11) to root@db2(192.168.2.12)..
Thu Dec 1 10:10:53 2011 – [debug] ok.

masterha_check_repl is used to check MySQL replicaton. This is chatty. Your are looking for the ‘MySQL Replication Health is OK’ at the end. You should pay attention to any warnings.

 masterha_check_repl --conf=/etc/MHA.cnf

—
Thu Dec 1 10:07:22 2011 – [info] Checking slave configurations..
Thu Dec 1 10:07:22 2011 – [warning] read_only=1 is not set on slave db2(192.168.0.11:3306).
Thu Dec 1 10:07:22 2011 – [warning] relay_log_purge=0 is not set on slave db2(192.168.0.12:3306).
—
MySQL Replication Health is OK.

You now have MHA installed. If the configure it wrong MHA will give some errors that are not helpful and die. check the system logs (/var/log/messages) and your MHA settings and try again.

We’ll be using Pacemaker and a command like this to fail over to a slave when the master dies.

masterha_master_switch –master_state=dead –dead_master_host=db1 –conf=/etc/MHA.cnf

Putting the H in High Availability (HA)

There are may ways a system can fail and even with MySQL running. How do you know when your dead? How do you make the new is system ready to accept data? Are your servers even on the same network? I’ve chosen PaceMaker because it is the most complete and flexible.

Install Pacemaker / CoroSync

Neither RedHat or CentOS supply PaceMaker packeges. RedHat support their own propitiatory clustering suite. CentOS does supply heartbeat. Thankfully PaceMaker project provide it in EPEL repository for Redhat 5.

Note we are installing PaceMaker for RedHat 5 not 6.2. At the time of this writing a version for 6.2 was not available. This create some dependence problems we have to work around.

 wget -O /etc/yum.repos.d/pacemaker.repo http://clusterlabs.org/rpm/epel-5/clusterlabs.repo
 yum -y install libtool-ltdl
 ln -s /usr/lib64/libltdl.so.7.2.1 /usr/lib64/libltdl.so.3
 yum -y install net-snmp
 wget http://www.clusterlabs.org/rpm/epel-5/x86_64/cluster-glue-1.0.6-1.6.el5.x86_64.rpm
 wget http://www.clusterlabs.org/rpm/epel-5/x86_64/cluster-glue-libs-1.0.6-1.6.el5.x86_64.rpm
 rpm -i cluster-glue-1.0.6-1.6.el5.x86_64.rpm --nodeps
 rpm -i cluster-glue-libs-1.0.6-1.6.el5.x86_64.rpm --nodeps
 yum -y install pacemaker corosync heartbeat

Again, make sure all the name resolves are in place on both DB1 and DB2. Heartbeat and Pacemaker provide the group communications and process management respectivly. The nodes must be able to communicate with each other.

Configure CoroSync

On both DB1 and DB2.

 corosync-keygen
 chown root:root /etc/corosync/authkey
 chmod 400 /etc/corosync/authkey

 vi /etc/corosync/corosync.conf

totem {
    version: 2
    token: 5000 # How long before declaring a token lost (ms)
    token_retransmits_before_loss_const: 20 # How many token retransmits
    # before forming a new configuration
    join: 1000 # How long to wait for join messages in the membership protocol (ms)
    consensus: 7500 # How long to wait for consensus to be achieved before
    # starting a new round of membership configuration (ms)
    vsftype: none # Turn off the virtual synchrony filter
    max_messages: 20 # Number of messages that may be sent by
    #one processor on receipt of the token
    secauth: off # Disable encryption
    threads: 0 # How many threads to use for encryption/decryption
    clear_node_high_bit: yes # Limit generated nodeids to 31-bits (positive signed integers)
    # Optionally assign a fixed node id (integer)
    # nodeid: 1234
    interface {
        ringnumber: 0
        # The following three values need to be set based on your environment
        bindnetaddr: 192.168.2.201
        mcastaddr: 239.255.42.0
        mcastport: 5405
        }
}
logging {
    fileline: off
    to_syslog: yes
    to_stderr: no
    syslog_facility: daemon
    debug: on
    timestamp: on
}
amf {
    mode: disabled
}

 vi /etc/corosync/service.d/pcmk

service {
    # Load the Pacemaker Cluster Resource Manager
    name: pacemaker
    ver: 0
}

If you didn’t edit this on both servers copy it from second server.

 rsync -rop /etc/corosync db2:/etc

Now start corosync on both servers.

 chkconfig corosync on
 /etc/init.d/corosync start

Check CoroSync healty. Look for ‘no faluts’ and both members show up as members.

 corosync-cfgtool -s

Printing ring status.
Local node ID -922572608
RING ID 0
id = 192.168.2.201
status = ring 0 active with no faults

 corosync-objctl | grep members

totem.join=1000 # How long to wait for join messages in the membership protocol (ms)
runtime.totem.pg.mrp.srp.members.-922572608.ip=r(0) ip(192.168.2.201)
runtime.totem.pg.mrp.srp.members.-922572608.join_count=1
runtime.totem.pg.mrp.srp.members.-922572608.status=joined
runtime.totem.pg.mrp.srp.members.-905795392.ip=r(0) ip(192.168.2.202)
runtime.totem.pg.mrp.srp.members.-905795392.join_count=1
runtime.totem.pg.mrp.srp.members.-905795392.status=joined

Patch for Pacemaker

Yves Trudeau had created a newer version of the mysql resource agent for Pacemaker. He keeps it at:
https://github.com/y-trudeau/. Just download it and copy it into the ocf resource heartbeat directory.

 wget https://raw.github.com/y-trudeau/resource-agents/master/heartbeat/mysql
 cp mysql /usr/lib/ocf/resource.d/heartbeat/mysql

Configure Pacemaker

Fencing is normally used and is enabled by default and should be! We’ll start my turning Shoot the other Node in the Head (stonith) off. TURN THIS BACK ON BEFORE YOU GO INTO PRODUCTION! You disable it with:

 crm configure property stonith-enabled="false"

If your cluster consists of just two nodes, switch the quorum feature off with:

 crm configure property no-quorum-policy=ignore

This is a bit of magic. I’ll explain after we’re done. Create file to configure crm and paste this document into it.

 crm configure node db2.grennan.com attributes IP="192.168.2.202"

 crm configure

Now your in the CRM configuration utility. Type in the blue part of this txt and edit the red parts to fit your configuration.

crm(live)configure# primitive failover-ip ocf:heartbeat:IPaddr \
 params ip="192.168.2.200" \
 operations $id="failover-ip-operations" \
 op monitor start-delay="0" interval="2"

crm(live)configure# primitive p_mysql ocf:heartbeat:mysql \
 params pid="/var/run/mysqld/mysqld.pid" socket="/data/mysql/mysql.sock" \
 test_passwd="P@ssw0rd" enable_creation="false" replication_user="root" \
 replication_passwd="P@ssw0rd" MHASupport="true" \
 operations $id="p_mysql-operations" \
 op monitor interval="20" timeout="30" depth="0" \
 op monitor interval="10" role="Master" timeout="30" depth="0" \
 op monitor interval="30" role="Slave" timeout="30" depth="0"

crm(live)configure# ms ms_mysql p_mysql meta clone-max="2" notify="true"

crm(live)configure# colocation col_ms_mysql_failover-ip inf: failover-ip ms_mysql:Master

crm(live)configure# order ord_ms_mysql_failover-ip inf: ms_mysql:promote failover-ip:start

crm(live)configure# commit

crm(live)configure# quit

Monitor Pacemaker

Back at the system prompt you can monitor the health of Pacemaker from the command line.

 crm_mon -Arf

============
Last updated: Fri Feb 10 14:28:53 2012
Last change: Fri Feb 10 14:26:15 2012 via crm_attribute on db1.grennan.com
Stack: openais
Current DC: db1.grennan.com – partition with quorum
Version: 1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558
2 Nodes configured, 2 expected votes
3 Resources configured.
============

Online: [ db1.grennan.com db2.grennan.com ]

Full list of resources:

failover-ip (ocf::heartbeat:IPaddr): Started db1.grennan.com
Master/Slave Set: ms_mysql [p_mysql]
Masters: [ db1.grennan.com ]
Slaves: [ db2.grennan.com ]

Node Attributes:
* Node db1.grennan.com:
+ IP : 192.168.2.201
+ master-p_mysql:0 : 3601
+ readerOK : 1
+ writerOK : 1
* Node db2.grennan.com:
+ IP : 192.168.2.202
+ master-p_mysql:1 : 1
+ readerOK : 0
+ writerOK : 0

Migration summary:

* Node db1.grennan.com:
* Node db2.grennan.com:

Install DRBD Management Console

The DRBD Management Console is a Java application that eases the burden of managing your DRBD and Pacemaker/Corosync or Heartbeat based cluster systems.

Download this Jar file to your workstation.

When you run it you will need to add the server logins and cluster set. You don’t need to edit anything. Skip any installations or configurations.

http://sourceforge.net/projects/lcmc/files/LCMC-1.2.3.jar/download

[I'll post a video of configuring the Management Console here.]

But wait there is MORE

Percona Tool kit

You install the Perconal Tool kit when you installed MySQL. :-) Percona Toolkit is a collection of advanced command-line tools used by Percona (http://www.percona.com/) support staff to perform a variety of MySQL and system tasks that are too difficult or complex to perform manually. As a DBA you will find them very useful. Here is a small sample:

• pt-duplicate-key-checker – Find duplicate indexes and foreign keys on MySQL tables.
• pt-heartbeat – Monitor MySQL replication delay.
• pt-index-usage – Read queries from a log and analyze how they use indexes.
• pt-query-advisor – Analyze queries and advise on possible problems.
• pt-query-digest – Analyze query execution logs and generate a query report, filter, replay, or transform queries for MySQL, PostgreSQL, memcached, and more.
• pt-query-profiler – Execute SQL statements and print statistics, or measure activity caused by other processes.
• pt-show-grants – Print MySQL grants so you can effectively replicate, compare and version-control them.
• pt-table-checksum – Perform an online replication consistency check, or checksum MySQL tables efficiently on one or many servers.
• pt-table-sync – Synchronize MySQL table data efficiently.
• pt-visual-explain – Format EXPLAIN output as a tree.

openark kit

Shlomi Noach maintains another great set of utility.

• oak-apply-ri: apply referential integrity on two columns with parent-child relationship.
• oak-block-account: block or release MySQL users accounts, disabling them or enabling them to login.
• oak-chunk-update: perform long, non-blocking UPDATE/DELETE operation in auto managed small chunks.
• oak-get-slave-lag: print slave replication lag and terminate with respective exit code.
• oak-hook-general-log: hook up and filter general log entries based on entry type or execution plan criteria.
• oak-kill-slow-queries: terminate long running queries.
• oak-modify-charset: change the character set (and collation) of a textual column.
• oak-online-alter-table: perform a non-blocking ALTER TABLE operation.
• oak-prepare-shutdown: make for a fast and safe MySQL shutdown.
• oak-purge-master-logs: purge master logs, depending on the state of replicating slaves.
• oak-repeat-query: repeat query execution until some condition holds.
• oak-security-audit: audit accounts, passwords, privileges and other security settings.
• oak-show-limits: show AUTO_INCREMENT “free space”.
• oak-show-replication-status: show how far behind are replicating slaves on a given master.

You can find them here. http://code.openark.org/forge/openark-kit

MyTOP

Mytop is a console-based (non-gui) tool for monitoring the threads and overall performance of MySQL. The original development was done by Jeremy D. Zawodny <Jeremy@Zawodny.com> and is well know after being published in the book, High Performance MySQL. I’ve updated it with as a tool for monitoring MyISAM and replication.

I would be amiss if I didn’t say something about innotop. Some consider this a  successors to mytop. It is if you are using InnoDB. I still find mytop useful. More than half of my tables are MyISAM.

 wget http://www.mysqlfanboy.com/mytop/mytop-1.9.tar.gz
 tar zxf mytop-1.9.tar.gz
 cd mytop-1.9
 perl Makefile.PL
 make install

 wget https://innotop.googlecode.com/files/innotop-1.8.0.tar.gz
 cd innotop-1.8.0
 perl Makefile.PL
 make install

Backup

Percona XtraBackup is a hot backup utility that doesn’t lock your database during the backup!

It’s best to run this from one of your slaves. I often create a local copy and then rsync the files off the backup system to a remote (off site) server using a cron script.

 yum -y install xtrabackup

 innobackupex --user=root --password=`P@ssw0rd` /tmp/Backup

Here is an example script.

export HOST=`/bin/hostname -a`
mkdir /root/Backup
/usr/bin/innobackupex –user=root –password=`P@ssword` /root/Backup
/usr/bin/xtrabackup –prepare –target-dir=/tmp/backup
/bin/find /tmp/Backup/* -mtime +1 -exec rm {} -Rf \;

Last word

If you’ve made it this fare, now is a good time to restart everything and do some testing.

 init 6

Tweet

I have video of two MySQL servers (one master, on slave) failing over from Master to Slave.  The Master becomes the Slave and vise-verse with the slave becoming read-only.  Next some load testing and then I’ll post how I did it with FREE software.

Click here to view the video on YouTube.

Tweet

What will this return?

select if((1 = true) and ( 1 is TRUE) and (NULL is not TRUE), TRUE, FALSE) as answer;

You should know,

NULL is never TRUE or FALSE but

NULL is always NOT TRUE or NOT FALSE.

NULL is a little like a vacuum.  Ancient Greek philosophers did not like to admit the existence of a vacuum, asking themselves “how can ‘nothing’ be something?”.    Does GOD exist in NULL?

If  select “” is NOT NULL; is TRUE what is “”?

select "" is NOT NULL;
+----------------+
| "" is NOT NULL |
+----------------+
|              1 |
+----------------+

Is the lack of an answer, an answer? Is the lack of datum, DATA?  Or should it be “”.  Should we record the blank spaces?

Given:

When should this happen?

for x = 1 to X;
INSERT INTO `test`(`data`) VALUES ( NULL);
next x;

Null happens.

Tweet

Tweet

MySQL doesn’t often crash, but, if you use MySQL on a production system you should have High Availability.

Maybe you’ve not heard of MHA.  Yoshinori Matsunobu only released it in July of this year (2011).  I’ve been reviewing it and I think you should too. There are a lot of people chasing after this “holy grail” and most systems are complex and / or hard to recover when they fail. MHA simple and easy to use.

MHA is a fail over tool. It’s designed to fail a master and promote a slave to a new master. (This is simply to say but you should read how it’s done.) It can monitor your master or you can manually fail over.  Failing back doesn’t happen. Your expected to fix the failed master and turn it into a slave.  The monitoring program (masterha_manager) will stop running after a fail over.

I’m not going to give details on how MHA works. I’ll assume you have a master and some number of slaves already working.

Your slave/s server should be ready to become masters. This means they have sever_id and bin-log set.  You should be able to run ‘show master status’ on your slave/s and get an answer. Slaves should also be set with read_only=1 and realy_log_purge=0;

Preparing the way 

Download the latest version of MHA from the Google Code server. http://code.google.com/p/mysql-master-ha/downloads/list

Every MySQL server needs to have a copy of the MHA node installed.

 yum install perl-DBD-MySQL.x86_64 

 rpm -i mha4mysql-node-0.52-0.noarch.rpm

The Manager

The Manageris a little more difficult.  It should be run on a system outside your cluster.  Should the Manager process fail your HA process fails with it.  (Quis custodiet ipsos custodes?)

There are a number of required Perl modules.  You may find you have to use CPAN to install these. CPAN will require you install perl-YAML, make and gcc.  (I found this the easiest for CentOS 6.2)  If you use CPAN you will need to tell RPM to skip dependencies.

• DBD::mysql
• Config::Tiny
• Log::Dispatch
• Parallel::ForkManager
• Time::HiRes

 rpm -i mha4mysql-manager-0.52-0.noarch.rpm --nodeps

This may not be needed. On some systems, because I installed on a x64 version (as you should be too) I found I had to copy the MHA libraries to the perl lib64 space.  You might want to make simlinks instead.

  cp -crp  /usr/lib/perl5/vendor_perl/MHA  /usr/lib64/perl5/vendor_perl

You only need one configuration file if you are not going to monitor multiple Master/Slave clusters.  You can name this configuration file anything you want.  Change these setting and make sure the directories exist and are writable.

On the server where the Manager will be running:

  vi /etc/MHA.cnf

[server default]
user=hauser
password=P@ssw0rd
manager_workdir=/var/log/masterha
manager_log=/var/log/masterha/MHA.log
remote_workdir=/var/log/masterha

[server1]
hostname=db1

[server2]
hostname=db2

The MHA will need access to each server so you need to add the user to each.  On all MySQL server add the hauser.

 mysql> grant all on *.* to 'hauser'@'192.168.%.%'  identified by 'P@ssw0rd';  mysql> flush privileges; 

Now is a good time to make sure all the server names resolve on each system and they know their own names.

MHA also need ssh access to each server. You need to create ssh keys and copy to the mysql servers.

Don’t do these steps if you already have ssh keys installed.

 ssh-keygen -t dsa -f ~/.ssh/id_dsa -N ""  

 cp ~/.ssh/id_dsa.pub ~/.ssh/authorized_keys

 scp -r ~/.ssh db2:

Check if you have ssl working with the masterha_check_ssh tool.  Look for the OK after the connect test.

From the Manager server:

 masterha_check_ssh  --conf=/etc/MHA.cnf

Thu Dec  1 10:10:52 2011 - [info] Starting SSH connection tests..
Thu Dec  1 10:10:53 2011 - [debug]
Thu Dec  1 10:10:52 2011 - [debug]  Connecting via SSH from root@db1(192.168.0.11) to root@db2(192.168.2.12)..
Thu Dec  1 10:10:53 2011 - [debug]   ok.

MHA need to control replication so you should test it with masterha_check_repl. This will outout a lot of chat.  Your looking for the ‘MySQL Replication Health is OK’ at the end.   You might also pay attention to any warnings.

 masterha_check_repl  --conf=/etc/MHA.cnf

---
Thu Dec  1 10:07:22 2011 - [info] Checking slave configurations..
Thu Dec  1 10:07:22 2011 - [warning]  read_only=1 is not set on slave db2(192.168.0.11:3306).
Thu Dec  1 10:07:22 2011 - [warning]  relay_log_purge=0 is not set on slave db2(192.168.0.12:3306).
---
MySQL Replication Health is OK.

You now have MHA installed.  If you configure it wrong MHA will give some errors that are not helpful and die. Check your settings again.

If you want MHA to monitor the status of your master server you need to run masterha_manager.  It does not normally run as a daemon.  If masterha_manage is not running your server will not automatically fail over.  This is a weakness in MHA.  You might want to use something like Pacemaker(Heartbeat) to monitor your systems and MHA to do the fail over.  Here is how I make MHA run in the background.

 nohup  masterha_manager  --conf=/etc/MHA.cnf &

You can check masterha_manager with masterha_check_status.

 masterha_check_status  --conf=/etc/MHA.cnf

Now the magic. If you want to manual fail over you can use masterha_master_switch and tell it the master is alive.

 masterha_master_switch  --master_state=alive --conf=/etc/MHA.cnf

If your using another monitoring system like Pacemaker you can have it run this command to automatically fail your master.

 masterha_master_switch --master_state=dead --dead_master_host=db1 --conf=/etc/MHA.cnf

If you find I’ve missed something or it works different on your OS please drop me a comment.

Tweet

Most existing examples I found  setup SSL authentication and encryption.  If you are handling  PCI DSS or HIPAA data you must encrypt the data on the wire, but passwords are all you need to authenticate the application to the data source.

 # mkdir /etc/mysql/certs
 # cd /etc/mysql/certs

This looks complicated, it’s not.  JUST PRESS ENTER when openssl ask you a question.  This will not make you any less secure.  Your data will be encrypted.   Note these keys will expire in three years (1095 days).

 # openssl genrsa 2048 > ca-key.pem
 # openssl req -new -x509 -nodes -days 1095 -key ca-key.pem -out ca-cert.pem
 # openssl req -newkey rsa:2048 -days 1095 -nodes -keyout server-key.pem -out server-req.pem
 # openssl rsa -in server-key.pem -out server-key.pem
 # openssl x509 -req -in server-req.pem -days 1095 -CA ca-cert.pem -CAkey ca-key.pem \
 -set_serial 01 -out server-cert.pem

With the keys generated you need to tell MySQL to use them. Add these lines to your my.cnf.

 # vi /etc/my.cnf

 ssl
 ssl-cipher=DHE-RSA-AES256-SHA
 ssl-ca=/etc/mysql/certs/ca-cert.pem
 ssl-cert=/etc/mysql/certs/server-cert.pem
 ssl-key=/etc/mysql/certs/server-key.pem

Now restart mysql.

 # service mysql restart

You should see SSL is enabled and mysql sees the keys.

 mysql> show variables like '%ssl%';
 +---------------+----------------------------------+
 | Variable_name | Value                            |
 +---------------+----------------------------------+
 | have_openssl  | YES  |
 | have_ssl      | YES  |
 | ssl_ca        | /etc/mysql/certs/ca-cert.pem     |
 | ssl_capath    |                                  |
 | ssl_cert      | /etc/mysql/certs/server-cert.pem |
 | ssl_cipher    | DHE-RSA-AES256-SHA               |
 | ssl_key       | /etc/mysql/certs/server-key.pem  |
 +---------------+----------------------------------+
 7 rows in set (0.00 sec)

Duplicate these keys and the configuration segment to all your servers.

Client

If the client system is not a server, copy of the MySQL SSL keys to it was well.  Then, you need to tell the client to use SSL.  Edit your user’s .my.cnf file and give it the keys too.

 # vi ~/.my.cnf

[client]
 ssl
 ssl-cipher=DHE-RSA-AES256-SHA
 ssl-ca=/etc/mysql/certs/ca-cert.pem

Thats it.  It should be working.  Look for ‘Cipher in use’.

 # mysql -e "\s"

mysql Ver 14.14 Distrib 5.1.59, for unknown-linux-gnu (x86_64) using readline 5.1

 Connection id: 7
 Current database:
 Current user: root@localhost
 SSL: Cipher in use is DHE-RSA-AES256-SHA
 ......
 Threads: 1 Questions: 22 Slow queries: 0 Opens: 15 Flush tables: 1 Open tables: 8 Queries per second avg: 2.0

Replication

On the master, you need to tell the replication user to require SSL connections. Replace the rep_user with your replication user’s ID.

 mysql> GRANT USAGE ON *.* TO 'rep_user'@'%'  REQUIRE SSL;
 mysql> flush privileges;

and on the slave tell it to connect to the master with SSL.  Then make sure you are still connecting. Change the IP and user name, show here,  to your settings.

 mysql> stop slave;
 mysql> CHANGE MASTER TO master_host='192.168.1.12', master_user='rep_user', \
MASTER_SSL=1, MASTER_SSL_CA = '/etc/mysql/certs/ca-cert.pem' ;
 mysql> start slave;
 mysql> show slave status;

Applications

Your application could be written in lots of different languages and I can’t go over each of them but here are some links to setup up SSL connection to MySQL in several of the most popular.  If you know where there are some better examples, please leave me a comment.

Python

Perl

JDBC

Tweet

This is more boring parts but you must lay a good foundation.

MySQL Setup:

All the resources must be in place before you configure Pacemaker. The default location for MySQL data is /var/lib/mysql.  You will be moving this to the DRBD file system in /data/mysql.   The default MySQL configuration is in /etc/my.cnf.  I move it to /data/mysql so updates will move with system fail overs.   To keep life simple I use sym-links to point from /var/lib/mysql and /etc/my.cnf to their new locations.
Install MySQL of your choise on DB1, DB2 and DB3.

# rpm -Uhv http://www.percona.com/downloads/percona-release/percona-release-0.0-1.x86_64.rpm
# yum -y install Percona-Server-server-51.x86_64 Percona-Server-client-51.x86_64 \
 Percona-Server-shared-51.x86_64 xtrabackup.x86_64
# service mysql start
# mysql -e "CREATE FUNCTION fnv1a_64 RETURNS INTEGER SONAME 'libfnv1a_udf.so'"
# mysql -e "CREATE FUNCTION fnv_64 RETURNS INTEGER SONAME 'libfnv_udf.so'"
# mysql -e "CREATE FUNCTION murmur_hash RETURNS INTEGER SONAME 'libmurmur_udf.so'"
# service mysql stop
# chkconfig mysql off

ON DB1, move MySQL data to the DRBD file system.

# df

Filesystem        1K-blocks   Used Available Use% Mounted on
 /dev/md2           11903664      1633608   9655620   15% /
 /dev/md1             505508        218159    261250    46% /tmp
 /dev/md0             256586         25967    217371    11% /boot
 tmpfs               2037412  16348   2021064    1% /dev/shm
 /dev/drbd1         30477552 176200  28753152    1% /data

On DB1 and DB3

# mkdir /data/mysql
# cp -prv /var/lib/mysql/* /data/mysql
# mv /var/lib/mysql /var/lib/mysql-empty
# ln -s /data/mysql /var/lib/mysql
# touch /data/mysql/my.cnf
# ln -sf /data/mysql/my.cnf /etc/my.cnf
# chown -R mysql.mysql /data/mysql
# chmod 644 /data/mysql/my.cnf

On DB2, make it ready for a fail over.

# mv /var/lib/mysql /var/lib/mysql-empty
# ln -sf /data/mysql /var/lib/mysql
# ln -sf /data/mysql/my.cnf /etc/my.cnf

Configure MySQL (/data/mysql/my.cnf)

I have a documented my.cnf file I start with. You need to change the server-id number on each server.  The minimum setting you will need are:

[mysqld]
 log-bin=mysql-bin
 server-id=1
 innodb_flush_log_at_trx_commit=2
 sync_binlog=1

on DB1

# service mysql start

Now we need to add a couple of user before we install the replicator.  We also need to set a password for the root user.  The root password is blank so just it return.

If you have skip-name-resolve set you will need to substitute the host names for IP addresses.

# mysql -h localhost -u root -p
  mysql> DROP USER ''@'db.grennan.com';
  mysql> DROP USER ''@'localhost';
  mysql> GRANT ALL on *.* TO 'root'@'%' IDENTIFIED by 'P@ssw0rd' with GRANT option;
  mysql> GRANT ALL on *.* TO 'root'@'localhost' IDENTIFIED by 'P@ssw0rd' with GRANT option;
  mysql> GRANT ALL ON *.* TO 'tungsten'@'127.0.0.1' IDENTIFIED BY 'P@ssw0rd' WITH GRANT OPTION;
  mysql> GRANT ALL ON *.* TO 'tungsten'@'localhost' IDENTIFIED BY 'P@ssw0rd' WITH GRANT OPTION;
  mysql> GRANT ALL ON *.* TO 'tungsten'@'db.grennan.com' IDENTIFIED BY 'P@ssw0rd' WITH GRANT OPTION;
  mysql> GRANT ALL ON *.* TO 'tungsten'@'db2.grennan.com' IDENTIFIED BY 'P@ssw0rd' WITH GRANT OPTION;
  mysql> GRANT ALL ON *.* TO 'tungsten'@'db3.grennan.com' IDENTIFIED BY 'P@ssw0rd' WITH GRANT OPTION;
  mysql> flush privileges;

Now that we can access MySQL from any of the servers we need to prepare each as a master and replication.  We can fix DB1 and then copy all of the data tables to the other servers.

 mysql> reset master; reset slave;
  mysql> exit
# service mysql stop
# rsync -rog --delete /data/mysql  root@db2:/data
# rsync -rog --delete /data/mysql  root@db3:/data

Remember we put the my.cnf file in the /data/mysql directory.  We need to make sure each has a unique server_id number.

# vi /etc/my.cnf
 server_id = 1
# ssh db2
# vi /etc/my.cnf
 server_id = 2
# exit
# ssh db3
 server_id = 3
# exit

Rather than type the password for MySQL each type we connect.  We can also setup a private my.cnf to prevent this. This should connect you to the master (RW) server from each host.

# vi ~/.my.cnf

[Client]
 user=root
 password=P@ssw0rd
 host=db.grennan.com
 socket=/data/mysql/mysql.sock

# scp .my.cnf db2:.
# scp .my.cnf db3:.

Next up,  Configuring Heartbeat and making failover work!

Tweet

You could do a full dump, drop everything and wait for the requests to restore. (This might be OK on a test database.)

We could ask MySQL for all the databases that have tables that have been updated in the last 30 days.

$ mysql information_schema -e  \
"select DISTINCT TABLE_SCHEMA as NAME from TABLES
  WHERE UPDATE_TIME IS NOT NULL and UPDATE_TIME > NOW() - INTERVAL 30 DAY"  \
> Databases

But this only gives a list of databases that have changed.  What about tables that are only read from?

Bin-Logs don’t help because they too only have updates.  What we really need is a long snapshot of the general log. But, starting the general log will just fill your disk space even more.

As long as all queries are remote we can use tcpdump.

Filter all the MySQL connections for  ‘FROM tablenames’.

$ tcpdump -i eth1 -s 0 -l -w - dst port 3306 | strings | \
 sed -n 's/\(.*\)\(from.*\)/\2/p' | awk '{print $2}' > TableList
 $ cat TableList | sort | uniq > ShortList

Some of the queries may reference databases.tablename so we can filter out the database names and add them to your Database list.

$ cat TableList | sed -n 's/\./ /p' | awk '{print $1}' | sort | uniq >> Databases

Now for the queries that only use table names. We need to find the database each belongs to.  This isn’t completely accurate because the table name may be in more then one databases.

$  for x in `cat TableList | sort | uniq` ;
 do
 mysql information_schema -e "select TABLE_SCHEMA from TABLES where TABLE_NAME='$x'\G"  \
 | grep TABLE | awk '{print $2}' ;
 done | sort | uniq >> Databases

To create the final list of database that should not be dropped.

$ sort Databases | uniq > KEEPTHESE ; cat KEEPTHESE

Now you can do the deed with some confidence your not removing anything being used.

P.S.  Did I tell you, Never do anything you can’t reverse.   Make a BACKUP FIRST!

Tweet

In Part 1 I when through the process of preparing a number of CentOS 5.6 servers. Now make the services they’ll preform more stable.

High Availability (HA)

I’ll be presenting two ways to provide redundant data and high available services. First, Pacemaker – with DRDB will duplicate your data at the disk partition level and watch for failures. Should the hardware failure, Pacemaker will take all the needed steps to start MySQL on the Hot Stand By (HSB). This is not perfect. Should someone run ‘rm *’ or drop a database DRDB will duplicate the loss on the HSB.

In another part, I’ll use Tungsten replicator. It offers a set of features that surpass the built-in MySQL replicator. The community version of Tungsten has global transaction IDs so even with many slaves, global transactions IDs make turning a slave into the master easy. Tungsten replicator is not a HA service. You have to manually fail to a new master. Tungsten Enterprise (if you have money) solves all the issues. With the tools the enterprise version supplies you can easily migrate the “master” server to any slave.

HA with Pacemaker:

Neither RedHat or CentOS supply PaceMaker packages. Redhat support their own propitiatory clustering suite. CentOS is suck trying to maintain compatibility with Redhat while still giving you a high availability system. CentOS does supply heartbeat and openais but not pacemaker. Thankful Redhat helps out by supporting the Fedora project and in turn Fedora provide an EPEL repository for Redhat 5.

The Pacemaker packages in Fedora’s EPEL directories are built against some additional packages that don’t exist on vanilla RHEL/CentOS installs. For more information on EPEL, see http://fedoraproject.org/wiki/EPEL/FAQ So before installing Pacemaker, you will first need to tell the machine how to find the EPEL packages Pacemaker depends on. To do this, download and install the EPEL package that matches your RHEL/CentOS version.

LINBIT is the primary maintainer of DRBD and offers a product and service portfolio for exactly what we are building here. They have produced a video that takes you through this same process using the the DRDB Console Manager. I’m going to take you through the same process by hand. I hope this way you will better understand the management touch points.

DRBD Installation:

I build two computer alike in every way (clones) and use DRBD to sync the data partitions on each. CentOS provides the packages needed.

 # wget http://www.clusterlabs.org/rpm/epel-5/clusterlabs.repo  
 # mv clusterlabs.repo /etc/yum.repos.d  
 # rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm  
 # yum -y install pacemaker.x86_64 heartbeat.x86_64 drbd83.x86_64 kmod-drbd83.x86_64
 # /sbin/chkconfig --add heartbeat  
 # depmod  
 # modprobe drbd

DRBD Configuration:

On both machines (db1 and db2), edit the DRBD configuration file. The host names must be the save as returned by ‘hostname’ command. You may also need to edit the host name in /etc/sysconfig/network. I have highlighted parts you will need to edit in red.

To replace the shared-secret, select at least half of the characters from the “63 random alpha-numeric characters” from Gibson Research Ultra High Security Passwords.

 # vi /etc/drbd.conf

 include "drbd.d/global.conf";
 include "drbd.d/*.res";

# vi /etc/drbd.d/global.conf

global { usage-count yes; }
common { startup { degr-wfc-timeout 0; }
net { cram-hmac-alg sha1; shared-secret R4x2alEkxtIg2kzbXqUL6l4uoTI7Ab7Qt; }
disk { on-io-error detach; } }

# hostname
 db1.grennan.com

# vi /etc/drbd.d/db.res

resource db { protocol C; syncer { rate 10M; }
on db1.grennan.com { device /dev/drbd0; disk /dev/md3; address 192.168.4.1:7788; flexible-meta-disk internal; }
on db2.grennan.com { device /dev/drbd0; disk /dev/md3; address 192.168.4.2:7788; flexible-meta-disk internal; } }

 # scp -r /etc/drbd.d db2:/etc  # scp -r /etc/drbd.conf db2:/etc

Manage DRDB processes:

If the disk was formatted during the OS install you may need to erase the ext3 file system info on both DB1 and DB2.

 # umount /data  # dd if=/dev/zero of=/dev/md3 count=2048

Write the DRBD meta data on both DB1 and DB2.

# drbdadm create-md db

On >>> DB1 only <<<

  # drbdadm adjust db 
  # drbdsetup /dev/drbd0 primary -o
  # service drbd start

On DB2

  # service drbd start

Did you miss the >>> DB1 only <<< above?   If you did you’ll need to re-run ‘drbdsetup /dev/drbd0 primary -o’ on DB1.

WAIT until the sync process completes.

Back on DB1 create the file system and mount it.

  # mkdir /data 
  # mkfs -j /dev/drbd0
  # tune2fs -c -1 -i 0 /dev/drbd0
  # mkdir /data
  # mount -o rw /dev/drbd0 /data

Don’t forget to make the /data directory on DB2.

  # mkdir /data/mysql
  # cat /proc/drbd

version: 8.3.8 (api:88/proto:86-94)
 GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:09

1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
 ns:2433920 nr:0 dw:0 dr:2433920 al:0 bm:148 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:28530088
 [==>.................] sync'ed: 21.9% (27860/30236)M delay_probe: 757
 finish: 4:57:11 speed: 1,248 (6,404) K/sec

In the next post I’ll go through installing MySQL in preparation for configuring Pacemaker. Then I’ll show you how to test fail over.

In the next part I will go over setting up MySQL.

Tweet