What will this return?

select if((1 = true) and ( 1 is TRUE) and (NULL is not TRUE), TRUE, FALSE) as answer;

You should know,

NULL is never TRUE or FALSE but

NULL is always NOT TRUE or NOT FALSE.

NULL is a little like a vacuum.  Ancient Greek philosophers did not like to admit the existence of a vacuum, asking themselves “how can ‘nothing’ be something?”.    Does GOD exist in NULL?

If  select “” is NOT NULL; is TRUE what is “”?

select "" is NOT NULL;
+----------------+
| "" is NOT NULL |
+----------------+
|              1 |
+----------------+

Is the lack of an answer, an answer? Is the lack of datum, DATA?  Or should it be “”.  Should we record the blank spaces?

Given:

When should this happen?

for x = 1 to X;
INSERT INTO `test`(`data`) VALUES ( NULL);
next x;

Null happens.

Tweet

Tweet

MySQL doesn’t often crash, but, if you use MySQL on a production system you should have High Availability.

Maybe you’ve not heard of MHA.  Yoshinori Matsunobu only released it in July of this year (2011).  I’ve been reviewing it and I think you should too. There are a lot of people chasing after this “holy grail” and most systems are complex and / or hard to recover when they fail. MHA simple and easy to use.

MHA is a fail over tool. It’s designed to fail a master and promote a slave to a new master. (This is simply to say but you should read how it’s done.) It can monitor your master or you can manually fail over.  Failing back doesn’t happen. Your expected to fix the failed master and turn it into a slave.  The monitoring program (masterha_manager) will stop running after a fail over.

I’m not going to give details on how MHA works. I’ll assume you have a master and some number of slaves already working.

Your slave/s server should be ready to become masters. This means they have sever_id and bin-log set.  You should be able to run ‘show master status’ on your slave/s and get an answer. Slaves should also be set with read_only=1 and realy_log_purge=0;

Preparing the way 

Download the latest version of MHA from the Google Code server. http://code.google.com/p/mysql-master-ha/downloads/list

Every MySQL server needs to have a copy of the MHA node installed.

 yum install perl-DBD-MySQL.x86_64 

 rpm -i mha4mysql-node-0.52-0.noarch.rpm

The Manager

The Manageris a little more difficult.  It should be run on a system outside your cluster.  Should the Manager process fail your HA process fails with it.  (Quis custodiet ipsos custodes?)

There are a number of required Perl modules.  You may find you have to use CPAN to install these. CPAN will require you install perl-YAML, make and gcc.  (I found this the easiest for CentOS 6.2)  If you use CPAN you will need to tell RPM to skip dependencies.

• DBD::mysql
• Config::Tiny
• Log::Dispatch
• Parallel::ForkManager
• Time::HiRes

 rpm -i mha4mysql-manager-0.52-0.noarch.rpm --nodeps

This may not be needed. On some systems, because I installed on a x64 version (as you should be too) I found I had to copy the MHA libraries to the perl lib64 space.  You might want to make simlinks instead.

  cp -crp  /usr/lib/perl5/vendor_perl/MHA  /usr/lib64/perl5/vendor_perl

You only need one configuration file if you are not going to monitor multiple Master/Slave clusters.  You can name this configuration file anything you want.  Change these setting and make sure the directories exist and are writable.

On the server where the Manager will be running:

  vi /etc/MHA.cnf

[server default]
user=hauser
password=P@ssw0rd
manager_workdir=/var/log/masterha
manager_log=/var/log/masterha/MHA.log
remote_workdir=/var/log/masterha

[server1]
hostname=db1

[server2]
hostname=db2

The MHA will need access to each server so you need to add the user to each.  On all MySQL server add the hauser.

 mysql> grant all on *.* to 'hauser'@'192.168.%.%'  identified by 'P@ssw0rd';  mysql> flush privileges; 

Now is a good time to make sure all the server names resolve on each system and they know their own names.

MHA also need ssh access to each server. You need to create ssh keys and copy to the mysql servers.

Don’t do these steps if you already have ssh keys installed.

 ssh-keygen -t dsa -f ~/.ssh/id_dsa -N ""  

 cp ~/.ssh/id_dsa.pub ~/.ssh/authorized_keys

 scp -r ~/.ssh db2:

Check if you have ssl working with the masterha_check_ssh tool.  Look for the OK after the connect test.

From the Manager server:

 masterha_check_ssh  --conf=/etc/MHA.cnf

Thu Dec  1 10:10:52 2011 - [info] Starting SSH connection tests..
Thu Dec  1 10:10:53 2011 - [debug]
Thu Dec  1 10:10:52 2011 - [debug]  Connecting via SSH from root@db1(192.168.0.11) to root@db2(192.168.2.12)..
Thu Dec  1 10:10:53 2011 - [debug]   ok.

MHA need to control replication so you should test it with masterha_check_repl. This will outout a lot of chat.  Your looking for the ‘MySQL Replication Health is OK’ at the end.   You might also pay attention to any warnings.

 masterha_check_repl  --conf=/etc/MHA.cnf

---
Thu Dec  1 10:07:22 2011 - [info] Checking slave configurations..
Thu Dec  1 10:07:22 2011 - [warning]  read_only=1 is not set on slave db2(192.168.0.11:3306).
Thu Dec  1 10:07:22 2011 - [warning]  relay_log_purge=0 is not set on slave db2(192.168.0.12:3306).
---
MySQL Replication Health is OK.

You now have MHA installed.  If you configure it wrong MHA will give some errors that are not helpful and die. Check your settings again.

If you want MHA to monitor the status of your master server you need to run masterha_manager.  It does not normally run as a daemon.  If masterha_manage is not running your server will not automatically fail over.  This is a weakness in MHA.  You might want to use something like Pacemaker(Heartbeat) to monitor your systems and MHA to do the fail over.  Here is how I make MHA run in the background.

 nohup  masterha_manager  --conf=/etc/MHA.cnf &

You can check masterha_manager with masterha_check_status.

 masterha_check_status  --conf=/etc/MHA.cnf

Now the magic. If you want to manual fail over you can use masterha_master_switch and tell it the master is alive.

 masterha_master_switch  --master_state=alive --conf=/etc/MHA.cnf

If your using another monitoring system like Pacemaker you can have it run this command to automatically fail your master.

 masterha_master_switch --master_state=dead --dead_master_host=db1 --conf=/etc/MHA.cnf

If you find I’ve missed something or it works different on your OS please drop me a comment.

Tweet

Most existing examples I found  setup SSL authentication and encryption.  If you are handling  PCI DSS or HIPAA data you must encrypt the data on the wire, but passwords are all you need to authenticate the application to the data source.

 # mkdir /etc/mysql/certs
 # cd /etc/mysql/certs

This looks complicated, it’s not.  JUST PRESS ENTER when openssl ask you a question.  This will not make you any less secure.  Your data will be encrypted.   Note these keys will expire in three years (1095 days).

 # openssl genrsa 2048 > ca-key.pem
 # openssl req -new -x509 -nodes -days 1095 -key ca-key.pem -out ca-cert.pem
 # openssl req -newkey rsa:2048 -days 1095 -nodes -keyout server-key.pem -out server-req.pem
 # openssl rsa -in server-key.pem -out server-key.pem
 # openssl x509 -req -in server-req.pem -days 1095 -CA ca-cert.pem -CAkey ca-key.pem \
 -set_serial 01 -out server-cert.pem

With the keys generated you need to tell MySQL to use them. Add these lines to your my.cnf.

 # vi /etc/my.cnf

 ssl
 ssl-cipher=DHE-RSA-AES256-SHA
 ssl-ca=/etc/mysql/certs/ca-cert.pem
 ssl-cert=/etc/mysql/certs/server-cert.pem
 ssl-key=/etc/mysql/certs/server-key.pem

Now restart mysql.

 # service mysql restart

You should see SSL is enabled and mysql sees the keys.

 mysql> show variables like '%ssl%';
 +---------------+----------------------------------+
 | Variable_name | Value                            |
 +---------------+----------------------------------+
 | have_openssl  | YES  |
 | have_ssl      | YES  |
 | ssl_ca        | /etc/mysql/certs/ca-cert.pem     |
 | ssl_capath    |                                  |
 | ssl_cert      | /etc/mysql/certs/server-cert.pem |
 | ssl_cipher    | DHE-RSA-AES256-SHA               |
 | ssl_key       | /etc/mysql/certs/server-key.pem  |
 +---------------+----------------------------------+
 7 rows in set (0.00 sec)

Duplicate these keys and the configuration segment to all your servers.

Client

If the client system is not a server, copy of the MySQL SSL keys to it was well.  Then, you need to tell the client to use SSL.  Edit your user’s .my.cnf file and give it the keys too.

 # vi ~/.my.cnf

[client]
 ssl
 ssl-cipher=DHE-RSA-AES256-SHA
 ssl-ca=/etc/mysql/certs/ca-cert.pem

Thats it.  It should be working.  Look for ‘Cipher in use’.

 # mysql -e "\s"

mysql Ver 14.14 Distrib 5.1.59, for unknown-linux-gnu (x86_64) using readline 5.1

 Connection id: 7
 Current database:
 Current user: root@localhost
 SSL: Cipher in use is DHE-RSA-AES256-SHA
 ......
 Threads: 1 Questions: 22 Slow queries: 0 Opens: 15 Flush tables: 1 Open tables: 8 Queries per second avg: 2.0

Replication

On the master, you need to tell the replication user to require SSL connections. Replace the rep_user with your replication user’s ID.

 mysql> GRANT USAGE ON *.* TO 'rep_user'@'%'  REQUIRE SSL;
 mysql> flush privileges;

and on the slave tell it to connect to the master with SSL.  Then make sure you are still connecting. Change the IP and user name, show here,  to your settings.

 mysql> stop slave;
 mysql> CHANGE MASTER TO master_host='192.168.1.12', master_user='rep_user', \
MASTER_SSL=1, MASTER_SSL_CA = '/etc/mysql/certs/ca-cert.pem' ;
 mysql> start slave;
 mysql> show slave status;

Applications

Your application could be written in lots of different languages and I can’t go over each of them but here are some links to setup up SSL connection to MySQL in several of the most popular.  If you know where there are some better examples, please leave me a comment.

Python

Perl

JDBC

Tweet

This is more boring parts but you must lay a good foundation.

MySQL Setup:

All the resources must be in place before you configure Pacemaker. The default location for MySQL data is /var/lib/mysql.  You will be moving this to the DRBD file system in /data/mysql.   The default MySQL configuration is in /etc/my.cnf.  I move it to /data/mysql so updates will move with system fail overs.   To keep life simple I use sym-links to point from /var/lib/mysql and /etc/my.cnf to their new locations.
Install MySQL of your choise on DB1, DB2 and DB3.

# rpm -Uhv http://www.percona.com/downloads/percona-release/percona-release-0.0-1.x86_64.rpm
# yum -y install Percona-Server-server-51.x86_64 Percona-Server-client-51.x86_64 \
 Percona-Server-shared-51.x86_64 xtrabackup.x86_64
# service mysql start
# mysql -e "CREATE FUNCTION fnv1a_64 RETURNS INTEGER SONAME 'libfnv1a_udf.so'"
# mysql -e "CREATE FUNCTION fnv_64 RETURNS INTEGER SONAME 'libfnv_udf.so'"
# mysql -e "CREATE FUNCTION murmur_hash RETURNS INTEGER SONAME 'libmurmur_udf.so'"
# service mysql stop
# chkconfig mysql off

ON DB1, move MySQL data to the DRBD file system.

# df

Filesystem        1K-blocks   Used Available Use% Mounted on
 /dev/md2           11903664      1633608   9655620   15% /
 /dev/md1             505508        218159    261250    46% /tmp
 /dev/md0             256586         25967    217371    11% /boot
 tmpfs               2037412  16348   2021064    1% /dev/shm
 /dev/drbd1         30477552 176200  28753152    1% /data

On DB1 and DB3

# mkdir /data/mysql
# cp -prv /var/lib/mysql/* /data/mysql
# mv /var/lib/mysql /var/lib/mysql-empty
# ln -s /data/mysql /var/lib/mysql
# touch /data/mysql/my.cnf
# ln -sf /data/mysql/my.cnf /etc/my.cnf
# chown -R mysql.mysql /data/mysql
# chmod 644 /data/mysql/my.cnf

On DB2, make it ready for a fail over.

# mv /var/lib/mysql /var/lib/mysql-empty
# ln -sf /data/mysql /var/lib/mysql
# ln -sf /data/mysql/my.cnf /etc/my.cnf

Configure MySQL (/data/mysql/my.cnf)

I have a documented my.cnf file I start with. You need to change the server-id number on each server.  The minimum setting you will need are:

[mysqld]
 log-bin=mysql-bin
 server-id=1
 innodb_flush_log_at_trx_commit=2
 sync_binlog=1

on DB1

# service mysql start

Now we need to add a couple of user before we install the replicator.  We also need to set a password for the root user.  The root password is blank so just it return.

If you have skip-name-resolve set you will need to substitute the host names for IP addresses.

# mysql -h localhost -u root -p
  mysql> DROP USER ''@'db.grennan.com';
  mysql> DROP USER ''@'localhost';
  mysql> GRANT ALL on *.* TO 'root'@'%' IDENTIFIED by 'P@ssw0rd' with GRANT option;
  mysql> GRANT ALL on *.* TO 'root'@'localhost' IDENTIFIED by 'P@ssw0rd' with GRANT option;
  mysql> GRANT ALL ON *.* TO 'tungsten'@'127.0.0.1' IDENTIFIED BY 'P@ssw0rd' WITH GRANT OPTION;
  mysql> GRANT ALL ON *.* TO 'tungsten'@'localhost' IDENTIFIED BY 'P@ssw0rd' WITH GRANT OPTION;
  mysql> GRANT ALL ON *.* TO 'tungsten'@'db.grennan.com' IDENTIFIED BY 'P@ssw0rd' WITH GRANT OPTION;
  mysql> GRANT ALL ON *.* TO 'tungsten'@'db2.grennan.com' IDENTIFIED BY 'P@ssw0rd' WITH GRANT OPTION;
  mysql> GRANT ALL ON *.* TO 'tungsten'@'db3.grennan.com' IDENTIFIED BY 'P@ssw0rd' WITH GRANT OPTION;
  mysql> flush privileges;

Now that we can access MySQL from any of the servers we need to prepare each as a master and replication.  We can fix DB1 and then copy all of the data tables to the other servers.

 mysql> reset master; reset slave;
  mysql> exit
# service mysql stop
# rsync -rog --delete /data/mysql  root@db2:/data
# rsync -rog --delete /data/mysql  root@db3:/data

Remember we put the my.cnf file in the /data/mysql directory.  We need to make sure each has a unique server_id number.

# vi /etc/my.cnf
 server_id = 1
# ssh db2
# vi /etc/my.cnf
 server_id = 2
# exit
# ssh db3
 server_id = 3
# exit

Rather than type the password for MySQL each type we connect.  We can also setup a private my.cnf to prevent this. This should connect you to the master (RW) server from each host.

# vi ~/.my.cnf

[Client]
 user=root
 password=P@ssw0rd
 host=db.grennan.com
 socket=/data/mysql/mysql.sock

# scp .my.cnf db2:.
# scp .my.cnf db3:.

Next up,  Configuring Heartbeat and making failover work!

Tweet

You could do a full dump, drop everything and wait for the requests to restore. (This might be OK on a test database.)

We could ask MySQL for all the databases that have tables that have been updated in the last 30 days.

$ mysql information_schema -e  \
"select DISTINCT TABLE_SCHEMA as NAME from TABLES
  WHERE UPDATE_TIME IS NOT NULL and UPDATE_TIME > NOW() - INTERVAL 30 DAY"  \
> Databases

But this only gives a list of databases that have changed.  What about tables that are only read from?

Bin-Logs don’t help because they too only have updates.  What we really need is a long snapshot of the general log. But, starting the general log will just fill your disk space even more.

As long as all queries are remote we can use tcpdump.

Filter all the MySQL connections for  ‘FROM tablenames’.

$ tcpdump -i eth1 -s 0 -l -w - dst port 3306 | strings | \
 sed -n 's/\(.*\)\(from.*\)/\2/p' | awk '{print $2}' > TableList
 $ cat TableList | sort | uniq > ShortList

Some of the queries may reference databases.tablename so we can filter out the database names and add them to your Database list.

$ cat TableList | sed -n 's/\./ /p' | awk '{print $1}' | sort | uniq >> Databases

Now for the queries that only use table names. We need to find the database each belongs to.  This isn’t completely accurate because the table name may be in more then one databases.

$  for x in `cat TableList | sort | uniq` ;
 do
 mysql information_schema -e "select TABLE_SCHEMA from TABLES where TABLE_NAME='$x'\G"  \
 | grep TABLE | awk '{print $2}' ;
 done | sort | uniq >> Databases

To create the final list of database that should not be dropped.

$ sort Databases | uniq > KEEPTHESE ; cat KEEPTHESE

Now you can do the deed with some confidence your not removing anything being used.

P.S.  Did I tell you, Never do anything you can’t reverse.   Make a BACKUP FIRST!

Tweet

In Part 1 I when through the process of preparing a number of CentOS 5.6 servers. Now make the services they’ll preform more stable.

High Availability (HA)

I’ll be presenting two ways to provide redundant data and high available services. First, Pacemaker – with DRDB will duplicate your data at the disk partition level and watch for failures. Should the hardware failure, Pacemaker will take all the needed steps to start MySQL on the Hot Stand By (HSB). This is not perfect. Should someone run ‘rm *’ or drop a database DRDB will duplicate the loss on the HSB.

In another part, I’ll use Tungsten replicator. It offers a set of features that surpass the built-in MySQL replicator. The community version of Tungsten has global transaction IDs so even with many slaves, global transactions IDs make turning a slave into the master easy. Tungsten replicator is not a HA service. You have to manually fail to a new master. Tungsten Enterprise (if you have money) solves all the issues. With the tools the enterprise version supplies you can easily migrate the “master” server to any slave.

HA with Pacemaker:

Neither RedHat or CentOS supply PaceMaker packages. Redhat support their own propitiatory clustering suite. CentOS is suck trying to maintain compatibility with Redhat while still giving you a high availability system. CentOS does supply heartbeat and openais but not pacemaker. Thankful Redhat helps out by supporting the Fedora project and in turn Fedora provide an EPEL repository for Redhat 5.

The Pacemaker packages in Fedora’s EPEL directories are built against some additional packages that don’t exist on vanilla RHEL/CentOS installs. For more information on EPEL, see http://fedoraproject.org/wiki/EPEL/FAQ So before installing Pacemaker, you will first need to tell the machine how to find the EPEL packages Pacemaker depends on. To do this, download and install the EPEL package that matches your RHEL/CentOS version.

LINBIT is the primary maintainer of DRBD and offers a product and service portfolio for exactly what we are building here. They have produced a video that takes you through this same process using the the DRDB Console Manager. I’m going to take you through the same process by hand. I hope this way you will better understand the management touch points.

DRBD Installation:

I build two computer alike in every way (clones) and use DRBD to sync the data partitions on each. CentOS provides the packages needed.

 # wget http://www.clusterlabs.org/rpm/epel-5/clusterlabs.repo  
 # mv clusterlabs.repo /etc/yum.repos.d  
 # rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm  
 # yum -y install pacemaker.x86_64 heartbeat.x86_64 drbd83.x86_64 kmod-drbd83.x86_64
 # /sbin/chkconfig --add heartbeat  
 # depmod  
 # modprobe drbd

DRBD Configuration:

On both machines (db1 and db2), edit the DRBD configuration file. The host names must be the save as returned by ‘hostname’ command. You may also need to edit the host name in /etc/sysconfig/network. I have highlighted parts you will need to edit in red.

To replace the shared-secret, select at least half of the characters from the “63 random alpha-numeric characters” from Gibson Research Ultra High Security Passwords.

 # vi /etc/drbd.conf

 include "drbd.d/global.conf";
 include "drbd.d/*.res";

# vi /etc/drbd.d/global.conf

global { usage-count yes; }
common { startup { degr-wfc-timeout 0; }
net { cram-hmac-alg sha1; shared-secret R4x2alEkxtIg2kzbXqUL6l4uoTI7Ab7Qt; }
disk { on-io-error detach; } }

# hostname
 db1.grennan.com

# vi /etc/drbd.d/db.res

resource db { protocol C; syncer { rate 10M; }
on db1.grennan.com { device /dev/drbd0; disk /dev/md3; address 192.168.4.1:7788; flexible-meta-disk internal; }
on db2.grennan.com { device /dev/drbd0; disk /dev/md3; address 192.168.4.2:7788; flexible-meta-disk internal; } }

 # scp -r /etc/drbd.d db2:/etc  # scp -r /etc/drbd.conf db2:/etc

Manage DRDB processes:

If the disk was formatted during the OS install you may need to erase the ext3 file system info on both DB1 and DB2.

 # umount /data  # dd if=/dev/zero of=/dev/md3 count=2048

Write the DRBD meta data on both DB1 and DB2.

# drbdadm create-md db

On >>> DB1 only <<<

  # drbdadm adjust db 
  # drbdsetup /dev/drbd0 primary -o
  # service drbd start

On DB2

  # service drbd start

Did you miss the >>> DB1 only <<< above?   If you did you’ll need to re-run ‘drbdsetup /dev/drbd0 primary -o’ on DB1.

WAIT until the sync process completes.

Back on DB1 create the file system and mount it.

  # mkdir /data 
  # mkfs -j /dev/drbd0
  # tune2fs -c -1 -i 0 /dev/drbd0
  # mkdir /data
  # mount -o rw /dev/drbd0 /data

Don’t forget to make the /data directory on DB2.

  # mkdir /data/mysql
  # cat /proc/drbd

version: 8.3.8 (api:88/proto:86-94)
 GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:09

1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
 ns:2433920 nr:0 dw:0 dr:2433920 al:0 bm:148 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:28530088
 [==>.................] sync'ed: 21.9% (27860/30236)M delay_probe: 757
 finish: 4:57:11 speed: 1,248 (6,404) K/sec

In the next post I’ll go through installing MySQL in preparation for configuring Pacemaker. Then I’ll show you how to test fail over.

In the next part I will go over setting up MySQL.

Tweet

THIS DOCUMENT IS BEING UPDATED – PLEASE WATCH FOR CHANGES!

After years of supporting MySQL, for many different companies, I’ve seen this story played out again and again.
The company:

• chooses a Database Management System (MySQL)
• installs the DBMS on a computer with other processes
• writes many programs to access the data (Without concern on how the queries are written.)
• moves DBMS to a computer of its own and writes more programs
• buy bigger computer to run the DBMS and writes more programs
• tires of DBMS response times and outages caused by developers working on production systems and hires a Database Administrator to fix the mess

This is a step by step description of  how I build a highly available, production MySQL servers. Like most things it life, these problems can be avoided with a little extra work at the start.

My first goal is to create a MySQL DBMS that with 99.999% up time. MySQL can distribute read requests but not writes. This make write service a single point of failure. To fix this you can either turn a slave server into the master or provide a hot standby to become the master.

My second goal is scalability. This is done by creating more read only slave servers. More slaves complicate turning one of them into the master should the master fail. The enterprise version Continuent’s Tungsten replicator fixes this. But, my goal is to do this on the cheep for start-ups while providing flexibility for the future. With DRBD and Pacemaker I can create a hot standby for the master server.

I have chosen these programs and utilities because they are free. Enterprise support is available for each.

Here is what I’m using:

• Scientific Linux 6.1
• DRBD disk replication for a hot standby
• MySQL 5.1.xx (Percona) or MeriaDB
• Heartbeat
• Pacemaker for failure recover to the hot standby
• Tunsten data replication
• MaaKit a toolkit for users, developers, and administrators of open-source databases
• OpenArk Kit – a set of utilities for MySQL
• xtrabackup for database backups
• mydumper for quick dumps
• MySQL Sandbox for testing
• mytop and innotop
• mysqlsniffer

I’ve worked hard to make this just cut and past. I enjoyed the work. I hope you do to.

NOTE:  Type the GREEN stuff, cut and past the BLUE stuff and edit the RED stuff to fit you needs.

Getting started:

Every good system starts with good hardware. The two thinks database servers hunger for are disk space and memory. You should supply your self with as much as you can afford.  Most of my production system run on 32 gig of memory and RAID-10 systems.  Two network ports or more is recommended.  One network port will be used with a crossover cable for the heartbeat function.

To test this installation I’m building on a VMware server.  If you’d like to know more about this hardware read my “Building a Home VMware server” post.

Building the Operating System – DB1

The operating system I’m using is CentOS 5.6 Scientfic Linux 6.1 64 bit.  You might choose Redhat 6.0. I’m being conservative and I’m trying to use free (I have no budget for this project) version of commercial products with enterprise support.

To help you understand the following instructions, for this example, I’m building a virtual machines (VM) with four (4) virtual SAS hard disks. I have split the four 15G virtual disks into four partitions.  The /boot and /tmp partitions are RAID-1 and the / (root) and /data partitions are RAID-5.  The sizes of these partitions depend you your needs but your /boot needs to be about 1G and the / needs to be about 12G.  I leave /data unassigned and un-formatted.  My layout looks like this.

Md0 - /boot    Md1 - /    Md2 – /tmp    Md3 - /data
Disk 0 - 15G    256M    4G    512M    11G
Disk 1 - 15G    256M    4G    512M    11G
Disk 2 - 15G    256M    4G    512M    11G
Disk 3 - 15G    256M    Checksum    512M    Checksum
Total    1G    12G    2G    33G

Install the OS:

Start with a “minimum installation” with as few applications installed as possible.   No desktop or server applications are needed.
The first step after the install is to update the installed packages.  With a minimum install you may find you need to fix a few things like your network connections.

# yum -y update

Even with a minimum install there is a little clean up.  I remove a few un-needed services like bluetooth, printing and there are a couple of packages we will need latter that where not installed.  It is better to install them now and avoid some dependency issues.  I remove supplied MySQL.

# rpm -e bluez-utils
# rpm -e smartmontools
# yum -y install ruby
# yum -y install perl-DBD-MySQL.x86_64
# yum -y install libdbi-dbd-mysql.x86_64
# rpm -e mysql --nodeps
# chkconfig iptables off
# chkconfig ip6tables off

After the OS is install and updated the disk looks like this.

# df

Filesystem 1K-blocks  Used Available Use% Mounted on

/dev/md2  11903664    1394068 9895160  13% /
/dev/md3  29995056    176200  28270608 1%  /data
/dev/md1  505508      10547   468862   3%  /tmp
/dev/md0  256586      22969   220369   10% /boot
tmpfs     2037380     0       2037380  0%  /dev/shm

Disable Security:

Because we move the MySQL data directy you will need to disable SELinux or update it.

To disable it, edit /etc/selinux/config and change the SELINUX line to SELINUX=disabled:

# vi /etc/selinux/config

SELINUX=disabled
# echo 0 >/selinux/enforce
# service iptables stop
# chkconfig iptables off
# service ip6tables stop
# chkconfig ip6tables off

If you don’t want to disable To update SELinux for the new data directory you will need to have the the selinux tools installed.

# yum -y install policycoreutils
# semanage fcontext -a -t mysqld_db_t "/data/mysql(/.*)?"
# restorecon -Rv /data/mysql

Network configuration:

# vi /etc/hosts
  192.168.2.22 db.grennan.com db   192.168.2.23 db1.grennan.com db1  192.168.2.24 db2.grennan.com db2   192.168.2.25 db3.grennan.com db3

Syncing Time:

Time Singularization is very important to maintaining data. You may want to edit the /etc/ntp.conf file to point to your primary NTP time server. CentOS and Redhat provide time servers for your use. I recommend using pool.ntp.org.

# yum install ntp
# chkconfig ntpd on
# ntpdate 0.pool.ntp.org
# service ntpd start

Building DB2/3

DB2 will become the hot standby server (drbd / heartbeat) and DB3 will  become the MySQL slave server (Tungsten).

If, during the OS install, you created and formatted the /data partition you will need to remove it from /etc/fstab.  On both DB1 and DB2,  edit /etc/fstab and remove the /data file system.

# vi /etc/fstab

/dev/md2                /                       ext3    defaults        1 1
/dev/md3                /data                   ext3    defaults        1 2
/dev/md1                /tmp                    ext3    defaults        1 2
/dev/md0                /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP-sdd3         swap                    swap    defaults        0 0
LABEL=SWAP-sdc3         swap                    swap    defaults        0 0
LABEL=SWAP-sdb3         swap                    swap    defaults        0 0
LABEL=SWAP-sda3         swap                    swap    defaults        0 0

Setup SSH:

# ssh-keygen -t dsa -f ~/.ssh/id_dsa -N ""
# cp ~/.ssh/id_dsa.pub ~/.ssh/authorized_keys
# scp -r ~/.ssh db2:

root@db2's password:
 id_dsa.pub         100%  610     0.6KB/s   00:00  id_dsa                   100%  668     0.7KB/s   00:00
 authorized_keys    100%  610     0.6KB/s   00:00  known_hosts              100%  398     0.4KB/s   00:00

Visualization

I can’t stand the color choices made for BASH so I set my own.

# vi ~/.bash_profile
export LS_COLORS='no=00:fi=00:di=00;33:ln=00;36:pi=40;33:so=00;35:bd=40;33 \ 
01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00; \ 
32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00; \ 
31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00; \
31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00; \ 
35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35:'

Tweet

Start with CentOS 5.6 x86_64 basic install.

Add the PostgreSQL Yum repository to your system.

 $ wget http://yum.pgrpms.org/reporpms/9.0/pgdg-centos-9.0-2.noarch.rpm
 $ rpm -i pgdg-centos-9.0-2.noarch.rpm

Another location for these is DAG. I have to tried these so your results may very.

You will need to exclude the packages CentOS provide by added two lines to the BASE and UPDATE sections of /etc/yum.repos.d/CentOS-Base.repo. They are:

exclude=postgresql*
exclude=geos*

You you are ready to install the needed packages. This includes proj version 4 and geos version 3.

 $ yum install postgresql90-contrib.x86_64
 $ yum install postgis90.x86_64
 $ yum install postgresql90-server
 $ yum install proj
 $ yum install geos
 $ yum install php-pear
 $ yum install php-devel

GEOS

There may be some dependencies you will have to work through. If you are using PDO and haven’t installed the pgsql PDO drivers, you can do it now:

 $ pecl install pdo_pgsql

PL/pgSQL

You you are ready to initialize the database files and start postgresql.

 $ service postgresql-9.0 initdb
 $ service postgresql-9.0 start

Now you can change to the postgres user and begin installing the functions for GIS. You have to start with defining the language.

 $ su – postgres
 $ psql
 # create language plpgsql ;
 # \q

Now you can create your database and add the GIS functions calls to it.

 $ createdb geos
 $ psql -d geos -f /usr/pgsql-9.0/share/contrib/postgis-1.5/postgis.sql
 $ psql -d geos -f /usr/pgsql-9.0/share/contrib/postgis-1.5/spatial_ref_sys.sql

You you can verify the install.

 $ psql geos
 # select postgis_full_version();
                                              postgis_full_version
——————————————————————————————————–
 POSTGIS=”1.5.2″ GEOS=”3.2.2-CAPI-1.6.2″ PROJ=”Rel. 4.7.1, 23 September 2009″ LIBXML=”2.6.26″ USE_STATS
(1 row)

For more on using PostGIS check out Jeremy’s “PostGIS part 2”.

.

Tweet

The bug number fixes are from SourceForge.  https://sourceforge.net/tracker/?atid=628964&group_id=101066&func=browse

# 2.5.5 MTG – (2011-07-21)
#    – Bug – Typo Ureadable Unreadable config file line 424 – ID: 3316825
#    – Bug – Change “#!/bin/bash” to “#!/usr/bin/env bash” – ID: 3292873
#    – Bug – problem with excludes – ID: 3169562
#    – Bug – Total disk space on symbolic links – ID: 3064547
#    – Added DEBUG option to only print the commands that will be executed.
#    – Bug – WHICH command didn’t work if there was a WHICH alias.
# VER 2.5.4: MTG – (2011-01-28)
#    – fixed bug in rsync process.
#    – Added the ability to backup only a single table by naming the table like
#      database.table
# VER 2.5.2-02:  MTG – (2010-12-29)
#    – Added file promission settings (chmod) to directory and file creation points.
# VER 2.5.2-01:  MTG – (2010-11-06)
#    – Added ‘-R’ to the mysqldump options to include stored procedures in the backup
#      by default. (suggested by Zack Evans)
# VER 2.5.2:  MTG – (2010-11-04)
#    – Added option to archive (rsync) the local backup files to a remote locations
#      using the COPYDIR varaible.
#    – Added option to copy files into a directory based on the hostname using the
#      variable HOSTNAME.  This allows the script to be run from a shared storage directory
#      ( SBM, NFS, NetApp) the data to be kept seperate places.
#    – Added option to backup all database schemas only using variable FULLSCHEMA.
#    – Added option to backup MySQL configuration file, my.cnf and remove files older then seven
#      days from the BACKUPDIR directory.
#    – Added –master-data=2 and –single-transaction to include a comment with the master server’s
#      the binary log coordinates. If used the CHANGE_MASTER_TO line must be uncommented.

The project is at https://sourceforge.net/projects/automysqlbackup/

You can download my version at http://www.mysqlfanboy.com/Files/automysqlbackup.sh

Email me or leave a comment if you have any trouble.

Tweet