<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>MySQL Fanboy &#187; Tips &amp; Tricks</title>
	<atom:link href="/category/tips-n-tricks/feed/" rel="self" type="application/rss+xml" />
	<link>http://www.mysqlfanboy.com</link>
	<description>Almost crazy about Opensource / Free  information.</description>
	<lastBuildDate>Sun, 01 Aug 2010 18:50:18 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.0.1</generator>
		<item>
		<title>Does Size or Type Matter?</title>
		<link>http://www.mysqlfanboy.com/2010/07/does-size-or-type-matter/</link>
		<comments>http://www.mysqlfanboy.com/2010/07/does-size-or-type-matter/#comments</comments>
		<pubDate>Tue, 27 Jul 2010 18:08:16 +0000</pubDate>
		<dc:creator>mark</dc:creator>
				<category><![CDATA[Tips & Tricks]]></category>
		<category><![CDATA[Tunning]]></category>
		<category><![CDATA[Data]]></category>
		<category><![CDATA[MySQL]]></category>
		<category><![CDATA[Profilling]]></category>
		<category><![CDATA[Query]]></category>
		<category><![CDATA[Tips]]></category>

		<guid isPermaLink="false">http://www.mysqlfanboy.com/?p=238</guid>
		<description><![CDATA[MySQL seems to be happy to convert types for you. Developers are rushed to complete their project and if the function works they just move on. But what is the costs of mixing your types? Does it matter if your are running across a million rows or more? Lets find out. Here is what the [...]]]></description>
			<content:encoded><![CDATA[<p>MySQL seems to be happy to convert types for you.  Developers are rushed to complete their project and if the function works they just move on.  But what is the costs of mixing your types?  Does it matter if your are running across a million rows or more?  Lets find out.</p>
<p>Here is what the programmers see.</p>
<pre style="padding-left: 30px;">mysql&gt; select <span style="color: #ff0000;"><strong>1+1</strong></span>;
+-----+
| 1+1 |
+-----+
|   2 |
+-----+
1 row in set (<span style="color: #008000;">0.00 sec</span>)

mysql&gt; select <span style="color: #ff0000;"><strong>"1"+"1"</strong></span>;
+---------+
| "1"+"1" |
+---------+
|       2 |
+---------+
1 row in set (<span style="color: #008000;">0.00 sec</span>)</pre>
<p><strong>Benchmark</strong></p>
<p>What if we do a thousand simple loops?  How long does the looping itself take?<strong> </strong></p>
<p>The <a href="http://dev.mysql.com/doc/refman/5.0/en/information-functions.html#function_benchmark"><code>BENCHMARK()</code></a> function           executes the expression <em><code>expr</code></em> repeatedly <em><code>count</code></em> times. It may be           used to time how quickly MySQL processes the expression. The           result value is always <code>0</code>.</p>
<pre>
<pre style="padding-left: 30px;">mysql&gt; select benchmark(1000000000, <span style="color: #ff0000;"><strong>1</strong></span>);
+--------------------------+
| benchmark(1000000000, 1) |
+--------------------------+
|                        0 |
+--------------------------+
1 row in set (<span style="color: #008000;">5.42 sec</span>)

mysql&gt; select benchmark(1000000000, <span style="color: #ff0000;"><strong>"1"</strong></span> );
+-----------------------------+
| benchmark(1000000000, "1" ) |
+-----------------------------+
|                           0 |
+-----------------------------+
1 row in set (<span style="color: #008000;">5.40 sec</span>)</pre>
</pre>
<p>So maybe type doesn&#8217;t matter? About five seconds just to loop but the type didn&#8217;t change it.   What if we add <span style="color: #ff0000;"><strong>1+&#8221;1&#8243;</strong></span>?</p>
<p><span id="more-238"></span></p>
<pre style="padding-left: 30px;">mysql&gt; select benchmark(1000000000, <span style="color: #ff0000;"><strong>1+1</strong></span>);
+----------------------------+
| benchmark(1000000000, 1+1) |
+----------------------------+
|                          0 |
+----------------------------+
1 row in set (<span style="color: #008000;">12.65 sec</span>)

mysql&gt; select benchmark(1000000000, <span style="color: #ff0000;"><strong>1+"1"</strong></span>);
+------------------------------+
| benchmark(1000000000, 1+"1") |
+------------------------------+
|                            0 |
+------------------------------+
1 row in set (<span style="color: #008000;">35.58 sec</span>)
<pre>mysql&gt; select benchmark(1000000000, <span style="color: #ff0000;"><strong>"1"+"1"</strong></span>);
+--------------------------------+
| benchmark(1000000000, "1"+"1") |
+--------------------------------+
|                              0 |
+--------------------------------+
1 row in set (<span style="color: #008000;">51.59 sec</span>)</pre>
<p>It looks like type does matter.  But does it always matter?</p>
<pre style="padding-left: 30px;">mysql&gt; select benchmark(1000000000, <span style="color: #ff0000;">sum(1+1)</span>);
+---------------------------------+
| benchmark(1000000000, sum(1+1)) |
+---------------------------------+
|                               0 |
+---------------------------------+
1 row in set (<span style="color: #008000;">9.69 sec</span>)

mysql&gt; select benchmark(1000000000, <span style="color: #ff0000;">sum("1"+"1")</span>);
+-------------------------------------+
| benchmark(1000000000, sum("1"+"1")) |
+-------------------------------------+
|                                   0 |
+-------------------------------------+
1 row in set (<span style="color: #008000;">9.94 sec</span>)

mysql&gt; select benchmark(1000000000, <span style="color: #ff0000;">sum("1.23456789"+"1.23456789")</span>);
+-------------------------------------------------------+
| benchmark(1000000000, sum("1.23456789"+"1.23456789")) |
+-------------------------------------------------------+
|                                                     0 |
+-------------------------------------------------------+
1 row in set (<span style="color: #008000;">10.32 sec</span>)</pre>
<p>So, not all functions are the same.  But it looks like size might matter!</p>
<pre style="padding-left: 30px;">mysql&gt; select benchmark(1000000000, <span style="color: #ff0000;">1.1+1.1</span>);
+--------------------------------+
| benchmark(1000000000, 1.1+1.1) |
+--------------------------------+
|                              0 |
+--------------------------------+
1 row in set (<span style="color: #008000;">34.90 sec</span>)

mysql&gt; select benchmark(1000000000, <span style="color: #ff0000;">"1.1"+"1.1"</span>);
+------------------------------------+
| benchmark(1000000000, "1.1"+"1.1") |
+------------------------------------+
|                                  0 |
+------------------------------------+
1 row in set (<span style="color: #008000;">1 min 15.32 sec</span>)

mysql&gt; select  benchmark(1000000000, <span style="color: #ff0000;">"1.123456789"+"1.123456789"</span>);
+----------------------------------------------------+
| benchmark(1000000000, "1.123456789"+"1.123456789") |
+----------------------------------------------------+
|                                                  0 |
+----------------------------------------------------+
1 row in set (<span style="color: #008000;">1 min 53.32 sec</span>)</pre>
<p>Sorry.  Looks like size does matter.<br />
This doesn't seem logical.</p>
<pre>mysql&gt; select benchmark(1000000000, <span style="color: #ff0000;">1=1</span>);
+----------------------------+
| benchmark(1000000000, 1=1) |
+----------------------------+
|                          0 |
+----------------------------+
1 row in set (<span style="color: #008000;">12.75 sec</span>)

mysql&gt; select benchmark(1000000000, <span style="color: #ff0000;">1="1"</span>);
+------------------------------+
| benchmark(1000000000, 1="1") |
+------------------------------+
|                            0 |
+------------------------------+
1 row in set (<span style="color: #008000;">40.78 sec</span>)
mysql&gt; select benchmark(1000000000, <span style="color: #ff0000;">1=true</span>);
+-------------------------------+
| benchmark(1000000000, 1=true) |
+-------------------------------+
|                             0 |
+-------------------------------+
1 row in set (<span style="color: #008000;">12.73 sec</span>)

mysql&gt; select benchmark(1000000000, <span style="color: #ff0000;">1="true"</span>);
+---------------------------------+
| benchmark(1000000000, 1="true") |
+---------------------------------+
|                               0 |
+---------------------------------+
1 row in set, <span style="color: #ff0000;">65535 warnings</span> (<span style="color: #008000;">3 min 5.72 sec</span>)
mysql&gt; select benchmark(1000000000, <span style="color: #ff0000;">"true"="true"</span>);
+--------------------------------------+
| benchmark(1000000000, "true"="true") |
+--------------------------------------+
|                                    0 |
+--------------------------------------+
1 row in set (<span style="color: #008000;">57.25 sec</span>)</pre>
<p>Maybe we should CAST our work?</p>
<pre style="padding-left: 30px;">mysql&gt; select benchmark(1000000000, <span style="color: #ff0000;">cast("1" as unsigned)</span>);
+----------------------------------------------+
| benchmark(1000000000, cast("1" as unsigned)) |
+----------------------------------------------+
|                                            0 |
+----------------------------------------------+
1 row in set (<span style="color: #008000;">32.27 sec</span>)

mysql&gt; select benchmark(1000000000, <span style="color: #ff0000;">cast("1" as unsigned) + cast("1" as unsigned)</span>);
+----------------------------------------------------------------------+
| benchmark(1000000000, cast("1" as unsigned) + cast("1" as unsigned)) |
+----------------------------------------------------------------------+
|                                                                    0 |
+----------------------------------------------------------------------+
1 row in set (<span style="color: #008000;">1 min 7.24 sec</span>)</pre>
</pre>
<p>Maybe not!<br />
Conclusion:  Be careful with your data types.  If you are taking user input, do the type conversion ONCE in your program.  Don&#8217;t let MySQL do the type conversions for you.<br />
query = &#8220;SELECT * FROM table where $INPUT = 1&#8243;;   could be doing your wrong.<br />
<img class="alignnone" src="http://mark.grennan.com/images/MarkGrennanSigniture.bmp" alt="" width="166" height="69" /></p>
<pre>References:

http://dev.mysql.com/doc/refman/5.0/en/information-functions.html#function_benchmark

http://dev.mysql.com/doc/refman/5.0/en/numeric-type-overview.html

http://dev.mysql.com/doc/refman/5.0/en/cast-functions.html</pre>
]]></content:encoded>
			<wfw:commentRss>http://www.mysqlfanboy.com/2010/07/does-size-or-type-matter/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Federated Tables</title>
		<link>http://www.mysqlfanboy.com/2010/07/federated-tables/</link>
		<comments>http://www.mysqlfanboy.com/2010/07/federated-tables/#comments</comments>
		<pubDate>Wed, 07 Jul 2010 15:55:47 +0000</pubDate>
		<dc:creator>mark</dc:creator>
				<category><![CDATA[Tips & Tricks]]></category>
		<category><![CDATA[Data]]></category>
		<category><![CDATA[Engine]]></category>
		<category><![CDATA[Federated]]></category>
		<category><![CDATA[MySQL]]></category>
		<category><![CDATA[remote server]]></category>
		<category><![CDATA[Tables]]></category>

		<guid isPermaLink="false">http://www.mysqlfanboy.com/?p=234</guid>
		<description><![CDATA[Your searching for how to create a join across two databases on two different servers and it can&#8217;t be done directly.   select  d1.a, d2.b from db1@server1 join db2@server2 where db1.c = db2.c; does not work. You learn about federated databases.  The federated storage engine allows accesses data in tables of remote databases.  Now how do [...]]]></description>
			<content:encoded><![CDATA[<p>Your searching for how to create a join across two databases on two different servers and it can&#8217;t be done directly.   <em><strong>select  d1.a, d2.b from db1@server1 join db2@server2 where db1.c = db2.c;</strong></em> does not work.</p>
<p>You learn about federated databases.  The federated storage engine allows accesses     data in tables of remote databases.  Now how do you make it work?</p>
<p>1) Check if the federated storage engine is supported.  <strong>Federation is OFF by default!</strong></p>
<pre style="padding-left: 30px;">mysql&gt; show engines;
+------------+---------+----------------------------------------------------------------+
| Engine     | Support | Comment                                                        |
+------------+---------+----------------------------------------------------------------+
| InnoDB     | YES     | Supports transactions, row-level locking, and foreign keys     |
| MyISAM     | DEFAULT | Default engine as of MySQL 3.23 with great performance         |
| BLACKHOLE  | YES     | /dev/null storage engine (anything you write to it disappears) |
| CSV        | YES     | CSV storage engine                                             |
| MEMORY     | YES     | Hash based, stored in memory, useful for temporary tables      |
| <strong>FEDERATED  </strong>| <strong>YES     </strong>| Federated MySQL storage engine                                 |
| ARCHIVE    | YES     | Archive storage engine                                         |
| MRG_MYISAM | YES     | Collection of identical MyISAM tables                          |
+------------+---------+----------------------------------------------------------------+
</pre>
<p>If it is not &#8220;Support&#8221;ed (on) you need to add &#8216;<strong>federated=ON</strong>&#8216; to the <strong>[mysqld]</strong> section of your <strong>/etc/my.cnf</strong> file.  I found this section to be a bit troublesome.  It must be &#8216;=ON&#8217; not &#8216;=YES&#8221; or even &#8216;=on&#8217;.   Most options allow these but the federated options is picky.  I&#8217;m running MySQL Enterprise 5.1.37.sp1.</p>
<p>2) If you don&#8217;t already have the database created, create the database on the storage server.  By &#8216;storage server&#8217; I mean the one where the data will be written to disk.</p>
<p>I like to create a user just for the purpose of connection the federated copy of the database to the true database.  This way, if the password gets changed or the user deleted, the federated system can continue to connect.</p>
<pre style="padding-left: 30px;">
<pre>mysql&gt; CREATE DATABASE xfiles;
mysql&gt; USE xfiles;
mysql&gt; CREATE TABLE cases(
 Name VARCHAR(20),
 case TINYINT(3),
) ENGINE = INNODB;</pre>
</pre>
<p>3) Now you can create the federated version of your data on the remote system.</p>
<pre style="padding-left: 30px;">mysql&gt; CREATE DATABASE xfiles;
mysql&gt; USE xfiles;
mysql&gt; CREATE TABLE cases(
 Name VARCHAR(20),
 case TINYINT(3),
) ENGINE = FEDERATED
CONNECTION = 'mysql://skiner:c0nsper@fbi/xfiles/cases';</pre>
<p>4) Check your work. The table status should show Engine: FEDERATED.</p>
<pre style="padding-left: 30px;">mysql&gt; use xfiles;
mysql&gt; show table status\G</pre>
<p>Now you can add records to the table and the data should show up in select on either server.</p>
<p>Enjoy.</p>
<p><img class="alignnone" title="Mark Grennan" src="http://mark.grennan.com/images/MarkGrennanSigniture.bmp" alt="" width="166" height="69" /></p>
]]></content:encoded>
			<wfw:commentRss>http://www.mysqlfanboy.com/2010/07/federated-tables/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>MyTOP Upated</title>
		<link>http://www.mysqlfanboy.com/2010/06/mytop/</link>
		<comments>http://www.mysqlfanboy.com/2010/06/mytop/#comments</comments>
		<pubDate>Thu, 24 Jun 2010 22:54:20 +0000</pubDate>
		<dc:creator>mark</dc:creator>
				<category><![CDATA[Code]]></category>
		<category><![CDATA[MyTOP]]></category>
		<category><![CDATA[Tips & Tricks]]></category>

		<guid isPermaLink="false">http://www.mysqlfanboy.com/?p=194</guid>
		<description><![CDATA[MyTOP is a console-based (non-gui) tool for monitoring the threads and overall performance of a MySQL. UPDATE &#8211; I just fond Jeremy did update MyTOP in 2009 and released it on GitHub.  He fixed the 64x and 5.x bugs. He also incremented the version number to 1.7.  So, I&#8217;m bumping my number to 1.8. Jeremy [...]]]></description>
			<content:encoded><![CDATA[<p><a href="http://jeremy.zawodny.com/mysql/mytop/">MyTOP</a> is a console-based (non-gui) tool for monitoring the threads and overall performance of a MySQL.</p>
<p><em><strong>UPDATE &#8211; I just fond Jeremy did update MyTOP in 2009 and released it on <a href="http://github.com/jzawodn/mytop">GitHub</a>.  He fixed the 64x and 5.x bugs. He also incremented the version number to 1.7.  So, I&#8217;m bumping my number to 1.8. </strong></em></p>
<p><a href="/wp-content/uploads/2010/06/MyTop-1..jpg"><img class="alignright size-full wp-image-190" title="Small-MyTop-1." src="/wp-content/uploads/2010/06/Small-MyTop-1..jpg" alt="" width="256" height="176" /></a>Jeremy D. Zawodny &lt;<a href="mailto:Jeremy@Zawodny.com">Jeremy@Zawodny.com</a>&gt; wrote the original in 2000 and has continued to update it until 2007. The 1.6 version works on MySQL up to version 4.x.</p>
<p>For weeks now and I&#8217;m been working on bringing it up to date.  When I started using version 1.6 it worked but didn&#8217;t return some data fields.  After fixing these bugs I began to ideas for  improvements.  Here is a quick list of what I have done.</p>
<ul>
<li>Added updates for MySQL 5.x.</li>
<li>Added &#8216;S&#8217; (slow) highlighting.</li>
<li>Added &#8216;C&#8217; to turn on and off Color.</li>
<li>Added &#8216;l&#8217; command to change color for long running queries.</li>
<li>Fixed a few documentation issues.</li>
<li>Added monitoring for  Slave status.</li>
<li>Added color to Queue hit ratio.</li>
<li>Added number of rows sorted per second.</li>
<li>Added a new column to display State of the query. (Sorting, Locked, Updating)</li>
<li>Added &#8216;t&#8217; to filter based on State.</li>
</ul>
<p><a href="/wp-content/uploads/2010/06/MyTop-2.jpg"><img class="alignright size-full wp-image-191" title="Small-MyTop-2" src="/wp-content/uploads/2010/06/Small-MyTop-2.jpg" alt="" width="256" height="176" /></a>How can you use it?  I was having a problem with a production system.  It has locks creating long delays in MyISAM tables. (Not new.) I used MyTOP version 1.7 (my release) with the <strong>&#8216;t&#8217; </strong>command to filter the query state for &#8216;Locked&#8221;. I also use the <strong>&#8216;i&#8221;</strong> command to reverse the sort order and filter for active connections. I found a &#8220;LOCK&#8221; run by root on the local host.  I then use &#8216;f&#8217; to display the full query request.  I copied it and killed it with the &#8216;k&#8217; command. Problem solved. You can&#8217;t do this with MySQL Administrator, MySQL Workbench or Toad for MySQL.</p>
<p>The color display makes monitoring what is happening easy.</p>
<p>I&#8217;m making my updates to MyTOP availible at  <a href="/mytop/">www.MySqlFanBoy.com/mytop</a>.</p>
<p>Enjoy.</p>
<p><img class="alignnone" src="http://mark.grennan.com/images/MarkGrennanSigniture.bmp" alt="" width="166" height="69" /></p>
]]></content:encoded>
			<wfw:commentRss>http://www.mysqlfanboy.com/2010/06/mytop/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>New AutoMySQLBackup Script</title>
		<link>http://www.mysqlfanboy.com/2010/04/new-automysqlbackup-script/</link>
		<comments>http://www.mysqlfanboy.com/2010/04/new-automysqlbackup-script/#comments</comments>
		<pubDate>Mon, 19 Apr 2010 16:31:31 +0000</pubDate>
		<dc:creator>mark</dc:creator>
				<category><![CDATA[Tips & Tricks]]></category>
		<category><![CDATA[Backup]]></category>
		<category><![CDATA[Code]]></category>
		<category><![CDATA[mg_]]></category>
		<category><![CDATA[MySQL]]></category>
		<category><![CDATA[Script]]></category>

		<guid isPermaLink="false">http://www.mysqlfanboy.com/?p=103</guid>
		<description><![CDATA[MySQL Backup Script has been around for a long time.  I have used it on and off for years but now I&#8217;ve needed to make some  improvements.   This script is based on VER. 2.6 &#8211; http://sourceforge.net/projects/automysqlbackup/ Copyright (c) 2002-2003 wipe_out@lycos.co.uk. I have added my own Copyright (c) 2010 mark@grennan.com &#8211; http://www.mysqlfanboy.com/Files/automysqlbackup.sh. But as the code [...]]]></description>
			<content:encoded><![CDATA[<p>MySQL Backup Script has been around for a long time.  I have used it on and off for years but now I&#8217;ve needed to make some  improvements.   This script is based on VER. 2.6 &#8211; <a href="http://sourceforge.net/projects/automysqlbackup/">http://sourceforge.net/projects/automysqlbackup/</a> Copyright (c) 2002-2003 wipe_out@lycos.co.uk.<br />
I have added my own Copyright (c) 2010 mark@grennan.com &#8211; <a href="/Files/automysqlbackup.sh">http://www.mysqlfanboy.com/Files/automysqlbackup.sh</a>. But as the code says:  This program is distributed in the hope that it will be useful,  but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.</p>
<p>My improvements include:</p>
<p># VER 2.6 Beta 5 &#8211; MTG &#8211; (2010-04-18)<br />
#    Added option to archive (rsync) the local backup files to a remote locations<br />
#     using the COPYDIR variable.<br />
#    Added option to copy files into a directory based on the host name using the<br />
#     variable HOSTNAME.  This allows the script to be run from a shared storage directory<br />
#     ( SBM, NFS, NetApp) the data to be kept separate.<br />
#    Added option to additionally backup all database schema only using variable FULLSCHEMA.<br />
#    Added option to backup MySQL configuration file, my.cnf and remove files older then seven<br />
#    days from the BACKUPDIR directory.<br />
#    Added &#8211;master-data=2 and &#8211;single-transaction to include a comment with the master server&#8217;s<br />
#    the binary log coordinates. If used the CHANGE_MASTER_TO line must be uncommitted.</p>
<p>Download it here: <a href="../Files/automysqlbackup.sh">http://www.mysqlfanboy.com/Files/automysqlbackup.sh</a></p>
<p><img title="Mark Grennan" src="http://mark.grennan.com/images/MarkGrennanSigniture.bmp" alt="Mark Grennan" width="166" height="69" /><br />
]]></content:encoded>
			<wfw:commentRss>http://www.mysqlfanboy.com/2010/04/new-automysqlbackup-script/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Developer Tips using MySQL</title>
		<link>http://www.mysqlfanboy.com/2010/04/developer-tips-using-mysql/</link>
		<comments>http://www.mysqlfanboy.com/2010/04/developer-tips-using-mysql/#comments</comments>
		<pubDate>Mon, 19 Apr 2010 16:17:02 +0000</pubDate>
		<dc:creator>mark</dc:creator>
				<category><![CDATA[Tips & Tricks]]></category>
		<category><![CDATA[Code]]></category>
		<category><![CDATA[Developer]]></category>
		<category><![CDATA[Examples]]></category>
		<category><![CDATA[MySQL]]></category>
		<category><![CDATA[Tips]]></category>

		<guid isPermaLink="false">http://www.mysqlfanboy.com/?p=44</guid>
		<description><![CDATA[I get ask, by application developers,  &#8220;how do you optimize MySQL&#8221;.  I do lots of things that don&#8217;t really relate to a developer. I analyze the percent of queries are being pulled from cache for instance.  What a developer can do to optimize the SQL they develop is a different questions.   So here is a [...]]]></description>
			<content:encoded><![CDATA[<p>I get ask, by application developers,  &#8220;how do you optimize MySQL&#8221;.  I do lots of things that don&#8217;t really relate to a developer. I analyze the percent of queries are being pulled from cache for instance.  What a developer can do to optimize the SQL they develop is a different questions.   So here is a quick list of things applications developers should know about MySQL.</p>
<p><a href="http://dev.mysql.com/doc/refman/5.1/en/using-explain.html">Explain</a> will analyze your query.</p>
<p>This example shows the possible indexes (keys) that could be used and the index that was selected.  2,262 rows where selected and then sorted (Using file sorts) and one record was returned (limit 1).</p>
<pre><strong>mysql&gt;</strong> <strong>explain</strong> SELECT 5/9*(temp_F-32) as t, 5/9*(dewpt_F-32) as td, speed_mps as spd, dir
 &gt; where stn='KLDM' and date_time&lt;'2010-02-12 18:15' and date_time&gt;'2010-02-12 17:45'
 &gt; order by ABS( date_time - CAST('2010-02-12 18:00:00' as datetime) ) limit 1;</pre>
<pre>+----+-------------+----------+------+-----------------------+------+---------+-------+------+-----------------------------+
| id | select_type | table    | type | possible_keys         | key  | key_len | ref   | rows | Extra                       |
+----+-------------+----------+------+-----------------------+------+---------+-------+------+-----------------------------+
|  1 | SIMPLE      | metar_nc | ref  | PRIMARY,stn,date_time | stn  | 8       | const | 2262 | Using where; Using filesort |
+----+-------------+----------+------+-----------------------+------+---------+-------+------+-----------------------------+</pre>
<p>Using <a href="http://dev.mysql.com/tech-resources/articles/using-new-query-profiler.html">profiling</a> can give you even more information;  Don’t forget to turn it off with a ‘<strong>set profiling=0</strong>’ when you are done.</p>
<pre><strong>mysql&gt;</strong> set profiling=1;
+--------------------+----------+
| Status             | Duration |
+--------------------+----------+
| starting           | 0.000110 |
| Opening tables     | 0.000014 |
| query end          | 0.000004 |
| freeing items      | 0.000008 |
| logging slow query | 0.000002 |
| cleaning up        | 0.000003 |
+--------------------+----------+
6 rows in set (0.01 sec)
<strong>mysql&gt;</strong> <strong>set profiling=0;</strong></pre>
<p><strong><br />
Indexing Basics</strong></p>
<p>Avoiding disk reads is the name of the game.  Indexes are presorted and small.  Two or three disk reads of an index can point to a large amount of data.</p>
<ul>
<li>MySQL Can only use Prefixes of the index<br />
mysql&gt; SELECT AVG(age) FROM user GROUP BY city;<br />
This is a complex query that needs to scan all the rows. You can make it traverse shorter indexes by adding an index on (city,age)</li>
<li>Index (A,B) can be used for<br />
WHERE A=5 ,  WHERE A=6 AND  B=5 , WHERE A=7 AND B&gt;5<br />
It can&#8217;t be used for  –  WHERE B=6 AND B&lt;2</li>
<li>Only Equality/List allows second key part usage<br />
WHERE A=5 AND B&gt;6 &#8211; will use 2 key parts<br />
IN (1,2) AND B=2 &#8211; will use 2 key parts</li>
<li>A&gt;5 and B=2 will use 1 key part only<br />
The B=2 will be checked while reading row/index only<br />
A=5 ORDER BY B – will use the index<br />
A&gt;5 ORDER BY B – will NOT use the index</li>
<li>For simple cross reference look ups, add the data to the index to skip the data read.<br />
SELECT name FROM login=”Jack123”;</li>
</ul>
<p>If this is a very common part of your code, make the index (login,name).  When the index is read the data is in memory. Don’t add every column you just double the disk space and magnify the disk access.</p>
<p><strong>More Tips</strong></p>
<ol>
<li>Check that all tables have PRIMARY KEYs <span style="text-decoration: line-through;">on columns with high cardinality</span>. Primary keys must be unique.<br />
A column like, `gender` low cardinality (selectivity), an id column (Int &#8211; auto increment) is a good candidate to become a primary key.</li>
<li>All joins (inter, outer, ‘,’) should have indexes.</li>
<li>Fields you often search on (appear frequently in WHERE, ORDER BY or GROUP BY clauses) need indexes.<br />
But don’t add too many: the worst thing you can do is to add an index on every column of a table.</li>
<li>Don&#8217;t use DISTINCT when you have or could use GROUP BY</li>
<li>Open to the server just before you are going to use it.  Unless you are using a persistent connection library, don&#8217;t open a database connections and then run minutes of calculations before making your query.  You may find your connections has been &#8220;gone away&#8221; before you make your query.</li>
<li>When your index many columns, create a hash column. Then your query will look like:<br />
SELECT *<br />
FROM table<br />
WHERE hash_column = MD5( CONCAT(col1, col2) )<br />
AND col1=&#8217;aaa&#8217; AND col2=&#8217;bbb&#8217;;</li>
<li>Use less RAM by declaring columns only as large as they need to be to hold the values stored in them.<br />
Use CHAR type when possible (instead of VARCHAR, BLOB or TEXT) — when values of a column have constant length: MD5-hash (32 symbols) or  ICAO or IATA airport code (4 and 3 symbols). This is also true for indexes.  If only the last 4 symbols are unique index only that part.</li>
<li>Use SQL_NO_CACHE when you are SELECT-ing frequently updated data or large sets of data.  This way you will not kick good data out of the cache.</li>
<li>Avoid wildcards at the start of LIKE queries.  (LIKE ‘%find%’).  Finding ‘1234find’ in 10000 records requires up to 40,000 searches.</li>
<li>Normalizing redundant data is good but don’t split a table because you have too many columns.</li>
<li>Think of storing users sessions data (or any non-critical / high access data) in <a href="http://dev.mysql.com/doc/refman/5.0/en/memory-storage-engine.html">MEMORY</a> table — it’s very fast.</li>
<li>Divide complex queries into several simpler ones — they have more chances to be cached, so will be quicker.</li>
<li>A column must be declared as NOT NULL if it really is. This speeds up table traversing.</li>
<li>If you usually retrieve rows in the same order like expr1, expr2, &#8230;, make <a href="http://dev.mysql.com/doc/refman/5.0/en/alter-table.html">ALTER TABLE &#8230; ORDER BY expr1, expr2</a>, &#8230; to optimize the table.</li>
<li>Don’t use PHP loop to fetch rows from database one by one just because you can — use IN instead, e.g.<br />
SELECT *<br />
FROM `table`<br />
WHERE `id` IN (1,7,13,42);</li>
<li>Reuse your database connections.  Opening a new connection to the database will add one or more seconds to your query.<br />
In PHP use mysql_pconnect() to open a persistent connection with mod_php. Perl provides persistent connections with Apache::DBI with mod_perl. Python does not have persistent connections in mod_python.  But you can maintain them in your application.  (<a href="http://www.modpython.org/FAQ/faqw.py">http://www.modpython.org/FAQ/faqw.py</a>)</li>
<li>When inserting data, insert only those values that differs from the default. This reduces the query parsing time.</li>
<li>Use <a href="http://dev.mysql.com/doc/refman/5.0/en/insert-delayed.html">INSERT DELAYED</a> or <a href="http://dev.mysql.com/doc/refman/5.0/en/insert.html">INSERT LOW_PRIORITY</a> (for MyISAM) to write to your change log table.<br />
Also, if it’s MyISAM, you can add DELAY_KEY_WRITE=1 option — this makes index updates faster because they are not flushed to disk until the table is closed.</li>
<li>For your web application, images and other binary assets should normally be stored as files.</li>
<li> That is, store only a reference to the file rather than the file itself in the database.</li>
</ol>
<p><img title="Mark Grennan" src="http://mark.grennan.com/images/MarkGrennanSigniture.bmp" alt="Mark Grennan" width="166" height="69" /><br />
]]></content:encoded>
			<wfw:commentRss>http://www.mysqlfanboy.com/2010/04/developer-tips-using-mysql/feed/</wfw:commentRss>
		<slash:comments>3</slash:comments>
		</item>
		<item>
		<title>Compiling Drizzle on CentOS 5.4</title>
		<link>http://www.mysqlfanboy.com/2010/04/72/</link>
		<comments>http://www.mysqlfanboy.com/2010/04/72/#comments</comments>
		<pubDate>Tue, 06 Apr 2010 21:36:00 +0000</pubDate>
		<dc:creator>mark</dc:creator>
				<category><![CDATA[Tips & Tricks]]></category>
		<category><![CDATA[Build]]></category>
		<category><![CDATA[CentOS]]></category>
		<category><![CDATA[Compile]]></category>
		<category><![CDATA[Drizzle]]></category>
		<category><![CDATA[RedHat]]></category>

		<guid isPermaLink="false">http://www.mysqlfanboy.com/?p=72</guid>
		<description><![CDATA[I was able to compile Drizzle on CentOS today thanks to Neil Armitage post on his website. Clean install centos 5.4 with Development Tools and Development Libraries yum groupinstall &#8220;Development Tools&#8221; yum groupinstall &#8220;Development Libraries&#8221; Setup the drizzle user account and allow it to sudo /usr/sbin/visudo uncomment %wheel ALL=(ALL) NOPASSWD: ALL useradd drizzle gpasswd -a [...]]]></description>
			<content:encoded><![CDATA[<p>I was able to compile Drizzle on CentOS today thanks to <a href="http://neilarmitage.com/2010/01/drizzle-on-centos-5-4/">Neil Armitage</a> post on his website.</p>
<p>Clean install centos 5.4 with Development Tools and Development  Libraries</p>
<blockquote><p>yum groupinstall &#8220;Development Tools&#8221;<br />
yum groupinstall &#8220;Development Libraries&#8221;</p></blockquote>
<p>Setup the drizzle user account and allow it to sudo</p>
<blockquote><p>/usr/sbin/visudo<br />
uncomment %wheel  ALL=(ALL)       NOPASSWD: ALL<br />
useradd drizzle<br />
gpasswd -a drizzle wheel</p></blockquote>
<p>Install Required Dependencies</p>
<blockquote><p>yum install autoconf autoconf.noarch bison-devel.x86_64 \<br />
bison.x86_64 bzr cpp.x86_64 e2fsprogs-devel.i386 e2fsprogs-devel.x86_64 \<br />
gcc gcc-c++.x86_64 gcc.x86_ glib2-devel glibc-devel.x86_64 glibc.x86_64 \<br />
gperf libevent-devel.x86_64 libstdc++.i386 libtool.x86_64  ncurses-devel.i386 \<br />
ncurses-devel.x86_64 ncurses.x86_64 pcre-devel.x86_64  pcre.i386 pcre.x86_64 \<br />
readline-devel.x86_64 readline.x86_64  zlib-devel.x86_64</p></blockquote>
<p>Install Protobufs</p>
<blockquote><p>wget  http://protobuf.googlecode.com/files/protobuf-2.3.0.tar.gz<br />
tar -xvf protobuf-2.3.0.tar.gz<br />
cd protobuf-2.3.0<br />
./configure<br />
make<br />
make install</p></blockquote>
<p>Install bzr</p>
<blockquote><p>wget  http://launchpad.net/bzr/2.1/2.1.0b4/+download/bzr-2.1.0b4.tar.gz<br />
tar -xvf bzr-2.1.0b4.tar.gz<br />
cd bzr-2.1.0b4<br />
python setup.py install</p></blockquote>
<p>Make the Local bzr Repo</p>
<blockquote><p>su &#8211; drizzle</p></blockquote>
<blockquote><p>mkdir ~/bzrwork<br />
bzr init-repo &#8211;2a  ~/bzrwork<br />
cd ~/bzrwork</p></blockquote>
<p>Build libdrizzle</p>
<blockquote><p>bzr branch lp:libdrizzle<br />
cd libdrizzle<br />
./config/autorun.sh<br />
./configure<br />
make<br />
sudo make install</p></blockquote>
<p>Build Drizzle</p>
<blockquote><p>cd ~/bzrwork<br />
bzr branch lp:drizzle<br />
cd drizzle<br />
./config/autorun.sh<br />
./configure<br />
make<br />
sudo make install</p></blockquote>
<p>Run the tests</p>
<blockquote><p>cd tests<br />
./test-run</p></blockquote>
<p>Configure and Start Drizzle</p>
<blockquote><p>sudo mkdir /usr/local/var<br />
sudo chown drizzle.drizzle /usr/local/var<br />
cd /usr/local<br />
/usr/local/sbin/drizzled &#8211;no-defaults  &#8211;drizzle-protocol-port=9306 \<br />
&#8211;basedir=$PWD &#8211;datadir=$PWD/var  &gt;&gt;  $PWD/var/drizzle.err 2&gt;&amp;1 &amp;</p></blockquote>
<p>Connect to drizzle</p>
<blockquote><p>drizzle</p></blockquote>
<p><img title="Mark Grennan" src="http://mark.grennan.com/images/MarkGrennanSigniture.bmp" alt="Mark Grennan" width="166" height="69" /><br />
]]></content:encoded>
			<wfw:commentRss>http://www.mysqlfanboy.com/2010/04/72/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>The OpenArk Kit</title>
		<link>http://www.mysqlfanboy.com/2010/03/the-openark-kit/</link>
		<comments>http://www.mysqlfanboy.com/2010/03/the-openark-kit/#comments</comments>
		<pubDate>Thu, 25 Mar 2010 20:07:42 +0000</pubDate>
		<dc:creator>mark</dc:creator>
				<category><![CDATA[Tips & Tricks]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[MySQL]]></category>
		<category><![CDATA[Tools]]></category>

		<guid isPermaLink="false">http://www.mysqlfanboy.com/?p=53</guid>
		<description><![CDATA[Shlomi Noach is DBA,  authorized MySQL instructor software developer and Winner of the MySQL Community Member of the Year, 2009. He has published his own collections of MySQL scripts and you might find them useful.  Shlomi calls them the &#8220;openark kit&#8220;. The available tools are: oak-apply-ri: apply referential integrity on two columns with parent-child relationship. [...]]]></description>
			<content:encoded><![CDATA[<p><a href="http://code.openark.org/blog/shlomi-noach">Shlomi Noach</a> is DBA,  authorized MySQL instructor software developer and Winner of the <a href="http://www.mysql.com/why-mysql/awards/community-2009.html">MySQL  Community Member of the Year, 2009</a>.</p>
<p>He has published his own collections of MySQL scripts and you might find them useful.  Shlomi calls them the &#8220;<a title="Permanent Link to openark kit" rel="bookmark" href="http://code.openark.org/forge/openark-kit">openark kit</a>&#8220;.</p>
<p>The available tools are:</p>
<ul>
<li><a title="oak-apply-ri" href="http://code.openark.org/forge/openark-kit/oak-apply-ri">oak-apply-ri</a>:  apply referential integrity on two columns with parent-child   relationship.</li>
<li><a title="oak-block-account" href="http://code.openark.org/forge/openark-kit/oak-block-account">oak-block-account</a>:  block or release MySQL users accounts, disabling them or enabling them  to login.</li>
<li><a title="oak-chunk-update" href="http://code.openark.org/forge/openark-kit/oak-chunk-update">oak-chunk-update</a>:  Perform long, non-blocking UPDATE/DELETE operation in auto managed  small chunks.</li>
<li><a title="oak-kill-slow-queries" href="http://code.openark.org/forge/openark-kit/oak-kill-slow-queries">oak-kill-slow-queries</a>:  terminate long running queries.</li>
<li><a title="oak-modify-charset" href="http://code.openark.org/forge/openark-kit/oak-modify-charset">oak-modify-charset</a>:  change the character set (and collation) of a textual column.</li>
<li><a title="oak-online-alter-table" href="http://code.openark.org/forge/openark-kit/oak-online-alter-table">oak-online-alter-table</a>:  Perform a non-blocking ALTER TABLE operation.</li>
<li><a title="oak-purge-master-logs" href="http://code.openark.org/forge/openark-kit/oak-purge-master-logs">oak-purge-master-logs</a>:  purge master logs, depending on the state of replicating slaves.</li>
<li><a title="oak-security-audit" href="http://code.openark.org/forge/openark-kit/oak-security-audit">oak-security-audit</a>:  audit accounts, passwords, privileges and other security settings.</li>
<li><a title="oak-show-limits" href="http://code.openark.org/forge/openark-kit/oak-show-limits">oak-show-limits</a>:  show AUTO_INCREMENT “free space”.</li>
<li><a title="oak-show-replication-status" href="http://code.openark.org/forge/openark-kit/oak-show-replication-status">oak-show-replication-status</a>:  show how far behind are replicating slaves on a given master.</li>
</ul>
<p>All tools are written in Python, and require Python 2.3 or newer, and  the python-mysqldb driver. Some tools require MySQL 5.0 or higher; see  the docs for each tool.</p>
<p>The openark kit is released under the <a href="http://www.opensource.org/licenses/bsd-license.php">BSD license</a>.</p>
<p>Thanks Shlomi.</p>
<p>Enjoy!</p>
]]></content:encoded>
			<wfw:commentRss>http://www.mysqlfanboy.com/2010/03/the-openark-kit/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Loading Bulk CSV Tables</title>
		<link>http://www.mysqlfanboy.com/2010/03/loading-big-csv-tables/</link>
		<comments>http://www.mysqlfanboy.com/2010/03/loading-big-csv-tables/#comments</comments>
		<pubDate>Tue, 23 Mar 2010 18:47:03 +0000</pubDate>
		<dc:creator>mark</dc:creator>
				<category><![CDATA[Tips & Tricks]]></category>
		<category><![CDATA[Big]]></category>
		<category><![CDATA[Bulk]]></category>
		<category><![CDATA[CSV]]></category>
		<category><![CDATA[Data]]></category>
		<category><![CDATA[Index]]></category>
		<category><![CDATA[Load]]></category>
		<category><![CDATA[Loading]]></category>
		<category><![CDATA[MyISAM]]></category>
		<category><![CDATA[MySQL]]></category>
		<category><![CDATA[Tables]]></category>

		<guid isPermaLink="false">http://www.mysqlfanboy.com/?p=18</guid>
		<description><![CDATA[In my job I use many data tables that are transient.  New weather data is received all the time and old data is purged.  Most of these table are received as CSV files.  The data is then loaded into MySQL tables and indexed to be used with geographic queries. Most of these tables never see [...]]]></description>
			<content:encoded><![CDATA[<p>In my job I use many data tables that are transient.  New weather data is received all the time and old data is purged.  Most of these table are received as <a href="http://en.wikipedia.org/wiki/Comma-separated_values">CSV files</a>.  The data is then loaded into MySQL tables and indexed to be used with geographic queries.</p>
<p>Most of these <a href="http://dev.mysql.com/doc/refman/5.1/en/csv-storage-engine.html">tables</a> never see an insert or update.  It would be nice if  you could build make these CVS tables read only and build byte pointer <a href="http://dev.mysql.com/doc/refman/5.1/en/se-csv-limitations.html">indexes</a> for each row.  (Maybe some day I&#8217;ll code this into MySQL.)</p>
<p>Most people load large data tables at night time with the LOCK &amp; LOAD method.  It goes like LOCK TABLE&#8230;; LOAD DATA INFILE&#8230;; UNLOCK TABLE.  In other words, nobody will read data or generate reports during while this is running.</p>
<p>With the script I developed I have been able to load 33,000,000 records from a CSV file into a MySQL table, with indexes, in 22m 36.282s minutes without creating long LOCK times effecting the users.</p>
<p>Here is what I’m doing.  This is a proof of concept script written in BASH.</p>
<p><span id="more-18"></span></p>
<p>In the ‘test’ database there are two tables.</p>
<p>forecast = MyISAM table with index<br />
NEWforecast = CSV table</p>
<pre>#!/bin/bash
echo "Truncate forecast file"
mysql test -Bse "truncate table forecast;"
count=`mysql test -Bse "select TABLE_ROWS from information_schema.tables where table_name = 'forecast';"`
echo "Count is now $count"

echo "Check Slave is truncated"
count=`mysql -h slave_ip -u dbaops –pP@ssw0rd -Bse "select TABLE_ROWS from information_schema.tables where table_name = 'forecast';"`
echo "Count is now $count"

# The size of the split file determines the time the MyISAM table will be locked.
    echo "splitting NEWforecast.CVS file into 100,000 records"
    split -l 100000 NEWforecast.CSV data_
# Time the for loop
time for x in data_*
do
# copy new data to MySQL CSV file
    echo "cp /home/dbaops/$x /data/mysql/test/NEWforecast.CSV"
    cp /home/dbaops/$x /data/mysql/test/NEWforecast.CSV
# copy same data to the SLAVE server
    scp /home/dbaops/$x 'dbaops:P@ssw0rd@slave_ip:/data/mysql/test/NEWforecast.CSV'
# Flush tables to load new data
    mysql test -Bse "flush tables;"
    mysql -h slave_ip -u dbaops –pP@ssw0rd -Bse "flush tables;"
# Insert from CVS to MyISAM with index – This command get replicated.
    mysql test -Bse "concurrent insert ignore into forecast select * from NEWforecast;"
    count=`mysql test -Bse "select TABLE_ROWS from information_schema.tables where table_name = 'forecast';"`
    echo "Count for this load is $count"
done

rm data_*

sleep 5
count=`mysql -h slave_ip  -u dbaops –pP@ssw0rd test -Bse "select TABLE_ROWS from information_schema.tables where table_name = 'forecast';"`

echo "Count on SLAVE is now $count"
</pre>
<p>I use the CONCURRENT keyword to enable inserts to happen concurrently, and if needed use &#8220;<a href="http://dev.mysql.com/doc/refman/5.1/en/concurrent-inserts.html">SET GLOBAL concurrent_insert=2</a>&#8220;.</p>
<p>Deleting the old records can be a trick too.  In the above example I just empty the table using the &#8216;TRUNCATE TABLE&#8221; command.  Having no data for the application to query may return strange results to the user.</p>
<p>Bulk deletes can also lock the table for a long amount of time.  A stored procedure can be used to loop through the data and remove all record in batches until there are none left.</p>
<pre>DROP PROCEDURE IF EXISTS forecast.delete_incrementally;
CREATE PROCEDURE `forecast.delete_incrementally`()
MODIFIES SQL DATA
BEGIN
  REPEAT
    DELETE FROM test.forecast
    WHERE valid_time_utc &lt;= SUBDATE(NOW(), INTERVAL 1 DAY)
    LIMIT 10000;
  UNTIL ROW_COUNT() = 0 END REPEAT;
END;</pre>
<p>For InnoDB:</p>
<p>It is a big advantage if the data in the CSV files is already ordered by primary key. (Because the InnoDB primary key is a clustered index, so it will organize the table physically to be in primary key order anyway.)</p>
<p>For the bulk insert, you should consider turning off foreign key checking and unique index checking.</p>
<p>UNIQUE_CHECKS=0;<br />
FOREIGN_KEY_CHECKS=0</p>
<p>Using InnoDB plugin, you can speed things up by inserting data into a table without indexes (only define primary key, of course), and then create the indexes separately with alter table. (on an existing table you can also consider dropping existing indexes, the benefit of this would depend case by case).</p>
<p>CSV Files</p>
<p>http://www.shinguz.ch/MySQL/CSV_tables.pdf</p>
<p>http://blogs.sun.com/carriergrademysql/entry/tips_for_bulk_loading</p>
<p>http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/</p>
<p>InnoDB</p>
<p>http://www.innodb.com/doc/innodb_plugin-1.0/innodb-create-index.html</p>
<p>http://www.mysqlperformanceblog.com/2008/04/23/testing-innodb-barracuda-format-with-compression/</p>
]]></content:encoded>
			<wfw:commentRss>http://www.mysqlfanboy.com/2010/03/loading-big-csv-tables/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>
