MySQL Fanboy » Tips http://www.mysqlfanboy.com Almost crazy about Opensource / Free information. Tue, 09 Aug 2011 17:09:08 +0000 en hourly 1 http://wordpress.org/?v=3.2.1 What is this MySQL file used for? http://www.mysqlfanboy.com/2011/02/what-is-this-mysql-file-used-for/ http://www.mysqlfanboy.com/2011/02/what-is-this-mysql-file-used-for/#comments Thu, 17 Feb 2011 00:25:42 +0000 mark http://www.mysqlfanboy.com/?p=516 MySQL keeps many different files, some contain real data, some contain meta data. Witch ones are important? Witch can your throw away?

This is my attempt to create a quick reference of all the files used by MySQL, whats in them, what can you do if they are missing, what can you do with them.

When I was working for Dell doing Linux support my first words to a customer where “DO YOU HAVE COMPLETE AND VERIFIED BACKUP?” Make one now before you think about doing anything I suggest here.

You should always try to manage your data through a MySQL client.  If things have gone very bad this may not be possible. MySQL may not start. If your file system get corrupt you may have missing files. Sometimes people create other files in the MySQL directory (BAD).  This should help you understand what is safe to remove.

Before you try to work with one of these files make sure you have the file permissions set correctly.

This may not be a complete list of files used my MySQL.  It most certainly doesn’t describe everything each table is used for. If you know of ways to replace a missing file or what happens to MySQL when a file is missing that I haven’t described here, please leave me a comment or email me.  I’ll update this document and give your a reference.

my.cnf

This file alters the default configuration settings. MySQL looks in the /etc directory for my.cnf. You should review this file to insure you are looking in the right place for all other MySQL files.  MySQL WILL run without it.  If you have trouble getting MySQL to start, read the error log then try moving are renaming this file.

There are other places MySQL looks for configuration files.

  • /etc/my.cnf
  • /etc/mysql/my.cnf
  • ~/my.cnf
  • <$MYSQLHOME>/my.cnf
  • ~/.my.cnf

MySQL will use only one configuration file if you use DEFAULTS-FILE = /etc/foo.bar.

mysql <directory>

This directory contains all your data.  This may not be its name.  On Linux servers the default location for MySQL files is /var/lib/mysql.  This directory is controlled by the “datadir” variable in your my.cnf file.

Do I need to say, deleting this directory deletes everything?

mysql/mysql<directory>

This directory is MySQL’s database and is inside the directory containing all your data (see above).  All the files in this directory are MyISAM tables for MySQL information like table and column privilages, users, functions and more.  MySQL will not run without this this database.

If you delete this directory, and all the files/tables within, you can recreate a blank set with the MySQL install utility.

mysql_install_db --user=mysql --datadir=/var/lib/mysql

After running this MySQL will rediscover any database that still exist (directory in the MySQL directory).  You will however have to recreate all users, permissions, functions, events, processes, and maybe more.

ibdata1

If your remove this file your InnoDB DATA IS GONE and MySQL will recreate an empty file.

If you are not using the innodb_file_per_table option (default), this file holds almost ALL of your data in InnoDB tables.  This file is all but useless without its corresponding  ‘.frm’ file for each table in the right database directory. If all you have is the .frm files you can recreate the structure of your tables. (See below.)

Idbdata1 can get really big. The default size is 10MB. MySQL will automatically extended it by the default size as needed. If MySQL crashed, some of your InnoDB data may be in your transaction logs (ib_logfile.*).

By design the InnoDB file does not shrink. The safest way to shrink this file is to take a complete backup, stop MySQL, remove the ib* files, start MySQL and restore all your data.  REALLY.

DatabaseName <directory in mysql>

Each MySQL database has a directory named after it.  Each directory holds the meta data for the database. Your MyISAM data is in this directory. InnoDB tables may be here if the innodb_file_per_table variable is used. By default InnoDB tables are in the ibdata1 file (see above).

If you delete the directory your data may be gone. MyISAM data WILL be lost.  InnoDB tables may still be in the ibdata1.  If so, you will need to recreate the meta data files to recover your data.

Creating a directory is almost equivalent to ‘create database’.  If a directory exists MySQL will show you have a database by that name. The create database command may also creates a db.opt file.

<TableName>.frm

This file is key to both InnoDB and MyISAM databases. It is the meta data to the location of your data. It contains the table column definitions.

If you remove this file MySQL will tell you your DATA doesn’t exist.  It does. Your data is still in the ibdata (.ibd) file or the ibdata1 file. You need to recreate the table to recreate this file. If you don’t know the exact structure of this table your out of luck.

Stop MySQL and move the .MYD and .MYI files to another directory.  (You might also make a backup copy.) Start MySQL and recreate this table. Stop MySQL and copy the .MYD and .MYI files back to the database directory and restart MySQL.

<TableName>.MYD

THIS IS YOUR MyISAM DATA. If this is all you have, and you know the data structure of the the table, all is not lost.  (See .frm Above.)  You may also need to recreate the .MYI index file.

<TableName>.MYI

This file contains the indexs for your table.  If it becomes corrupt or is deleted you can recreate it using the ‘REPAIR TABLE table_name USE_FRM;’ command.

/tmp/#<TableName>.[ frm MYD MYI ]

MySQL creates temporary files in the /tmp director (C:\windows\temp) when sorting (ORDER BY or GROUP BY).  It also creates temp files on a replication server when  the LOAD DATA LOCAL INFILE… command is used. Replication will fail if these files are removed.

<TableName>.ibd

THIS IS YOUR InnoDB DATA. If this is all you have, and you know the data structure of the the table, all is not lost.  (See .frm Above.) Unlike MyISAM tables the indexes are contained in this file with your data.

MySQL doesn’t create this file unless you are using the innodb_file_per_table option.

If you have a “Clean” .ibd file you can import it into a database with the command “ALTER TABLE tbl_name IMPORT TABLESPACE;”

<TableName>.CSV

THIS IS YOUR CSV DATA.  This file contains comma separated text data. These file do not have indexes.

<TableName>.CSM

This file contains meta data for CSV and archive tables.  I have not found what is stored here. I do know it tells MySQL if you are logging to the general logs.

ib_logfile*

This file contains your un-committed transactions data. MySQL uses it to recover from a crash.

If you shut down InnoDB cleanly, you can remove them. MySQL will recreate them.

If you change the size of innodb_log_file_size, you will need to recreate these files by stopping MySQL cleanly and deleting them.

mysql-bin.*

This “Bin Log” files contain any change made to any database. Each transaction is assigned a MASTER_LOG_POS(ion).  These files are not created by default. They are used for replication and point-in-time recovery.

You can stop the server and remove these files IF you remove the mysql-bin.index file as well.  MySQL creates a new bin log file each time it starts or the logs are flushed.  Deleting these files will the server is running will break replication.

mysql-bin.index

This file is used by MySQL to keep a Bin Log list.  It is a simple text file like;

./mysql-bin.000001

./mysql-bin.000002

./mysql-bin.000003

If you remove this file, MySQL will recreate it with only the newest bin log name. If you need to remove old bin logs use the command “purge binary logs [to mysql-bin.######] [before “yyyy-mm-dd”]”.

You can control the number of bin logs using the expire_logs_days variable.

mysqld.log

This is MySQL’s primary administration log. MySQL reports starts and stops as well as some warning and errors in this file.  If MySQL crashes, mysqld_safe will restart it. This log will report this.

You can delete this file if needed.

slow.log

The slow query log consists of all SQL statements that took more than long_query_time seconds to execute and (as of MySQL 5.1.21) required at least min_examined_row_limit rows to be examined.

You can delete this file if needed.

db.opt

Database characteristics, like the CHARACTER SET clause are stored in the db.opt file. You may have strange query results if this file is missing. MySQL will use it’s default. You can recreate this file my altering the table with the correct settings.

mysql.pid

The PID file hold the process ID number for the running server. MySQL creates this file and scripts that start and stop MySQL use it to control MySQL.

MySQL will remove this file when is stops. You should not delete it if MySQL is running. If mysql is NOT running and the file exists MySQL may have crashed and you should delete this file.

master.info
relay-log.info

These are found only on the slave servers. Slaves read the commands from those files in order to replicate what the master has done. Simularly to the bin log, it will create files with numbered extensions as needed.

Master.info is a simple text file. This is a store of the master command is used by mysql when it restarts to make the slave connection.

15
mysql-bin.006004
1006945586
192.168.1.2
repl_user
P@sSw0Rd
3306
60
0

Removing these files will breat replication and / or through your slave out of sync.

*relay-bin.index

The “*” is either the hostname or the same as the .PID file.  This file is the index for the relay log used in replication.  Relay bin files are found only on slave servers.

<TableName>.TRG

Contains the list of triggers belonging to a given table

<TrigerName>.TRN

Contains some metadata on the trigger

<TableName>.ARZ

It’s the file containing the data for an ARCHIVE table.

<TableName>.par

Contains the definition of a partition

tablename#P#partitionname.{MYD,ARZ,ibd}
contains the data related to a given partition

References:

FULL DISK
http://dev.mysql.com/doc/refman/5.1/en/full-disk.html

MySQL Database Backup .MYI and .MYD
http://www.aeonity.com/frost/mysql-database-backup-myi-myd

Recovering from Crashes
http://dev.mysql.com/tech-resources/articles/recovering-from-crashes.html

Using Per-Table Tablespaces

http://dev.mysql.com/doc/refman/5.0/en/innodb-multiple-tablespaces.html

Troubleshooting InnoDB Data Dictionary Operations
http://dev.mysql.com/doc/refman/5.1/en/innodb-troubleshooting-datadict.html

Giuseppe Maxia – who writes a great blog about mysql at http://datacharmer.blogspot.com/ contributed several files I missed.

]]> http://www.mysqlfanboy.com/2011/02/what-is-this-mysql-file-used-for/feed/ 5 It’s about Time. http://www.mysqlfanboy.com/2011/01/its_about_time/ http://www.mysqlfanboy.com/2011/01/its_about_time/#comments Thu, 27 Jan 2011 21:08:27 +0000 mark http://www.mysqlfanboy.com/?p=301 WHAT TIME IS IT?

This post started with a simple question: “Does the function NOW() get executed every time a row is examined?” According to the manual,  “Functions that return the current date or time each are evaluated only once per query …. multiple references to a function such as NOW() … produce the same result. …. (However,) as of MySQL 5.0.12, SYSDATE() returns the time (the row is) executes. “

  • CURDATE() returns the current date.
  • CURTIME() returns the current time.
  • UTC_DATE() returns the current UTC date.
  • UTC_TIME() returns the current UTC time.
  • NOW() return the current date and time.
  • UTC_TIMESTAMP() returns the current UTC date and time.
  • SYSDATE() returns the date and time at which the function executes.
  • LOCALTIME(), LOCALTIME(), LOCALTIMESTAMP(), LOCALTIMESTAMP() are synonyms for NOW().

All these functions return some value of a date / time. For example, WEEK() returns the week number for date.

Here is an example of a common query.

SELECT  stn, temp, lat, lon WHERE valid_time > NOW() – INTERVAL 1 HOUR;

This is the right way to make this query because NOW() is only executed once.  If SYSDATE() was used, each row selected would call SYSDATE().

So, if NOW() is only executed once and SYSDATE() is executed every time, we should see this difference if we BENCHMARK these functions.

mysql> SELECT BENCHMARK( 10000000, NOW() );
+------------------------------+
| BENCHMARK( 10000000, NOW() ) |
+------------------------------+
|                            0 |
+------------------------------+
1 row in set (0.07 sec)

+----------------------------------+
| BENCHMARK( 10000000, SYSDATE() ) |
+----------------------------------+
|                                0 |
+----------------------------------+
1 row in set (5.49 sec)

And it seems to be true. But what about real SQL statements. Will this hold up? Here is a simple table.

CREATE TABLE `TableName1` (
 `id` int(11) NOT NULL AUTO_INCREMENT,
 `datetime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
 PRIMARY KEY (`id`)
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1

I used this SQL statement to put records in the table. It adds a new records with a new date/time with each query. Like:

INSERT INTO `stage`.`TableName1`(`id`,`dtime`) VALUES ( NULL,NOW())

What if we then query the table every second. Will NOW() be processed as a part of the results?  I used the SLEEP() function to delay each record.

SELECT id, dtime, (SELECT NOW()) FROM TableName1 WHERE NOT SLEEP(1);

This query took ten seconds to run for a table with ten rows and returns these results.

id	dtime	                (SELECT NOW())
 1	2010-08-06 15:55:57	2010-08-06 16:05:06
 2	2010-08-06 15:56:12	2010-08-06 16:05:06
 3	2010-08-06 15:56:16	2010-08-06 16:05:06
 4	2010-08-06 15:56:20	2010-08-06 16:05:06
 5	2010-08-06 15:56:30	2010-08-06 16:05:06
 6	2010-08-06 16:48:23	2010-08-06 16:05:06
 7	2010-08-06 16:48:26	2010-08-06 16:05:06
 8	2010-08-06 16:48:27	2010-08-06 16:05:06
 9	2010-08-06 16:48:28	2010-08-06 16:05:06
 10	2010-08-06 16:48:29	2010-08-06 16:05:06

As the documentation states, the NOW() is NOT processes every time.

What if  NOW() is a part of a JOIN?

SELECT t.id, t.dtime, d.n  FROM TableName1 AS t, (SELECT 1, NOW() AS n) AS d  WHERE NOT SLEEP(1);

Again, this SELECT takes ten seconds to run for a table with ten rows.

id	dtime                 	n
 1	2010-08-06 15:55:57	2010-08-06 17:09:00
 2	2010-08-06 15:56:12	2010-08-06 17:09:00
 3	2010-08-06 15:56:16	2010-08-06 17:09:00
 4	2010-08-06 15:56:20	2010-08-06 17:09:00
 5	2010-08-06 15:56:30	2010-08-06 17:09:00
 6	2010-08-06 16:48:23	2010-08-06 17:09:00
 7	2010-08-06 16:48:26	2010-08-06 17:09:00
 8	2010-08-06 16:48:27	2010-08-06 17:09:00
 9	2010-08-06 16:48:28	2010-08-06 17:09:00
 10	2010-08-06 16:48:29	2010-08-06 17:09:00

And again the NOW() function is only executed once.

The WHERE clause is harder to test. I inserted a row in the table only seconds apart. I then quickly ran the SELECT again to see if the NOW() matched the times in the table.

SELECT t.id, t.dtime FROM TableName1 AS t WHERE NOT SLEEP(1) AND dtime = (SELECT NOW() - INTERVAL 15 SECOND)  ;

Here is the data table.

id	dtime
 1	2010-08-06 17:35:00
 2	2010-08-06 17:35:00
 3	2010-08-06 17:35:01
 4	2010-08-06 17:35:03
 5	2010-08-06 17:35:04
 6	2010-08-06 17:35:04
 7	2010-08-06 17:35:04
 8	2010-08-06 17:35:05
 9	2010-08-06 17:35:05
 10	2010-08-06 17:35:05
 11	2010-08-06 17:35:06
 12	2010-08-06 17:35:06
 13	2010-08-06 17:35:06
 14	2010-08-06 17:35:07
 15	2010-08-06 17:35:07
 16	2010-08-06 17:35:07
 17	2010-08-06 17:35:08
 18	2010-08-06 17:35:08
 19	2010-08-06 17:35:08
 20	2010-08-06 17:35:09
 21	2010-08-06 17:35:09
 22	2010-08-06 17:35:09
 23	2010-08-06 17:35:09
 24	2010-08-06 17:35:10
 25	2010-08-06 17:35:10

The select statement above only took three seconds to run and returned three rows all the same.

 id	dtime
 11	2010-08-06 17:35:06
 12	2010-08-06 17:35:06
 13	2010-08-06 17:35:06

MySQL is evaluating dtime = (SELECT NOW() – INTERVAL 15 SECOND) before the SLEEP(1).  MySQL only ran the SLEEP(1) for each record that matched.  I reversed the two parts of the AND and  the order of the didn’t matter.

Next I tried these functions with SYSDATE().  I found it is executed every time it is evaluated.  This SELECT took 17 seconds and return a different time for each.

SELECT id, dtime, (SELECT SYSDATE()) FROM TableName1 WHERE NOT SLEEP(1);
id    dtime                 (SELECT SYSDATE())
1    2010-08-06 18:28:42    2010-08-06 19:00:57
2    2010-08-06 18:28:42    2010-08-06 19:00:58
3    2010-08-06 18:28:42    2010-08-06 19:00:59
4    2010-08-06 18:28:43    2010-08-06 19:01:00
5    2010-08-06 18:28:43    2010-08-06 19:01:01
6    2010-08-06 18:28:43    2010-08-06 19:01:02
7    2010-08-06 18:28:43    2010-08-06 19:01:03
8    2010-08-06 18:28:44    2010-08-06 19:01:04
9    2010-08-06 18:28:44    2010-08-06 19:01:05
10    2010-08-06 18:28:44    2010-08-06 19:01:06
11    2010-08-06 18:28:44    2010-08-06 19:01:07
12    2010-08-06 18:28:45    2010-08-06 19:01:08
13    2010-08-06 18:28:45    2010-08-06 19:01:09
14    2010-08-06 18:28:45    2010-08-06 19:01:10
15    2010-08-06 18:28:45    2010-08-06 19:01:11
16    2010-08-06 18:28:46    2010-08-06 19:01:12
17    2010-08-06 18:28:46    2010-08-06 19:01:13

The SET TIMESTAMP statement affects the value returned by NOW() but not by SYSDATE(). This means that timestamp settings in the binary log have no effect on invocations of SYSDATE().

CACHING

MySQL doesn’t cache nondeterministic functions like NOW() and SYSDATE().  If your application processes the same data base queries within the same second or if you need the two queries to return the same results, you will need to turn NOW() into a string.

SELECT stn, temp, lat, lon WHERE valid_time > ‘2010-05-06 23:20:00’ – INTERVAL 1 HOUR;

Even better, do the math in your program.  Here are some benchmarks show this outcome.

SELECT stn, temp, lat, lon WHERE valid_time > ‘2010-05-06 22:20:00’;

+------------------------------------------------------------------------+
| BENCHMARK( 10000000, NOW() -  INTERVAL 1 HOUR = '2011-01-27 19:00:00') |
+------------------------------------------------------------------------+
|                                                                      0 |
+------------------------------------------------------------------------+
1 row in set (0.36 sec)

mysql> SELECT BENCHMARK( 10000000, SYSDATE() -  INTERVAL 1 HOUR = '2011-01-27 19:00:00');
+----------------------------------------------------------------------------+
| BENCHMARK( 10000000, SYSDATE() -  INTERVAL 1 HOUR = '2011-01-27 19:00:00') |
+----------------------------------------------------------------------------+
|                                                                          0 |
+----------------------------------------------------------------------------+
1 row in set (2.42 sec)

mysql> SELECT BENCHMARK( 10000000, '2010-05-06 23:20:00' -  INTERVAL 1 HOUR = '2011-01-27 19:00:00');
+----------------------------------------------------------------------------------------+
| BENCHMARK( 10000000, '2010-05-06 23:20:00' -  INTERVAL 1 HOUR = '2011-01-27 19:00:00') |
+----------------------------------------------------------------------------------------+
|                                                                                      0 |
+----------------------------------------------------------------------------------------+
1 row in set (3.98 sec)

mysql> SELECT BENCHMARK( 10000000, '2010-05-06 23:20:00' = '2011-01-27 19:00:00');
+---------------------------------------------------------------------+
| BENCHMARK( 10000000, '2010-05-06 23:20:00' = '2011-01-27 19:00:00') |
+---------------------------------------------------------------------+
|                                                                   0 |
+---------------------------------------------------------------------+
1 row in set (0.22 sec)

REFERENCE:

http://dev.mysql.com/doc/refman/5.0/en/functions.html

http://dev.mysql.com/doc/refman/5.0/en/date-and-time-functions.html

]]> http://www.mysqlfanboy.com/2011/01/its_about_time/feed/ 6 MySQL GIS – Part 1 http://www.mysqlfanboy.com/2010/08/mysql-gis-part-1/ http://www.mysqlfanboy.com/2010/08/mysql-gis-part-1/#comments Mon, 23 Aug 2010 21:49:55 +0000 mark http://www.mysqlfanboy.com/?p=314 In my business (weather) we use lots map based (Geo) information.  Almost every table has latitude and longitude. Working with this kind of data can be exciting and frustrating.  This should give you a quick start into GIS with MySQL.

“A geographic information system (GIS), or geographical information system, is any system that captures, stores, analyzes, manages, and presents data that are linked to location. In the simplest terms, GIS is the merging of cartography, statistical analysis, and database technology. GIS systems are used in cartography, remote sensing, land surveying, utility management, natural resource management, photogrammetry, geography, urban planning, emergency management, navigation, and localized search engines.” – Wikipedia

GIS / Mapping Systems work with both text data and graphical data.  Applications and utilities often blur the lines between the two types and make understanding difficult.  Map servers blend raster images, with point or polygon data, and bitmap images to make complete images to display in the user’s client application.  For this post I will concentrate on the text type “data”.  The type we can index in a MySQL database.

THE SEARCH

After months of reading, [1]  I’m writing this post to describes what I have learned about how to get started using GEO coding data as quickly as possible.  I found very little piratical information on GIS and MySQL.  The MySQL manual covers the functions but doesn’t supply much practical information on GEO.  Anders Karlsson wrote a nice and short story about GIS that give me a good start.

The best information has be written by Florin Duroiu in his post titled “Political boundaries overlay in Google maps”. A good part of my post is based on his work.

STEP BY STEP

Below are the detailed needed to to produce a MySQL database with the Points of Interest (POI).  This is based on CentOS 5.5 with MySQL 5.1.

yum install gdal
mkdir geo
mkdir data
mkdir data/Oklahoma
cd geo/data/Oklahoma
wget http://downloads.cloudmade.com/north_america/united_states/oklahoma/oklahoma.shapefiles.zip
unzip oklahoma.shapefiles.zip
mysql -e 'create database geo'
ogr2ogr -f "MySQL" MySQL:"geo,user=root,host=localhost,password=" -nln oklahoma_poi -lco engine=MYISAM oklahoma_poi.shp
mysql geo -e 'desc oklahoma_poi'

+----------+--------------+------+-----+---------+----------------+
| Field    | Type         | Null | Key | Default | Extra          |
+----------+--------------+------+-----+---------+----------------+
| OGR_FID  | int(11)      | NO   | PRI | NULL    | auto_increment |
| SHAPE    | geometry     | NO   | MUL | NULL    |                |
| category | varchar(30)  | YES  |     | NULL    |                |
| name     | varchar(113) | YES  |     | NULL    |                |
+----------+--------------+------+-----+---------+----------------+

mysql geo -e "select name, category, Y(SHAPE) as lat, X(SHAPE) as lng  from oklahoma_poi where NAME like 'School:Putnam%'"

+-------------------------------------+--------------------------------+------------+-------------+
| name                                | category                       | lat        | lng         |
+-------------------------------------+--------------------------------+------------+-------------+
| School:Putnam City West High School | Government and Public Services |  35.492557 | -97.6605975 |
| School:Putnam City North School     | Government and Public Services | 35.5892209 | -97.6372648 |
| School:Putnam City School           | Government and Public Services | 35.5122794 | -97.6142079 |
| School:Putnam High School           | Government and Public Services | 35.5214459 | -97.6086523 |
| School:Putnam Heights Academy       | Government and Public Services | 35.5081143 | -97.5397619 |
+-------------------------------------+--------------------------------+------------+-------------+

In a coming set of post I’ll go over:

  • The “Data” types your will find and how to convert between them.
  • What data is available and where can you find it?
  • More examples on what you can do with GIS data.
  • Viewing our GIS data.
  • How to collect your own GIS data.
  • Good and bad examples of searching GIS data.
  • Optimizing MySQL GIS.  Is it really worth using?

[1] Books: GIS for Dummies – Author: Michael N. DeMers – John Wiley & Sons (2009) – ISBN: 0470236825
Open Source GIS: A GRASS GIS Approach. Third Edition.-  Author: Markus Neteler and Helena Mitasova – ISBN: 978-0-38735767-6
Web Mapping Illustrated: Using Open Source GIS Toolkits – Author: Tyler Mitchell – ISBN: 9780596008659



]]> http://www.mysqlfanboy.com/2010/08/mysql-gis-part-1/feed/ 2 Does Size or Type Matter? http://www.mysqlfanboy.com/2010/07/does-size-or-type-matter/ http://www.mysqlfanboy.com/2010/07/does-size-or-type-matter/#comments Tue, 27 Jul 2010 18:08:16 +0000 mark http://www.mysqlfanboy.com/?p=238 MySQL seems to be happy to convert types for you. Developers are rushed to complete their project and if the function works they just move on. But what is the costs of mixing your types? Does it matter if your are running across a million rows or more? Lets find out.

Here is what the programmers see.

mysql> select 1+1;
+-----+
| 1+1 |
+-----+
|   2 |
+-----+
1 row in set (0.00 sec)

mysql> select "1"+"1";
+---------+
| "1"+"1" |
+---------+
|       2 |
+---------+
1 row in set (0.00 sec)

Benchmark

What if we do a thousand simple loops?  How long does the looping itself take?

The BENCHMARK() function executes the expression expr repeatedly count times. It may be used to time how quickly MySQL processes the expression. The result value is always 0.

mysql> select benchmark(1000000000, 1);
+--------------------------+
| benchmark(1000000000, 1) |
+--------------------------+
|                        0 |
+--------------------------+
1 row in set (5.42 sec)

mysql> select benchmark(1000000000, "1" );
+-----------------------------+
| benchmark(1000000000, "1" ) |
+-----------------------------+
|                           0 |
+-----------------------------+
1 row in set (5.40 sec)

So maybe type doesn’t matter? About five seconds just to loop but the type didn’t change it.   What if we add 1+”1″?

mysql> select benchmark(1000000000, 1+1);
+----------------------------+
| benchmark(1000000000, 1+1) |
+----------------------------+
|                          0 |
+----------------------------+
1 row in set (12.65 sec)

mysql> select benchmark(1000000000, 1+"1");
+------------------------------+
| benchmark(1000000000, 1+"1") |
+------------------------------+
|                            0 |
+------------------------------+
1 row in set (35.58 sec)

mysql> select benchmark(1000000000, "1"+"1");
+--------------------------------+
| benchmark(1000000000, "1"+"1") |
+--------------------------------+
|                              0 |
+--------------------------------+
1 row in set (51.59 sec)


It looks like type does matter.  But does it always matter?


mysql> select benchmark(1000000000, sum(1+1));
+---------------------------------+
| benchmark(1000000000, sum(1+1)) |
+---------------------------------+
|                               0 |
+---------------------------------+
1 row in set (9.69 sec)

mysql> select benchmark(1000000000, sum("1"+"1"));
+-------------------------------------+
| benchmark(1000000000, sum("1"+"1")) |
+-------------------------------------+
|                                   0 |
+-------------------------------------+
1 row in set (9.94 sec)

mysql> select benchmark(1000000000, sum("1.23456789"+"1.23456789"));
+-------------------------------------------------------+
| benchmark(1000000000, sum("1.23456789"+"1.23456789")) |
+-------------------------------------------------------+
|                                                     0 |
+-------------------------------------------------------+
1 row in set (10.32 sec)


So, not all functions are the same.  But it looks like size might matter!


mysql> select benchmark(1000000000, 1.1+1.1);
+--------------------------------+
| benchmark(1000000000, 1.1+1.1) |
+--------------------------------+
|                              0 |
+--------------------------------+
1 row in set (34.90 sec)

mysql> select benchmark(1000000000, "1.1"+"1.1");
+------------------------------------+
| benchmark(1000000000, "1.1"+"1.1") |
+------------------------------------+
|                                  0 |
+------------------------------------+
1 row in set (1 min 15.32 sec)

mysql> select  benchmark(1000000000, "1.123456789"+"1.123456789");
+----------------------------------------------------+
| benchmark(1000000000, "1.123456789"+"1.123456789") |
+----------------------------------------------------+
|                                                  0 |
+----------------------------------------------------+
1 row in set (1 min 53.32 sec)


Sorry.  Looks like size does matter.

This doesn't seem logical.


mysql> select benchmark(1000000000, 1=1);
+----------------------------+
| benchmark(1000000000, 1=1) |
+----------------------------+
|                          0 |
+----------------------------+
1 row in set (12.75 sec)

mysql> select benchmark(1000000000, 1="1");
+------------------------------+
| benchmark(1000000000, 1="1") |
+------------------------------+
|                            0 |
+------------------------------+
1 row in set (40.78 sec)
mysql> select benchmark(1000000000, 1=true);
+-------------------------------+
| benchmark(1000000000, 1=true) |
+-------------------------------+
|                             0 |
+-------------------------------+
1 row in set (12.73 sec)

mysql> select benchmark(1000000000, 1="true");
+---------------------------------+
| benchmark(1000000000, 1="true") |
+---------------------------------+
|                               0 |
+---------------------------------+
1 row in set, 65535 warnings (3 min 5.72 sec)
mysql> select benchmark(1000000000, "true"="true");
+--------------------------------------+
| benchmark(1000000000, "true"="true") |
+--------------------------------------+
|                                    0 |
+--------------------------------------+
1 row in set (57.25 sec)


Maybe we should CAST our work?


mysql> select benchmark(1000000000, cast("1" as unsigned));
+----------------------------------------------+
| benchmark(1000000000, cast("1" as unsigned)) |
+----------------------------------------------+
|                                            0 |
+----------------------------------------------+
1 row in set (32.27 sec)

mysql> select benchmark(1000000000, cast("1" as unsigned) + cast("1" as unsigned));
+----------------------------------------------------------------------+
| benchmark(1000000000, cast("1" as unsigned) + cast("1" as unsigned)) |
+----------------------------------------------------------------------+
|                                                                    0 |
+----------------------------------------------------------------------+
1 row in set (1 min 7.24 sec)

Maybe not!
Conclusion:  Be careful with your data types.  If you are taking user input, do the type conversion ONCE in your program.  Don’t let MySQL do the type conversions for you.
query = “SELECT * FROM table where $INPUT = 1″;   could be doing your wrong.

References:

http://dev.mysql.com/doc/refman/5.0/en/information-functions.html#function_benchmark

http://dev.mysql.com/doc/refman/5.0/en/numeric-type-overview.html

http://dev.mysql.com/doc/refman/5.0/en/cast-functions.html

]]> http://www.mysqlfanboy.com/2010/07/does-size-or-type-matter/feed/ 0 MySQL Network Connections http://www.mysqlfanboy.com/2010/05/mysql-network-connections/ http://www.mysqlfanboy.com/2010/05/mysql-network-connections/#comments Fri, 07 May 2010 20:59:39 +0000 mark http://www.mysqlfanboy.com/?p=139 If your MySQL server has hundreds of clients (applications) and tens of thousands of queries per second,  MySQL default network settings may NOT be for you.  Network performance is not often a significant factor in the performance of MySQL.  That said, there are things to consider.

If you are building new applications make these changes now.  Developer expectations are hard to change.  My example below will break your application if  developers open a database connections and then spend ten minutes playing with their play doe before making a query.  If you have working applications make changes slowly.

Middleware provides database connections pooling. This allows database connections to be used by multiple applications. Any program that runs as a deamon or service can use database pooling using libdbi. Applications written in PHP or CGI code don’t pool DB connections. They open and closes the connection each time a user hit a page.

The Apache module mod_dbd can manages database connections. On non-threaded platforms, it provides a persistent connection. On threaded platform, it provides an altogether more scalable and efficient connection pool, as described in this article at ApacheTutor. Note that mod_dbd supersedes the modules presented in that article.

PHP provides mysql_pconnect. It first tries to find a (persistent) link that’s already open with the same host, username and password (NOT PORT). If one is found, an identifier for it will be returned instead of opening a new connection.  Second, the connection to the SQL server will not be closed when the execution of the script ends. Instead, the link will remain open for future use (mysql_close() will not close links established by mysql_pconnect()).

Back to MySQL.

Max Connections has a default of 100 (<= 5.1.14) to 151 (>= 5.1.15).  If you exceed this number you will receive a “Too many connections error”.  This is often the first setting people find they have to changed as their usage grows.

Max User Connections controls the maximum number of simultaneous connections allowed per MySQL user account. By default this is set to unlimited (zero).  The default is fine. Your system will not run out of connections or memory if you have set Max Connections correctly.  Except a bad applications can “steel” all the connections from all the others.  Hopefully you have assigned a different user for each application.  Set this to the maximum number of connections any given user (application) will make.

Wait timeout is the number of seconds the server waits for activity on a non-interactive connection before closing it. The default is eight hours. (28800 seconds)  This is a crazy amount of time.  How long would you stay on the phone if no one was speaking?  Fifteen seconds is a good place to start. If it gives you problems increase it by fifteen until your good. Applications that open a connections and then process their data for minutes before making a request will be effected by a short time setting.

Interactive timeout is the number of seconds the server waits for activity on an interactive connection before closing it. This is only used by applications using the  mysql_real_connect() function of the MySQL API. I set this to 600. My idea is to give more sophisticated users more grace.  (Think connection pools have to reconnect.)

Connect timeout is the number of seconds that the MySQL server waits for a connect packet before responding with Bad handshake. The default is ten seconds.  Five would be better.

Max connections errors is a strange setting. IO stops after this number of bad connections is made from a host. You can unblock blocked hosts with the FLUSH HOSTS statement.  The default for this is ten (10). On a “good” network this shouldn’t have to be changed.  Because “the dabase should never stop” most DBAs make this a very large number. If you have a over loaded network you should put your database on a network of its own.  Or, you can try increasing this value.

max_allowed_packet controls the size of your BLOBs.  If you are storing pictures in your database, you may need to set this.  The default is one megabyte (1M). The protocol limit for max_allowed_packet is 1GB.  Set it to the biggest BLOB you need to store. The value should be a multiple of 1024 bytes.

Here are my my.cnf settings for a server with over 200 PHP and Perl applications and peeking at 4,000 connections per second.

# Network management
port                    = 3306
max_connections         = 5000  # number of simultaneous client connections allowed
max_user_connections    = 200   # number of connections a user can make, 0 = Unlimited
wait_timeout            = 15    # seconds to waits for activity on non interactive connection
interactive_timeout     = 600   # seconds to waits for activity on interactive connection
connect_timeout         = 5     # on connect seconds to waits for a connect packet
max_connect_errors      = 1000000 # IO stops after this number of bad connections one good connect reset
max_allowed_packet      = 10M   # How big is your BLOB?

Mark Grennan

]]> http://www.mysqlfanboy.com/2010/05/mysql-network-connections/feed/ 1 MySQL master/slave support merged into Linux-HA http://www.mysqlfanboy.com/2010/04/mysql-masterslave-support-merged-into-linux-ha/ http://www.mysqlfanboy.com/2010/04/mysql-masterslave-support-merged-into-linux-ha/#comments Wed, 21 Apr 2010 21:07:12 +0000 mark http://www.mysqlfanboy.com/?p=105 (Re-posted from Florian’s blog.)

MySQL replication support for the Pacemaker cluster manager (the stuff that we explained in this webinar) has made it into the Linux-HA resource agents default branch. If you are interested in testing — and you should! — please read the extended announcement. Feedback is extremely welcome on the linux-ha-dev mailing list.

We are expecting to release this as part of resource-agents 1.0.4, in late May/early June.

]]> http://www.mysqlfanboy.com/2010/04/mysql-masterslave-support-merged-into-linux-ha/feed/ 0 Developer Tips using MySQL http://www.mysqlfanboy.com/2010/04/developer-tips-using-mysql/ http://www.mysqlfanboy.com/2010/04/developer-tips-using-mysql/#comments Mon, 19 Apr 2010 16:17:02 +0000 mark http://www.mysqlfanboy.com/?p=44 I get ask, by application developers,  “how do you optimize MySQL”.  I do lots of things that don’t really relate to a developer. I analyze the percent of queries are being pulled from cache for instance.  What a developer can do to optimize the SQL they develop is a different questions.   So here is a quick list of things applications developers should know about MySQL.

Explain will analyze your query.

This example shows the possible indexes (keys) that could be used and the index that was selected.  2,262 rows where selected and then sorted (Using file sorts) and one record was returned (limit 1).

mysql> explain SELECT 5/9*(temp_F-32) as t, 5/9*(dewpt_F-32) as td, speed_mps as spd, dir
 > where stn='KLDM' and date_time<'2010-02-12 18:15' and date_time>'2010-02-12 17:45'
 > order by ABS( date_time - CAST('2010-02-12 18:00:00' as datetime) ) limit 1;
+----+-------------+----------+------+-----------------------+------+---------+-------+------+-----------------------------+
| id | select_type | table    | type | possible_keys         | key  | key_len | ref   | rows | Extra                       |
+----+-------------+----------+------+-----------------------+------+---------+-------+------+-----------------------------+
|  1 | SIMPLE      | metar_nc | ref  | PRIMARY,stn,date_time | stn  | 8       | const | 2262 | Using where; Using filesort |
+----+-------------+----------+------+-----------------------+------+---------+-------+------+-----------------------------+

Using profiling can give you even more information;  Don’t forget to turn it off with a ‘set profiling=0’ when you are done.

mysql> set profiling=1;
+--------------------+----------+
| Status             | Duration |
+--------------------+----------+
| starting           | 0.000110 |
| Opening tables     | 0.000014 |
| query end          | 0.000004 |
| freeing items      | 0.000008 |
| logging slow query | 0.000002 |
| cleaning up        | 0.000003 |
+--------------------+----------+
6 rows in set (0.01 sec)
mysql> set profiling=0;


Indexing Basics

Avoiding disk reads is the name of the game.  Indexes are presorted and small.  Two or three disk reads of an index can point to a large amount of data.

  • MySQL Can only use Prefixes of the index
    mysql> SELECT AVG(age) FROM user GROUP BY city;
    This is a complex query that needs to scan all the rows. You can make it traverse shorter indexes by adding an index on (city,age)
  • Index (A,B) can be used for
    WHERE A=5 ,  WHERE A=6 AND  B=5 , WHERE A=7 AND B>5
    It can’t be used for  –  WHERE B=6 AND B<2
  • Only Equality/List allows second key part usage
    WHERE A=5 AND B>6 – will use 2 key parts
    IN (1,2) AND B=2 – will use 2 key parts
  • A>5 and B=2 will use 1 key part only
    The B=2 will be checked while reading row/index only
    A=5 ORDER BY B – will use the index
    A>5 ORDER BY B – will NOT use the index
  • For simple cross reference look ups, add the data to the index to skip the data read.
    SELECT name FROM login=”Jack123”;

If this is a very common part of your code, make the index (login,name).  When the index is read the data is in memory. Don’t add every column you just double the disk space and magnify the disk access.

More Tips

  1. Check that all tables have PRIMARY KEYs on columns with high cardinality. Primary keys must be unique.
    A column like, `gender` low cardinality (selectivity), an id column (Int – auto increment) is a good candidate to become a primary key.
  2. All joins (inter, outer, ‘,’) should have indexes.
  3. Fields you often search on (appear frequently in WHERE, ORDER BY or GROUP BY clauses) need indexes.
    But don’t add too many: the worst thing you can do is to add an index on every column of a table.
  4. Don’t use DISTINCT when you have or could use GROUP BY
  5. Open to the server just before you are going to use it.  Unless you are using a persistent connection library, don’t open a database connections and then run minutes of calculations before making your query.  You may find your connections has been “gone away” before you make your query.
  6. When your index many columns, create a hash column. Then your query will look like:
    SELECT *
    FROM table
    WHERE hash_column = MD5( CONCAT(col1, col2) )
    AND col1=’aaa’ AND col2=’bbb’;
  7. Use less RAM by declaring columns only as large as they need to be to hold the values stored in them.
    Use CHAR type when possible (instead of VARCHAR, BLOB or TEXT) — when values of a column have constant length: MD5-hash (32 symbols) or  ICAO or IATA airport code (4 and 3 symbols). This is also true for indexes.  If only the last 4 symbols are unique index only that part.
  8. Use SQL_NO_CACHE when you are SELECT-ing frequently updated data or large sets of data.  This way you will not kick good data out of the cache.
  9. Avoid wildcards at the start of LIKE queries.  (LIKE ‘%find%’).  Finding ‘1234find’ in 10000 records requires up to 40,000 searches.
  10. Normalizing redundant data is good but don’t split a table because you have too many columns.
  11. Think of storing users sessions data (or any non-critical / high access data) in MEMORY table — it’s very fast.
  12. Divide complex queries into several simpler ones — they have more chances to be cached, so will be quicker.
  13. A column must be declared as NOT NULL if it really is. This speeds up table traversing.
  14. If you usually retrieve rows in the same order like expr1, expr2, …, make ALTER TABLE … ORDER BY expr1, expr2, … to optimize the table.
  15. Don’t use PHP loop to fetch rows from database one by one just because you can — use IN instead, e.g.
    SELECT *
    FROM `table`
    WHERE `id` IN (1,7,13,42);
  16. Reuse your database connections.  Opening a new connection to the database will add one or more seconds to your query.
    In PHP use mysql_pconnect() to open a persistent connection with mod_php. Perl provides persistent connections with Apache::DBI with mod_perl. Python does not have persistent connections in mod_python.  But you can maintain them in your application.  (http://www.modpython.org/FAQ/faqw.py)
  17. When inserting data, insert only those values that differs from the default. This reduces the query parsing time.
  18. Use INSERT DELAYED or INSERT LOW_PRIORITY (for MyISAM) to write to your change log table.
    Also, if it’s MyISAM, you can add DELAY_KEY_WRITE=1 option — this makes index updates faster because they are not flushed to disk until the table is closed.
  19. For your web application, images and other binary assets should normally be stored as files.
  20. That is, store only a reference to the file rather than the file itself in the database.

Mark Grennan

]]> http://www.mysqlfanboy.com/2010/04/developer-tips-using-mysql/feed/ 3