QUESTION: Is the database truly transactional? If so read and understand this carefully.

InnoDB does row level locking. If your SQL is only inserting records the rest of the database should not be effected. The index for other changes is not updated until the commit. Other queries don’t know about the new records until then. However, In row-level locking, InnoDB normally uses next-key locking. That means that besides index records, InnoDB can also lock the “gap” preceding an index record to block insertions by other sessions in the gap immediately before the index record. A next-key lock refers to a lock that locks an index record and the gap before it. A gap lock refers to a lock that locks only the gap before some index record.

By default, InnoDB operates in REPEATABLE READ transaction isolation level and with the innodb_locks_unsafe_for_binlog system variable disabled. In this case, InnoDB uses next-key locks for searches and index scans, which prevents phantom rows. Phantom rows area problem related to replication. If you are not replicating this server or database you might want to change this setting.

Gap locking can be disabled explicitly. This occurs if you change the transaction isolation level to READ COMMITTED or enable the innodb_locks_unsafe_for_binlog system variable. Under these circumstances, gap locking is disabled for searches and index scans and is used only for foreign-key constraint checking and duplicate-key checking.

How InnoDB locking is handles is described at http://dev.mysql.com/doc/refman/5.1/en/innodb-lock-modes.html and http://dev.mysql.com/doc/refman/5.1/en/innodb-record-level-locks.html .

As for create a multi-value insert, you are on the right track. This will lower you insert time. You might also check if the client can and is using compression.

About your thousands of ‘INSERT INTO xxx VALUES (a,b,c)’ transactions. InnoDB tables I hope? MyISAM locks the entire table for an insert. If you change this to a multi-value insert you will be locking the table for a longer time. Yes, thousands of one line inserts may be a WAN problem, stacking up thousands of SELECT operations waiting on the LOCK to clear may also give you issues. The way it is now give breathing room for other transactions. InnoDB tables do row level locking and should not be an issue.

Be very careful if you use mysql-proxy in production. Any update of the code may change how it works and break your use.

You might also look at SQL Relay. The code is more stable. You might have to hack what you need into the code it self. I’m not sure if it has a scripting language.