If the first column of your CSV file is in sorted order you can do a binary search to find your data.  But what if you need to find data in the second or third column?

If you have a separate index file pointing to the first byte of each line you could seek to that position in the CSV file and get your data.  Given a file with only your needed column and the byte offset and length of the line you can search the index to find the pointer to the position in the CSV file.

SAMPLE DATA
“record”,”title”,”data”
1,”test1”,”data1”
2,”test3”,”data3”
3,”test2”,”data2”

SAMPLE INDEX
“test1”,24,17
“test2”,58,17
“test3”,41,17
“title”,0,23

Here is a simple perl program to create just such an index.

#!/usr/bin/perl
# csvindex.pl

use strict;
use warnings;

use Text::CSV_XS;

my $file = $ARGV[0];

my $csv = Text::CSV_XS->new ({
binary    => 1,
auto_diag => 1,
sep_char  => ','    # not really needed as this is the default
});

open(my $data, '<:encoding(utf8)', $file) or die "Could not open '$file'\n";

my $head = 0;
my $tail = 0 ;

while (my $fields = $csv->getline( $data )) {
$tail = tell $data ;
printf("\"%s\",%d,%d\n",$fields->[$ARGV[1]], $head, $tail-$head);
$head = $tail;
}

This command create an index of the second column (number start with 0).

$ perl csvindex.pl 2009.csv 1 | sort > 2009.cidx

A binary search can be done with the linux ‘look’ command.

$ look \"OK\" 2009.cidx
"OK",2134,59

Here is a simple program to seek into a file and print ‘n’ number of characters.

#!/usr/bin/perl
# csvseek.pl
use strict;
use warnings;

use Text::CSV_XS;

my $file = $ARGV[0];

open(my $data, '<:encoding(utf8)', $file) or die "Could not open '$file'\n";

seek $data, $ARGV[1], 0 ;

my $line = "" ;

read $data, $line, $ARGV[2];
chomp($line);

print $line . "\n" ;

Here is how it is used.

$ perl csvseek.pl  2009.csv 2134 59
OKLAHOMA,OK,40,"4,765","1,251","2,160","1,024",6,"9,206",

I’d like to see these tools added to CSVKit.

What I’d really like to see is this index/s added to MySQL’s CSV engine.  Would anyone like to help me get this done?

Tweet

The bug number fixes are from SourceForge.  https://sourceforge.net/tracker/?atid=628964&group_id=101066&func=browse

# 2.5.5 MTG – (2011-07-21)
#    – Bug – Typo Ureadable Unreadable config file line 424 – ID: 3316825
#    – Bug – Change “#!/bin/bash” to “#!/usr/bin/env bash” – ID: 3292873
#    – Bug – problem with excludes – ID: 3169562
#    – Bug – Total disk space on symbolic links – ID: 3064547
#    – Added DEBUG option to only print the commands that will be executed.
#    – Bug – WHICH command didn’t work if there was a WHICH alias.
# VER 2.5.4: MTG – (2011-01-28)
#    – fixed bug in rsync process.
#    – Added the ability to backup only a single table by naming the table like
#      database.table
# VER 2.5.2-02:  MTG – (2010-12-29)
#    – Added file promission settings (chmod) to directory and file creation points.
# VER 2.5.2-01:  MTG – (2010-11-06)
#    – Added ‘-R’ to the mysqldump options to include stored procedures in the backup
#      by default. (suggested by Zack Evans)
# VER 2.5.2:  MTG – (2010-11-04)
#    – Added option to archive (rsync) the local backup files to a remote locations
#      using the COPYDIR varaible.
#    – Added option to copy files into a directory based on the hostname using the
#      variable HOSTNAME.  This allows the script to be run from a shared storage directory
#      ( SBM, NFS, NetApp) the data to be kept seperate places.
#    – Added option to backup all database schemas only using variable FULLSCHEMA.
#    – Added option to backup MySQL configuration file, my.cnf and remove files older then seven
#      days from the BACKUPDIR directory.
#    – Added –master-data=2 and –single-transaction to include a comment with the master server’s
#      the binary log coordinates. If used the CHANGE_MASTER_TO line must be uncommented.

The project is at https://sourceforge.net/projects/automysqlbackup/

You can download my version at http://www.mysqlfanboy.com/Files/automysqlbackup.sh

Email me or leave a comment if you have any trouble.

Tweet

Do your queries, using that table, get blocked because the table is locked?

Do the waiting queries create idle connections slowing down the table load?

Do you wish you could just replace the table?

Years ago I was told you can replace CSV tables by simply replacing the CSV file. I figured this would also be true of a MyISAM file and it is. I use this perl script to replace MyISAM tables forcast and current observation weather data. The processing and tables are created on another computer. Weather forecasting is CPU and database expensive. I then copy (rsync) the files to the production system and run this script.

#!/usr/bin/perl
################################################################################
################################################################################
# mg_hot_replace_table.pl - Hot Replace a MySQL table.
#
# 2010-05-01 Mark Grennan - Weather Decision Technonigy Inc.
#
################################################################################
################################################################################
 use DBI;
 use File::Copy;
 use Getopt::Long;
 use Pod::Usage;
################################################################################
# These items need to be modified for your needs.
################################################################################
 $mydir="/data/mysql";           # MySQL data directory
 $stage="$mydir/stage";          # Directory containing the NEW data
 # database information
 $db="point_forecast";           # Database that containes the table to be replaced.
 $tb="uv_hourly";                # Table being replaced.
 $host="127.0.0.1";              # IP or FQDN of the MySQL server (localhost)
 $port="3306";                   # Port used by localhost (3306)
 $userid="dba";                  # User with access to database.
 $passwd="dbap@sswd";            # Password for the user.
################################################################################
# take command line options
################################################################################
 $help=0;
 &GetOptions("mydir=s" => \$mydir, "stage=s" => \$stage, "u=s" => \$userid, "p=s" => \$passwd, "port=i" => \$port, "db=s" => \$db, "tb=s" => \$tb, 'help|?' => \$help);
 if ($help) {
 &usage;
 exit 1;
 }

################################################################################
# main
################################################################################

#&check_table;
print "Connecting to database $db.$tb on $host:$port\n";
# make connection to database
$connectionInfo="DBI:mysql:$db:$host:$port";
$dbh = DBI->connect($connectionInfo,$userid,$passwd) || die ;
&lock_table;
&flush_table;
&move_files;
&flush_table;
$sth->finish()
 or die "Couldn't execute statement: " . $sth->errstr;
print "Disconnecting from MySQL.\n";
$dbh->disconnect        # disconnect from database
 or die "Couldn't execute statement: " . $sth->errstr;

print "DONE\n";
exit 0;

################################################################################
# Subrutines
################################################################################
sub lock_table {                # Lock table to prevent READs
 print "Locking Table\n";
 $query = "LOCK TABLES $tb WRITE;";
 $sth = $dbh->prepare($query);
 $sth->execute()
 or die "Couldn't execute statement: " . $sth->errstr;
}
sub unlock_table {              # Un-Lock table to allow use.
 print "UnLocking Table\n";
 $query = "UNLOCK TABLES;";
 $sth = $dbh->prepare($query);
 $sth->execute()
 or die "Couldn't execute statement: " . $sth->errstr;
}
sub flush_table {               # Flush table cache data.
 print "Flushing Table\n";
 $query = "FLUSH TABLE $tb";
 $sth = $dbh->prepare($query);
 $sth->execute()
 or die "Couldn't execute statement: " . $sth->errstr;
}
sub move_files {                        # Move new data to table files being used.
 print "Moving Table Files\n";
#    `rm -f /data/mysql/$db/$tb.frm /data/mysql/$db/$tb.MYD /data/mysql/$db/$tb.MYI`;
#    `mv $stage/$tb.frm  $stage/$tb.MYD $stage/$tb.MYI /data/mysql/$db`;
 if (-e "$stage/$tb.frm" and -e "$stage/$tb.MYD" and -e "$stage/$tb.MYI" ) {
 move("$stage/$tb.frm", "$mydir/$db");
 move("$stage/$tb.MYD", "$mydir/$db");
 move("$stage/$tb.MYI", "$mydir/$db");
 } else {
 print "There is a file missing! Files NOT moved.\n";
 }
}
sub check_table {
 $status = `mysqlcheck $stage $tb`;
 print "Status is  $status";
 if ($status) {
 print "New data table is BAD!\n";
 exit 2;
 }
}
sub usage ($) {
 my ($message)= @_;
 if ( $message )
 {
 print STDERR "$message\n";
 }

print <<HERE;

$0 [ OPTIONS ]
Options are:
 -mydir=      Directory containing mysql data ($mydir)
 -stage=      Staging database ($stage) where new data table is held
 -u=          User name ($userid)
 -p=          Password ($passwd) -port= Port ($port)
 -db=         Destination database holding destination table ($db)
 -tb=         Table ($tb)
 -help        Prints this help
HERE
}

Tweet