Perl Log Processing Goodies

Here are some Perl goodies I forget to write down once I’ve remembered them long enough to solve my problem.

Line range

If you want to print all the lines in a file after a certain line number:

perl -ne 'print if (2655641 .. -1)' some.log

This will print from line 2655641 to the last line (-1).

Omiting chunks

Sometimes you have chunks of lines, or multi-line records, that you want to skip.

2014-09-24    
DBD::mysql Integer Type Coercion

DBD::mysql knows that certain MySQL column types are integers with this call:

$sth->execute;
$nums = $sth->{'mysql_is_num'} || [];

You can iterate over $nums and see which columns are numeric-type columns. However, when you pull the data out, it’s converted to a string. So, what used to be as simple as this:

my $accounts = $sth->fetchall_arrayref({});

now requires this kind of trickery:

my @accounts = ();
my $is_num   = $sth->{'mysql_is_num'} || [];
while (my $row = $sth->fetchrow_arrayref) {
    push @accounts, {
        map {
            $Acct_Fields[$_] =>   ## key
              ($is_num->[$_]      ## check to see if it's int
                ? 0 + $row->[$_]  ## coerce to integer/numeric
                : $row->[$_])     ## leave it alone
        } (0 .. $#Acct_Fields)    ## number of keys
    };
}

This seems to work as a primary test to see if a scalar ($val) is an number-ish thing:

2014-09-10    
Use git-crypt to Store Secrets in Git

git-crypt ( github) keeps your secrets safe in a git repository. It decrypts on checkout and encrypts at commit using standard git hooks. Once configured, it is completely transparent.

I had a situation where the secrets were already in the repo and I needed to encrypt them (if you’re in this situation, you should also change your secrets because git log -p).

2014-07-15    
Perl List of Anonymous Hashes

Here’s a nice trick from perlfunc’s map entry:

To force an anon hash constructor use "+{":

    @hashes = map +{ lc($_) => 1 }, @array # EXPR, so needs
                                           # comma at end

to get a list of anonymous hashes each with only one entry apiece.
2014-07-08    
Renaming Hash Keys

Say you have a hash:

my %hash = (foo => 'fooey',
            bar => 'barey',
            baz => 'bazey',
            blech => 'blechey');

and you want to rename some of the hash keys, create another hash that holds the new names of the keys:

## old-name => new-name
my %new = (foo => 'foooo',
           bar => 'baaar');

Then this is a fast way to rename the keys. Measured on 2.4Ghz i5 MacBook Pro.

2013-11-07    
emacs Artist Mode

I don’t use emacs’ artist mode often enough to remember all of the keystrokes and finding the reference sometimes is effort than it’s worth.

To enable:

M-x artist-mode

to disable:

C-cC-c

Here is the list of commands from the artist.el file:

C-cC-aC-r  artist-toggle-rubber-banding
C-cC-al    artist-select-op-line
C-cC-aL    artist-select-op-straight-line
C-cC-ar    artist-select-op-rectangle
C-cC-aR    artist-select-op-square
C-cC-as    artist-select-op-square
C-cC-ap    artist-select-op-poly-line
C-cC-aP    artist-select-op-straight-poly-line
C-cC-ae    artist-select-op-ellipse
C-cC-ac    artist-select-op-circle
C-cC-at    artist-select-op-text-see-thru
C-cC-aT    artist-select-op-text-overwrite
C-cC-aS    artist-select-op-spray-can
C-cC-az    artist-select-op-spray-set-size
C-cC-aC-d  artist-select-op-erase-char
C-cC-aE    artist-select-op-erase-rectangle
C-cC-av    artist-select-op-vaporize-line
C-cC-aV    artist-select-op-vaporize-lines
C-cC-aC-k  artist-select-op-cut-rectangle
C-cC-aM-w  artist-select-op-copy-rectangle
C-cC-aC-y  artist-select-op-paste
C-cC-af    artist-select-op-flood-fill
2013-09-05    
Profiling with Devel::NYTProf

To selectively profile during only a portion of the code, invoke like this:

$ NYTPROF=start=no perl -d:NYTProf t/load.t -v

Then inside t/load.t add the following directives which tell Devel::NYTProf to start and stop profiling:

DB::enable_profile();
... ## profile this section
DB::finish_profile();

Add use Devel::NYTProf; in either the test file or in the module.

2013-08-20    
git bisect

Git has a nice feature called bisect that’s immensely useful for finding out where and when something broke.

For this example, we’ll use small integers for commit ids because they’re easier to reason about and orderly. In reality, they’ll be long SHA hashes.

Let’s say you have a commit history like this:

commit 12
Author: Joe

commit 11
Author: Scott

commit 10
Author: Scott

commit 9
Author: Scott

commit 8
Author: Miles

commit 7
Author: Joe

commit 6
Author: Dave

commit 5
Author: Bob

commit 4
Author: Joe

commit 3
Author: Dave

Pretend you’re Dave (your last commit was commit 6) and you do a git pull and an ‘install-dev’ and see that the site’s busted, CSS all over the place, whatever. Git bisect to the rescue. First, tell git you’re ready to have it help you find the problem:

2013-08-15    
SQL Fast Column Add

I found a fast way to add a new column to a table.

MySQL

  1. Create the new table like the old table (foo), but with the new column you want in it. Make the new table with a different name (foo_new):

  2. Add the records from the old table into the new table:

     INSERT INTO foo_new (col1, col2, col3)
          SELECT col1, col2, col3 FROM foo
    
  3. If you have a lot of records, you can disable index updates during insert:

2013-05-23    
Duplicate an SD Card in OS X

I made these notes as a result of my experimentation with a Raspberry Pi. I kept messing up the software on the SD card and needed to start over, but it took a long time to go through all of the RPi setup steps. By copying the partitions, I was able to restore relatively quickly.

Get the info of each partition you want to copy on your SD card using diskutil:

2013-04-28