Using Stow on a Mac

For various reasons, I build some utilities outside of things like MacPorts and Brew, mostly interpreters where I want to ensure that my environment is my environment. I need all of Python, Ruby, and Perl, and sometimes different versions of Ruby as well.

Normally I build with –prefix=$HOME/app/name-maj.min then link app to whatever version I want to be the default. In $HOME/bin I make links to $HOME/pkg/app/bin and mostly everything just works. Except then I started running into problems with man pages, and setting this environment up can be a pain in my login shells. So I went looking for something that automated this. There’s just no other sensible way to do this on a Unix system.

By the way, at (the University of) Waterloo we have a(n) (in)famously obtuse packaging / configuration management system called xhier. Any time I think about this, I wish I had the GOOD parts of said system available. It has lots of nifty toys to automate building, configuration, and managing links, as well as a utility called showpath that helps you to manage both your PATH and MANPATH variables. And I’ve often declared that anybody who thinks about automating, including things like Stow, is at risk of re-inventing xhier.

Stow is a bit of an annoyance in that it, itself, requires Perl. The entire reason I like having my own separate Perl instance, in particular, is because of a long history of CPAN screwing me over. So I put my own version aside, run CPAN, and no operating system update blows my stuff away, and no CPAN will touch my OS stuff. I tried the MacPorts version of Stow and just couldn’t make it cooperate in integrating into my above flow – it would make links just fine in $HOME/stow, but it wouldn’t remove apps.

What finally worked was using my own $HOME/pkg/perl CPAN utility to install Stow. Then I was able to:

cd ~/pkg
stow -t ~/stow ruby-1.9.3
stow -D -t ~/stow ruby-1.9.3

and everything seems to work as expected. The above did not, with the MacPorts version, but it’s still at 1.3 and perhaps there’s issues with the older version, I don’t know. I don’t really like Perl very much any more, so I’m not inclined to delve into the whats wherefores and whys. I thought about looking at how the MacPorts version differed (in configuration) from mine, but then I cracked another beer, drank it and a few more, then laid down for a while and the feeling passed.

(Edited to account for Giles’ grammar correction and general persnickitiness. Thanks, Giles!)

packetfu followup

I went back to packetfu today (see here for my first talk about it) for a large collection of pcaps I have (about 30GB worth) hoping to use it to help me quickly get an overview of what I’ve seen.

Fortunately, a lot of the hoops I had to jump through previously are now non-issues; using gem to install pcaprub and packetfu itself made it dead simple. Even more fortunately, I didn’t bother trying it on my Mac initially, guessing – correctly, as it turns out – that RAM usage would go through the roof. I ^C’ed my test ruby script, pasted from my first post on the subject, after it hit 7.5GB resident on my 8GB test machine. I have Ruby 1.9.1 on that box, so I don’t think there’s much more I can do to optimise. Sadly, I guess I’m stuck with tcpdump and looping shell scripts for the time being, and I’ll try to follow up with the author, although I don’t know what help I can provide.

Doing pcap stuff with Ruby on a Mac

This is probably old news to anybody who’s used to Ruby on Macs, but I’m not.  I like using MacPorts when I can, so that’s where my Ruby runs from.

I went looking for pcap modules and found a bunch, but the most promising seemed to be packetfu.  It came with a caveat: “PacketFu is reported to work on OS X, assuming you can get pcaprub installed correctly.”

So the first step was to get pcaprub going.  You can get it at rubyforge, the trick is it has a README, a single C source file, and a small ruby script. It turns out the secret is to do ‘ruby extconf.rb’, which will create a Makefile for you, and you can then ‘sudo make install’.

The author of packetfu recommends sticking with sources, available here. Check them out like this: ‘svn checkout http://packetfu.googlecode.com/svn/trunk/ packetfu-read-only’ – I like doing this in ~/src, but your tastes may differ.  Use the pcaprub_linux directory, and again, ‘ruby extconf.rb’ followed by a make / make install.  The test script bombs out – by default it wants to use the fw0 device.  Nevertheless, I pressed on and did as suggested: ‘cd .. && sudo ruby setup.rb’ which finished with no failures.

Note that following these steps will pollute your MacPorts ruby install with files about which it knows nothing – I generally like to avoid this, but it seems like scripting languages insist on having their own packaging systems that make it not quite impossible to work with other such systems.

Then I wrote a little test script that should just count packets:

#!/opt/local/bin/ruby

require 'packetfu'
filename = ARGV[0] || exit
count = 0
incap = PacketFu::Read.f2a(:file => filename)

incap.each do |pkt|
 p = PacketFu::Packet.parse(pkt)
 count += 1
end

puts "#{count} packets"

I would not recommend doing as I did and turning it loose on a 711MB pcap trace – top showed the process using 1.9GB of memory before I managed to get a responsive terminal to kill it.  It fared slightly better on a 12MB trace (a subset of the larger ones) and correctly counted 85411 packets, but still hit 265MB:

real    4m0.359s
user    3m16.012s
sys    0m1.891s

I’m a fairly-incompetent Ruby programmer, but the above seems fairly straightforward – I’m not sure I’d want to turn this parser loose on anything of any real size.  While 711MB is a pretty big trace to be parsing, 12 is pretty small.  To be fair, I had other processes running at the time, including a Windows 7 VM that was patching (unbeknownst to me) but it’s a 4GB dual core machine.  I’d been hoping to more easily pull various bits of data out of the larger file, but it looks like I’m stuck with tcpdump and tshark for now.

ETA: I had been going to post again about xtractr, but it’s not a 100% Mac/Ruby solution. It provides an interface so that I could use Mac Ruby installs to talk to an xtractr instance, but so far as I can tell, there’s no 100% Mac solution. So I’m running it on an Ubuntu box, but I won’t bother detailing how I got it going – it was pretty straightforward, although it required some work I’d already done to make gems sane on an 8.04 system.