Documentation and maturity

There are three stages of maturity when it comes to getting things work and documenting how you did it.

1) I got it working, hurray! Run away before it breaks.

2) I got it working, hurray! And I took notes while I was getting it working, so I’m good. Run away before it breaks

3) I got it working, hurray! And I took notes while I was getting it working, so I’d better start from scratch and make sure my notes work.

chkconfig, mysql

A crash today brought home a couple of things that I should have known before but didn’t, and now I do. I’m writing them here so I don’t forget, but also because google results were annoyingly useless without digging.

#1: chkconfig. That funny line in the init script that says something like “# chkconfig: 23456 88 22”. It’s easy to find out that the first batch of numbers is runlevels, but less easy to find out the meaning of the second set. They’re priorities; the higher the first number, the later in the boot process it starts. The higher the second number, the later in the shutdown process it’s killed. Also, if you change those numbers – this is obvious in retrospect, but I forgot – you need to re-run “chkconfig on servicename”.

This is apparently all defined in the LSB. Back in my day, there was no that thur LSB, never mind chkconfig. You just put scripts in /etc/rc.d and crossed your fingers.

#2: misamchk. This worked fairly well on my 72GB RAM server with a ~25GB database:

myisamchk --fast --update-state --force --sort_buffer_size=1G \
--key_buffer_size=2G --read_buffer_size=512M --write_buffer_size=512M \

The myisamchk manpage talks about “a lot of memory” being 512M so obviously it could use a bit of modernizing. That, or it really is MyFirstDatabase.

And this kind of stuff reminds me that while I need to keep my hand in, I’m just as happy to no longer be a sysadmin for a primary job duty.

(Edited Aug 2012 – neither should typing be, chkconfig on, not chkconfig start.)

Firesheep: black hats for the masses

The following will also be used as part of my day job.  To ITSec types, it’s likely old hat, but it may be new to some of my readers.  Some terminology in here is my-job-specific, but I did try to generalize it as much as possible.

What is it?

Firesheep is a readily-available and easily-installed add on for Firefox, a commonly-used web browser.  It allows the person using it to, within certain parameters, steal web cookies used for authorization on many common websites, mostly social networking but also, most notably, Hotmail.  This is a form of attack commonly known as sidejacking.  Note that while this document largely talks about uw-wireless, everything about that network applies equally to any open or unsecured wireless network, such as uw-guest or those commonly found in coffee shops, restaurants, and other places offering free service to customers.

Please note that discussing a tool does not condone its use.  In particular, the use of Firesheep on a campus network is a violation of the principles laid down in the Guidelines on Use of Waterloo Computing and Network Resources and, as noted by these Guidelines, is subject to discipline under the appropriate University Policy or Policies.  Use of Firesheep here, or anywhere else, may furthermore violate local laws and regulations on privacy, mischief, or wiretapping.

What does it mean?

Firesheep makes it easy for anybody who can click a mouse to be able to access these websites as if they were the person whose credentials they’ve stolen – and that person may not even be aware of the accesses.

How does it work?

The victim has to be on the same wireless access point, using the same wireless network, as the attacker.  That implies a certain physical proximity.  The wireless network must be open, and the victim must be using the network at the same time as the attacker, and must be accessing these sites (but see below, the victim may not necessarily know they were using a particular website, embedded content can cause problems).

When Firesheep is started, the attacker’s computer starts passively watching the network for authentication credentials.  When it sees such, it saves them and presents an icon for the attacker to click; this allows the attacker to easily access the website as the victim.

It should be noted that a user may inadvertently access a website.  Many websites will have badges to follow the site’s author on Twitter, or link to them on Facebook.  Sometimes the simple act of loading that website can cause your browser to send and receive authentication cookies, and therefore expose this information to the attacker.

I don’t use social networks.

Firesheep doesn’t only work on social networks, so you still may not be safe.  Web services as varied as Evernote, Cisco, eBay, Amazon, and Slicehost have had Firesheep handlers written for them.

Why doesn’t the University stop it from happening?

This attack takes advantage of two weaknesses in the way victims might access vulnerable websites.

The first weakness is that open networks are precisely as the name implies.  All clients transmit all data unencrypted using their wireless radio.  Like any radio, anybody can listen in.  This means that unless the client takes extra precautions to encrypt data, such as using TLS/SSL encryption on the data stream itself, that data is exposed for anybody within range who’s listening.  This means that many websites which do encryption properly, such as almost all banking websites, are not vulnerable to authentication theft in the way that Firesheep accomplishes it.

The second weakness is in the way the vulnerable websites perform authentication and authorization.  Without getting too technical, these sites rely on cookies, and merely having that cookie implies both that you are who you say you are, and you’re allowed to access the content you’ve requested.  Not all websites have this issue; as noted, banking websites don’t rely on this model.  Other sites such as GMail and corporate applications at the University of Waterloo don’t either, and so your credentials there are safe from this attack.

What can I do to protect myself?

The simplest thing you can do is not use open wireless networks.  These are ones which do not require a key or password to access.  uw-wireless is one such network, but many coffee shops and restaurants and other companies provide such networks for the convenience of their customers.  This also affords attackers certain conveniences.

The next-best thing is to not use sites vulnerable to Firesheep attack whilst connected to an open wireless network.  Be aware that some sites you visit may effectively force you to give up your credentials anyway, as noted under How Does It Work?

Some clients customized for use by some social networks, such as Tweetdeck, may allow these networks to be used in a manner that does not expose authorization credentials to Firesheep.  That does not necessarily mean that these clients always operate in a safe manner.

Some people have authored Firefox extensions which could potentially warn you about the use of Firesheep on your network segment.  The use of these extensions (the most commonly mentioned are FireShepherd and BlackSheep) is prohibited on University wireless networks, as they have a deleterious effect on the operation of the campus network and, not incidentally, of the remote service.


Developuction: when a service or host intended for development gets pressed into production. Usually this is done in a reaction to some external event, and nearly always results in fail somewhere down the road. If sysadmins are lucky, it just means the service collapses under a load it wasn’t designed to handle. If they’re not lucky, it means that somebody took a shortcut that compromised security somewhere, and the box gets well and thoroughly pwned. This is nearly always the fault of the sysadmin and not the developer who wrote the code, or the manager who caused the service to be pressed into production.
Developuction is a fact of life for many system administrators, and points to one or more serious issues in their shops. For instance, they could lack professionalism. This is generally ignorance, which is sometimes willful. They could be not granted any authority over technical decisions, in which case management needs to understand why it is that professionals are hired. But ultimately, it’s a sign that short-term answers are favoured over sustainable long-term stability. Sometimes shops get lucky with developuction, and later they run into trouble when this luck is confused with competence.
We’ve all done developuction at some point or another. Sometimes it really is the best way to solve a problem. The key is understanding when it’s acceptable, and being able to properly analyze the risks involved. It helps if you’ve not treated the development machines as a place where processes don’t matter because “it’s just a dev box.”

IT as a service organization

All my professional career, I’ve been a member of service organizations. Granted, by some standards, that has not been for very long, but since I graduated from university, I’ve been a technician for a small business that sold and repaired PCs, I’ve been an owner of a business doing exactly the same thing, I’ve been a sysadmin type in an academic support department, and I’m now a system administrator in an academic unit.
A faculty member once said to me, “It’s a relief to hear that you consider yourself to be in a service organization.” I took that to mean that he didn’t feel we (always) behaved as such. But how could it be organized any other way? We all report, ultimately, to a faculty member, and in our organization, it’s faculty members who make the requests and drive the agenda. He could have been remarking that my own behaviour contraindicated this belief, but I don’t think so. I will avoid false modesty and say that I know most faculty members and grad students for whom I do work are extremely happy with my work, as is my manager.
So why the statement? Has a clear mandate not been given to us, and if not, why not? (The fact that I ask this question gives a pretty clear answer: not really.) How do IT organizations perceive themselves? I used to think that mission statements were foolish, but I’ve changed my mind over the last several years. They allow an organization to not only publish a self-perception, but to ensure that staff members either are aware of this perception, or at least to not give them an excuse for being unaware of such. They allow people dealing with that organization to know what they’re getting into and what to expect.
Ultimately, a service organization needs to define itself in terms of the person or group of people to whom it is giving service. I’m not entirely sure that’s been done – not for our own group in particular, and not for most other service groups on campus. Even if statements of mission have been made, it’s not clear that they have been promulgated or are under regular review to ensure they’re still applicable.
Are you in a service organization? If not, how do you know that?