Friday, February 17, 2006

What a bitch

So work today was kind of a pain.

I set out to finish the work I have been doing for a couple of days, and got a bit distracted. I forgot they were moving all the equipment from the colo datacenter back here to the office. So I had to shut it all down so it could be tranported. And then just as they were driving back to the office with the equipment, I looked over at the wireless thermostat on my desk and noticed that it was 79 degrees in the computer room. It is usually about 70 degrees (averages between 69 and 72). This means that the air conditioning is broken and it will be 110 degrees in about half an hour. So I opened the door, put a fan in the doorway, and spent the next hour shutting down non-essential systems.

Then I made the fatal mistake. I instructed Vlad to power off Sun02 and Sun03 (which I had already issued shutdown commands to). Unfortunately he hit the power switch on Sun01 and Sun02. Now these are not your ordinary PCs. These are big-iron multiprocessor enterprise servers. And let's just say powering off one of these puppies without the proper shutdown procedure can cause pretty big problems. And that's exactly what happened. The root partition was screwed up on Sun01 and It took all afternoon to backup the data and get the system back on-line.

And then a little while later, four developers simultaneously screamed "what the hell??" as their sessions were terminated. Turns up Ben pulled the wrong power cable and knocked out seven servers. Ouch. The irony of course is that we had redundant power on the chassis and he unplugged both power cables leading to the servers.

Ok, enough bitching about work. Apparently GridApp has a three day weekend. This was an error. The CEO only found out last week and was literally like, "what do you mean we have off for Presidents' Day??" Too funny. We have been assured that we should not expect to get off for President's Day next year.

Had dinner with Ben at the Thai place I like on 20th street. Good stuff. I think it's the only sit-down restaurant I have come across in Manhattan that has $8.95 entrees. It's a nice restaurant with white tablecloths, candles and dimmed lights, but the food is inexpensive.

Afterwards I met up for drinks at Broadway Station with Dan, Matt, Jeff, and Dan's old roommate.

Not sure how I'm going to spend my weekend yet. Was hoping to get together with kdb but it looks like that isn't going to happen. I don't think she's pissed at me, but just genuinely bored/unenthused about the idea of hanging out.