iDogg

Red lights of impending doom

I walked in this morning and got logged in.  Shortly after that, I was approached by a coworker stating that the server room was really loud.  I wandered down and heard the server fans screaming away before I even got to the door.  As soon as I opened the door I was blasted by a head wave as hot as some of our mid summer heat waves here in eastern PA.  At first, I felt like I was walking into some crazy bizarro server room.  Every server was flashing amber or red and there was a slight odor of burning electronics.

I quickly realized the 1 year old AC unit had crapped out. The server room has two doors so I opened both doors and grabbed a few box fans to start moving air through the room. I then started to shut down most of the servers and a large chunk of the network gear. Somewhere during that, I called our facilities department and alerted them of the situation. They began to work on the AC unit and determined that a fuse blew, which has been replaced and the unit is back on. The server room is still cooling down now and I have slowly been bringing servers and other equipment back online.

There are a few issues here. Ultimately, things break and we need to have some sort of control of the environment of that room or at the very least, a way to monitor it. First of all, I should have an independent temperature sensor that will alert me when it gets too hot in there. If someone hadn’t alerted me to the problem when they did, there could have been a shit load of issues if equipment started to fail.

The other issue is that I’ve always been toying around with a network monitor system, but I’ve never been able to settle on anything. I’ve mucked around with Zenoss and nagios but I’ve never stuck with anything. I need to just find a proper system and knuckle down and get it configured.

There are lessons to be learned from this morning!

iScsi SAN or NAS with VMWare

A while back I posted something about iScsi SANS.  What I’d really like to see, once all is said and done, is to be able to virtualize almost every server we have.  I’ve been eyeing up the Lefthand redundant storage solution but the price is always an issue.  Up until recently, the likelyhood of moving forward with virtualizing our servers and storage has been out of the question due to funding.  Now, I’ve seen a small crack of light from the door accidently being opened up a crack so I’m just going to pretend it’s going to open and I’ll be showered with budget numbers and blank purchase orders.

I just noticed a post by a netapp employee, Dave.  I haven’t thought of running VMWare over NFS to a NAS.  To me, without looking at pricing*.  Any thoughts on this?

*Sales people, when someone from a K-12 Public School District calls for pricing, it’s not because they’re ready to buy.  We’re calling because if we can’t afford it, we’re not buying it.  I don’t even bother getting too deep into a possible project without knowing the pricing because that’s the bottom line regardless of features.  To the bosses, it’s as simple as why and how much?  When I see “call for pricing”, I cringe.  When someone tells me I have to sign an NDA to get pricing, goodbye.

Sysadmin Forums?

I’ve always been a fan of web based message boards since UBB came out many moons ago.  Doing a quick google showed a few forums out there, some part of bigger boards like ars, and some that have seemed to be pretty dead.

If there isn’t anything dedicated to sys admins our there that is active, I wonder if there would be any desire to see one here?

Apparently, the best way to fix a problem…

…is to walk away from it for about a month. In this case anyway.

We had a lot of legacy cruft in our network that has been and continues to be slowly phased out.  We have recently replaced our content filter running Bordermanager with a Barracuda 610 web filter appliance…sort of.  There is a legacy web based app which we have to run through the summer.  When I cut everyone over to the Barracuda away from Bordermanager, it broke the web application.  Right now, students go directly through the Barracuda, everyone else goes through the old proxy box which in turn goes through the Barracuda.  That wasn’t really a large issue, but the proxy server has been locking up cold lately.  The only thing that really changes on that box is the web cache processed by the proxy nlm.  Maybe it was choking on something there…or I’m just grasping at straws.  Either way, I don’t think it matters much, since ultimately that box needs to go away.

I finally got to spend a couple of minutes on the problem today.  NSLookups on the hostname worked fine.  But I couldn’t ping it from my test machine.  I could ping it from my own machine without any issues.  I figured out what was going on.  The web app server is on the old legacy 10.0.0.0/8 network where my workstation and the proxy server all reside.  So, routing is no issue there.  The default gateway of the proxy and my machine is our core 4507.  Unfortunatly, due to more legacy issues, the default gateway of the web app server is an old pfsense firewall connected hooked to our old ISP connection so it can’t have a proper gateway to the inside of the network.  That goofy setup is only around because of DNS issues and the fact that Oracle application servers, at least our version, are incredibly stupid and need a complete reload if you want to change the hostname.  There’s more to that story, but suffice it to say, we have to keep this setup until the summer.

To fix the whole mess, or rather duct tape it together, I just configured static routes into pfsense for the rest of the network.  So, maybe walking away for an hour would have worked, I just chose a month.  Also, this is a testament to keeping things simple.

“Breaking” my laptop

I made the mistake of creating a DHCP exception based on the MAC address of my wired interface on my laptop.  I promptly forgot I did it.  It wouldn’t be an issue in a small environment, but we have roughly a dozen data access vlans which could be recieving DHCP addresses.  I kept running into the problem where at work, I couldn’t get an address from our DHCP server.  The complicate things further, but ultimately made me realize what the problem was, both my Windows and Linux installs bombed out pulling an address.  I knew the issue wasn’t the actual NIC as it worked fine at home and on my little pix 501 test box.  Yesterday I realized that I had put the exception into one of the DHCP scopes.

When my laptop tried to pull an address, it would see the DHCP exception and try to assign it a specific address.  That address was in a VLAN that wasn’t allowed in most of buildings/areas.  It’s always fun to sabotage yourself!

Adding a zero to files in a directory

I had a directory of pictures that I needed to prefix a zero to in the filename.  This is a simple bash script to do it.  I’m posting it here for when I forget what I did.  ;)  Cygwin on my work box running XP barfed running on it.  I used my Opensuse install on my laptop and just mounted a share on my XP machine and ran it from linux and it worked like a charm.

#!/bin/bash
for f in `ls`
do
echo “Prefixing Zero to filename $f…”
mv $f 0$f
done

Simple file sharing and Sophos

We’re a Novell shop. No domain. Workstation policies are handled via zenworks. We have recently switched over to Sophos for virus scanning. Because we’re not running in a windows domain environment, I’m having difficulties forcing the virus scanner out using the Sophos enterprise console. I can manually install the client and the client will grab updates and policy settings from a Novell “share”, but the enterprise console is crippled.

Here’s the scenario. Workstation running XP PRO SP3. 2003 Server running the enterprise console. Both machines are in the same workgroup. I have a user on the server with the same username and password as a user on the workstation. The workstation user is in the ‘administrators’ group. I have a workstation policy(via zenworks) which will turn off simple file sharing at logon. At that point, I can push the sophos scanner out to the machine and/or simply go to \\machinename\C$ and see it’s C: drive. If the machine is sitting at the logon screen, I can’t push the sophos client out or even go to \\machinename\C$ to see the C: drive.

Is this by design? If so, is there anyway around it?

Lack of motivation

I figure with any job, regardless how much you love it, you will run into times when you completely lack motivation.  I’m in a job where, beyond my own personal motivation, the only real motivation is not to get fired(Peter Gibbons, 1999).  I’m at an organization where good performance is rarely met with monetary rewards and making a vertical career moves means you have to “move out to move up”.  While I have gripes with the situation, it’s my own problem and I choose to be where I am.  Beyond all of that, my own motivation is what really keeps me going; my motivation to learn and suceed.

I’m going to make an assumption that not everyone is motivated 100% of the time.  How do you combat long droughts of motivation?  Beyond that, how do you know if it’s not just a matter of motivation and you have become stagnant in a position?

SAN appreciation station

I’ve begun some initial looks at some SAN hardware for shared storage.  I’m just looking at iSCSI.  Fibre Channel is out of the question due to price.  I already have a home grown iScsi SAN/cluster running on top of NetWare, but it’s limited to iScsi Initiator access using NSS.  While its doing it’s job, it’s basically featureless.  Moreover, while the cluster resources are redundant, the shared storage is not.  There are some iSCSI appliances out there that will allow for redundancy between units.

That’s the major feature I’m looking for.  I hate the idea of having all of our eggs in one basket.  Even our core switches have redundant supervisor blades and power supplies and we have redundant port channel links to the IDFs spread out on separate gig blades in the event that one of those eat themselves.  We really need that redundancy with our storage.  We have a nice infrastructure in place to allow for iSCSI traffic between redundant sites.  Our main campus is a couple of miles away from our secondary campus, but we own the fiber on the poles in between.  So it’s one consitant LAN, no WAN links to worry about.  Beyond that, I’ve read a bit about snapshots and being able to back up data without the hassle of agent software on remote servers that has a tendancy to crap out.

Initially I’m looking at two devices, Lefthand’s Multisite San and EMC’s CLARiiON AX4.  If anyone has any advice, I’m all ears…er, eyes.

Verizon, ugh

In what seems to be a common theme with voice and data providers, I find it’s impossible to find anyone at these companies who know what any other part of the company is doing and who is responsible for what.  Verizon “owns” the DNS records for our domain at the moment and the person who handled all of that for us is not longer working at our organization.  I’ve been trying to get in touch with our sales “rep”, who comically won’t answer her phone and her voicemail box is full.  Prior to that, I was given the run around by someone at Verizon support who claimed they needed to snail mail me a form which would take 2 weeks to get to me.  It’s been over a month and I haven’t recieved anything. Does it have to be this hard to get anything done?