I admin the traditional “server room” setup.  Multiple servers, each with their own OS and their own specific task.  Sometimes I’m forced to double up applications or tasks on one box.  Generally, I prefer to keep one specific duty to one server.  That way, I can make things less complicated during an update, and losing a single box generally won’t seize the motor so to speak.  The downside to this is that I have to have more hardware on hand.  We have a fairly modest server setup.  Most of the servers live in a standard 19″ Dell rack.  I have a mishmosh of 2U and 1U Dell Poweredge boxes of various age and capacity.  I have a few non Dell boxes in another rack.  Being a non profit, we tend to run servers until the wheels fall off.

Every piece of critical equipment is hooked to one of four stand alone UPS units.  That includes servers, network switches, monitors and KVMs.  When I first started in the position, we dismantled a large freezer which freed up capacity on the building backup generator.  With the help of district staff, we were able to put in four 20 amp curcuits which in the event of losing utility power, would be fed from generator power.  The UPS batteries only needed to hold on for a minute or two until the generator came online and the power was regulated.

That’s enough back story.  The other afternoon, the main transformer for the building caught fire and we lost power.  The generator came online, but not all of my batteries switched back to “utility” power.  I began to lose servers.  To make matter worse, the fire was 30 feet or so from the generator which is powered by natural gas.  The fire department had us kill the motor.  The rest of the servers went offline and the room was dark and quiet.  This episode exposed a bunch of problems with our data center.  First, one of the emergency circuits was tripped.  Each circuit runs 12 gage copper and can’t really be changed to a 30 amp breaker without new wiring.  Second, the emergency generator is almost at its max on load.  The load is spread out best as possible, but the total load is getting to be too much.  Finally, even if I could keep all of the equipment running, I can’t cool the room during an extended power outage.  That forces me to bring non critical equipment offline.  The insult to injury during this event was, no lights in the room were tied into the emergency circuits.  Try working with a flashlite in your mouth.  Good fun all around.

This is your rock and hard place.  I can’t lessen the load without spending money on a virtual infrastructure.  Conversely, to increase “IT” capacity and allow for air conditioning, we need an all new generator and new circuits run to the data center.  From my point of view, we have to do both.  We can keep the 20 amp circuits assuming we go the VM route.  That way, I can just virtualize all the one off servers onto a few bare metal boxes depending on the strategy.  Secondly, some provisions have to be made to incorporate cooling into the mix.

And finally, a light or two would be nice…