Ratio of People to Cake: Untested Updates and a Dark Day at the Office
I came across this story in an online IT digest. I think it speaks volumes about learing from your mistakes. After you read it, what are some experiences you have had in the IT field that have made you learn a lesson you have never forgotten?
IT, for many companies, is an afterthought. If it works, no one even knows there are IT guys behind it and they even question WHY they need an IT department. The IT department is always the last group to get the flyers for the company parties, picnics, new hires, birthday cake in the break room. One fateful day, I even failed to get the memo the company would be closed for the day — and to this day, I wish I wouldn’t have got out of bed that morning…
As I pulled up on my motorcycle, I noticed the parking lot was eerily empty, but that wasn’t so odd. Many times I had come in slightly earlier than normal and the parking lot was almost empty. Add to that, the recent construction and machinery deliveries expected that mandated the lot be cleared for cranes and other heavy equipment. So, without giving it another thought, I proceeded to the rear parking lot. To my surprise, that lot was also empty. Again, maybe I was just early. And as for the third-shift manufacturing employees? Maybe, just maybe, they all carpooled or got rides in. But, from this point on, I can’t help but look back and wonder, “Why didn’t I just leave?!?”
I walked into the building. It was dark, except for the twinkle of machinery, flashing status lights and the occasional office light left on. Confused and determined to get to the bottom of the situation, which was starting to bear an eerie similarity to the Stephen King thriller “The Langoliers,” I continued toward my office. I stopped at the snack machine to taste something to make sure it hadn’t gone stale, and that I was still in my reality. I got to my office and decided to call the only person I knew who would be working even if the plant was closed.
Me: “Why’s the plant shut down?”
Her: “Um, hello… Everyone knew the plant was closed. The memo went out last week.”
Me: “Ah! The memo. I was left out of… again!!”
Her: “Wait, are you IN the plant?” She began to laugh hysterically.
Me: “Goodbye!!” I gently put down the phone. (I hope you can read through that sarcasm.)
After a few minutes of full on adult tantrum, I reconciled with the fact that I should be used to this by now — even though this was more than the typical “cake in the break room” mishap. But, I was going to make this into a productive time. So I figured I stay in the office for two hours, do some quick maintenance on our ERP server and go home in time to have lunch with the family. It started out well enough. I downloaded some Windows Updates, installed some driver updates, made some driver updates, backed up the database, and did a couple of reboots. It was starting to seem like it was going to be an OK morning. I queued up a mix of Jack Johnson, Metallica, The Beatles, The Eagles, some ’80s hip-hop. I was determined: it was going to be a good day in “forgotten IT land.” And then…
I decide to brave it a little and install some of the latest Windows Updates, which I had not tested yet, on the server. What could possibly go wrong, right? Very small download, quick install, then a simple reboot — I’ll be home in time to make lunch. NOT!
Reboot attempt #1: Nothing. I waited 20 minutes.
Reboot attempt #2: Gone. All gone! Repair attempts, “Safe Mode” attempts, IT Super Secret tools — nothing works! Why did I get up this morning?
After eight more hours or server installation, software restores, database restore and reconfiguration, updates (the tested ones this time), I finally got the server back to a working, stable and — may I say — optimized state. After the panic subsided, I got on my motorcycle and headed home, arriving just in time to heat up leftovers for dinner.
I learned three things that day:
1. Always test your Windows Updates before applying them to your live environment.
2. Always make backups before you make any updates or changes. (I did, and I was glad.)
3. If the lights are out, no one’s home — and you shouldn’t be, either.
I plan all my updates, nowadays. I also make sure I have the right contacts available in case something goes wrong, and I make sure I have a quick recovery route. And, I ask for a plant schedule every two weeks. I haven’t missed cake in the break room ever since.
