Went Atomic

Thirteen hours yesterday. Eleven hours today. Cleared off old backups. Created new ones. Staged the latest Microsoft patches. In the office by six in the morning. Then a printing upgrade. And the reboots. What happens next? Of course things went atomic. But thankfully (as if it were a consolation prize) only the Citrix environment burst in flames. Several hundred folks completely offline. Others unable to open our primary customer-management applications. Spent the better part of two hours on the phone with Citrix support. Still unable to identify the problem. Maybe some additional patches would help. Maybe pull off the new patches, one at a time. Hours of work, with no guarantee of success any time soon.

Fortunately for Your Humble Narrator, he made snapshots of all the servers involved. Powered down the affected systems. Snapped back. Powered up. Voila. Everything restored to its former glory. More than a dozen hours of work erased with a couple of mouse clicks. Alas. Easy come, easy go.

The users were able to connect. The applications fired up. Sunlight through the storm clouds. And I nearly puked when it  all drew to a close. The severity of the outage finally biting into my old bones. Damage I'd wrought. Three hours multiplied by several hundred folks. Add up the cost of their loss of productivity and it likely amounts to more than my annual income. Not a cross I bear lightly.

Shortly thereafter, the Prez calls. His cell to mine. Wants to hear directly what I'd done. Was nice about it, of course. Gave him the exact times and efforts. Promised to make it right next time. And off we go, each to our own adventures. Not fun being on his radar. I crept away. As quickly and quietly as possible. No chest thumping or braggadocio from This Old Nerd. Lived to fight another day. Hopefully less atomic next time.