Today at the company where I work, our remotely hosted server went down suddenly. Nobody know why, and then the shit started to hit the fan, with emails flying back and forth, and phones ringing off the hook in search of an answer to that one timeless question, WTF ? Of course, everything filtered towards me eventually because I'm considered responsible for the server (even though I'm no where near it and if I don't have a network connection to it, I can't do shit). After several phone calls back and forth between our former partners (who happen to be in the same building as our ISP and still maintain our server), and our ISP, and our firewall technician (who, oddly, is outsourced by our ISP and doesn't technically work for them), it was discovered that the machine was alive, but not responding to any network traffic, so the quickest and simplest solution was to just get somebody to reboot the machine. Eventually, one of our former partners' employees went down to the cage and rebooted the server. Later, in an email explaining what had happened, that same employee (who happened to be their sysadmin) explained that their DVD backup had frozen the whole box and that this situation had never happened before. (This during our peak period for the day).
Now, I can believe that (I've had weird shit like that happen to me, so I can't really bitch at the guy.) But it brings to mind the need for good software and why good software should be written : if you write software, and it fucks up, then the managers of those using your software don't see it as your fuckup, they see it as their employees' fuckup, and hold them responsible. I hate the thought of being one of those employees (and very nearly was one today). This has brought forth some small amount of Karmic inspiration to put more time into testing and finding those odd failure modes for my own software that you wouldn't think can happen, but can because you eventually hit an odd and unexpected set of circumstances. Just something to think about if you're a serious developer and you're reading this.