How Mouse Poop Killed our VMware vCenter Server
With all the redundancy built into a server these days apparently a mouse is able to bring down a server. We had our VMware VirtualCenter running on an old Dell PowerEdge 2650 and the other day I was told it just wouldn't boot. This morning I went about troubleshooting, the LCD on the front said that there was no memory so I opened up the case to make sure the memory was seated correctly. As soon as I opened it I noticed what looked like rust on what I believe is the northbridge and other areas of the motherboard. Upon further investigation I noticed that it looked like mouse poop. Normally this probably would have seemed ridiculous but we were aware that there was a mouse in the office since an employee had seen it and we had noticed that it had been eating our food and pooping. The mouse seems to have gained entry into the server through one of the PCI slots that did not have a cover on it.
We have an extra working Dell PowerEdge 2650 so I put the two hard drives which were configured with RAID and thankfully the server booted and all is well. So, I guess a lesson learned is in addition to server redundancy, backups, and all the other fancy nice things that go along with a stable network infrastructure you need to make sure you don't have rodents in your server room, or at least make sure your PCI slots are covered.
