View Single Post
Lt. Commander
Join Date: Dec 2007
Posts: 120
# 25
02-05-2010, 11:26 PM
Originally Posted by Khabok

NOC guys at Cryptic? Good luck to you guys, and ignore the idiots that have no idea what you're going through right now.
God bless them every one. And if they are not the owners of their hardware.... if they are contracting it through a telco.... and they have to rely on telco monkeys .... and oh my...

I'm on vacation and just the thoughts of the possible problems gives me angina.

As an aside... we had one time when a switch wasn't handling spanning tree properly and took out 32 servers that had NICs which didn't establish link integrity until driver load during boot. The result was that ARPs never made it to the network because the switch went into port blocking *permanently*.

By the time I made it into the NOC and identified the firmware problem with the switch, 12 engineers were sitting there needing psychiatric care because they changed out all the network hardware and 5 servers under the belief it was a level 1 event. It was a level 1 event.... but all the network hardware had the firmware bug- and there was no patch. So no matter what they did nothing worked.

The solution?

6 8 port Netgear hubs from Best Buy. And flash the switches the next maintenance window with updated firmware.

It was the only time in my life I felt like both a hero and a villain. The applauded me on the way out... and hated me for the bill.