This just in from TENET 9 Feb 15:47:
Hi All,
There was a brief yet critical outage on the network just now, however the network has now stabilized.
On Monday we experienced a failure of one of the ES20 line cards in the Cape Town router. This caused a brief outage during this time, and in order to rectify this, we moved one of the backbone circuits off the faulty card onto a backup card. Because of issues this faulty card was however propagating onto the network even after this port move, the decision was taken to replace the faulty card at first available opportunity under an emergency change procedure. These cards are meant to be hotswappable in the routers as per Cisco’s documentation, however, on removal of the faulty card and insertion of the new card, the router malfunctioned and rebooted the secondary working card on which the alternate backbone link was terminated. This resulted in BOTH backbone links out of Cape Town being offline while the cards reloaded and the router resynched.
The cards have now both loaded correctly and the router is back functioning normally with the faulty card removed and a new card inserted into the system.
TENET apologizes for the inconvenience caused.
Many Thanks
Andrew Alston