The root cause was a faulty supervisor card on a router. It was scheduled to be replaced tomorrow at 6pm GMT. When the router crashed again today they decided to perform the maintenance early to stabilize things. We saw a similar issue on Tuesday afternoon. Hard down for 5 minutes and then about an hour of lingering bgp issues. Sent from my mobile device. -------- Original message -------- From: Chris Stone via Outages Date:09/04/2014 6:56 AM (GMT-07:00) To: outages@outages.org Subject: Re: [outages] Major Level3 outages? Just got this from L3 on my ticket: *** CASCADED EXTERNAL NOTES 04-Sep-2014 12:26:58 GMT From CASE: 8320222 - Event The IP NOC has advised that the field technician successfully completed the card replacement and the IP Technical Service Center has been engaged to verify restorations with the impacted customers. *** CASCADED EXTERNAL NOTES 04-Sep-2014 12:33:52 GMT From CASE: 8320222 - Event The IP Technical Service Center has verified restorations for the impacted customers, and services will be monitored by the IP NOC to ensure stability. If additional issues are experienced, please contact the Level 3 Technical Service Center at 1-877-4LEVEL3. Chris On 09/04/2014 06:25 AM, Morgan Miskell wrote:

Might be localized. East Coast (Charlotte) everything appears normal.

Full BGP Tables loaded, no large withdrawals etc...

On Thu, 2014-09-04 at 06:01 -0600, Chris Stone via Outages wrote:

...

Seeing drops in everything I've tested in the Denver area on Level3. My BGP sessions with them are all down since 05:53:11. Ticket just opened with Level3, but wondering is anyone else has any insight.

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages