We saw a large number of our Teleworkers lose Internet access about the 1:30pm EST time frame. Issue cleared about 10 minutes later. From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Raman Sud Sent: Monday, August 01, 2011 1:53 PM To: outages@outages.org; michael.self@Phonefusion.com Subject: Re: [outages] Level3 TFN's We are seeing issues in phoenix on the transit side Sent from my Android phone using TouchDown (www.nitrodesk.com) -----Original Message----- From: Michael R. Self [michael.self@Phonefusion.com] Received: Monday, 01 Aug 2011, 10:51am To: outages@outages.org [outages@outages.org] Subject: [outages] Level3 TFN's Anybody else seeing Level3 inbound traffic (toll free) problems? I have about 40,000 TFN's that seem to be not working. Mike. ********************************************************************** This e-mail message and any attachments contain confidential information from Medco. If you are not the intended recipient, you are hereby notified that disclosure, printing, copying, distribution, or the taking of any action in reliance on the contents of this electronic information is strictly prohibited. If you have received this e-mail message in error, please immediately notify the sender by reply message and then delete the electronic message and any attachments.

Wow. Something must have been up. I noticed some issues with paths that normally take XO (Bay Area) -> TWTC and XO (Bay Area) -> Qwest right around the same time. I don't believe that we use Level3 for these paths, but we definitely took a latency and path reliability hit. Makes me wonder what the root cause here is. Keeping on monitoring.... Cheers, jof On Mon, Aug 1, 2011 at 10:56 AM, Bower, Zeek <Zeek_Bower@medco.com> wrote:

We lost roughly 300 Texas area teleworkers from 13:20 to 13:35 EST.  Mostly Comcast and Time Warner connections, most of which are normally utilizing level3 backbone connections.

Zeek Bower

From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Michael R. Self Sent: Monday, August 01, 2011 1:34 PM To: outages@outages.org

Subject: [outages] Level3 TFN's

Anybody else seeing Level3 inbound traffic (toll free) problems?

I have about 40,000 TFN's that seem to be not working.

Mike.

________________________________ This e-mail message and any attachments contain confidential information from Medco. If you are not the intended recipient, you are hereby notified that disclosure, printing, copying, distribution, or the taking of any action in reliance on the contents of this electronic information is strictly prohibited. If you have received this e-mail message in error, please immediately notify the sender by reply message and then delete the electronic message and any attachments.

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

From L3's portal: 8/1/11 7:12:56 PM GMT The IP NOC reports that the network equipment self restored, resolving the routing issue and restoring services at approximately 17:55 GMT. The IP NOC states that they will continue to work with Level 3 OPS Engineers and the equipment vendor to isolate the root cause of the service interruption. The Level 3 TSC has confirmed that all customer services have been restored and the IP NOC will continue to monitor for stability. 8/1/11 6:29:26 PM GMT The IP NOC reports that a routing issue failure between Dallas, TX and Los Angeles, CA is impacting IP services in multiple markets. The IP NOC has engaged the equipment vendor, as well as Level 3 OPS Engineers and continues to investigate to isolate the issue at this time. On 8/1/11 2:39 PM, Chris Wright wrote:

L3 master case 40976066, DFW route reflector took a dive @ 17:22 UTC.

-Chris

On Mon, Aug 1, 2011 at 1:59 PM, Bower, Zeek <Zeek_Bower@medco.com <mailto:Zeek_Bower@medco.com>> wrote:

We lost roughly 300 Texas area teleworkers from 13:20 to 13:35 EST. Mostly Comcast and Time Warner connections, most of which are normally utilizing level3 backbone connections.

*Zeek Bower*

------------------------------------------------------------------------ This e-mail message and any attachments contain confidential information from Medco. If you are not the intended recipient, you are hereby notified that disclosure, printing, copying, distribution, or the taking of any action in reliance on the contents of this electronic information is strictly prohibited. If you have received this e-mail message in error, please immediately notify the sender by reply message and then delete the electronic message and any attachments.

_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages