Confirmed. Lisle, IL Ryan Prihoda Systems Administrator dynaConnections www.dynaconnections.com On 02/02/2015 12:04 PM, Alex Forster via Outages wrote:

Confirmed in northern Delaware.

*Alex Forster* Network Engineer, Linode

On 2/2/15 12:52 PM, Mauricio Lizano via Outages wrote:

...

I just saw Windstream circuits go down across the board, from CA to PA. I saw circuits down from 11:34 to 11:44 Central. Most are recovering on their own but a few are not.

Anyone else feeling the pain?

Thanks, -- Mauricio Lizano

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

We didn't see sites go completely down. We just saw packet loss around 80-90% which is pretty much the same as being down. -Richey From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Bob Colon via Outages Sent: Monday, February 02, 2015 1:28 PM To: outages@outages.org Subject: Re: [outages] Windstream Outage Confirmed here as well. We had sites in IL and FL drop off. I'm seeing very high packet loss as well, one site I tested had 84% packet loss Robert Colon | Tier 3 Technical Support | EthoStream LLC | P: (877) 282-2519 x2114 | F: (414) 258-8307 | 20800 Swenson Dr. Suite 175 Milwaukee, WI 53226 | www.telkonet.com <http://www.telkonet.com/> | www.ethostream.com <http://www.ethostream.com/> | @Telkonet <https://twitter.com/telkonet> On 2/2/2015 12:17 PM, Ryan Prihoda via Outages wrote: Confirmed. Lisle, IL Ryan Prihoda Systems Administrator dynaConnections www.dynaconnections.com On 02/02/2015 12:04 PM, Alex Forster via Outages wrote: Confirmed in northern Delaware. Alex Forster Network Engineer, Linode On 2/2/15 12:52 PM, Mauricio Lizano via Outages wrote: I just saw Windstream circuits go down across the board, from CA to PA. I saw circuits down from 11:34 to 11:44 Central. Most are recovering on their own but a few are not. Anyone else feeling the pain? Thanks, -- Mauricio Lizano _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _____ <http://www.telkonet.com> <http://www.ethostream.com> This e-mail and any files transmitted with it are private and confidential and are solely for the use of the addressee. It may contain material which is legally privileged. If you are not the addressee or the person responsible for delivering to the addressee, be advised that you have received this e-mail in error and that any use of it is strictly prohibited. If you have received this e-mail in error please contact us at (414) 223-0473.

They are sending out this e-mail: Windstream Hosted Solutions is experiencing a nationwide MPLS event affecting all customers. Our engineers are engaged and troubleshooting this event. Because of high call volume, we ask that you do not call the Network Operations Center rather open tickets online. William Collier-Byrd will@collier-byrd.net On Mon, Feb 2, 2015 at 10:26 AM, Sass, Amy via Outages <outages@outages.org> wrote:

We are still down in Illinois. First started for us about 11:30ish.

Amy Sass

CITES SR. Windows Systems Admin, University of Illinois at Urbana-Champaign

Lync Service Manager

2530 DCL

1304 W. Springfield Ave.

Urbana, IL 61801

(217)265-0531

*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Mauricio Lizano via Outages *Sent:* Monday, February 2, 2015 11:53 AM *To:* outages@outages.org *Subject:* [outages] Windstream Outage

I just saw Windstream circuits go down across the board, from CA to PA. I saw circuits down from 11:34 to 11:44 Central. Most are recovering on their own but a few are not.

Anyone else feeling the pain?

Thanks,

--

Mauricio Lizano

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

This is what we received from Windstream this morning regarding the outage: "Worklog Update: Hello, if you are receiving this message, your Windstream trouble ticket was associated to a known network event. At this time, the issue has been resolved and your services should be restored. A future planned core link had an inconsistant ISIS metric that temporarily rerouted many PE BGP sessions causing VPN route reflectors to bounce, inet-bgp session to drop and large amounts of packet loss. The configuration was corrected and the outage was resolved. If there are any further issues please contact Windstream Enterprise Repair at xxx-xxx-xxxx and reference your ticket number. Thank you." Amy Sass CITES SR. Windows Systems Admin, University of Illinois at Urbana-Champaign Lync Service Manager 2530 DCL 1304 W. Springfield Ave. Urbana, IL 61801 (217)265-0531 -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Julian Y Koh via Outages Sent: Tuesday, February 3, 2015 9:40 AM To: outages Subject: Re: [outages] Windstream Outage On Mon Feb 02 2015 12:30:42 CST, William Byrd via Outages <outages@outages.org> wrote:

Windstream Hosted Solutions is experiencing a nationwide MPLS event affecting all customers. Our engineers are engaged and troubleshooting this event. Because of high call volume, we ask that you do not call the Network Operations Center rather open tickets online.

Any other info available yet? We’ve asked our account team, but nothing yet. Curious that MPLS is the problem given that they had the previous MPLS meltdown back in November. -- Julian Y. Koh Acting Associate Director, Telecommunications and Network Services Northwestern University Information Technology (NUIT) 2001 Sheridan Road #G-166 Evanston, IL 60208 847-467-5780 NUIT Web Site: <http://www.it.northwestern.edu/> PGP Public Key:<http://bt.ittns.northwestern.edu/julian/pgppubkey.html> _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages