TW/Level 3 issues?

Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well. [image: https://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3X...] *Brad Riemann* Senior Systems and Network Engineer Tel: 847.352.4850 x125 briemann@nmi.com | www.nmi.com

We just lost all connectivity in L3 Tampa peered with 3356 and the now useless 4323. BGP is still up to our routers, packets not making it in our out though. David From: Outages <outages-bounces@outages.org> on behalf of Brad Riemann via Outages <outages@outages.org> Reply-To: Brad Riemann <briemann@nmi.com> Date: Wednesday, November 2, 2016 at 10:57 AM To: "outages@outages.org" <outages@outages.org> Subject: [outages] TW/Level 3 issues? Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well. [ttps://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3XqIWdQQR4sDljaGNwzEaPyLYO0Thml] Brad Riemann Senior Systems and Network Engineer Tel: 847.352.4850 x125 briemann@nmi.com<mailto:briemann@nmi.com> | www.nmi.com<http://www.nmi.com/>

Same here all our L3 circuits are uber spotty down in dallas right now. Alex From: Outages [mailto:outages-bounces@outages.org] On Behalf Of David Hubbard via Outages Sent: Wednesday, November 02, 2016 10:00 AM To: Brad Riemann; outages@outages.org Subject: Re: [outages] TW/Level 3 issues? We just lost all connectivity in L3 Tampa peered with 3356 and the now useless 4323. BGP is still up to our routers, packets not making it in our out though. David From: Outages <outages-bounces@outages.org> on behalf of Brad Riemann via Outages <outages@outages.org> Reply-To: Brad Riemann <briemann@nmi.com> Date: Wednesday, November 2, 2016 at 10:57 AM To: "outages@outages.org" <outages@outages.org> Subject: [outages] TW/Level 3 issues? Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well. [Description: ttps://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3XqIWdQQR4sDljaGNwzEaPyLYO0Thml] Brad Riemann Senior Systems and Network Engineer Tel: 847.352.4850 x125 briemann@nmi.com<mailto:briemann@nmi.com> | www.nmi.com<http://www.nmi.com/> [Supreme Lending] ________________________________ Notice to Recipient: The information contained in this email is confidential and intended solely for the addressees. Any unauthorized access, use, reproduction, or dissemination is prohibited. If you have received this e-mail in error, please notify me immediately and delete this copy from your system. Everett Financial, Inc., dba Supreme Lending, its subsidiaries and/or its affiliates shall not assume legal liability or responsibility for any incorrect, misleading, or altered information contained herein. Any rate, payment, or cost quoted above is subject to change and may increase. Please see your Loan Estimate for your actual loan terms. NMLS ID# 2129 [Equal Housing Lender]

Same here, Level 3 connectivity between our Allen Texas office and our Dallas office dropped, as well as connectivity between Allen and Memphis TN. -------- Eric Henson Server Solutions Architect PFS p: 972.881.2900 x 3104 m: 972.948.3424 www.pfsweb.com<http://www.pfsweb.com/> From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Alex Holmes via Outages Sent: Wednesday, November 2, 2016 10:10 AM To: David Hubbard; Brad Riemann Cc: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] TW/Level 3 issues? Same here all our L3 circuits are uber spotty down in dallas right now. Alex From: Outages [mailto:outages-bounces@outages.org] On Behalf Of David Hubbard via Outages Sent: Wednesday, November 02, 2016 10:00 AM To: Brad Riemann; outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] TW/Level 3 issues? We just lost all connectivity in L3 Tampa peered with 3356 and the now useless 4323. BGP is still up to our routers, packets not making it in our out though. David From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> on behalf of Brad Riemann via Outages <outages@outages.org<mailto:outages@outages.org>> Reply-To: Brad Riemann <briemann@nmi.com<mailto:briemann@nmi.com>> Date: Wednesday, November 2, 2016 at 10:57 AM To: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] TW/Level 3 issues? Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well. [Description: ttps://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3XqIWdQQR4sDljaGNwzEaPyLYO0Thml] Brad Riemann Senior Systems and Network Engineer Tel: 847.352.4850 x125 briemann@nmi.com<mailto:briemann@nmi.com> | www.nmi.com<http://www.nmi.com/> [Supreme Lending] ________________________________ Notice to Recipient: The information contained in this email is confidential and intended solely for the addressees. Any unauthorized access, use, reproduction, or dissemination is prohibited. If you have received this e-mail in error, please notify me immediately and delete this copy from your system. Everett Financial, Inc., dba Supreme Lending, its subsidiaries and/or its affiliates shall not assume legal liability or responsibility for any incorrect, misleading, or altered information contained herein. Any rate, payment, or cost quoted above is subject to change and may increase. Please see your Loan Estimate for your actual loan terms. NMLS ID# 2129 [Equal Housing Lender]

We had outages for TW/Level3 in Suwanee, GA. Rob++ ----- Original Message -----
From: "Eric Henson via Outages" <outages@outages.org> To: outages@outages.org Sent: Wednesday, November 2, 2016 10:33:14 AM Subject: Re: [outages] TW/Level 3 issues?
Same here, Level 3 connectivity between our Allen Texas office and our Dallas office dropped, as well as connectivity between Allen and Memphis TN.
From: Outages [ mailto:outages-bounces@outages.org ] On Behalf Of Alex Holmes via Outages Sent: Wednesday, November 2, 2016 10:10 AM To: David Hubbard; Brad Riemann Cc: outages@outages.org Subject: Re: [outages] TW/Level 3 issues?
Same here all our L3 circuits are uber spotty down in dallas right now.
From: Outages [ mailto:outages-bounces@outages.org ] On Behalf Of David Hubbard via Outages Sent: Wednesday, November 02, 2016 10:00 AM To: Brad Riemann; outages@outages.org Subject: Re: [outages] TW/Level 3 issues?
We just lost all connectivity in L3 Tampa peered with 3356 and the now useless 4323. BGP is still up to our routers, packets not making it in our out though.
From: Outages < outages-bounces@outages.org > on behalf of Brad Riemann via Outages < outages@outages.org > Reply-To: Brad Riemann < briemann@nmi.com > Date: Wednesday, November 2, 2016 at 10:57 AM To: " outages@outages.org " < outages@outages.org > Subject: [outages] TW/Level 3 issues?
Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well.
-- pubkey: http://www.talgas.com/misc/rob_windsor-pubkey.txt Internet: windsor@warthog.com Life: Rob@Allen.Texas.USA.Earth "They couldn't hit an elephant at this distance." -- Major General John Sedgwick

L3 Just confirmed with us that they are having a Nationwide Outage for unknown reason. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Rob Windsor via Outages Sent: Wednesday, November 02, 2016 10:44 AM To: outages@outages.org Subject: Re: [outages] TW/Level 3 issues? We had outages for TW/Level3 in Suwanee, GA. Rob++ ----- Original Message -----
From: "Eric Henson via Outages" <outages@outages.org> To: outages@outages.org Sent: Wednesday, November 2, 2016 10:33:14 AM Subject: Re: [outages] TW/Level 3 issues?
Same here, Level 3 connectivity between our Allen Texas office and our Dallas office dropped, as well as connectivity between Allen and Memphis TN.
From: Outages [ mailto:outages-bounces@outages.org ] On Behalf Of Alex Holmes via Outages Sent: Wednesday, November 2, 2016 10:10 AM To: David Hubbard; Brad Riemann Cc: outages@outages.org Subject: Re: [outages] TW/Level 3 issues?
Same here all our L3 circuits are uber spotty down in dallas right now.
From: Outages [ mailto:outages-bounces@outages.org ] On Behalf Of David Hubbard via Outages Sent: Wednesday, November 02, 2016 10:00 AM To: Brad Riemann; outages@outages.org Subject: Re: [outages] TW/Level 3 issues?
We just lost all connectivity in L3 Tampa peered with 3356 and the now useless 4323. BGP is still up to our routers, packets not making it in our out though.
From: Outages < outages-bounces@outages.org > on behalf of Brad Riemann via Outages < outages@outages.org > Reply-To: Brad Riemann < briemann@nmi.com > Date: Wednesday, November 2, 2016 at 10:57 AM To: " outages@outages.org " < outages@outages.org > Subject: [outages] TW/Level 3 issues?
Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well.
-- pubkey: http://www.talgas.com/misc/rob_windsor-pubkey.txt Internet: windsor@warthog.com Life: Rob@Allen.Texas.USA.Earth "They couldn't hit an elephant at this distance." -- Major General John Sedgwick _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages [Supreme Lending] ________________________________ Notice to Recipient: The information contained in this email is confidential and intended solely for the addressees. Any unauthorized access, use, reproduction, or dissemination is prohibited. If you have received this e-mail in error, please notify me immediately and delete this copy from your system. Everett Financial, Inc., dba Supreme Lending, its subsidiaries and/or its affiliates shall not assume legal liability or responsibility for any incorrect, misleading, or altered information contained herein. Any rate, payment, or cost quoted above is subject to change and may increase. Please see your Loan Estimate for your actual loan terms. NMLS ID# 2129 [Equal Housing Lender]

From out ticket in on Level3's portal about a half hour ago: 11/02/2016 15:35:17 GMT - An unknown issue is affecting Voice and IP (VPN and Internet IP services) nationwide. Multiple internal teams have been engaged and are investigating at this time. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve. Next update by: 2016-11-02 16:35 GMT
Sent from Yahoo Mail on Android On Wed, Nov 2, 2016 at 11:33 AM, Eric Henson via Outages<outages@outages.org> wrote: Same here, Level 3 connectivity between our Allen Texas office and our Dallas office dropped, as well as connectivity between Allen and Memphis TN. -------- Eric Henson Server Solutions Architect PFS p:972.881.2900 x 3104 m:972.948.3424 www.pfsweb.com From: Outages [mailto:outages-bounces@outages.org]On Behalf Of Alex Holmes via Outages Sent: Wednesday, November 2, 2016 10:10 AM To: David Hubbard; Brad Riemann Cc: outages@outages.org Subject: Re: [outages] TW/Level 3 issues? Same here all our L3 circuits are uber spotty down in dallas right now. Alex From: Outages [mailto:outages-bounces@outages.org]On Behalf Of David Hubbard via Outages Sent: Wednesday, November 02, 2016 10:00 AM To: Brad Riemann; outages@outages.org Subject: Re: [outages] TW/Level 3 issues? We just lost all connectivity in L3 Tampa peered with 3356 and the now useless 4323. BGP is still up to our routers, packets not making it in our out though. David From: Outages <outages-bounces@outages.org> on behalf of Brad Riemann via Outages <outages@outages.org> Reply-To: Brad Riemann <briemann@nmi.com> Date: Wednesday, November 2, 2016 at 10:57 AM To: "outages@outages.org" <outages@outages.org> Subject: [outages] TW/Level 3 issues? Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well. | | Brad Riemann Senior Systems and Network Engineer Tel: 847.352.4850 x125 briemann@nmi.com | www.nmi.com | Notice to Recipient: The information contained in this email is confidential and intended solely for the addressees. Any unauthorized access, use, reproduction, or dissemination is prohibited. If you have received this e-mail in error, please notify me immediately and delete this copy from your system. Everett Financial, Inc., dba Supreme Lending, its subsidiaries and/or its affiliates shall not assume legal liability or responsibility for any incorrect, misleading, or altered information contained herein. Any rate, payment, or cost quoted above is subject to change and may increase. Please see your Loan Estimate for your actual loan terms. NMLS ID# 2129

We saw IP transit traffic die going across twtc as4323 from Raleigh NC, although BGP sessions stayed up Started approx 10:50am and recovered shortly after 11am EDT On Wed, Nov 2, 2016 at 10:59 AM, David Hubbard via Outages < outages@outages.org> wrote:
We just lost all connectivity in L3 Tampa peered with 3356 and the now useless 4323. BGP is still up to our routers, packets not making it in our out though.
David
*From: *Outages <outages-bounces@outages.org> on behalf of Brad Riemann via Outages <outages@outages.org> *Reply-To: *Brad Riemann <briemann@nmi.com> *Date: *Wednesday, November 2, 2016 at 10:57 AM *To: *"outages@outages.org" <outages@outages.org> *Subject: *[outages] TW/Level 3 issues?
Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well.
[image: ttps://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3XqIWdQQR4sDljaGNwzEaPyLYO0Thml]
*Brad Riemann*
Senior Systems and Network Engineer
Tel: 847.352.4850 x125
briemann@nmi.com | www.nmi.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

We are having problems in Dallas with PRIs and some MPLS circuits in the Dallas area. David Warren, Systems Engineer Winstead PC | 500 Winstead Building | 2728 N. Harwood Street | Dallas, Texas 75201 214.745.5778 (Office) | 214.745.5390 (Fax) | dwarren@winstead.com<mailto:dwarren@winstead.com> | www.winstead.com<http://www.winstead.com/> From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Brad Riemann via Outages Sent: Wednesday, November 02, 2016 9:57 AM To: outages@outages.org Subject: [outages] TW/Level 3 issues? Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well. [https://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3X...] Brad Riemann Senior Systems and Network Engineer Tel: 847.352.4850 x125 briemann@nmi.com<mailto:briemann@nmi.com> | www.nmi.com<http://www.nmi.com/> ________________________________ Information contained in this transmission is attorney privileged and confidential. It is intended for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copy of this communication is strictly prohibited. If you have received this communication in error, please immediately notify us by telephone. --

Thanks Guys, talked with Level 3, their internal system doesn’t have any outage information at this time, but the noc engineer said that all of their customers seemed to be calling in. Anyway, not seeing any more issues so if anyone wants to rag on L3/TWC we can take this to outage-discussions ;p [image: https://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3X...] *Brad Riemann* Senior Systems and Network Engineer Tel: 847.352.4850 x125 briemann@nmi.com | www.nmi.com *From:* Outages [mailto:outages-bounces@outages.org <outages-bounces@outages.org>] *On Behalf Of *Brad Riemann via Outages *Sent:* Wednesday, November 02, 2016 9:57 AM *To:* outages@outages.org *Subject:* [outages] TW/Level 3 issues? Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well. [image: https://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3X...] *Brad Riemann* Senior Systems and Network Engineer Tel: 847.352.4850 x125 briemann@nmi.com | www.nmi.com ------------------------------ Information contained in this transmission is attorney privileged and confidential. It is intended for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copy of this communication is strictly prohibited. If you have received this communication in error, please immediately notify us by telephone. --

Confirmed - we saw some MPLS circuits drop between NJ, NY and Houston along with some IP circuits BGP sessions drop in NYC.Looks like it's clearing up now. On Wed, Nov 2, 2016 at 10:57 AM, Brad Riemann via Outages < outages@outages.org> wrote:
Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well.
[image: https://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3X...]
*Brad Riemann*
Senior Systems and Network Engineer
Tel: 847.352.4850 x125
briemann@nmi.com | www.nmi.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

We lost Level 3 MPLS between Verona WI and Amsterdam from about 09:50am CDT to about 10:00am CDT. It bounced several times during that ten-minute time frame. We’re still seeing some evidence of instability or packet loss, but no hard downs. From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Jayson Navitsky via Outages Sent: Wednesday, November 2, 2016 10:01 AM To: Brad Riemann <briemann@nmi.com> Cc: outages@outages.org Subject: Re: [outages] TW/Level 3 issues? Confirmed - we saw some MPLS circuits drop between NJ, NY and Houston along with some IP circuits BGP sessions drop in NYC.Looks like it's clearing up now. On Wed, Nov 2, 2016 at 10:57 AM, Brad Riemann via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well. [https://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3X...] Brad Riemann Senior Systems and Network Engineer Tel: 847.352.4850 x125<tel:847.352.4850%20x125> briemann@nmi.com<mailto:briemann@nmi.com> | www.nmi.com<http://www.nmi.com/> _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages

I had a brief out on a TW/L3 P2P circuit in San Diego, CA at 7:52am Pacific. Coincidence? Seems odd, to me, that there was such a widespread brief incident? Neil From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> on behalf of "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Reply-To: Jayson Navitsky <jnavitsky@gmail.com<mailto:jnavitsky@gmail.com>> Date: Wednesday, November 2, 2016 at 8:01 AM To: Brad Riemann <briemann@nmi.com<mailto:briemann@nmi.com>> Cc: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: Re: [outages] TW/Level 3 issues? Confirmed - we saw some MPLS circuits drop between NJ, NY and Houston along with some IP circuits BGP sessions drop in NYC.Looks like it's clearing up now. On Wed, Nov 2, 2016 at 10:57 AM, Brad Riemann via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well. [https://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3X...] Brad Riemann Senior Systems and Network Engineer Tel: 847.352.4850 x125<tel:847.352.4850%20x125> briemann@nmi.com<mailto:briemann@nmi.com> | www.nmi.com<http://www.nmi.com/> _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages

We also had slight issues in Germany via GBLX (level3) starting around the time when it was first reported here. Probably no coincidence... BR Max M. On 02.11.2016 16:13, Neil Robst via Outages wrote:
I had a brief out on a TW/L3 P2P circuit in San Diego, CA at 7:52am Pacific. Coincidence? Seems odd, to me, that there was such a widespread brief incident?
Neil
From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> on behalf of "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Reply-To: Jayson Navitsky <jnavitsky@gmail.com<mailto:jnavitsky@gmail.com>> Date: Wednesday, November 2, 2016 at 8:01 AM To: Brad Riemann <briemann@nmi.com<mailto:briemann@nmi.com>> Cc: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: Re: [outages] TW/Level 3 issues?
Confirmed - we saw some MPLS circuits drop between NJ, NY and Houston along with some IP circuits BGP sessions drop in NYC.Looks like it's clearing up now.
On Wed, Nov 2, 2016 at 10:57 AM, Brad Riemann via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well.
[https://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3X...]
Brad Riemann Senior Systems and Network Engineer Tel: 847.352.4850 x125<tel:847.352.4850%20x125> briemann@nmi.com<mailto:briemann@nmi.com> | www.nmi.com<http://www.nmi.com/>
_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Seeing the same thing on a Level3 Amsterdam, NL circuit. Dropped 14:51 UTC, recovered 14:53 UTC. Ben Montour Network Engineer Workiva On Wed, Nov 2, 2016 at 10:32 AM, Max Mühlbronner via Outages < outages@outages.org> wrote:
We also had slight issues in Germany via GBLX (level3) starting around the time when it was first reported here. Probably no coincidence...
BR
Max M.
On 02.11.2016 16:13, Neil Robst via Outages wrote:
I had a brief out on a TW/L3 P2P circuit in San Diego, CA at 7:52am Pacific. Coincidence? Seems odd, to me, that there was such a widespread brief incident?
Neil
From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org> <outages-bounces@outages.org>> on behalf of "outages@outages.org<mailto:outages@outages.org> <outages@outages.org>" <outages@outages.org<mailto:outages@outages.org> <outages@outages.org>> Reply-To: Jayson Navitsky <jnavitsky@gmail.com<mailto:jnavitsky@gmail.com> <jnavitsky@gmail.com>> Date: Wednesday, November 2, 2016 at 8:01 AM To: Brad Riemann <briemann@nmi.com<mailto:briemann@nmi.com> <briemann@nmi.com>> Cc: "outages@outages.org<mailto:outages@outages.org> <outages@outages.org>" <outages@outages.org<mailto:outages@outages.org> <outages@outages.org>> Subject: Re: [outages] TW/Level 3 issues?
Confirmed - we saw some MPLS circuits drop between NJ, NY and Houston along with some IP circuits BGP sessions drop in NYC.Looks like it's clearing up now.
On Wed, Nov 2, 2016 at 10:57 AM, Brad Riemann via Outages <outages@outages.org<mailto:outages@outages.org> <outages@outages.org>> wrote: Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well.
[https://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3X...]
Brad Riemann Senior Systems and Network Engineer Tel: 847.352.4850 x125<tel:847.352.4850%20x125>briemann@nmi.com<mailto:briemann@nmi.com> <briemann@nmi.com> | www.nmi.com<http://www.nmi.com/> <http://www.nmi.com/>
_______________________________________________ Outages mailing listOutages@outages.org<mailto:Outages@outages.org> <Outages@outages.org>https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Yup, we had level 3 circuits drop in UK, Japan, Australia and US. Looks like a very widespread issues. Has cleared up for us here. Cheers, Ashley Tayles On 2 Nov 2016 15:40, "Ben Montour via Outages" <outages@outages.org> wrote:
Seeing the same thing on a Level3 Amsterdam, NL circuit. Dropped 14:51 UTC, recovered 14:53 UTC.
Ben Montour Network Engineer Workiva
On Wed, Nov 2, 2016 at 10:32 AM, Max Mühlbronner via Outages < outages@outages.org> wrote:
We also had slight issues in Germany via GBLX (level3) starting around the time when it was first reported here. Probably no coincidence...
BR
Max M.
On 02.11.2016 16:13, Neil Robst via Outages wrote:
I had a brief out on a TW/L3 P2P circuit in San Diego, CA at 7:52am Pacific. Coincidence? Seems odd, to me, that there was such a widespread brief incident?
Neil
From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org> <outages-bounces@outages.org>> on behalf of "outages@outages.org<mailto:outages@outages.org> <outages@outages.org>" <outages@outages.org<mailto:outages@outages.org> <outages@outages.org>> Reply-To: Jayson Navitsky <jnavitsky@gmail.com<mailto:jnavitsky@gmail.com> <jnavitsky@gmail.com>> Date: Wednesday, November 2, 2016 at 8:01 AM To: Brad Riemann <briemann@nmi.com<mailto:briemann@nmi.com> <briemann@nmi.com>> Cc: "outages@outages.org<mailto:outages@outages.org> <outages@outages.org>" <outages@outages.org<mailto:outages@outages.org> <outages@outages.org>> Subject: Re: [outages] TW/Level 3 issues?
Confirmed - we saw some MPLS circuits drop between NJ, NY and Houston along with some IP circuits BGP sessions drop in NYC.Looks like it's clearing up now.
On Wed, Nov 2, 2016 at 10:57 AM, Brad Riemann via Outages <outages@outages.org<mailto:outages@outages.org> <outages@outages.org>> wrote: Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well.
[https://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3X...]
Brad Riemann Senior Systems and Network Engineer Tel: 847.352.4850 x125<tel:847.352.4850%20x125>briemann@nmi.com<mailto:briemann@nmi.com> <briemann@nmi.com> | www.nmi.com<http://www.nmi.com/> <http://www.nmi.com/>
_______________________________________________ Outages mailing listOutages@outages.org<mailto:Outages@outages.org> <Outages@outages.org>https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Observed two bounces between NYC and Charelston as well at 10:52 and 10:53 EST From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Brad Riemann via Outages Sent: Wednesday, November 2, 2016 10:57 AM To: outages@outages.org Subject: [outages] TW/Level 3 issues? Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well. [https://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3X...] Brad Riemann Senior Systems and Network Engineer Tel: 847.352.4850 x125 briemann@nmi.com<mailto:briemann@nmi.com> | www.nmi.com<http://www.nmi.com/>

We saw our TWTC circuit go from partial route feeds to full routes this morning. Looks like more merger changes within the network. From: Outages <outages-bounces@outages.org> on behalf of Andrew Bunde via Outages <outages@outages.org> Reply-To: Andrew Bunde <abunde@tower-research.com> Date: Wednesday, November 2, 2016 at 10:01 AM To: 'Brad Riemann' <briemann@nmi.com>, "'outages@outages.org'" <outages@outages.org> Subject: Re: [outages] TW/Level 3 issues? Observed two bounces between NYC and Charelston as well at 10:52 and 10:53 EST From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Brad Riemann via Outages Sent: Wednesday, November 2, 2016 10:57 AM To: outages@outages.org Subject: [outages] TW/Level 3 issues? Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well. [ttps://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3XqIWdQQR4sDljaGNwzEaPyLYO0Thml] Brad Riemann Senior Systems and Network Engineer Tel: 847.352.4850 x125 briemann@nmi.com<mailto:briemann@nmi.com> | www.nmi.com<http://www.nmi.com/>

We saw momentary impact to non-datacenter links in San Antonio and Austin. However it seems to have stabilized. Nolan Berry Linux Systems Engineer DNS Engineering Rackspace Hosting ________________________________ From: Outages <outages-bounces@outages.org> on behalf of John East via Outages <outages@outages.org> Sent: Wednesday, November 2, 2016 10:12 AM To: Andrew Bunde; 'Brad Riemann'; 'outages@outages.org' Subject: Re: [outages] TW/Level 3 issues? We saw our TWTC circuit go from partial route feeds to full routes this morning. Looks like more merger changes within the network. From: Outages <outages-bounces@outages.org> on behalf of Andrew Bunde via Outages <outages@outages.org> Reply-To: Andrew Bunde <abunde@tower-research.com> Date: Wednesday, November 2, 2016 at 10:01 AM To: 'Brad Riemann' <briemann@nmi.com>, "'outages@outages.org'" <outages@outages.org> Subject: Re: [outages] TW/Level 3 issues? Observed two bounces between NYC and Charelston as well at 10:52 and 10:53 EST From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Brad Riemann via Outages Sent: Wednesday, November 2, 2016 10:57 AM To: outages@outages.org Subject: [outages] TW/Level 3 issues? Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well. [ttps://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3XqIWdQQR4sDljaGNwzEaPyLYO0Thml] Brad Riemann Senior Systems and Network Engineer Tel: 847.352.4850 x125 briemann@nmi.com<mailto:briemann@nmi.com> | www.nmi.com<http://www.nmi.com/>

I’ve been seeing a large percentage of calls failing for about an hour ½ inbound to some servers I have hosted in MI that go through Lvl3/TWTelcom. Not sure if it’s related… I’m still trying to track down the issues. From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Brad Riemann via Outages Sent: Wednesday, November 2, 2016 9:57 AM To: outages@outages.org Subject: [outages] TW/Level 3 issues? Anyone seeing anything, have links with high (80-100%) packetloss in DC and Chicago, our P2P just dropped as well. [https://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3X...] Brad Riemann Senior Systems and Network Engineer Tel: 847.352.4850 x125 briemann@nmi.com<mailto:briemann@nmi.com> | www.nmi.com<http://www.nmi.com/> ***** Confidentiality Notification***** Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Opinions, conclusions and other information in this message that do not relate to the official business of Goode Technology Group shall be understood as neither given nor endorsed by GTG.
participants (18)
-
Adolfo Celis
-
Alex Holmes
-
Andrew Bunde
-
Ashley Tayles
-
Ben Montour
-
Brad Riemann
-
David Hubbard
-
Eric Henson
-
jawitherell@yahoo.com
-
Jayson Navitsky
-
Jeremy Charles
-
John East
-
Mark Price
-
Max Mühlbronner
-
Neil Robst
-
Nolan Berry
-
Rob Windsor
-
Warren, David