We were having trouble with reaching some customer sites through Level3 Chicago (one customer is connected to TWTC for example). We shut down our BGP and all is well. Someone is contacting Level3 now, but curious if anyone else is seeing similar. -- Chris Adams <cma@cmadams.net>
Confirming. Seeing issues here in Mpls, MN from our customers around the USA logging into a COLO fed from Cogent and Level3. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Chris Adams via Outages Sent: Tuesday, December 15, 2015 1:07 PM To: outages@outages.org Subject: [outages] Level3 Chicago? We were having trouble with reaching some customer sites through Level3 Chicago (one customer is connected to TWTC for example). We shut down our BGP and all is well. Someone is contacting Level3 now, but curious if anyone else is seeing similar. -- Chris Adams <cma@cmadams.net> _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages Avionte accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information provided, unless that information is subsequently confirmed in writing. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
My datacenter is in Madison Wisconsin and we are experiencing issues from stores across the US having issues connecting to the VPN. Looks like there is definitely something going on there. Working on getting a root cause. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Chris Adams via Outages Sent: Tuesday, December 15, 2015 1:07 PM To: outages@outages.org Subject: [outages] Level3 Chicago? We were having trouble with reaching some customer sites through Level3 Chicago (one customer is connected to TWTC for example). We shut down our BGP and all is well. Someone is contacting Level3 now, but curious if anyone else is seeing similar. -- Chris Adams <cma@cmadams.net> _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Seeing the same here, my VPN endpoint is in Milwaukee, I'm trying to connect in Nashville, TN and not able to connect via AT&T Uverse. Traceroute seems to die on the ATT network, but I assume this is the last hop before hitting level 3? traceroute to 66.195.129.28 (66.195.129.28), 30 hops max, 60 byte packets 1 192.168.10.1 (192.168.10.1) 0.756 ms 1.188 ms 1.099 ms 2 99-98-180-2.lightspeed.nsvltn.sbcglobal.net (99.98.180.2) 20.989 ms 21.522 ms 22.053 ms 3 99.131.205.92 (99.131.205.92) 22.597 ms 23.212 ms 23.743 ms 4 12.83.112.21 (12.83.112.21) 27.580 ms 24.973 ms 28.006 ms 5 12.122.141.233 (12.122.141.233) 33.381 ms 37.744 ms 38.289 ms 6 * * * 7 * * * 8 * * * 9 * * * Signature To ensure prompt service, please use "Reply All" when responding Robert Colon | TCSA Tier 3 | EthoStream LLC | P: (877) 282-2519 x2114 | F: (414) 258-8307 | 20800 Swenson Drive, Suite 175 Waukesha, WI 53186 | www.telkonet.com <http://www.telkonet.com/> | www.ethostream.com <http://www.ethostream.com/> | @Telkonet <https://twitter.com/telkonet> On 12/15/2015 1:11 PM, Savage, Tony via Outages wrote:
My datacenter is in Madison Wisconsin and we are experiencing issues from stores across the US having issues connecting to the VPN. Looks like there is definitely something going on there. Working on getting a root cause.
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Chris Adams via Outages Sent: Tuesday, December 15, 2015 1:07 PM To: outages@outages.org Subject: [outages] Level3 Chicago?
We were having trouble with reaching some customer sites through Level3 Chicago (one customer is connected to TWTC for example). We shut down our BGP and all is well.
Someone is contacting Level3 now, but curious if anyone else is seeing similar.
-- Chris Adams <cma@cmadams.net> _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- ------------------------------ <http://www.telkonet.com> <http://www.ethostream.com> This e-mail and any files transmitted with it are private and confidential and are solely for the use of the addressee. It may contain material which is legally privileged. If you are not the addressee or the person responsible for delivering to the addressee, be advised that you have received this e-mail in error and that any use of it is strictly prohibited. If you have received this e-mail in error please contact us at (414) 223-0473.
Coming from Atlanta to my facility in Chicago 1 <1 ms <1 ms <1 ms 10.233.208.1 2 <1 ms <1 ms <1 ms 172.16.0.12 3 1 ms <1 ms <1 ms 68.208.9.1 4 3 ms 3 ms 3 ms 12.250.9.101 5 7 ms 7 ms 7 ms cr1.attga.ip.att.net [12.122.141.186] 6 4 ms 7 ms 7 ms 12.122.141.233 7 * * * Request timed out. Same thing... advertising on backup ISP and working fine. No word on my open ticket with L3 From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Bob Colon via Outages Sent: Tuesday, December 15, 2015 1:16 PM To: outages@outages.org Subject: Re: [outages] Level3 Chicago? Seeing the same here, my VPN endpoint is in Milwaukee, I'm trying to connect in Nashville, TN and not able to connect via AT&T Uverse. Traceroute seems to die on the ATT network, but I assume this is the last hop before hitting level 3? traceroute to 66.195.129.28 (66.195.129.28), 30 hops max, 60 byte packets 1 192.168.10.1 (192.168.10.1) 0.756 ms 1.188 ms 1.099 ms 2 99-98-180-2.lightspeed.nsvltn.sbcglobal.net (99.98.180.2) 20.989 ms 21.522 ms 22.053 ms 3 99.131.205.92 (99.131.205.92) 22.597 ms 23.212 ms 23.743 ms 4 12.83.112.21 (12.83.112.21) 27.580 ms 24.973 ms 28.006 ms 5 12.122.141.233 (12.122.141.233) 33.381 ms 37.744 ms 38.289 ms 6 * * * 7 * * * 8 * * * 9 * * * To ensure prompt service, please use "Reply All" when responding Robert Colon | TCSA Tier 3 | EthoStream LLC | P: (877) 282-2519 x2114 | F: (414) 258-8307 | 20800 Swenson Drive, Suite 175 Waukesha, WI 53186 | www.telkonet.com<http://www.telkonet.com> | www.ethostream.com<http://www.ethostream.com/> | @Telkonet<https://twitter.com/telkonet> On 12/15/2015 1:11 PM, Savage, Tony via Outages wrote: My datacenter is in Madison Wisconsin and we are experiencing issues from stores across the US having issues connecting to the VPN. Looks like there is definitely something going on there. Working on getting a root cause. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Chris Adams via Outages Sent: Tuesday, December 15, 2015 1:07 PM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] Level3 Chicago? We were having trouble with reaching some customer sites through Level3 Chicago (one customer is connected to TWTC for example). We shut down our BGP and all is well. Someone is contacting Level3 now, but curious if anyone else is seeing similar. -- Chris Adams <cma@cmadams.net><mailto:cma@cmadams.net> _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages ________________________________ [Description: Image removed by sender.][Description: Image removed by sender.]<http://www.telkonet.com>[Description: Image removed by sender.]<http://www.ethostream.com/> This e-mail and any files transmitted with it are private and confidential and are solely for the use of the addressee. It may contain material which is legally privileged. If you are not the addressee or the person responsible for delivering to the addressee, be advised that you have received this e-mail in error and that any use of it is strictly prohibited. If you have received this e-mail in error please contact us at (414) 223-0473.
Just got off the phone with Level3, this basically explains the issue. *** CASCADED EXTERNAL NOTES 15-Dec-2015 20:20:05 GMT From CASE: 10125524 - Event The IP NOC reported that an unknown issue was impacting IP services for multiple users across the East Coast. Traffic between Chicago, IL and Atlanta, GA was experiencing latency and routing issues. The IP NOC made adjustments to normalize traffic. Tier III Technical Support and the IP NOC will continue investigations to isolate the root cause. A final notification will be sent when stability has been confirmed.
On Dec 15, 2015, at 2:23 PM, Ryan Rittgarn via Outages <outages@outages.org> wrote:
Coming from Atlanta to my facility in Chicago
1 <1 ms <1 ms <1 ms 10.233.208.1 2 <1 ms <1 ms <1 ms 172.16.0.12 3 1 ms <1 ms <1 ms 68.208.9.1 4 3 ms 3 ms 3 ms 12.250.9.101 5 7 ms 7 ms 7 ms cr1.attga.ip.att.net <http://cr1.attga.ip.att.net/> [12.122.141.186] 6 4 ms 7 ms 7 ms 12.122.141.233 7 * * * Request timed out.
Same thing… advertising on backup ISP and working fine. No word on my open ticket with L3
From: Outages [mailto:outages-bounces@outages.org <mailto:outages-bounces@outages.org>] On Behalf Of Bob Colon via Outages Sent: Tuesday, December 15, 2015 1:16 PM To: outages@outages.org <mailto:outages@outages.org> Subject: Re: [outages] Level3 Chicago?
Seeing the same here, my VPN endpoint is in Milwaukee, I'm trying to connect in Nashville, TN and not able to connect via AT&T Uverse. Traceroute seems to die on the ATT network, but I assume this is the last hop before hitting level 3?
traceroute to 66.195.129.28 (66.195.129.28), 30 hops max, 60 byte packets 1 192.168.10.1 (192.168.10.1) 0.756 ms 1.188 ms 1.099 ms 2 99-98-180-2.lightspeed.nsvltn.sbcglobal.net <http://99-98-180-2.lightspeed.nsvltn.sbcglobal.net/> (99.98.180.2) 20.989 ms 21.522 ms 22.053 ms 3 99.131.205.92 (99.131.205.92) 22.597 ms 23.212 ms 23.743 ms 4 12.83.112.21 (12.83.112.21) 27.580 ms 24.973 ms 28.006 ms 5 12.122.141.233 (12.122.141.233) 33.381 ms 37.744 ms 38.289 ms 6 * * * 7 * * * 8 * * * 9 * * *
To ensure prompt service, please use "Reply All" when responding
Robert Colon | TCSA Tier 3 | EthoStream LLC | P: (877) 282-2519 x2114 | F: (414) 258-8307 | 20800 Swenson Drive, Suite 175 Waukesha, WI 53186 | www.telkonet.com <http://www.telkonet.com/> | www.ethostream.com <http://www.ethostream.com/> | @Telkonet <https://twitter.com/telkonet>
On 12/15/2015 1:11 PM, Savage, Tony via Outages wrote: My datacenter is in Madison Wisconsin and we are experiencing issues from stores across the US having issues connecting to the VPN. Looks like there is definitely something going on there. Working on getting a root cause.
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org <mailto:outages-bounces@outages.org>] On Behalf Of Chris Adams via Outages Sent: Tuesday, December 15, 2015 1:07 PM To: outages@outages.org <mailto:outages@outages.org> Subject: [outages] Level3 Chicago?
We were having trouble with reaching some customer sites through Level3 Chicago (one customer is connected to TWTC for example). We shut down our BGP and all is well.
Someone is contacting Level3 now, but curious if anyone else is seeing similar.
-- Chris Adams <cma@cmadams.net> <mailto:cma@cmadams.net> _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages> _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
<~WRD000.jpg><~WRD000.jpg> <http://www.telkonet.com/><~WRD000.jpg> <http://www.ethostream.com/> This e-mail and any files transmitted with it are private and confidential and are solely for the use of the addressee. It may contain material which is legally privileged. If you are not the addressee or the person responsible for delivering to the addressee, be advised that you have received this e-mail in error and that any use of it is strictly prohibited. If you have received this e-mail in error please contact us at (414) 223-0473.
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
FYI *** CASCADED EXTERNAL NOTES 15-Dec-2015 23:10:18 GMT From CASE: 10125524 - Event Event Conclusion Summary Outage Start: December 15, 2015 18:18 GMT Outage Stop: December 15, 2015 19:20 GMT Root Cause: A router addition and subsequent card failure Fix Action: The IP NOC made adjustments to normalize traffic. Reason for Outage (RFO) Summary: The IP NOC investigated the reported trouble and determined that the incident resulted from a router addition and subsequent card failure. Traffic between Chicago, IL and Atlanta, GA was impacted for multiple users across the East Coast. The IP NOC made adjustments to normalize traffic and no further impact is anticipated. Corrective Actions: The defective card will be replaced during an upcoming maintenance window, but is not expected to be service affecting. Mike On Dec 15, 2015 12:26, "Joshua McDonald via Outages" <outages@outages.org> wrote:
Just got off the phone with Level3, this basically explains the issue.
*** CASCADED EXTERNAL NOTES 15-Dec-2015 20:20:05 GMT From CASE: 10125524 - Event The IP NOC reported that an unknown issue was impacting IP services for multiple users across the East Coast. Traffic between Chicago, IL and Atlanta, GA was experiencing latency and routing issues. The IP NOC made adjustments to normalize traffic. Tier III Technical Support and the IP NOC will continue investigations to isolate the root cause. A final notification will be sent when stability has been confirmed.
On Dec 15, 2015, at 2:23 PM, Ryan Rittgarn via Outages < outages@outages.org> wrote:
Coming from Atlanta to my facility in Chicago
1 <1 ms <1 ms <1 ms 10.233.208.1 2 <1 ms <1 ms <1 ms 172.16.0.12 3 1 ms <1 ms <1 ms 68.208.9.1 4 3 ms 3 ms 3 ms 12.250.9.101 5 7 ms 7 ms 7 ms cr1.attga.ip.att.net [12.122.141.186] 6 4 ms 7 ms 7 ms 12.122.141.233 7 * * * Request timed out.
Same thing… advertising on backup ISP and working fine. No word on my open ticket with L3
*From:* Outages [mailto:outages-bounces@outages.org <outages-bounces@outages.org>] *On Behalf Of *Bob Colon via Outages *Sent:* Tuesday, December 15, 2015 1:16 PM *To:* outages@outages.org *Subject:* Re: [outages] Level3 Chicago?
Seeing the same here, my VPN endpoint is in Milwaukee, I'm trying to connect in Nashville, TN and not able to connect via AT&T Uverse. Traceroute seems to die on the ATT network, but I assume this is the last hop before hitting level 3?
traceroute to 66.195.129.28 (66.195.129.28), 30 hops max, 60 byte packets 1 192.168.10.1 (192.168.10.1) 0.756 ms 1.188 ms 1.099 ms 2 99-98-180-2.lightspeed.nsvltn.sbcglobal.net (99.98.180.2) 20.989 ms 21.522 ms 22.053 ms 3 99.131.205.92 (99.131.205.92) 22.597 ms 23.212 ms 23.743 ms 4 12.83.112.21 (12.83.112.21) 27.580 ms 24.973 ms 28.006 ms 5 12.122.141.233 (12.122.141.233) 33.381 ms 37.744 ms 38.289 ms 6 * * * 7 * * * 8 * * * 9 * * *
To ensure prompt service, please use "Reply All" when responding
Robert Colon | TCSA Tier 3 | EthoStream LLC | P: (877) 282-2519 x2114 | F: (414) 258-8307 | 20800 Swenson Drive, Suite 175 Waukesha, WI 53186 | www.telkonet.com | www.ethostream.com | @Telkonet <https://twitter.com/telkonet>
On 12/15/2015 1:11 PM, Savage, Tony via Outages wrote:
My datacenter is in Madison Wisconsin and we are experiencing issues from stores across the US having issues connecting to the VPN. Looks like there is definitely something going on there. Working on getting a root cause.
-----Original Message-----
From: Outages [mailto:outages-bounces@outages.org <outages-bounces@outages.org>] On Behalf Of Chris Adams via Outages
Sent: Tuesday, December 15, 2015 1:07 PM
To: outages@outages.org
Subject: [outages] Level3 Chicago?
We were having trouble with reaching some customer sites through Level3 Chicago (one customer is connected to TWTC for example). We shut down our BGP and all is well.
Someone is contacting Level3 now, but curious if anyone else is seeing similar.
--
Chris Adams <cma@cmadams.net> <cma@cmadams.net>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
------------------------------
<~WRD000.jpg><~WRD000.jpg> <http://www.telkonet.com/><~WRD000.jpg> <http://www.ethostream.com/>
This e-mail and any files transmitted with it are private and confidential and are solely for the use of the addressee. It may contain material which is legally privileged. If you are not the addressee or the person responsible for delivering to the addressee, be advised that you have received this e-mail in error and that any use of it is strictly prohibited. If you have received this e-mail in error please contact us at (414) 223-0473. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We're seeing the exact same behavior into Chicago (car4) . Started about 12:15pm. We're dropping our session now. On 12/15/15 1:06 PM, Chris Adams via Outages wrote:
We were having trouble with reaching some customer sites through Level3 Chicago (one customer is connected to TWTC for example). We shut down our BGP and all is well.
Someone is contacting Level3 now, but curious if anyone else is seeing similar.
participants (8)
-
Andy Brezinsky -
Bob Colon -
Chris Adams -
Joshua McDonald -
Mike Davis -
Ryan Rittgarn -
Savage, Tony -
Tariq Khan