February 2014 - Outages - Outages Lists

Re: [outages] Level3 Routing...
by Clayton Zekelman 01 Mar '14

01 Mar '14

We had a report from a customer earlier this morning, but they advised the problem went away. We connect to Level3 at Southfield MI. At 09:35 AM 25/02/2014, *Network wrote: >Content-Language: en-US >Content-Type: multipart/alternative; > >boundary="_000_391AAB48E84BB24288BED210F379E1DC4DEA4C02NOCMAILDAG2Inte_" > >Latency issue with Level 3 here in Detroit area > >Anton > > > >From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Luke Rockwell >Sent: Tuesday, February 18, 2014 12:24 PM >To: Petter Bruland >Cc: Outages(a)outages.org >Subject: Re: [outages] Level3 Routing... > >nothing on the level 3 network events page I have access to > >Luke > >On Tue, Feb 18, 2014 at 9:17 AM, Petter Bruland ><<mailto:Petter.Bruland@allegiantair.com>Petter.Bruland(a)allegiantair.com> >wrote: >Not seeing any issues with Level3 in Las Vegas. > >-Petter > >From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Nick Olsen >Sent: Tuesday, February 18, 2014 6:04 AM >To: Morgan Miskell; <mailto:Outages@outages.org>Outages(a)outages.org > >Subject: Re: [outages] Level3 Routing... > >Not seeing anything homed off of Level 3 Orlando. > >Traceroutes to a few common places look fine. > >~474815 Routes from 3356. >Nick Olsen >Network Operations >(855) FLSPEED x106 > > >---------- >From: "Morgan Miskell" ><<mailto:morgan.miskell@caro.net>morgan.miskell(a)caro.net> >Sent: Tuesday, February 18, 2014 9:00 AM >To: "<mailto:Outages@outages.org>Outages(a)outages.org" ><<mailto:outages@outages.org>outages(a)outages.org> >Subject: [outages] Level3 Routing... > >Anyone seeing Level3 routing issues? We just observed a massive drop in >routes in the BGP feeds and seeing some strange ICMP issues along their >backbone in terms of traceroutes along their backbone. They might be >adding some filters based on the recent DDOS attacks, I can ping >interfaces but traceroutes go to hell... >-- >Morgan A. Miskell >CaroNet Data Centers >704-643-8330 x206 >---------------------------------------------------------------------------- >The information contained in this e-mail is confidential and is intended >only for the named recipient(s). If you are not the intended recipient >you must not copy, distribute, or take any action or reliance on it. If >you have received this e-mail in error, please notify the sender. Any >unauthorized disclosure of the information contained in this e-mail is >strictly prohibited. >---------------------------------------------------------------------------- > >_______________________________________________ >Outages mailing list ><mailto:Outages@outages.org>Outages(a)outages.org >https://puck.nether.net/mailman/listinfo/outages > >_______________________________________________ >Outages mailing list ><mailto:Outages@outages.org>Outages(a)outages.org >https://puck.nether.net/mailman/listinfo/outages > >_______________________________________________ >Outages mailing list >Outages(a)outages.org >https://puck.nether.net/mailman/listinfo/outages --- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409

10 14

DNS server issues
by Petter Bruland 28 Feb '14

28 Feb '14

We're working with eNomCentral, which is our domain registrar and hosted DNS provider, on a DNS resolution issue. They reported that the issue seems to be with multiple IPSs and are looking into this issue. Anyone else seen issues with DNS resolution out there this morning (morning PST time...) Thanks -Petter

4 6

Connectivity issue to BVI?
by Frank Bulk 28 Feb '14

28 Feb '14

Our email server is reporting it can't send email to several domains that reside on 208.91.197.241. That IP address is associated with 208.91.196.0/23, which is Confluence Networks, which appears to be based in the BVI. A traceroute shows it stopping in Houston with Cogent. Is there an issue with connectivity to the BVI? Frank nagios:/tmp# tcptraceroute 208.91.197.241 25 Selected device eth0.3, address 96.31.0.5, port 50890 for outgoing packets Tracing the path to 208.91.197.241 on TCP port 25 (smtp), 30 hops max 1 router-core-inside.mtcnet.net (96.31.0.254) 0.238 ms 0.180 ms 0.180 ms 2 sxct.sxcy.mtcnet.net (167.142.156.197) 0.198 ms 0.119 ms 0.115 ms 3 premier.sxcy-mlx.fbnt.netins.net (173.215.60.5) 1.633 ms 1.587 ms 1.598 ms 4 ins-db1-te-13-2-219.desm.netins.net (167.142.60.157) 6.775 ms 7.003 ms 6.778 ms 5 ins-dc2-et-8-3.desm.netins.net (167.142.67.25) 6.837 ms 6.745 ms 6.855 ms 6 ins-wc1-po20.wdmn.netins.net (167.142.67.34) 6.831 ms 6.833 ms 6.800 ms 7 * * * 8 te0-7-0-25.ccr21.ord01.atlas.cogentco.com (154.54.0.77) 14.537 ms 14.550 ms 14.480 ms 9 be2156.ccr21.mci01.atlas.cogentco.com (154.54.6.85) 27.077 ms 26.814 ms 27.000 ms 10 be2012.ccr21.dfw01.atlas.cogentco.com (154.54.2.114) 36.752 ms 37.051 ms 37.054 ms 11 be2145.mpd21.iah01.atlas.cogentco.com (154.54.25.217) 41.932 ms 42.041 ms 41.963 ms 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * Destination not reached nagios:/etc/cron.daily# Origin-AS: 40034 Prefix: 208.91.196.0/23 AS-Path: 293 6461 3900 40034 AS-Org-Name: Confluence Networks Inc Org-Name: Confluence Networks Inc Net-Name: CONFLUENCE-NETWORK-INC Cache-Date: 1393498184 Latitude: 18.416670 Longitude: -64.616670 City: ROAD TOWN Region: BRITISH VIRGIN ISLANDS Country: VIRGIN ISLANDS, BRITISH Country-Code: VG

3 4

Weird issue with PacWest
by Darren Schreiber 27 Feb '14

27 Feb '14

We got a report from a client that inbound calls to only one of their blocks are not working. What's strange is we have only been able to recreate it from AT&T. Calls from Sprint are being delivered. Anyone else experiencing this and can explain why this would happen? Client claims Verizon, T-Mobile and AT&T can't reach their block. Yet I can.

4 4

mailchimp.com outage
by jd 26 Feb '14

26 Feb '14

can anyone reach mailchimp.com? can't reach it from anywhere. has been like this for about 40 minutes 3 te-3-4.car2.Tustin1.Level3.net (4.53.178.213) 8.116 ms 8.118 ms 8.240 ms 4 vl-3102-ve-133.ebr1.Tustin1.Level3.net (4.69.160.13) 7.431 ms 7.431 ms 7.660 ms 5 ae-6-6.ebr1.LosAngeles1.Level3.net (4.69.153.221) 7.655 ms 7.651 ms 7.649 ms 6 ae-91-91.csw4.LosAngeles1.Level3.net (4.69.137.14) 7.634 ms ae-61-61.csw1.LosAngeles1.Level3.net (4.69.137.2) 7.322 ms ae-81-81.csw3.LosAngeles1.Level3.net (4.69.137.10) 7.259 ms 7 ae-2-70.edge1.LosAngeles9.Level3.net (4.69.144.74) 7.251 ms * 7.229 ms 8 xo-level3.Losangeles9.Level3.net (4.68.110.206) 7.221 ms 7.218 ms 7.204 ms 9 207.88.14.217.ptr.us.xo.net (207.88.14.217) 69.524 ms 73.410 ms 69.405 ms 10 vb15.rar3.dallas-tx.us.xo.net (207.88.12.45) 66.400 ms 66.398 ms 66.392 ms 11 te-4-0-0.rar3.atlanta-ga.us.xo.net (207.88.12.1) 66.117 ms 66.114 ms 59.732 ms 12 ae0d0.cir2.atlanta6-ga.us.xo.net (207.88.13.9) 58.478 ms 58.309 ms 58.445 ms 13 * * * 14 * * * 3 ggr2.la2ca.ip.att.net (12.122.128.101) 6.070 ms 6.064 ms 6.056 ms 4 205.158.79.241.ptr.us.xo.net (205.158.79.241) 11.785 ms 12.024 ms 12.017 ms 5 207.88.14.217.ptr.us.xo.net (207.88.14.217) 56.226 ms 65.201 ms 56.200 ms 6 vb15.rar3.dallas-tx.us.xo.net (207.88.12.45) 55.187 ms 58.156 ms 58.132 ms 7 te-4-0-0.rar3.atlanta-ga.us.xo.net (207.88.12.1) 53.121 ms 58.421 ms 58.411 ms 8 ae0d0.cir2.atlanta6-ga.us.xo.net (207.88.13.9) 51.894 ms 51.887 ms 51.877 ms 9 * 67.106.215.22.ptr.us.xo.net (67.106.215.22) 102.552 ms 102.550 ms 10 * * * 11 * * * 3 te-3-4.car2.Tustin1.Level3.net (4.53.178.213) 8.219 ms 8.212 ms 8.205 ms 4 vl-3102-ve-133.ebr1.Tustin1.Level3.net (4.69.160.13) 7.585 ms 7.578 ms 7.569 ms 5 ae-6-6.ebr1.LosAngeles1.Level3.net (4.69.153.221) 8.894 ms 8.889 ms 7.729 ms 6 ae-81-81.csw3.LosAngeles1.Level3.net (4.69.137.10) 8.117 ms ae-91-91.csw4.LosAngeles1.Level3.net (4.69.137.14) 9.007 ms ae-61-61.csw1.LosAngeles1.Level3.net (4.69.137.2) 7.242 ms 7 * ae-4-90.edge1.LosAngeles9.Level3.net (4.69.144.202) 7.112 ms * 8 xo-level3.Losangeles9.Level3.net (4.68.110.206) 7.334 ms 7.326 ms 7.067 ms 9 207.88.14.217.ptr.us.xo.net (207.88.14.217) 63.310 ms 68.788 ms 69.126 ms 10 vb15.rar3.dallas-tx.us.xo.net (207.88.12.45) 70.964 ms 70.502 ms 71.089 ms 11 te-4-0-0.rar3.atlanta-ga.us.xo.net (207.88.12.1) 60.081 ms 60.061 ms 59.566 ms 12 ae0d0.cir2.atlanta6-ga.us.xo.net (207.88.13.9) 58.308 ms 58.278 ms 58.272 ms 13 * * * 14 * * * 3 so320.cr1.lsancajw.telepacific.net (65.60.127.70) 9.166 ms 9.177 ms 9.170 ms 4 gi22.lax-core01.mpowercom.com (208.57.218.169) 9.143 ms 9.137 ms 9.128 ms 5 * * *

3 2

Re: [outages] Level3 Routing...
by Nick Olsen 25 Feb '14

25 Feb '14

Not seeing anything homed off of Level 3 Orlando. Traceroutes to a few common places look fine. ~474815 Routes from 3356. Nick Olsen Network Operations (855) FLSPEED x106 ---------------------------------------- From: "Morgan Miskell" <morgan.miskell(a)caro.net> Sent: Tuesday, February 18, 2014 9:00 AM To: "Outages(a)outages.org" <outages(a)outages.org> Subject: [outages] Level3 Routing... Anyone seeing Level3 routing issues? We just observed a massive drop in routes in the BGP feeds and seeing some strange ICMP issues along their backbone in terms of traceroutes along their backbone. They might be adding some filters based on the recent DDOS attacks, I can ping interfaces but traceroutes go to hell... -- Morgan A. Miskell CaroNet Data Centers 704-643-8330 x206 ---------------------------------------------------------------------------- The information contained in this e-mail is confidential and is intended only for the named recipient(s). If you are not the intended recipient you must not copy, distribute, or take any action or reliance on it. If you have received this e-mail in error, please notify the sender. Any unauthorized disclosure of the information contained in this e-mail is strictly prohibited. ---------------------------------------------------------------------------- _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

6 5

Peer1 Routing Failure
by Graham Nelson-Zutter 24 Feb '14

24 Feb '14

Peer1’s core NOC (1.877.504.0091) confirms network wide “routing issues”. All of our IPs in Vancouver were unreachable for approximately 29 minutes. Other cities were also offline. Is anyone using Peer1 still offline? Graham Nelson-Zutter graham(a)telephonic.ca CTO, Co-founder +1 514 228 1262 Ext 102 +1 877 638 3848 Toll-Free +1 604 608 4764 Fax Telephonic Communications Inc. #502-2818 Main Street Vancouver, B.C. Canada V5T0C1

2 1

NSI seems to be having issues...
by eric-list＠truenet.com 24 Feb '14

24 Feb '14

I'm getting calls about worldnic.com servers not responding. Nothing on twitter: https://twitter.com/netsolcares and I can't seem to find any status pages. Sincerely, Eric Tykwinski TrueNet, Inc. P: 610-429-8300 F: 610-429-3222

2 1

Southeast US (Fiber Outage between Bremen and Carrollton Georgia)
by Kraig Beahn 22 Feb '14

22 Feb '14

Evening - Any SP aware of a maintenance event taking place between Bremen Georgia and Carrollton Georgia, affecting multiple 10G circuits on our network as well as Earthlink and a few other carriers to include Google's primary in/out of Atlanta? Licensed to us as "ring protected", however, as a result of this outage, come to find out this appears to be configured as a collapsed ring. Thank you kindly for any information anyone may be able to provide. Kraig Beahn L2Networks Corp.

1 0

Possible outage affecting Amazon.com?
by chris＠htswireless.com 21 Feb '14

21 Feb '14

Is anyone else seeing traffic routing all over the place trying to get to Amazon.com? Chris

7 7