May 2013 - Outages - Outages Lists

Frontier.com homepage unaccessible
by Frank Bulk 21 Jun '13

21 Jun '13

It's seems Frontier Communications home page has been down over IPv4 and IPv6 since 12:34 pm Central. Port 80 is open over v4 and v6, just the page doesn't load. =========================================================== nagios:/# wget -4 www.frontier.com --2013-05-09 13:24:53-- http://www.frontier.com/ Resolving www.frontier.com... 66.133.172.205 Connecting to www.frontier.com|66.133.172.205|:80... connected. HTTP request sent, awaiting response... ^C =========================================================== nagios:/# wget -6 www.frontier.com --2013-05-09 13:25:07-- http://www.frontier.com/ Resolving www.frontier.com... 2001:1960:21:1::92 Connecting to www.frontier.com|2001:1960:21:1::92|:80... connected. HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers. Retrying. --2013-05-09 13:25:08-- (try: 2) http://www.frontier.com/ Connecting to www.frontier.com|2001:1960:21:1::92|:80... connected. HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers. Retrying. --2013-05-09 13:25:10-- (try: 3) http://www.frontier.com/ Connecting to www.frontier.com|2001:1960:21:1::92|:80... connected. HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers. Retrying. ^C =========================================================== nagios:/# tcptraceroute www.frontier.com Selected device eth0.3, address 96.31.0.5, port 50820 for outgoing packets Tracing the path to www.frontier.com (66.133.172.205) on TCP port 80 (www), 30 hops max 1 router-core-inside.mtcnet.net (96.31.0.254) 0.248 ms 0.194 ms 0.184 ms 2 sxct.movl.mtcnet.net (167.142.156.194) 0.195 ms 0.131 ms 0.116 ms 3 premier.movl-mlx.fbnt.netins.net (173.215.60.1) 1.932 ms 1.914 ms 1.922 ms 4 ins-kb1-te-13-2-220.kmrr.netins.net (167.142.64.253) 7.582 ms 7.513 ms 7.483 ms 5 ins-kc2-et-8-3.kmrr.netins.net (167.142.67.37) 9.148 ms 9.180 ms 9.126 ms 6 ins-dc1-po20.desm.netins.net (167.142.67.29) 9.327 ms 9.437 ms 9.139 ms 7 te-3-3.car2.KansasCity1.Level3.net (4.53.34.113) 15.643 ms 15.800 ms 15.647 ms 8 ae-11-11.car1.KansasCity1.Level3.net (4.69.135.233) 44.503 ms 44.517 ms 44.595 ms 9 ae-5-5.ebr2.Dallas1.Level3.net (4.69.135.230) 44.475 ms 44.443 ms 44.420 ms 10 ae-62-62.csw1.Dallas1.Level3.net (4.69.151.129) 56.816 ms 50.240 ms 44.854 ms 11 ae-63-63.ebr3.Dallas1.Level3.net (4.69.151.134) 44.511 ms 44.640 ms 44.660 ms 12 ae-7-7.ebr3.Atlanta2.Level3.net (4.69.134.22) 44.635 ms 44.603 ms 44.463 ms 13 ae-73-73.ebr2.Atlanta2.Level3.net (4.69.148.254) 44.559 ms 44.565 ms 44.595 ms 14 ae-2-52.edge1.Atlanta4.Level3.net (4.69.150.79) 44.627 ms 44.418 ms 44.442 ms 15 4.53.232.34 44.695 ms 76.316 ms 44.644 ms 16 ae2---0.cor02.atln.ga.frontiernet.net (74.40.2.81) 44.794 ms 44.675 ms 44.624 ms 17 ae1---0.cor02.asbn.va.frontiernet.net (74.40.2.198) 58.282 ms 144.739 ms 58.297 ms 18 ae3---0.cor01.roch.ny.frontiernet.net (74.40.5.17) 71.606 ms 71.562 ms 71.494 ms 19 ge--0-0-0---0.car01.roch.ny.frontiernet.net (74.40.5.74) 70.920 ms 70.830 ms 70.909 ms 20 te5-4.br5.roch.ny.frontiernet.net (74.45.102.6) 404.081 ms 71.598 ms 71.562 ms 21 * frontier.com (66.133.172.205) [open] 68.823 ms 71.625 ms nagios:/# =========================================================== nagios:/# tcptraceroute6 www.frontier.com traceroute to frontier.com (2001:1960:21:1::92) from 2607:fe28:0:1000::5, port 80, from port 59200, 30 hops max, 60 bytes packets 1 router-core.mtcnet.net (2607:fe28:0:1000::1) 0.250 ms 0.194 ms 0.201 ms 2 sxct.movl.mtcnet.net (2607:fe28:11:1002::194) 0.181 ms 0.125 ms 0.130 ms 3 v6-premier.movl-mlx.fbnt.netins.net (2001:5f8:7f0a:1::1) 1.967 ms 1.887 ms 1.909 ms 4 v6-ins-kb1-te-13-2-220.kmrr.netins.net (2001:5f8:2:2::1) 7.595 ms 7.477 ms 7.512 ms 5 v6-ins-kc2-te-8-3.kmrr.netins.net (2001:5f8::11:1) 7.568 ms 7.494 ms 7.584 ms 6 v6-ins-dc1-po20.desm.netins.net (2001:5f8::8:1) 9.210 ms 9.202 ms 9.148 ms 7 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 15.738 ms 15.912 ms 15.657 ms 8 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 15.632 ms 15.751 ms 15.632 ms 9 vl-4049.edge2.Dallas1.Level3.net (2001:1900:4:1::379) 24.750 ms 24.706 ms 24.654 ms 10 vl-4080.edge3.Dallas1.Level3.net (2001:1900:4:1::82) 24.780 ms 24.713 ms 24.751 ms 11 vl-4061.car1.Atlanta2.Level3.net (2001:1900:4:1::96) 44.731 ms 44.774 ms 44.759 ms 12 vl-51.edge1.Atlanta4.Level3.net (2001:1900:1c:1::10) 44.866 ms 44.896 ms 44.806 ms 13 2001:1900:2100::1a (2001:1900:2100::1a) 71.300 ms 44.481 ms 44.571 ms 14 ae2---0.cor02.atln.ga.frontiernet.net (2001:1960:c:1f::) 52.316 ms 44.854 ms 44.776 ms 15 ae1---0.cor02.asbn.va.frontiernet.net (2001:1960:c:3c::1) 58.473 ms 58.343 ms 58.461 ms 16 ae3---0.cor01.roch.ny.frontiernet.net (2001:1960:c:3d::) 124.867 ms 71.000 ms 70.933 ms 17 ge--0-1-0---0.car02.roch.ny.frontiernet.net (2001:1960:c:28::1) 69.445 ms 69.408 ms 69.299 ms 18 frontier.com (2001:1960:21:1::92) 69.200 ms [open] 69.149 ms 69.079 ms nagios:/# =========================================================== Frank

9 11

IPv6 tunnels in FRA1 on HE.net down?
by Constantine A. Murenin 13 Jun '13

13 Jun '13

As of a couple of minutes ago, my IPv6 tunnel seems to have no connectivity, weirdly other than being able to access ordns.he.net (2001:470:20::2) just fine, but not ns{2,3,4,5}.he.net, or any other IPv6 host.

6 14

Hurricane Electric down - Bay Area
by Blaine Reeve 30 May '13

30 May '13

My session with Hurricane Electric is down in the Bay Area and LA - One Wilshire, CA. Can't get to their website right now either. Blaine M Reeve | Vice President of Operations O1 Communications 5190 Golden Foothill Pkwy El Dorado Hills, CA 95762 office: 916.235.2005 | cell: 916.396.5992 email: breeve(a)o1.com<mailto:breeve@o1.com> | web: www.o1.com<http://www.o1.com/>

4 4

IPv6 access to Hurricane Electric's home page down?
by Frank Bulk (iname.com) 30 May '13

30 May '13

I noticed that IPv4 and IPv6 access to www.he.net went down at 5:30 pm Central. Traceroutes show different things, but basically it's getting stuck somewhere in HE's core. nagios:# tcptraceroute www.he.net Selected device eth0.3, address 96.31.0.5, port 36642 for outgoing packets Tracing the path to www.he.net (216.218.186.2) on TCP port 80 (www), 30 hops max 1 router-core-inside.mtcnet.net (96.31.0.254) 0.240 ms 0.201 ms 0.179 ms 2 sxct.sxcy.mtcnet.net (167.142.156.197) 0.208 ms 0.136 ms 0.127 ms 3 premier.sxcy-mlx.fbnt.netins.net (173.215.60.5) 1.939 ms 1.920 ms 1.904 ms 4 ins-db1-te-13-2-219.desm.netins.net (167.142.60.157) 7.886 ms 7.835 ms 7.957 ms 5 ins-dc2-et-8-3.desm.netins.net (167.142.67.25) 7.795 ms 7.759 ms 7.803 ms 6 ins-db3-te-0-7-0-0.desm.netins.net (167.142.67.182) 8.190 ms 7.977 ms 7.973 ms 7 ord1.ins.indatelservices.com (216.176.4.29) 23.014 ms 23.230 ms 22.932 ms 8 10gigabitethernet4-1.core1.chi1.he.NET (206.223.119.37) 22.819 ms 22.835 ms 22.776 ms 9 * * * 10 * * * 11 * * * 12 * * * nagios:# tcptraceroute6 www.he.net traceroute to he.net (2001:470:0:76::2) from 2607:fe28:0:1000::5, port 80, from port 60791, 30 hops max, 60 bytes packets 1 router-core.mtcnet.net (2607:fe28:0:1000::1) 0.240 ms 0.200 ms 0.181 ms 2 sxct.movl.mtcnet.net (2607:fe28:11:1002::194) 0.226 ms 0.856 ms 0.139 ms 3 v6-premier.movl-mlx.fbnt.netins.net (2001:5f8:7f0a:1::1) 1.962 ms 1.923 ms 1.888 ms 4 v6-ins-kb1-te-13-2-220.kmrr.netins.net (2001:5f8:2:2::1) 6.315 ms 6.354 ms 6.399 ms 5 v6-ins-kc1-te-9-2.kmrr.netins.net (2001:5f8::10:1) 6.360 ms 6.412 ms 6.306 ms 6 sl-crs2-chi-te1-2-0-1.v6.sprintlink.net (2600:4::6) 16.130 ms 15.853 ms 15.736 ms 7 sl-crs1-chi-po0-0-0-3.v6.sprintlink.net (2600:0:2:1239:144:232:26:8) 17.236 ms 16.344 ms 16.133 ms 8 sl-st30-chi-bu-20.v6.sprintlink.net (2600:0:2:1239:144:232:1:102) 17.649 ms 17.857 ms 17.544 ms 9 10gigabitethernet4-1.core1.chi1.he.net (2001:504:0:4::6939:1) 25.946 ms 16.488 ms 16.484 ms 10 10gigabitethernet5-7.core1.mci3.he.net (2001:470:0:270::2) 28.448 ms 38.399 ms 28.468 ms 11 10gigabitethernet5-1.core1.den1.he.net (2001:470:0:240::1) 40.525 ms 48.500 ms 48.141 ms 12 10gigabitethernet13-5.core1.sjc2.he.net (2001:470:0:1b4::1) 75.170 ms 74.256 ms 72.699 ms 13 * * * 14 * * * 15 * * * 16 * * * [fbulk@puck /]$ traceroute www.he.net traceroute to www.he.net (216.218.186.2), 30 hops max, 60 byte packets 1 ge-0-7-0-22.r05.chcgil09.us.bb.gin.ntt.net (204.42.253.5) 0.738 ms 0.650 ms 0.610 ms 2 ae-0.teliasonera.chcgil09.us.bb.gin.ntt.net (129.250.8.178) 0.219 ms 0.194 ms 0.172 ms 3 chi-bb1-link.telia.net (80.91.246.168) 0.290 ms chi-bb1-link.telia.net (80.91.246.116) 31.168 ms chi-bb1-link.telia.net (80.91.246.118) 31.363 ms 4 sjo-bb1-link.telia.net (213.248.80.25) 51.322 ms sjo-bb1-link.telia.net (213.155.132.180) 51.297 ms 51.275 ms 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * [fbulk@puck /]$ traceroute6 www.he.net traceroute to www.he.net (2001:470:0:76::2), 30 hops max, 80 byte packets 1 ge-0-7-0-22.r05.chcgil09.us.bb.gin.ntt.net (2001:418:3f4::1) 0.725 ms 0.622 ms 0.666 ms 2 ae-6.r20.chcgil09.us.bb.gin.ntt.net (2001:418:0:2000::29) 80.228 ms 0.395 ms 0.446 ms 3 10gigabitethernet4-1.core1.chi1.he.net (2001:504:0:4::6939:1) 0.267 ms 0.402 ms 0.299 ms 4 10gigabitethernet5-7.core1.mci3.he.net (2001:470:0:270::2) 12.659 ms 10gigabitethernet4-1.core1.den1.he.net (2001:470:0:293::2) 24.342 ms 24.299 ms 5 10gigabitethernet13-5.core1.sjc2.he.net (2001:470:0:1b4::1) 59.785 ms 59.189 ms 59.450 ms 6 10gigabitethernet13-5.core1.sjc2.he.net (2001:470:0:1b4::1) 59.539 ms 58.630 ms * 7 * * * 8 * * * 9 * * * 10 * * * Incidentally, I noticed that ipv6.cablelabs.com went down at the same time, too. Frank

3 2

Fwd: FYI StartCom / StartSSL Down [AR]
by Jason Hellenthal 30 May '13

30 May '13

Could list admin unsub this... It seems permanently down. Begin forwarded message: > From: CenturyLink Autoresponse <qwestrsp(a)CenturyLink.com> > Date: May 24, 2013, 12:45:10 EDT > To: Jason Hellenthal <jhellenthal(a)dataix.net> > Subject: [outages] FYI StartCom / StartSSL Down [AR] > > > Thank you for contacting CenturyLink; we appreciate your business. This > email address is no longer active or cannot accept messages. We > apologize for the inconvenience. > > CenturyLink's online portal, Control Center, lets you efficiently manage > your network, make service changes, pay bills, and much more; go to https://controlcenter.centurylink.com/ > > Key Contacts > Residential: 800-244-1111 > Small Business: 800-603-6000 > Large Business: 800-777-9594 > Government and Education: 800-777-9594 > Federal Government: 800-879-1023 > Mid-Markets: 800-763-5843 > > Please visit our website at http://www.centurylink.com >

5 4

Earthlink/STS issues in Miami (again!)
by Eric Whitehill 30 May '13

30 May '13

Earthlink/STS apparently had a VLX2020 fail last night in south Florida. No ETR, but have been told it's a major outage, and when I pressed for an ETR I was told they'd get back to me. I showed it went down last night around 20:15 Eastern Time, and it kind of came back up at 8:13 Eastern, but I've been told to expect it to bounce until it's resolved. Not sure if anyone else is affected by this, but figuring it's a VLX2020, I'm sure there are. If anyone else who has been affected by this outage and the one on 5-6-13 can give me a jingle, let me know. Trying to figure out if we can coordinate DR plans or something here. Thanks folks, -Eric

2 1

Outage with TWCBC in the Chicago Area
by Michael Peterman 30 May '13

30 May '13

Boy, what a day. So I just saw some strange routes die. I am connected to a time warner cable connection and I guess from my side I can't get around anything that is in the Chicago area. My route times out at 66.109.6.206 at ae-6-0.cr0.Chi30.tbone.rr.com And I can't even get to google.com Anyone else seeing this issue? Michael Peterman

1 0

IL Comcast Outage
by Michael Peterman 30 May '13

30 May '13

I just was a few of the Comcast modems go down. Anyone else seeing an outages in the CHI area? Michael Peterman President Small Business Technology Solutions "You worry about running your business; we will worry about the technology" Main Office 7513 7th Avenue Suite 3 Kenosha, WI 53143 Phone: (262) 671-4898 ext. 202 Cell: (262) 496-3306 www.seeus4it.com<http://www.seeus4it.com/> Michael(a)seeus4it.com<mailto:Michael@seeus4it.com> [Description: Facebook]<http://www.facebook.com/seeus4it>[Description: Twitter]<http://www.twitter.com/seeus4it>[Description: Linkedin]<http://www.linkedin.com/companies/263975>[Description: RSS]<http://blog.seeus4it.com/?feed=rss2>

2 2

Fiber cut in SC coast
by Joshua Johnson 29 May '13

29 May '13

Any word on the fiber/cable cut impacting (at least) Comcast in the Charledton/Goose Creek area? Thanks.

1 0

Chronic packet loss/latency in LAX (4 providers potentially involved)
by Jeremy Chadwick 28 May '13

28 May '13

Hi folks, For the past 3-4 weeks (likely longer), there has been a peering point in Los Angeles that has been repeatedly overtaxed every night for roughly 5-6 hours. Specifically high latency and packet loss. It tends to start at roughly the same time every day (1800 PDT), and tends to dissipate around the same time as well (2300 PDT or so) -- i.e. prime-time. This is not ICMP prioritisation -- this is real packet loss (I can tell via SSH, fetching mail (IMAP), etc.). However in recent days (past week or so), the issue has persisted even until hours as late as 0100 PDT, and today (for example) has existed since as early as 1500 PDT. So the situation as I said is getting worse. Because of asymmetrical routing, it's impossible for me to tell where peering-wise the actual issue is: a) between Comcast and GTT/nLayer b) between Comcast and Tata/AS6453 c) between NTT/Verio and Tata/AS6453 Of those 4 providers I've only a relationship with one; me getting all four to talk simultaneously, while the issue is happening, is virtually impossible. However it happens every night like clockwork, for a long period of time, so resolution shouldn't be all that difficult. I do periodic mtrs (think traceroute + ping combined) between my home connection and my VPS. Probe durations are 40 seconds, at intervals of 60 seconds. I store roughly 2-3 months of data, and I can make all this data available. Here's examples taken from tonight, both directions. Source and destination IPs are provided. Source IP: 67.180.84.87 (Comcast; Mountain View, CA) Dest IP: 206.125.172.42 (ARP Networks; Sylmar, CA) === Mon May 27 19:36:00 PDT 2013 (1369708560) HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.3 0.3 0.2 0.4 2.|-- 67.180.84.1 0.0% 40 40 11.4 24.9 10.1 61.7 3.|-- 68.85.191.249 0.0% 40 40 11.3 11.7 8.9 27.7 4.|-- 69.139.199.106 0.0% 40 40 14.6 13.3 10.4 16.1 5.|-- 68.86.91.45 0.0% 40 40 15.9 19.6 13.1 34.3 6.|-- 68.86.88.58 0.0% 40 40 25.0 22.8 20.0 27.2 7.|-- 68.86.88.190 0.0% 40 40 21.5 21.9 19.6 38.2 8.|-- 173.167.57.138 10.0% 40 36 68.5 79.4 54.1 92.3 9.|-- 69.31.127.141 0.0% 40 40 81.4 95.1 64.5 128.8 10.|-- 69.31.127.130 35.0% 40 26 69.5 69.6 43.0 96.1 11.|-- 69.174.121.73 0.0% 40 40 67.0 72.3 39.6 90.4 12.|-- 67.199.135.102 10.0% 40 36 39.1 49.2 36.7 249.8 13.|-- 206.125.172.42 5.0% 40 38 37.1 36.0 34.2 49.9 === END DNS resolution for relevant hops: $ host 68.86.88.190 190.88.86.68.in-addr.arpa domain name pointer pos-0-4-0-0-pe01.600wseventh.ca.ibone.comcast.net. $ host 173.167.57.138 Host 138.57.167.173.in-addr.arpa. not found: 3(NXDOMAIN) $ host 69.31.127.141 141.127.31.69.in-addr.arpa domain name pointer ae0-110g.cr1.lax1.us.nlayer.net. Source IP: 206.125.172.42 (ARP Networks; Sylmar, CA) Dest IP: 67.180.84.87 (Comcast; Mountain View, CA) === Mon May 27 19:37:00 PDT 2013 (1369708620) HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst 1.|-- 206.125.172.41 0.0% 40 40 1.5 12.8 0.9 181.8 2.|-- 208.79.88.135 0.0% 40 40 0.6 1.0 0.5 11.6 3.|-- 129.250.198.185 0.0% 40 40 0.8 0.9 0.7 1.4 4.|-- 129.250.2.221 0.0% 40 40 1.0 1.0 0.9 1.5 5.|-- 64.86.252.65 32.5% 40 27 0.5 0.8 0.5 8.6 | `|-- 216.6.84.65 6.|-- 173.167.59.185 55.0% 40 18 19.6 19.3 15.7 75.1 7.|-- 68.86.82.61 40.0% 40 24 17.9 17.5 15.9 19.5 8.|-- 68.86.88.57 5.0% 40 38 25.2 25.7 24.1 27.5 9.|-- 68.86.90.94 7.5% 40 37 25.8 26.6 24.7 28.7 10.|-- 69.139.198.81 5.0% 40 38 26.6 26.6 26.3 27.0 11.|-- 68.85.191.254 7.5% 40 37 47.9 46.0 30.5 50.5 12.|-- 67.180.84.87 7.5% 40 37 36.0 36.1 34.0 50.3 === END DNS resolution for relevant hops (hop #5 indicates round-robin'ing between two different interfaces presumably on the same Tata/AS6453 router): $ host 129.250.2.221 221.2.250.129.in-addr.arpa domain name pointer ae-3.r05.lsanca03.us.bb.gin.ntt.net. $ host 64.86.252.65 65.252.86.64.in-addr.arpa domain name pointer ix-11-2-1-0.tcore2.LVW-LosAngeles.as6453.net. $ host 216.6.84.65 65.84.6.216.in-addr.arpa domain name pointer ix-9-1-2-0.tcore2.LVW-LosAngeles.as6453.net. $ host 173.167.59.185 185.59.167.173.in-addr.arpa domain name pointer xe-1-2-0-0-pe01.onewilshire.ca.ibone.comcast.net. If relevant providers are on this list and can respond, that would be awesome. If they can't respond but can instead start the ball rolling on rectifying this, I think myself and many other 'netizens would really appreciate it. -- | Jeremy Chadwick jdc(a)koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |

5 5