June 2013 - Outages - Outages Lists

Outage: outages
by Jay Ashworth 27 Jun '13

27 Jun '13

Just in case anyone's wondering, yes, wiki.outages.org appears to be stuck. We're looking into it. Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274

1 0

anyone seeing problems from DTAG to GoDaddy AS26496?
by Eugen Leitl 27 Jun '13

27 Jun '13

Can't reproduce it myself, since not DTAG, but it's been purportedly an issue for last 2-3 days. Sorry if this is the wrong list, and should go to nanog.

1 0

GMail sending issues
by Blair Trosper 26 Jun '13

26 Jun '13

Our entire organization and three of my other accounts are getting this pop-up when sending a message: Oops... a server error occurred and your email was not sent. (#793) But, as usual, everything is "totally fine" according to the GApps status page: http://www.google.com/appsstatus#hl=en&v=status&ts=1372272841152

7 7

PG&E power outage central California
by Jay Hennigan 24 Jun '13

24 Jun '13

Widespread power outage ongoing for over two hours from San Luis Obispo through Santa Ynez Valley. No ETR as of yet. -- -- Jay Hennigan - CCIE #7880 - Network Engineering - jay(a)impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV

1 0

GoDaddy Nameservers
by Mike Phipps 23 Jun '13

23 Jun '13

I just noticed that several of our hosted websites are not working at the moment, and they're failing due to DNS record resolution errors. I checked into it and they're all domain names that are using GoDaddy for name service. Anyone else seeing this?

4 5

Frontier.com homepage unaccessible
by Frank Bulk 21 Jun '13

21 Jun '13

It's seems Frontier Communications home page has been down over IPv4 and IPv6 since 12:34 pm Central. Port 80 is open over v4 and v6, just the page doesn't load. =========================================================== nagios:/# wget -4 www.frontier.com --2013-05-09 13:24:53-- http://www.frontier.com/ Resolving www.frontier.com... 66.133.172.205 Connecting to www.frontier.com|66.133.172.205|:80... connected. HTTP request sent, awaiting response... ^C =========================================================== nagios:/# wget -6 www.frontier.com --2013-05-09 13:25:07-- http://www.frontier.com/ Resolving www.frontier.com... 2001:1960:21:1::92 Connecting to www.frontier.com|2001:1960:21:1::92|:80... connected. HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers. Retrying. --2013-05-09 13:25:08-- (try: 2) http://www.frontier.com/ Connecting to www.frontier.com|2001:1960:21:1::92|:80... connected. HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers. Retrying. --2013-05-09 13:25:10-- (try: 3) http://www.frontier.com/ Connecting to www.frontier.com|2001:1960:21:1::92|:80... connected. HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers. Retrying. ^C =========================================================== nagios:/# tcptraceroute www.frontier.com Selected device eth0.3, address 96.31.0.5, port 50820 for outgoing packets Tracing the path to www.frontier.com (66.133.172.205) on TCP port 80 (www), 30 hops max 1 router-core-inside.mtcnet.net (96.31.0.254) 0.248 ms 0.194 ms 0.184 ms 2 sxct.movl.mtcnet.net (167.142.156.194) 0.195 ms 0.131 ms 0.116 ms 3 premier.movl-mlx.fbnt.netins.net (173.215.60.1) 1.932 ms 1.914 ms 1.922 ms 4 ins-kb1-te-13-2-220.kmrr.netins.net (167.142.64.253) 7.582 ms 7.513 ms 7.483 ms 5 ins-kc2-et-8-3.kmrr.netins.net (167.142.67.37) 9.148 ms 9.180 ms 9.126 ms 6 ins-dc1-po20.desm.netins.net (167.142.67.29) 9.327 ms 9.437 ms 9.139 ms 7 te-3-3.car2.KansasCity1.Level3.net (4.53.34.113) 15.643 ms 15.800 ms 15.647 ms 8 ae-11-11.car1.KansasCity1.Level3.net (4.69.135.233) 44.503 ms 44.517 ms 44.595 ms 9 ae-5-5.ebr2.Dallas1.Level3.net (4.69.135.230) 44.475 ms 44.443 ms 44.420 ms 10 ae-62-62.csw1.Dallas1.Level3.net (4.69.151.129) 56.816 ms 50.240 ms 44.854 ms 11 ae-63-63.ebr3.Dallas1.Level3.net (4.69.151.134) 44.511 ms 44.640 ms 44.660 ms 12 ae-7-7.ebr3.Atlanta2.Level3.net (4.69.134.22) 44.635 ms 44.603 ms 44.463 ms 13 ae-73-73.ebr2.Atlanta2.Level3.net (4.69.148.254) 44.559 ms 44.565 ms 44.595 ms 14 ae-2-52.edge1.Atlanta4.Level3.net (4.69.150.79) 44.627 ms 44.418 ms 44.442 ms 15 4.53.232.34 44.695 ms 76.316 ms 44.644 ms 16 ae2---0.cor02.atln.ga.frontiernet.net (74.40.2.81) 44.794 ms 44.675 ms 44.624 ms 17 ae1---0.cor02.asbn.va.frontiernet.net (74.40.2.198) 58.282 ms 144.739 ms 58.297 ms 18 ae3---0.cor01.roch.ny.frontiernet.net (74.40.5.17) 71.606 ms 71.562 ms 71.494 ms 19 ge--0-0-0---0.car01.roch.ny.frontiernet.net (74.40.5.74) 70.920 ms 70.830 ms 70.909 ms 20 te5-4.br5.roch.ny.frontiernet.net (74.45.102.6) 404.081 ms 71.598 ms 71.562 ms 21 * frontier.com (66.133.172.205) [open] 68.823 ms 71.625 ms nagios:/# =========================================================== nagios:/# tcptraceroute6 www.frontier.com traceroute to frontier.com (2001:1960:21:1::92) from 2607:fe28:0:1000::5, port 80, from port 59200, 30 hops max, 60 bytes packets 1 router-core.mtcnet.net (2607:fe28:0:1000::1) 0.250 ms 0.194 ms 0.201 ms 2 sxct.movl.mtcnet.net (2607:fe28:11:1002::194) 0.181 ms 0.125 ms 0.130 ms 3 v6-premier.movl-mlx.fbnt.netins.net (2001:5f8:7f0a:1::1) 1.967 ms 1.887 ms 1.909 ms 4 v6-ins-kb1-te-13-2-220.kmrr.netins.net (2001:5f8:2:2::1) 7.595 ms 7.477 ms 7.512 ms 5 v6-ins-kc2-te-8-3.kmrr.netins.net (2001:5f8::11:1) 7.568 ms 7.494 ms 7.584 ms 6 v6-ins-dc1-po20.desm.netins.net (2001:5f8::8:1) 9.210 ms 9.202 ms 9.148 ms 7 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 15.738 ms 15.912 ms 15.657 ms 8 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 15.632 ms 15.751 ms 15.632 ms 9 vl-4049.edge2.Dallas1.Level3.net (2001:1900:4:1::379) 24.750 ms 24.706 ms 24.654 ms 10 vl-4080.edge3.Dallas1.Level3.net (2001:1900:4:1::82) 24.780 ms 24.713 ms 24.751 ms 11 vl-4061.car1.Atlanta2.Level3.net (2001:1900:4:1::96) 44.731 ms 44.774 ms 44.759 ms 12 vl-51.edge1.Atlanta4.Level3.net (2001:1900:1c:1::10) 44.866 ms 44.896 ms 44.806 ms 13 2001:1900:2100::1a (2001:1900:2100::1a) 71.300 ms 44.481 ms 44.571 ms 14 ae2---0.cor02.atln.ga.frontiernet.net (2001:1960:c:1f::) 52.316 ms 44.854 ms 44.776 ms 15 ae1---0.cor02.asbn.va.frontiernet.net (2001:1960:c:3c::1) 58.473 ms 58.343 ms 58.461 ms 16 ae3---0.cor01.roch.ny.frontiernet.net (2001:1960:c:3d::) 124.867 ms 71.000 ms 70.933 ms 17 ge--0-1-0---0.car02.roch.ny.frontiernet.net (2001:1960:c:28::1) 69.445 ms 69.408 ms 69.299 ms 18 frontier.com (2001:1960:21:1::92) 69.200 ms [open] 69.149 ms 69.079 ms nagios:/# =========================================================== Frank

9 11

Facebook web interface partial outage 2120 EDT 6/18
by Jay Ashworth 18 Jun '13

18 Jun '13

Some subset of FB appears to be down for some people from some places for about the last 10 minutes; according to both my experience and reports from a couple dozen people on twitter. I'm on Sprint 4G in Tampa. Got some error messages, and now no response to homepage or internal links... in Chrome. Clearing cookies and local storage didn't help. Fresh login on Firefox on the same machine *did* work; I suspect a load balancer broke, and Chrome has the IP internally cached and won't drop it. Win7 resolves www.facebook.com to 69.171.242.27, star.c10r.facebook.com Can't post an mtr; I'm on Windows. Most of the things that appear to be outage reports are in other languages, mostly South American, I think, Portuguese, Spanish... ... and see? You write a detailed enough report, and it will come back before you're done. Anyway, followups to -discuss, unless your email is @facebook.com, and you *know* what was going on. :-) Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274

1 0

Cogent 216 W Jackson, Chicago Opteman Outage?
by Erich Kaiser 17 Jun '13

17 Jun '13

Anyone else seeing an opteman outage @ 216 W Jackson in Chicago?

2 2

Re: [outages] REMINDER: On/Off Topic for Outages
by Jay Ashworth 15 Jun '13

15 Jun '13

>From a list subscriber, anonymized cause it was sent to me directly: > On 6/14/13 12:56 PM, Jay Ashworth wrote: > > The perceived S/N on outages seems to be dropping slightly again > > lately, so please remember: > > Is the list for small things like someone's DSL and such ? Seems rather > noisy to me and only 'larger' outages should be posted but i'm not sure > what is and isn't appropriate. In short: no. :-) If you have a direct support channel, such as for an uplink connection you pay someone for, that's whom you ought to be talking to. If your DSL link can only talk to some sites, and your collocated cablemodem can get to everything, that's more on topic. If there's a single destination site you can't get to, *and you've done some homework* that suggests it's either the target or some transit link in between, that, too, would be good fodder for the list. Our -- well, my :-) -- twin goals for the list are to keep informed those people who get queries about "why can't I get to X", either from paying customers of their own, or from inside "customers" in their business, about things which are known to be problematic on any given day, and, a layer down, the hope that the people involved in the outages will get some early warning, since in many cases, the people seeing the outage have no formal reporting channel to tell them about it. Hopefully that helps a bit. [ Followups to -discuss ] Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274

1 0

REMINDER: On/Off Topic for Outages
by Jay Ashworth 14 Jun '13

14 Jun '13

The perceived S/N on outages seems to be dropping slightly again lately, so please remember: If it's a report that something appears out, a confirmation that something is out, or an inquiry directly related to either (concerning scope or necessary further data to confirm or deny), then it's on-topiv. As soon as you're tempted to reply to a posting and branch off into "and it probably wouldn't happen if they had..." or "and this happens a lot..." or any other such editorial comments -- and *certainly* if you are replying to one -- you should move your reply to -discussion, CCing the person to whom you're replying, in case they're not on that list. And, in other news: an additional reminder that if you get a message CCd to both you and a list you're already on anyway, it's nearly always because the sender's email client does not have a Reply-To-List button, as nearly all consumer MUAs do not. (I've had a ticket open on Zimbra for 4 years for this with no motion.) We try to keep the list as useful as possible to as large a number of subscribers as possible, but we understand that some audiences have disjoint requirements; we apologize if the rules we *do* try to enforce leave still too much traffic on the list for you. </moderator> And Happy Flag Day! Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274

1 0