June 2012 - Outages - Outages Lists

Fwd: RE: FYI Netflix is down
by Seth Mattinen 30 Jun '12

30 Jun '12

FYI from Nanog. -------- Original Message -------- Subject: RE: FYI Netflix is down Date: Sat, 30 Jun 2012 03:42:52 +0000 From: James Laszko <jamesl(a)mythostech.com> To: Grant Ridder <shortdudey123(a)gmail.com>, Jason Baugher <jason(a)thebaughers.com> CC: nanog(a)nanog.org <nanog(a)nanog.org> To further expand: 8:21 PM PDT We are investigating connectivity issues for a number of instances in the US-EAST-1 Region. 8:31 PM PDT We are investigating elevated errors rates for APIs in the US-EAST-1 (Northern Virginia) region, as well as connectivity issues to instances in a single availability zone. 8:40 PM PDT We can confirm that a large number of instances in a single Availability Zone have lost power due to electrical storms in the area. We are actively working to restore power. -----Original Message----- From: Grant Ridder [mailto:shortdudey123@gmail.com] Sent: Friday, June 29, 2012 8:42 PM To: Jason Baugher Cc: nanog(a)nanog.org Subject: Re: FYI Netflix is down >From Amazon Amazon Elastic Compute Cloud (N. Virginia) (http://status.aws.amazon.com/) 8:21 PM PDT We are investigating connectivity issues for a number of instances in the US-EAST-1 Region. 8:31 PM PDT We are investigating elevated errors rates for APIs in the US-EAST-1 (Northern Virginia) region, as well as connectivity issues to instances in a single availability zone. -Grant On Fri, Jun 29, 2012 at 10:40 PM, Jason Baugher <jason(a)thebaughers.com>wrote: > Seeing some reports of Pinterest and Instagram down as well. Amazon > cloud services being implicated. > > > On 6/29/2012 10:22 PM, Joe Blanchard wrote: > >> Seems that they are unreachable at the moment. Called and theres a >> recorded message stating they are aware of an issue, no details. >> >> -Joe >> >> >> > > >

4 3

Windstream (Paetec)
by Rob Windsor 29 Jun '12

29 Jun '12

Heavy flapping witnessed in Irvine CA, Morris IL, Norcross, GA (pretty much coast-to-coast, different LECs).

1 0

Comcast - East Coast issues ?
by Brandon Applegate 27 Jun '12

27 Jun '12

Hello, Having multiple customers call in about reachability - common thread is their side users are all on Comcast. Not to mention the fact that we can't even hit www.comcast.com from our network currently. We have Level3 (Cincinnati) and Qwest (Chicago). Traceroutes from us outbound seem to prefer Qwest and then die in Comcast. I can post traceroutes if need be - just wanted to see if anyone knew anything before I call my upstreams and wade through phone tree hell. -- Brandon Applegate - CCIE 10273 PGP Key fingerprint: 8779 B023 7637 CEC8 C5C6 4052 664D 7E08 3CBB 1739 "SH1-0151. This is the serial number, of our orbital gun."

10 18

www.justice.gov down on ipv6?
by Cameron Byrne 26 Jun '12

26 Jun '12

It looks like www.justice.gov is down on IPv6. This is likely causing some heart burn for dual stack network that don't have happy eyeballs If you know someone at DOJ, you may want to give them a heads up to fix it (pull the AAAA DNS record until the servers are ready) wget -6 www.justice.gov --2012-06-26 19:27:19-- http://www.justice.gov/ Resolving www.justice.gov (www.justice.gov) 2607:f330:8400:100::50 Connecting to www.justice.gov (www.justice.gov)|2607:f330:8400:100::50|:80... failed: Operation timed out. Retrying. --2012-06-26 19:29:31-- (try: 2) http://www.justice.gov/ Connecting to www.justice.gov (www.justice.gov)|2607:f330:8400:100::50|:80... failed: Operation timed out. Retrying. --2012-06-26 19:31:45-- (try: 3) http://www.justice.gov/ Connecting to www.justice.gov (www.justice.gov)|2607:f330:8400:100::50|:80... failed: Operation timed out. Retrying. --2012-06-26 19:34:01-- (try: 4) http://www.justice.gov/

1 0

w3.org dtd files really slow?
by Mike 18 Jun '12

18 Jun '12

We're running web apps that require the use of W3's dtd files. We're getting some serious timeouts. $ time curl -I http://www.w3.org/TR/html4/HTMLlat1.ent HTTP/1.1 200 OK Date: Mon, 18 Jun 2012 15:27:20 GMT Server: Apache/2 Last-Modified: Fri, 24 Dec 1999 23:26:00 GMT ETag: "2eb4-35c73f0bc5200" Accept-Ranges: bytes Content-Length: 11956 Cache-Control: max-age=7776000 Expires: Sun, 16 Sep 2012 15:27:20 GMT P3P: policyref="http://www.w3.org/2001/05/P3P/p3p.xml" Connection: close Content-Type: application/xml-external-parsed-entity real 0m29.297s user 0m0.007s sys 0m0.005s As well as when we try caching the file locally, we're getting a ton of app errors when it tries to consume it. Anyone else? -Michael

6 5

Elavon Payment gateway timing out when processing
by Marco Prechel 15 Jun '12

15 Jun '12

Is anyone else seeing issues processing payments through Elavon/VirtualMerchant? Their gateway is timing out when processing payments, can't process through merchant portal either. All numbers we have for Elavon are ringing busy/disco. Anyone seeing similar?

4 4

Ookla/Speedtest.net - various symptoms
by Jay Ashworth 14 Jun '12

14 Jun '12

I'm having trouble getting their speedtest.net site to load tonight, from Road Runner Tampa Bay, on three separate browsers on two separate laptops. Same symptom: it gets to "Loading 100%", and then, on a netbook on FF13 or Chrome, it hangs there; on my Linux ThinkPad, I get a red message saying something about the configuration file not loading - I can't give exact language cause now the session is resetting on me. New problem in this location and with this ThinkPad; never seen this before. speedtest.speakeasy.net works fine, but is much slower (1-2Mbps) than the RR Lightning line here (40-50Mbps) would lead me to expect, testing to ATL. Trace: HOST: elphaba.baylink.com Loss% Snt Last Avg Best Wrst StDev 1.|-- 65-35-160-1.res.bhn.net 0.0% 20 27.4 26.0 9.1 54.4 8.5 2.|-- ten0-0-0-4.tamp11-ser1.bh 40.0% 20 10.2 10.1 8.7 12.3 1.2 3.|-- ten0-5-0-1.tamp05-car1.bh 40.0% 20 16.9 19.3 13.7 43.3 7.8 4.|-- ten0-14-0-3.tamp20-car1.b 40.0% 20 18.4 18.9 12.8 25.8 4.0 5.|-- hun0-1-0-0.tamp20-cbr3.bh 40.0% 20 19.1 18.3 11.9 44.7 8.6 6.|-- ae-11-11.cr0.dfw10.tbone. 35.0% 20 36.2 42.8 35.7 75.3 14.2 7.|-- 107.14.19.154 35.0% 20 42.1 41.9 37.6 49.0 3.3 8.|-- ae-3-0.pr0.dfw10.tbone.rr 40.0% 20 39.4 38.1 37.2 39.4 0.8 9.|-- fe1-0.ar02.dal01.pccwbtn. 45.0% 20 38.2 41.4 37.1 57.3 7.3 10.|-- netriver.ge2-16.br01.sea0 42.1% 19 99.9 102.4 98.9 120.0 6.0 11.|-- static-74-209-160-12.lynn 52.6% 19 100.4 101.2 98.9 104.4 1.5 HOST: elphaba.baylink.com Loss% Snt Last Avg Best Wrst StDev 1.|-- 65.35.160.1 0.0% 20 26.2 26.4 12.6 50.1 8.4 2.|-- 71.44.3.70 30.0% 20 10.4 19.9 9.0 76.4 20.3 3.|-- 72.31.117.144 35.0% 20 16.3 15.7 11.9 18.4 1.7 4.|-- 72.31.211.106 40.0% 20 21.1 28.9 13.3 110.0 26.9 5.|-- 72.31.117.158 45.0% 20 17.8 24.8 13.3 102.9 26.0 6.|-- 66.109.10.12 45.0% 20 37.1 45.8 34.9 125.1 26.4 7.|-- 107.14.19.154 45.0% 20 42.7 43.5 38.1 59.4 6.1 8.|-- 66.109.6.209 40.0% 20 37.3 50.3 37.2 112.4 23.9 9.|-- 63.218.23.13 33.3% 18 49.2 57.9 36.4 224.8 53.8 10.|-- 63.216.14.162 50.0% 18 101.6 100.9 98.6 104.1 1.6 11.|-- 74.209.160.12 33.3% 18 98.8 103.5 98.3 143.5 12.7 -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274

5 6

Dubuque IA CO Problems
by Dantzig, Brian 12 Jun '12

12 Jun '12

We lost voice and data circuits coming out of the Dubuque CO. Both copper T1s and our Fiber node are down. I tried calling a few local businesses in Dubuque and I get fast busy's. Nothing yet from my carriers. Anyone have any other information. Brian Dantzig

3 6

Re: [outages] Dubuque IA CO Problems
by Frank Bulk 12 Jun '12

12 Jun '12

Update received from the wireless carrier while I was composing this previous message: We've been notified of an issue w/ our customers calling to CenturyLink customers in Dubuque: Calls to and from CenturyLink in the Dubuque area will likely fail until this issue is resolved. We are not seeing any affected cell sites at this time. <redacted> went off the air at around the same time, so it could be related. The NOC is working w/ CenturyLink to get this restored as quickly as possible. CenturyLink has dispatched a tech to the site. The redacted site (in addition to Scales Mound) is within 20 miles of Dubuque. Frank -----Original Message----- From: Frank Bulk [mailto:frnkblk@iname.com] Sent: Tuesday, June 12, 2012 12:05 PM To: 'Dantzig, Brian'; 'outages(a)outages.org' Subject: RE: Dubuque IA CO Problems I assume this Dubuque, Iowa? The only thing I've seen thus far is that at 11:25 am Central the wireless carrier we're associated with notified us of an outage in Scales Mound, IL, which 30 miles east of Dubuque. I'm not sure if that cell site is fed out or through Dubuque, but considering the timing and relative location, the two could be connected. Frank -----Original Message----- From: outages-bounces(a)outages.org [mailto:outages-bounces@outages.org] On Behalf Of Dantzig, Brian Sent: Tuesday, June 12, 2012 11:27 AM To: outages(a)outages.org Subject: [outages] Dubuque IA CO Problems We lost voice and data circuits coming out of the Dubuque CO. Both copper T1s and our Fiber node are down. I tried calling a few local businesses in Dubuque and I get fast busy's. Nothing yet from my carriers. Anyone have any other information. Brian Dantzig _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

1 0

Peeringdb down?
by Vinny_Abello＠Dell.com 08 Jun '12

08 Jun '12

I already sent an email to support(a)peeringdb.com<mailto:support@peeringdb.com> about this, but I'm uncertain they are receiving email properly, so I'll mention it here as well. I hadn't seen mention of it already anywhere. I'm seeing MySQL errors when attempting to query peeringdb.com this morning. I'm assuming the admins or friends of admins lurk here. :) IE: Database error: Invalid SQL: .....<edited> MySQL Error: 1030 (Got error 28 from storage engine) Session halted. -Vinny

3 3