Looks like a problem inside godaddy as I can trace into their network but the last hop appears to be unreachable 4: 198.32.160.35 11.850ms asymm 6 5: phx1-ar3-xe-2-1-0-0.us.twtelecom.net 72.053ms asymm 12 6: 206-15-85-10.static.twtelecom.net 68.600ms asymm 14 7: 208.109.112.253 60.058ms asymm 10 8: 208.109.112.253 111.869ms asymm 10 9: 208.109.113.58 60.585ms asymm 10 10: no reply Is comcast really hosting on timewarner? :) Lightpath Im guessing? chris On Mon, Sep 10, 2012 at 2:02 PM, Mark Price <mprice@tqhosting.com> wrote:

I noticed the same problem when trying to get to some websites

Additionally, Internetpulse.net shows some issues: http://www.internetpulse.net/

On Mon, Sep 10, 2012 at 1:53 PM, chris <tknchris@gmail.com> wrote:

...

It appears PDNS01 and PDNS02 are not resolving. Anyone else seeing the issue?

PDNS01.DOMAINCONTROL.COM PDNS02.DOMAINCONTROL.COM

thanks chris

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Although i'm not sure how you confirm this, Anonymous is claiming responsibility depending on who you listen to: https://www.google.com/search?q=godaddy+outage+anonymous On Mon, Sep 10, 2012 at 2:08 PM, Jonathan Smith <jonathan.smith@venture-computers.com> wrote:

NS43.DOMAINCONTROL.COM

NS44.DOMAINCONTROL.COM

Aren’t working either.

Calls to their support line give me a busy signal, and their website’s not loading for me.

It’s not just you.

From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of chris

Sent: Monday, September 10, 2012 1:54 PM To: outages@outages.org Subject: [outages] Godaddy / Premium DNS outage?

It appears PDNS01 and PDNS02 are not resolving. Anyone else seeing the issue?

PDNS01.DOMAINCONTROL.COM

PDNS02.DOMAINCONTROL.COM

thanks

chris

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

I am seeing this as well. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Mon, Sep 10, 2012 at 2:14 PM, Drew Linsalata <drew.linsalata@gmail.com>wrote:

Its more than just Premium DNS. We're seeing a slew of customer domains going dark, all of which are hosted on any number of nsxx.domaincontrol.com authoritative servers.

On Mon, Sep 10, 2012 at 1:53 PM, chris <tknchris@gmail.com> wrote:

...

It appears PDNS01 and PDNS02 are not resolving. Anyone else seeing the issue?

PDNS01.DOMAINCONTROL.COM PDNS02.DOMAINCONTROL.COM

thanks chris

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

On Mon, Sep 10, 2012 at 1:02 PM, Jared Mauch <jared@puck.nether.net> wrote:

Looks like many networks are not rfc2182 compliant. If you suffered an outage, you should read this.

(See http://tools.ietf.org/html/rfc2182 please)

Funny--I *just* had that conversation with a GoDaddy customer this morning and they dismissed it. Once their VoIP server is reachable again, I'll have to give them a call... ;) -A

Just to follow up. godaddy says it was not a ddos. http://www.godaddy.com/newscenter/release-view.aspx?news_item_id=410 phil

On 9/11/12 10:39 AM, Ben Carleton wrote:

"corrupted router data tables..." Any ideas on what this refers to? BGP?

It's either referring to a BGP/OSPF/IS-IS (insert dynamic routing protocol here) bug or a line of corporate-speak BS that sounds technical enough to where most people aren't going to ask for details. ~Seth

Frankly, "we were ddos'd" is a much better sell than "we screwed up our internal systems horribly because we're idiots.". Surprised they went this direction. On Sep 11, 2012, at 10:53 AM, "francis.daigneault@bell.ca" <francis.daigneault@bell.ca> wrote:

It's the easy answer to avoid admitting they where DDOS, when you can't explain something or you don't want to admit something blame it on "corruption"

The outages start and stop at the time predicted by the hacker who "admit" it.. how could that happen.. coincidence ?

-=--=--=--=--=--=--=--=--=--=--=--=--=--=--=-- Francis Daigneault (CCNP,MCSE,ECSA|LPT) KNOWLEDGE IS FREE !

-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Seth Mattinen Sent: September-11-12 1:48 PM To: outages@outages.org Subject: Re: [outages] Godaddy / Premium DNS outage?

On 9/11/12 10:39 AM, Ben Carleton wrote:

...

"corrupted router data tables..." Any ideas on what this refers to? BGP?

It's either referring to a BGP/OSPF/IS-IS (insert dynamic routing protocol here) bug or a line of corporate-speak BS that sounds technical enough to where most people aren't going to ask for details.

~Seth _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Your right! perhaps DDOS does not mean any data have been compromise, it simply mean someone decide to piss you off.., and personally I prefer honesty, or more detail answer then "corrupted data" which is too easy.. let see what they going to say on that one.. http://pastebin.com/WMb4Qg7H I guess it's a internal leak.. I'm not good enough in programation to say if this is fake or real or publicly available, clearly there missing stuff.. like the DB :) perhaps for me it appear a bit more serious and making sense then the godaddy story about yesterday failure. -=--=--=--=--=--=--=--=--=--=--=--=--=--=--=-- Francis Daigneault (CCNP,MCSE,ECSA|LPT) KNOWLEDGE IS FREE !   -----Original Message----- From: Cary Smith [mailto:csmith@pickmta.com] Sent: September-11-12 4:40 PM To: Corey Quinn; Daigneault, Francis (P008790) Cc: outages@outages.org Subject: RE: [outages] Godaddy / Premium DNS outage? I'm not surprised Go-Daddy went in the direction of "internal network events that corrupted router data tables". It is definitely the lesser of the two evils. To come out and admit they were hacked or experienced a DDOS would make Customers feel their information on Go-Daddy servers was potentially compromised. Hence the CEO's statement, " At no time was any customer data at risk or were any of our systems compromised." Unless there's a government investigation, which is doubtful, none of us will really ever know what truly happened. Cary W. Smith MTA 1567 East 93rd. Avenue Merrillville, IN 46410 Office - 219-750-1803 Cell - 219-765-1251 e-mail - cary@pickmta.com -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Corey Quinn Sent: Tuesday, September 11, 2012 1:02 PM To: francis.daigneault@bell.ca Cc: outages@outages.org Subject: Re: [outages] Godaddy / Premium DNS outage? Frankly, "we were ddos'd" is a much better sell than "we screwed up our internal systems horribly because we're idiots.". Surprised they went this direction. On Sep 11, 2012, at 10:53 AM, "francis.daigneault@bell.ca" <francis.daigneault@bell.ca> wrote:

It's the easy answer to avoid admitting they where DDOS, when you can't explain something or you don't want to admit something blame it on "corruption"

The outages start and stop at the time predicted by the hacker who "admit" it.. how could that happen.. coincidence ?

-=--=--=--=--=--=--=--=--=--=--=--=--=--=--=-- Francis Daigneault (CCNP,MCSE,ECSA|LPT) KNOWLEDGE IS FREE !

-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Seth Mattinen Sent: September-11-12 1:48 PM To: outages@outages.org Subject: Re: [outages] Godaddy / Premium DNS outage?

On 9/11/12 10:39 AM, Ben Carleton wrote:

...

"corrupted router data tables..." Any ideas on what this refers to? BGP?

It's either referring to a BGP/OSPF/IS-IS (insert dynamic routing protocol here) bug or a line of corporate-speak BS that sounds technical enough to where most people aren't going to ask for details.

~Seth _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages ________________________________ DISCLAIMER: The information in this message is considered confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, or distribution of the message, or any action or omission taken by you in reliance on it, is prohibited and may be unlawful. Please immediately contact the sender if you have received this message in error.

On Tue, Sep 11, 2012 at 3:01 PM, Darius Jahandarie <djahandarie@gmail.com> wrote:

What a bunch of FUD ... I hope that on a list of network professionals we can put that sort of speculation to rest.

Amen to that. Beyond, let's wait until further information comes out instead of jumping on finger pointing trying to "force" a vendor to defend against unsubstantiated claims about a handful of files from 2010 and '302' responses on some SQLi attempts. Also, perhaps a thread can be posted to 'outages-discussion' for followup on whether or not you believe GoDaddy's claims, and to discuss some credible information that might exist to the contrary, rather than in 'outages' proper.

I made a commitment to get everyone on here the RFO. Its presented in layman's terms: http://inside.godaddy.com/inside-story-happened-godaddy-com-sept-10-2012/ Regards, Jason LeBlanc Network Engineering (480)505-8800 x7202 Please contact my direct supervisor mdob@godaddy.com with any feedback. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Cary Smith Sent: Tuesday, September 11, 2012 1:40 PM To: Corey Quinn; francis.daigneault@bell.ca Cc: outages@outages.org Subject: Re: [outages] Godaddy / Premium DNS outage? I'm not surprised Go-Daddy went in the direction of "internal network events that corrupted router data tables". It is definitely the lesser of the two evils. To come out and admit they were hacked or experienced a DDOS would make Customers feel their information on Go-Daddy servers was potentially compromised. Hence the CEO's statement, " At no time was any customer data at risk or were any of our systems compromised." Unless there's a government investigation, which is doubtful, none of us will really ever know what truly happened. Cary W. Smith MTA 1567 East 93rd. Avenue Merrillville, IN 46410 Office - 219-750-1803 Cell - 219-765-1251 e-mail - cary@pickmta.com -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Corey Quinn Sent: Tuesday, September 11, 2012 1:02 PM To: francis.daigneault@bell.ca Cc: outages@outages.org Subject: Re: [outages] Godaddy / Premium DNS outage? Frankly, "we were ddos'd" is a much better sell than "we screwed up our internal systems horribly because we're idiots.". Surprised they went this direction. On Sep 11, 2012, at 10:53 AM, "francis.daigneault@bell.ca" <francis.daigneault@bell.ca> wrote:

It's the easy answer to avoid admitting they where DDOS, when you can't explain something or you don't want to admit something blame it on "corruption"

The outages start and stop at the time predicted by the hacker who "admit" it.. how could that happen.. coincidence ?

-=--=--=--=--=--=--=--=--=--=--=--=--=--=--=-- Francis Daigneault (CCNP,MCSE,ECSA|LPT) KNOWLEDGE IS FREE !

-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Seth Mattinen Sent: September-11-12 1:48 PM To: outages@outages.org Subject: Re: [outages] Godaddy / Premium DNS outage?

On 9/11/12 10:39 AM, Ben Carleton wrote:

...

"corrupted router data tables..." Any ideas on what this refers to? BGP?

It's either referring to a BGP/OSPF/IS-IS (insert dynamic routing protocol here) bug or a line of corporate-speak BS that sounds technical enough to where most people aren't going to ask for details.

~Seth _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages ________________________________ DISCLAIMER: The information in this message is considered confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, or distribution of the message, or any action or omission taken by you in reliance on it, is prohibited and may be unlawful. Please immediately contact the sender if you have received this message in error. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

He won't be able to answer, otherwise his employer would get upset. Thanks for posting the link. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Fri, Oct 5, 2012 at 2:29 PM, ryanL <ryan.landry@gmail.com> wrote:

so, redistributed eBGP into IGP? how does one 210x their routing table? i appreciate the openness, but it doesn't sound like a "perfect storm" to me. it sounds like someone screwed up.

On Fri, Oct 5, 2012 at 10:18 AM, Jason LeBlanc <jleblanc@godaddy.com>wrote:

...

I made a commitment to get everyone on here the RFO. Its presented in layman's terms:

http://inside.godaddy.com/inside-story-happened-godaddy-com-sept-10-2012/

Regards,

Jason LeBlanc Network Engineering (480)505-8800 x7202

Please contact my direct supervisor mdob@godaddy.com with any feedback.

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

as this is not an active outage please move to discussion thank you. Regards, Bradley D. Bopp Director Of Engineering http://www.nationalnet.com http://as22384.peeringdb.net NationalNet is committed to the highest level of Customer Service available in the Web Hosting business. If you have any questions, comments or concerns feel free to contact us at 678-247-7000 ext 1 (or toll-free, 888-4-NATNET). On Oct 6, 2012, at Oct 6, 2012, 8:35 AM, Warren Kumari <warren@kumari.net> wrote:

-- Warren Kumari Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Friday, October 5, 2012 at 2:29 PM, ryanL wrote:

...

so, redistributed eBGP into IGP? how does one 210x their routing table? i appreciate the openness, but it doesn't sound like a "perfect storm" to me. it sounds like someone screwed up.

Could be someone screwed up, could have been caused by a bug.

Your assumption that this was redistribution of eBGP -> IGP is not really backed up by the text. if I had to guess, I'd suppose that they have devices (like cisco 4900 series) that don't have enough memory / FIB / TCAM for full tables. Normally they only send partial routes (such as internal only / large aggregates / something) and accidentally or though a bug dropped filter and overflowed these devices…

But, whatever the case, GoDaddy has released a somewhat reasonable post-mortem, sounds like they are / have addressed the issue and os it shouldn't happen again, etc. As many know, I'm not a real of GoDaddy, but credit where it is due…

Running a large network is non-trivial, sometimes s%#@t happens -- at least GoDaddy was willing to publish something and not just hand wave this away…

We keep kvetching that folk never publish postmortems when they have an outage -- lets not attack them when they do…

W

...

On Fri, Oct 5, 2012 at 10:18 AM, Jason LeBlanc <jleblanc@godaddy.com (mailto:jleblanc@godaddy.com)> wrote:

...

I made a commitment to get everyone on here the RFO. Its presented in layman's terms:

http://inside.godaddy.com/inside-story-happened-godaddy-com-sept-10-2012/

Regards,

Jason LeBlanc Network Engineering (480)505-8800 x7202 (tel:%28480%29505-8800%20x7202)

Please contact my direct supervisor mdob@godaddy.com (mailto:mdob@godaddy.com) with any feedback.

---

Outages mailing list Outages@outages.org (mailto:Outages@outages.org) https://puck.nether.net/mailman/listinfo/outages

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

On 09/11/2012 01:53 PM, francis.daigneault-t/WHVqD6nWw@public.gmane.org wrote:

...

It's the easy answer to avoid admitting they where DDOS, when you can't explain something or you don't want to admit something blame it on "corruption" The outages start and stop at the time predicted by the hacker who "admit" it.. how could that happen.. coincidence ?

What if 'corruption' in this case meant 'tampering'? Is it possible that their routing tables were corrupted either by injection or someone compromising a router and reconfiguring it?

It's been a few years since I last heard about route injection (at DefCon, wasn't it?) so I don't know whether or not it's widely possible, or at least plausible in the case of GoDaddy.

I don't think this is the case. I've done my own (rudimentary) analysis, and their routers were unaffected as far as the BGP DFZ is concerned (check http://bgplay.routeviews.org or http://bgp.potaroo.net against their route announcements and you won't see any unusual activity). Normally, NANOG tends to be quick on routing issues if they affect the Internet, but everyone seemed to be more focused on the DNS issues at the time. Some actually claiming connectivity (albeit intermittent) by IP. Going from what they've said, chatter on the Internet (and on this list), and the conjectures this is what I've concluded: If it were a BGP issue, it would have to be an incredibly nasty iBGP problem. The nature of the problem would've had to shown intermittent connectivity as there are posted traceroutes (under this subject thread) during the outage showing reachability to GoDaddy. Having said that, to take 5-12 hours to recover doesn't really sound accurate at all, even if it was a misconfiguration. Diagnostics and active monitoring software which undoubtedly they have in place can usually pinpoint the problem. But supposing they don't have their act together at all, it's still a more convenient excuse than the alternatives. If it was a security breach, this will surmount to more negative publicity, media outlets spreading information, and basically scaring off their customer base. They also participate in the browser CA oligarchy, so I imagine people would think twice before purchasing an SSL certificate from GoDaddy. If it was a DoS/DDoS attack, they're basically admitting their capacity and equipment is inadequate. Competitors will target their customer base claiming they're DDoS resilient (which for the most part is BS marketing), and they will probably lose a hearty chunk of customers. Lastly, GoDaddy may have no understanding or comprehension of the events that occured during the outage, outside of what we already know. This could be plausible assuming they weren't retaining logs of all their network devices, or did not have enough information (such as storing netflow data during a DoS/DDoS). I'm not going to argue that spreading disinformation is ethical or unethical given the circumstances, but I assume it would be the safest position for the company. Also given that two of their employees, in this list, are quick to promise they wouldn't spread disinformation, it just makes everything seem more suspicious. Also, every announcement is quick to denounce that the outage is due to a security breach or DoS/DDoS attack. This just seems to be a tactic to save face, and to avoid blowback from media outlets that could pick it up, sensationalize it, and result in more negative press. At least by claiming it was their fault either intentionally or unintentionally, they can at least regain customer confidence and goodwill, dish out a bunch of service credit, and everyone can forget about it. But, from all the discussions so far, and what they've actually said, nothing adds up. In any case, the best bet is to wait for more information. - Naveen

http://www.godaddy.com/newscenter/release-view.aspx?news_item_id=410 nice.. this doesn't make sense Raman Sud Director, Network Infrastructure   PhoenixNAP LLC 3402 East University Drive Phoenix AZ 85034 Phone: 480.449.8834 Fax: 480.682.3236 http://www.phoenixnap.com PGP Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBB765DA2 THIS MESSAGE IS INTENDED ONLY FOR THE USE OF THE  INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED, AND MAY CONTAIN INFORMATION  THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE UNDER APPLICABLE  LAW.  IF THE READER OF THIS MESSAGE IS NOT THE INTENDED RECIPIENT, YOU ARE HEREBY NOTIFIED THAT ANY DISSEMINATION, DISTRIBUTION OR  COPYING OF THIS COMMUNICATION IS STRICTLY PROHIBITED, AND THAT THIS EMAIL AND  ALL COPIES AND ATTACHMENTS SHOULD BE IMMEDIATELY  DESTROYED. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Ben Carleton Sent: Tuesday, September 11, 2012 10:40 AM To: Phil Dyer Cc: outages@outages.org Subject: Re: [outages] Godaddy / Premium DNS outage? "corrupted router data tables..." Any ideas on what this refers to? BGP? On Sep 11, 2012, at 12:49 PM, "Phil Dyer" <phil@cluestick.net> wrote:

Just to follow up. godaddy says it was not a ddos.

http://www.godaddy.com/newscenter/release-view.aspx?news_item_id=410

phil _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Speculation - TCAM overload. Check out this talk at NANOG 54: https://www.nanog.org/meetings/nanog54/presentations/Monday/HughesLT.pdf On Sep 11, 2012, at 11:05 AM, Colin Johnston <colinj@mx5.org.uk> wrote:

On 11 Sep 2012, at 18:58, Raman Sud <ramans@phoenixnap.com> wrote:

...

http://www.godaddy.com/newscenter/release-view.aspx?news_item_id=410

nice.. this doesn't make sense

It does make sense from UK traffic perspective, some AS's worked for queries, some AS's did not get a connection at all.

Colin Johnston

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages