November 2010 - Outages - Outages Lists

Comcast DNS outage on the evening of Sunday, November 28
by Frank Bulk - iName.com 30 Nov '10

30 Nov '10

There was all kinds of press coverage about the outage, but not a lot of technical detail. Based on reports, Comcast Internet customers in Massachusetts, Connecticut, Maryland, Virginia, New York, New Hampshire, and parts of Washington, D.C. were without access their DNS server(s) from about 8:30 pm Eastern to 11:30 pm Eastern, though some say it started as early at 8 pm and ended after midnight. NBC was told by a Comcast technician that outages were spread out over six states in the Northeast and Mid-Atlantic. Comcast said that phone and CATV customers were not affected. According to published reports, Comcast was performing regular maintenance on their DNS servers and something went wrong. A Comcast spokesman said on Monday that "Last night, our engineers identified a server issue that was affecting Internet service in Boston, parts of New England and the DC/Beltway areas". Comcast did not share how many customers were affected. BTW, did anyone look at http://dns.comcast.net/status.php during this outage? Regards, Frank

2 1

Re: [outages] Big Level 3 loss problem in Washington DC area?
by Bill Wichers 24 Nov '10

24 Nov '10

We're peered w/ level(3) in southeast michigan and didn't notice anything unusual. Sorry for the too-much-included reply -- this little blackberry is limited in it's editing abilities... [Sent using Blackberry Messaging] ----- Original Message ----- From: virendra rode <virendra.rode(a)outages.org> To: outages(a)outages.org <outages(a)outages.org> Sent: Wed Nov 24 19:13:38 2010 Subject: Re: [outages] Big Level 3 loss problem in Washington DC area? On 11/24/2010 02:05 PM, Terry Hardie wrote: > Anyone else seeing this? We've shutdown our links with Level 3 in the > Ashburn, VA area.... > ------------------------------- Hmm, we had customers calling from Southeast Michigan complaining about slow access and at times high latency via one of our peers which happens to be level3. It appears problem went away when we looked early afternoon about 12:30pm PST. regards, /virendra _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

1 0

Big Level 3 loss problem in Washington DC area?
by Terry Hardie 24 Nov '10

24 Nov '10

Anyone else seeing this? We've shutdown our links with Level 3 in the Ashburn, VA area.... -- Terry Hardie - terry(a)cotendo.com - +1-510-206-2366 Cotendo Inc. - www.cotendo.com - Operations Engineer Skype: cotendoterry

5 5

Re: [outages] Big Level 3 loss problem in Washington DC area?
by Patrick Shoemaker 24 Nov '10

24 Nov '10

Yes, was seeing up to 50% loss on a route through Level3 in DC beginning at 4:52 EST. Seems to have cleared up as of 5:22. ------Original Message------ From: Terry Hardie Sender: outages-bounces(a)outages.org To: outages(a)outages.org Subject: [outages] Big Level 3 loss problem in Washington DC area? Sent: Nov 24, 2010 5:05 PM Anyone else seeing this? We've shutdown our links with Level 3 in the Ashburn, VA area.... -- Terry Hardie - terry(a)cotendo.com - +1-510-206-2366 Cotendo Inc. - www.cotendo.com - Operations Engineer Skype: cotendoterry _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages Sent from my Cellular One BlackBerry.

1 0

Diamondcard.us?
by Aria Stewart 22 Nov '10

22 Nov '10

Anyone heard information on why diamondcard.us (VoIP services) is down? Aria Stewart The Internet Company

1 0

Level 3 Fiber Cut - So. Florida
by Scott, Robert D. 22 Nov '10

22 Nov '10

Level 3 has confirmed a fiber cut between Miami and Tampa FL.; more specifically, based on our equipment, between Deem City and Goodno. Level 3 Master Ticket: 4480302 Robert D. Scott Robert(a)ufl.edu Senior Network Engineer 352-273-0113 Phone CNS - Network Services 352-392-2061 CNS Phone Tree University of Florida 352-392-9440 FAX Florida Lambda Rail 352-294-3571 FLR NOC Gainesville, FL 32611 321-663-0421 Cell 3216630421(a)messaging.sprintpcs.com

1 0

Comcast Chicago Fiber Cut
by Brandon Galbraith 21 Nov '10

21 Nov '10

Anyone else hear of a Comcast fiber cut in Chicago? Our monitoring is showing several client sites down, and they all have Comcast business connections in common. Thanks! -brandon -- Brandon Galbraith US Voice: 630.492.0464

2 3

Facebook down?
by Ujjval Karihaloo 19 Nov '10

19 Nov '10

Is Facebook down again?

24 32

Re: [outages] Facebook down?
by Zachary Hanna 18 Nov '10

18 Nov '10

If facebook made a major bad BGP route announcement and wreaked havoc across a bunch of peers, I would see it as more relevant. Or if a DDOS impacted one of their upstream providers. -zth Zachary Hanna ----- Original Message ----- From: outages-bounces(a)outages.org <outages-bounces(a)outages.org> To: outages(a)outages.org <outages(a)outages.org> Sent: Thu Nov 18 16:11:08 2010 Subject: Re: [outages] Facebook down? On Thu, Nov 18, 2010 at 3:30 PM, Jay Hennigan <jay(a)west.net> wrote: >[,,,] > I agree that Facebook is meaningless from an infrastructure standpoint > but disagree that it is meaningless from an operations standpoint. Seconded. And if it's not meaningless to your operations, it is to many others'. We can be slightly inclusive here, for sites which are well enough known that an outage will be Big News, or we can split the list into a bunch of sublists which only contain certain types of outages, segregating it so that people who really only truly care about fiber and routing can only get fiber and routing outages. The bandwidth of discussion for the non-net/infrastructure stuff is low enough that it's not worth getting worked up about and splitting things, for me. Your mileage may vary, but we've now wasted more bandwidth on list arguing about Facebook being relevant than we actually have had notifications of website outages over the last year-ish, so I think that the point of diminishing returns has vanished behind us in the mirror. If those who object want an infrastructure-only outages list, please propose one. Or apply mail filtering or the D key. -- -george william herbert george.herbert(a)gmail.com _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

1 0

Earthlink/mindspring MX
by Seth Mattinen 18 Nov '10

18 Nov '10

All of the MX servers for earthlink.com and mindspring.com seem to be unreachable; noticed after deferrals tripped the high water mark. I tested on an outside DSL connection as well with no response as well. ~Seth

4 3