January 2014 - Outages - Outages Lists

Does anyone know what's going on with Level3/CenturyLink?
by Owen Roth 30 Jan '14

30 Jan '14

So: http://www.internetpulse.net/Main.aspx?Metric=PL is showing 12.5% packet loss over the last hour for CenturyLink<-->Level3? Is anyone else seeing this? Trying to do match this problem against an earlier issue I had. Regards, ------------ Owen Roth Snr Network Engineer Impulse Advanced Communications owen(a)impulse.net 805-884-6332

2 1

CL hl-priv20 MPLS is down
by Karl Putland 30 Jan '14

30 Jan '14

More CL MPLS issues... hl-priv20 is down --Karl Karl Putland Senior Engineer *SimpleSignal* Anywhere: 303-242-8608 <http://www.simplesignal.com/explainer_video.php>

1 0

1-28-2014 - AT&T outage in Maryland area
by Bertram, Matthew 29 Jan '14

29 Jan '14

We've got a managed MPLS site down in MD, I've been given the following update from AT&T: 01/28/2014,11:41:26:[AT&T ] Activity Type Code Desc: GENERAL COMMENTS Activity Type Code: GEN THE FOLLOWING STATUS INFORMATION ON YOUR TROUBLE HAS BEEN SENT FROM AT&T NETWORK OPERATIONS. Currently 22 T3's still down, no further restoration ability, at least 1hr eta f or tech troubleshooting. My guess is either a fiber fault somewhere or significant equipment failure. - Matt

3 4

Re: [outages] Century Link MPLS Hiccups
by Lou Labenskyj 28 Jan '14

28 Jan '14

Ours are all showing green Thanks Lou Labenskyj On Jan 28, 2014, at 9:19 AM, "Karl Roesler" <kjroesler(a)cbecompanies.com> wrote: Just saw one of our CL MPLS connections to the Philippines go down about 5 minutes ago. Anyone else still experiencing hiccups today? Karl >>> Karl Putland <karl(a)simplesignal.com> 1/27/2014 6:10 PM >>> No alerts on for us on CL MPLS. --Karl Karl Putland Senior Engineer SimpleSignal Anywhere: 303-242-8608 <mime-attachment.png> <mime-attachment.png> > On Mon, Jan 27, 2014 at 4:46 PM, Lou Labenskyj <llabenskyj(a)bashas.com> wrote: > Our MPLS has hiccupped twice in the last half hour. Anyone else seeing outages? We are in AZ. > > Thanks > > Lou Labenskyj > Bashas' Inc. > Director of Infrastructure, Development & Security > (480) 802-5477 Voice > (480) 895-5293 Fax > (480) 895-3377 PCI Hotline > > > > _______________________________________________ > Outages mailing list > Outages(a)outages.org > https://puck.nether.net/mailman/listinfo/outages THIS MESSAGE AND ACCOMPANYING DOCUMENTS ARE COVERED BY THE ELECTRONIC COMMUNICATIONS PRIVACY ACT, 18 U.S.C. §§ 2510-2521, AS AMENDED, AND CONTAINS INFORMATION INTENDED FOR THE SPECIFIED INDIVIDUAL(S) ONLY. THIS INFORMATION IS CONFIDENTIAL. IF YOU ARE NOT THE INTENDED RECIPIENT OR AN AGENT RESPONSIBLE FOR DELIVERING IT TO THE INTENDED RECIPIENT, YOU ARE HEREBY NOTIFIED THAT YOU HAVE RECEIVED THIS DOCUMENT IN ERROR AND THAT ANY REVIEW, DISSEMINATION, COPYING, OR THE TAKING OF ANY ACTION BASED ON THE CONTENTS OF THIS INFORMATION IS STRICTLY PROHIBITED. IF YOU HAVE RECEIVED THIS COMMUNICATION IN ERROR, PLEASE NOTIFY US IMMEDIATELY BY E-MAIL, AND DELETE THE ORIGINAL MESSAGE. IN AN EFFORT TO PROTECT PRIVATE CONSUMER INFORMATION AND TO MAINTAIN STRICT COMPLIANCE WITH THE HEALTH INSURANCE PORTABILITY AND ACCOUNTABILITY ACT (HIPAA), P.L.104-191, AS AMENDED AND THE GRAMM-LEACH-BLILEY ACT (GLBA), P.L.106-102, CBE COMPANIES, INC., AND ALL OF ITS SUBSIDIARIES AND AFFILIATES, DOES NOT SEND AND ENCOURAGES OTHERS NOT TO SEND NONPUBLIC PERSONAL INFORMATION (NPI) OR PERSONAL HEALTH INFORMATION (PHI) VIA UNSECURED E-MAIL.

1 0

Century Link MPLS Hiccups
by Lou Labenskyj 27 Jan '14

27 Jan '14

Our MPLS has hiccupped twice in the last half hour. Anyone else seeing outages? We are in AZ. Thanks Lou Labenskyj Bashas' Inc. Director of Infrastructure, Development & Security (480) 802-5477 Voice (480) 895-5293 Fax (480) 895-3377 PCI Hotline

4 3

ATT UVerse problems with default DNS server
by Edward Vielmetti 27 Jan '14

27 Jan '14

On ATT UVerse in Michigan (48104), having problems reaching some web sites including MLive.com. I checked "Down detector" for ATT (http://downdetector.com/status/att) and it's reporting lots of problems nationally. The comments suggest to change your DNS server and all will be well, so I did (to Google's 8.8.8.8) and all is indeed well. FYI. A little writeup here http://vielmetti.typepad.com/vacuum/2014/01/27-january-2014-att-uverse-dns-… to capture the moment. thanks! Ed -- Edward Vielmetti +1 734 330 2465 edward.vielmetti(a)gmail.com

3 2

Google outage after-action
by Jay Ashworth 25 Jan '14

25 Jan '14

Here's the formal one, 'pon the Google Blog, though not in any great detail. In short: whatever they build configurations with, for pushing to servers via cfengine or puppet or whatever they use, had a bug, and built a broken config, and didn't catch it, and it got pushed. http://googleblog.blogspot.com/2014/01/todays-outage-for-several-google.html No details on whether there was contributory control-plane congestion, as often causes, or extends, AWS outages. Ironically, Google's Site Reliability Engineering team *was doing an AMA when the outage hit*. No comment from the 'plex, either, on whether those are connected, and the AMAers declined to comment. http://www.reddit.com/r/IAmA/comments/1w1y5m/we_are_the_google_site_reliabi… Highlights: Q: Pager?!?! Why don't you use text messages for that? A: "Pager" is a synonynm for "A beepy thing that goes 'beep'." and I can assure you that Google does not use nagios. Source: I used to be a Google SRE. and Q: Sooo....what's it like there when a Google service goes down? How much freaking out is done? A: Very little freaking out actually, we have a well-oiled process for this that all services use - we use thoroughly documented incident management procedures, so people understand their role explicitly and can act very quickly. We also exercise this processes regularly as part of our DiRT testing. Running regular service-specific drills is also a big part of making sure that once something goes wrong, we’re straight on it. Folo: In a highly parallel environment, it should not be necessary that all pants are shat in unison. The map-reduce pattern allows each dump to be taken in its own time, and then collected once all shits are complete. Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

3 2

Power outage - Cologix 3 - Montreal, QC Canada
by Eric Dugas 24 Jan '14

24 Jan '14

We lost a full cabinet (with dual electric feed) at 1250 Rene-Levesque, Montreal, QC Canada. A lot of carriers are present at that datacenter, Level3, Cogent, Hibernia, etc.. Just received this email from Cologix support: "Hello, Please be advised that Cologix is experiencing a commercial power issue at our Cologix 3, 1250 René Levesque datacenter. Our technicians are informed and working on repairing the problem. An update will be provided to you in the next 30 minutes. Regards," Eric Dugas edugas(a)zerofail.com Network Management Group ZEROFAIL / AS40191 Montreal (514) 848-9198 x310 New York (212) 796-5853 x310 Toronto (416) 800-4209 x310 1-888-GO-0FAIL (460-3245) Facebook | Twitter | LinkedIn | Google+ www.zerofail.com

3 5

Re: [outages] Google Outage now clear; after actions and press reports
by Evan Anderson 24 Jan '14

24 Jan '14

Dashboard reports that the issues have been resolved. http://www.google.com/appsstatus#hl=en&v=status&ts=1390595430189 On Jan 24, 2014 3:20 PM, Mike <miketheman(a)gmail.com> wrote: > > Article is from Aug 2013 - current? > > > On Fri, Jan 24, 2014 at 3:11 PM, Jay Ashworth <jra(a)baylink.com> wrote: >> >> They're coming out all over; this one from Marketplace and the Verge >> cites a 40% drop in overall traffic while they were down. >> >> http://www.marketplace.org/topics/tech/down-goes-google-down-goes-internet >> >> Cheers, >> -- jra >> >> -- >> Jay R. Ashworth Baylink jra(a)baylink.com >> Designer The Things I Think RFC 2100 >> Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII >> St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274 >> _______________________________________________ >> Outages mailing list >> Outages(a)outages.org >> https://puck.nether.net/mailman/listinfo/outages > >

1 0

Google Outage now clear; after actions and press reports
by Jay Ashworth 24 Jan '14

24 Jan '14

They're coming out all over; this one from Marketplace and the Verge cites a 40% drop in overall traffic while they were down. http://www.marketplace.org/topics/tech/down-goes-google-down-goes-internet Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

2 2