September 2015 - Outages

Akamai Cert Issues today
by Jim Witherell 03 Oct '15

03 Oct '15

e noticed SSL warnings based around Akamai's "a248.e.akamai.net" certificate today. NET::ERR_CERT_COMMON_NAME_INVALID is the most common error we're seeing. Can anyone comment on what may be going on? Looks like the cert was renewed or issued on 8/27/2015. Wonder why we are noticing the errors from multiple points on the internet now? Jim Witherell Cincinnati OH

10 16

Re: [outages] Facebook throwing 500 errors?
by William Byrd 29 Sep '15

29 Sep '15

Hey Rafael - I am sure it would make a very interesting post mortem post, and I will ask if anyone would be willing to put something together to share externally. Right now I can't share any details, so I'll just have to apologize (again) for the inconvenience and see if we can get something posted. William Collier-Byrd will(a)collier-byrd.net On Tue, Sep 29, 2015 at 8:25 AM, Rafael Possamai <rafael(a)e2wsolutions.com> wrote: > Hi Will, > > Are you able to disclose any high level details of what happened? It's > just interesting to know how such a large infrastructure, with assumed > redundancies, can go down completely. I know AWS had a similar outage, > which lasted even longer, but it was a series of cascading problems due to > human error. Was it an SDN bad config push? > > > > > On Mon, Sep 28, 2015 at 2:59 PM, William Collier-Byrd via Outages < > outages(a)outages.org> wrote: > >> I am starting to feel like a broken record, but we're aware and working >> on it. >> >> -- >> Will Collier-Byrd >> >> >> >> >> On Mon, Sep 28, 2015 at 12:56 PM -0700, "Mike Phipps via Outages" < >> outages(a)outages.org> wrote: >> >> Confirmed on T-Mobile LTE and Comcast Business Class in metro Detroit. It >>> is that same message. >>> >>> On Mon, Sep 28, 2015 at 3:27 PM, Bob Colon via Outages < >>> outages(a)outages.org> wrote: >>> >>>> Yup....getting the same here, >>>> >>>> To ensure prompt service, please use "Reply All" when responding >>>> >>>> Robert Colon | TCSA Tier 3 | EthoStream LLC | P: (877) 282-2519 x2114 >>>> | F: (414) 258-8307 | 20800 Swenson Drive, Suite 175 Waukesha, WI >>>> 53186 | <http://www.telkonet.com/>www.telkonet.com | www.ethostream.com >>>> | @Telkonet <https://twitter.com/telkonet> >>>> >>>> On 9/28/2015 2:23 PM, James Hartig via Outages wrote: >>>> >>>> Anyone else getting: >>>> >>>> Sorry, something went wrong. >>>> We're working on it and we'll get it fixed as soon as we can. >>>> >>>> when you go to Facebook? >>>> -- >>>> James Hartig >>>> >>>> >>>> _______________________________________________ >>>> Outages mailing listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages >>>> >>>> >>>> >>>> ------------------------------ >>>> >>>> <http://www.telkonet.com> <http://www.ethostream.com> >>>> >>>> This e-mail and any files transmitted with it are private and >>>> confidential and are solely for the use of the addressee. It may contain >>>> material which is legally privileged. If you are not the addressee or the >>>> person responsible for delivering to the addressee, be advised that you >>>> have received this e-mail in error and that any use of it is strictly >>>> prohibited. If you have received this e-mail in error please contact us at (414) >>>> 223-0473. >>>> >>>> _______________________________________________ >>>> Outages mailing list >>>> Outages(a)outages.org >>>> https://puck.nether.net/mailman/listinfo/outages >>>> >>>> >>> >> _______________________________________________ >> Outages mailing list >> Outages(a)outages.org >> https://puck.nether.net/mailman/listinfo/outages >> >> >

6 6

Re: [outages] Facebook throwing 500 errors?
by William Byrd 29 Sep '15

29 Sep '15

Hah, I stopped at the day and figured it was from the outage last week. At any rate, I have a few threads going internally to ask about publishing post mortems regarding these most recent outages. William Collier-Byrd will(a)collier-byrd.net On Tue, Sep 29, 2015 at 10:54 AM, Erik Sundberg <ESundberg(a)nitelusa.com> wrote: > Thats from September 2010, a bit old i say. > > > > > Erik Sundberg > Sr. Network Engineer > Nitel > 1101 West Lake Street,6th Fl > Chicago, IL 60607 > Desk: 773-661-5532 > Cell: 708-710-7419 > NOC 24/7: 866-892-0915 > Email: esundberg(a)nitelusa.com > http://www.nitelusa.com > On Sep 29, 2015, at 12:50 PM, Scott Voll via Outages <outages(a)outages.org> > wrote: > > > https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-… > > Guess it's already out there. > > Scott > > > On Tue, Sep 29, 2015 at 9:36 AM, William Byrd via Outages < > outages(a)outages.org> wrote: > >> Hey Rafael - >> >> I am sure it would make a very interesting post mortem post, and I will >> ask if anyone would be willing to put something together to share >> externally. Right now I can't share any details, so I'll just have to >> apologize (again) for the inconvenience and see if we can get something >> posted. >> >> William Collier-Byrd >> will(a)collier-byrd.net >> >

1 0

Significant XO outage in the Cleveland area
by Jd 29 Sep '15

29 Sep '15

5 4

github timing out
by Grant Ridder 29 Sep '15

29 Sep '15

I am seeing time outs to github.com (resolves to 192.30.252.128) from SF bay area. -Grant

2 2

Facebook down?
by Scott Voll 29 Sep '15

29 Sep '15

http://downforeveryoneorjustme.com/facebook.com Shows it's down. Scott

3 2

Facebook throwing 500 errors?
by James Hartig 29 Sep '15

29 Sep '15

Anyone else getting: Sorry, something went wrong. We're working on it and we'll get it fixed as soon as we can. when you go to Facebook? -- James Hartig

24 25

Re: [outages] Facebook firm-down*
by William Byrd 24 Sep '15

24 Sep '15

Site should be recovering now. Our apologies for the service interruptions. William Collier-Byrd will(a)collier-byrd.net On Thu, Sep 24, 2015 at 9:50 AM, Chaim Rieger <chaim.rieger(a)gmail.com> wrote: > Seems to be back, at least for me… > > > On Sep 24, 2015, at 9:45 AM, William Byrd via Outages <outages(a)outages.org> > wrote: > > We are aware and actively working on the issue. > > William Collier-Byrd > will(a)collier-byrd.net > > On Thu, Sep 24, 2015 at 9:43 AM, Jay Ashworth via Outages < > outages(a)outages.org> wrote: > >> As of about 10 minutes ago, Facebook is giving out a "Sorry, something >> went >> wrong" message to at least some subset of users which includes me. >> >> The live map at downdetector.com shows large hotspots in the NE and the >> UK. >> I get to mine via Sprint LTE, which terminates in Lenexa KS, I think; >> there's a hotspot in the map in FL as well, but not as big yet. >> >> DD reports they've been having trouble today since 0731EDT, but I hadn't >> noticed it until now. >> >> Those who do eyeball support might want to know. >> >> Cheers, >> -- jra >> >> * Firm-down: Not quite a hard-down; the edge caching server are up and >> giving a >> message suggesting the stuff behind them is dead. >> >> -- >> Jay R. Ashworth Baylink >> jra(a)baylink.com >> Designer The Things I Think RFC >> 2100 >> Ashworth & Associates http://www.bcp38.info 2000 Land >> Rover DII >> St Petersburg FL USA BCP38: Ask For It By Name! +1 727 >> 647 1274 >> _______________________________________________ >> Outages mailing list >> Outages(a)outages.org >> https://puck.nether.net/mailman/listinfo/outages >> > > _______________________________________________ > Outages mailing list > Outages(a)outages.org > https://puck.nether.net/mailman/listinfo/outages > > >

2 1

Facebook firm-down*
by Jay Ashworth 24 Sep '15

24 Sep '15

As of about 10 minutes ago, Facebook is giving out a "Sorry, something went wrong" message to at least some subset of users which includes me. The live map at downdetector.com shows large hotspots in the NE and the UK. I get to mine via Sprint LTE, which terminates in Lenexa KS, I think; there's a hotspot in the map in FL as well, but not as big yet. DD reports they've been having trouble today since 0731EDT, but I hadn't noticed it until now. Those who do eyeball support might want to know. Cheers, -- jra * Firm-down: Not quite a hard-down; the edge caching server are up and giving a message suggesting the stuff behind them is dead. -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

3 2

AT&T and Telia.net
by Edward Spoon 22 Sep '15

22 Sep '15

Can't get to yahoo.com and others from an AT&T (AS7018) circuit. Trace routes die in either Chicago or Seattle at telia.net. Been going on for at least 3 hours. traceroute to yahoo.com (98.138.253.109), 64 hops max, 40 byte packets 1 172.18.0.1 (172.18.0.1) 0.575 ms 0.468 ms 0.305 ms 2 xxxxxxxxxxxxxxxxxxxxxxxx 0.957 ms 1.030 ms 1.081 ms 3 cr1.nwrla.ip.att.net (12.123.153.38) 253.161 ms 289.990 ms 259.564 ms 4 cr1.nwrla.ip.att.net (12.123.153.38) 253.579 ms 273.143 ms 259.231 ms 5 cr2.attga.ip.att.net (12.122.18.1) 281.024 ms cr1.attga.ip.att.net (12.122.2.145) 263.013 ms cr2.attga.ip.att.net (12.122.18.1) 284.453 ms 6 12.122.117.97 (12.122.117.97) 312.646 ms gar24.attga.ip.att.net (12.122.141.181) 274.122 ms 249.340 ms 7 192.205.33.42 (192.205.33.42) 284.968 ms 289.257 ms 285.251 ms 8 chi-b21-link.telia.net (62.115.113.25) 322.581 ms chi-b21-link.telia.net (62.115.113.19) 314.945 ms chi-b21-link.telia.net (62.115.143.62) 293.350 ms 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * *

1 0