Re: [outages] Facebook throwing 500 errors?
Hey Rafael - I am sure it would make a very interesting post mortem post, and I will ask if anyone would be willing to put something together to share externally. Right now I can't share any details, so I'll just have to apologize (again) for the inconvenience and see if we can get something posted. William Collier-Byrd will@collier-byrd.net On Tue, Sep 29, 2015 at 8:25 AM, Rafael Possamai <rafael@e2wsolutions.com> wrote:
Hi Will,
Are you able to disclose any high level details of what happened? It's just interesting to know how such a large infrastructure, with assumed redundancies, can go down completely. I know AWS had a similar outage, which lasted even longer, but it was a series of cascading problems due to human error. Was it an SDN bad config push?
On Mon, Sep 28, 2015 at 2:59 PM, William Collier-Byrd via Outages < outages@outages.org> wrote:
I am starting to feel like a broken record, but we're aware and working on it.
-- Will Collier-Byrd
On Mon, Sep 28, 2015 at 12:56 PM -0700, "Mike Phipps via Outages" < outages@outages.org> wrote:
Confirmed on T-Mobile LTE and Comcast Business Class in metro Detroit. It
is that same message.
On Mon, Sep 28, 2015 at 3:27 PM, Bob Colon via Outages < outages@outages.org> wrote:
Yup....getting the same here,
To ensure prompt service, please use "Reply All" when responding
Robert Colon | TCSA Tier 3 | EthoStream LLC | P: (877) 282-2519 x2114 | F: (414) 258-8307 | 20800 Swenson Drive, Suite 175 Waukesha, WI 53186 | <http://www.telkonet.com/>www.telkonet.com | www.ethostream.com | @Telkonet <https://twitter.com/telkonet>
On 9/28/2015 2:23 PM, James Hartig via Outages wrote:
Anyone else getting:
Sorry, something went wrong. We're working on it and we'll get it fixed as soon as we can.
when you go to Facebook? -- James Hartig
_______________________________________________ Outages mailing listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages
------------------------------
<http://www.telkonet.com> <http://www.ethostream.com>
This e-mail and any files transmitted with it are private and confidential and are solely for the use of the addressee. It may contain material which is legally privileged. If you are not the addressee or the person responsible for delivering to the addressee, be advised that you have received this e-mail in error and that any use of it is strictly prohibited. If you have received this e-mail in error please contact us at (414) 223-0473.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-o... Guess it's already out there. Scott On Tue, Sep 29, 2015 at 9:36 AM, William Byrd via Outages < outages@outages.org> wrote:
Hey Rafael -
I am sure it would make a very interesting post mortem post, and I will ask if anyone would be willing to put something together to share externally. Right now I can't share any details, so I'll just have to apologize (again) for the inconvenience and see if we can get something posted.
William Collier-Byrd will@collier-byrd.net
On Tue, Sep 29, 2015 at 8:25 AM, Rafael Possamai <rafael@e2wsolutions.com> wrote:
Hi Will,
Are you able to disclose any high level details of what happened? It's just interesting to know how such a large infrastructure, with assumed redundancies, can go down completely. I know AWS had a similar outage, which lasted even longer, but it was a series of cascading problems due to human error. Was it an SDN bad config push?
On Mon, Sep 28, 2015 at 2:59 PM, William Collier-Byrd via Outages < outages@outages.org> wrote:
I am starting to feel like a broken record, but we're aware and working on it.
-- Will Collier-Byrd
On Mon, Sep 28, 2015 at 12:56 PM -0700, "Mike Phipps via Outages" < outages@outages.org> wrote:
Confirmed on T-Mobile LTE and Comcast Business Class in metro Detroit.
It is that same message.
On Mon, Sep 28, 2015 at 3:27 PM, Bob Colon via Outages < outages@outages.org> wrote:
Yup....getting the same here,
To ensure prompt service, please use "Reply All" when responding
Robert Colon | TCSA Tier 3 | EthoStream LLC | P: (877) 282-2519 x2114 | F: (414) 258-8307 | 20800 Swenson Drive, Suite 175 Waukesha, WI 53186 | <http://www.telkonet.com/>www.telkonet.com | www.ethostream.com | @Telkonet <https://twitter.com/telkonet>
On 9/28/2015 2:23 PM, James Hartig via Outages wrote:
Anyone else getting:
Sorry, something went wrong. We're working on it and we'll get it fixed as soon as we can.
when you go to Facebook? -- James Hartig
_______________________________________________ Outages mailing listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages
------------------------------
<http://www.telkonet.com> <http://www.ethostream.com>
This e-mail and any files transmitted with it are private and confidential and are solely for the use of the addressee. It may contain material which is legally privileged. If you are not the addressee or the person responsible for delivering to the addressee, be advised that you have received this e-mail in error and that any use of it is strictly prohibited. If you have received this e-mail in error please contact us at (414) 223-0473.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
That was actually from the previous Outage from last week. William Collier-Byrd will@collier-byrd.net On Tue, Sep 29, 2015 at 10:44 AM, Scott Voll <svoll.voip@gmail.com> wrote:
https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-o...
Guess it's already out there.
Scott
On Tue, Sep 29, 2015 at 9:36 AM, William Byrd via Outages < outages@outages.org> wrote:
Hey Rafael -
I am sure it would make a very interesting post mortem post, and I will ask if anyone would be willing to put something together to share externally. Right now I can't share any details, so I'll just have to apologize (again) for the inconvenience and see if we can get something posted.
William Collier-Byrd will@collier-byrd.net
On Tue, Sep 29, 2015 at 8:25 AM, Rafael Possamai <rafael@e2wsolutions.com
wrote:
Hi Will,
Are you able to disclose any high level details of what happened? It's just interesting to know how such a large infrastructure, with assumed redundancies, can go down completely. I know AWS had a similar outage, which lasted even longer, but it was a series of cascading problems due to human error. Was it an SDN bad config push?
On Mon, Sep 28, 2015 at 2:59 PM, William Collier-Byrd via Outages < outages@outages.org> wrote:
I am starting to feel like a broken record, but we're aware and working on it.
-- Will Collier-Byrd
On Mon, Sep 28, 2015 at 12:56 PM -0700, "Mike Phipps via Outages" < outages@outages.org> wrote:
Confirmed on T-Mobile LTE and Comcast Business Class in metro Detroit.
It is that same message.
On Mon, Sep 28, 2015 at 3:27 PM, Bob Colon via Outages < outages@outages.org> wrote:
Yup....getting the same here,
To ensure prompt service, please use "Reply All" when responding
Robert Colon | TCSA Tier 3 | EthoStream LLC | P: (877) 282-2519 x2114 | F: (414) 258-8307 | 20800 Swenson Drive, Suite 175 Waukesha, WI 53186 | <http://www.telkonet.com/>www.telkonet.com | www.ethostream.com | @Telkonet <https://twitter.com/telkonet>
On 9/28/2015 2:23 PM, James Hartig via Outages wrote:
Anyone else getting:
Sorry, something went wrong. We're working on it and we'll get it fixed as soon as we can.
when you go to Facebook? -- James Hartig
_______________________________________________ Outages mailing listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages
------------------------------
<http://www.telkonet.com> <http://www.ethostream.com>
This e-mail and any files transmitted with it are private and confidential and are solely for the use of the addressee. It may contain material which is legally privileged. If you are not the addressee or the person responsible for delivering to the addressee, be advised that you have received this e-mail in error and that any use of it is strictly prohibited. If you have received this e-mail in error please contact us at (414) 223-0473.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
That post is dated 2010. On Tue, Sep 29, 2015 at 10:46 AM, William Byrd via Outages < outages@outages.org> wrote:
That was actually from the previous Outage from last week.
William Collier-Byrd will@collier-byrd.net
On Tue, Sep 29, 2015 at 10:44 AM, Scott Voll <svoll.voip@gmail.com> wrote:
https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-o...
Guess it's already out there.
Scott
On Tue, Sep 29, 2015 at 9:36 AM, William Byrd via Outages < outages@outages.org> wrote:
Hey Rafael -
I am sure it would make a very interesting post mortem post, and I will ask if anyone would be willing to put something together to share externally. Right now I can't share any details, so I'll just have to apologize (again) for the inconvenience and see if we can get something posted.
William Collier-Byrd will@collier-byrd.net
On Tue, Sep 29, 2015 at 8:25 AM, Rafael Possamai < rafael@e2wsolutions.com> wrote:
Hi Will,
Are you able to disclose any high level details of what happened? It's just interesting to know how such a large infrastructure, with assumed redundancies, can go down completely. I know AWS had a similar outage, which lasted even longer, but it was a series of cascading problems due to human error. Was it an SDN bad config push?
On Mon, Sep 28, 2015 at 2:59 PM, William Collier-Byrd via Outages < outages@outages.org> wrote:
I am starting to feel like a broken record, but we're aware and working on it.
-- Will Collier-Byrd
On Mon, Sep 28, 2015 at 12:56 PM -0700, "Mike Phipps via Outages" < outages@outages.org> wrote:
Confirmed on T-Mobile LTE and Comcast Business Class in metro Detroit.
It is that same message.
On Mon, Sep 28, 2015 at 3:27 PM, Bob Colon via Outages < outages@outages.org> wrote:
> Yup....getting the same here, > > To ensure prompt service, please use "Reply All" when responding > > Robert Colon | TCSA Tier 3 | EthoStream LLC | P: (877) 282-2519 > x2114 | F: (414) 258-8307 | 20800 Swenson Drive, Suite 175 > Waukesha, WI 53186 | <http://www.telkonet.com/>www.telkonet.com | > www.ethostream.com | @Telkonet <https://twitter.com/telkonet> > > On 9/28/2015 2:23 PM, James Hartig via Outages wrote: > > Anyone else getting: > > Sorry, something went wrong. > We're working on it and we'll get it fixed as soon as we can. > > when you go to Facebook? > -- > James Hartig > > > _______________________________________________ > Outages mailing listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages > > > > ------------------------------ > > <http://www.telkonet.com> <http://www.ethostream.com> > > This e-mail and any files transmitted with it are private and > confidential and are solely for the use of the addressee. It may contain > material which is legally privileged. If you are not the addressee or the > person responsible for delivering to the addressee, be advised that you > have received this e-mail in error and that any use of it is strictly > prohibited. If you have received this e-mail in error please contact us at (414) > 223-0473. > > _______________________________________________ > Outages mailing list > Outages@outages.org > https://puck.nether.net/mailman/listinfo/outages > >
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
or 5 years ago? "By Robert Johnson on Thursday, 23 September 2010 at 20:29" On Tue, Sep 29, 2015 at 2:02 PM William Byrd via Outages < outages@outages.org> wrote:
That was actually from the previous Outage from last week.
William Collier-Byrd will@collier-byrd.net
On Tue, Sep 29, 2015 at 10:44 AM, Scott Voll <svoll.voip@gmail.com> wrote:
https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-o...
Guess it's already out there.
Scott
On Tue, Sep 29, 2015 at 9:36 AM, William Byrd via Outages < outages@outages.org> wrote:
Hey Rafael -
I am sure it would make a very interesting post mortem post, and I will ask if anyone would be willing to put something together to share externally. Right now I can't share any details, so I'll just have to apologize (again) for the inconvenience and see if we can get something posted.
William Collier-Byrd will@collier-byrd.net
On Tue, Sep 29, 2015 at 8:25 AM, Rafael Possamai < rafael@e2wsolutions.com> wrote:
Hi Will,
Are you able to disclose any high level details of what happened? It's just interesting to know how such a large infrastructure, with assumed redundancies, can go down completely. I know AWS had a similar outage, which lasted even longer, but it was a series of cascading problems due to human error. Was it an SDN bad config push?
On Mon, Sep 28, 2015 at 2:59 PM, William Collier-Byrd via Outages < outages@outages.org> wrote:
I am starting to feel like a broken record, but we're aware and working on it.
-- Will Collier-Byrd
On Mon, Sep 28, 2015 at 12:56 PM -0700, "Mike Phipps via Outages" < outages@outages.org> wrote:
Confirmed on T-Mobile LTE and Comcast Business Class in metro Detroit.
It is that same message.
On Mon, Sep 28, 2015 at 3:27 PM, Bob Colon via Outages < outages@outages.org> wrote:
> Yup....getting the same here, > > To ensure prompt service, please use "Reply All" when responding > > Robert Colon | TCSA Tier 3 | EthoStream LLC | P: (877) 282-2519 > x2114 | F: (414) 258-8307 | 20800 Swenson Drive, Suite 175 > Waukesha, WI 53186 | <http://www.telkonet.com/>www.telkonet.com | > www.ethostream.com | @Telkonet <https://twitter.com/telkonet> > > On 9/28/2015 2:23 PM, James Hartig via Outages wrote: > > Anyone else getting: > > Sorry, something went wrong. > We're working on it and we'll get it fixed as soon as we can. > > when you go to Facebook? > -- > James Hartig > > > _______________________________________________ > Outages mailing listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages > > > > ------------------------------ > > <http://www.telkonet.com> <http://www.ethostream.com> > > This e-mail and any files transmitted with it are private and > confidential and are solely for the use of the addressee. It may contain > material which is legally privileged. If you are not the addressee or the > person responsible for delivering to the addressee, be advised that you > have received this e-mail in error and that any use of it is strictly > prohibited. If you have received this e-mail in error please contact us at (414) > 223-0473. > > _______________________________________________ > Outages mailing list > Outages@outages.org > https://puck.nether.net/mailman/listinfo/outages > >
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Thats from September 2010, a bit old i say. Erik Sundberg Sr. Network Engineer Nitel 1101 West Lake Street,6th Fl Chicago, IL 60607 Desk: 773-661-5532 Cell: 708-710-7419 NOC 24/7: 866-892-0915 Email: esundberg@nitelusa.com<mailto:esundberg@nitelusa.com> http://www.nitelusa.com On Sep 29, 2015, at 12:50 PM, Scott Voll via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-o... Guess it's already out there. Scott On Tue, Sep 29, 2015 at 9:36 AM, William Byrd via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Hey Rafael - I am sure it would make a very interesting post mortem post, and I will ask if anyone would be willing to put something together to share externally. Right now I can't share any details, so I'll just have to apologize (again) for the inconvenience and see if we can get something posted. William Collier-Byrd will@collier-byrd.net<mailto:will@collier-byrd.net> On Tue, Sep 29, 2015 at 8:25 AM, Rafael Possamai <rafael@e2wsolutions.com<mailto:rafael@e2wsolutions.com>> wrote: Hi Will, Are you able to disclose any high level details of what happened? It's just interesting to know how such a large infrastructure, with assumed redundancies, can go down completely. I know AWS had a similar outage, which lasted even longer, but it was a series of cascading problems due to human error. Was it an SDN bad config push? On Mon, Sep 28, 2015 at 2:59 PM, William Collier-Byrd via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: I am starting to feel like a broken record, but we're aware and working on it. -- Will Collier-Byrd On Mon, Sep 28, 2015 at 12:56 PM -0700, "Mike Phipps via Outages" <outages@outages.org<mailto:outages@outages.org>> wrote: Confirmed on T-Mobile LTE and Comcast Business Class in metro Detroit. It is that same message. On Mon, Sep 28, 2015 at 3:27 PM, Bob Colon via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Yup....getting the same here, To ensure prompt service, please use "Reply All" when responding Robert Colon | TCSA Tier 3 | EthoStream LLC | P: (877) 282-2519 x2114<tel:%28877%29%20282-2519%20x2114> | F: (414) 258-8307<tel:%28414%29%20258-8307> | 20800 Swenson Drive, Suite 175 Waukesha, WI 53186 | <http://www.telkonet.com/> www.telkonet.com<http://www.telkonet.com> | www.ethostream.com<http://www.ethostream.com/> | @Telkonet<https://twitter.com/telkonet> On 9/28/2015 2:23 PM, James Hartig via Outages wrote: Anyone else getting: Sorry, something went wrong. We're working on it and we'll get it fixed as soon as we can. when you go to Facebook? -- James Hartig _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages ________________________________ [http://www.telkonet.com/images/signature/telkonetlogo.gif][http://www.telkonet.com/images/signature/telkosig.gif]<http://www.telkonet.com>[http://www.ethostream.com/images/signature/ethosig.gif]<http://www.ethostream.com> This e-mail and any files transmitted with it are private and confidential and are solely for the use of the addressee. It may contain material which is legally privileged. If you are not the addressee or the person responsible for delivering to the addressee, be advised that you have received this e-mail in error and that any use of it is strictly prohibited. If you have received this e-mail in error please contact us at (414) 223-0473<tel:%28414%29%20223-0473>. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
That post is from an outage back in 2010, not the outage from yesterday... Signature To ensure prompt service, please use "Reply All" when responding Robert Colon | TCSA Tier 3 | EthoStream LLC | P: (877) 282-2519 x2114 | F: (414) 258-8307 | 20800 Swenson Drive, Suite 175 Waukesha, WI 53186 | www.telkonet.com <http://www.telkonet.com/> | www.ethostream.com <http://www.ethostream.com/> | @Telkonet <https://twitter.com/telkonet> On 9/29/2015 12:44 PM, Scott Voll via Outages wrote:
https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-o...
Guess it's already out there.
Scott
On Tue, Sep 29, 2015 at 9:36 AM, William Byrd via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Hey Rafael -
I am sure it would make a very interesting post mortem post, and I will ask if anyone would be willing to put something together to share externally. Right now I can't share any details, so I'll just have to apologize (again) for the inconvenience and see if we can get something posted.
William Collier-Byrd will@collier-byrd.net <mailto:will@collier-byrd.net>
On Tue, Sep 29, 2015 at 8:25 AM, Rafael Possamai <rafael@e2wsolutions.com <mailto:rafael@e2wsolutions.com>> wrote:
Hi Will,
Are you able to disclose any high level details of what happened? It's just interesting to know how such a large infrastructure, with assumed redundancies, can go down completely. I know AWS had a similar outage, which lasted even longer, but it was a series of cascading problems due to human error. Was it an SDN bad config push?
On Mon, Sep 28, 2015 at 2:59 PM, William Collier-Byrd via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
I am starting to feel like a broken record, but we're aware and working on it.
-- Will Collier-Byrd
On Mon, Sep 28, 2015 at 12:56 PM -0700, "Mike Phipps via Outages" <outages@outages.org <mailto:outages@outages.org>> wrote:
Confirmed on T-Mobile LTE and Comcast Business Class in metro Detroit. It is that same message.
On Mon, Sep 28, 2015 at 3:27 PM, Bob Colon via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Yup....getting the same here,
To ensure prompt service, please use "Reply All" when responding
Robert Colon | TCSA Tier 3 | EthoStream LLC | P: (877) 282-2519 x2114 <tel:%28877%29%20282-2519%20x2114> | F: (414) 258-8307 <tel:%28414%29%20258-8307> | 20800 Swenson Drive, Suite 175 Waukesha, WI 53186 | www.telkonet.com <http://www.telkonet.com> | www.ethostream.com <http://www.ethostream.com/> | @Telkonet <https://twitter.com/telkonet>
On 9/28/2015 2:23 PM, James Hartig via Outages wrote:
Anyone else getting:
Sorry, something went wrong. We're working on it and we'll get it fixed as soon as we can.
when you go to Facebook? -- James Hartig
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
------------------------------------------------------------------------
<http://www.telkonet.com><http://www.ethostream.com>
This e-mail and any files transmitted with it are private and confidential and are solely for the use of the addressee. It may contain material which is legally privileged. If you are not the addressee or the person responsible for delivering to the addressee, be advised that you have received this e-mail in error and that any use of it is strictly prohibited. If you have received this e-mail in error please contact us at (414) 223-0473 <tel:%28414%29%20223-0473>.
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- ------------------------------ <http://www.telkonet.com> <http://www.ethostream.com> This e-mail and any files transmitted with it are private and confidential and are solely for the use of the addressee. It may contain material which is legally privileged. If you are not the addressee or the person responsible for delivering to the addressee, be advised that you have received this e-mail in error and that any use of it is strictly prohibited. If you have received this e-mail in error please contact us at (414) 223-0473.
participants (6)
-
Benjamin Monjay -
Bob Colon -
Erik Sundberg -
Scott Voll -
Wayne Egerer -
William Byrd