Discord: Sever outages and Increased API errors
Discord seems to be having issues. It is showing on Down Detector. The splash page says: The Server Status page shows: Mostly FYI. -- TTFN, patrick
We should be back now. As you can probably expect we've had a very large amount of growth recently and hit a file descriptor limit on a central system that caused some cascading failures. On Mon, Mar 16, 2020 at 2:49 PM Patrick W. Gilmore via Outages < outages@outages.org> wrote:
Discord seems to be having issues. It is showing on Down Detector. The splash page says:
The Server Status page shows:
Mostly FYI.
-- TTFN, patrick
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
If anyone is interested we published a post mortem on our status page for this incident. https://status.discordapp.com/incidents/62gt9cgjwdgf On Mon, Mar 16, 2020 at 2:49 PM Patrick W. Gilmore via Outages < outages@outages.org> wrote:
Discord seems to be having issues. It is showing on Down Detector. The splash page says:
The Server Status page shows:
Mostly FYI.
-- TTFN, patrick
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I am sufficiently interested that I'm seriously considering writing a book on this topic actually. That was a very nice write-up. Do I correctly understand, then, that the root cause was the broken golang library function / routine? (For the moment, we will pass on the topic of the irony of having to come up with an alternate name for your own language because otherwise people won't be able to find things about it... in the search engine you run that you wrote it for :-) Way to go Google.) On March 20, 2020 11:54:08 PM EDT, Gary Josack via Outages <outages@outages.org> wrote:
If anyone is interested we published a post mortem on our status page for this incident. https://status.discordapp.com/incidents/62gt9cgjwdgf
On Mon, Mar 16, 2020 at 2:49 PM Patrick W. Gilmore via Outages < outages@outages.org> wrote:
Discord seems to be having issues. It is showing on Down Detector. The splash page says:
The Server Status page shows:
Mostly FYI.
-- TTFN, patrick
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Whenever you decide to publish such a book, I’ve got your title and cover art ready... Put me down for a pre-order. -Brad [A close up of a bird Description automatically generated] From: Outages <outages-bounces@outages.org> on behalf of Jay Ashworth via Outages <outages@outages.org> Reply-To: Jay Ashworth <jra@baylink.com> Date: Friday, March 20, 2020 at 9:07 PM To: Gary Josack <gary@byoteki.com>, Gary Josack via Outages <outages@outages.org>, "Patrick W. Gilmore" <patrick@ianai.net> Cc: outages <outages@outages.org> Subject: [EXTERNAL] Re: [outages] Discord: Sever outages and Increased API errors I am sufficiently interested that I'm seriously considering writing a book on this topic actually. That was a very nice write-up. Do I correctly understand, then, that the root cause was the broken golang library function / routine? (For the moment, we will pass on the topic of the irony of having to come up with an alternate name for your own language because otherwise people won't be able to find things about it... in the search engine you run that you wrote it for :-) Way to go Google.) On March 20, 2020 11:54:08 PM EDT, Gary Josack via Outages <outages@outages.org> wrote: If anyone is interested we published a post mortem on our status page for this incident. https://status.discordapp.com/incidents/62gt9cgjwdgf<https://urldefense.com/v3/__https:/status.discordapp.com/incidents/62gt9cgjwdgf__;!!PIZeeW5wscynRQ!9IlozgcLDbWntVNJYRLbLSjlkY1RnEa2iRitGoMKZBTwtilpUibKlSbLxIv4M9w-XQ$> On Mon, Mar 16, 2020 at 2:49 PM Patrick W. Gilmore via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Discord seems to be having issues. It is showing on Down Detector. The splash page says: [https://files.slack.com/files-pri/T2F7AJRMY-FV5C2QE4A/image.png] [cid:170fb3917e5a2da56b91] The Server Status page shows: [cid:170fb3917e5774fd4222] Mostly FYI. -- TTFN, patrick _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages<https://urldefense.com/v3/__https:/puck.nether.net/mailman/listinfo/outages__;!!PIZeeW5wscynRQ!9IlozgcLDbWntVNJYRLbLSjlkY1RnEa2iRitGoMKZBTwtilpUibKlSbLxIs82I34TA$> -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Yeah, but that's an owl, with a 360-degree neck. Shouldn't it have been facing away from the camera? :-) On March 22, 2020 2:03:04 AM EDT, "Chapman, Brad (NBCUniversal)" <Brad.Chapman@nbcuni.com> wrote:
Whenever you decide to publish such a book, I’ve got your title and cover art ready...
Put me down for a pre-order.
-Brad
[A close up of a bird Description automatically generated]
From: Outages <outages-bounces@outages.org> on behalf of Jay Ashworth via Outages <outages@outages.org> Reply-To: Jay Ashworth <jra@baylink.com> Date: Friday, March 20, 2020 at 9:07 PM To: Gary Josack <gary@byoteki.com>, Gary Josack via Outages <outages@outages.org>, "Patrick W. Gilmore" <patrick@ianai.net> Cc: outages <outages@outages.org> Subject: [EXTERNAL] Re: [outages] Discord: Sever outages and Increased API errors
I am sufficiently interested that I'm seriously considering writing a book on this topic actually. That was a very nice write-up. Do I correctly understand, then, that the root cause was the broken golang library function / routine?
(For the moment, we will pass on the topic of the irony of having to come up with an alternate name for your own language because otherwise people won't be able to find things about it... in the search engine you run that you wrote it for :-) Way to go Google.) On March 20, 2020 11:54:08 PM EDT, Gary Josack via Outages <outages@outages.org> wrote: If anyone is interested we published a post mortem on our status page for this incident. https://status.discordapp.com/incidents/62gt9cgjwdgf<https://urldefense.com/v3/__https:/status.discordapp.com/incidents/62gt9cgjwdgf__;!!PIZeeW5wscynRQ!9IlozgcLDbWntVNJYRLbLSjlkY1RnEa2iRitGoMKZBTwtilpUibKlSbLxIv4M9w-XQ$>
On Mon, Mar 16, 2020 at 2:49 PM Patrick W. Gilmore via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Discord seems to be having issues. It is showing on Down Detector. The splash page says: [https://files.slack.com/files-pri/T2F7AJRMY-FV5C2QE4A/image.png] [cid:170fb3917e5a2da56b91]
The Server Status page shows: [cid:170fb3917e5774fd4222]
Mostly FYI.
-- TTFN, patrick
_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages<https://urldefense.com/v3/__https:/puck.nether.net/mailman/listinfo/outages__;!!PIZeeW5wscynRQ!9IlozgcLDbWntVNJYRLbLSjlkY1RnEa2iRitGoMKZBTwtilpUibKlSbLxIs82I34TA$>
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Nodes announce themselves to etcd by issuing an HTTP PUT, etcd client uses Python's builtin HTTP client, and sends a PUT request (along with the Content-Length header) in one packet, and the request body in another packet. We determined that the connection was reset after sending the first packet, but before the second packet could be sent.
Looks like service discovery poisoning, linked to python lib. Is it so? (Even when its not DNS, it looks very much like DNS) On Fri, Mar 20, 2020 at 11:05 PM Jay Ashworth via Outages < outages@outages.org> wrote:
I am sufficiently interested that I'm seriously considering writing a book on this topic actually. That was a very nice write-up. Do I correctly understand, then, that the root cause was the broken golang library function / routine?
(For the moment, we will pass on the topic of the irony of having to come up with an alternate name for your own language because otherwise people won't be able to find things about it... in the search engine you run that you wrote it for :-) Way to go Google.)
On March 20, 2020 11:54:08 PM EDT, Gary Josack via Outages < outages@outages.org> wrote:
If anyone is interested we published a post mortem on our status page for this incident. https://status.discordapp.com/incidents/62gt9cgjwdgf
On Mon, Mar 16, 2020 at 2:49 PM Patrick W. Gilmore via Outages < outages@outages.org> wrote:
Discord seems to be having issues. It is showing on Down Detector. The splash page says:
The Server Status page shows:
Mostly FYI.
-- TTFN, patrick
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Andrei Hurynovich Ops&IT @ Charlesworth Research, Chicago
participants (5)
-
Andrei Hurynovich -
Chapman, Brad (NBCUniversal) -
Gary Josack -
Jay Ashworth -
Patrick W. Gilmore