Atlanta 56 Marietta - MMR Power issue
Looks like there is a power issue in the DR/Telx 56 Marietta facility. Just happened about 7:41 CST. No update as of yet. Erich Kaiser The Fusion Network
Having an issue in the same region. We lost connectivity around 12:36 UTC. Trying to see what’s happening. Kindest Regards, Andrey Leo www.cogecodata.com<http://www.cogecodata.com/> | www.peer1.com<http://www.peer1.com/> From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Erich Kaiser via Outages Sent: 12 July 2017 13:56 To: outages@outages.org Subject: [outages] Atlanta 56 Marietta - MMR Power issue Looks like there is a power issue in the DR/Telx 56 Marietta facility. Just happened about 7:41 CST. No update as of yet. Erich Kaiser The Fusion Network
Saw our two circuits to 2nd floor and 4th floor go down at ~8:36:50 EDT and the circuit to 2nd floor still hasn't come back up. *Alex Forster* Network Engineer, Linode On Wed, Jul 12, 2017 at 8:55 AM, Erich Kaiser via Outages < outages@outages.org> wrote:
Looks like there is a power issue in the DR/Telx 56 Marietta facility. Just happened about 7:41 CST.
No update as of yet.
Erich Kaiser The Fusion Network
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Once upon a time, Erich Kaiser via Outages <outages@outages.org> said:
Looks like there is a power issue in the DR/Telx 56 Marietta facility. Just happened about 7:41 CST.
Our cabinet on the 6th floor appears to have lost power (DC and AC) briefly between 08:36:54 (last remote log message) and 08:38:58 (first device booted) EDT. -- Chris Adams <cma@cmadams.net>
We lost something in Telx, but not in Equinix. Device came back, but Telx IX ports still down. Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype) https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:04, Chris Adams via Outages <outages@outages.org> wrote:
Once upon a time, Erich Kaiser via Outages <outages@outages.org> said:
Looks like there is a power issue in the DR/Telx 56 Marietta facility. Just happened about 7:41 CST.
Our cabinet on the 6th floor appears to have lost power (DC and AC) briefly between 08:36:54 (last remote log message) and 08:38:58 (first device booted) EDT.
-- Chris Adams <cma@cmadams.net> _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We also lost connectivity to some of our sip providers, our gear is on the 7th floor not sure where they are. Sent from my iPad On Jul 12, 2017, at 8:11 AM, Marty Strong via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: We lost something in Telx, but not in Equinix. Device came back, but Telx IX ports still down. Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com<mailto:marty@cloudflare.com> +44 7584 906 055 smartflare (Skype) https://www.peeringdb.com/asn/13335 On 12 Jul 2017, at 14:04, Chris Adams via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Once upon a time, Erich Kaiser via Outages <outages@outages.org<mailto:outages@outages.org>> said: Looks like there is a power issue in the DR/Telx 56 Marietta facility. Just happened about 7:41 CST. Our cabinet on the 6th floor appears to have lost power (DC and AC) briefly between 08:36:54 (last remote log message) and 08:38:58 (first device booted) EDT. -- Chris Adams <cma@cmadams.net<mailto:cma@cmadams.net>> _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
I'm guessing they just now transferred back to grid power. *Alex Forster* Network Engineer, Linode On Wed, Jul 12, 2017 at 9:12 AM, John Bergoon via Outages < outages@outages.org> wrote:
We also lost connectivity to some of our sip providers, our gear is on the 7th floor not sure where they are.
Sent from my iPad
On Jul 12, 2017, at 8:11 AM, Marty Strong via Outages <outages@outages.org> wrote:
We lost something in Telx, but not in Equinix.
Device came back, but Telx IX ports still down.
Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 <+44%207584%20906055> smartflare (Skype)
https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:04, Chris Adams via Outages <outages@outages.org> wrote:
Once upon a time, Erich Kaiser via Outages <outages@outages.org> said:
Looks like there is a power issue in the DR/Telx 56 Marietta facility.
Just happened about 7:41 CST.
Our cabinet on the 6th floor appears to have lost power (DC and AC)
briefly between 08:36:54 (last remote log message) and 08:38:58 (first
device booted) EDT.
--
Chris Adams <cma@cmadams.net>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We are a provider on the 8th floor no power issues but logs show customers links down On Wed, Jul 12, 2017 at 9:13 AM John Bergoon via Outages < outages@outages.org> wrote:
We also lost connectivity to some of our sip providers, our gear is on the 7th floor not sure where they are.
Sent from my iPad
On Jul 12, 2017, at 8:11 AM, Marty Strong via Outages <outages@outages.org> wrote:
We lost something in Telx, but not in Equinix.
Device came back, but Telx IX ports still down.
Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype)
https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:04, Chris Adams via Outages <outages@outages.org> wrote:
Once upon a time, Erich Kaiser via Outages <outages@outages.org> said:
Looks like there is a power issue in the DR/Telx 56 Marietta facility.
Just happened about 7:41 CST.
Our cabinet on the 6th floor appears to have lost power (DC and AC)
briefly between 08:36:54 (last remote log message) and 08:38:58 (first
device booted) EDT.
--
Chris Adams <cma@cmadams.net>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We're on two ... just coming up now. On 7/12/17 8:15 AM, Jamie Stephens via Outages wrote:
We are a provider on the 8th floor no power issues but logs show customers links down On Wed, Jul 12, 2017 at 9:13 AM John Bergoon via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We also lost connectivity to some of our sip providers, our gear is on the 7th floor not sure where they are.
Sent from my iPad
On Jul 12, 2017, at 8:11 AM, Marty Strong via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We lost something in Telx, but not in Equinix.
Device came back, but Telx IX ports still down.
Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com <mailto:marty@cloudflare.com> +44 7584 906 055 smartflare (Skype)
https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:04, Chris Adams via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Once upon a time, Erich Kaiser via Outages <outages@outages.org <mailto:outages@outages.org>> said:
Looks like there is a power issue in the DR/Telx 56 Marietta facility. Just happened about 7:41 CST.
Our cabinet on the 6th floor appears to have lost power (DC and AC) briefly between 08:36:54 (last remote log message) and 08:38:58 (first device booted) EDT.
-- Chris Adams <cma@cmadams.net <mailto:cma@cmadams.net>> _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Looks like everything is coming back up now. No explanation just yet. Andrey Leo Network Analyst I T(US) 1-866-484-2588 | T(UK) 0800-840-7491 30-31 Town Quay Southampton, Hampshire SO14 2AQ UK cogecopeer1.com<http://www.cogecopeer1.com/> [cid:image001.png@01D2FB1A.1B95A880]<http://www.cogecopeer1.com/> From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Jamie Stephens via Outages Sent: 12 July 2017 14:16 To: John Bergoon; outages@outages.org; Marty Strong Subject: Re: [outages] Atlanta 56 Marietta - MMR Power issue We are a provider on the 8th floor no power issues but logs show customers links down On Wed, Jul 12, 2017 at 9:13 AM John Bergoon via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: We also lost connectivity to some of our sip providers, our gear is on the 7th floor not sure where they are. Sent from my iPad On Jul 12, 2017, at 8:11 AM, Marty Strong via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: We lost something in Telx, but not in Equinix. Device came back, but Telx IX ports still down. Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com<mailto:marty@cloudflare.com> +44 7584 906 055 smartflare (Skype) https://www.peeringdb.com/asn/13335 On 12 Jul 2017, at 14:04, Chris Adams via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Once upon a time, Erich Kaiser via Outages <outages@outages.org<mailto:outages@outages.org>> said: Looks like there is a power issue in the DR/Telx 56 Marietta facility. Just happened about 7:41 CST. Our cabinet on the 6th floor appears to have lost power (DC and AC) briefly between 08:36:54 (last remote log message) and 08:38:58 (first device booted) EDT. -- Chris Adams <cma@cmadams.net<mailto:cma@cmadams.net>> _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Telx IX ports came back up. Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype) https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:21, NSC Global <nsc.global@cogecopeer1.com> wrote:
Looks like everything is coming back up now.
No explanation just yet.
ANDREY LEO Network Analyst I
T(US) 1-866-484-2588 | T(UK) 0800-840-7491 30-31 Town Quay Southampton, Hampshire SO14 2AQ UK cogecopeer1.com <image001.png>
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Jamie Stephens via Outages Sent: 12 July 2017 14:16 To: John Bergoon; outages@outages.org; Marty Strong Subject: Re: [outages] Atlanta 56 Marietta - MMR Power issue
We are a provider on the 8th floor no power issues but logs show customers links down On Wed, Jul 12, 2017 at 9:13 AM John Bergoon via Outages <outages@outages.org> wrote: We also lost connectivity to some of our sip providers, our gear is on the 7th floor not sure where they are.
Sent from my iPad
On Jul 12, 2017, at 8:11 AM, Marty Strong via Outages <outages@outages.org> wrote:
We lost something in Telx, but not in Equinix.
Device came back, but Telx IX ports still down.
Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype)
https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:04, Chris Adams via Outages <outages@outages.org> wrote:
Once upon a time, Erich Kaiser via Outages <outages@outages.org> said: Looks like there is a power issue in the DR/Telx 56 Marietta facility. Just happened about 7:41 CST.
Our cabinet on the 6th floor appears to have lost power (DC and AC) briefly between 08:36:54 (last remote log message) and 08:38:58 (first device booted) EDT.
-- Chris Adams <cma@cmadams.net> _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We came back up about a minute ago. NSC Global via Outages wrote:
Looks like everything is coming back up now.
No explanation just yet.
*Andrey Leo*
Network Analyst I
*T(US)* *1-866-484-2588* | T(UK) *0800-840-7491*
30-31 Town Quay Southampton, Hampshire SO14 2AQ UK *cogecopeer1.com* <http://www.cogecopeer1.com/>
*From:*Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Jamie Stephens via Outages *Sent:* 12 July 2017 14:16 *To:* John Bergoon; outages@outages.org; Marty Strong *Subject:* Re: [outages] Atlanta 56 Marietta - MMR Power issue
We are a provider on the 8th floor no power issues but logs show customers links down
On Wed, Jul 12, 2017 at 9:13 AM John Bergoon via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We also lost connectivity to some of our sip providers, our gear is on the 7th floor not sure where they are.
Sent from my iPad
On Jul 12, 2017, at 8:11 AM, Marty Strong via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We lost something in Telx, but not in Equinix.
Device came back, but Telx IX ports still down.
Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com <mailto:marty@cloudflare.com> +44 7584 906 055 smartflare (Skype)
https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:04, Chris Adams via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Once upon a time, Erich Kaiser via Outages <outages@outages.org <mailto:outages@outages.org>> said:
Looks like there is a power issue in the DR/Telx 56 Marietta facility.
Just happened about 7:41 CST.
Our cabinet on the 6th floor appears to have lost power (DC and AC)
briefly between 08:36:54 (last remote log message) and 08:38:58 (first
device booted) EDT.
--
Chris Adams <cma@cmadams.net <mailto:cma@cmadams.net>>
_______________________________________________
Outages mailing list
Outages@outages.org <mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free. www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Received this from DRT
RED Event - Generator Failure / Critical Load Impact 56 Marietta Atlanta GA
During scheduled maintenance, the engineering team at 56 Marietta reports that the generator failed resulting in a loss of critical IT load to equipment in the space. The site team is investigating the source of the issue and scope of the impact. Further updates will be provided as additional information becomes available.
Thank you, Digital Realty Operations
Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype) https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:23, Jay Weekley via Outages <outages@outages.org> wrote:
We came back up about a minute ago.
NSC Global via Outages wrote:
Looks like everything is coming back up now.
No explanation just yet.
*Andrey Leo*
Network Analyst I
*T(US)* *1-866-484-2588* | T(UK) *0800-840-7491*
30-31 Town Quay Southampton, Hampshire SO14 2AQ UK *cogecopeer1.com* <http://www.cogecopeer1.com/>
*From:*Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Jamie Stephens via Outages *Sent:* 12 July 2017 14:16 *To:* John Bergoon; outages@outages.org; Marty Strong *Subject:* Re: [outages] Atlanta 56 Marietta - MMR Power issue
We are a provider on the 8th floor no power issues but logs show customers links down
On Wed, Jul 12, 2017 at 9:13 AM John Bergoon via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We also lost connectivity to some of our sip providers, our gear is on the 7th floor not sure where they are.
Sent from my iPad
On Jul 12, 2017, at 8:11 AM, Marty Strong via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We lost something in Telx, but not in Equinix.
Device came back, but Telx IX ports still down.
Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com <mailto:marty@cloudflare.com> +44 7584 906 055 smartflare (Skype)
https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:04, Chris Adams via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Once upon a time, Erich Kaiser via Outages <outages@outages.org <mailto:outages@outages.org>> said:
Looks like there is a power issue in the DR/Telx 56 Marietta facility.
Just happened about 7:41 CST.
Our cabinet on the 6th floor appears to have lost power (DC and AC)
briefly between 08:36:54 (last remote log message) and 08:38:58 (first
device booted) EDT.
-- Chris Adams <cma@cmadams.net <mailto:cma@cmadams.net>>
_______________________________________________
Outages mailing list
Outages@outages.org <mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free. www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Ok. But why was there no battery backup?
On Jul 12, 2017, at 9:33 AM, Marty Strong via Outages <outages@outages.org> wrote:
Received this from DRT
RED Event - Generator Failure / Critical Load Impact 56 Marietta Atlanta GA
During scheduled maintenance, the engineering team at 56 Marietta reports that the generator failed resulting in a loss of critical IT load to equipment in the space. The site team is investigating the source of the issue and scope of the impact. Further updates will be provided as additional information becomes available.
Thank you, Digital Realty Operations
Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype)
https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:23, Jay Weekley via Outages <outages@outages.org> wrote:
We came back up about a minute ago.
NSC Global via Outages wrote:
Looks like everything is coming back up now.
No explanation just yet.
*Andrey Leo*
Network Analyst I
*T(US)* *1-866-484-2588* | T(UK) *0800-840-7491*
30-31 Town Quay Southampton, Hampshire SO14 2AQ UK *cogecopeer1.com* <http://www.cogecopeer1.com/>
*From:*Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Jamie Stephens via Outages *Sent:* 12 July 2017 14:16 *To:* John Bergoon; outages@outages.org; Marty Strong *Subject:* Re: [outages] Atlanta 56 Marietta - MMR Power issue
We are a provider on the 8th floor no power issues but logs show customers links down
On Wed, Jul 12, 2017 at 9:13 AM John Bergoon via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We also lost connectivity to some of our sip providers, our gear is on the 7th floor not sure where they are.
Sent from my iPad
On Jul 12, 2017, at 8:11 AM, Marty Strong via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We lost something in Telx, but not in Equinix.
Device came back, but Telx IX ports still down.
Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com <mailto:marty@cloudflare.com> +44 7584 906 055 smartflare (Skype)
https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:04, Chris Adams via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Once upon a time, Erich Kaiser via Outages <outages@outages.org <mailto:outages@outages.org>> said:
Looks like there is a power issue in the DR/Telx 56 Marietta facility.
Just happened about 7:41 CST.
Our cabinet on the 6th floor appears to have lost power (DC and AC)
briefly between 08:36:54 (last remote log message) and 08:38:58 (first
device booted) EDT.
-- Chris Adams <cma@cmadams.net <mailto:cma@cmadams.net>>
_______________________________________________
Outages mailing list
Outages@outages.org <mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free. www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
1 Telx route server still down: Route server connections dropped for sure in the tie network: 198.32.132.5 32299 0 0 0 16 56:04 Connect 198.32.132.6 32299 3553 37 0 14 15:56 Establ inet.0: 8364/12500/12500/0 -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Matt Hoppes via Outages Sent: Wednesday, July 12, 2017 8:35 AM To: Marty Strong <marty@cloudflare.com> Cc: outages@outages.org Subject: Re: [outages] Atlanta 56 Marietta - MMR Power issue Ok. But why was there no battery backup?
On Jul 12, 2017, at 9:33 AM, Marty Strong via Outages <outages@outages.org> wrote:
Received this from DRT
RED Event - Generator Failure / Critical Load Impact 56 Marietta Atlanta GA
During scheduled maintenance, the engineering team at 56 Marietta reports that the generator failed resulting in a loss of critical IT load to equipment in the space. The site team is investigating the source of the issue and scope of the impact. Further updates will be provided as additional information becomes available.
Thank you, Digital Realty Operations
Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype)
https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:23, Jay Weekley via Outages <outages@outages.org> wrote:
We came back up about a minute ago.
NSC Global via Outages wrote:
Looks like everything is coming back up now.
No explanation just yet.
*Andrey Leo*
Network Analyst I
*T(US)* *1-866-484-2588* | T(UK) *0800-840-7491*
30-31 Town Quay Southampton, Hampshire SO14 2AQ UK *cogecopeer1.com* <http://www.cogecopeer1.com/>
*From:*Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Jamie Stephens via Outages *Sent:* 12 July 2017 14:16 *To:* John Bergoon; outages@outages.org; Marty Strong *Subject:* Re: [outages] Atlanta 56 Marietta - MMR Power issue
We are a provider on the 8th floor no power issues but logs show customers links down
On Wed, Jul 12, 2017 at 9:13 AM John Bergoon via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We also lost connectivity to some of our sip providers, our gear is on the 7th floor not sure where they are.
Sent from my iPad
On Jul 12, 2017, at 8:11 AM, Marty Strong via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We lost something in Telx, but not in Equinix.
Device came back, but Telx IX ports still down.
Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com <mailto:marty@cloudflare.com> +44 7584 906 055 smartflare (Skype)
https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:04, Chris Adams via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Once upon a time, Erich Kaiser via Outages <outages@outages.org <mailto:outages@outages.org>> said:
Looks like there is a power issue in the DR/Telx 56 Marietta facility.
Just happened about 7:41 CST.
Our cabinet on the 6th floor appears to have lost power (DC and AC)
briefly between 08:36:54 (last remote log message) and 08:38:58 (first
device booted) EDT.
-- Chris Adams <cma@cmadams.net <mailto:cma@cmadams.net>>
_______________________________________________
Outages mailing list
Outages@outages.org <mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free. www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Just came back for us ... On 7/12/17 8:39 AM, Kody Vicknair via Outages wrote:
1 Telx route server still down:
Route server connections dropped for sure in the tie network:
198.32.132.5 32299 0 0 0 16 56:04 Connect 198.32.132.6 32299 3553 37 0 14 15:56 Establ inet.0: 8364/12500/12500/0
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Matt Hoppes via Outages Sent: Wednesday, July 12, 2017 8:35 AM To: Marty Strong <marty@cloudflare.com> Cc: outages@outages.org Subject: Re: [outages] Atlanta 56 Marietta - MMR Power issue
Ok. But why was there no battery backup?
On Jul 12, 2017, at 9:33 AM, Marty Strong via Outages <outages@outages.org> wrote:
Received this from DRT
RED Event - Generator Failure / Critical Load Impact 56 Marietta Atlanta GA
During scheduled maintenance, the engineering team at 56 Marietta reports that the generator failed resulting in a loss of critical IT load to equipment in the space. The site team is investigating the source of the issue and scope of the impact. Further updates will be provided as additional information becomes available.
Thank you, Digital Realty Operations
Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype)
https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:23, Jay Weekley via Outages <outages@outages.org> wrote:
We came back up about a minute ago.
NSC Global via Outages wrote:
Looks like everything is coming back up now.
No explanation just yet.
*Andrey Leo*
Network Analyst I
*T(US)* *1-866-484-2588* | T(UK) *0800-840-7491*
30-31 Town Quay Southampton, Hampshire SO14 2AQ UK *cogecopeer1.com* <http://www.cogecopeer1.com/>
*From:*Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Jamie Stephens via Outages *Sent:* 12 July 2017 14:16 *To:* John Bergoon; outages@outages.org; Marty Strong *Subject:* Re: [outages] Atlanta 56 Marietta - MMR Power issue
We are a provider on the 8th floor no power issues but logs show customers links down
On Wed, Jul 12, 2017 at 9:13 AM John Bergoon via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We also lost connectivity to some of our sip providers, our gear is on the 7th floor not sure where they are.
Sent from my iPad
On Jul 12, 2017, at 8:11 AM, Marty Strong via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We lost something in Telx, but not in Equinix.
Device came back, but Telx IX ports still down.
Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com <mailto:marty@cloudflare.com> +44 7584 906 055 smartflare (Skype)
https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:04, Chris Adams via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Once upon a time, Erich Kaiser via Outages <outages@outages.org <mailto:outages@outages.org>> said:
Looks like there is a power issue in the DR/Telx 56 Marietta facility.
Just happened about 7:41 CST.
Our cabinet on the 6th floor appears to have lost power (DC and AC)
briefly between 08:36:54 (last remote log message) and 08:38:58 (first
device booted) EDT.
-- Chris Adams <cma@cmadams.net <mailto:cma@cmadams.net>>
_______________________________________________
Outages mailing list
Outages@outages.org <mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free. www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Guaranteed there is battery, but you would have to know more about the type of maintenance they were doing. Maybe they were working on the UPS and having to run on generator. Tom Sands Principal Architect Rackspace 1-800-961-4454 On 7/12/17 8:35 AM, Matt Hoppes via Outages wrote:
Ok. But why was there no battery backup?
On Jul 12, 2017, at 9:33 AM, Marty Strong via Outages <outages@outages.org> wrote:
Received this from DRT
RED Event - Generator Failure / Critical Load Impact 56 Marietta Atlanta GA
During scheduled maintenance, the engineering team at 56 Marietta reports that the generator failed resulting in a loss of critical IT load to equipment in the space. The site team is investigating the source of the issue and scope of the impact. Further updates will be provided as additional information becomes available.
Thank you, Digital Realty Operations Regards, Marty Strong
Cloudflare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype)
https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:23, Jay Weekley via Outages <outages@outages.org> wrote:
We came back up about a minute ago.
NSC Global via Outages wrote:
Looks like everything is coming back up now.
No explanation just yet.
*Andrey Leo*
Network Analyst I
*T(US)* *1-866-484-2588* | T(UK) *0800-840-7491*
30-31 Town Quay Southampton, Hampshire SO14 2AQ UK *cogecopeer1.com* <http://www.cogecopeer1.com/>
*From:*Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Jamie Stephens via Outages *Sent:* 12 July 2017 14:16 *To:* John Bergoon; outages@outages.org; Marty Strong *Subject:* Re: [outages] Atlanta 56 Marietta - MMR Power issue
We are a provider on the 8th floor no power issues but logs show customers links down
On Wed, Jul 12, 2017 at 9:13 AM John Bergoon via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We also lost connectivity to some of our sip providers, our gear is on the 7th floor not sure where they are.
Sent from my iPad
On Jul 12, 2017, at 8:11 AM, Marty Strong via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We lost something in Telx, but not in Equinix.
Device came back, but Telx IX ports still down.
Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com <mailto:marty@cloudflare.com> +44 7584 906 055 smartflare (Skype)
https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:04, Chris Adams via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Once upon a time, Erich Kaiser via Outages <outages@outages.org <mailto:outages@outages.org>> said:
Looks like there is a power issue in the DR/Telx 56 Marietta facility.
Just happened about 7:41 CST.
Our cabinet on the 6th floor appears to have lost power (DC and AC)
briefly between 08:36:54 (last remote log message) and 08:38:58 (first
device booted) EDT.
-- Chris Adams <cma@cmadams.net <mailto:cma@cmadams.net>>
_______________________________________________
Outages mailing list
Outages@outages.org <mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free. www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
My biggest concern is we lost both A and B power. Which I would expect to be on separate "Power Grids / Infrastructure" Here was the maintenance notice sent out. July 12, 2017 MAINTENANCE NOTIFICATION - 15-MINUTE-REMINDER - MTST# 00443066 - ATL1 Valued Customer, Digital Realty engineers will be performing maintenance to add capacity to existing PDUs (Power Distribution Units) at our ATL1 (56 Marietta Street, Atlanta, GA) facility on the dates/times shown below. All work associated should be seamless and your systems should not see any loss of power. As with any maintenance we do not expect any power outages, but there is always a chance that critical systems could be impacted during these procedures. Please ensure that if you subscribe to redundant power sources that your equipment is properly distributed. Maintenance TST# 00443066 Location: ATL1 (56 Marietta, Atlanta) Floor(s) / Suite(s): Atlanta Sub-Basement;Atlanta Basement;Atlanta 1st FL;Atlanta 2nd FL;Atlanta 3rd FL;Atlanta 4th FL;Atlanta 7th FL;Atlanta 8th FL;Atlanta 9th FL;Atlanta 10th FL Maintenance Type: Planned Maintenance Priority: YELLOW Event - PSA (Potentially Service Affecting) Product(s)/Asset(s) Affected: Electrical Distribution Description: Critical Power Redistribution Phase 1-PDU Source Selector Feeds Upgrade Start Date/Time: 07/12/2017 07:00 AM EDT End Date/Time: 07/13/2017 07:00 PM EDT Duration: 7am - 7pm Daily Related Work Orders: Related Circuit IDs: Please refer any questions to: The Global Command Center at gcc@digitalrealty.com or 888-835-9832 Erik Sundberg Sr. Network Engineering Network Engineering Department p: 773.661.5532 c: 708.710.7419 e: esundberg@nitelusa.com Main: 888.450.2100 NOC 24/7: 866.892.0915 350 North Orleans Street, Suite 1300N Chicago, IL 60654 www.nitelusa.com Managed Telecom Services MPLS | Ethernet | Private Line | Internet | Voice | Security -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Matt Hoppes via Outages Sent: Wednesday, July 12, 2017 8:35 AM To: Marty Strong Cc: outages@outages.org Subject: Re: [outages] Atlanta 56 Marietta - MMR Power issue Ok. But why was there no battery backup?
On Jul 12, 2017, at 9:33 AM, Marty Strong via Outages <outages@outages.org> wrote:
Received this from DRT
RED Event - Generator Failure / Critical Load Impact 56 Marietta Atlanta GA
During scheduled maintenance, the engineering team at 56 Marietta reports that the generator failed resulting in a loss of critical IT load to equipment in the space. The site team is investigating the source of the issue and scope of the impact. Further updates will be provided as additional information becomes available.
Thank you, Digital Realty Operations
Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype)
https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:23, Jay Weekley via Outages <outages@outages.org> wrote:
We came back up about a minute ago.
NSC Global via Outages wrote:
Looks like everything is coming back up now.
No explanation just yet.
*Andrey Leo*
Network Analyst I
*T(US)* *1-866-484-2588* | T(UK) *0800-840-7491*
30-31 Town Quay Southampton, Hampshire SO14 2AQ UK *cogecopeer1.com* <http://www.cogecopeer1.com/>
*From:*Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Jamie Stephens via Outages *Sent:* 12 July 2017 14:16 *To:* John Bergoon; outages@outages.org; Marty Strong *Subject:* Re: [outages] Atlanta 56 Marietta - MMR Power issue
We are a provider on the 8th floor no power issues but logs show customers links down
On Wed, Jul 12, 2017 at 9:13 AM John Bergoon via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We also lost connectivity to some of our sip providers, our gear is on the 7th floor not sure where they are.
Sent from my iPad
On Jul 12, 2017, at 8:11 AM, Marty Strong via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We lost something in Telx, but not in Equinix.
Device came back, but Telx IX ports still down.
Regards, Marty Strong -------------------------------------- Cloudflare - AS13335 Network Engineer marty@cloudflare.com <mailto:marty@cloudflare.com> +44 7584 906 055 smartflare (Skype)
https://www.peeringdb.com/asn/13335
On 12 Jul 2017, at 14:04, Chris Adams via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Once upon a time, Erich Kaiser via Outages <outages@outages.org <mailto:outages@outages.org>> said:
Looks like there is a power issue in the DR/Telx 56 Marietta facility.
Just happened about 7:41 CST.
Our cabinet on the 6th floor appears to have lost power (DC and AC)
briefly between 08:36:54 (last remote log message) and 08:38:58 (first
device booted) EDT.
-- Chris Adams <cma@cmadams.net <mailto:cma@cmadams.net>>
_______________________________________________
Outages mailing list
Outages@outages.org <mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free. www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
Once upon a time, Erik Sundberg via Outages <outages@outages.org> said:
My biggest concern is we lost both A and B power. Which I would expect to be on separate "Power Grids / Infrastructure"
Yeah, we have both AC and DC powered equipment in our cabinet, and it appears that both failed at the same time. -- Chris Adams <cma@cmadams.net>
My guess is they were on generator during the work, and probably have STSs upstream of the PDUs and when the generator failed the sources didn't transfer to another bus. The DC system is probably feed from the same system. On Wed, Jul 12, 2017 at 9:47 AM Chris Adams via Outages <outages@outages.org> wrote:
Once upon a time, Erik Sundberg via Outages <outages@outages.org> said:
My biggest concern is we lost both A and B power. Which I would expect to be on separate "Power Grids / Infrastructure"
Yeah, we have both AC and DC powered equipment in our cabinet, and it appears that both failed at the same time. -- Chris Adams <cma@cmadams.net> _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Brendan Mannella TeraSwitch Inc. Main - 1.412.945.7045 Direct - 1.412.945.7049 eFax - 1.412.945.7049 Colocation . Cloud . Connectivity
participants (14)
-
Alex Forster -
Brendan Mannella -
Bryan Holloway -
Chris Adams -
Erich Kaiser -
Erik Sundberg -
Jamie Stephens -
Jay Weekley -
John Bergoon -
Kody Vicknair -
Marty Strong -
Matt Hoppes -
NSC Global -
Tom Sands