Re: [outages] Linode outage in Newark, NJ starting at 3:30 pm
The fault was related to an automatically operated circuit breaker on one of several generators. It would not close automatically, and had to be persuaded, which took some time. Further details will be made available to customers as soon as the post-event investigation is complete. ----- Original Message ----- From: outages-bounces@outages.org <outages-bounces@outages.org> To: reza a <reza@lethalnetworks.com>; outages@outages.org <outages@outages.org> Sent: Fri Aug 31 18:52:35 2012 Subject: Re: [outages] Linode outage in Newark, NJ starting at 3:30 pm It is common to have not only strings of batteries but an ATS, or automatic transfer switch. In this case, it sounds like the fault was on the ATS side of the plant based on the IR below. - LB -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of reza a Sent: Friday, August 31, 2012 3:29 PM To: outages@outages.org Subject: Re: [outages] Linode outage in Newark, NJ starting at 3:30 pm Is it common practice to have an array of batteries in between the colo and the generators to give you time to start the generators? ----- Original Message ----- From: "Sadiq Saif" <sadiq@asininetech.com> To: "Jack Carrozzo" <jack@crepinc.com> Cc: outages@outages.org Sent: Friday, August 31, 2012 2:59:16 PM Subject: Re: [outages] Linode outage in Newark, NJ starting at 3:30 pm This is the incident report from NAC: Subsequent to the utility power failure at our Cedar Knolls facility the generator that powers systems A and D did not start automatically. Manual intervention was required to get generator power running. Unfortunately before the generator was manually engaged customers on these systems experienced a complete power loss. We are still investigating why the generator did not engage automatically.Utility power has been restored, and transfer back to utility was successful. On Fri, Aug 31, 2012 at 3:59 PM, Jack Carrozzo <jack@crepinc.com> wrote:
3:52pm (EDT): NAC has informed us of a power issue affecting at least some portion of the datacenter. As soon as power is restored we are poised to execute our recovery procedures to all affected systems.
On Fri, Aug 31, 2012 at 3:46 PM, Smith, Kyle <ksmith@litle.com> wrote:
FYI,
We started getting alerts at roughly ~3:30
- Kyle
The information in this message is for the intended recipient(s) only and may be the proprietary and/or confidential property of Litle & Co., LLC, and thus protected from disclosure. If you are not the intended recipient(s), or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any use, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify Litle & Co. immediately by replying to this message and then promptly deleting it and your reply permanently from your computer.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sadiq S O< ascii ribbon campaign - stop html mail - www.asciiribbon.org _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Alex, Thanks for the information! Good luck with those relays! Blake Pfankuch Cisco, Microsoft, Adtran and VMware Certification Information available upon request. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Alex Rubenstein Sent: Friday, August 31, 2012 5:34 PM To: 'Lonnie.Bozeman@coresite.com'; 'reza@lethalnetworks.com'; 'outages@outages.org' Subject: Re: [outages] Linode outage in Newark, NJ starting at 3:30 pm The fault was related to an automatically operated circuit breaker on one of several generators. It would not close automatically, and had to be persuaded, which took some time. Further details will be made available to customers as soon as the post-event investigation is complete. ----- Original Message ----- From: outages-bounces@outages.org <outages-bounces@outages.org> To: reza a <reza@lethalnetworks.com>; outages@outages.org <outages@outages.org> Sent: Fri Aug 31 18:52:35 2012 Subject: Re: [outages] Linode outage in Newark, NJ starting at 3:30 pm It is common to have not only strings of batteries but an ATS, or automatic transfer switch. In this case, it sounds like the fault was on the ATS side of the plant based on the IR below. - LB -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of reza a Sent: Friday, August 31, 2012 3:29 PM To: outages@outages.org Subject: Re: [outages] Linode outage in Newark, NJ starting at 3:30 pm Is it common practice to have an array of batteries in between the colo and the generators to give you time to start the generators? ----- Original Message ----- From: "Sadiq Saif" <sadiq@asininetech.com> To: "Jack Carrozzo" <jack@crepinc.com> Cc: outages@outages.org Sent: Friday, August 31, 2012 2:59:16 PM Subject: Re: [outages] Linode outage in Newark, NJ starting at 3:30 pm This is the incident report from NAC: Subsequent to the utility power failure at our Cedar Knolls facility the generator that powers systems A and D did not start automatically. Manual intervention was required to get generator power running. Unfortunately before the generator was manually engaged customers on these systems experienced a complete power loss. We are still investigating why the generator did not engage automatically.Utility power has been restored, and transfer back to utility was successful. On Fri, Aug 31, 2012 at 3:59 PM, Jack Carrozzo <jack@crepinc.com> wrote:
3:52pm (EDT): NAC has informed us of a power issue affecting at least some portion of the datacenter. As soon as power is restored we are poised to execute our recovery procedures to all affected systems.
On Fri, Aug 31, 2012 at 3:46 PM, Smith, Kyle <ksmith@litle.com> wrote:
FYI,
We started getting alerts at roughly ~3:30
- Kyle
The information in this message is for the intended recipient(s) only and may be the proprietary and/or confidential property of Litle & Co., LLC, and thus protected from disclosure. If you are not the intended recipient(s), or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any use, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify Litle & Co. immediately by replying to this message and then promptly deleting it and your reply permanently from your computer.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sadiq S O< ascii ribbon campaign - stop html mail - www.asciiribbon.org _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (2)
-
Alex Rubenstein -
Blake Pfankuch