Re: [outages] Linode Fremont outage
So the outage was OK to have since they had staff on site? On May 30, 2015 09:01, Paul WALL via Outages <outages@outages.org> wrote: It's a good thing Hurricane Electric had electricians on site to deal with the outage. If it were an Equinix or Telx site, the response would have assuredly been far worse. Paul _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Here it is right from HE You can always call us. Sorry I did not see this sooner Walt At the Fremont 2 facility at 48233 Warm Springs Blvd, Fremont CA on May 29th at approximately 6:30 PM PST PG&E (the power utility) had an outage. Generators 1,2,4,5,6,7,8 started correctly and transferred. Customers in suites 1100,1200,1400,1500 experienced no issues. Generator 3 experienced an electromechanical failure. We are in the process of remediating this. Customers in suite 1300 are currently affected. The estimate time to restoration is 9:30 PM. We apologize for the inconvenience. Sent from my iPhone
On May 30, 2015, at 5:39 PM, Alex Rubenstein via Outages <outages@outages.org> wrote:
So the outage was OK to have since they had staff on site?
On May 30, 2015 09:01, Paul WALL via Outages <outages@outages.org> wrote: It's a good thing Hurricane Electric had electricians on site to deal with the outage.
If it were an Equinix or Telx site, the response would have assuredly been far worse.
Paul _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Are suites tied to just one generator? Frank From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Walt Wollny via Outages Sent: Saturday, May 30, 2015 9:50 PM To: Alex Rubenstein Cc: outages@outages.org Subject: Re: [outages] Linode Fremont outage Here it is right from HE You can always call us. Sorry I did not see this sooner Walt At the Fremont 2 facility at 48233 Warm Springs Blvd, Fremont CA <x-apple-data-detectors://0> on May 29th at approximately 6:30 PM PST <x-apple-data-detectors://2> PG&E (the power utility) had an outage. Generators 1,2,4,5,6,7,8 started correctly and transferred. Customers in suites 1100,1200,1400,1500 experienced no issues. Generator 3 experienced an electromechanical failure. We are in the process of remediating this. Customers in suite 1300 are currently affected. The estimate time to restoration is 9:30 PM <x-apple-data-detectors://3> . We apologize for the inconvenience. Sent from my iPhone On May 30, 2015, at 5:39 PM, Alex Rubenstein via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: So the outage was OK to have since they had staff on site? On May 30, 2015 09:01, Paul WALL via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: It's a good thing Hurricane Electric had electricians on site to deal with the outage. If it were an Equinix or Telx site, the response would have assuredly been far worse. Paul _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
On Sat, May 30, 2015 at 8:04 PM, Frank Bulk via Outages <outages@outages.org
wrote:
Are suites tied to just one generator?
Unless things have changed since I was last there, yes. HE do not offer redundant power (at least, not at FMT2). HE offer a great price point, but as the saying goes, you get what you pay for! Scott
Hmm, that's not really what I envision as 'redundant' :-/ Is there an overview somewhere how this has been sorted for the other datacenters Linode uses? Are they better equipped? --
On 31 May 2015, at 16:09, Alex Forster via Outages <outages@outages.org> wrote:
The redundancy was grid + generator. Unfortunately both failed.
<signature-logo.png> Alex Forster Network Engineer | Linode From: Scott Howard via Outages <outages@outages.org> Sent: Sunday, May 31, 2015 4:37AM To: Frank Bulk <frnkblk@iname.com> Cc: Outages Mailing List <outages@outages.org> Subject: Re: [outages] Linode Fremont outage
On Sat, May 30, 2015 at 8:04 PM, Frank Bulk via Outages <outages@outages.org> wrote: Are suites tied to just one generator?
Unless things have changed since I was last there, yes. HE do not offer redundant power (at least, not at FMT2).
HE offer a great price point, but as the saying goes, you get what you pay for!
Scott
_______________________________________________ Outages mailing list
Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
There must be more to the story because the equipment on a particular generator seems to have gotten damaged, no? Greg On May 31, 2015 9:39 AM, "Alex Forster via Outages" <outages@outages.org> wrote:
The redundancy was grid + generator. Unfortunately both failed.
[image: (Logo)] Alex Forster Network Engineer | Linode *From:* Scott Howard via Outages <outages@outages.org> <outages@outages.org> *Sent:* Sunday, May 31, 2015 4:37AM *To:* Frank Bulk <frnkblk@iname.com> <frnkblk@iname.com> *Cc:* Outages Mailing List <outages@outages.org> <outages@outages.org> *Subject:* Re: [outages] Linode Fremont outage
On Sat, May 30, 2015 at 8:04 PM, Frank Bulk via Outages < outages@outages.org> wrote:
Are suites tied to just one generator?
Unless things have changed since I was last there, yes. HE do not offer redundant power (at least, not at FMT2).
HE offer a great price point, but as the saying goes, you get what you pay for!
Scott
_______________________________________________ Outages mailing listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
So, As a reminder a few years back Amazon had a similar situation back east. (I forget which location) The utility power went off and one of the Generators was not working correctly. Before they were able to get it fixed the UPS Juice went out and a section of servers went off-line. A few minutes later power was returned. (Either utility or generator I don't recall) But it took apx 1 day to get most the servers (And Virtual servers on them) back online. However there was some servers / virtual servers that were not recoverable. From my experience loss of power on a single machine generally doesn't hurt the machine to badly. However when you have Hundreds go off-line there are bound to be a few with issues. It just seems to be that way. Also I do not Fault HE or Linode for only have Grid + Generator. However I bet there are some UPS in there too. People tend to believe that there is just so much redundancy that how could a thing happen. People need to see reality and as mentioned before you do get what you pay for. However most people do not want to pay for tons of redundancy. And even when you do I foresee there are things that can still take you down. If there is more to the story sure I would like to hear as well. But honestly I am guessing it was a failure with no foul play happening. Sincerely, Mark On 5/31/2015 7:44 AM, Greg Ihnen via Outages wrote:
There must be more to the story because the equipment on a particular generator seems to have gotten damaged, no?
Greg
On May 31, 2015 9:39 AM, "Alex Forster via Outages" <outages@outages.org <mailto:outages@outages.org>> wrote:
The redundancy was grid + generator. Unfortunately both failed.
(Logo)
Alex Forster Network Engineer | Linode
*From:* Scott Howard via Outages <outages@outages.org> <mailto:outages@outages.org> *Sent:* Sunday, May 31, 2015 4:37AM *To:* Frank Bulk <frnkblk@iname.com> <mailto:frnkblk@iname.com> *Cc:* Outages Mailing List <outages@outages.org> <mailto:outages@outages.org> *Subject:* Re: [outages] Linode Fremont outage
On Sat, May 30, 2015 at 8:04 PM, Frank Bulk via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Are suites tied to just one generator?
Unless things have changed since I was last there, yes. HE do not offer redundant power (at least, not at FMT2).
HE offer a great price point, but as the saying goes, you get what you pay for!
Scott
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I'm glad HE is fixing the generator. Is weekly testing also being contemplated? It would be a plus not to discover that the generator is broken when the power goes out. 9:30 PM turned into 11:10 PM. Total outage for my systems was 4 hrs, 57 min. http://status.linode.com/ On Sat, May 30, 2015 at 07:49:42PM -0700, Walt Wollny via Outages wrote:
Here it is right from HE
You can always call us. Sorry I did not see this sooner
Walt
At the Fremont 2 facility at 48233 Warm Springs Blvd, Fremont CA on May 29th at approximately 6:30 PM PST PG&E (the power utility) had an outage.
Generators 1,2,4,5,6,7,8 started correctly and transferred.
Customers in suites 1100,1200,1400,1500 experienced no issues.
Generator 3 experienced an electromechanical failure. We are in the process of remediating this.
Customers in suite 1300 are currently affected.
The estimate time to restoration is 9:30 PM.
We apologize for the inconvenience.
Sent from my iPhone
On May 30, 2015, at 5:39 PM, Alex Rubenstein via Outages <outages@outages.org> wrote:
So the outage was OK to have since they had staff on site?
On May 30, 2015 09:01, Paul WALL via Outages <outages@outages.org> wrote: It's a good thing Hurricane Electric had electricians on site to deal with the outage.
If it were an Equinix or Telx site, the response would have assuredly been far worse.
Paul _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
9:30 pm Pacific hasn't arrived, yet. Frank -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Nate Itkin via Outages Sent: Saturday, May 30, 2015 10:06 PM To: outages@outages.org Subject: Re: [outages] Linode Fremont outage I'm glad HE is fixing the generator. Is weekly testing also being contemplated? It would be a plus not to discover that the generator is broken when the power goes out. 9:30 PM turned into 11:10 PM. Total outage for my systems was 4 hrs, 57 min. http://status.linode.com/ On Sat, May 30, 2015 at 07:49:42PM -0700, Walt Wollny via Outages wrote:
Here it is right from HE
You can always call us. Sorry I did not see this sooner
Walt
At the Fremont 2 facility at 48233 Warm Springs Blvd, Fremont CA on May 29th at approximately 6:30 PM PST PG&E (the power utility) had an outage.
Generators 1,2,4,5,6,7,8 started correctly and transferred.
Customers in suites 1100,1200,1400,1500 experienced no issues.
Generator 3 experienced an electromechanical failure. We are in the process of remediating this.
Customers in suite 1300 are currently affected.
The estimate time to restoration is 9:30 PM.
We apologize for the inconvenience.
Sent from my iPhone
On May 30, 2015, at 5:39 PM, Alex Rubenstein via Outages <outages@outages.org> wrote:
So the outage was OK to have since they had staff on site?
On May 30, 2015 09:01, Paul WALL via Outages <outages@outages.org> wrote: It's a good thing Hurricane Electric had electricians on site to deal with the outage.
If it were an Equinix or Telx site, the response would have assuredly been far worse.
Paul _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Ok, I'm the idiot -- this was yesterday's outage. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk via Outages Sent: Saturday, May 30, 2015 10:09 PM To: outages@konadogs.net; outages@outages.org Subject: Re: [outages] Linode Fremont outage 9:30 pm Pacific hasn't arrived, yet. Frank -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Nate Itkin via Outages Sent: Saturday, May 30, 2015 10:06 PM To: outages@outages.org Subject: Re: [outages] Linode Fremont outage I'm glad HE is fixing the generator. Is weekly testing also being contemplated? It would be a plus not to discover that the generator is broken when the power goes out. 9:30 PM turned into 11:10 PM. Total outage for my systems was 4 hrs, 57 min. http://status.linode.com/ On Sat, May 30, 2015 at 07:49:42PM -0700, Walt Wollny via Outages wrote:
Here it is right from HE
You can always call us. Sorry I did not see this sooner
Walt
At the Fremont 2 facility at 48233 Warm Springs Blvd, Fremont CA on May 29th at approximately 6:30 PM PST PG&E (the power utility) had an outage.
Generators 1,2,4,5,6,7,8 started correctly and transferred.
Customers in suites 1100,1200,1400,1500 experienced no issues.
Generator 3 experienced an electromechanical failure. We are in the process of remediating this.
Customers in suite 1300 are currently affected.
The estimate time to restoration is 9:30 PM.
We apologize for the inconvenience.
Sent from my iPhone
On May 30, 2015, at 5:39 PM, Alex Rubenstein via Outages <outages@outages.org> wrote:
So the outage was OK to have since they had staff on site?
On May 30, 2015 09:01, Paul WALL via Outages <outages@outages.org> wrote: It's a good thing Hurricane Electric had electricians on site to deal with the outage.
If it were an Equinix or Telx site, the response would have assuredly been far worse.
Paul _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Yes, and the outage was only five hours. From about 2015-05-29 18:30 PT to about 23:30 PT, on a Friday. (Not until 21:30, as originally promised.) Also, it was a "limited issue", according to HE. Not sure what that means -- it looks like the whole Linode network in Fremont has been down, plus some other customers, too, even though supposedly only one out of the eight power generators was affected. Did anyone NOT have had their server power-cycled? IIRC, this is by far not the first power-related issue over in Fremont. In other words, business as usual! C. On 30 May 2015 at 17:39, Alex Rubenstein via Outages <outages@outages.org> wrote:
So the outage was OK to have since they had staff on site?
On May 30, 2015 09:01, Paul WALL via Outages <outages@outages.org> wrote:
It's a good thing Hurricane Electric had electricians on site to deal with the outage.
If it were an Equinix or Telx site, the response would have assuredly been far worse.
Paul _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (11)
-
Alex Forster -
Alex Rubenstein -
Constantine A. Murenin -
DTNX Postmaster -
Frank Bulk -
Greg Ihnen -
Jim Popovitch -
Mark Keymer -
Nate Itkin -
Scott Howard -
Walt Wollny