8x8 outage/111 8th Ave NYC
Our hosts there are not responding as of 10:58p EDT. We've got no news but are assuming it's Sandy coming to town. --j
I have no idea what is going on there. However, much of Manhattan is under water, under mandatory evacuation, and certainly no fuel deliveries are being made. I am at my house in 07874. Considerable damage and lots of trees down. We own datacenters in 07927, which still has utility power; and another in 07054 which is on generator with utility voltage roughly 14% high (about 550 volts). We have equipment in 60 Hudson - as of now, it is still up. We have lost one span of fiber between Morristown, NJ and Newark, NJ. The situation is not great. I am grateful our centers are 100's of feet above sea level. From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jim Meyer Sent: Monday, October 29, 2012 11:19 PM To: outages@outages.org Subject: [outages] 8x8 outage/111 8th Ave NYC Our hosts there are not responding as of 10:58p EDT. We've got no news but are assuming it's Sandy coming to town. --j
Large section of Manhattan is without power currently. Flooding too so the generator might be a little waterlogged. Chuck On Mon, Oct 29, 2012 at 11:18 PM, Jim Meyer <jim@geekdaily.org> wrote:
Our hosts there are not responding as of 10:58p EDT.
We've got no news but are assuming it's Sandy coming to town.
--j
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
There are widespread power outages downtown, with much of Manhattan south of 39th st going dark starting at 2200 Eastern. (end) On Oct 29, 2012 11:31 PM, "Jim Meyer" <jim@geekdaily.org> wrote:
Our hosts there are not responding as of 10:58p EDT.
We've got no news but are assuming it's Sandy coming to town.
--j
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
They have shut sections of the grid off.. Lonnie Bozeman Operations Manager CORESITE (NYSE: COR) 900 N. Alameda St. Suite 200 Los Angeles, CA 90012 USA +1 213.327.1204 | Office +1 213.598.1739 | Cell +1 213.327.1289 | eFax Lonnie.Bozeman@CoreSite.com| www.CoreSite.com Jim Meyer <jim@geekdaily.org> wrote: Our hosts there are not responding as of 10:58p EDT. We've got no news but are assuming it's Sandy coming to town. --j
Because salt water and electricity are a bad combination, NY has shut down large sections of power as a precaution in preparation for expected storm surges that are likely to flood many of the underground facilities. These precautionary outages will very likely make it a whole lot easier to restore power much sooner after Sandy finishes pounding the area. It's actually a surprisingly wise thing for them to have done. However, for it to work out well, they would have had to offline the back-up generators and disconnect the battery systems as well. I don't know to what extent that will have happened. Owen On Oct 29, 2012, at 20:39 , Lonnie Bozeman <Lonnie.Bozeman@coresite.com> wrote:
They have shut sections of the grid off..
Lonnie Bozeman Operations Manager
CORESITE (NYSE: COR) 900 N. Alameda St. Suite 200 Los Angeles, CA 90012 USA +1 213.327.1204 | Office +1 213.598.1739 | Cell +1 213.327.1289 | eFax
Lonnie.Bozeman@CoreSite.com| www.CoreSite.com
Jim Meyer <jim@geekdaily.org> wrote: Our hosts there are not responding as of 10:58p EDT.
We've got no news but are assuming it's Sandy coming to town.
--j
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
There are also unexpected outages. 111 8th was not in the area to be intentionally powered down. (end) On Oct 30, 2012 12:22 AM, "Owen DeLong" <owen@delong.com> wrote:
Because salt water and electricity are a bad combination, NY has shut down large sections of power as a precaution in preparation for expected storm surges that are likely to flood many of the underground facilities. These precautionary outages will very likely make it a whole lot easier to restore power much sooner after Sandy finishes pounding the area.
It's actually a surprisingly wise thing for them to have done. However, for it to work out well, they would have had to offline the back-up generators and disconnect the battery systems as well. I don't know to what extent that will have happened.
Owen
On Oct 29, 2012, at 20:39 , Lonnie Bozeman <Lonnie.Bozeman@coresite.com> wrote:
They have shut sections of the grid off..
Lonnie Bozeman Operations Manager
CORESITE (NYSE: COR) 900 N. Alameda St. Suite 200 Los Angeles, CA 90012 USA +1 213.327.1204 | Office +1 213.598.1739 | Cell +1 213.327.1289 | eFax
Lonnie.Bozeman@CoreSite.com| www.CoreSite.com
Jim Meyer <jim@geekdaily.org> wrote: Our hosts there are not responding as of 10:58p EDT.
We've got no news but are assuming it's Sandy coming to town.
--j
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Info from Zayo: The local power provider in the NY Market, ConEd, has intentionally de-energized power systems in the New York metropolitan area to reduce system damage as salt water has entered the underground infrastructure. Additionally, power outages have been observed in the Newark metropolitan area affecting the Zayo facility at 165 Halsey. All Zayo facilities that have sustained a loss of commercial power are currently operating normally on generator backup, and refueling arrangements are being made for all Zayo owned generators at this time to ensure continuous operation of Zayo facilities and equipment. A current readout by site of our current generator runtime is as follows: - *60 Hudson (NY)* - 1st Floor MMR (Meet Me Room) 18 Hours of generator backup, with additional fuel onsite to replenish the generator - 12th Floor has 30+ hours or generator back up - 15th and 19th Colocation Floor 30+ hours or generator back up which is provided by the building **Fuel supplier has tankers inside the city of New York to deliver fuel to Zayo when necessary.* - *111 8th Ave (NY)* - Has 90,000 gallons of fuel, which is able to provide up to 4 days of power - *165 Halsey (NJ)* - 5th Floor has 17+ hours of generator backup - 10th Floor has 30+ hours of generator backup At this point, we are experiencing multiple fiber cuts as a result of the storm which has resulted in service interruptions to transport services in the Pennsylvania, New Jersey and New York markets. On Mon, Oct 29, 2012 at 9:07 PM, Owen DeLong <owen@delong.com> wrote:
Because salt water and electricity are a bad combination, NY has shut down large sections of power as a precaution in preparation for expected storm surges that are likely to flood many of the underground facilities. These precautionary outages will very likely make it a whole lot easier to restore power much sooner after Sandy finishes pounding the area.
It's actually a surprisingly wise thing for them to have done. However, for it to work out well, they would have had to offline the back-up generators and disconnect the battery systems as well. I don't know to what extent that will have happened.
Owen
On Oct 29, 2012, at 20:39 , Lonnie Bozeman <Lonnie.Bozeman@coresite.com> wrote:
They have shut sections of the grid off..
Lonnie Bozeman Operations Manager
CORESITE (NYSE: COR) 900 N. Alameda St. Suite 200 Los Angeles, CA 90012 USA +1 213.327.1204 | Office +1 213.598.1739 | Cell +1 213.327.1289 | eFax
Lonnie.Bozeman@CoreSite.com| www.CoreSite.com
Jim Meyer <jim@geekdaily.org> wrote: Our hosts there are not responding as of 10:58p EDT.
We've got no news but are assuming it's Sandy coming to town.
--j
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Chris Sheats yawnbox@gmail.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Tracker was updated as well. regards, /virendra On 10/29/2012 09:25 PM, Chris Sheats wrote:
Info from Zayo:
The local power provider in the NY Market, ConEd, has intentionally de-energized power systems in the New York metropolitan area to reduce system damage as salt water has entered the underground infrastructure. Additionally, power outages have been observed in the Newark metropolitan area affecting the Zayo facility at 165 Halsey. All Zayo facilities that have sustained a loss of commercial power are currently operating normally on generator backup, and refueling arrangements are being made for all Zayo owned generators at this time to ensure continuous operation of Zayo facilities and equipment. A current readout by site of our current generator runtime is as follows:
- *60 Hudson (NY)* - 1st Floor MMR (Meet Me Room) 18 Hours of generator backup, with additional fuel onsite to replenish the generator - 12th Floor has 30+ hours or generator back up - 15th and 19th Colocation Floor 30+ hours or generator back up which is provided by the building
**Fuel supplier has tankers inside the city of New York to deliver fuel to Zayo when necessary.*
- *111 8th Ave (NY)* - Has 90,000 gallons of fuel, which is able to provide up to 4 days of power
- *165 Halsey (NJ)* - 5th Floor has 17+ hours of generator backup - 10th Floor has 30+ hours of generator backup
At this point, we are experiencing multiple fiber cuts as a result of the storm which has resulted in service interruptions to transport services in the Pennsylvania, New Jersey and New York markets.
On Mon, Oct 29, 2012 at 9:07 PM, Owen DeLong <owen@delong.com> wrote:
Because salt water and electricity are a bad combination, NY has shut down large sections of power as a precaution in preparation for expected storm surges that are likely to flood many of the underground facilities. These precautionary outages will very likely make it a whole lot easier to restore power much sooner after Sandy finishes pounding the area.
It's actually a surprisingly wise thing for them to have done. However, for it to work out well, they would have had to offline the back-up generators and disconnect the battery systems as well. I don't know to what extent that will have happened.
Owen
On Oct 29, 2012, at 20:39 , Lonnie Bozeman <Lonnie.Bozeman@coresite.com> wrote:
They have shut sections of the grid off..
Lonnie Bozeman Operations Manager
CORESITE (NYSE: COR) 900 N. Alameda St. Suite 200 Los Angeles, CA 90012 USA +1 213.327.1204 | Office +1 213.598.1739 | Cell +1 213.327.1289 | eFax
Lonnie.Bozeman@CoreSite.com| www.CoreSite.com
Jim Meyer <jim@geekdaily.org> wrote: Our hosts there are not responding as of 10:58p EDT.
We've got no news but are assuming it's Sandy coming to town.
--j
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iF4EAREIAAYFAlCPXwgACgkQ3HuimOHfh+HwpAD+O7bYXoT/u2zR6y9L5L/62ba/ Hvtif3Pdwt4m+j3eNL8A/1E7f5rXXfDYtPGGJgXF9BO7fvdruIv90pzWX7Qk6tmM =DiSW -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Just received this update by one of Zayo's customer. In preparation for this potential power event, here's the readout by site of Zayos generator backup capacity is as follows: Hopefully this helps if you are located in one of these locations and /or floors: * 60 Hudson (NY) * 8th Floor MMR (Meet Me Room) 18 Hours of generator backup, with additional fuel onsite to replenish the generator * 12th Floor has 30+ hours or generator back up * 19th and 20th Colocation Floors 30+ hours or generator back up which is provided by the building *Fuel supplier has tankers inside the city of New York to deliver fuel to Zayo when necessary. * 111 8th Ave (NY) * Has 90,000 gallons of fuel, which is able to provide up to 4 days of power * 165 Halsey (NJ) * 5th Floor has 17+ hours of generator backup * 10th Floor has 30+ hours of generator backup * 401 N Broad (PA) * 2nd Floor has 30+ hours of generator backup * 3rd Floor has 50+ hours of generator backup * 1 Summer St (CT) * The building has 16+ hours of generator backup * 111 Market Place (MD) * Has 36+ hours of generator backup * 2100 M St (DC) * The building has 20+ hours of generator backup regards, /virendra On 10/29/2012 09:07 PM, Owen DeLong wrote:
Because salt water and electricity are a bad combination, NY has shut down large sections of power as a precaution in preparation for expected storm surges that are likely to flood many of the underground facilities. These precautionary outages will very likely make it a whole lot easier to restore power much sooner after Sandy finishes pounding the area.
It's actually a surprisingly wise thing for them to have done. However, for it to work out well, they would have had to offline the back-up generators and disconnect the battery systems as well. I don't know to what extent that will have happened.
Owen
On Oct 29, 2012, at 20:39 , Lonnie Bozeman <Lonnie.Bozeman@coresite.com> wrote:
They have shut sections of the grid off..
Lonnie Bozeman Operations Manager
CORESITE (NYSE: COR) 900 N. Alameda St. Suite 200 Los Angeles, CA 90012 USA +1 213.327.1204 | Office +1 213.598.1739 | Cell +1 213.327.1289 | eFax
Lonnie.Bozeman@CoreSite.com| www.CoreSite.com
Jim Meyer <jim@geekdaily.org> wrote: Our hosts there are not responding as of 10:58p EDT.
We've got no news but are assuming it's Sandy coming to town.
--j
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iF4EAREIAAYFAlCPVwgACgkQ3HuimOHfh+ETpQD/SI4Zm41ISjcr5P6kSsFNHktX AYSUieFqfL4UZw//k0EA/1qAnkE73ks3oS7XgBbciMFTg3WslrFGjr8uCj1KA4DD =BGej -----END PGP SIGNATURE-----
If you'd like to "see" the outage, NY Times has a webcam on the 51st floor now. < http://www.nytimes.com/interactive/2012/10/28/nyregion/nyt-webcam.html?smid=...
-- Eric (riding out our version of the storm in the Binghamton, NY area) On Oct 29, 2012 11:31 PM, "Jim Meyer" <jim@geekdaily.org> wrote:
Our hosts there are not responding as of 10:58p EDT.
We've got no news but are assuming it's Sandy coming to town.
--j
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Reported at http://www.webhostingtalk.com/showthread.php?t=1205042, "Voxel/Internap are down at 111 8th Avenue" Ryan On Mon, Oct 29, 2012 at 11:57 PM, Eric Adler <eaptech@gmail.com> wrote:
If you'd like to "see" the outage, NY Times has a webcam on the 51st floor now. < http://www.nytimes.com/interactive/2012/10/28/nyregion/nyt-webcam.html?smid=...
-- Eric (riding out our version of the storm in the Binghamton, NY area) On Oct 29, 2012 11:31 PM, "Jim Meyer" <jim@geekdaily.org> wrote:
Our hosts there are not responding as of 10:58p EDT.
We've got no news but are assuming it's Sandy coming to town.
--j
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
XO reports they're on generators at 111 8th Ave and are still seeing issues to that data center. Internap also has big issues, likely to go dead ~3:30-5:30a EDT 10/30/2012: http://pastebin.com/d9espQRw Init7 reports issues, but still has connectivity to NYIIX or Equinix NYC: http://www.init7.net/en/status/&ticket=6088 Meanwhile, light reading on what folks did to gear up for Sandy: http://www.datacenterknowledge.com/archives/2012/10/29/northeast-data-center... It'll be interesting to see what mattered and what didn't. --j Power outage maps: http://www.governor.ny.gov/poweroutageinfo Data center maps: http://www.datacentermap.com/ ConEd map with outages, weather radar: http://apps.coned.com/stormcenter_external_oru/default.html On Oct 29, 2012, at 8:18 PM, Jim Meyer wrote:
Our hosts there are not responding as of 10:58p EDT.
We've got no news but are assuming it's Sandy coming to town.
--j
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (10)
-
Alex Rubenstein -
Charles Mills -
Chris Sheats -
Dan Stralka -
Eric Adler -
Jim Meyer -
Lonnie Bozeman -
Owen DeLong -
Ryan McIntosh -
virendra rode