Evocative EWR1 (formerly INAP NYJ004)
We are hearing of a site-wide emergency at Evocative EWR1, formerly INAP NYJ004, at 1 Enterprise in Secaucus NJ. We have lost both sides of power to our cage. Anyone seeing similar? Any scuttlebutt? Support is non-responsive. -cjp
Same. We can't reach any server. You sure it's power vs network?
On Mon, Jul 10, 2023, at 19:10, Alex wrote:
Same. We can't reach any server. You sure it's power vs network?
We saw network loss around 1750 EDT. Things briefly came back, and I saw that my hardware had been hard booted around 1755 EDT. Lost everything again around 1805 EDT. -cjp
There was a power issue. On Mon, Jul 10, 2023, 7:18 PM Christopher J. Pilkington via Outages < outages@outages.org> wrote:
On Mon, Jul 10, 2023, at 19:10, Alex wrote:
Same. We can't reach any server. You sure it's power vs network?
We saw network loss around 1750 EDT. Things briefly came back, and I saw that my hardware had been hard booted around 1755 EDT.
Lost everything again around 1805 EDT.
-cjp _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
https://status.dedicated.com/incidents/44
We've been informed that an electrical room experienced a fire, was put out by retardant, and the datacenter is in emergency power off status at the requirement of on-site fire fighters.
On Mon, Jul 10, 2023, 7:23 PM Ross Tajvar <ross@tajvar.io> wrote:
There was a power issue.
On Mon, Jul 10, 2023, 7:18 PM Christopher J. Pilkington via Outages < outages@outages.org> wrote:
On Mon, Jul 10, 2023, at 19:10, Alex wrote:
Same. We can't reach any server. You sure it's power vs network?
We saw network loss around 1750 EDT. Things briefly came back, and I saw that my hardware had been hard booted around 1755 EDT.
Lost everything again around 1805 EDT.
-cjp _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Confirmed, my coworker said he's seeing fire trucks there. - Alex
Hi guys. Has anyone received an update from them?
Alex - looks like dedicated.com is still posting good updates. Everyone else - a friend in the area sent me this picture: https://imgur.com/a/QsxldPj He also said it "smells like marshmallows", so...maybe someone is taking the opportunity to make smores? :) On Mon, Jul 10, 2023 at 8:26 PM Alex <mr.alexey@gmail.com> wrote:
Hi guys. Has anyone received an update from them?
We received the same update from the datacenter ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Power remains off at our data center in Secaucus/EWR1 per the local fire marshall. We have had an electrical failure with one of our redundant UPS' that started to smoke and then had a small fire in the UPS room. The fire department was dispatched and the fire was extinguished quickly. The fire department subsequently cut power to the entire data center and disabled our generators while they and the utility verify electrical system. We have been working with both the fire department and the utility to expedite this process. We are currently waiting on the fire marshall and local utility to reenergize the site. We are completely dependent upon their inspection and approval. We are hoping to get an update that we can share in the next hour. At the current time, the fire department is controlling access to the building and we will not be able to let customers in. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Looks like EWR1 will be down at least until the afternoon.
ROFL we were making the same joke about gracing palms. The update we received: "Once we receive approval and utility is restored, we will turn up critical systems. This will take approximately 5 hours. After the critical systems are restored, we will be turning up the carriers and then will start to turn the servers back on." 9am+5=2pm+$carriers_time_to_on, I'm betting on after 5pm counting the time it will take the inspector to sigh off. - Alex
Utility power has been restored to the site! The ETA for bringing up the critical infrastructure systems is approximately 5 hours.
participants (3)
-
Alex -
Christopher J. Pilkington -
Ross Tajvar