Equinix CH1 is overheating due to some chiller failures. The vendor is on-site troubleshooting. Ambient temperature is currently 28 degrees higher than normal. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
IBX: CH1 IBX Address: 350 E Cermak Rd, Chicago, 60616 Number: INC0025932 Date and Time of Occurrence: 2024-01-14 08:20:49 Site Local Time To view the incident and its updates, please use Service Insight. New update on Incident INC0025932: Equinix IBX Site Staff reports that temperatures have stabilized with an average temperature of 120 F degrees. One chiller is now fully operational, and another chiller is online but operating at reduced capacity. Equinix IBX Critical Facility Engineers continue to work on restoring the required 4-out-of-6 chillers to return cooling back to normal operations. Portable chillers are currently enroute with an estimated arrival time of 21:00 Site Local Time. Upon arrival, IBX Critical Facility Engineers and our vendor partner will work diligently to get the portable chillers online as quickly as possible. We will send another update once the portable chillers are operational. If not already implemented, Equinix site staff highly recommends that customers power down non-essential loads within the data center and/or transfer those operations to backup sites to assist with efforts at lowering the temperature within the facility. The severe weather in the Chicago area, contributing to the initial failure, continues to hamper efforts at resolution, but our teams are working diligently to resolve the issue as quickly as possible. IBX Critical Facility Engineers continue to supply outside air to decrease the collocation area's temperature, and all fans have been deployed to support cooling the IBX. • Service Offering: CH1 - Location - Floor • Affected Assets: 05 • Asset Condition: Temperature above operating range • Situation status: Under investigation The next update will be provided in approximately 30 minutes or earlier if available. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mike Hammett via Outages" <outages@outages.org> To: "outages" <outages@outages.org> Sent: Sunday, January 14, 2024 2:04:51 PM Subject: [outages] Equinix CH1 Overheating Equinix CH1 is overheating due to some chiller failures. The vendor is on-site troubleshooting. Ambient temperature is currently 28 degrees higher than normal. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Open the doors? On January 14, 2024 9:52:24 PM EST, Mike Hammett via Outages <outages@outages.org> wrote:
IBX: CH1 IBX Address: 350 E Cermak Rd, Chicago, 60616 Number: INC0025932 Date and Time of Occurrence: 2024-01-14 08:20:49 Site Local Time
To view the incident and its updates, please use Service Insight. New update on Incident INC0025932: Equinix IBX Site Staff reports that temperatures have stabilized with an average temperature of 120 F degrees. One chiller is now fully operational, and another chiller is online but operating at reduced capacity. Equinix IBX Critical Facility Engineers continue to work on restoring the required 4-out-of-6 chillers to return cooling back to normal operations. Portable chillers are currently enroute with an estimated arrival time of 21:00 Site Local Time. Upon arrival, IBX Critical Facility Engineers and our vendor partner will work diligently to get the portable chillers online as quickly as possible. We will send another update once the portable chillers are operational.
If not already implemented, Equinix site staff highly recommends that customers power down non-essential loads within the data center and/or transfer those operations to backup sites to assist with efforts at lowering the temperature within the facility.
The severe weather in the Chicago area, contributing to the initial failure, continues to hamper efforts at resolution, but our teams are working diligently to resolve the issue as quickly as possible. IBX Critical Facility Engineers continue to supply outside air to decrease the collocation area's temperature, and all fans have been deployed to support cooling the IBX. • Service Offering: CH1 - Location - Floor • Affected Assets: 05 • Asset Condition: Temperature above operating range • Situation status: Under investigation
The next update will be provided in approximately 30 minutes or earlier if available.
----- Mike Hammett Intelligent Computing Solutions
Midwest Internet Exchange
The Brothers WISP
----- Original Message -----
From: "Mike Hammett via Outages" <outages@outages.org> To: "outages" <outages@outages.org> Sent: Sunday, January 14, 2024 2:04:51 PM Subject: [outages] Equinix CH1 Overheating
Equinix CH1 is overheating due to some chiller failures. The vendor is on-site troubleshooting. Ambient temperature is currently 28 degrees higher than normal.
----- Mike Hammett Intelligent Computing Solutions
Midwest Internet Exchange
The Brothers WISP
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
They tried that:
IBX Critical Facility Engineers have deployed all available floor fans, opened all available doors and have engaged local rental company to source additional fans and portable spot coolers.
On Sun, Jan 14, 2024 at 10:19 PM Jay Ashworth via Outages < outages@outages.org> wrote:
Open the doors?
On January 14, 2024 9:52:24 PM EST, Mike Hammett via Outages < outages@outages.org> wrote:
IBX: CH1 IBX Address: 350 E Cermak Rd, Chicago, 60616 Number: INC0025932 Date and Time of Occurrence: 2024-01-14 08:20:49 Site Local Time
To view the incident and its updates, please use Service Insight. New update on Incident INC0025932: Equinix IBX Site Staff reports that temperatures have stabilized with an average temperature of 120 F degrees. One chiller is now fully operational, and another chiller is online but operating at reduced capacity. Equinix IBX Critical Facility Engineers continue to work on restoring the required 4-out-of-6 chillers to return cooling back to normal operations. Portable chillers are currently enroute with an estimated arrival time of 21:00 Site Local Time. Upon arrival, IBX Critical Facility Engineers and our vendor partner will work diligently to get the portable chillers online as quickly as possible. We will send another update once the portable chillers are operational.
If not already implemented, Equinix site staff highly recommends that customers power down non-essential loads within the data center and/or transfer those operations to backup sites to assist with efforts at lowering the temperature within the facility.
The severe weather in the Chicago area, contributing to the initial failure, continues to hamper efforts at resolution, but our teams are working diligently to resolve the issue as quickly as possible. IBX Critical Facility Engineers continue to supply outside air to decrease the collocation area's temperature, and all fans have been deployed to support cooling the IBX. • Service Offering: CH1 - Location - Floor • Affected Assets: 05 • Asset Condition: Temperature above operating range • Situation status: Under investigation
The next update will be provided in approximately 30 minutes or earlier if available.
----- Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> <https://www.linkedin.com/company/intelligent-computing-solutions> <https://twitter.com/ICSIL> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix> <https://www.linkedin.com/company/midwest-internet-exchange> <https://twitter.com/mdwestix> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> ------------------------------ *From: *"Mike Hammett via Outages" <outages@outages.org> *To: *"outages" <outages@outages.org> *Sent: *Sunday, January 14, 2024 2:04:51 PM *Subject: *[outages] Equinix CH1 Overheating
Equinix CH1 is overheating due to some chiller failures. The vendor is on-site troubleshooting. Ambient temperature is currently 28 degrees higher than normal.
----- Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> <https://www.linkedin.com/company/intelligent-computing-solutions> <https://twitter.com/ICSIL> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix> <https://www.linkedin.com/company/midwest-internet-exchange> <https://twitter.com/mdwestix> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Windy.com tells me that the outside temperature in Chicago right now is -8F. I'm not sure why it wouldn't work... On January 14, 2024 10:20:56 PM EST, Ross Tajvar <ross@tajvar.io> wrote:
They tried that:
IBX Critical Facility Engineers have deployed all available floor fans, opened all available doors and have engaged local rental company to source additional fans and portable spot coolers.
On Sun, Jan 14, 2024 at 10:19 PM Jay Ashworth via Outages < outages@outages.org> wrote:
Open the doors?
On January 14, 2024 9:52:24 PM EST, Mike Hammett via Outages < outages@outages.org> wrote:
IBX: CH1 IBX Address: 350 E Cermak Rd, Chicago, 60616 Number: INC0025932 Date and Time of Occurrence: 2024-01-14 08:20:49 Site Local Time
To view the incident and its updates, please use Service Insight. New update on Incident INC0025932: Equinix IBX Site Staff reports that temperatures have stabilized with an average temperature of 120 F degrees. One chiller is now fully operational, and another chiller is online but operating at reduced capacity. Equinix IBX Critical Facility Engineers continue to work on restoring the required 4-out-of-6 chillers to return cooling back to normal operations. Portable chillers are currently enroute with an estimated arrival time of 21:00 Site Local Time. Upon arrival, IBX Critical Facility Engineers and our vendor partner will work diligently to get the portable chillers online as quickly as possible. We will send another update once the portable chillers are operational.
If not already implemented, Equinix site staff highly recommends that customers power down non-essential loads within the data center and/or transfer those operations to backup sites to assist with efforts at lowering the temperature within the facility.
The severe weather in the Chicago area, contributing to the initial failure, continues to hamper efforts at resolution, but our teams are working diligently to resolve the issue as quickly as possible. IBX Critical Facility Engineers continue to supply outside air to decrease the collocation area's temperature, and all fans have been deployed to support cooling the IBX. • Service Offering: CH1 - Location - Floor • Affected Assets: 05 • Asset Condition: Temperature above operating range • Situation status: Under investigation
The next update will be provided in approximately 30 minutes or earlier if available.
----- Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> <https://www.linkedin.com/company/intelligent-computing-solutions> <https://twitter.com/ICSIL> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix> <https://www.linkedin.com/company/midwest-internet-exchange> <https://twitter.com/mdwestix> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> ------------------------------ *From: *"Mike Hammett via Outages" <outages@outages.org> *To: *"outages" <outages@outages.org> *Sent: *Sunday, January 14, 2024 2:04:51 PM *Subject: *[outages] Equinix CH1 Overheating
Equinix CH1 is overheating due to some chiller failures. The vendor is on-site troubleshooting. Ambient temperature is currently 28 degrees higher than normal.
----- Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> <https://www.linkedin.com/company/intelligent-computing-solutions> <https://twitter.com/ICSIL> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix> <https://www.linkedin.com/company/midwest-internet-exchange> <https://twitter.com/mdwestix> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Getting into to -discussion territory here, but I suspect they had difficulty circulating the air sufficiently. Having cold air isn't enough; you have to swap it out for the hot air pretty often. On Sun, Jan 14, 2024, 10:22 PM Jay Ashworth <jra@baylink.com> wrote:
Windy.com tells me that the outside temperature in Chicago right now is -8F. I'm not sure why it wouldn't work...
On January 14, 2024 10:20:56 PM EST, Ross Tajvar <ross@tajvar.io> wrote:
They tried that:
IBX Critical Facility Engineers have deployed all available floor fans, opened all available doors and have engaged local rental company to source additional fans and portable spot coolers.
On Sun, Jan 14, 2024 at 10:19 PM Jay Ashworth via Outages < outages@outages.org> wrote:
Open the doors?
On January 14, 2024 9:52:24 PM EST, Mike Hammett via Outages < outages@outages.org> wrote:
IBX: CH1 IBX Address: 350 E Cermak Rd, Chicago, 60616 Number: INC0025932 Date and Time of Occurrence: 2024-01-14 08:20:49 Site Local Time
To view the incident and its updates, please use Service Insight. New update on Incident INC0025932: Equinix IBX Site Staff reports that temperatures have stabilized with an average temperature of 120 F degrees. One chiller is now fully operational, and another chiller is online but operating at reduced capacity. Equinix IBX Critical Facility Engineers continue to work on restoring the required 4-out-of-6 chillers to return cooling back to normal operations. Portable chillers are currently enroute with an estimated arrival time of 21:00 Site Local Time. Upon arrival, IBX Critical Facility Engineers and our vendor partner will work diligently to get the portable chillers online as quickly as possible. We will send another update once the portable chillers are operational.
If not already implemented, Equinix site staff highly recommends that customers power down non-essential loads within the data center and/or transfer those operations to backup sites to assist with efforts at lowering the temperature within the facility.
The severe weather in the Chicago area, contributing to the initial failure, continues to hamper efforts at resolution, but our teams are working diligently to resolve the issue as quickly as possible. IBX Critical Facility Engineers continue to supply outside air to decrease the collocation area's temperature, and all fans have been deployed to support cooling the IBX. • Service Offering: CH1 - Location - Floor • Affected Assets: 05 • Asset Condition: Temperature above operating range • Situation status: Under investigation
The next update will be provided in approximately 30 minutes or earlier if available.
----- Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> <https://www.linkedin.com/company/intelligent-computing-solutions> <https://twitter.com/ICSIL> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix> <https://www.linkedin.com/company/midwest-internet-exchange> <https://twitter.com/mdwestix> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> ------------------------------ *From: *"Mike Hammett via Outages" <outages@outages.org> *To: *"outages" <outages@outages.org> *Sent: *Sunday, January 14, 2024 2:04:51 PM *Subject: *[outages] Equinix CH1 Overheating
Equinix CH1 is overheating due to some chiller failures. The vendor is on-site troubleshooting. Ambient temperature is currently 28 degrees higher than normal.
----- Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> <https://www.linkedin.com/company/intelligent-computing-solutions> <https://twitter.com/ICSIL> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix> <https://www.linkedin.com/company/midwest-internet-exchange> <https://twitter.com/mdwestix> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (5)
-
Dave Cohen via Outages -
J. Hellenthal via Outages
-
Jay Ashworth via Outages
-
Mike Hammett via Outages
-
Ross Tajvar via Outages