Word we got from our fiber provider there is that the entire building (ATT) is on battery backup as the generators either did not come online or were damaged in the blast. They told us to expect outages as battery backups start to fail. -- Larry Smith lesmith@ecsis.net On Fri December 25 2020 13:20, William Peebles via Outages wrote:

Haven’t heard anything from my end on direct technical problems as a result of the explosion, but after any major event like this it’s highly likely that voice and data lines will be overloaded with traffic from nearby residents

Sent from Outlook<https://aka.ms/qtex0l> for iOS ________________________________ From: Outages <outages-bounces@outages.org> on behalf of Kurt Duncan via Outages <outages@outages.org> Sent: Friday, December 25, 2020 2:09:15 PM To: outages@outages.org <outages@outages.org> Subject: [outages] Explosion in Nashville

Anyone else seeing phone or internet issues due to the AT&T building being damaged by the explosion in Nashville ?

Thank God for the first responders working today who protect us. -- Kurt Duncan

From a NY Times story: “AT&T is experiencing an outage in the area because gas lines have been shut off in the area, impacting the building’s backup natural gas generators.” -Brian

On Dec 25, 2020, at 1:35 PM, Larry Smith via Outages <outages@outages.org> wrote:

Word we got from our fiber provider there is that the entire building (ATT) is on battery backup as the generators either did not come online or were damaged in the blast. They told us to expect outages as battery backups start to fail.

-- Larry Smith lesmith@ecsis.net

...

On Fri December 25 2020 13:20, William Peebles via Outages wrote: Haven’t heard anything from my end on direct technical problems as a result of the explosion, but after any major event like this it’s highly likely that voice and data lines will be overloaded with traffic from nearby residents

Sent from Outlook<https://aka.ms/qtex0l> for iOS ________________________________ From: Outages <outages-bounces@outages.org> on behalf of Kurt Duncan via Outages <outages@outages.org> Sent: Friday, December 25, 2020 2:09:15 PM To: outages@outages.org <outages@outages.org> Subject: [outages] Explosion in Nashville

Anyone else seeing phone or internet issues due to the AT&T building being damaged by the explosion in Nashville ?

Thank God for the first responders working today who protect us. -- Kurt Duncan

---

Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Latest update from AT&T: AT&T Corporate Real Estate teams are now gathering on site after pausing work last night due a fire that reignited on site around midnight. The Fire Marshall has joined the AT&T team to provide official approval to proceed. Last night, the teams were able to accomplish testing the busbar, placing the temporary generators, completing power cabling, and opening all the breakers on the individual floors. It is estimated that this round of power related work is about 75% complete. The target is for energizing is 1200 CST. A partial analysis by the AT&T Structural Engineer initial found no concrete or steel damage that caused major concern. Once team re-engage this morning, they will complete their full analysis and ensure it is safe for restoration crews to proceed. AT&T Mechanical Engineers have confirmed there is roughly 3 feet of water in the basement with potential damage to mechanical equipment in that area. It is also suspected that commercial power may not be available for some time and as a result power will come from AT&T and Tennessee Emergency Management Agency generators. Additional technician support will be made available rotating over a 24 hour period. Core CBB teams continue to work on the secondary plan to reroute around the Nashville 2nd Ave location. They had some success this morning at 05:05 CT when core connectivity was established for the Winchester, KY office. Partial 911 connectivity has been restored in that specific area. Work will continue on the CBB reroute plans in several locations around the Southeast Region. We have installed a SatCOLT positioned in the area of the CO NSVLTNMT building and is up and taking traffic to aid in communication. The secondary SatCOLT arrived and has been turned up to provide additional coverage. The next update will be issued at approximately 13:00 CT, unless there is a significant change in status On Fri, Dec 25, 2020, 7:49 PM Ross Tajvar <ross@tajvar.io> wrote:

Got this internally, sharing here:

Here is the update from AT&T:

...

The front 1/3rd of their building is structuraly damaged. That includes

fiber entry.

...

They have no power and there is major flooding on all floors including the basement. The Fire Department is en route with pump trucks to help with the flooding. They have Building Engineers and Electricians on the way to look into the structure and getting power turned on. They have 2 850KW generators on the way and 2 1MW generators on standby. They have new A/C Chillers on the way (Existing Chiller plant was destroyed)

On Fri, Dec 25, 2020 at 3:38 PM Jay R. Ashworth via Outages < outages@outages.org> wrote:

...

----- Original Message -----

...

From: "Kurt Duncan via Outages" <outages@outages.org>

...

Anyone else seeing phone or internet issues due to the AT&T building being damaged by the explosion in Nashville ?

Let's please continue to confine replies here to specific information about network impacts, and keep the speculation threads to -discuss; thanks to whomever already started that thread. I would name them, but the caffeine hasn't kicked in yet.

If you're in that building in Nash, we'll be particularly interested in anything you're allowed to tell us in between bailing.

Cheers, -- jr '<admin/>' a -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Sorry I don't follow. Isn't the point of this list to keep people informed? On Sat, Dec 26, 2020, 1:31 PM Luis Diaz <noc@riotgames.com> wrote:

May want to consider moving this discussion to something more private.

On Sat, Dec 26, 2020 at 9:30 AM Ross Tajvar via Outages < outages@outages.org> wrote:

...

Latest update from AT&T:

AT&T Corporate Real Estate teams are now gathering on site after pausing work last night due a fire that reignited on site around midnight. The Fire Marshall has joined the AT&T team to provide official approval to proceed. Last night, the teams were able to accomplish testing the busbar, placing the temporary generators, completing power cabling, and opening all the breakers on the individual floors. It is estimated that this round of power related work is about 75% complete. The target is for energizing is 1200 CST. A partial analysis by the AT&T Structural Engineer initial found no concrete or steel damage that caused major concern. Once team re-engage this morning, they will complete their full analysis and ensure it is safe for restoration crews to proceed. AT&T Mechanical Engineers have confirmed there is roughly 3 feet of water in the basement with potential damage to mechanical equipment in that area. It is also suspected that commercial power may not be available for some time and as a result power will come from AT&T and Tennessee Emergency Management Agency generators. Additional technician support will be made available rotating over a 24 hour period.

Core CBB teams continue to work on the secondary plan to reroute around the Nashville 2nd Ave location. They had some success this morning at 05:05 CT when core connectivity was established for the Winchester, KY office. Partial 911 connectivity has been restored in that specific area. Work will continue on the CBB reroute plans in several locations around the Southeast Region.

We have installed a SatCOLT positioned in the area of the CO NSVLTNMT building and is up and taking traffic to aid in communication. The secondary SatCOLT arrived and has been turned up to provide additional coverage.

The next update will be issued at approximately 13:00 CT, unless there is a significant change in status

On Fri, Dec 25, 2020, 7:49 PM Ross Tajvar <ross@tajvar.io> wrote:

...

Got this internally, sharing here:

Here is the update from AT&T:

...

The front 1/3rd of their building is structuraly damaged. That

includes fiber entry.

...

They have no power and there is major flooding on all floors including the basement. The Fire Department is en route with pump trucks to help with the flooding. They have Building Engineers and Electricians on the way to look into the structure and getting power turned on. They have 2 850KW generators on the way and 2 1MW generators on standby. They have new A/C Chillers on the way (Existing Chiller plant was destroyed)

On Fri, Dec 25, 2020 at 3:38 PM Jay R. Ashworth via Outages < outages@outages.org> wrote:

...

----- Original Message -----

...

From: "Kurt Duncan via Outages" <outages@outages.org>

...

Anyone else seeing phone or internet issues due to the AT&T building being damaged by the explosion in Nashville ?

Let's please continue to confine replies here to specific information about network impacts, and keep the speculation threads to -discuss; thanks to whomever already started that thread. I would name them, but the caffeine hasn't kicked in yet.

If you're in that building in Nash, we'll be particularly interested in anything you're allowed to tell us in between bailing.

Cheers, -- jr '<admin/>' a -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

I appreciate the updates. I’m still getting outage alerts from Intrado due to the ongoing work. John Bergoon 2513666986 On Dec 26, 2020, at 12:52 PM, Jason Kuehl via Outages <outages@outages.org> wrote:  CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. This is turning into a mix of information. Its going to be that way for a while until we get more stable, when it's just outage notification updates. On Sat, Dec 26, 2020 at 1:38 PM Ross Tajvar via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Sorry I don't follow. Isn't the point of this list to keep people informed? On Sat, Dec 26, 2020, 1:31 PM Luis Diaz <noc@riotgames.com<mailto:noc@riotgames.com>> wrote: May want to consider moving this discussion to something more private. On Sat, Dec 26, 2020 at 9:30 AM Ross Tajvar via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Latest update from AT&T: AT&T Corporate Real Estate teams are now gathering on site after pausing work last night due a fire that reignited on site around midnight. The Fire Marshall has joined the AT&T team to provide official approval to proceed. Last night, the teams were able to accomplish testing the busbar, placing the temporary generators, completing power cabling, and opening all the breakers on the individual floors. It is estimated that this round of power related work is about 75% complete. The target is for energizing is 1200 CST. A partial analysis by the AT&T Structural Engineer initial found no concrete or steel damage that caused major concern. Once team re-engage this morning, they will complete their full analysis and ensure it is safe for restoration crews to proceed. AT&T Mechanical Engineers have confirmed there is roughly 3 feet of water in the basement with potential damage to mechanical equipment in that area. It is also suspected that commercial power may not be available for some time and as a result power will come from AT&T and Tennessee Emergency Management Agency generators. Additional technician support will be made available rotating over a 24 hour period. Core CBB teams continue to work on the secondary plan to reroute around the Nashville 2nd Ave location. They had some success this morning at 05:05 CT when core connectivity was established for the Winchester, KY office. Partial 911 connectivity has been restored in that specific area. Work will continue on the CBB reroute plans in several locations around the Southeast Region. We have installed a SatCOLT positioned in the area of the CO NSVLTNMT building and is up and taking traffic to aid in communication. The secondary SatCOLT arrived and has been turned up to provide additional coverage. The next update will be issued at approximately 13:00 CT, unless there is a significant change in status On Fri, Dec 25, 2020, 7:49 PM Ross Tajvar <ross@tajvar.io<mailto:ross@tajvar.io>> wrote: Got this internally, sharing here: Here is the update from AT&T:

The front 1/3rd of their building is structuraly damaged. That includes fiber entry. They have no power and there is major flooding on all floors including the basement. The Fire Department is en route with pump trucks to help with the flooding. They have Building Engineers and Electricians on the way to look into the structure and getting power turned on. They have 2 850KW generators on the way and 2 1MW generators on standby. They have new A/C Chillers on the way (Existing Chiller plant was destroyed)

On Fri, Dec 25, 2020 at 3:38 PM Jay R. Ashworth via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: ----- Original Message -----

From: "Kurt Duncan via Outages" <outages@outages.org<mailto:outages@outages.org>>

Anyone else seeing phone or internet issues due to the AT&T building being damaged by the explosion in Nashville ?

Let's please continue to confine replies here to specific information about network impacts, and keep the speculation threads to -discuss; thanks to whomever already started that thread. I would name them, but the caffeine hasn't kicked in yet. If you're in that building in Nash, we'll be particularly interested in anything you're allowed to tell us in between bailing. Cheers, -- jr '<admin/>' a -- Jay R. Ashworth Baylink jra@baylink.com<mailto:jra@baylink.com> Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274 _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages -- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com<mailto:jason.w.kuehl@gmail.com> _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

It is. So unless that message that you reposted came to you with a flag line suggesting it should not be disseminated, you're welcome to post it here. Cheers, -- jr '<admin/>' a On December 26, 2020 1:35:29 PM EST, Ross Tajvar via Outages <outages@outages.org> wrote:

Sorry I don't follow. Isn't the point of this list to keep people informed?

On Sat, Dec 26, 2020, 1:31 PM Luis Diaz <noc@riotgames.com> wrote:

...

May want to consider moving this discussion to something more private.

On Sat, Dec 26, 2020 at 9:30 AM Ross Tajvar via Outages < outages@outages.org> wrote:

...

Latest update from AT&T:

AT&T Corporate Real Estate teams are now gathering on site after pausing work last night due a fire that reignited on site around midnight. The Fire Marshall has joined the AT&T team to provide official approval to proceed. Last night, the teams were able to accomplish testing the busbar, placing the temporary generators, completing power cabling, and opening all the breakers on the individual floors. It is estimated that this round of power related work is about 75% complete. The target is for energizing is 1200 CST. A partial analysis by the AT&T Structural Engineer initial found no concrete or steel damage that caused major concern. Once team re-engage this morning, they will complete their full analysis and ensure it is safe for restoration crews to proceed. AT&T Mechanical Engineers have confirmed there is roughly 3 feet of water in the basement with potential damage to mechanical equipment in that area. It is also suspected that commercial power may not be available for some time and as a result power will come from AT&T and Tennessee Emergency Management Agency generators. Additional technician support will be made available rotating over a 24 hour period.

Core CBB teams continue to work on the secondary plan to reroute around the Nashville 2nd Ave location. They had some success this morning at 05:05 CT when core connectivity was established for the Winchester, KY office. Partial 911 connectivity has been restored in that specific area. Work will continue on the CBB reroute plans in several locations around the Southeast Region.

We have installed a SatCOLT positioned in the area of the CO NSVLTNMT building and is up and taking traffic to aid in communication. The secondary SatCOLT arrived and has been turned up to provide additional coverage.

The next update will be issued at approximately 13:00 CT, unless there is a significant change in status

On Fri, Dec 25, 2020, 7:49 PM Ross Tajvar <ross@tajvar.io> wrote:

...

Got this internally, sharing here:

Here is the update from AT&T:

...

The front 1/3rd of their building is structuraly damaged. That

includes fiber entry.

...

They have no power and there is major flooding on all floors including the basement. The Fire Department is en route with pump trucks to help with the flooding. They have Building Engineers and Electricians on the way to look into the structure and getting power turned on. They have 2 850KW generators on the way and 2 1MW generators on standby. They have new A/C Chillers on the way (Existing Chiller plant was destroyed)

On Fri, Dec 25, 2020 at 3:38 PM Jay R. Ashworth via Outages < outages@outages.org> wrote:

...

----- Original Message -----

...

From: "Kurt Duncan via Outages" <outages@outages.org>

...

Anyone else seeing phone or internet issues due to the AT&T building being damaged by the explosion in Nashville ?

Let's please continue to confine replies here to specific information about network impacts, and keep the speculation threads to -discuss; thanks to whomever already started that thread. I would name them, but the caffeine hasn't kicked in yet.

If you're in that building in Nash, we'll be particularly interested in anything you're allowed to tell us in between bailing.

Cheers, -- jr '<admin/>' a -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

This is from 11pm. Some duplicate info here, but some other stuff as well: Latest AT&T Update: Within the Nashville Central Office, the 2nd, 3rd, 4th and 6th floors currently have power to the rectifiers. The 6th floor rectifiers are currently being energized to charge batteries. Power has been applied to the Air Conditioning risers, and Gensets are turned up. One handler is turned up on each of floors 2 and 3. Pumping of the three feet of water in the basement has been completed; however, the FBI has not yet released this portion of the building back to AT&T. It has been confirmed that the four utility service feeds are offline, and the utility transformer bus duct experienced damage. On Sat, Dec 26, 2020, 10:45 PM Walt Amason <wamason@kelleycompanies.com> wrote:

Apologies if this has been already posted.

8pm update:

All - Teams have completed their walk-through with the Fire Marshal and local authorities. Officials have given teams approval to begin restoring power to the 2nd and 3rd floors. After the 2nd and 3rd floors are energized, teams will begin working to restore power to the 1st, 4th and 5th floors. A fuel truck was brought in to refill the fuel tanks on the generators and a Caterpilar technician is onsite getting the generators ready to run. Teams will start one generator at a time. Once generators are running, teams can begin energizing power plants and charging batteries. Many batteries have reversed polarity and it is unclear at this time, if those batteries can be recharged. AT&T Structural Engineer has found no concrete or steel damage that would impair restoring power and service in the building that caused major concern.

Core CBB teams continue to work on the secondary plan to reroute around the Nashville 2nd Ave location. Service in and around the Lexington/Winchester, KY has returned for 3G and 4G service when CBB service was rerouted around the Nashville 2nd Ave location. 3G Data service is still degraded within that specific area. Work will continue with the CBB reroute plans across several locations around the Southeast Region including the Nashville, MTSO (Brick Church Pike).

Two SATCOLTs are positioned on the East and West sides of the NSVLTNMT Central Office and both are on-air and taking traffic to aid in communication. There are currently 6 assets on air – the 2 at the CO plus 4 that are supporting FN Customer Requests. There are 4 assets on route to their respective deployment locations- 2 in Alabama and 2 in TN. One asset has arrived at its intended deployment location in TN and is in the process of being turned up. Another 3 requests have been prioritized and assets that just arrived at the staging location are being deployed to their target locations in TN. Issues remain at various locations in GA, KY, & MS.

On Sat, Dec 26, 2020 at 6:08 PM Jay Ashworth via Outages < outages@outages.org> wrote:

...

It is. So unless that message that you reposted came to you with a flag line suggesting it should not be disseminated, you're welcome to post it here.

Cheers, -- jr '<admin/>' a

On December 26, 2020 1:35:29 PM EST, Ross Tajvar via Outages < outages@outages.org> wrote:

...

Sorry I don't follow. Isn't the point of this list to keep people informed?

On Sat, Dec 26, 2020, 1:31 PM Luis Diaz <noc@riotgames.com> wrote:

...

May want to consider moving this discussion to something more private.

On Sat, Dec 26, 2020 at 9:30 AM Ross Tajvar via Outages < outages@outages.org> wrote:

...

Latest update from AT&T:

AT&T Corporate Real Estate teams are now gathering on site after pausing work last night due a fire that reignited on site around midnight. The Fire Marshall has joined the AT&T team to provide official approval to proceed. Last night, the teams were able to accomplish testing the busbar, placing the temporary generators, completing power cabling, and opening all the breakers on the individual floors. It is estimated that this round of power related work is about 75% complete. The target is for energizing is 1200 CST. A partial analysis by the AT&T Structural Engineer initial found no concrete or steel damage that caused major concern. Once team re-engage this morning, they will complete their full analysis and ensure it is safe for restoration crews to proceed. AT&T Mechanical Engineers have confirmed there is roughly 3 feet of water in the basement with potential damage to mechanical equipment in that area. It is also suspected that commercial power may not be available for some time and as a result power will come from AT&T and Tennessee Emergency Management Agency generators. Additional technician support will be made available rotating over a 24 hour period.

Core CBB teams continue to work on the secondary plan to reroute around the Nashville 2nd Ave location. They had some success this morning at 05:05 CT when core connectivity was established for the Winchester, KY office. Partial 911 connectivity has been restored in that specific area. Work will continue on the CBB reroute plans in several locations around the Southeast Region.

We have installed a SatCOLT positioned in the area of the CO NSVLTNMT building and is up and taking traffic to aid in communication. The secondary SatCOLT arrived and has been turned up to provide additional coverage.

The next update will be issued at approximately 13:00 CT, unless there is a significant change in status

On Fri, Dec 25, 2020, 7:49 PM Ross Tajvar <ross@tajvar.io> wrote:

...

Got this internally, sharing here:

Here is the update from AT&T: > > The front 1/3rd of their building is structuraly damaged. That includes fiber entry. > They have no power and there is major flooding on all floors including the basement. > The Fire Department is en route with pump trucks to help with the flooding. > They have Building Engineers and Electricians on the way to look into the structure and getting power turned on. > They have 2 850KW generators on the way and 2 1MW generators on standby. > They have new A/C Chillers on the way (Existing Chiller plant was destroyed)

On Fri, Dec 25, 2020 at 3:38 PM Jay R. Ashworth via Outages < outages@outages.org> wrote:

> ----- Original Message ----- > > From: "Kurt Duncan via Outages" <outages@outages.org> > > > Anyone else seeing phone or internet issues due to the AT&T > building being > > damaged by the explosion in Nashville ? > > Let's please continue to confine replies here to specific > information about > network impacts, and keep the speculation threads to -discuss; > thanks to > whomever already started that thread. I would name them, but the > caffeine > hasn't kicked in yet. > > If you're in that building in Nash, we'll be particularly interested > in > anything you're allowed to tell us in between bailing. > > Cheers, > -- jr '<admin/>' a > -- > Jay R. Ashworth Baylink > jra@baylink.com > Designer The Things I Think > RFC 2100 > Ashworth & Associates http://www.bcp38.info 2000 > Land Rover DII > St Petersburg FL USA BCP38: Ask For It By Name! +1 > 727 647 1274 > _______________________________________________ > Outages mailing list > Outages@outages.org > https://puck.nether.net/mailman/listinfo/outages > _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

-- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

--

*Walt Amason* | Director of Information Technology | Kelley Companies *cell* (601) 382-5888 | *office* (601) 735-2541 | *website* www.kelleycompanies.com