December 2020 - Outages - Outages Lists

FiOS issue in NYC Area?
by Izzy Goldstein - TeleGo 16 Feb '21

16 Feb '21

Many of our clients who are using fios are reporting issues with their voip phones when i switch them to their backup ISP (Optimum etc) it works fine anyone else experiencing any fios issues ? -- Izzy Goldstein Chief Technology Officer Main: (212) 477-1000 x 2085 <(212)%20477-1000> Direct: (929) 477-2085 Website: www.telego.com <http://www.telego.net/> <http://www.telego.com/> Confidentiality Notice: This e-mail may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error please notify us immediately by email reply and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of TeleGo (T). Employees of TeleGo are expressly required not to make defamatory statements and not to infringe or authorize any infringement of copyright or any other legal right by email communications. Any such communication is contrary to TeleGo policy and outside the scope of the employment of the individual concerned. TeleGo will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising. TeleGo Hosted PBX <https://youtu.be/DaT8YAZ4V0w>

7 18

Fwd: Nashville
by Jay R. Ashworth 30 Dec '20

30 Dec '20

This is from yesterday ----- Forwarded Message ----- > From: "Eric Kuhnke" > To: "Sean Donelan" > Cc: "NANOG" <nanog(a)nanog.org> > Sent: Tuesday, December 29, 2020 6:06:00 PM > Subject: Re: Nashville > From a few days ago. Obviously centralizing lots of ss7/pstn stuff all in > one place has a long recovery time when it's physically damaged. Something > to think about for entities that own and operate traditional telco COs and > their plans for disaster recovery. > > > Nv1 > > Here is the latest update: 6:46AM 12/27: > > Work continues restoring service to the CRS routers in the Nashville > Central Office. One router remains out of service and the other is in > service with some links remaining out of service. > > The working bridge will reconvene at 08:00 CT with the following action > plan: > Additional cabling added to the first portable generator to enable full > load capabilities (08:00 CT) > Pigtails with camlocks installed for easy swap; investigate possibility to > land generator on the emergency service board to give the site N+1 with a > manual ability to choose anyone. (08:00 CT) > check small power plants on floors 4 and 6 (08:00 CT) > Investigate water damage on 1st floor and energize if safe (08:00 CT > Air handlers for floors 4,5 and 6 (09:00 CT) > complete all transport work > Turn up SS7 > Turn up 911 service - Approximately noon or after) > Turn up switching service. > TDM Switching team will reconvene at 09:00 CT and the Signaling team will > reconvene at 11:00 CT on 12/27/2020. > DMS equipment on the 1st floor will be assessed for water damage. Switching > teams will monitor power and HVAC restoration and will begin switch > restoration as soon as the go ahead is provided by the power team. > > Recovery Priorities: > 1. 4th & 5th floors (Specify transport equipment needed to clear MTSO SS7 > isolation & Datakit needed for Local Switch restoration). Transport SMEs > currently working to turn up transport equipment > 2. 6th floor (ESINET Groomers) > 3. 10th and 8th floors (N4E) – Trunks > 4. 1st floor (DMS: DS1, 5E: DS3) - Local POTS > 5. 1st floor (DMS: DS0, DS2 | 5E: DS6) – Trunks > 6. 11th floor (DMS: 01T) – Trunks > 7. 4th floor (STP and SCP with mates up in Donelson) > > The next update will be issued at approximately 09:00 CT on December 27. > > > > Nv2 > > As of 09:00 CT: Teams worked through the night to restore service and > improve conditions at the Nashville 2nd Ave Central Office. Since the > initial service impact, over 75% of the Out of Service Mobility Sites have > been restored. Certain call flows may be limited and should improve as > additional restoration activities complete. > The generator that is currently powering equipment on the 2nd and 3rd > floor, was refueled and ran with no issues through the night. Overnight, > the batteries connected to it, continued to charge. Teams have placed > additional power cables, which once connected, will allow the working > generator, to better handle the load in the building. In order to > accomplish this, the generator will need to be shut down for 15-30 minutes > this morning, so teams can connect the new cables to the system. The power > team reports they are still on target to restore power and cooling to the > 5th and 6th floor by approximately 12:00 CT. Also, a portable chiller will > be delivered this morning and strategically placed, in case it is needed to > assist in cooling the office. > > There is a Call Center at 333 Commerce, in Nashville that does not have > network or phone services available. Corporate Real Estate (CRE) reports > there is some damage to that office, but the extent of the damage will not > be known until they can gain access to the site. Because of this, the > impacted Call Center ceased operations until further notice. > > DMS switching equipment on the 1st floor will be assessed for water damage. > Switching teams will monitor power and HVAC restoration. Equipment power > ups will begin, as soon as the go ahead is provided by the power team. > > Two SatCOLTs remain positioned on the East and West sides of the NSVLTNMT > Central Office providing critical communication for teams working > restoration efforts. There are 17 assets deployed in the field- 15 are on > air (the 2 at the CO and 13 supporting FN Customer Requests) and 2 are in > hot-standby for FN Customers where macro service recently recovered. There > is 1 asset staged at a deployment site in KY where macro service restored, > and 8 additional assets are on route to Nashville today to fulfill pending > FN Customer requests. Incoming requests continue to be triaged. The ones in > areas where service looks to have been restored, are being held, while the > others are being prioritized to be dispatched upon. > > The next update will be issued at approximately 14:00 CT, unless there is a > significant change in status. > > > > Nv3 > > AT&T Nashville update below, received at 3:35PM 12/27. > > Since the initial service impact, over 95% of the Out of Service Mobility > Sites have been restored. Certain call flows may be limited and should > improve as additional restoration activities complete. > > Electricians have installed the additional power cables from the generator, > to the emergency bus. These new cables will allow the generator to support > more of the load, of the building. The portable chiller requested, has > arrived on-site, and is available to assist in cooling, if needed. > Generally speaking, there are four (4) phases of restoration per floor (Air > Handler restoral, Power restoral, Transport Equipment restoral, and > Switch/Application Equipment restoral). Teams report that Air Handlers are > up and running, and all power plants are on floors 2 through 7 are online. > Given significant progress made, floors 2 through 7, are ready for > technology turn up. Relative to Priority Transport related equipment, > approximately 90% of the elements have been turned up on floors 2 through > 7. The Power team is currently working on Floors 8 through 11 (N4E). The > first floor is not accessible, at this time. Once access is granted by > federal and local authorities, further assessment and restoration efforts > will begin. > > The generator is currently supporting approximately 50% of its capacity, > and alternative plans are being considered to handle the full load of the > building. Teams continue to work proactively in effort to identify > potential issues and are actively engaged working to restore services and > repair infrastructure. > > AT&T Network Disaster Recovery (NDR) has eleven (11) SatCOLTs in service > (TN, AL, GA). Two (2) of the eleven (11) are deployed at the Nashville, TN > Central Office to provide coverage for the AT&T response teams as well as > FirstNet (FN) customers. One (1) COLT is in hot-standby (TN). Six (6) COLTs > are en-route to deployment sites in TN and AL. Three (3) COLTs are being > demobilized in Alabama and coming back to Nashville for new assignments and > five (5) additional COLTs are en-route to the Nashville area to support > additional requests. > > The next update will be issued at approximately 19:00 CT, unless there is a > significant change in status. > > > > On Mon, Dec 28, 2020, 5:59 PM Sean Donelan <sean(a)donelan.com> wrote: > >> >> AT&T statement says nearly all services have been restore in Nashville as >> of Monday, 5pm CST >> >> They are working on permanent repairs. >> >> https://about.att.com/pages/disaster_relief/nashville.html >> >> >> AT&T's Network Disaster Recovery group faces management questions nearly >> every year to justifying their budget. While no one wants disasters, >> business continuity has to be part of the business. There are also mutual >> aid agreements between companies, but I don't know how many were invoked >> for this incident. >> >> >> https://about.att.com/ecms/dam/pages/disaster_relief/NDR_edited_04.22.19.pdf -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

1 0

Comcast and DreamHost Objects Issues
by Jiordan Villareal 28 Dec '20

28 Dec '20

Is there anyone who works on the neteng side of Comcast who can reach out to me off-list? We're not having success with the CS side concerning a potential blackhole. Much appreciated and happy holidays! Jiordan Villareal | Network Administrator | DreamHost Jiordan.Villareal(a)DreamHost.com | www.dreamhost.com o: 323-375-3849 c: 818-430-8432

1 0

Explosion in Nashville
by Kurt Duncan 26 Dec '20

26 Dec '20

Anyone else seeing phone or internet issues due to the AT&T building being damaged by the explosion in Nashville ? Thank God for the first responders working today who protect us. -- Kurt Duncan

18 21

Hurricane Electric in New York City
by Ethan Hayon 26 Dec '20

26 Dec '20

Is anyone else experiencing issues accessing specific services through Hurricane Electric in NYC? The he.net looking glass is even unable to communicate with the core1.nyc5.he.net node at this time. -- Ethan Hayon Natural Wireless

2 1

Google After-Action on the email 550 that *followed* the Big Outage
by Jay R. Ashworth 25 Dec '20

25 Dec '20

PDF, sorry: http://www.google.com/appsstatus/ir/4et50yp2ckm8otv.pdf Summary: """ Background: The Gmail SMTP inbound service uses a configuration system that allows specific service options and flags to be changed while the service is already deployed in production. The "gmail.com" domain name is specified as one of these configuration options. An ongoing migration was in effect to update this underlying configuration system to meet Google internal best practices.A configuration change during this migration shifted the formatting behavior of a service option so that it incorrectly provided an invalid domain name, instead of the intended "gmail.com" domain name, to the Google SMTP inbound service. As a result, the service incorrectly transformed lookups of certain email addresses ending in "@gmail.com" into non-existent email addresses. """ In short: "Whoops." Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

1 0

Local News coverage of Nashville
by Jay R. Ashworth 25 Dec '20

25 Dec '20

https://www.youtube.com/watch?v=iag6cTWpgq8 As usual, conversations to -discuss Cheers, -- jr '<admin/>' a -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

1 0

Re: [outages] Explosion in Nashville
by Matt Hoppes 25 Dec '20

25 Dec '20

Why would you use natural gas for your backup power??? That just causes issues like this. > On Dec 25, 2020, at 4:32 PM, Brian Knight via Outages <outages(a)outages.org> wrote: >

7 10

AT&T SF Fiber - Major Packet Loss
by Mike McLaughlin 18 Dec '20

18 Dec '20

I've been seeing 30-40% packet loss just inside AT&T's network (after an NTT and Telia handoffs) in San Francisco starting around 5:13PM PST. Specifically, it starts dropping at sffca21crs.ip.att.net (12.122.114.30). ATT Fiber DIA customer. Where the loss starts I'd guess it's a large region impacted.

5 5

random gmail bounce issues ?
by mike tancsa 15 Dec '20

15 Dec '20

Anyone else getting random bounce back issues with gmail accounts ? Was getting about 10% bounce backs on our test email that we send to from the outside. The gmail twitter feed seems to have other users around the world chiming in too and https://downdetector.ca/status/gmail/ has a big spike << 550-5.1.1 The email account that you tried to reach does not exist. Please try <<< 550-5.1.1 double-checking the recipient's email address for typos or <<< 550-5.1.1 unnecessary spaces. Learn more at <<< 550 5.1.1 https://support.google.com/mail/?p=NoSuchUser t14si83243ilq.122 - gsmtp 550 5.1.1 <xxxxxxx(a)gmail.com>... User unknown <<< 503 5.5.1 RCPT first. t14si83243ilq.122 - gsmtp

12 13