June 2021 - Outages - Outages Lists

Broadview networks - no outgoing calls?
by Izzy Goldstein - TeleGo 30 Jun '21

30 Jun '21

herein the East Cost - NYC Area several of our broadview networks Circuits are experiencing issues making outgoing calls -- Izzy Goldstein Chief Technology Officer Main: (212) 477-1000 x 2085 <(212)%20477-1000> Direct: (929) 477-2085 Website: www.telego.com <http://www.telego.net/> <http://www.telego.com/> Confidentiality Notice: This e-mail may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error please notify us immediately by email reply and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of TeleGo (T). Employees of TeleGo are expressly required not to make defamatory statements and not to infringe or authorize any infringement of copyright or any other legal right by email communications. Any such communication is contrary to TeleGo policy and outside the scope of the employment of the individual concerned. TeleGo will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising. TeleGo Hosted PBX <https://youtu.be/DaT8YAZ4V0w>

1 0

Dallas AT&T outage
by Mike Lewinski 28 Jun '21

28 Jun '21

We have more than two dozen AT&T circuits down in Dallas. First we were told no trouble found and had our ticket set to auto-close. Then we were told cause is a failed line card on 100Gbps transport. Now we're told the 100Gbps outage is a fiber cut, ETR is 10:30am on Tues. June 29. The thing is, it's not affecting every NNI we have at the Infomart site. Is anyone else in the Dallas area experiencing an AT&T outage? Mike

1 0

Mobile calling issues - Michigan/Detroit
by Anthony Concepcion 24 Jun '21

24 Jun '21

Anyone seeing issues with mobile calls not completing in the Detroit/Rochester Hills area? We're receiving reports of calls not completing to our network from tmobile/sprint..

5 4

Re: [outages] Outages Digest, Vol 157, Issue 13
by Corey Caruso 24 Jun '21

24 Jun '21

We are seeing calls to toll free fail from Verizon Wireless via Lumen SIP. T-Mobile and ATT are fine. DIDs are fine. ________________________________ From: Outages <outages-bounces(a)outages.org> on behalf of outages-request(a)outages.org <outages-request(a)outages.org> Sent: Thursday, June 24, 2021 9:51:41 AM To: outages(a)outages.org <outages(a)outages.org> Subject: Outages Digest, Vol 157, Issue 13 Send Outages mailing list submissions to outages(a)outages.org To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request(a)outages.org You can reach the person managing the list at outages-owner(a)outages.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..." Today's Topics: 1. Re: MS Office 365 Access Issues (Jeff Wozniak) 2. Frontier in central FL having issues? (Douglas Haber) 3. Mobile calling issues - Michigan/Detroit (Anthony Concepcion) 4. Re: Mobile calling issues - Michigan/Detroit (Ufer, Robert) ---------------------------------------------------------------------- Message: 1 Date: Mon, 21 Jun 2021 14:46:41 -0500 From: Jeff Wozniak <jeffwozniak(a)gmail.com> To: Nathan C <master2004134(a)gmail.com> Cc: Ben Mize <bmize(a)corp.tecinfo.net>, outages(a)outages.org Subject: Re: [outages] MS Office 365 Access Issues Message-ID: <CAOiuhjr7HrKEtzX+bpmVTTcsL2DALLk9yo8ECZQPifu92rO2wg(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" Same, coming out of MKE. On Mon, Jun 21, 2021, 2:33 PM Nathan C via Outages <outages(a)outages.org> wrote: > Working great for me from up in Maine, for what it's worth. > > Nate > > On Mon, Jun 21, 2021 at 3:06 PM Ben Mize via Outages <outages(a)outages.org> > wrote: > >> Anyone having an issue using the MS Teams desktop application or reaching >> teams.microsoft.com? >> >> >> >> We are able to ping the name teams.microsft.com, however we cannot >> access the webpage for Teams login. >> >> >> >> Just curious if this is us or a CDN/upstream peer. >> >> >> >> Thanks, >> >> >> >> *Ben Mize* >> >> *Operations Director* >> >> TecInfo Communications >> >> bmize(a)corp.tecinfo.net >> >> 662.686.9009 >> >> >> >> http://www.tecinfo.net/ >> >> >> >> The information transmitted is intended only for the person or entity to >> which it is addressed and may contain confidential, proprietary, and/or >> privileged material. Any review, retransmission, dissemination or other use >> of, or taking of any action in reliance upon this information by persons or >> entities without the written permission of TecInfo is prohibited. If you >> received this in error, please contact the sender and delete the material >> from all computers. >> >> >> >> >> _______________________________________________ >> Outages mailing list >> Outages(a)outages.org >> https://puck.nether.net/mailman/listinfo/outages >> > _______________________________________________ > Outages mailing list > Outages(a)outages.org > https://puck.nether.net/mailman/listinfo/outages >

1 0

Frontier in central FL having issues?
by Douglas Haber 22 Jun '21

22 Jun '21

I just had two buildings in Plant City, FL on the same road go down about 10 minutes ago (~913 AM). Only buildings I service in that county. Anyone else have exposure to Frontier? Maybe it's just we're lucky there? -- Douglas Haber Managing Member Garden State Computing, LLC Tel: (973) 636-7350 www.gardenstatecomputing.com<http://www.gardenstatecomputing.com/> support(a)gardenstatecomputing.com<mailto:support@gardenstatecomputing.com>

1 0

MS Office 365 Access Issues
by Ben Mize 21 Jun '21

21 Jun '21

Anyone having an issue using the MS Teams desktop application or reaching teams.microsoft.com? We are able to ping the name teams.microsft.com, however we cannot access the webpage for Teams login. Just curious if this is us or a CDN/upstream peer. Thanks, Ben Mize Operations Director TecInfo Communications bmize(a)corp.tecinfo.net<mailto:bmize@corp.tecinfo.net> 662.686.9009 http://www.tecinfo.net/ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential, proprietary, and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities without the written permission of TecInfo is prohibited. If you received this in error, please contact the sender and delete the material from all computers.

3 2

Re: [outages] Verizon Internet extreme slowness
by Matthew Huff 21 Jun '21

21 Jun '21

Our peering is at Equinix NY2 in Secaucus NJ. We are still having issues with traffic crossing Verizon even if the source and destination are all within Verizon. The dropped packets impacting VoIP and IPSEC appear to be resolved, but we are having extreme slowness access Websites directly on Verizon or CDNS peered on Verizon. Sites such as tdbank, Yahoo, etc are so slow as to be unusable, even Verizon’s Enterprise support portal, while sites external to Verizon appear to load fine. Verizon is claiming that it is related to a fire in Flatbush, Brooklyn and should be resolved, but we are still seeing issues since last Thursday. Anyone else seeing this? Matthew Huff | Director of Technical Operations | OTA Management LLC Office: 914-460-4039 mhuff(a)ox.com<mailto:mhuff@ox.com> | www.ox.com<http://www.ox.com> ........................................................................................................................................... From: steve <steve(a)nysys.com> Sent: Friday, June 18, 2021 8:34 AM To: Matthew Huff <mhuff(a)ox.com>; outages(a)outages.org Subject: RE: [outages] Verizon Internet extreme slowness We are still down in upstate NY. Anyone else? Sent from my Verizon, Samsung Galaxy smartphone -------- Original message -------- From: Matthew Huff via Outages <outages(a)outages.org<mailto:outages@outages.org>> Date: 6/18/21 7:00 AM (GMT-05:00) To: outages(a)outages.org<mailto:outages@outages.org> Subject: [outages] Verizon Internet extreme slowness We are directly peered with Verizon (AS 701) and we started seeing issues with "slowness" yesterday even to other IPs within AS701. Overnight the performance dropped off a cliff. Data transfers that normal happen at ~3MBs dropped to 18KBs. I stopped our BGP advertisement via AS701 and setup a route filter to block Verizon's routes and our data is back to normal. Anyone else here any reports of Verizon issues? Matthew Huff | Director of Technical Operations | OTA Management LLC Office: 914-460-4039 mhuff(a)ox.com<mailto:mhuff@ox.com> | www.ox.com<http://www.ox.com> ........................................................................................................................................... _______________________________________________ Outages mailing list Outages(a)outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages

1 0

Verizon Internet extreme slowness
by Matthew Huff 18 Jun '21

18 Jun '21

We are directly peered with Verizon (AS 701) and we started seeing issues with "slowness" yesterday even to other IPs within AS701. Overnight the performance dropped off a cliff. Data transfers that normal happen at ~3MBs dropped to 18KBs. I stopped our BGP advertisement via AS701 and setup a route filter to block Verizon's routes and our data is back to normal. Anyone else here any reports of Verizon issues? Matthew Huff | Director of Technical Operations | OTA Management LLC Office: 914-460-4039 mhuff(a)ox.com | www.ox.com ...........................................................................................................................................

2 1

Spectrum DFW routing issues
by David Eddleman 15 Jun '21

15 Jun '21

Looks like there's a router in Dallas that's dropping certain routes. 96.34.3.69 seems to have issues forwarding certain connections along. C:\Users\David>tracert 63.97.76.216 Tracing route to 63.97.76.216 over a maximum of 30 hops 1 <1 ms 1 ms <1 ms 192.168.1.1 2 * * * Request timed out. 3 10 ms 10 ms 10 ms 159-111-145-012.res.spectrum.com [159.111.145.12] 4 13 ms 14 ms 10 ms crr01ftwotx-bue-4.ftwo.tx.charter.com [96.34.113.116] 5 14 ms 11 ms 15 ms bbr01dllstx-bue-2.dlls.tx.charter.com [96.34.2.32] 6 12 ms 11 ms 12 ms prr01dllstx-bue-3.dlls.tx.charter.com [96.34.3.69] 7 * * * Request timed out. 8 * * * Request timed out. 9 * * * Request timed out. And just to prove it's not a one-network issue: C:\Users\David>tracert 204.2.229.106 Tracing route to 204.2.229.106 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms 192.168.1.1 2 * * * Request timed out. 3 18 ms 9 ms 9 ms 159-111-145-012.res.spectrum.com [159.111.145.12] 4 10 ms 12 ms 15 ms crr01ftwotx-bue-4.ftwo.tx.charter.com [96.34.113.116] 5 26 ms 15 ms 22 ms bbr01dllstx-bue-2.dlls.tx.charter.com [96.34.2.32] 6 12 ms 13 ms 12 ms prr01dllstx-bue-3.dlls.tx.charter.com [96.34.3.69] 7 * * * Request timed out. 8 * * * Request timed out. 9 * * * Request timed out. C:\Users\David>tracert 204.2.229.88 Tracing route to 204.2.229.88 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms 192.168.1.1 2 * * * Request timed out. 3 10 ms 17 ms 9 ms 159-111-145-012.res.spectrum.com [159.111.145.12] 4 11 ms 10 ms 10 ms crr01ftwotx-bue-4.ftwo.tx.charter.com [96.34.113.116] 5 17 ms 15 ms 15 ms bbr01dllstx-bue-2.dlls.tx.charter.com [96.34.2.32] 6 14 ms 15 ms 13 ms prr01dllstx-bue-3.dlls.tx.charter.com [96.34.3.69] 7 * * * Request timed out. 8 * * * Request timed out. 9 * * * Request timed out. C:\Users\David>tracert 195.82.50.9 Tracing route to 195.82.50.9 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms 192.168.1.1 2 * * * Request timed out. 3 9 ms 9 ms 8 ms 159-111-145-012.res.spectrum.com [159.111.145.12] 4 14 ms 17 ms 15 ms crr01ftwotx-bue-4.ftwo.tx.charter.com [96.34.113.116] 5 14 ms 15 ms 15 ms bbr01dllstx-bue-2.dlls.tx.charter.com [96.34.2.32] 6 11 ms 11 ms 11 ms prr01dllstx-bue-3.dlls.tx.charter.com [96.34.3.69] 7 * * * Request timed out. 8 * * * Request timed out. 9 * * * Request timed out. Bizarrely, portions of some of those networks are reachable. C:\Users\David>tracert 204.2.229.107 Tracing route to 204.2.229.107 over a maximum of 30 hops 1 1 ms 4 ms <1 ms 192.168.1.1 2 * * * Request timed out. 3 10 ms 15 ms 17 ms 159-111-145-012.res.spectrum.com [159.111.145.12] 4 15 ms 12 ms 9 ms crr01ftwotx-bue-4.ftwo.tx.charter.com [96.34.113.116] 5 13 ms 15 ms 15 ms bbr01dllstx-bue-2.dlls.tx.charter.com [96.34.2.32] 6 16 ms 11 ms 11 ms prr01dllstx-bue-3.dlls.tx.charter.com [96.34.3.69] 7 10 ms 12 ms 10 ms dls-b23-link.ip.twelve99.net [62.115.180.82] 8 11 ms 10 ms 10 ms ntt-ic325660-dls-b23.ip.twelve99-cust.net [62.115.156.249] 9 11 ms 12 ms 16 ms ae-7.r25.dllstx09.us.bb.gin.ntt.net [129.250.5.25] 10 49 ms 47 ms 48 ms ae-2.r25.snjsca04.us.bb.gin.ntt.net [129.250.4.154] 11 47 ms 75 ms 51 ms ae-5.r24.snjsca04.us.bb.gin.ntt.net [129.250.3.146] 12 52 ms 50 ms 54 ms ae-4.r00.scrmca02.us.bb.gin.ntt.net [129.250.7.57] 13 53 ms 52 ms 67 ms xe-0-1-0-1-1.r00.scrmca02.us.ce.gin.ntt.net [192.80.16.2] 14 51 ms * 51 ms 204.2.229.234 15 52 ms 53 ms 50 ms 204.2.229.107 Trace complete. Already let Spectrum know after having to jump through some hoops, but if anyone is experiencing similar behavior let me know and I can give you a contact to reach out to. David Eddleman

1 0

DNS SERVFAIL for nist.gov
by Matthew Huff 15 Jun '21

15 Jun '21

We have to query and compare against NIST time servers for FINRA compliance. This morning I noticed our systems are unable to DNS query the NIST time servers. Neither our local resolvers or google (8.8.8.8) work. [root@bacall log]# dig @8.8.8.8 time-a-g.nist.gov ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 time-a-g.nist.gov ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36018 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;time-a-g.nist.gov. IN A ;; Query time: 6 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Mon Jun 14 06:27:45 EDT 2021 ;; MSG SIZE rcvd: 46 [root@bacall log]# dig @8.8.8.8 nist.gov in soa ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 nist.gov in soa ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17779 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;nist.gov. IN SOA ;; Query time: 5 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Mon Jun 14 06:31:59 EDT 2021 ;; MSG SIZE rcvd: 37 The time servers are documented here: https://tf.nist.gov/tf-cgi/servers.cgi Using the IP addresses work, it look like the nist.gov domain is offline. Matthew Huff | Director of Technical Operations | OTA Management LLC Office: 914-460-4039 mhuff(a)ox.com | www.ox.com ...........................................................................................................................................

7 10