July 2023 - Outages - Outages Lists

flapping issues reaching East Coast AWS
by William Kern 07 Aug '23

07 Aug '23

We have seen flapping all morning reaching our AWS East Coast servers from our Thousand Oaks location. They drop off for a minute or so and then come back. It seems to occur at 3-5 minute intervals for several cycles, then everything is fine for 30 minutes or so. Then the alarms start to come in again so wash rinse and repeat. The issue is definately not local, as our local monitoring to our peer connections are fine as are our connections to AWS WestCoast servers. The only complaint I got so far was from a customer on the East cost trying to VPN to a server here and getting booted out a lot. Traceroutes imply there may be an issue with Frontier as the few times I have caught the problem while logged into the remote East Coast server the path outbound from AWS East goes to Dallas and then seems to have issue reaching Los Angeles. I also show those traceroute pages moving around a bit between those locations. The connection from here to the the East Cost show weirdness involving Charter so maybe the issue is on the reverse trip. this is pretty typical. 8 lag-401.dllstx976iw-bcr00.netops.charter.com (66.109.5.229) 36.002 ms 36.316 ms lag-16.dllstx976iw-bcr00.netops.charter.com (66.109.6.1) 36.451 ms 9 lag-0.pr3.dfw10.netops.charter.com (66.109.5.121) 39.187 ms lag-302.pr3.dfw10.netops.charter.com (209.18.43.77) 42.279 ms 45.338 ms 10 99.83.71.242 (99.83.71.242) 35.427 ms 99.83.71.240 (99.83.71.240) 36.389 ms 99.82.176.170 (99.82.176.170) 35.339 ms 11 * 150.222.206.169 (150.222.206.169) 34.071 ms * 12 * * 15.230.48.42 (15.230.48.42) 33.717 ms Finally its not within AWS as our WestCoast servers can maintain a steady ping with the affected East Coast sites. So is anyone else seeing weirdness on cross country trips? William Kern PixelGate

4 6

myboostmobile.com DNS issue
by Matthew McGehrin 28 Jul '23

28 Jul '23

Hello. Reviewing some of my mail queues, I am seeing delayed emails being sent to myboostmobile.com. Checking DNS it doesn't appear to be resolving at the moment. URL: https://dns.google/query?name=myboostmobile.com > { > "Status": 2 /* SERVFAIL */, > "TC": false, > "RD": true, > "RA": true, > "AD": false, > "CD": false, > "Question": [ > { > "name": "myboostmobile.com.", > "type": 1 /* A */ > } > ], > "Comment": "Name servers refused query (lame delegation?) [156.154.130.100, 156.154.131.100]." > } URL: https://dns.google/query?name=sms.myboostmobile.com > { > "Status": 2 /* SERVFAIL */, > "TC": false, > "RD": true, > "RA": true, > "AD": false, > "CD": false, > "Question": [ > { > "name": "sms.myboostmobile.com.", > "type": 1 /* A */ > } > ], > "Comment": "Name servers refused query (lame delegation?) [156.154.130.100, 156.154.131.100]." > } Thank you.

1 0

Zayo in Amsterdam
by Jason Berenson 26 Jul '23

26 Jul '23

Just got confirmation from Zayo that they are having a widespread outage in Amsterdam which is affecting one of my circuits. BGP to the peer IP was up but broken within their network. Does anyone have any insight into what's happening? Zayo wouldn't tell me much yet.

1 0

earthlink smtp ssl error
by jimmy keffer 26 Jul '23

26 Jul '23

is anyone else seeing ssl errors from earthlink this am ?

1 0

Zayo - Western Chicago Suburbs - Packetloss
by Nate Burke 19 Jul '23

19 Jul '23

Anyone heard of Zayo issues in Chicago this week? I've had steadily increasing packet loss, now reaching about 30%, and lower ingress speed on my Zayo Circuit from various internet destinations over the last 48 hours. Especially with GRE and HTTPS traffic. I've had a ticket open with zayo since yesterday morning but they have 0 updates. I can ping the Gateway address, and another zayo address with no loss, so it does not appear to be related to the physical circuit. Outbound traffic sent via Zayo does not appear to have any issues reaching destinations. As the day has gone on, there have been more customer complaints from customers using VPN Connections, and HTTPS (O365) apps over the zayo circuit. As soon as I migrate Customer IP blocks off the Zayo circuit for ingress, customers report things improve. Between my Windstream and Zayo connection, I can get almost full bandwidth with a native iperf, but only 5mb/s through a GRE tunnel. Before 3 days ago, this would function normally. Traceroutes appear to show direct connections between zayo and my other providers, but I'm seeing the issue across multiple providers, as well as upstream from Zayo. The fact that there have been no updates from zayo lead me to believe that either they're working on a bigger problem, and don't know what it is yet, or this issue is too hard and nobody want's to tackle the ticket. Nate

1 0

DRFortress network issues in Honollulu, HI?
by Tomas Lynch 14 Jul '23

14 Jul '23

We have all of our transits down in Honolulu. We are hosted by DRFortress, they are saying that maybe Spectrum is having issues but I don't know how this will affect our providers (Cogent and GTT). Is anybody else experiencing issues there? Thanks, Tomas Lynch

2 1

Peering out of Spectrum/Charter issues in Midwest, WI Area
by Michael Peterman 12 Jul '23

12 Jul '23

I can confirm seeing the same thing. Our COLO in MKE has to go to our remote office is having to route traffic around TX and other Southern States. Inbound traffic looks ok, not great, but outbound traffic is being handed off to HE and then the high packet loss and ping mess starts Michael Peterman -----Original Message----- From: Outages <outages-bounces(a)outages.org> On Behalf Of outages-request(a)outages.org Sent: Wednesday, July 12, 2023 1:48 PM To: outages(a)outages.org Subject: Outages Digest, Vol 177, Issue 6 Send Outages mailing list submissions to outages(a)outages.org To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request(a)outages.org You can reach the person managing the list at outages-owner(a)outages.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..." Today's Topics: 1. Re: Peering out of Spectrum/Charter issues? (George Metz) 2. Re: Peering out of Spectrum/Charter issues? (Cary Wiedemann) ---------------------------------------------------------------------- Message: 1 Date: Wed, 12 Jul 2023 14:46:01 -0400 From: George Metz <george.metz(a)gmail.com> To: "Warren, David" <dwarren(a)winstead.com>, outages-discussion <outages-discussion(a)outages.org> Cc: "outages(a)outages.org" <outages(a)outages.org> Subject: Re: [outages] Peering out of Spectrum/Charter issues? Message-ID: <CANjVB-ihqcR3m=P4etKNXk=u0w=2ehk+ggQPyY7kSqB1JCwouw(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" We also were seeing this in Maine - a few minutes ago it looks like Spectrum stopped handing off to HE and all the latency and packet loss we were seeing dropped off. On Wed, Jul 12, 2023 at 2:33?PM Warren, David via Outages < outages(a)outages.org> wrote: > > > We have seen similar. We have seen increased spikes in latency and > drops as well to two of our datacenters in Texas. The common theme > for our users with high latency and drops is they are using Spectrum home internet. > Everyone not using Spectrum is not having any issues. We are mainly > located in Texas. > > > > > > *From:* Outages <outages-bounces(a)outages.org> *On Behalf Of *James > Sharp via Outages > *Sent:* Wednesday, July 12, 2023 12:50 PM > *To:* outages(a)outages.org > *Subject:* [outages] Peering out of Spectrum/Charter issues? > > > > > > We're seeing massive latency increases (normal 50-60ms, now up to > 4000-6000ms) between our customers and our datacenter in LA, mostly > from customers and end users on Spectrum. Downdetector is showing a > spike in issues. Anyone else seeing something similar? > > -- > James Sharp > > Senior Voice Engineer > > Engineering | Convoso, Inc > > [image: Image removed by sender.] > > 818-337-6365 <%20818-337-6365> > > [image: Image removed by sender.] > > jsharp(a)convoso.com > > [image: Image removed by sender.] > > www.convoso.com > > [image: Image removed by sender.] > > 22837 Ventura Blvd. Suite 300 Woodland Hills, CA 91364 > > [image: Image removed by sender.] > > > > ------------------------------ > > Information contained in this transmission is attorney privileged and > confidential. It is intended for the use of the individual or entity > named above. If the reader of this message is not the intended > recipient, you are hereby notified that any dissemination, > distribution or copy of this communication is strictly prohibited. If > you have received this communication in error, please immediately notify us by telephone. > _______________________________________________ > Outages mailing list > Outages(a)outages.org > https://puck.nether.net/mailman/listinfo/outages >

1 0

Peering out of Spectrum/Charter issues?
by James Sharp 12 Jul '23

12 Jul '23

We're seeing massive latency increases (normal 50-60ms, now up to 4000-6000ms) between our customers and our datacenter in LA, mostly from customers and end users on Spectrum. Downdetector is showing a spike in issues. Anyone else seeing something similar? -- JamesSharp Senior Voice Engineer Engineering | Convoso, Inc 818-337-6365 <tel: 818-337-6365> jsharp(a)convoso.com <mailto:jsharp@convoso.com> www.convoso.com <//www.convoso.com> 22837 Ventura Blvd. Suite 300 Woodland Hills, CA 91364

5 4

Evocative EWR1 (formerly INAP NYJ004)
by Christopher J. Pilkington 12 Jul '23

12 Jul '23

We are hearing of a site-wide emergency at Evocative EWR1, formerly INAP NYJ004, at 1 Enterprise in Secaucus NJ. We have lost both sides of power to our cage. Anyone seeing similar? Any scuttlebutt? Support is non-responsive. -cjp

3 12

Verizon Packet loss to GTT in the NE
by David 11 Jul '23

11 Jul '23

Hi all, I'm seeing some packet loss and high TTL's going from Verizon Fios PVD to GTT in EWR. My team states they are seeing packet loss to my node. Verizon hold times are over an hour at the moment. Anyone seeing anything similar: Problem hop area: 5 7 ms 8 ms 7 ms ae6.cr3-bos1.ip4.gtt.net [72.29.220.221] 6 58 ms 58 ms 63 ms ae15.cr1-nyc4.ip4.gtt.net [89.149.133.193] 1 <1 ms <1 ms <1 ms 192.168.10.1 2 4 ms 1 ms 4 ms lo0-100.PRVDRI-VFTTP-314.verizon-gni.net [108.34.146.1] 3 7 ms 7 ms 8 ms 100.41.27.146 4 * * * Request timed out. 5 7 ms 8 ms 7 ms ae6.cr3-bos1.ip4.gtt.net [72.29.220.221] 6 58 ms 58 ms 63 ms ae15.cr1-nyc4.ip4.gtt.net [89.149.133.193] 7 57 ms 55 ms 54 ms ip4.gtt.net [209.120.128.170] 8 * * * Request timed out. 9 58 ms * 55 ms 209.204.240.122 - David

2 2