May 2023 - Outages - Outages Lists

eBay is down
by Nathanael Dick 25 May '23

25 May '23

5 5

Re: [outages] AS37468 (Angola Cables) - Route Leak?
by Phil Lavin 25 May '23

25 May '23

Return path is via NTT and looks OK: My traceroute [v0.94] http-lb-01 (88.99.88.69) -> 216.147.4.201 2023-05-25T12:34:28+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. fe-fw-01.lavin.me.uk 0.0% 19 0.2 0.1 0.1 0.2 0.0 2. 10.69.69.1 0.0% 18 0.3 0.2 0.2 0.3 0.0 3. static.225.8.4.46.clients.your-server.de 0.0% 18 0.5 0.8 0.4 5.3 1.1 4. core23.fsn1.hetzner.com 0.0% 18 0.7 1.7 0.5 9.7 2.6 5. core0.fra.hetzner.com 0.0% 18 5.2 5.1 5.1 5.2 0.0 6. 213.198.82.129 0.0% 18 8.2 5.8 5.5 8.2 0.6 7. ae-3.r20.frnkge13.de.bb.gin.ntt.net 0.0% 18 5.5 8.5 5.2 48.7 10.1 8. ae-4.r20.vienat02.at.bb.gin.ntt.net 0.0% 18 22.0 18.1 16.6 22.0 1.7 9. ae-0.r21.vienat02.at.bb.gin.ntt.net 0.0% 18 26.8 21.8 17.1 29.9 3.4 10. ae-10.r23.sngpsi07.sg.bb.gin.ntt.net 94.1% 18 162.1 162.1 162.1 162.1 0.0 11. ae-1.r00.sngpsi07.sg.bb.gin.ntt.net 0.0% 18 156.9 157.1 156.9 157.2 0.1 12. ae-0.amazon.sngpsi07.sg.bb.gin.ntt.net 0.0% 18 158.7 159.6 158.3 164.3 2.0 13. (waiting for reply) 14. (waiting for reply) 15. (waiting for reply) 16. (waiting for reply) 17. (waiting for reply) 18. (waiting for reply) 19. (waiting for reply) 20. (waiting for reply) 21. (waiting for reply) 22. (waiting for reply) 23. 216.147.4.201 70.6% 18 508.8 515.5 504.3 523.0 8.4

1 0

AS37468 (Angola Cables) - Route Leak?
by Phil Lavin 25 May '23

25 May '23

Hey Folks, Seeing massive packet loss on routes from AWS to Hetzner today. First was AWS USW2 -> Hetzner (e.g. 88.99.88.69). Traffic was transiting via AS37468 (Angola Cables) in Coresite IX. Now got loss from AWS apse1/apse2. Traffic transiting via AS37468 on Equinix IX. I e-mailed Hetzner NOC but they don’t seem entirely bothered, currently. My traceroute [v0.94] jcasc-rtc01 (10.36.21.252) -> phil.lavin.me.uk 2023-05-25T11:50:13+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ip-10-36-17-199.ap-southeast- 0.0% 18 0.1 0.6 0.1 7.7 1.8 2. ec2-18-141-171-15.ap-southeas 0.0% 18 4.3 17.5 0.9 95.7 24.0 ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com 3. (waiting for reply) 4. (waiting for reply) 5. (waiting for reply) 6. (waiting for reply) 7. (waiting for reply) 8. 100.65.10.161 0.0% 17 0.6 1.4 0.3 9.6 2.4 100.65.11.65 9. 150.222.108.75 0.0% 17 2.1 3.0 1.0 13.1 3.3 52.93.10.74 10. 150.222.108.82 0.0% 17 2.2 2.4 1.3 10.2 2.1 11. 52.93.11.115 0.0% 17 5.8 2.4 1.2 9.7 2.5 52.93.10.185 12. 37468-sg1-ix.equinix.com 68.8% 17 268.6 279.0 262.4 312.8 19.8 13. 102.219.127.3 93.8% 17 420.0 420.0 420.0 420.0 0.0 14. (waiting for reply) 15. 195.66.227.209 62.5% 17 467.9 456.9 440.3 488.2 18.5 16. core6.par.hetzner.com 75.0% 17 455.6 464.1 455.6 486.3 14.8 17. core11.nbg1.hetzner.com 75.0% 17 459.5 467.5 445.9 497.3 21.8 18. core23.fsn1.hetzner.com 68.8% 17 478.4 441.0 390.0 478.4 32.0 19. ex9k1.dc1.fsn1.hetzner.com 87.5% 17 460.4 454.2 448.1 460.4 8.7 20. vmh02.lavin.me.uk 75.0% 17 433.3 424.2 382.7 441.9 27.9 21. (waiting for reply) 22. http-lb-01.lavin.me.uk 75.0% 17 436.8 438.6 385.0 493.3 44.2 I’ve been out of the BGP game for a few years now. Anybody have visibility of the extent of the leaks? Phil

4 3

TDS - Fiber Cut in Eastern Wisconsin
by Gregory Strike 24 May '23

24 May '23

As of 2:20 p.m. May 24, TDS is aware of an outage impacting TDS services in Eastern Wisconsin, including the Fond du Lac, Appleton, and Green Bay areas due to a fiber cut. Some of our customers have been experiencing high amounts of packet loss as and service outages a result. Greg

1 0

sea-us cable outage
by scott 17 May '23

17 May '23

Hey, Does anyone have any good info on the SEA-US cable outage? Getting crap information from all directions. "it's somewhere between Guam and LA. That's only 4000 miles of cable. Thanks!

1 0

Spectrum - Midwest US residential
by vom513 10 May '23

10 May '23

Anyone had issues the past 2 days ? I have Spectrum in Cincinnati, OH and for the past 2 days or so I’ve had sites not loading, my monitoring systems alerting etc. Seems to have been both v4+v6 (based on the alerts I got). Wish I could catch it with a traceroute / RIPE atlas test but I haven’t been able to yet. My guess is it’s something backbone/peering related. I say this because when there is an OSP / last mile / CMTS type outage - Spectrum has been pretty good lately about pushing an alert to their app. No trucks around that I’ve seen, cable modem hasn’t rebooted, levels all about the same etc. I also called as usually they will have a message saying that they know about an outage in the area. None of this this time. I feel like those alerts only take into account “last mile” type issues. I’m currently on our backup ISP and I will try again tomorrow. Thanks.

3 2

Lumen/Cirion
by Mehmet Akcin 10 May '23

10 May '23

Hello We are seeing massive issues from Brazil, Argentina, Peru, Colombia, toward Europe/US/Asia between former Level3 now separate companies Lumen(USA) vs Cirion(LATAM) Anyone else seeing routes not announced, and other issues from Latin America beyond? Mehmet -- Mehmet +1-424-298-1903

4 3

Twitter down-ish?
by Andy Ringsmuth 04 May '23

04 May '23

Tried from multiple ISPs in Lincoln, Neb. Twitter loads, but goes to a “Nothing to see here. Looks like this page doesn’t exist.” That’s for the main twitter.com <http://twitter.com/> homepage or looking at any person’s feed. PLEASE lets not do a bunch of “me too” replies to outages. I set the “Reply-To” to go to -discussion. Sent at 9:10 a.m. CDT to also check list response time. ---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 202-1230 andy(a)andyring.com

1 0

OVH Canada change invoice date on customer server accounts and suspended thousands of customers today at 0800 for none Payment
by Jeff Kilpatrick 02 May '23

02 May '23

OVH Canada changed invoice data on customer server accounts and suspended thousands of customers today at 0800 for none Payment. as of 11:45 services still not restored despite promises by staff to resort service in 'a few minutes'. -- Jeff Kilpatrick

2 1

Lumen outage DFW market. 2023-05-01 14:17:39 GMT
by Joseph Jackson 01 May '23

01 May '23

Outage on the Lumen IP Transit network in DFW this morning. Update from Lumen NOC - 2023-05-01 14:17:39 GMT - OnMay 01, 2023, at 14:02 GMT Lumen identified a service impact in Dallas, TX. As this network fault is impacting multiple clients, the event has increased visibility with Lumen leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority. I know its impacting our circuits in 2323 Bryan. Hope everyone else is having a lovely Monday. Joseph

1 0