Outages

Download

outages@outages.org

September 2026

4 participants
2 discussions

AT&T (AS7018) to Cloudflare (AS13335) congestion at DFW - 16% loss, 250ms+ avg, sustained
by James Bennett 20 Sep '26

20 Sep '26

Seeing sustained congestion on the AS7018 to AS13335 interconnect serving DFW, from AT&T Fiber residential in the Tulsa OK metro. AS path is 7018 13335, directly adjacent, no transit between them. Source: 76.235.9.173 (AS7018, Owasso OK) Colo: DFW, per speed.cloudflare.com/cdn-cgi/trace Window: 2026-09-20 21:27-21:45 CDT (UTC-5), and sustained for at least 2 hours prior; Cloudflare Radar suggests 90+ days TCP/443 MTR, 200 packets, sequential, figures at the FINAL destination: 104.16.0.1 Cloudflare 16.5% loss best 75.1 avg 264.5 StDev 440.7 1.1.1.1 Cloudflare 0.0% loss best 73.6 avg 256.1 StDev 450.3 151.101.13.140 Fastly 0.0% loss best 22.2 avg 23.4 StDev 0.8 8.8.8.8 Google 0.0% loss best 8.4 avg 10.0 StDev 1.1 The discontinuity is exactly at the AS boundary: 5. AS7018 32.130.107.240 0.0% 200 7.0 7.8 7.0 10.5 0.7 6. AS7018 32.130.17.1 0.0% 200 7.7 8.2 7.3 10.3 0.7 7. AS13335 141.101.74.79 0.0% 200 79.3 338.6 74.7 7209. 760.2 9. AS13335 one.one.one.one 0.0% 200 85.7 256.1 73.6 3155. 450.3 8.2ms to 338.6ms across one hop. Tulsa to Dallas is ~250 miles so even the 74ms floor is well above what the distance implies, and StDev of 450-760 looks like a saturated buffer rather than a detour. Same host to Google at the same time leaves via the adjacent AS7018 router 32.130.26.105, reaches AS15169 in 9.5ms and the destination in 10.0ms, StDev 1.1. Routing the same destinations over a commercial VPN from the same host in the same minute removes the symptom entirely: 1.1.1.1 direct 21.7% loss 78.7ms via VPN 0% loss 25.2ms 104.16.0.1 direct 21.7% loss 78.5ms via VPN 0% loss 25.2ms 8.8.8.8 direct 0.0% loss 8.2ms via VPN 0% loss 44.0ms Note the VPN makes Google 5x worse while making Cloudflare 3x better with zero loss, so this is not a case of the VPN simply being faster. Ruled out: customer equipment (reproduces on the AT&T BGW's own diagnostics, upstream of anything I own); address family (BGW has native IPv6, Cloudflare measures 72-79ms over v6 vs 77-82ms over v4, Google ~7ms on both); last mile (Google and Fastly clean on the same interface in the same minute, no interface errors, no bufferbloat). Headline numbers are TCP/443, not ICMP, so this is not control-plane rate limiting. Cloudflare's own Radar data for AS7018 shows p75 latency holding at 48-60ms for 90 days while AS701 holds at 20-22ms: https://radar.cloudflare.com/quality/as7018 Is anyone else seeing this out of DFW, and is there a capacity augment already in flight on that interconnect? -- *James Bennett* Sownley Business Solutions LLC 9905 S Pennsylvania Ave Ste A, Oklahoma City, OK 73159 US *james(a)sownley.com <james(a)sownley.com>* · *918-727-5224* Not interested? Reply "unsubscribe."

1 0

switchvox outage ?
by Izzy Goldstein - TeleGo 10 Sep '26

10 Sep '26

anyone else experiencing issues with digium / switchvox ? We're unable to spin up new servers, they all fail during the initialization steps. Existing servers also went offline, and the server gives a "500 Internal Server Error." -- Izzy Goldstein Chief Technology Officer - DevOps Master Main: (212) 477-1000 x 2085 <(212)%20477-1000> Direct: (929) 477-2085 Website: https://telego.com/ Confidentiality Notice: This e-mail may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error please notify us immediately by email reply and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of TeleGo (T). Employees of TeleGo are expressly required not to make defamatory statements and not to infringe or authorize any infringement of copyright or any other legal right by email communications. Any such communication is contrary to TeleGo policy and outside the scope of the employment of the individual concerned. TeleGo will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising. TeleGo Hosted PBX <https://youtu.be/DaT8YAZ4V0w>

4 5