January 2024 - Outages - Outages Lists

AT&T Fiber Partial Outage
by James Hartig via Outages 26 Jan '24

26 Jan '24

I have AT&T fiber in the Chicago area and I'm having trouble reaching several different sites, including AWS, GitHub, OVH, and some DigitalOcean DCs. I don't seem to have any issues with GCP, Cloudflare, Facebook, Twitter. traceroute to cognito-idp.eu-central-1.amazonaws.com (3.67.102.134), 30 hops max, 60 byte packets 1 192.168.0.1 0.750 ms 0.742 ms 0.737 ms 2 192.168.1.254 1.284 ms 1.459 ms 1.790 ms 3 107.207.60.1 2.451 ms 2.446 ms 2.441 ms 4 71.151.17.222 2.694 ms 2.727 ms 2.379 ms 5 12.242.114.41 8.454 ms 4.815 ms 9.285 ms 6 * * * traceroute to proof.ovh.net (141.95.207.211), 30 hops max, 60 byte packets 1 192.168.0.1 0.923 ms 0.723 ms 0.916 ms 2 192.168.1.254 1.279 ms 1.473 ms 1.766 ms 3 107.207.60.1 2.491 ms 2.486 ms 2.781 ms 4 71.151.17.222 2.455 ms 2.810 ms 2.807 ms 5 12.242.114.41 10.741 ms 10.013 ms 6.730 ms 6 * * * I don't see any outages listed for my area in AT&T but I can still access a lot of the Internet so I don't believe it's a local outage. _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

3 3

Re: [outages] Twelve99 / AWS usw2 significant loss
by Phil Lavin via Outages 26 Jan '24

26 Jan '24

Thanks, ytti. I have raised a case with AWS but I expect it to be as unproductive as usual. In this type of issue, there is often a friendly Engineer keeping an eye on NANOG list or IRC channel who can mitigate it. Hoping that to be the case this time, also. > On 26 Jan 2024, at 08:40, Saku Ytti <saku(a)ytti.fi> wrote: > > On Fri, 26 Jan 2024 at 10:23, Phil Lavin via NANOG <nanog(a)nanog.org> wrote: > > >> 88.99.88.67 to 216.147.3.209: >> Host Loss% Snt Last Avg Best Wrst StDev >> 1. 10.88.10.254 0.0% 176 0.2 0.1 0.1 0.3 0.1 >> 7. nug-b1-link.ip.twelve99.net 0.0% 176 3.3 3.5 3.1 24.1 1.6 >> 8. hbg-bb2-link.ip.twelve99.net 86.9% 175 18.9 18.9 18.7 19.2 0.1 >> 9. ldn-bb2-link.ip.twelve99.net 92.0% 175 30.5 30.6 30.4 30.8 0.1 >> 10. nyk-bb1-link.ip.twelve99.net 4.6% 175 99.5 99.5 99.3 100.1 0.2 >> 11. sjo-b23-link.ip.twelve99.net 56.3% 175 296.8 306.0 289.7 315.0 5.5 >> 12. amazon-ic-366608.ip.twelve99-cust.net 80.5% 175 510.0 513.5 500.7 539.7 8.4 > > This implies the problem is not on this path, because #10 is not > experiencing it, possibly because it happens to return a packet via > another option, but certainly shows the problem didn't happen in this > direction yet at #10, but because #8 and #9 saw it, they already saw > it on the other direction. > > >> 44.236.47.236 to 178.63.26.145: >> Host Loss% Snt Last Avg Best Wrst StDev >> 1. ip-10-96-50-153.us-west-2.compute.internal 0.0% 267 0.2 0.2 0.2 0.4 0.0 >> 11. port-b3-link.ip.twelve99.net 0.0% 267 5.8 5.9 5.6 11.8 0.5 >> 12. palo-b24-link.ip.twelve99.net 4.9% 267 21.1 21.5 21.0 58.4 3.1 >> 13. sjo-b23-link.ip.twelve99.net 0.0% 266 21.4 22.7 21.3 86.2 6.5 >> 14. nyk-bb1-link.ip.twelve99.net 58.1% 266 432.7 422.7 407.2 438.5 6.5 >> 15. ldn-bb2-link.ip.twelve99.net 98.1% 266 485.6 485.4 481.6 491.1 3.9 >> 16. hbg-bb2-link.ip.twelve99.net 92.5% 266 504.1 499.8 489.8 510.1 5.9 >> 17. nug-b1-link.ip.twelve99.net 55.5% 266 523.5 519.6 504.4 561.7 7.6 >> 18. hetzner-ic-340780.ip.twelve99-cust.net 53.6% 266 524.4 519.2 506.0 545.5 6.9 >> 19. core22.fsn1.hetzner.com 70.2% 266 521.7 519.2 498.5 531.7 6.6 >> 20. static.213-239-254-150.clients.your-server.de 33.2% 266 382.4 375.4 364.9 396.5 4.1 >> 21. static.145.26.63.178.clients.your-server.de 62.0% 266 529.9 518.4 506.9 531.3 6.1 > > This suggests the congestion point is from sjo to nyk, in 1299, not AWS at all. > > You could try to fix SPORT/DPORT, and do several SPORT options, to see > if loss goes away with some, to determine if all LAG members are full > or just one. > > > At any rate, this seems business as usual, sometimes internet is very > lossy, you should contact your service provider, which I guess is AWS > here, so they can contact their service provider or 1299. > > -- > ++ytti _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

2 1

Microsoft Teams appears to be having issues ...
by Bryan Holloway via Outages 26 Jan '24

26 Jan '24

https://downdetector.com/status/teams/ _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

5 4

Office365 issues
by Bruce Freshwater via Outages 26 Jan '24

26 Jan '24

Above and beyond Teams out of order messages and delays, we now seeing reports of mail bounces from with multiple customers: [cid:image001.png@01DA5069.3009D5B0] Regards, Bruce Freshwater Direct: 412.722.1701 Mobile: 412.292.0282 Main: 866.707.5869 Fax: 412.774.2469 Email: BFreshwater(a)SierraExperts.com<mailto:BFreshwater@SierraExperts.com> Request Help: Support Portal (SIM)<sim.sierraincidentmanager.com> Sierra Experts [cid:image002.jpg@01D1C645.CF870330] 2 Robinson Plaza, Suite 300, Pittsburgh, PA 15205-1017 Our Services: Managed Services<http://www.sierraexperts.com/managed-services/> | Development & Design<http://www.sierraexperts.com/development-and-design/> | Telephony & VoIP<http://www.sierraexperts.com/telephony-and-voip/> | Hardware & Software<http://www.sierraexperts.com/hardware-and-software/> | Data Center<http://sierradatacenters.com/> Stay Connected: Website<http://www.sierraexperts.com/> | Awards<http://www.sierraexperts.com/about-us/awards/> | Facebook<https://www.facebook.com/SierraExperts> | X<https://twitter.com/sierraexperts> | LinkedIn<https://www.linkedin.com/company/sierraexperts> | Blog<http://www.sierraexperts.com/about-us/blog/> [cid:image001.png@01D573C1.8B84DB80] _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

1 0

Twelve99 / AWS usw2 significant loss
by Phil Lavin via Outages 26 Jan '24

26 Jan '24

Hi Folks, Cross posting to outages and nanog. Can anybody from Twelve99 / AWS investigate high loss and latency around SJO between Twelve99 and AWS? Traces below: 88.99.88.67 to 216.147.3.209: My traceroute [v0.94] phil (10.88.10.10) -> 216.147.3.209 2024-01-26T08:16:16+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 10.88.10.254 0.0% 176 0.2 0.1 0.1 0.3 0.1 2. fe-fw-01.lavin.me.uk 0.0% 176 0.4 0.4 0.2 0.7 0.1 3. 10.69.69.1 0.0% 176 0.5 0.4 0.2 0.7 0.1 4. static.225.8.4.46.clients.your-server.de 0.0% 176 0.8 1.0 0.5 8.2 1.0 5. core22.fsn1.hetzner.com 0.6% 176 0.8 2.0 0.7 39.0 4.6 6. static.213-239-254-234.clients.your-server.de 0.0% 176 3.2 3.8 3.1 36.5 3.3 7. nug-b1-link.ip.twelve99.net 0.0% 176 3.3 3.5 3.1 24.1 1.6 8. hbg-bb2-link.ip.twelve99.net 86.9% 175 18.9 18.9 18.7 19.2 0.1 9. ldn-bb2-link.ip.twelve99.net 92.0% 175 30.5 30.6 30.4 30.8 0.1 10. nyk-bb1-link.ip.twelve99.net 4.6% 175 99.5 99.5 99.3 100.1 0.2 11. sjo-b23-link.ip.twelve99.net 56.3% 175 296.8 306.0 289.7 315.0 5.5 12. amazon-ic-366608.ip.twelve99-cust.net 80.5% 175 510.0 513.5 500.7 539.7 8.4 13. (waiting for reply) …... 26. 216.147.3.209 59.1% 150 534.2 534.4 513.5 549.7 8.1 44.236.47.236 to 178.63.26.145: jcasc-rtc01 (10.96.54.240) -> hc-ping.com 2024-01-26T08:10:24+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ip-10-96-50-153.us-west-2.compute.internal 0.0% 267 0.2 0.2 0.2 0.4 0.0 2. ec2-34-221-151-225.us-west-2.compute.amazonaw 0.0% 267 6.8 15.9 0.7 114.1 24.1 3. (waiting for reply) 4. (waiting for reply) 5. (waiting for reply) 6. 241.0.2.5 0.0% 267 0.4 0.4 0.3 1.2 0.1 7. 242.0.42.195 0.0% 267 1.2 1.7 0.5 23.9 3.0 8. 240.2.144.12 0.0% 267 5.2 5.2 5.2 27.4 1.4 9. amazon-ic-375418.ip.twelve99-cust.net 0.0% 267 5.3 5.2 5.2 6.1 0.1 10. port-b4-link.ip.twelve99.net 0.0% 267 5.3 5.5 4.9 35.5 3.0 11. port-b3-link.ip.twelve99.net 0.0% 267 5.8 5.9 5.6 11.8 0.5 12. palo-b24-link.ip.twelve99.net 4.9% 267 21.1 21.5 21.0 58.4 3.1 13. sjo-b23-link.ip.twelve99.net 0.0% 266 21.4 22.7 21.3 86.2 6.5 14. nyk-bb1-link.ip.twelve99.net 58.1% 266 432.7 422.7 407.2 438.5 6.5 15. ldn-bb2-link.ip.twelve99.net 98.1% 266 485.6 485.4 481.6 491.1 3.9 16. hbg-bb2-link.ip.twelve99.net 92.5% 266 504.1 499.8 489.8 510.1 5.9 17. nug-b1-link.ip.twelve99.net 55.5% 266 523.5 519.6 504.4 561.7 7.6 18. hetzner-ic-340780.ip.twelve99-cust.net 53.6% 266 524.4 519.2 506.0 545.5 6.9 19. core22.fsn1.hetzner.com 70.2% 266 521.7 519.2 498.5 531.7 6.6 20. static.213-239-254-150.clients.your-server.de 33.2% 266 382.4 375.4 364.9 396.5 4.1 21. static.145.26.63.178.clients.your-server.de 62.0% 266 529.9 518.4 506.9 531.3 6.1 Thanks _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

1 1

bgp.he.net down?
by Hank Nussbacher via Outages 16 Jan '24

16 Jan '24

Anyone know why bgp.he.net is down? I get: We're sorry, but something went wrong. We've been notified about this issue and we'll take a look at it shortly. Thank, Hank _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

9 10

Shaw High Packet Loss
by Lyndon Nerenberg (VE7TFX/VE6BBM) via Outages 15 Jan '24

15 Jan '24

For the last hour I've been seeing high packet loss rates inside Shaw's western Canada network. E.g.: --- orthanc.ca ping statistics --- 152 packets transmitted, 8 packets received, 94.7% packet loss round-trip min/avg/max/stddev = 109.441/285.678/696.621/194.853 ms That's from Vancouver residential to a colo server in L.A. A typical traceroute shows the choke point seems to be Calgary: traceroute to orthanc.ca (208.79.93.154), 64 hops max, 72 byte packets 1 * 192.168.0.1 (192.168.0.1) 20.742 ms 4.571 ms 2 24.86.192.1 (24.86.192.1) 15.915 ms 12.080 ms 16.014 ms 3 rc1bb-be147-1.vc.shawcable.net (64.59.150.185) 11.852 ms 16.537 ms 31.073 ms 4 24.244.59.217 (24.244.59.217) 12.509 ms 12.295 ms 12.035 ms 5 24.83.252.66 (24.83.252.66) 11.529 ms 24.933 ms 197.457 ms 6 rc1st-be214.vc.shawcable.net (24.244.58.65) 13.587 ms 10.923 ms 104.439 ms 7 * * rc3no-be11-1.cg.shawcable.net (66.163.72.69) 58.938 ms 8 rc3so-be23.cg.shawcable.net (66.163.71.118) 50.948 ms 27.276 ms 25.755 ms 9 66.163.71.42 (66.163.71.42) 24.937 ms 30.755 ms 23.542 ms 10 rc4ec-be13.il.shawcable.net (66.163.65.18) 96.292 ms * 90.125 ms 11 equinix-ch2.cr2.ord1.amcbb.net (208.115.136.124) 53.652 ms * * 12 ae1-4.cr1.ord1.atlanticmetro.net (174.128.183.2) 173.172 ms * 248.078 ms 13 ve521.cr2.lax0.atlanticmetro.net (208.78.26.210) 126.758 ms * * 14 xe-0-0-3-0.cr2.lax2.atlanticmetro.net (208.78.31.37) 104.574 ms 234.672 ms * 15 * ae3-4.cr2.lax3.atlanticmetro.net (173.205.176.40) 114.315 ms * 16 arpnetworks-dmarc.lax3.atlanticmetro.net (173.205.184.132) 179.245 ms * * 17 * orthanc.ca (208.79.93.154) 191.936 ms 102.680 ms Anyone else seeing this? Or know what's going on? --lyndon _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

3 3

Equinix CH1 Overheating
by Mike Hammett via Outages 14 Jan '24

14 Jan '24

Equinix CH1 is overheating due to some chiller failures. The vendor is on-site troubleshooting. Ambient temperature is currently 28 degrees higher than normal. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

5 8

Spectrum latency in SATX area
by William Peebles via Outages 03 Jan '24

03 Jan '24

We have a few remote employees with Spectrum in different locations in the San Antonio area who have had a lot of VPN disconnects today due to high latency between their machines and our firewall in Boston. Spectrum customer service gave them the typical run-around speech about how it's our problem and not theirs, but even Google and Cloudflare are giving higher pings than usual Pinging 8.8.8.8 with 32 bytes of data: Reply from 8.8.8.8: bytes=32 time=41ms TTL=57 Reply from 8.8.8.8: bytes=32 time=35ms TTL=57 Reply from 8.8.8.8: bytes=32 time=32ms TTL=57 Reply from 8.8.8.8: bytes=32 time=38ms TTL=57 --- Pinging (redacted) with 32 bytes of data: Reply from (redacted): bytes=32 time=76ms TTL=239 Reply from (redacted): bytes=32 time=69ms TTL=239 Reply from (redacted): bytes=32 time=64ms TTL=239Reply from (redacted): bytes=32 time=62ms TTL=239 Sent with [Proton Mail](https://proton.me/) secure email. _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

3 3