June 2023 - Outages - Outages Lists

AT&T SFO to Twitter -- possible routing issue or BGP hijack
by Jeremy Chadwick 29 Jun '23

29 Jun '23

Found twitter.com was not loading tonight. Dug in. $ host www.twitter.com www.twitter.com is an alias for twitter.com. twitter.com has address 104.244.42.129 twitter.com mail is handled by 30 ASPMX2.GOOGLEMAIL.com. twitter.com mail is handled by 20 alt2.aspmx.l.google.com. twitter.com mail is handled by 20 alt1.aspmx.l.google.com. twitter.com mail is handled by 10 aspmx.l.google.com. twitter.com mail is handled by 30 ASPMX3.GOOGLEMAIL.com. $ dig ns twitter.com +short b.r06.twtrdns.net. a.r06.twtrdns.net. c.r06.twtrdns.net. d.r06.twtrdns.net. a.u06.twtrdns.net. b.u06.twtrdns.net. c.u06.twtrdns.net. d.u06.twtrdns.net. $ dig @b.r06.twtrdns.net a twitter.com +short 104.244.42.129 And 104.244.42.129 does indeed point to Twitter (per WHOIS/ARIN), so doesn't appear to be a DNS-related thing. Onward we go: $ mtr www.twitter.com Packets Pings Host Loss% Snt Rcv Last Avg Best Wrst 1. 192.168.1.254 0.0% 9 9 0.6 0.6 0.5 0.8 2. 172-10-232-1.lightspeed.sntcca.sbcglobal.net (172.10.232.1) 0.0% 9 9 2.3 2.3 1.5 4.3 3. 71.148.149.42 (71.148.149.42) 0.0% 8 8 2.6 2.7 1.7 3.9 4. 12.242.117.22 (12.242.117.22) 0.0% 8 8 4.6 6.1 3.6 8.2 5. att-gw.sfo.pccw.net (192.205.32.82) 0.0% 8 8 6.7 6.5 5.3 8.1 6. Bundle-Ether45.br04.osa01.pccwbtn.net (63.223.26.30) 0.0% 8 8 121.6 121.1 119.6 122.8 7. 63-222-51-222.static.pccwglobal.net (63.222.51.222) 0.0% 8 8 154.3 154.4 153.3 155.8 8. (waiting for reply) 9. 104.244.42.129 (104.244.42.129) 0.0% 8 8 151.5 152.6 151.4 153.8 $ mtr -z www.twitter.com Packets Pings Host Loss% Snt Rcv Last Avg Best Wrst 1. AS??? 192.168.1.254 0.0% 8 8 0.8 0.6 0.3 0.8 2. AS7018 172-10-232-1.lightspeed.sntcca.sbcglobal.net (172.10.2 0.0% 8 8 1.3 2.2 1.0 4.1 3. AS7018 71.148.149.42 (71.148.149.42) 0.0% 8 8 2.2 3.4 1.9 5.0 4. AS7018 12.242.117.22 (12.242.117.22) 0.0% 8 8 6.0 6.1 4.5 7.6 5. AS7018 att-gw.sfo.pccw.net (192.205.32.82) 0.0% 8 8 5.6 6.2 4.8 7.7 6. AS3491 Bundle-Ether45.br04.osa01.pccwbtn.net (63.223.26.30) 0.0% 8 8 120.4 120.7 120.0 121.5 7. AS3491 63-222-51-222.static.pccwglobal.net (63.222.51.222) 0.0% 8 8 154.1 155.4 153.6 161.5 8. (waiting for reply) 9. AS13414 104.244.42.129 (104.244.42.129) 0.0% 7 7 152.7 152.2 150.8 154.0 AS3491 (pccwbtn.net) is PCCW Global, though a WHOIS on pccwbtn.net says they're PCCW-HKT out of Hong Kong, which would explains the huge jump in latency (6ms -> 121ms) since I'm located in California. 63.223.26.30 is also PCCW Global. PeeringDB says https://www.peeringdb.com/net/674 (AT&T) has a looking glass server at http://route-server.ip.att.net/ but the webserver is not listening on TCP port 80, nor 443: $ telnet route-server.ip.att.net 80 Trying 12.0.1.28... telnet: connect to address 12.0.1.28: Connection refused telnet: Unable to connect to remote host $ telnet route-server.ip.att.net 443 Trying 12.0.1.28... telnet: connect to address 12.0.1.28: Connection refused telnet: Unable to connect to remote host And for those that want source and destinations: src IP: 107.197.104.143 (AT&T Fibre) dst IP: 104.244.42.129 (Twitter) -- | Jeremy Chadwick jdc_at_koitsu.org | | UNIX Systems Administrator PGP 0x2A389531 | | Making life hard for others since 1977. |

3 3

8x8 issues?
by William Kern 23 Jun '23

23 Jun '23

For the past two days we have lost our 8x8 desktop phones between 9 and 10 PST for 10-20 minutes or so. Internet access is entirely normal (as it should be <Grin>). so its definately an 8x8 thing or at least much further away from us. The 8x8 status board doesn't correlate anything in regards to timing, though they show other problems in the early mornings. However Downdetector show spikes that correlate with our outages. Does anyone have any info such as a known DDOS. Sincerely, William Kern PixelGate Networks

2 2

AT&T Las Vegas, NV Issues
by Mike McGurty 20 Jun '23

20 Jun '23

We have verbal reports that correlate with network monitoring of AT&T MPLS and AT&T Wireless down in Las Vegas, NV. Anyone else have AT&T services in that area that are impacted? -- MICHAEL MCGURTY *HE/HIM/HIS* *NETWORK ENGINEER* *AMERICAN EAGLE*t / +1.724.779.5652 e / x15652 540 Keystone Drive, Warrendale, PA 15086 __________________________________________________________________ This electronic message contains information from American Eagle Outfitters, Inc. that may be privileged and/or confidential. If you are not the intended recipient, any disclosure, copy, distribution or use of the contents of this message is prohibited. If you have received this email in error, please contact the sender immediately and destroy this message. Thank you.

1 0

AWS Down again
by Andreas Dolleschal 17 Jun '23

17 Jun '23

Seem AWS has again a major outage. A lot of services are not reachable since about 30 minutes.

7 7

Trouble reaching Geneva
by Miller, Jon 17 Jun '23

17 Jun '23

For the last week we have had trouble reaching a web site in Geneva, Switzerland. A trace route to 193.5.93.80 times out (most of the time). I see this from two Cogent connections and a Frontier connection. Since Cogent and Frontier both successfully hand off to twelve99.net, they aren't going to offer any help. Do you all see this as well? This message and any attachments may contain legally privileged or confidential information, and are intended only for the individual or entity identified above as the addressee. If you are not the addressee, or if this message has been addressed to you in error, you are not authorized to read, copy, or distribute this message and any attachments, and we ask that you please delete this message and attachments (including all copies) and notify the sender. Delivery of this message and any attachments to any person other than the intended recipient(s) is not intended in any way to waive confidentiality or a privilege. All personal messages express views only of the individual sender, and may not be copied or distributed without this statement.

5 4

Quintillion Fiber Cut
by Kevin Shymkiw 16 Jun '23

16 Jun '23

Likely doesn't impact most on this list, but it is a sub-sea fiber cut in an interesting place - https://alaskapublic.org/2023/06/12/cut-cable-causes-weeks-long-north-slope… Looks like it will be down until sometime in July when the sea ice melts enough for the ships to get in and make repairs. I know Quintillion has been looking to run cables both to Asia and Europe using these arctic routes, but I wonder how common these types of long outages could be as we put more and more infra in these arctic climates. Kevin

1 0

MS 365 down globally
by Clayton Dukes 05 Jun '23

05 Jun '23

Looks like MS Email is down globally. Of course, I realize some of you may not get this email :) Does Anyone know the cause? https://twitter.com/MSFT365Status/status/1665734492122742790 ______________________________________________________________ Clayton Dukes ______________________________________________________________

7 7