Found twitter.com was not loading tonight. Dug in. $ host www.twitter.com www.twitter.com is an alias for twitter.com. twitter.com has address 104.244.42.129 twitter.com mail is handled by 30 ASPMX2.GOOGLEMAIL.com. twitter.com mail is handled by 20 alt2.aspmx.l.google.com. twitter.com mail is handled by 20 alt1.aspmx.l.google.com. twitter.com mail is handled by 10 aspmx.l.google.com. twitter.com mail is handled by 30 ASPMX3.GOOGLEMAIL.com. $ dig ns twitter.com +short b.r06.twtrdns.net. a.r06.twtrdns.net. c.r06.twtrdns.net. d.r06.twtrdns.net. a.u06.twtrdns.net. b.u06.twtrdns.net. c.u06.twtrdns.net. d.u06.twtrdns.net. $ dig @b.r06.twtrdns.net a twitter.com +short 104.244.42.129 And 104.244.42.129 does indeed point to Twitter (per WHOIS/ARIN), so doesn't appear to be a DNS-related thing. Onward we go: $ mtr www.twitter.com Packets Pings Host Loss% Snt Rcv Last Avg Best Wrst 1. 192.168.1.254 0.0% 9 9 0.6 0.6 0.5 0.8 2. 172-10-232-1.lightspeed.sntcca.sbcglobal.net (172.10.232.1) 0.0% 9 9 2.3 2.3 1.5 4.3 3. 71.148.149.42 (71.148.149.42) 0.0% 8 8 2.6 2.7 1.7 3.9 4. 12.242.117.22 (12.242.117.22) 0.0% 8 8 4.6 6.1 3.6 8.2 5. att-gw.sfo.pccw.net (192.205.32.82) 0.0% 8 8 6.7 6.5 5.3 8.1 6. Bundle-Ether45.br04.osa01.pccwbtn.net (63.223.26.30) 0.0% 8 8 121.6 121.1 119.6 122.8 7. 63-222-51-222.static.pccwglobal.net (63.222.51.222) 0.0% 8 8 154.3 154.4 153.3 155.8 8. (waiting for reply) 9. 104.244.42.129 (104.244.42.129) 0.0% 8 8 151.5 152.6 151.4 153.8 $ mtr -z www.twitter.com Packets Pings Host Loss% Snt Rcv Last Avg Best Wrst 1. AS??? 192.168.1.254 0.0% 8 8 0.8 0.6 0.3 0.8 2. AS7018 172-10-232-1.lightspeed.sntcca.sbcglobal.net (172.10.2 0.0% 8 8 1.3 2.2 1.0 4.1 3. AS7018 71.148.149.42 (71.148.149.42) 0.0% 8 8 2.2 3.4 1.9 5.0 4. AS7018 12.242.117.22 (12.242.117.22) 0.0% 8 8 6.0 6.1 4.5 7.6 5. AS7018 att-gw.sfo.pccw.net (192.205.32.82) 0.0% 8 8 5.6 6.2 4.8 7.7 6. AS3491 Bundle-Ether45.br04.osa01.pccwbtn.net (63.223.26.30) 0.0% 8 8 120.4 120.7 120.0 121.5 7. AS3491 63-222-51-222.static.pccwglobal.net (63.222.51.222) 0.0% 8 8 154.1 155.4 153.6 161.5 8. (waiting for reply) 9. AS13414 104.244.42.129 (104.244.42.129) 0.0% 7 7 152.7 152.2 150.8 154.0 AS3491 (pccwbtn.net) is PCCW Global, though a WHOIS on pccwbtn.net says they're PCCW-HKT out of Hong Kong, which would explains the huge jump in latency (6ms -> 121ms) since I'm located in California. 63.223.26.30 is also PCCW Global. PeeringDB says https://www.peeringdb.com/net/674 (AT&T) has a looking glass server at http://route-server.ip.att.net/ but the webserver is not listening on TCP port 80, nor 443: $ telnet route-server.ip.att.net 80 Trying 12.0.1.28... telnet: connect to address 12.0.1.28: Connection refused telnet: Unable to connect to remote host $ telnet route-server.ip.att.net 443 Trying 12.0.1.28... telnet: connect to address 12.0.1.28: Connection refused telnet: Unable to connect to remote host And for those that want source and destinations: src IP: 107.197.104.143 (AT&T Fibre) dst IP: 104.244.42.129 (Twitter) -- | Jeremy Chadwick jdc_at_koitsu.org | | UNIX Systems Administrator PGP 0x2A389531 | | Making life hard for others since 1977. |
On Sat, Jun 24, 2023 at 06:17:12AM +0000, Jeremy Chadwick via Outages wrote:
$ telnet route-server.ip.att.net 80 Trying 12.0.1.28... telnet: connect to address 12.0.1.28: Connection refused telnet: Unable to connect to remote host
It's a telnet service: $ telnet route-server.ip.att.net Trying 12.0.1.28... Connected to route-server.ip.att.net. Escape character is '^]'. -------------- route-server.ip.att.net --------------- --------- AT&T IP Services Route Monitor ----------- The information available through route-server.ip.att.net is offered by AT&T's Internet engineering organization to the Internet community. This router maintains eBGP peerings with customer-facing routers throughout the AT&T IP Services Backbone: [snip]
*This*, folks and peoples, is a good problem report. Strive to provide this much detail, or more. The price of free help is you gotta show your work: http://www.catb.org/~esr/faqs/smart-questions.html Cheers, -- jr '<admin/>' a ----- Original Message -----
From: "Jeremy Chadwick via Outages" <outages@outages.org> To: outages@outages.org Sent: Saturday, June 24, 2023 2:17:12 AM Subject: [outages] AT&T SFO to Twitter -- possible routing issue or BGP hijack
Found twitter.com was not loading tonight. Dug in.
$ host www.twitter.com www.twitter.com is an alias for twitter.com. twitter.com has address 104.244.42.129 twitter.com mail is handled by 30 ASPMX2.GOOGLEMAIL.com. twitter.com mail is handled by 20 alt2.aspmx.l.google.com. twitter.com mail is handled by 20 alt1.aspmx.l.google.com. twitter.com mail is handled by 10 aspmx.l.google.com. twitter.com mail is handled by 30 ASPMX3.GOOGLEMAIL.com.
$ dig ns twitter.com +short b.r06.twtrdns.net. a.r06.twtrdns.net. c.r06.twtrdns.net. d.r06.twtrdns.net. a.u06.twtrdns.net. b.u06.twtrdns.net. c.u06.twtrdns.net. d.u06.twtrdns.net.
$ dig @b.r06.twtrdns.net a twitter.com +short 104.244.42.129
And 104.244.42.129 does indeed point to Twitter (per WHOIS/ARIN), so doesn't appear to be a DNS-related thing. Onward we go:
$ mtr www.twitter.com Packets Pings Host Loss% Snt Rcv Last Avg Best Wrst 1. 192.168.1.254 0.0% 9 9 0.6 0.6 0.5 0.8 2. 172-10-232-1.lightspeed.sntcca.sbcglobal.net (172.10.232.1) 0.0% 9 9 2.3 2.3 1.5 4.3 3. 71.148.149.42 (71.148.149.42) 0.0% 8 8 2.6 2.7 1.7 3.9 4. 12.242.117.22 (12.242.117.22) 0.0% 8 8 4.6 6.1 3.6 8.2 5. att-gw.sfo.pccw.net (192.205.32.82) 0.0% 8 8 6.7 6.5 5.3 8.1 6. Bundle-Ether45.br04.osa01.pccwbtn.net (63.223.26.30) 0.0% 8 8 121.6 121.1 119.6 122.8 7. 63-222-51-222.static.pccwglobal.net (63.222.51.222) 0.0% 8 8 154.3 154.4 153.3 155.8 8. (waiting for reply) 9. 104.244.42.129 (104.244.42.129) 0.0% 8 8 151.5 152.6 151.4 153.8
$ mtr -z www.twitter.com
Packets Pings Host Loss% Snt Rcv Last Avg Best Wrst 1. AS??? 192.168.1.254 0.0% 8 8 0.8 0.6 0.3 0.8 2. AS7018 172-10-232-1.lightspeed.sntcca.sbcglobal.net (172.10.2 0.0% 8 8 1.3 2.2 1.0 4.1 3. AS7018 71.148.149.42 (71.148.149.42) 0.0% 8 8 2.2 3.4 1.9 5.0 4. AS7018 12.242.117.22 (12.242.117.22) 0.0% 8 8 6.0 6.1 4.5 7.6 5. AS7018 att-gw.sfo.pccw.net (192.205.32.82) 0.0% 8 8 5.6 6.2 4.8 7.7 6. AS3491 Bundle-Ether45.br04.osa01.pccwbtn.net (63.223.26.30) 0.0% 8 8 120.4 120.7 120.0 121.5 7. AS3491 63-222-51-222.static.pccwglobal.net (63.222.51.222) 0.0% 8 8 154.1 155.4 153.6 161.5 8. (waiting for reply) 9. AS13414 104.244.42.129 (104.244.42.129) 0.0% 7 7 152.7 152.2 150.8 154.0
AS3491 (pccwbtn.net) is PCCW Global, though a WHOIS on pccwbtn.net says they're PCCW-HKT out of Hong Kong, which would explains the huge jump in latency (6ms -> 121ms) since I'm located in California. 63.223.26.30 is also PCCW Global.
PeeringDB says https://www.peeringdb.com/net/674 (AT&T) has a looking glass server at http://route-server.ip.att.net/ but the webserver is not listening on TCP port 80, nor 443:
$ telnet route-server.ip.att.net 80 Trying 12.0.1.28... telnet: connect to address 12.0.1.28: Connection refused telnet: Unable to connect to remote host $ telnet route-server.ip.att.net 443 Trying 12.0.1.28... telnet: connect to address 12.0.1.28: Connection refused telnet: Unable to connect to remote host
And for those that want source and destinations:
src IP: 107.197.104.143 (AT&T Fibre) dst IP: 104.244.42.129 (Twitter)
-- | Jeremy Chadwick jdc_at_koitsu.org | | UNIX Systems Administrator PGP 0x2A389531 | | Making life hard for others since 1977. |
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
To close this matter out: looks like someone noticed and now Twitter traffic is going through PCCW Global's San Jose location. The latency is still higher than it should be (ideally), but a 3x improvement is better than none. Packets Pings Host Loss% Snt Rcv Last Avg Best Wrst 1. AS??? 192.168.1.254 0.0% 7 7 0.4 0.5 0.2 0.7 2. AS7018 172-10-232-1.lightspeed.sntcca.sbcglobal.net (172.10.2 0.0% 7 7 2.1 2.5 1.5 3.4 3. AS7018 71.148.149.42 (71.148.149.42) 0.0% 7 7 2.7 2.8 1.3 4.0 4. AS7018 12.242.117.22 (12.242.117.22) 0.0% 7 7 7.0 6.1 4.0 8.6 5. AS7018 att-gw.sfo.pccw.net (192.205.32.82) 0.0% 7 7 7.6 6.7 5.6 7.7 6. AS3491 Twitter.BE16.br04.sjo01.pccwbtn.net (63.218.179.202) 0.0% 7 7 66.0 64.4 63.3 66.0 7. (waiting for reply) 8. AS13414 104.244.42.65 (104.244.42.65) 0.0% 7 7 55.2 54.7 53.9 55.7 -- | Jeremy Chadwick jdc_at_koitsu.org | | UNIX Systems Administrator PGP 0x2A389531 | | Making life hard for others since 1977. | On Sat, Jun 24, 2023 at 06:17:12AM +0000, Jeremy Chadwick via Outages wrote:
Found twitter.com was not loading tonight. Dug in.
$ host www.twitter.com www.twitter.com is an alias for twitter.com. twitter.com has address 104.244.42.129 twitter.com mail is handled by 30 ASPMX2.GOOGLEMAIL.com. twitter.com mail is handled by 20 alt2.aspmx.l.google.com. twitter.com mail is handled by 20 alt1.aspmx.l.google.com. twitter.com mail is handled by 10 aspmx.l.google.com. twitter.com mail is handled by 30 ASPMX3.GOOGLEMAIL.com.
$ dig ns twitter.com +short b.r06.twtrdns.net. a.r06.twtrdns.net. c.r06.twtrdns.net. d.r06.twtrdns.net. a.u06.twtrdns.net. b.u06.twtrdns.net. c.u06.twtrdns.net. d.u06.twtrdns.net.
$ dig @b.r06.twtrdns.net a twitter.com +short 104.244.42.129
And 104.244.42.129 does indeed point to Twitter (per WHOIS/ARIN), so doesn't appear to be a DNS-related thing. Onward we go:
$ mtr www.twitter.com Packets Pings Host Loss% Snt Rcv Last Avg Best Wrst 1. 192.168.1.254 0.0% 9 9 0.6 0.6 0.5 0.8 2. 172-10-232-1.lightspeed.sntcca.sbcglobal.net (172.10.232.1) 0.0% 9 9 2.3 2.3 1.5 4.3 3. 71.148.149.42 (71.148.149.42) 0.0% 8 8 2.6 2.7 1.7 3.9 4. 12.242.117.22 (12.242.117.22) 0.0% 8 8 4.6 6.1 3.6 8.2 5. att-gw.sfo.pccw.net (192.205.32.82) 0.0% 8 8 6.7 6.5 5.3 8.1 6. Bundle-Ether45.br04.osa01.pccwbtn.net (63.223.26.30) 0.0% 8 8 121.6 121.1 119.6 122.8 7. 63-222-51-222.static.pccwglobal.net (63.222.51.222) 0.0% 8 8 154.3 154.4 153.3 155.8 8. (waiting for reply) 9. 104.244.42.129 (104.244.42.129) 0.0% 8 8 151.5 152.6 151.4 153.8
$ mtr -z www.twitter.com
Packets Pings Host Loss% Snt Rcv Last Avg Best Wrst 1. AS??? 192.168.1.254 0.0% 8 8 0.8 0.6 0.3 0.8 2. AS7018 172-10-232-1.lightspeed.sntcca.sbcglobal.net (172.10.2 0.0% 8 8 1.3 2.2 1.0 4.1 3. AS7018 71.148.149.42 (71.148.149.42) 0.0% 8 8 2.2 3.4 1.9 5.0 4. AS7018 12.242.117.22 (12.242.117.22) 0.0% 8 8 6.0 6.1 4.5 7.6 5. AS7018 att-gw.sfo.pccw.net (192.205.32.82) 0.0% 8 8 5.6 6.2 4.8 7.7 6. AS3491 Bundle-Ether45.br04.osa01.pccwbtn.net (63.223.26.30) 0.0% 8 8 120.4 120.7 120.0 121.5 7. AS3491 63-222-51-222.static.pccwglobal.net (63.222.51.222) 0.0% 8 8 154.1 155.4 153.6 161.5 8. (waiting for reply) 9. AS13414 104.244.42.129 (104.244.42.129) 0.0% 7 7 152.7 152.2 150.8 154.0
AS3491 (pccwbtn.net) is PCCW Global, though a WHOIS on pccwbtn.net says they're PCCW-HKT out of Hong Kong, which would explains the huge jump in latency (6ms -> 121ms) since I'm located in California. 63.223.26.30 is also PCCW Global.
PeeringDB says https://www.peeringdb.com/net/674 (AT&T) has a looking glass server at http://route-server.ip.att.net/ but the webserver is not listening on TCP port 80, nor 443:
$ telnet route-server.ip.att.net 80 Trying 12.0.1.28... telnet: connect to address 12.0.1.28: Connection refused telnet: Unable to connect to remote host $ telnet route-server.ip.att.net 443 Trying 12.0.1.28... telnet: connect to address 12.0.1.28: Connection refused telnet: Unable to connect to remote host
And for those that want source and destinations:
src IP: 107.197.104.143 (AT&T Fibre) dst IP: 104.244.42.129 (Twitter)
-- | Jeremy Chadwick jdc_at_koitsu.org | | UNIX Systems Administrator PGP 0x2A389531 | | Making life hard for others since 1977. |
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (3)
-
Jay R. Ashworth -
Jeremy Chadwick -
Job Snijders