DT sells transit services in the US as AS3320. So I would say there is congestion on the link between Comcast and DTAG in San Jose. If Layer42 is your uplink I would let them know because it is very likely they have a paid relationship with DTAG. Mike On Oct 25, 2013, at 8:49 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Here's another one of these incredibly annoying situations that baffles me (and of course figuring out who's responsible is not something I have the power to do).
Been watching this one happen every day, usually between the hours of 1700 and 2000 PDT (UTC-0700). Packet loss and latency lasts anywhere between 30 to 75 minutes. From today around 1915:
src IP: 204.8.213.80 dst IP: 76.102.14.35
Host Loss% Snt Last Avg Best Wrst StDev 1. --- 0.0% 161 21.5 1.4 0.3 59.3 6.0 2. --- 0.0% 160 0.4 1.2 0.3 41.4 4.6 3. --- 0.0% 160 0.3 2.2 0.2 44.5 6.5 4. --- 0.0% 160 1.5 3.3 1.2 47.2 6.7 5. xe0-1-0-1.core1.sv8.layer42.net 0.0% 160 2.2 2.1 1.9 2.4 0.1 6. 193.159.165.73 0.0% 160 4.9 4.1 1.9 8.1 1.3 7. 80.156.163.154 24.5% 160 1000. 933.0 6.3 1738. 624.0 8. pos-1-12-0-0-cr01.sanjose.ca.ibone.comcast 27.7% 160 987.5 932.7 7.2 1706. 647.7 9. he-0-6-0-0-ar01.sfsutro.ca.sfba.comcast.ne 26.4% 160 981.4 917.1 7.2 1719. 642.9 10. te-0-4-0-10-ur05.santaclara.ca.sfba.comcas 30.8% 160 980.0 895.1 9.0 1710. 636.9 11. te-6-0-acr03.santaclara.ca.sfba.comcast.ne 17.0% 160 967.3 614.0 6.9 1224. 518.1 12. c-76-102-14-35.hsd1.ca.comcast.net 18.9% 160 970.2 609.1 16.2 1229. 517.8
src IP: 76.102.14.35 dst IP: 204.8.213.80
Host Loss% Snt Rcv Last Avg Best Wrst 1. gw.home.lan 0.0% 563 563 0.3 0.2 0.2 1.9 2. 76.102.12.1 0.0% 563 563 8.4 8.8 7.9 22.8 3. te-0-2-0-5-ur06.santaclara.ca.sfba.comcast. 0.0% 563 563 8.7 8.9 8.1 23.1 4. te-1-1-0-9-ar01.oakland.ca.sfba.comcast.net 0.0% 563 563 13.5 12.4 9.7 30.2 5. be-90-ar01.sfsutro.ca.sfba.comcast.net 0.0% 563 563 12.5 13.7 11.1 41.5 6. he-3-8-0-0-cr01.sanjose.ca.ibone.comcast.ne 0.0% 563 563 13.2 15.0 12.5 25.0 7. be-14-pe02.11greatoaks.ca.ibone.comcast.net 0.0% 562 562 16.5 16.5 15.6 43.1 8. 23-30-206-94-static.hfc.comcastbusiness.net 0.0% 562 562 16.4 22.5 15.6 40.2 9. 0.xe-2-0-8.XL3.SJC7.ALTER.NET 0.7% 562 558 20.3 29.6 15.3 124.7 10. 0.so-4-0-0.XL1.SJC1.ALTER.NET 1.1% 562 556 26.6 22.9 16.8 174.2 0.so-7-0-0.XL1.SJC1.ALTER.NET 11. POS1-0.XR1.SJC1.ALTER.NET 23.0% 562 432 196.6 150.0 17.8 459.6 12. 193.ATM7-0.GW4.SJC1.ALTER.NET 24.6% 562 424 1133. 669.3 17.9 1835. 13. ???
What interests me: hops #6 and #7 shown in the first mtr:
193.159.165.73 = AS3320 = Deutsche Telekom AG 80.156.163.154 = AS3320 = Deutsche Telekom AG
Using Layer42's own looking glass long after the issue was over:
lg-west>show ip bgp 76.102.14.35 BGP routing table entry for 76.96.0.0/11, version 4070745 Paths: (1 available, best #1, table default) Not advertised to any peer Refresh Epoch 1 8121 3320 7922 69.36.239.8 from 69.36.239.8 (69.36.239.8) Origin IGP, metric 301, localpref 100, valid, external, best rx pathid: 0, tx pathid: 0x0
lg-west>traceroute 76.102.14.35 Type escape sequence to abort. Tracing the route to c-76-102-14-35.hsd1.ca.comcast.net (76.102.14.35) VRF info: (vrf in name/id, vrf out name/id) 1 69.36.229.177 [AS 8121] 1 msec 1 msec 1 msec 2 xe0-0-0-4.core1.sv8.layer42.net (65.50.198.161) [AS 8121] 3 msec 2 msec 2 msec 3 193.159.165.73 [AS 3320] 4 msec 2 msec 3 msec 4 80.156.163.154 [AS 3320] 4 msec 5 msec 3 msec 5 pos-1-12-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.87.141) [AS 7922] 7 msec 6 msec 4 msec 6 he-0-5-0-0-ar01.sfsutro.ca.sfba.comcast.net (68.86.91.46) [AS 7922] 6 msec 5 msec 8 msec 7 te-0-4-0-6-ur05.santaclara.ca.sfba.comcast.net (69.139.198.169) [AS 7922] 5 msec 6 msec 7 msec 8 te-6-0-acr03.santaclara.ca.sfba.comcast.net (68.86.249.66) [AS 7922] 5 msec 6 msec 5 msec 9 c-76-102-14-35.hsd1.ca.comcast.net (76.102.14.35) [AS 7922] 14 msec 18 msec 14 msec
Two final things to note:
1. When the packet loss/latency issue is not occurring (e.g. presently), the paths are still the same. If packets were actually going from Sunnyvale (Layer42) to Germany I'd expect a lot higher than 4ms-7ms latency consistently,
2. The source/destination of 204.8.213.80 is a box at my place of work; border of network intentionally filters. We do peer with Layer42 directly, but I have not pursued this with them yet (as said initially, it's hard for me to accurately point fingers).
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages