A couple of things - Hop 8's IP is an AT&T so likely an interface on an AT&T router, since you're headed towards it in your traceroute (next_hop). - Have you tried something like hping that will allow you to use TCP for your test? - Have you contacted InterNAP and told them to open a ticket with AT&T to open a ticket with AT&T using the data you have? Mike On Mar 8, 2014, at 4:53 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Since roughly Friday, I've been seeing what appears to be packet loss somewhere within Comcast/AT&T network mesh. Source and destination IPs are provided below as well, ditto with some mtrs from src->dst and dst->src. I keep periodic mtrs (both directions) going all the way back to 03/04. I can make all of those logs available if asked.
The issue started on 03/07 @ 21:33 PST suddenly -- not a "gradual" increase -- and lasted until an undetermined time (very hard to tell from mtrs) but I'd estimate ~02:00 PST on 03/08 (today).
The issue then appeared to start back up again ~07:00 PST, though it's hard to give an exact time (seems sort of a gradual increase, thus hard to pinpoint). It's been ongoing since.
The loss varies from 3% to 20%, but you can definitely "feel" it across an SSH session, so it's not ICMP prio.
I will make myself clear: it's very hard to "show" someone the way this problem manifests itself, because the packet loss will vary all over the place between different hops. It *definitely* starts at a particular point and "trickles down", but due to the fact that the loss is a smaller percentage, there are times where a hop will suddenly show 0%. TL;DR -- You'd really have to see a longer log (say, an hour's worth) to be able to say "ah yes, this really is a problem" and not blow it off as ICMP prio.
And as usual, there's one of those "mystery routers" (hop #8 in the first example) that peering providers looooooove to use as a scapegoat when it comes to shifting blame, ex. provider A says "that's a device owned by provider B", provider B says "that device is provider A's responsibility", and neither side does anything about the issue. However I should note that the "mystery router" usually does show some degree of loss even when this issue isn't occurring (likely ICMP prio on the device), but that makes it even more difficult to determine where the issue begins.
src IP: 76.102.14.35 (Comcast; Mountain View, CA) dst IP: 204.109.61.174 (RootBSD; Dallas, TX)
=== Sat Mar 8 16:22:00 PST 2014 (1394324520) Start: Sat Mar 8 16:22:00 2014 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.4 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 30 30 8.0 8.8 8.0 12.2 3.|-- te-0-2-0-5-ur06.santaclara.ca.sfba.comcast.net (68.86.249.253) 0.0% 30 30 8.2 9.0 8.2 16.5 4.|-- te-1-1-0-1-ar01.oakland.ca.sfba.comcast.net (69.139.198.94) 0.0% 30 30 11.9 12.2 10.1 15.0 5.|-- be-90-ar01.sfsutro.ca.sfba.comcast.net (68.85.155.14) 0.0% 30 30 12.0 12.5 10.1 15.1 6.|-- he-3-8-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.94.85) 0.0% 30 30 13.0 14.0 11.8 18.0 7.|-- pos-0-3-0-0-pe01.11greatoaks.ca.ibone.comcast.net (68.86.87.18) 0.0% 30 30 15.6 17.2 15.3 19.8 8.|-- 192.205.37.1 70.0% 30 9 54.8 67.7 53.6 102.2 9.|-- cr2.sffca.ip.att.net (12.122.86.202) 13.3% 30 26 65.2 63.5 61.0 65.7 10.|-- cr2.la2ca.ip.att.net (12.122.31.133) 6.7% 30 28 63.3 63.5 60.9 75.2 11.|-- cr2.dlstx.ip.att.net (12.122.28.177) 3.3% 30 29 65.2 63.7 61.1 69.7 12.|-- ggr6.dlstx.ip.att.net (12.122.138.113) 6.7% 30 28 60.3 64.5 59.9 153.6 13.|-- 12.90.228.14 6.7% 30 28 60.3 60.7 60.2 62.5 14.|-- border1.pc1-bbnet1.dal004.pnap.net (216.52.191.19) 3.3% 30 29 60.3 60.2 59.8 60.5 15.|-- giglinx-60.border1.dal004.pnap.net (216.52.189.46) 3.3% 30 29 59.9 60.2 59.8 61.4 16.|-- 204.109.62.46 6.7% 30 28 60.1 60.5 60.1 62.7 17.|-- mambo.koitsu.org (204.109.61.174) 3.3% 30 29 60.7 60.9 60.1 63.4 === END
src IP: 204.109.61.174 (RootBSD; Dallas, TX) dst IP: 76.102.14.35 (Comcast; Mountain View, CA)
=== Sat Mar 8 16:22:00 PST 2014 (1394324520) Start: Sat Mar 8 16:22:00 2014 HOST: mambo.koitsu.org Loss% Snt Rcv Last Avg Best Wrst 1.|-- 204.109.61.173 0.0% 30 30 0.5 1.3 0.4 15.0 2.|-- 204.109.62.45 0.0% 30 30 0.5 0.5 0.3 1.2 3.|-- border1.ge1-6.giglinx-60.dal004.pnap.net (216.52.189.45) 0.0% 30 30 0.5 0.6 0.4 4.7 4.|-- core3.pc1-bbnet1.ext1a.dal.pnap.net (216.52.191.41) 0.0% 30 30 0.9 1.0 0.9 1.2 5.|-- dax-edge-03.inet.qwest.net (67.133.189.93) 0.0% 30 30 0.6 2.0 0.5 22.8 6.|-- 63-235-82-234.dia.static.qwest.net (63.235.82.234) 0.0% 30 30 1.4 1.3 1.0 1.7 7.|-- be-13-cr01.dallas.tx.ibone.comcast.net (68.86.82.141) 0.0% 30 30 1.3 2.7 1.0 4.9 8.|-- he-0-14-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.85.141) 0.0% 30 30 35.6 33.6 31.8 35.7 9.|-- he-1-8-0-0-ar01.oakland.ca.sfba.comcast.net (68.86.89.54) 3.3% 30 29 52.8 53.5 51.5 55.5 10.|-- te-0-4-0-5-ur06.santaclara.ca.sfba.comcast.net (68.86.143.97) 0.0% 30 30 52.1 52.2 51.9 52.3 11.|-- te-6-0-acr03.santaclara.ca.sfba.comcast.net (68.86.249.66) 6.7% 30 28 53.0 53.0 52.8 53.8 12.|-- c-76-102-14-35.hsd1.ca.comcast.net (76.102.14.35) 3.3% 30 29 60.4 60.5 59.9 63.8 === END
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages