Comcast link to 6453 full as usual I am sure. We need an update from backdoor santa. -- Steve Rubin <ser@layer42.net> Layer42 Networks On Dec 20, 2011, at 19:13, Jeremy Chadwick <outages@jdc.parodius.com> wrote:

I've been dealing with this for weeks now. The asymmetric nature of the problem means *4* ISPs would have to get involved to really figure out what's causing this. What a PITA.

* Packet loss increasing between as6453.net and Tata Communications * Packet loss increasing between GBLX and nLayer (at the PAIX?)

Ingress traffic (work->home) shows the first problem:

Source IP: 204.8.209.251 Dest IP: 67.180.84.87

|---------------------------------------------------------------------------------- --------| | WinMTR statistics | | Host - % | Sent | Recv | Best | Avrg | Wrst | Last | |------------------------------------------------|------|------|------|------|------|------| | xe1-8.core2.sv1.layer42.net - 0 | 198 | 198 | 3 | 9 | 83 | 3 | | sjo-bb1-link.telia.net - 0 | 198 | 198 | 2 | 4 | 31 | 3 | | tata-ic-140123-sjo-bb1.c.telia.net - 20 | 109 | 88 | 3 | 8 | 21 | 8 | |if-4-3254.tcore2.PDI-PaloAlto.as6453.net - 0 | 198 | 198 | 3 | 6 | 55 | 6 | | if-2-2.tcore1.PDI-PaloAlto.as6453.net - 0 | 198 | 198 | 3 | 5 | 64 | 4 | | 66.198.127.14 - 5 | 168 | 161 | 14 | 15 | 25 | 15 | |pos-1-10-0-0-cr01.sanjose.ca.ibone.comcast.net - 4 | 172 | 166 | 14 | 16 | 23 | 15 | |pos-1-8-0-0-cr01.sacramento.ca.ibone.comcast.net - 2 | 186 | 183 | 17 | 19 | 29 | 18 | |pos-0-13-0-0-ar01.oakland.ca.sfba.comcast.net - 4 | 174 | 168 | 19 | 20 | 29 | 20 | |te-9-8-ur04.santaclara.ca.sfba.comcast.net - 1 | 194 | 193 | 12 | 13 | 173 | 13 | | 68.85.191.250 - 3 | 182 | 178 | 20 | 37 | 236 | 31 | | c-67-180-84-87.hsd1.ca.comcast.net - 2 | 190 | 188 | 28 | 32 | 74 | 29 | |________________________________________________|______|______|______|______|______|______|

Egress traffic (home->work) shows the 2nd problem.

Source IP: 67.180.84.87 Dest IP: 204.8.209.251 Packets Pings Host Loss% Snt Rcv Last Avg Best Wrst 1. gw.home.lan 0.0% 192 192 0.2 0.5 0.2 1.0 2. c-67-180-84-1.hsd1.ca.comcast.net 0.0% 192 192 27.1 26.2 10.3 139.9 3. te-5-5-ur04.santaclara.ca.sfba.comcast.net 0.0% 192 192 10.3 11.9 8.0 139.2 4. te-0-4-0-2-ar01.oakland.ca.sfba.comcast.net 0.0% 192 192 12.1 13.2 10.2 46.3 5. pos-2-0-0-0-cr01.sacramento.ca.ibone.comcas 0.0% 192 192 12.6 15.3 12.0 86.6 6. pos-0-4-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 192 192 17.3 19.2 15.0 69.0 7. pos-0-1-0-0-pe01.11greatoaks.ca.ibone.comca 0.0% 192 192 22.7 24.9 19.4 118.6 8. 208.178.58.1 0.0% 192 192 23.7 25.9 17.7 88.6 9. ae9-20G.scr3.SNV2.gblx.net 0.0% 192 192 20.1 22.9 18.5 101.3 10. po4-20G.ar1.PAO2.gblx.net 0.0% 192 192 51.4 116.7 18.8 1955. 11. xe-2-0-4.ar1.pao1.us.nlayer.net 5.2% 192 182 40.5 55.3 30.7 124.3 12. ae0-80g.cr1.pao1.us.nlayer.net 2.6% 192 187 38.6 54.2 28.5 128.6 13. as8121.ae0-3001.cr1.pao1.us.nlayer.net 1.0% 192 190 70.6 36.8 26.7 105.8 14. xe1-4.core1.svk.layer42.net 2.1% 192 188 27.6 33.3 27.2 86.9 15. ???

This is not ICMP prioritisation -- I feel this latency and packet loss when using Remote Desktop and SSH. It's really quite annoying.

Obviously the asymmetric nature means either #1 or #2 could explain the other. Issue smells of a capacity problem (e.g. bottleneck), but heck if I know.

I look forward to insights, and/or comments from anyone at any of these companies, including off-list if need be.

-- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages