Not sure when this started, could have been going on for hours. I just got in to work and attempted to RDP into my home machine -- no go. The asymmetric nature of routing makes this a little difficult to determine where the actual loss is, but I imagine what's shown in WinMTR is correct -- ingress packets are (were, keep reading) getting lost at some point. Note in the ingress mtr the Worst field. Also note that on the ingress side, packets were going from San Jose->Atlanta->Fort Worth->Los Angeles. I can't tell if this was a Comcast<->TATA Communications problem, or if it was a GBLX<->nLayer issue. I'm inclined to believe the latter. And as usual these days, the instant I begin writing about the problem, it stops. Whatever this was, however, was pretty major. I wish I had a start time. End time was approximately 15:28 PDT. I have not checked BGPlay to look for routing/reconvergence. Egress (work -> home): |------------------------------------------------------------------------------------------| | WinMTR statistics | | Host - % | Sent | Recv | Best | Avrg | Wrst | Last | |------------------------------------------------|------|------|------|------|------|------| | {snip} | | xe1-8.core2.sv1.layer42.net - 0 | 254 | 254 | 3 | 4 | 49 | 3 | | sjo-bb1-link.telia.net - 0 | 254 | 254 | 2 | 4 | 30 | 3 | | tata-ic-140123-sjo-bb1.c.telia.net - 31 | 115 | 80 | 3 | 7 | 16 | 3 | |if-4-3254.tcore2.PDI-PaloAlto.as6453.net - 1 | 250 | 249 | 3 | 7 | 91 | 4 | | if-2-2.tcore1.PDI-PaloAlto.as6453.net - 0 | 254 | 254 | 3 | 5 | 63 | 5 | | 66.198.127.14 - 34 | 106 | 70 | 160 | 163 | 167 | 164 | |pos-1-10-0-0-cr01.sanjose.ca.ibone.comcast.net - 35 | 108 | 71 | 149 | 160 | 165 | 149 | |pos-1-8-0-0-cr01.sacramento.ca.ibone.comcast.net - 25 | 130 | 98 | 152 | 167 | 171 | 152 | |pos-0-13-0-0-ar01.oakland.ca.sfba.comcast.net - 33 | 112 | 76 | 0 | 168 | 171 | 162 | |te-9-8-ur04.santaclara.ca.sfba.comcast.net - 39 | 99 | 61 | 0 | 166 | 168 | 166 | | 68.85.191.250 - 33 | 111 | 75 | 167 | 188 | 295 | 167 | | c-67-180-84-87.hsd1.ca.comcast.net - 27 | 124 | 91 | 0 | 180 | 203 | 174 | |________________________________________________|______|______|______|______|______|______| Ingress (home -> work): Packets Pings Host Loss% Snt Rcv Last Avg Best Wrst 1. gw.home.lan 0.0% 72 72 0.3 0.6 0.2 1.1 2. c-67-180-84-1.hsd1.ca.comcast.net 0.0% 71 71 24.7 55.5 12.1 1484. 3. te-5-5-ur04.santaclara.ca.sfba.comcast.net 0.0% 71 71 9.0 12.6 8.4 49.8 4. te-0-4-0-2-ar01.oakland.ca.sfba.comcast.net 0.0% 71 71 10.8 14.6 10.5 47.7 5. pos-2-0-0-0-cr01.sacramento.ca.ibone.comcas 0.0% 71 71 13.2 15.3 11.9 35.6 6. pos-0-4-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 71 71 16.6 18.4 14.7 59.7 7. pos-0-1-0-0-pe01.11greatoaks.ca.ibone.comca 0.0% 71 71 22.1 24.0 19.5 52.6 8. 208.178.58.1 0.0% 71 71 19.8 24.6 17.7 58.4 9. ae9-20G.scr3.SNV2.gblx.net 0.0% 71 71 20.5 22.8 18.8 53.3 10. ae0.ar4.ATL2.gblx.net 0.0% 71 71 77.5 76.9 73.8 101.3 11. xe-3-0-4.ar1.atl1.us.nlayer.net 2.8% 71 69 131.9 150.7 76.2 253.8 12. ae0-70g.cr1.atl1.us.nlayer.net 0.0% 71 71 75.1 113.2 73.7 172.0 13. xe-2-2-1.cr1.dfw1.us.nlayer.net 0.0% 71 71 89.9 115.7 73.8 192.8 14. xe-5-3-0.cr1.lax1.us.nlayer.net 0.0% 71 71 80.6 117.2 74.3 195.8 15. xe-1-2-0.cr1.sjc1.us.nlayer.net 0.0% 71 71 78.8 121.3 76.5 209.1 16. ae1-50g.cr1.pao1.us.nlayer.net 0.0% 71 71 77.7 118.1 77.1 197.1 17. as8121.ae0-3001.cr1.pao1.us.nlayer.net 1.4% 71 70 77.9 118.6 77.1 182.2 18. xe1-4.core1.svk.layer42.net 0.0% 71 71 78.7 117.0 76.5 174.9 19. ??? Below is how things look like now: Egress (work -> home): looks the same as the above WinMTR, sans packet loss. Ingress (home -> work): Packets Pings Host Loss% Snt Rcv Last Avg Best Wrst 1. gw.home.lan 0.0% 6 6 0.2 0.4 0.2 0.9 2. c-67-180-84-1.hsd1.ca.comcast.net 0.0% 5 5 12.4 21.7 12.4 27.4 3. te-5-5-ur04.santaclara.ca.sfba.comcast.net 0.0% 5 5 10.5 10.2 9.7 10.6 4. te-0-4-0-2-ar01.oakland.ca.sfba.comcast.net 0.0% 5 5 12.1 11.6 11.0 12.1 5. pos-2-0-0-0-cr01.sacramento.ca.ibone.comcas 0.0% 5 5 14.5 14.7 12.9 15.8 6. pos-0-4-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 5 5 14.7 16.4 14.7 17.6 7. pos-0-1-0-0-pe01.11greatoaks.ca.ibone.comca 0.0% 5 5 20.9 22.2 20.8 24.7 8. 208.178.58.1 0.0% 5 5 25.3 26.7 18.8 41.6 9. ae9-20G.scr3.SNV2.gblx.net 0.0% 5 5 18.7 19.8 18.7 20.7 10. te6-4-10G.ar1.PAO2.gblx.net 0.0% 5 5 19.5 20.5 19.5 21.7 11. xe-2-0-4.ar1.pao1.us.nlayer.net 0.0% 5 5 23.7 23.4 21.1 27.1 12. ae0-80g.cr1.pao1.us.nlayer.net 0.0% 5 5 18.7 20.1 18.7 21.1 13. as8121.ae0-3001.cr1.pao1.us.nlayer.net 0.0% 5 5 22.8 23.2 20.0 26.6 14. xe1-4.core1.svk.layer42.net 0.0% 5 5 21.1 22.9 19.3 34.9 15. ??? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |