Something weird is going on with Comcast in Los Angeles: small packets appear to be getting discarded with a loss rate of ~6-10% while larger packets (I've tried 1000 and 1200) seem to make it through without any loss. Things like SSH and IMAP+SSL are impacted consistently (particularly during key exchange), while HTTP does not appear to be, so the issue appears to be affecting both TCP and ICMP (have not tested UDP). First sign of this issue, according to my logs, is at 11:51 PT (UTC-8), and has gradually gotten worse over the past few hours. Looks like the issue is with the forward path (Source->Destination listed below), as the return path only shows loss at the final destination. (I currently cannot provide a return-path mtr because the only way I can access my box is through VNC, and I don't like doing things like screenshots of a VNC window, so you'll just have to trust me) Nothing anomalous shows up when using L3's looking glass to ping things, and the only Comcast LG I know of is in NYC. I most likely won't see Email responses from anyone (directly or to the list) until this problem is addressed, since my mail is hosted on shown box and cannot currently be fetched due to the problem. Frustrating. Source IP: 69.181.142.213 Destination IP: 174.136.108.146 ICMP 56 bytes: Packets Pings Host Loss% Snt Rcv Last Avg Best Wrst 1. gw.home.lan (192.168.1.1) 0.0% 185 185 0.2 0.2 0.2 0.3 2. 69.181.136.1 0.0% 185 185 8.9 9.9 7.0 21.1 3. te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 185 185 8.7 9.9 7.9 17.0 4. te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134 0.0% 185 185 10.7 12.9 9.7 20.7 5. he-1-5-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.90.93) 98.4% 184 3 49496 49006 47906 49615 6. he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102) 5.4% 184 174 11.9 14.1 10.9 23.5 7. 50.242.148.34 6.5% 184 172 11.7 13.4 10.4 69.9 8. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142) 66.7% 184 61 44.9 20.6 18.0 44.9 9. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142) 59.0% 184 75 18.7 19.8 18.0 26.0 10. ae-27-27.car2.LosAngeles1.Level3.net (4.69.201.18) 2.2% 184 180 20.8 63.1 18.2 293.3 11. GIGLINX-INC.car2.LosAngeles1.Level3.net (4.30.195.86) 6.0% 184 173 19.2 22.2 19.1 39.3 12. mambo.koitsu.org (174.136.108.146) 6.5% 184 172 18.8 20.6 18.5 36.9 Note: Loss shown at above hops #5, #8, and #9 are normal according to historic data (i.e. ICMP prioritisation is in effect) -- however loss at hop #6 is new. ICMP 1200 bytes: Packets Pings Host Loss% Snt Rcv Last Avg Best Wrst 1. gw.home.lan (192.168.1.1) 0.0% 43 43 0.3 0.3 0.2 0.4 2. 69.181.136.1 0.0% 43 43 9.7 10.4 8.6 14.3 3. te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 42 42 10.0 10.0 8.7 16.0 4. te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134 0.0% 42 42 11.7 13.3 10.3 18.9 5. ??? 6. he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102) 0.0% 42 42 14.4 15.9 12.4 25.1 7. 50.242.148.34 0.0% 42 42 12.7 13.6 11.2 23.1 8. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142) 9.8% 42 37 20.0 20.7 18.8 25.0 9. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142) 0.0% 42 42 19.1 21.1 18.9 29.9 10. ae-27-27.car2.LosAngeles1.Level3.net (4.69.201.18) 0.0% 42 42 64.3 105.7 20.9 412.5 11. GIGLINX-INC.car2.LosAngeles1.Level3.net (4.30.195.86) 0.0% 42 42 20.6 22.5 19.9 31.2 12. mambo.koitsu.org (174.136.108.146) 0.0% 42 42 20.7 22.0 20.1 26.4 -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |