
Those of you having issues, can you check to see if they're still occurring? Thanks, John On Thu, Feb 19, 2015 at 4:25 PM, Jeremy Chadwick via Outages < outages@outages.org> wrote:
Here's another destination showing the same problem (I can't even load http://www.dslreports.com/ at present -- handshake works, but either client->server GET request gets lost, or payload response gets lost, followed by TCP retrans, eventually RST from the server, rinse lather repeat), with things again looking centralised around LA.
Hopefully having multiple test endpoints will come in handy.
Source IP: 69.181.142.213 Destination IP: 64.91.255.98
Packets Pings Host Loss% Snt Rcv Last Avg Best Wrst 1. gw.home.lan (192.168.1.1) 0.0% 171 171 0.2 0.2 0.2 0.4 2. 69.181.136.1 0.0% 171 171 12.7 9.6 7.7 19.2 3. te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 171 171 9.4 9.2 8.0 17.1 4. te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134 0.0% 171 171 12.1 12.2 9.5 18.4 5. he-1-5-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.90.93) 98.8% 171 2 50456 50095 49735 50456 6. he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102) 2.9% 171 166 14.1 13.5 10.9 22.7 7. he-2-12-0-0-cr01.seattle.wa.ibone.comcast.net (68.86.86.229) 1.8% 171 168 29.2 30.2 27.4 36.3 8. he-0-15-0-0-cr01.denver.co.ibone.comcast.net (68.86.86.213) 9.9% 171 154 50.1 47.6 44.9 53.1 9. he-5-13-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.85.245) 4.1% 171 163 72.9 72.3 68.6 84.7 10. hu-0-10-0-1.pe03.350ecermak.il.ibone.comcast.net (68.86.86.6) 5.3% 170 161 69.8 69.4 68.1 84.7 11. 50.242.150.126 7.1% 170 158 106.0 89.7 67.4 400.5 50.242.150.130 12. lw-dc3-core1-te8-16.rtr.liquidweb.com (209.59.157.244) 9.4% 170 154 75.7 80.9 75.2 242.7 13. lw-dc3-dist15-po5.rtr.liquidweb.com (69.167.128.241) 2.4% 170 166 75.0 75.9 74.7 83.6 14. www.dslreports.com (64.91.255.98) 5.3% 170 161 74.2 74.3 73.2 82.5
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Thu, Feb 19, 2015 at 03:07:04PM -0800, Jeremy Chadwick wrote:
Something weird is going on with Comcast in Los Angeles: small packets appear to be getting discarded with a loss rate of ~6-10% while larger packets (I've tried 1000 and 1200) seem to make it through without any loss. Things like SSH and IMAP+SSL are impacted consistently (particularly during key exchange), while HTTP does not appear to be, so the issue appears to be affecting both TCP and ICMP (have not tested UDP).
First sign of this issue, according to my logs, is at 11:51 PT (UTC-8), and has gradually gotten worse over the past few hours.
Looks like the issue is with the forward path (Source->Destination listed below), as the return path only shows loss at the final destination. (I currently cannot provide a return-path mtr because the only way I can access my box is through VNC, and I don't like doing things like screenshots of a VNC window, so you'll just have to trust me)
Nothing anomalous shows up when using L3's looking glass to ping things, and the only Comcast LG I know of is in NYC.
I most likely won't see Email responses from anyone (directly or to the list) until this problem is addressed, since my mail is hosted on shown box and cannot currently be fetched due to the problem. Frustrating.
Source IP: 69.181.142.213 Destination IP: 174.136.108.146
ICMP 56 bytes:
Packets Pings
Host Loss% Snt Rcv Last Avg Best Wrst 1. gw.home.lan (192.168.1.1) 0.0% 185 185 0.2 0.2 0.2 0.3 2. 69.181.136.1 0.0% 185 185 8.9 9.9 7.0 21.1 3. te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 185 185 8.7 9.9 7.9 17.0 4. te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134 0.0% 185 185 10.7 12.9 9.7 20.7 5. he-1-5-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.90.93) 98.4% 184 3 49496 49006 47906 49615 6. he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102) 5.4% 184 174 11.9 14.1 10.9 23.5 7. 50.242.148.34 6.5% 184 172 11.7 13.4 10.4 69.9 8. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142) 66.7% 184 61 44.9 20.6 18.0 44.9 9. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142) 59.0% 184 75 18.7 19.8 18.0 26.0 10. ae-27-27.car2.LosAngeles1.Level3.net (4.69.201.18) 2.2% 184 180 20.8 63.1 18.2 293.3 11. GIGLINX-INC.car2.LosAngeles1.Level3.net (4.30.195.86) 6.0% 184 173 19.2 22.2 19.1 39.3 12. mambo.koitsu.org (174.136.108.146) 6.5% 184 172 18.8 20.6 18.5 36.9
Note: Loss shown at above hops #5, #8, and #9 are normal according to historic data (i.e. ICMP prioritisation is in effect) -- however loss at hop #6 is new.
ICMP 1200 bytes:
Packets Pings
Host Loss% Snt Rcv Last Avg Best Wrst 1. gw.home.lan (192.168.1.1) 0.0% 43 43 0.3 0.3 0.2 0.4 2. 69.181.136.1 0.0% 43 43 9.7 10.4 8.6 14.3 3. te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 42 42 10.0 10.0 8.7 16.0 4. te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134 0.0% 42 42 11.7 13.3 10.3 18.9 5. ??? 6. he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102) 0.0% 42 42 14.4 15.9 12.4 25.1 7. 50.242.148.34 0.0% 42 42 12.7 13.6 11.2 23.1 8. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142) 9.8% 42 37 20.0 20.7 18.8 25.0 9. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142) 0.0% 42 42 19.1 21.1 18.9 29.9 10. ae-27-27.car2.LosAngeles1.Level3.net (4.69.201.18) 0.0% 42 42 64.3 105.7 20.9 412.5 11. GIGLINX-INC.car2.LosAngeles1.Level3.net (4.30.195.86) 0.0% 42 42 20.6 22.5 19.9 31.2 12. mambo.koitsu.org (174.136.108.146) 0.0% 42 42 20.7 22.0 20.1 26.4
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages