My first inclination was to believe there was(?) an issue in Washington DC affecting Verizon given the data presented, but I couldn't definitively say "yeah that's where it is" because of a couple things: 1. It would have been nice to get a return traceroute/mtr from Site 2 back to wherever, both when the problem was occurring and when it wasn't. 2. I had great difficultly figuring out where each of these traceroutes and mtrs were run from and to. The descriptions given in your preceding paragraphs are hard to follow: "mtr to the equipment on TWC", "traceroute back to my network from TWC", "loss between TWC and Verizon", "traceroute back from Site 1 (TWC)". The latter is easiest accomplished by providing source/destination IP addresses for every single mtr/traceroute you provide (I tend to put them at the top; here's an example[1]), but since some privacy is required here, you could have used labels consistently across the board (e.g. "Site 1 (ISP)", "Site 2 (ISP)", "My Network (ISP)"). [1]: https://puck.nether.net/pipermail/outages/2014-March/006777.html -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | On Tue, Nov 04, 2014 at 04:29:59PM -0500, Michael Hertrick via Outages wrote:
Greetings Outages.org recipients,
Since yesterday, 03-NOV-2014 around 07:12AM EST, two systems on separate Time-Warner Cable (TWC) accounts have been experiencing intermittent and severe packet loss between TWC in NYC and my network in Ashburn, VA.
I've run MTR to the equipment on TWC. The loss I see in that direction is on the last two hops only. When I traceroute back to my network from TWC, and see packet loss, it's between TWC and Verizon.
Does anyone have any insight into this problem that, for the moment, appears to be between TWC and Verizon in the DC metro area?
Regards, Michael Hertrick Neovera, Inc.
Site 1:
My traceroute [v0.85] myhost (0.0.0.0) Tue Nov 4 16:14:02 2014 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev
<snip>
6. 154.54.10.210 0.1% 3506 3.2 13.6 1.7 191.0 25.8 7. 66.109.6.182 0.1% 3506 16.7 20.3 7.9 190.6 24.9 8. 107.14.17.179 0.1% 3506 10.6 20.3 8.0 130.1 24.8 9. 66.109.6.29 0.0% 3506 10.1 20.2 7.9 134.6 24.8 10. 107.14.19.25 0.1% 3506 10.0 20.5 8.4 207.6 24.8 11. 184.152.112.140 0.1% 3506 9.0 19.2 8.7 224.8 25.0 12. 68.173.214.14 0.0% 3506 10.9 16.1 8.8 179.2 21.0 68.173.214.10 13. 68.173.214.6 0.0% 3506 9.8 12.5 9.1 182.0 12.4 14. 67.244.103.36 31.6% 3505 28.2 24.3 14.6 114.4 11.9 15. <REDACTED> 32.1% 3505 22.7 24.0 15.0 173.4 12.4
Site 2:
My traceroute [v0.85] myhost (0.0.0.0) Tue Nov 4 16:14:03 2014 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev
<snip>
7. 154.54.10.210 0.1% 3501 2.5 13.7 1.6 148.2 26.0 8. 66.109.6.182 0.1% 3501 10.1 20.1 7.9 111.2 24.6 9. 66.109.6.31 0.1% 3501 15.0 19.9 7.8 122.6 24.7 10. 66.109.9.31 0.0% 3501 12.1 20.2 7.7 183.8 25.1 11. 107.14.19.25 0.1% 3501 12.4 20.6 8.1 186.1 24.9 12. 184.152.112.106 0.1% 3501 10.4 19.3 8.6 124.7 24.9 13. 68.173.212.78 0.0% 3501 9.6 17.7 9.0 183.3 23.7 68.173.212.74 14. 68.173.212.70 0.0% 3501 11.1 10.6 9.3 194.4 5.0 15. 74.66.0.15 31.6% 3501 22.5 23.6 12.0 143.5 11.5 16. <REDACTED> 31.9% 3501 36.2 23.2 11.8 181.6 11.4
Sample traceroute back from Site 1 (TWC) when the problem was at its worst: <snip> 3 68.173.214.1 10 msec 10 msec 10 msec 4 184.152.112.71 20 msec 20 msec 10 msec 5 107.14.19.22 10 msec 20 msec 10 msec 6 66.109.6.26 20 msec 20 msec 107.14.17.172 20 msec 7 66.109.9.30 20 msec 66.109.6.28 20 msec 20 msec 8 66.109.6.30 20 msec 20 msec 107.14.17.178 20 msec 9 66.109.6.183 20 msec 20 msec 20 msec 10 204.148.11.85 10 msec 20 msec * 11 * 152.63.41.153 30 msec * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * <snip>
Sample traceroute back from Site 1 (TWC) when the problem was not occurring: <snip> 3 68.173.214.1 10 msec 10 msec 10 msec 4 184.152.112.71 10 msec 10 msec 20 msec 5 107.14.19.22 10 msec 10 msec 40 msec 6 66.109.6.26 20 msec 20 msec 107.14.17.172 30 msec 7 66.109.9.30 20 msec 66.109.6.28 20 msec 10 msec 8 107.14.17.178 20 msec 30 msec 20 msec 9 66.109.6.169 20 msec 20 msec 107.14.19.133 20 msec 10 204.148.11.85 20 msec 20 msec 30 msec 11 152.63.41.153 20 msec 152.63.41.169 60 msec 152.63.41.153 20 msec 12 152.63.33.253 30 msec 152.63.35.157 20 msec 152.63.38.250 20 msec 13 152.179.50.54 20 msec 20 msec 20 msec <snip> _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages