Time Warner Cable through Verizon (Wash. DC Metro): Packet Loss
Greetings Outages.org recipients, Since yesterday, 03-NOV-2014 around 07:12AM EST, two systems on separate Time-Warner Cable (TWC) accounts have been experiencing intermittent and severe packet loss between TWC in NYC and my network in Ashburn, VA. I've run MTR to the equipment on TWC. The loss I see in that direction is on the last two hops only. When I traceroute back to my network from TWC, and see packet loss, it's between TWC and Verizon. Does anyone have any insight into this problem that, for the moment, appears to be between TWC and Verizon in the DC metro area? Regards, Michael Hertrick Neovera, Inc. Site 1:
My traceroute [v0.85] myhost (0.0.0.0) Tue Nov 4 16:14:02 2014 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev
<snip>
6. 154.54.10.210 0.1% 3506 3.2 13.6 1.7 191.0 25.8 7. 66.109.6.182 0.1% 3506 16.7 20.3 7.9 190.6 24.9 8. 107.14.17.179 0.1% 3506 10.6 20.3 8.0 130.1 24.8 9. 66.109.6.29 0.0% 3506 10.1 20.2 7.9 134.6 24.8 10. 107.14.19.25 0.1% 3506 10.0 20.5 8.4 207.6 24.8 11. 184.152.112.140 0.1% 3506 9.0 19.2 8.7 224.8 25.0 12. 68.173.214.14 0.0% 3506 10.9 16.1 8.8 179.2 21.0 68.173.214.10 13. 68.173.214.6 0.0% 3506 9.8 12.5 9.1 182.0 12.4 14. 67.244.103.36 31.6% 3505 28.2 24.3 14.6 114.4 11.9 15. <REDACTED> 32.1% 3505 22.7 24.0 15.0 173.4 12.4
Site 2:
My traceroute [v0.85] myhost (0.0.0.0) Tue Nov 4 16:14:03 2014 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev
<snip>
7. 154.54.10.210 0.1% 3501 2.5 13.7 1.6 148.2 26.0 8. 66.109.6.182 0.1% 3501 10.1 20.1 7.9 111.2 24.6 9. 66.109.6.31 0.1% 3501 15.0 19.9 7.8 122.6 24.7 10. 66.109.9.31 0.0% 3501 12.1 20.2 7.7 183.8 25.1 11. 107.14.19.25 0.1% 3501 12.4 20.6 8.1 186.1 24.9 12. 184.152.112.106 0.1% 3501 10.4 19.3 8.6 124.7 24.9 13. 68.173.212.78 0.0% 3501 9.6 17.7 9.0 183.3 23.7 68.173.212.74 14. 68.173.212.70 0.0% 3501 11.1 10.6 9.3 194.4 5.0 15. 74.66.0.15 31.6% 3501 22.5 23.6 12.0 143.5 11.5 16. <REDACTED> 31.9% 3501 36.2 23.2 11.8 181.6 11.4
Sample traceroute back from Site 1 (TWC) when the problem was at its worst: <snip> 3 68.173.214.1 10 msec 10 msec 10 msec 4 184.152.112.71 20 msec 20 msec 10 msec 5 107.14.19.22 10 msec 20 msec 10 msec 6 66.109.6.26 20 msec 20 msec 107.14.17.172 20 msec 7 66.109.9.30 20 msec 66.109.6.28 20 msec 20 msec 8 66.109.6.30 20 msec 20 msec 107.14.17.178 20 msec 9 66.109.6.183 20 msec 20 msec 20 msec 10 204.148.11.85 10 msec 20 msec * 11 * 152.63.41.153 30 msec * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * <snip> Sample traceroute back from Site 1 (TWC) when the problem was not occurring: <snip> 3 68.173.214.1 10 msec 10 msec 10 msec 4 184.152.112.71 10 msec 10 msec 20 msec 5 107.14.19.22 10 msec 10 msec 40 msec 6 66.109.6.26 20 msec 20 msec 107.14.17.172 30 msec 7 66.109.9.30 20 msec 66.109.6.28 20 msec 10 msec 8 107.14.17.178 20 msec 30 msec 20 msec 9 66.109.6.169 20 msec 20 msec 107.14.19.133 20 msec 10 204.148.11.85 20 msec 20 msec 30 msec 11 152.63.41.153 20 msec 152.63.41.169 60 msec 152.63.41.153 20 msec 12 152.63.33.253 30 msec 152.63.35.157 20 msec 152.63.38.250 20 msec 13 152.179.50.54 20 msec 20 msec 20 msec <snip>
----- Original Message -----
From: "Michael Hertrick via Outages" <outages@outages.org>
Since yesterday, 03-NOV-2014 around 07:12AM EST, two systems on separate Time-Warner Cable (TWC) accounts have been experiencing intermittent and severe packet loss between TWC in NYC and my network in Ashburn, VA.
I've run MTR to the equipment on TWC. The loss I see in that direction is on the last two hops only. When I traceroute back to my network from TWC, and see packet loss, it's between TWC and Verizon.
Well, the second to last hops have different addresses, but that doesn't mean they're not the same concentrator; annoying as hell that mtr -r won't show both names and IPs, isn't it? Are the remote TWC locations both near one another? Do you have reports from them that they have trouble getting other places on the internet? Speed tests showing problems? etc? Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/04/2014 04:48 PM, Jay Ashworth via Outages wrote:
----- Original Message -----
From: "Michael Hertrick via Outages" <outages@outages.org>
Since yesterday, 03-NOV-2014 around 07:12AM EST, two systems on separate Time-Warner Cable (TWC) accounts have been experiencing intermittent and severe packet loss between TWC in NYC and my network in Ashburn, VA.
I've run MTR to the equipment on TWC. The loss I see in that direction is on the last two hops only. When I traceroute back to my network from TWC, and see packet loss, it's between TWC and Verizon.
Well, the second to last hops have different addresses, but that doesn't mean they're not the same concentrator; annoying as hell that mtr -r won't show both names and IPs, isn't it?
Since my original e-mail, I've checked the fine man page and I should have run `mtr -wbc <seconds> <address>` to get names and addresses together.
Are the remote TWC locations both near one another?
They're about a mile or less apart.
Do you have reports from them that they have trouble getting other places on the internet? Speed tests showing problems? etc?
The folks on-site in NYC have reported being able to reach their other, usual web sites on the Internet without issue. It's actually difficult for me to get anyone on-site to run any tests because they typically leave when things aren't working right. I know... problem solved, right? (I wish) I see that Albert Yuen with Raymond Fong MD replied to this thread and has corroborated my story. He may be in a better position to run additional tests from the TWC@NYC side of things than I am.
Cheers, -- jra
- -- Regards, Michael Hertrick Neovera, Inc. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlRZsIgACgkQcJVdtfpkLb8T7ACfTlCaugCxxpBMMucqHOneD6PQ PzcAnRajQetbecDlnlSytoXIin+p3LXJ =wq5G -----END PGP SIGNATURE-----
Here is a representative sample of TWC-VZ in New York: http://i.imgur.com/A33IESb.png
From a cursory look at advertisements, it looks like they're trying to establish a direct adjacency, though things aren't going swimmingly.
On Wed, Nov 5, 2014 at 12:07 AM, Michael Hertrick via Outages <outages@outages.org> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/04/2014 04:48 PM, Jay Ashworth via Outages wrote:
----- Original Message -----
From: "Michael Hertrick via Outages" <outages@outages.org>
Since yesterday, 03-NOV-2014 around 07:12AM EST, two systems on separate Time-Warner Cable (TWC) accounts have been experiencing intermittent and severe packet loss between TWC in NYC and my network in Ashburn, VA.
I've run MTR to the equipment on TWC. The loss I see in that direction is on the last two hops only. When I traceroute back to my network from TWC, and see packet loss, it's between TWC and Verizon.
Well, the second to last hops have different addresses, but that doesn't mean they're not the same concentrator; annoying as hell that mtr -r won't show both names and IPs, isn't it?
Since my original e-mail, I've checked the fine man page and I should have run `mtr -wbc <seconds> <address>` to get names and addresses together.
Are the remote TWC locations both near one another?
They're about a mile or less apart.
Do you have reports from them that they have trouble getting other places on the internet? Speed tests showing problems? etc?
The folks on-site in NYC have reported being able to reach their other, usual web sites on the Internet without issue. It's actually difficult for me to get anyone on-site to run any tests because they typically leave when things aren't working right. I know... problem solved, right? (I wish)
I see that Albert Yuen with Raymond Fong MD replied to this thread and has corroborated my story. He may be in a better position to run additional tests from the TWC@NYC side of things than I am.
Cheers, -- jra
- -- Regards, Michael Hertrick Neovera, Inc.
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1
iEYEARECAAYFAlRZsIgACgkQcJVdtfpkLb8T7ACfTlCaugCxxpBMMucqHOneD6PQ PzcAnRajQetbecDlnlSytoXIin+p3LXJ =wq5G -----END PGP SIGNATURE----- _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/05/2014 11:40 AM, Adam Rothschild wrote:
Here is a representative sample of TWC-VZ in New York:
http://i.imgur.com/A33IESb.png
From a cursory look at advertisements, it looks like they're trying to establish a direct adjacency, though things aren't going swimmingly.
I'll say... Thanks for the info, Adam. Anyway, I was about to make some BGP configuration modifications when I noticed that TWC was no longer carrying my traffic to WDC and handing it off to Verizon. It's being handed off to TATA in NYC before heading down to WDC on NTT. I'm not sure precisely when that change occurred, but my ping monitors to the TWC sites have been stable for about 2.5 hours. If the problem returns for me, I'll most likely take Verizon out of the path altogether and won't be of any further use for troubleshooting/testing this issue. Thanks, all. - -- Regards, Michael Hertrick Neovera, Inc. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlRaYQ0ACgkQcJVdtfpkLb86HACcCmEfv7I03Y9RbY1gVNK1ICUv 5z0An0aJVlaKCg0z3Pu66ZTtxyK12/HD =joil -----END PGP SIGNATURE-----
Interesting that you say that. We have been having problems with customers in NY that are using Verizon FiOS. What is interesting is clients in NJ that are using Verizon FiOS are taking the exact same path (layer 3) with no problem. We are also having issues with traffic coming from equipment that is hosted by Verizon in St. Denis, France. I suspect there is some layer1/2 device out there in NY that is having issues. On Wed, Nov 5, 2014 at 12:40 PM, Michael Hertrick via Outages < outages@outages.org> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/05/2014 11:40 AM, Adam Rothschild wrote:
Here is a representative sample of TWC-VZ in New York:
http://i.imgur.com/A33IESb.png
From a cursory look at advertisements, it looks like they're trying to establish a direct adjacency, though things aren't going swimmingly.
I'll say... Thanks for the info, Adam.
Anyway, I was about to make some BGP configuration modifications when I noticed that TWC was no longer carrying my traffic to WDC and handing it off to Verizon. It's being handed off to TATA in NYC before heading down to WDC on NTT.
I'm not sure precisely when that change occurred, but my ping monitors to the TWC sites have been stable for about 2.5 hours.
If the problem returns for me, I'll most likely take Verizon out of the path altogether and won't be of any further use for troubleshooting/testing this issue.
Thanks, all.
- -- Regards, Michael Hertrick Neovera, Inc. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1
iEYEARECAAYFAlRaYQ0ACgkQcJVdtfpkLb86HACcCmEfv7I03Y9RbY1gVNK1ICUv 5z0An0aJVlaKCg0z3Pu66ZTtxyK12/HD =joil -----END PGP SIGNATURE----- _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
My first inclination was to believe there was(?) an issue in Washington DC affecting Verizon given the data presented, but I couldn't definitively say "yeah that's where it is" because of a couple things: 1. It would have been nice to get a return traceroute/mtr from Site 2 back to wherever, both when the problem was occurring and when it wasn't. 2. I had great difficultly figuring out where each of these traceroutes and mtrs were run from and to. The descriptions given in your preceding paragraphs are hard to follow: "mtr to the equipment on TWC", "traceroute back to my network from TWC", "loss between TWC and Verizon", "traceroute back from Site 1 (TWC)". The latter is easiest accomplished by providing source/destination IP addresses for every single mtr/traceroute you provide (I tend to put them at the top; here's an example[1]), but since some privacy is required here, you could have used labels consistently across the board (e.g. "Site 1 (ISP)", "Site 2 (ISP)", "My Network (ISP)"). [1]: https://puck.nether.net/pipermail/outages/2014-March/006777.html -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | On Tue, Nov 04, 2014 at 04:29:59PM -0500, Michael Hertrick via Outages wrote:
Greetings Outages.org recipients,
Since yesterday, 03-NOV-2014 around 07:12AM EST, two systems on separate Time-Warner Cable (TWC) accounts have been experiencing intermittent and severe packet loss between TWC in NYC and my network in Ashburn, VA.
I've run MTR to the equipment on TWC. The loss I see in that direction is on the last two hops only. When I traceroute back to my network from TWC, and see packet loss, it's between TWC and Verizon.
Does anyone have any insight into this problem that, for the moment, appears to be between TWC and Verizon in the DC metro area?
Regards, Michael Hertrick Neovera, Inc.
Site 1:
My traceroute [v0.85] myhost (0.0.0.0) Tue Nov 4 16:14:02 2014 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev
<snip>
6. 154.54.10.210 0.1% 3506 3.2 13.6 1.7 191.0 25.8 7. 66.109.6.182 0.1% 3506 16.7 20.3 7.9 190.6 24.9 8. 107.14.17.179 0.1% 3506 10.6 20.3 8.0 130.1 24.8 9. 66.109.6.29 0.0% 3506 10.1 20.2 7.9 134.6 24.8 10. 107.14.19.25 0.1% 3506 10.0 20.5 8.4 207.6 24.8 11. 184.152.112.140 0.1% 3506 9.0 19.2 8.7 224.8 25.0 12. 68.173.214.14 0.0% 3506 10.9 16.1 8.8 179.2 21.0 68.173.214.10 13. 68.173.214.6 0.0% 3506 9.8 12.5 9.1 182.0 12.4 14. 67.244.103.36 31.6% 3505 28.2 24.3 14.6 114.4 11.9 15. <REDACTED> 32.1% 3505 22.7 24.0 15.0 173.4 12.4
Site 2:
My traceroute [v0.85] myhost (0.0.0.0) Tue Nov 4 16:14:03 2014 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev
<snip>
7. 154.54.10.210 0.1% 3501 2.5 13.7 1.6 148.2 26.0 8. 66.109.6.182 0.1% 3501 10.1 20.1 7.9 111.2 24.6 9. 66.109.6.31 0.1% 3501 15.0 19.9 7.8 122.6 24.7 10. 66.109.9.31 0.0% 3501 12.1 20.2 7.7 183.8 25.1 11. 107.14.19.25 0.1% 3501 12.4 20.6 8.1 186.1 24.9 12. 184.152.112.106 0.1% 3501 10.4 19.3 8.6 124.7 24.9 13. 68.173.212.78 0.0% 3501 9.6 17.7 9.0 183.3 23.7 68.173.212.74 14. 68.173.212.70 0.0% 3501 11.1 10.6 9.3 194.4 5.0 15. 74.66.0.15 31.6% 3501 22.5 23.6 12.0 143.5 11.5 16. <REDACTED> 31.9% 3501 36.2 23.2 11.8 181.6 11.4
Sample traceroute back from Site 1 (TWC) when the problem was at its worst: <snip> 3 68.173.214.1 10 msec 10 msec 10 msec 4 184.152.112.71 20 msec 20 msec 10 msec 5 107.14.19.22 10 msec 20 msec 10 msec 6 66.109.6.26 20 msec 20 msec 107.14.17.172 20 msec 7 66.109.9.30 20 msec 66.109.6.28 20 msec 20 msec 8 66.109.6.30 20 msec 20 msec 107.14.17.178 20 msec 9 66.109.6.183 20 msec 20 msec 20 msec 10 204.148.11.85 10 msec 20 msec * 11 * 152.63.41.153 30 msec * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * <snip>
Sample traceroute back from Site 1 (TWC) when the problem was not occurring: <snip> 3 68.173.214.1 10 msec 10 msec 10 msec 4 184.152.112.71 10 msec 10 msec 20 msec 5 107.14.19.22 10 msec 10 msec 40 msec 6 66.109.6.26 20 msec 20 msec 107.14.17.172 30 msec 7 66.109.9.30 20 msec 66.109.6.28 20 msec 10 msec 8 107.14.17.178 20 msec 30 msec 20 msec 9 66.109.6.169 20 msec 20 msec 107.14.19.133 20 msec 10 204.148.11.85 20 msec 20 msec 30 msec 11 152.63.41.153 20 msec 152.63.41.169 60 msec 152.63.41.153 20 msec 12 152.63.33.253 30 msec 152.63.35.157 20 msec 152.63.38.250 20 msec 13 152.179.50.54 20 msec 20 msec 20 msec <snip> _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (5)
-
Adam Rothschild -
Dovid B -
Jay Ashworth -
Jeremy Chadwick -
Michael Hertrick