
Anyone else seeing packet loss and connectivity issues on Comcast in NoVA? We have several clients on Comcast Business with 1000ms+ pings and dropped packets.

Can you provide some traceroutes? Thanks, John On Thu, Feb 25, 2016 at 9:16 AM, Kenneth Padgett via Outages < outages@outages.org> wrote:
Anyone else seeing packet loss and connectivity issues on Comcast in NoVA? We have several clients on Comcast Business with 1000ms+ pings and dropped packets.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

It's sporadic, but here's an example: Tracing route to google-public-dns-a.google.com [8.8.8.8] over a maximum of 30 hops: 1 2 ms 1 ms 1 ms [custIP]-static.hfc.comcastbusiness.net 2 63 ms 23 ms 29 ms 96.120.106.81 3 18 ms 15 ms 17 ms xe-1-3-0-32767-sur01.catharpin.va.bad.comcast.net [68.85.147.5] 4 24 ms 22 ms 22 ms ae-40-0-ar01.whitemarsh.md.bad.comcast.net [68.85.133.53] 5 63 ms 18 ms 36 ms ae-0-0-ar01.capitolhghts.md.bad.comcast.net [68.85.67.201] 6 71 ms 55 ms 43 ms hu-0-1-0-3-cr02.ashburn.va.ibone.comcast.net [68.86.90.229] 7 394 ms 445 ms * he-0-14-0-1-pe04.ashburn.va.ibone.comcast.net [68.86.86.42] 8 74 ms 72 ms 65 ms as36040-2-c.ashburn.va.ibone.comcast.net [75.149.229.86] 9 116 ms 143 ms 122 ms 209.85.242.140 10 23 ms 25 ms 26 ms 209.85.143.210 11 * 432 ms 26 ms 216.239.48.160 12 28 ms 26 ms 26 ms 216.239.48.235 13 * * * Request timed out. 14 33 ms 29 ms 25 ms google-public-dns-a.google.com [8.8.8.8] Trace complete. Pinging 8.8.8.8 with 32 bytes of data: Reply from 8.8.8.8: bytes=32 time=27ms TTL=42 Reply from 8.8.8.8: bytes=32 time=151ms TTL=42 Reply from 8.8.8.8: bytes=32 time=27ms TTL=42 Reply from 8.8.8.8: bytes=32 time=29ms TTL=42 Reply from 8.8.8.8: bytes=32 time=27ms TTL=42 Reply from 8.8.8.8: bytes=32 time=26ms TTL=42 Reply from 8.8.8.8: bytes=32 time=545ms TTL=42 Reply from 8.8.8.8: bytes=32 time=30ms TTL=42 Reply from 8.8.8.8: bytes=32 time=42ms TTL=42 Reply from 8.8.8.8: bytes=32 time=26ms TTL=42 Reply from 8.8.8.8: bytes=32 time=24ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=203ms TTL=42 Reply from 8.8.8.8: bytes=32 time=386ms TTL=42 Request timed out. Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=24ms TTL=42 Reply from 8.8.8.8: bytes=32 time=171ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=28ms TTL=42 Reply from 8.8.8.8: bytes=32 time=26ms TTL=42 Reply from 8.8.8.8: bytes=32 time=27ms TTL=42 Reply from 8.8.8.8: bytes=32 time=29ms TTL=42 Reply from 8.8.8.8: bytes=32 time=33ms TTL=42 Reply from 8.8.8.8: bytes=32 time=34ms TTL=42 Reply from 8.8.8.8: bytes=32 time=28ms TTL=42 Reply from 8.8.8.8: bytes=32 time=125ms TTL=42 Reply from 8.8.8.8: bytes=32 time=24ms TTL=42 Reply from 8.8.8.8: bytes=32 time=28ms TTL=42 Request timed out. Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=29ms TTL=42 Reply from 8.8.8.8: bytes=32 time=31ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=23ms TTL=42 Reply from 8.8.8.8: bytes=32 time=27ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=26ms TTL=42 Reply from 8.8.8.8: bytes=32 time=42ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=24ms TTL=42 Reply from 8.8.8.8: bytes=32 time=27ms TTL=42 Reply from 8.8.8.8: bytes=32 time=26ms TTL=42 Reply from 8.8.8.8: bytes=32 time=32ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Request timed out. Reply from 8.8.8.8: bytes=32 time=42ms TTL=42 Reply from 8.8.8.8: bytes=32 time=26ms TTL=42 Reply from 8.8.8.8: bytes=32 time=24ms TTL=42 Reply from 8.8.8.8: bytes=32 time=409ms TTL=42 Reply from 8.8.8.8: bytes=32 time=31ms TTL=42 On Thu, Feb 25, 2016 at 11:20 AM, John Neiberger <jneiberger@gmail.com> wrote:
Can you provide some traceroutes?
Thanks, John
On Thu, Feb 25, 2016 at 9:16 AM, Kenneth Padgett via Outages < outages@outages.org> wrote:
Anyone else seeing packet loss and connectivity issues on Comcast in NoVA? We have several clients on Comcast Business with 1000ms+ pings and dropped packets.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Thanks. Can you run MTR, as well? I've run thousands of ping tests to 8.8.8.8 from cr02.ashburn.va.ibone with no loss and an RTT averaging around 10ms. I'd like to what MTR looks like after it's been running for a bit. Thanks again, John On Thu, Feb 25, 2016 at 9:37 AM, Kenneth Padgett <kenneth@itlifesaver.com> wrote:
It's sporadic, but here's an example:
Tracing route to google-public-dns-a.google.com [8.8.8.8] over a maximum of 30 hops:
1 2 ms 1 ms 1 ms [custIP]-static.hfc.comcastbusiness.net 2 63 ms 23 ms 29 ms 96.120.106.81 3 18 ms 15 ms 17 ms xe-1-3-0-32767-sur01.catharpin.va.bad.comcast.net [68.85.147.5] 4 24 ms 22 ms 22 ms ae-40-0-ar01.whitemarsh.md.bad.comcast.net [68.85.133.53] 5 63 ms 18 ms 36 ms ae-0-0-ar01.capitolhghts.md.bad.comcast.net [68.85.67.201] 6 71 ms 55 ms 43 ms hu-0-1-0-3-cr02.ashburn.va.ibone.comcast.net [68.86.90.229] 7 394 ms 445 ms * he-0-14-0-1-pe04.ashburn.va.ibone.comcast.net [68.86.86.42] 8 74 ms 72 ms 65 ms as36040-2-c.ashburn.va.ibone.comcast.net [75.149.229.86] 9 116 ms 143 ms 122 ms 209.85.242.140 10 23 ms 25 ms 26 ms 209.85.143.210 11 * 432 ms 26 ms 216.239.48.160 12 28 ms 26 ms 26 ms 216.239.48.235 13 * * * Request timed out. 14 33 ms 29 ms 25 ms google-public-dns-a.google.com [8.8.8.8]
Trace complete.
Pinging 8.8.8.8 with 32 bytes of data: Reply from 8.8.8.8: bytes=32 time=27ms TTL=42 Reply from 8.8.8.8: bytes=32 time=151ms TTL=42 Reply from 8.8.8.8: bytes=32 time=27ms TTL=42 Reply from 8.8.8.8: bytes=32 time=29ms TTL=42 Reply from 8.8.8.8: bytes=32 time=27ms TTL=42 Reply from 8.8.8.8: bytes=32 time=26ms TTL=42 Reply from 8.8.8.8: bytes=32 time=545ms TTL=42 Reply from 8.8.8.8: bytes=32 time=30ms TTL=42 Reply from 8.8.8.8: bytes=32 time=42ms TTL=42 Reply from 8.8.8.8: bytes=32 time=26ms TTL=42 Reply from 8.8.8.8: bytes=32 time=24ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=203ms TTL=42 Reply from 8.8.8.8: bytes=32 time=386ms TTL=42 Request timed out. Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=24ms TTL=42 Reply from 8.8.8.8: bytes=32 time=171ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=28ms TTL=42 Reply from 8.8.8.8: bytes=32 time=26ms TTL=42 Reply from 8.8.8.8: bytes=32 time=27ms TTL=42 Reply from 8.8.8.8: bytes=32 time=29ms TTL=42 Reply from 8.8.8.8: bytes=32 time=33ms TTL=42 Reply from 8.8.8.8: bytes=32 time=34ms TTL=42 Reply from 8.8.8.8: bytes=32 time=28ms TTL=42 Reply from 8.8.8.8: bytes=32 time=125ms TTL=42 Reply from 8.8.8.8: bytes=32 time=24ms TTL=42 Reply from 8.8.8.8: bytes=32 time=28ms TTL=42 Request timed out. Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=29ms TTL=42 Reply from 8.8.8.8: bytes=32 time=31ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=23ms TTL=42 Reply from 8.8.8.8: bytes=32 time=27ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=26ms TTL=42 Reply from 8.8.8.8: bytes=32 time=42ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=24ms TTL=42 Reply from 8.8.8.8: bytes=32 time=27ms TTL=42 Reply from 8.8.8.8: bytes=32 time=26ms TTL=42 Reply from 8.8.8.8: bytes=32 time=32ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Request timed out. Reply from 8.8.8.8: bytes=32 time=42ms TTL=42 Reply from 8.8.8.8: bytes=32 time=26ms TTL=42 Reply from 8.8.8.8: bytes=32 time=24ms TTL=42 Reply from 8.8.8.8: bytes=32 time=409ms TTL=42 Reply from 8.8.8.8: bytes=32 time=31ms TTL=42
On Thu, Feb 25, 2016 at 11:20 AM, John Neiberger <jneiberger@gmail.com> wrote:
Can you provide some traceroutes?
Thanks, John
On Thu, Feb 25, 2016 at 9:16 AM, Kenneth Padgett via Outages < outages@outages.org> wrote:
Anyone else seeing packet loss and connectivity issues on Comcast in NoVA? We have several clients on Comcast Business with 1000ms+ pings and dropped packets.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Yeah, I can run something similar. No *nix boxes at these sites. Here's some more examples of what we're seeing: 1 2 ms 2 ms 1 ms [site1]-richmond.hfc.comcastbusiness.net 2 16 ms 18 ms 13 ms 96.120.18.169 3 676 ms 869 ms 590 ms c-71-63-100-206.hsd1.va.comcast.net [71.63.100.206] 4 1194 ms 105 ms 173 ms [site2]-static.hfc.comcastbusiness.net Ping statistics for [site1 to site3]: Packets: Sent = 519, Received = 370, Lost = 149 (28% loss), Approximate round trip times in milli-seconds: Minimum = 35ms, Maximum = 2483ms, Average = 379ms Ping statistics for [site1 to site2]: Packets: Sent = 209, Received = 209, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 26ms, Maximum = 2072ms, Average = 518ms On Thu, Feb 25, 2016 at 11:47 AM, John Neiberger <jneiberger@gmail.com> wrote:
Thanks. Can you run MTR, as well? I've run thousands of ping tests to 8.8.8.8 from cr02.ashburn.va.ibone with no loss and an RTT averaging around 10ms. I'd like to what MTR looks like after it's been running for a bit.
Thanks again, John
On Thu, Feb 25, 2016 at 9:37 AM, Kenneth Padgett <kenneth@itlifesaver.com> wrote:
It's sporadic, but here's an example:
Tracing route to google-public-dns-a.google.com [8.8.8.8] over a maximum of 30 hops:
1 2 ms 1 ms 1 ms [custIP]-static.hfc.comcastbusiness.net 2 63 ms 23 ms 29 ms 96.120.106.81 3 18 ms 15 ms 17 ms xe-1-3-0-32767-sur01.catharpin.va.bad.comcast.net [68.85.147.5] 4 24 ms 22 ms 22 ms ae-40-0-ar01.whitemarsh.md.bad.comcast.net [68.85.133.53] 5 63 ms 18 ms 36 ms ae-0-0-ar01.capitolhghts.md.bad.comcast.net [68.85.67.201] 6 71 ms 55 ms 43 ms hu-0-1-0-3-cr02.ashburn.va.ibone.comcast.net [68.86.90.229] 7 394 ms 445 ms * he-0-14-0-1-pe04.ashburn.va.ibone.comcast.net [68.86.86.42] 8 74 ms 72 ms 65 ms as36040-2-c.ashburn.va.ibone.comcast.net [75.149.229.86] 9 116 ms 143 ms 122 ms 209.85.242.140 10 23 ms 25 ms 26 ms 209.85.143.210 11 * 432 ms 26 ms 216.239.48.160 12 28 ms 26 ms 26 ms 216.239.48.235 13 * * * Request timed out. 14 33 ms 29 ms 25 ms google-public-dns-a.google.com [8.8.8.8]
Trace complete.
Pinging 8.8.8.8 with 32 bytes of data: Reply from 8.8.8.8: bytes=32 time=27ms TTL=42 Reply from 8.8.8.8: bytes=32 time=151ms TTL=42 Reply from 8.8.8.8: bytes=32 time=27ms TTL=42 Reply from 8.8.8.8: bytes=32 time=29ms TTL=42 Reply from 8.8.8.8: bytes=32 time=27ms TTL=42 Reply from 8.8.8.8: bytes=32 time=26ms TTL=42 Reply from 8.8.8.8: bytes=32 time=545ms TTL=42 Reply from 8.8.8.8: bytes=32 time=30ms TTL=42 Reply from 8.8.8.8: bytes=32 time=42ms TTL=42 Reply from 8.8.8.8: bytes=32 time=26ms TTL=42 Reply from 8.8.8.8: bytes=32 time=24ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=203ms TTL=42 Reply from 8.8.8.8: bytes=32 time=386ms TTL=42 Request timed out. Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=24ms TTL=42 Reply from 8.8.8.8: bytes=32 time=171ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=28ms TTL=42 Reply from 8.8.8.8: bytes=32 time=26ms TTL=42 Reply from 8.8.8.8: bytes=32 time=27ms TTL=42 Reply from 8.8.8.8: bytes=32 time=29ms TTL=42 Reply from 8.8.8.8: bytes=32 time=33ms TTL=42 Reply from 8.8.8.8: bytes=32 time=34ms TTL=42 Reply from 8.8.8.8: bytes=32 time=28ms TTL=42 Reply from 8.8.8.8: bytes=32 time=125ms TTL=42 Reply from 8.8.8.8: bytes=32 time=24ms TTL=42 Reply from 8.8.8.8: bytes=32 time=28ms TTL=42 Request timed out. Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=29ms TTL=42 Reply from 8.8.8.8: bytes=32 time=31ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=23ms TTL=42 Reply from 8.8.8.8: bytes=32 time=27ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=26ms TTL=42 Reply from 8.8.8.8: bytes=32 time=42ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Reply from 8.8.8.8: bytes=32 time=24ms TTL=42 Reply from 8.8.8.8: bytes=32 time=27ms TTL=42 Reply from 8.8.8.8: bytes=32 time=26ms TTL=42 Reply from 8.8.8.8: bytes=32 time=32ms TTL=42 Reply from 8.8.8.8: bytes=32 time=25ms TTL=42 Request timed out. Reply from 8.8.8.8: bytes=32 time=42ms TTL=42 Reply from 8.8.8.8: bytes=32 time=26ms TTL=42 Reply from 8.8.8.8: bytes=32 time=24ms TTL=42 Reply from 8.8.8.8: bytes=32 time=409ms TTL=42 Reply from 8.8.8.8: bytes=32 time=31ms TTL=42
On Thu, Feb 25, 2016 at 11:20 AM, John Neiberger <jneiberger@gmail.com> wrote:
Can you provide some traceroutes?
Thanks, John
On Thu, Feb 25, 2016 at 9:16 AM, Kenneth Padgett via Outages < outages@outages.org> wrote:
Anyone else seeing packet loss and connectivity issues on Comcast in NoVA? We have several clients on Comcast Business with 1000ms+ pings and dropped packets.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

On 2/25/2016 2:14 PM, Kenneth Padgett via Outages wrote:
Yeah, I can run something similar. No *nix boxes at these sites.
winmtr? (http://winmtr.net/)

Got that and PingPlotter running now, looks pretty good at the moment. It's highly intermittent with times of serious latency, 2000ms+ when it's bad, that's when we're getting the calls from users. Been that way since this AM. KP On Thu, Feb 25, 2016 at 2:20 PM, Mark Wiater via Outages < outages@outages.org> wrote:
On 2/25/2016 2:14 PM, Kenneth Padgett via Outages wrote:
Yeah, I can run something similar. No *nix boxes at these sites.
winmtr? (http://winmtr.net/)
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

I just saw another report of someone at a colo who was noticing intermittent bad response times from 8.8.8.8. They switched to 8.8.4.4 and response times improved. When you notice it acting up again, try 8.8.4.4 and see if it behaves differently. John On Thu, Feb 25, 2016 at 12:41 PM, Kenneth Padgett via Outages < outages@outages.org> wrote:
Got that and PingPlotter running now, looks pretty good at the moment. It's highly intermittent with times of serious latency, 2000ms+ when it's bad, that's when we're getting the calls from users. Been that way since this AM.
KP
On Thu, Feb 25, 2016 at 2:20 PM, Mark Wiater via Outages < outages@outages.org> wrote:
On 2/25/2016 2:14 PM, Kenneth Padgett via Outages wrote:
Yeah, I can run something similar. No *nix boxes at these sites.
winmtr? (http://winmtr.net/)
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

On 2/25/2016 13:41, Kenneth Padgett via Outages wrote:
Got that and PingPlotter running now, looks pretty good at the moment. It's highly intermittent with times of serious latency, 2000ms+ when it's bad, that's when we're getting the calls from users. Been that way since this AM.
Whenever I see something like this (above) after a report of possible congestion or failure, I think of the old days when I lived in the area there would be a "Sigalert" regarding congestion on the Harbor Freeway, and everybody and his brother would jump in his care to go have a look. The congestion would persist hours (and days) after the original cause (usually somebody getting a speeding ticket--which in itself would have been interesting to see). -- sed quis custodiet ipsos custodes? (Juvenal)

There's also a PowerShell version of MTR that I wrote. https://gist.github.com/tylerapplebaum/dc527a3bd875f11871e2 -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Mark Wiater via Outages Sent: Thursday, February 25, 2016 11:21 AM To: outages@outages.org Subject: Re: [outages] Comcast internet in Northern VA On 2/25/2016 2:14 PM, Kenneth Padgett via Outages wrote:
Yeah, I can run something similar. No *nix boxes at these sites.
winmtr? (http://winmtr.net/) _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages Attention: Information contained in this message and or attachments is intended only for the recipient(s) named above and may contain confidential and or privileged material that is protected under State or Federal law. If you are not the intended recipient, any disclosure, copying, distribution or action taken on it is prohibited. If you believe you have received this email in error, please contact the sender with a copy to compliance@ochin.org, delete this email and destroy all copies.
participants (5)
-
John Neiberger
-
Kenneth Padgett
-
Larry Sheldon
-
Mark Wiater
-
Tyler Applebaum