I'm not on the Comcast backbone team, but for what it's worth I just ran MTR to 96.120.14.165 and to router1-fmt.linode.com for quite a while and never lost a single packet from my desktop inside Comcast. I don't think I'm hitting the problematic node, though. If I'm reading the results above correctly, in all cases the problem begins when traffic hits po4-20G.ar2.SJC2.gblx.net. Am I misreading that? Regards, John On Sat, Jan 19, 2013 at 1:20 PM, Constantine A. Murenin <mureninc@gmail.com>wrote:
On Sat, Jan 19, 2013 at 11:52:04AM -0800, Constantine A. Murenin wrote:
On Sat, 19 Jan 2013 at 11:04:14 -0800, Dylan N wrote:
I get packetloss on te-1-12-0-0-ar01.sfsutro.ca.sfba.comcast.net to my linode @ fmt at weekends and at nights, looks more like just too much load rather than an outage
In my case, all packet loss seem to be within gblx.net (e.g. no packet loss between Comcast and hetzner.de).
What I find strange is that the latency is still very good, even when there is a significant packet loss.
But my ssh connection between Comcast and Linode/Fremont is completely unusable for several seconds at a time during these events.
The loss seems to have continued throughout last night:
2013-01-19T01:50:04-0800 HOST: li163-### Snt Rcv Loss% Best Avg Wrst StDev 1. router1-fmt.linode.com 600 600 0.0% 0.5 6.6 268.8 18.5 2. 10gigabitethernet2-3.core1.fmt1.he.net 600 600 0.0% 0.5 12.8 1385. 66.3 3. 10gigabitethernet1-2.core1.sjc2.he.net 600 600 0.0% 0.7 4.4 28.9 5.1 4. Port-channel100.ar3.SJC2.gblx.net 600 600 0.0% 0.8 28.2 234.0 54.8 5. po6-20G.ar4.SJC2.gblx.net 600 579 3.5% 0.9 32.1 888.1 66.0 6. lag8.ar5.SJC2.gblx.net 600 570 5.0% 0.9 4.4 20.9 5.1 7. 208.178.58.2 600 572 4.7% 1.8 4.1 14.4 1.3 8. pos-2-7-0-0-cr01.sanjose.ca.ibone.comcast.net 600 575 4.2% 4.8 7.7 43.1 2.9 9. pos-1-4-0-0-cr01.sacramento.ca.ibone.comcast.net 600 574 4.3% 7.1 9.3 20.2 1.5 10. te-7-3-ar02.saltlakecity.ut.utah.comcast.net 600 573 4.5% 7.3 11.9 115.7 17.6 11. te-3-1-ur08.sacramento.ca.ccal.comcast.net 600 574 4.3% 7.5 8.3 155.6 6.6 12. te-1-0-0-ten05.sacramento.ca.ccal.comcast.net 600 581 3.2% 14.1 14.3 25.1 1.3 13. 74-93-180-##-Sacramento.hfc.comcastbusiness.net 600 572 4.7% 14.1 20.5 83.9 8.8 2013-01-19T02:00:04-0800
2013-01-19T01:50:44-0800 HOST: hype.####.########## Snt Rcv Loss% Best Avg Wrst StDev 1. logout.lan 60 60 0.0% 2.1 7.1 35.2 6.5 2. 192.168.1.1 60 60 0.0% 2.5 8.6 41.3 6.7 3. 10.1.10.1 60 60 0.0% 3.1 10.3 49.8 8.4 4. 96.120.14.165 60 60 0.0% 11.3 22.7 75.3 11.7 5. te-6-5-ur08.sacramento.ca.ccal.comcast.net 60 60 0.0% 12.3 22.8 66.1 12.1 6. te-0-2-0-2-ar03.sacramento.ca.sacra.comcast.net 60 60 0.0% 11.8 26.3 101.6 18.3 7. pos-0-3-0-0-cr01.sacramento.ca.ibone.comcast.net 60 60 0.0% 14.8 26.8 85.3 14.1 8. pos-0-3-0-0-cr01.sanjose.ca.ibone.comcast.net 60 60 0.0% 15.6 28.2 71.6 10.3 9. pos-0-1-0-0-pe01.11greatoaks.ca.ibone.comcast.net 60 60 0.0% 20.8 30.6 92.6 12.2 10. 208.178.58.1 60 60 0.0% 18.6 34.8 98.6 16.6 11. po4-20G.ar2.SJC2.gblx.net 60 60 0.0% 17.5 61.6 273.3 66.9 12. po1-20G.ar3.SJC2.gblx.net 60 47 21.7% 17.8 48.2 238.7 51.7 13. Hurrican-Electric-LLC.Port-channel100.ar3.SJC2.gblx.net 60 44 26.7% 17.1 30.5 69.8 11.6 14. 10gigabitethernet1-1.core1.fmt1.he.net 60 43 28.3% 21.5 55.2 598.8 87.9 15. router1-fmt.linode.com 60 44 26.7% 18.3 44.0 187.1 34.4 16. li163-###.members.linode.com 60 43 28.3% 17.8 29.0 68.1 8.6 2013-01-19T01:51:55-0800
It's been a 61% packet loss in the last minute... Ridiculous.
2013-01-19T12:15:15-0800 HOST: hype.####.########## Snt Rcv Loss% Best Avg Wrst StDev 1. logout.lan 60 60 0.0% 2.1 18.8 222.7 50.1 2. 192.168.1.1 60 60 0.0% 2.3 20.4 421.0 65.5 3. 10.1.10.1 60 60 0.0% 3.0 22.9 357.1 57.2 4. 96.120.14.165 60 60 0.0% 9.6 29.3 306.7 54.4 5. te-6-5-ur08.sacramento.ca.ccal.comcast.net 60 60 0.0% 10.3 34.2 265.3 51.9 6. te-0-2-0-2-ar03.sacramento.ca.sacra.comcast.net 60 60 0.0% 9.8 33.1 185.6 43.3 7. pos-0-4-0-0-cr01.sacramento.ca.ibone.comcast.net 60 60 0.0% 11.8 32.3 234.3 42.8 8. pos-0-3-0-0-cr01.sanjose.ca.ibone.comcast.net 60 60 0.0% 15.6 31.7 282.1 41.5 9. pos-0-1-0-0-pe01.11greatoaks.ca.ibone.comcast.net 60 60 0.0% 18.1 36.0 221.3 38.0 10. 208.178.58.1 60 60 0.0% 17.1 40.8 366.8 51.5 11. po4-20G.ar2.SJC2.gblx.net 60 59 1.7% 16.8 58.3 291.6 68.0 12. po1-20G.ar3.SJC2.gblx.net 60 46 23.3% 16.8 67.3 258.6 66.5 13. Hurrican-Electric-LLC.Port-channel100.ar3.SJC2.gblx.net 60 48 20.0% 17.3 34.9 155.8 32.0 14. 10gigabitethernet1-1.core1.fmt1.he.net 60 19 68.3% 18.3 50.4 117.8 32.2 15. router1-fmt.linode.com 60 22 63.3% 19.1 66.3 287.3 64.2 16. li163-###.members.linode.com 60 23 61.7% 18.3 42.5 213.8 48.7 2013-01-19T12:16:27-0800
Anyone from he.net / gblx.net / comcast.net to comment?
Best regards, Constantine. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages