
Oh yes, I've been seeing this for some time, probably since mid-April. At times we've seen Comcast prepending their AS multiple times on announcements through Level3, causing outbound to Comcast traffic via congested routes with Tata, rather than what would otherwise be the shorter, uncongested route via Level3. (Return traffic from Comcast in my case returns via Level3.) For traffic from a server in San Francisco connected via Level3 and XO, connecting to two Comcast cable modems in San Francisco, I see traffic go out via XO, Tata, Comcast via LA, with what looks like major congestion, then back up to SF. With the appearance of major congestion on the Tata<->Comcast link. I don't have any traffic which visiblity traverses NTT or nLayer, so I couldn't say if the scope goes beyond the Tata-Comcast link. It's particularly annoying receiving LA latency + 5-6% packet loss going to/from well connected locations a couple miles apart in SF. I'm experiencing it now composing this message over ssh. Server in SF to Comcast in SF: ... 5. te1-0-0d0.mcr1.fremont-ca.xo.net 0.0% 1.4 5.9 1.2 82.0 15.8 6. vb1500.rar3.sanjose-ca.us.xo.net 0.0% 11.1 8.4 1.5 13.3 3.5 7. 207.88.14.226.ptr.us.xo.net 6.0% 25.3 8.0 1.8 41.9 12.9 8. if-10-12.icore1.SQN-SanJose.as6453 44.9% 6.0 7.0 2.0 14.9 4.3 9. Vlan3260.icore2.SQN-SanJose.as6453 0.0% 3.1 6.6 2.3 13.5 3.4 10. if-4-2145.tcore1.SQN-SanJose.as645 0.0% 3.4 3.7 2.7 8.0 0.9 11. if-3-2.tcore2.LVW-LosAngeles.as645 0.0% 32.4 27.9 14.7 57.1 9.7 12. xe-0-2-0-0-pe01.onewilshire.ca.ibo 8.2% 39.4 30.3 28.3 72.3 7.1 13. te-2-0-0-6-cr01.losangeles.ca.ibon 6.1% 32.2 30.7 28.6 32.6 1.2 14. he-0-11-0-0-cr01.sacramento.ca.ibo 4.1% 30.6 31.5 29.6 33.3 1.0 15. he-0-4-0-0-ar01.oakland.ca.sfba.co 10.2% 30.5 32.3 30.4 34.3 1.1 16. te-8-4-ur04.santaclara.ca.sfba.com 8.2% 32.6 33.0 32.5 41.9 1.4 17. te-17-10-cdn08.sf19th.ca.sfba.comc 8.2% 43.0 60.5 31.2 830.2 117.4 ... The ComcastSF -> ServerInSF goes out via Level3 and sees no loss until the final hop, suggesting the problem is with the reverse path, above. (Most of the time, sometimes the first Comcast hops going out are very congested too, though that's not usually the case). And here's two smokepings of the issue, in which you can clearly see the timing of the issue you mentioned. SF server to SF cable modem, 2mi apart. http://cl.ly/image/040U3O2F0810 (last 30 hours) http://cl.ly/image/1D053a0T1k2I (last 10 days)