Acknowledged. I'm in Mountain View but you can clearly see where the reconvergence takes place in the mtrs (between Oakland and SF Sutro, at least from the forward path). Naturally I have no way of knowing where the actual issue is. I'm sure you know by now most of our (as a customer) packets get sent either down to San Jose or out of Oakland (and I believe the Oakland network is something that got rolled out somewhat recently; I remember seeing it suddenly appear in my mtrs one night not that long ago). I sent a tweet to @ComcastCares pointing them to this thread. I also imagine the issue will get worse since it's now 1700 and people are just starting to get off work... -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | On Wed, Oct 02, 2013 at 04:58:57PM -0700, Scott Howard wrote:
Not just Oakland - I'm seeing issues in both the South Bay as well as on the Peninsular so I'm guessing it's pretty much the whole SF Bay Area.
Same symptoms as you're seeing - packet loss that seems to be within their network, and as a result fairly ordinary speeds.
Scott
On Wed, Oct 2, 2013 at 4:54 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
This just happened -- looks like something in Comcast's network within Oakland broke, reconverged (quickly and seamless), but issues remain.
Ren Provo et al, if you're lurking, might want to make someone aware of this (if they aren't already). :-)
src: 76.102.14.35 dst: 208.79.90.130
=== Wed Oct 2 16:14:00 PDT 2013 (1380755640) Start: Wed Oct 2 16:14:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.4 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 9.6 10.8 8.1 73.5 3.|-- 68.86.249.253 0.0% 40 40 8.9 10.2 8.3 62.2 4.|-- 69.139.199.98 0.0% 40 40 10.6 13.1 9.8 47.4 5.|-- 68.85.155.14 0.0% 40 40 12.0 16.8 11.4 87.7 6.|-- 68.86.90.157 0.0% 40 40 14.6 17.8 12.6 87.5 7.|-- 68.86.89.210 0.0% 40 40 23.7 24.7 20.6 92.9 8.|-- 68.86.87.210 0.0% 40 40 25.9 25.0 20.7 99.0 9.|-- 173.167.57.138 0.0% 40 40 22.1 24.7 21.3 89.0 10.|-- 69.31.127.141 0.0% 40 40 20.5 22.0 19.8 67.1 11.|-- 69.31.124.114 0.0% 40 40 20.6 21.6 20.0 45.9 12.|-- 67.199.135.102 0.0% 40 40 39.9 36.5 21.1 231.0 13.|-- 208.79.90.130 0.0% 40 40 21.2 20.7 20.2 21.7 === END
=== Wed Oct 2 16:15:00 PDT 2013 (1380755700) Start: Wed Oct 2 16:15:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 8.1 9.4 8.1 13.6 3.|-- 68.86.249.253 0.0% 40 40 8.7 9.7 8.4 14.1 4.|-- 68.86.143.98 0.0% 40 40 13.1 13.1 10.2 21.1 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 9.|-- 173.167.57.138 0.0% 40 40 33.0 33.9 31.6 37.1 10.|-- 69.31.127.141 5.0% 40 38 30.2 31.4 29.7 39.3 11.|-- 69.31.124.114 7.5% 40 37 31.8 38.5 29.8 104.0 12.|-- 67.199.135.102 7.5% 40 37 31.1 36.6 30.6 119.7 13.|-- 208.79.90.130 7.5% 40 37 30.9 31.9 30.4 38.9 === END
68.86.249.253 = te-0-2-0-5-ur06.santaclara.ca.sfba.comcast.net 69.139.199.98 = te-1-1-0-11-ar01.oakland.ca.sfba.comcast.net 68.85.155.14 = be-90-ar01.sfsutro.ca.sfba.comcast.net 68.86.143.98 = te-1-1-0-5-ar01.oakland.ca.sfba.comcast.net
And packet loss has gradually been getting worse (impacting all forms of packets, not just TCP).
=== Wed Oct 2 16:45:00 PDT 2013 (1380757500) Start: Wed Oct 2 16:45:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 8.6 9.9 8.1 14.3 3.|-- 68.86.249.253 0.0% 40 40 9.2 10.2 8.4 16.7 4.|-- 68.86.143.98 0.0% 40 40 13.9 13.1 10.6 16.6 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 9.|-- 173.167.57.138 15.0% 40 34 34.6 34.8 31.7 41.7 10.|-- 69.31.127.141 25.0% 40 30 31.3 32.6 30.0 41.7 11.|-- 69.31.124.114 5.0% 40 38 31.8 33.5 30.5 58.6 12.|-- 67.199.135.102 12.8% 39 34 30.9 37.1 30.9 77.3 13.|-- 208.79.90.130 7.7% 39 36 32.9 32.3 30.8 37.3 === END
And here's what I see on the return path, for a brief run:
src: 208.79.90.130 dst: 76.102.14.35
Host Loss% Snt Rcv Last Avg Best Wrst 1. 208.79.90.129 0.0% 93 93 1.5 6.1 0.9 163.0 2. s7.lax.arpnetworks.com 0.0% 93 93 0.7 11.2 0.5 150.5 3. ge0-15.as01.lax07.mzima.net 0.0% 93 93 4.1 2.4 0.4 64.6 4. ae3-50g.cr1.lax1.us.nlayer.net 0.0% 93 93 0.6 2.2 0.4 56.0 5. ae1-50g.ar1.lax1.us.nlayer.net 1.1% 93 92 3.1 3.3 1.3 6.4 6. be-102-pe01.600wseventh.ca.ibone.comcast.ne 0.0% 93 93 3.8 3.1 1.1 5.4 7. pos-0-8-0-0-cr01.losangeles.ca.ibone.comcas 0.0% 93 93 3.0 2.9 1.5 6.1 8. he-2-15-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 92 92 8.6 10.3 8.1 12.5 9. he-0-9-0-0-ar01.sfsutro.ca.sfba.comcast.net 0.0% 92 92 12.5 11.6 9.8 13.6 10. be-100-ar01.oakland.ca.sfba.comcast.net 3.3% 92 88 22.0 25.2 21.9 49.6 11. te-0-4-0-2-ur06.santaclara.ca.sfba.comcast. 3.3% 92 89 23.8 23.8 23.0 25.4 12. te-6-0-acr03.santaclara.ca.sfba.comcast.net 9.8% 92 83 23.8 24.4 22.6 46.3 13. c-76-102-14-35.hsd1.ca.comcast.net 6.5% 92 86 36.3 31.7 30.5 38.3
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages