Comcast in Silicon Valley -- Oakland or thereabouts just broke?
This just happened -- looks like something in Comcast's network within Oakland broke, reconverged (quickly and seamless), but issues remain. Ren Provo et al, if you're lurking, might want to make someone aware of this (if they aren't already). :-) src: 76.102.14.35 dst: 208.79.90.130 === Wed Oct 2 16:14:00 PDT 2013 (1380755640) Start: Wed Oct 2 16:14:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.4 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 9.6 10.8 8.1 73.5 3.|-- 68.86.249.253 0.0% 40 40 8.9 10.2 8.3 62.2 4.|-- 69.139.199.98 0.0% 40 40 10.6 13.1 9.8 47.4 5.|-- 68.85.155.14 0.0% 40 40 12.0 16.8 11.4 87.7 6.|-- 68.86.90.157 0.0% 40 40 14.6 17.8 12.6 87.5 7.|-- 68.86.89.210 0.0% 40 40 23.7 24.7 20.6 92.9 8.|-- 68.86.87.210 0.0% 40 40 25.9 25.0 20.7 99.0 9.|-- 173.167.57.138 0.0% 40 40 22.1 24.7 21.3 89.0 10.|-- 69.31.127.141 0.0% 40 40 20.5 22.0 19.8 67.1 11.|-- 69.31.124.114 0.0% 40 40 20.6 21.6 20.0 45.9 12.|-- 67.199.135.102 0.0% 40 40 39.9 36.5 21.1 231.0 13.|-- 208.79.90.130 0.0% 40 40 21.2 20.7 20.2 21.7 === END === Wed Oct 2 16:15:00 PDT 2013 (1380755700) Start: Wed Oct 2 16:15:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 8.1 9.4 8.1 13.6 3.|-- 68.86.249.253 0.0% 40 40 8.7 9.7 8.4 14.1 4.|-- 68.86.143.98 0.0% 40 40 13.1 13.1 10.2 21.1 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 9.|-- 173.167.57.138 0.0% 40 40 33.0 33.9 31.6 37.1 10.|-- 69.31.127.141 5.0% 40 38 30.2 31.4 29.7 39.3 11.|-- 69.31.124.114 7.5% 40 37 31.8 38.5 29.8 104.0 12.|-- 67.199.135.102 7.5% 40 37 31.1 36.6 30.6 119.7 13.|-- 208.79.90.130 7.5% 40 37 30.9 31.9 30.4 38.9 === END 68.86.249.253 = te-0-2-0-5-ur06.santaclara.ca.sfba.comcast.net 69.139.199.98 = te-1-1-0-11-ar01.oakland.ca.sfba.comcast.net 68.85.155.14 = be-90-ar01.sfsutro.ca.sfba.comcast.net 68.86.143.98 = te-1-1-0-5-ar01.oakland.ca.sfba.comcast.net And packet loss has gradually been getting worse (impacting all forms of packets, not just TCP). === Wed Oct 2 16:45:00 PDT 2013 (1380757500) Start: Wed Oct 2 16:45:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 8.6 9.9 8.1 14.3 3.|-- 68.86.249.253 0.0% 40 40 9.2 10.2 8.4 16.7 4.|-- 68.86.143.98 0.0% 40 40 13.9 13.1 10.6 16.6 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 9.|-- 173.167.57.138 15.0% 40 34 34.6 34.8 31.7 41.7 10.|-- 69.31.127.141 25.0% 40 30 31.3 32.6 30.0 41.7 11.|-- 69.31.124.114 5.0% 40 38 31.8 33.5 30.5 58.6 12.|-- 67.199.135.102 12.8% 39 34 30.9 37.1 30.9 77.3 13.|-- 208.79.90.130 7.7% 39 36 32.9 32.3 30.8 37.3 === END And here's what I see on the return path, for a brief run: src: 208.79.90.130 dst: 76.102.14.35 Host Loss% Snt Rcv Last Avg Best Wrst 1. 208.79.90.129 0.0% 93 93 1.5 6.1 0.9 163.0 2. s7.lax.arpnetworks.com 0.0% 93 93 0.7 11.2 0.5 150.5 3. ge0-15.as01.lax07.mzima.net 0.0% 93 93 4.1 2.4 0.4 64.6 4. ae3-50g.cr1.lax1.us.nlayer.net 0.0% 93 93 0.6 2.2 0.4 56.0 5. ae1-50g.ar1.lax1.us.nlayer.net 1.1% 93 92 3.1 3.3 1.3 6.4 6. be-102-pe01.600wseventh.ca.ibone.comcast.ne 0.0% 93 93 3.8 3.1 1.1 5.4 7. pos-0-8-0-0-cr01.losangeles.ca.ibone.comcas 0.0% 93 93 3.0 2.9 1.5 6.1 8. he-2-15-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 92 92 8.6 10.3 8.1 12.5 9. he-0-9-0-0-ar01.sfsutro.ca.sfba.comcast.net 0.0% 92 92 12.5 11.6 9.8 13.6 10. be-100-ar01.oakland.ca.sfba.comcast.net 3.3% 92 88 22.0 25.2 21.9 49.6 11. te-0-4-0-2-ur06.santaclara.ca.sfba.comcast. 3.3% 92 89 23.8 23.8 23.0 25.4 12. te-6-0-acr03.santaclara.ca.sfba.comcast.net 9.8% 92 83 23.8 24.4 22.6 46.3 13. c-76-102-14-35.hsd1.ca.comcast.net 6.5% 92 86 36.3 31.7 30.5 38.3 -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
Not just Oakland - I'm seeing issues in both the South Bay as well as on the Peninsular so I'm guessing it's pretty much the whole SF Bay Area. Same symptoms as you're seeing - packet loss that seems to be within their network, and as a result fairly ordinary speeds. Scott On Wed, Oct 2, 2013 at 4:54 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
This just happened -- looks like something in Comcast's network within Oakland broke, reconverged (quickly and seamless), but issues remain.
Ren Provo et al, if you're lurking, might want to make someone aware of this (if they aren't already). :-)
src: 76.102.14.35 dst: 208.79.90.130
=== Wed Oct 2 16:14:00 PDT 2013 (1380755640) Start: Wed Oct 2 16:14:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.4 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 9.6 10.8 8.1 73.5 3.|-- 68.86.249.253 0.0% 40 40 8.9 10.2 8.3 62.2 4.|-- 69.139.199.98 0.0% 40 40 10.6 13.1 9.8 47.4 5.|-- 68.85.155.14 0.0% 40 40 12.0 16.8 11.4 87.7 6.|-- 68.86.90.157 0.0% 40 40 14.6 17.8 12.6 87.5 7.|-- 68.86.89.210 0.0% 40 40 23.7 24.7 20.6 92.9 8.|-- 68.86.87.210 0.0% 40 40 25.9 25.0 20.7 99.0 9.|-- 173.167.57.138 0.0% 40 40 22.1 24.7 21.3 89.0 10.|-- 69.31.127.141 0.0% 40 40 20.5 22.0 19.8 67.1 11.|-- 69.31.124.114 0.0% 40 40 20.6 21.6 20.0 45.9 12.|-- 67.199.135.102 0.0% 40 40 39.9 36.5 21.1 231.0 13.|-- 208.79.90.130 0.0% 40 40 21.2 20.7 20.2 21.7 === END
=== Wed Oct 2 16:15:00 PDT 2013 (1380755700) Start: Wed Oct 2 16:15:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 8.1 9.4 8.1 13.6 3.|-- 68.86.249.253 0.0% 40 40 8.7 9.7 8.4 14.1 4.|-- 68.86.143.98 0.0% 40 40 13.1 13.1 10.2 21.1 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 9.|-- 173.167.57.138 0.0% 40 40 33.0 33.9 31.6 37.1 10.|-- 69.31.127.141 5.0% 40 38 30.2 31.4 29.7 39.3 11.|-- 69.31.124.114 7.5% 40 37 31.8 38.5 29.8 104.0 12.|-- 67.199.135.102 7.5% 40 37 31.1 36.6 30.6 119.7 13.|-- 208.79.90.130 7.5% 40 37 30.9 31.9 30.4 38.9 === END
68.86.249.253 = te-0-2-0-5-ur06.santaclara.ca.sfba.comcast.net 69.139.199.98 = te-1-1-0-11-ar01.oakland.ca.sfba.comcast.net 68.85.155.14 = be-90-ar01.sfsutro.ca.sfba.comcast.net 68.86.143.98 = te-1-1-0-5-ar01.oakland.ca.sfba.comcast.net
And packet loss has gradually been getting worse (impacting all forms of packets, not just TCP).
=== Wed Oct 2 16:45:00 PDT 2013 (1380757500) Start: Wed Oct 2 16:45:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 8.6 9.9 8.1 14.3 3.|-- 68.86.249.253 0.0% 40 40 9.2 10.2 8.4 16.7 4.|-- 68.86.143.98 0.0% 40 40 13.9 13.1 10.6 16.6 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 9.|-- 173.167.57.138 15.0% 40 34 34.6 34.8 31.7 41.7 10.|-- 69.31.127.141 25.0% 40 30 31.3 32.6 30.0 41.7 11.|-- 69.31.124.114 5.0% 40 38 31.8 33.5 30.5 58.6 12.|-- 67.199.135.102 12.8% 39 34 30.9 37.1 30.9 77.3 13.|-- 208.79.90.130 7.7% 39 36 32.9 32.3 30.8 37.3 === END
And here's what I see on the return path, for a brief run:
src: 208.79.90.130 dst: 76.102.14.35
Host Loss% Snt Rcv Last Avg Best Wrst 1. 208.79.90.129 0.0% 93 93 1.5 6.1 0.9 163.0 2. s7.lax.arpnetworks.com 0.0% 93 93 0.7 11.2 0.5 150.5 3. ge0-15.as01.lax07.mzima.net 0.0% 93 93 4.1 2.4 0.4 64.6 4. ae3-50g.cr1.lax1.us.nlayer.net 0.0% 93 93 0.6 2.2 0.4 56.0 5. ae1-50g.ar1.lax1.us.nlayer.net 1.1% 93 92 3.1 3.3 1.3 6.4 6. be-102-pe01.600wseventh.ca.ibone.comcast.ne 0.0% 93 93 3.8 3.1 1.1 5.4 7. pos-0-8-0-0-cr01.losangeles.ca.ibone.comcas 0.0% 93 93 3.0 2.9 1.5 6.1 8. he-2-15-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 92 92 8.6 10.3 8.1 12.5 9. he-0-9-0-0-ar01.sfsutro.ca.sfba.comcast.net 0.0% 92 92 12.5 11.6 9.8 13.6 10. be-100-ar01.oakland.ca.sfba.comcast.net 3.3% 92 88 22.0 25.2 21.9 49.6 11. te-0-4-0-2-ur06.santaclara.ca.sfba.comcast. 3.3% 92 89 23.8 23.8 23.0 25.4 12. te-6-0-acr03.santaclara.ca.sfba.comcast.net 9.8% 92 83 23.8 24.4 22.6 46.3 13. c-76-102-14-35.hsd1.ca.comcast.net 6.5% 92 86 36.3 31.7 30.5 38.3
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Acknowledged. I'm in Mountain View but you can clearly see where the reconvergence takes place in the mtrs (between Oakland and SF Sutro, at least from the forward path). Naturally I have no way of knowing where the actual issue is. I'm sure you know by now most of our (as a customer) packets get sent either down to San Jose or out of Oakland (and I believe the Oakland network is something that got rolled out somewhat recently; I remember seeing it suddenly appear in my mtrs one night not that long ago). I sent a tweet to @ComcastCares pointing them to this thread. I also imagine the issue will get worse since it's now 1700 and people are just starting to get off work... -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | On Wed, Oct 02, 2013 at 04:58:57PM -0700, Scott Howard wrote:
Not just Oakland - I'm seeing issues in both the South Bay as well as on the Peninsular so I'm guessing it's pretty much the whole SF Bay Area.
Same symptoms as you're seeing - packet loss that seems to be within their network, and as a result fairly ordinary speeds.
Scott
On Wed, Oct 2, 2013 at 4:54 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
This just happened -- looks like something in Comcast's network within Oakland broke, reconverged (quickly and seamless), but issues remain.
Ren Provo et al, if you're lurking, might want to make someone aware of this (if they aren't already). :-)
src: 76.102.14.35 dst: 208.79.90.130
=== Wed Oct 2 16:14:00 PDT 2013 (1380755640) Start: Wed Oct 2 16:14:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.4 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 9.6 10.8 8.1 73.5 3.|-- 68.86.249.253 0.0% 40 40 8.9 10.2 8.3 62.2 4.|-- 69.139.199.98 0.0% 40 40 10.6 13.1 9.8 47.4 5.|-- 68.85.155.14 0.0% 40 40 12.0 16.8 11.4 87.7 6.|-- 68.86.90.157 0.0% 40 40 14.6 17.8 12.6 87.5 7.|-- 68.86.89.210 0.0% 40 40 23.7 24.7 20.6 92.9 8.|-- 68.86.87.210 0.0% 40 40 25.9 25.0 20.7 99.0 9.|-- 173.167.57.138 0.0% 40 40 22.1 24.7 21.3 89.0 10.|-- 69.31.127.141 0.0% 40 40 20.5 22.0 19.8 67.1 11.|-- 69.31.124.114 0.0% 40 40 20.6 21.6 20.0 45.9 12.|-- 67.199.135.102 0.0% 40 40 39.9 36.5 21.1 231.0 13.|-- 208.79.90.130 0.0% 40 40 21.2 20.7 20.2 21.7 === END
=== Wed Oct 2 16:15:00 PDT 2013 (1380755700) Start: Wed Oct 2 16:15:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 8.1 9.4 8.1 13.6 3.|-- 68.86.249.253 0.0% 40 40 8.7 9.7 8.4 14.1 4.|-- 68.86.143.98 0.0% 40 40 13.1 13.1 10.2 21.1 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 9.|-- 173.167.57.138 0.0% 40 40 33.0 33.9 31.6 37.1 10.|-- 69.31.127.141 5.0% 40 38 30.2 31.4 29.7 39.3 11.|-- 69.31.124.114 7.5% 40 37 31.8 38.5 29.8 104.0 12.|-- 67.199.135.102 7.5% 40 37 31.1 36.6 30.6 119.7 13.|-- 208.79.90.130 7.5% 40 37 30.9 31.9 30.4 38.9 === END
68.86.249.253 = te-0-2-0-5-ur06.santaclara.ca.sfba.comcast.net 69.139.199.98 = te-1-1-0-11-ar01.oakland.ca.sfba.comcast.net 68.85.155.14 = be-90-ar01.sfsutro.ca.sfba.comcast.net 68.86.143.98 = te-1-1-0-5-ar01.oakland.ca.sfba.comcast.net
And packet loss has gradually been getting worse (impacting all forms of packets, not just TCP).
=== Wed Oct 2 16:45:00 PDT 2013 (1380757500) Start: Wed Oct 2 16:45:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 8.6 9.9 8.1 14.3 3.|-- 68.86.249.253 0.0% 40 40 9.2 10.2 8.4 16.7 4.|-- 68.86.143.98 0.0% 40 40 13.9 13.1 10.6 16.6 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 9.|-- 173.167.57.138 15.0% 40 34 34.6 34.8 31.7 41.7 10.|-- 69.31.127.141 25.0% 40 30 31.3 32.6 30.0 41.7 11.|-- 69.31.124.114 5.0% 40 38 31.8 33.5 30.5 58.6 12.|-- 67.199.135.102 12.8% 39 34 30.9 37.1 30.9 77.3 13.|-- 208.79.90.130 7.7% 39 36 32.9 32.3 30.8 37.3 === END
And here's what I see on the return path, for a brief run:
src: 208.79.90.130 dst: 76.102.14.35
Host Loss% Snt Rcv Last Avg Best Wrst 1. 208.79.90.129 0.0% 93 93 1.5 6.1 0.9 163.0 2. s7.lax.arpnetworks.com 0.0% 93 93 0.7 11.2 0.5 150.5 3. ge0-15.as01.lax07.mzima.net 0.0% 93 93 4.1 2.4 0.4 64.6 4. ae3-50g.cr1.lax1.us.nlayer.net 0.0% 93 93 0.6 2.2 0.4 56.0 5. ae1-50g.ar1.lax1.us.nlayer.net 1.1% 93 92 3.1 3.3 1.3 6.4 6. be-102-pe01.600wseventh.ca.ibone.comcast.ne 0.0% 93 93 3.8 3.1 1.1 5.4 7. pos-0-8-0-0-cr01.losangeles.ca.ibone.comcas 0.0% 93 93 3.0 2.9 1.5 6.1 8. he-2-15-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 92 92 8.6 10.3 8.1 12.5 9. he-0-9-0-0-ar01.sfsutro.ca.sfba.comcast.net 0.0% 92 92 12.5 11.6 9.8 13.6 10. be-100-ar01.oakland.ca.sfba.comcast.net 3.3% 92 88 22.0 25.2 21.9 49.6 11. te-0-4-0-2-ur06.santaclara.ca.sfba.comcast. 3.3% 92 89 23.8 23.8 23.0 25.4 12. te-6-0-acr03.santaclara.ca.sfba.comcast.net 9.8% 92 83 23.8 24.4 22.6 46.3 13. c-76-102-14-35.hsd1.ca.comcast.net 6.5% 92 86 36.3 31.7 30.5 38.3
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I just checked into this. Looks like some sort of transport issue in sfba. The appropriate teams have been working on it for a little while. I haven't heard what the expected time to repair is yet. John On Oct 2, 2013 5:57 PM, "Jeremy Chadwick" <jdc@koitsu.org> wrote:
This just happened -- looks like something in Comcast's network within Oakland broke, reconverged (quickly and seamless), but issues remain.
Ren Provo et al, if you're lurking, might want to make someone aware of this (if they aren't already). :-)
src: 76.102.14.35 dst: 208.79.90.130
=== Wed Oct 2 16:14:00 PDT 2013 (1380755640) Start: Wed Oct 2 16:14:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.4 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 9.6 10.8 8.1 73.5 3.|-- 68.86.249.253 0.0% 40 40 8.9 10.2 8.3 62.2 4.|-- 69.139.199.98 0.0% 40 40 10.6 13.1 9.8 47.4 5.|-- 68.85.155.14 0.0% 40 40 12.0 16.8 11.4 87.7 6.|-- 68.86.90.157 0.0% 40 40 14.6 17.8 12.6 87.5 7.|-- 68.86.89.210 0.0% 40 40 23.7 24.7 20.6 92.9 8.|-- 68.86.87.210 0.0% 40 40 25.9 25.0 20.7 99.0 9.|-- 173.167.57.138 0.0% 40 40 22.1 24.7 21.3 89.0 10.|-- 69.31.127.141 0.0% 40 40 20.5 22.0 19.8 67.1 11.|-- 69.31.124.114 0.0% 40 40 20.6 21.6 20.0 45.9 12.|-- 67.199.135.102 0.0% 40 40 39.9 36.5 21.1 231.0 13.|-- 208.79.90.130 0.0% 40 40 21.2 20.7 20.2 21.7 === END
=== Wed Oct 2 16:15:00 PDT 2013 (1380755700) Start: Wed Oct 2 16:15:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 8.1 9.4 8.1 13.6 3.|-- 68.86.249.253 0.0% 40 40 8.7 9.7 8.4 14.1 4.|-- 68.86.143.98 0.0% 40 40 13.1 13.1 10.2 21.1 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 9.|-- 173.167.57.138 0.0% 40 40 33.0 33.9 31.6 37.1 10.|-- 69.31.127.141 5.0% 40 38 30.2 31.4 29.7 39.3 11.|-- 69.31.124.114 7.5% 40 37 31.8 38.5 29.8 104.0 12.|-- 67.199.135.102 7.5% 40 37 31.1 36.6 30.6 119.7 13.|-- 208.79.90.130 7.5% 40 37 30.9 31.9 30.4 38.9 === END
68.86.249.253 = te-0-2-0-5-ur06.santaclara.ca.sfba.comcast.net 69.139.199.98 = te-1-1-0-11-ar01.oakland.ca.sfba.comcast.net 68.85.155.14 = be-90-ar01.sfsutro.ca.sfba.comcast.net 68.86.143.98 = te-1-1-0-5-ar01.oakland.ca.sfba.comcast.net
And packet loss has gradually been getting worse (impacting all forms of packets, not just TCP).
=== Wed Oct 2 16:45:00 PDT 2013 (1380757500) Start: Wed Oct 2 16:45:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 8.6 9.9 8.1 14.3 3.|-- 68.86.249.253 0.0% 40 40 9.2 10.2 8.4 16.7 4.|-- 68.86.143.98 0.0% 40 40 13.9 13.1 10.6 16.6 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 9.|-- 173.167.57.138 15.0% 40 34 34.6 34.8 31.7 41.7 10.|-- 69.31.127.141 25.0% 40 30 31.3 32.6 30.0 41.7 11.|-- 69.31.124.114 5.0% 40 38 31.8 33.5 30.5 58.6 12.|-- 67.199.135.102 12.8% 39 34 30.9 37.1 30.9 77.3 13.|-- 208.79.90.130 7.7% 39 36 32.9 32.3 30.8 37.3 === END
And here's what I see on the return path, for a brief run:
src: 208.79.90.130 dst: 76.102.14.35
Host Loss% Snt Rcv Last Avg Best Wrst 1. 208.79.90.129 0.0% 93 93 1.5 6.1 0.9 163.0 2. s7.lax.arpnetworks.com 0.0% 93 93 0.7 11.2 0.5 150.5 3. ge0-15.as01.lax07.mzima.net 0.0% 93 93 4.1 2.4 0.4 64.6 4. ae3-50g.cr1.lax1.us.nlayer.net 0.0% 93 93 0.6 2.2 0.4 56.0 5. ae1-50g.ar1.lax1.us.nlayer.net 1.1% 93 92 3.1 3.3 1.3 6.4 6. be-102-pe01.600wseventh.ca.ibone.comcast.ne 0.0% 93 93 3.8 3.1 1.1 5.4 7. pos-0-8-0-0-cr01.losangeles.ca.ibone.comcas 0.0% 93 93 3.0 2.9 1.5 6.1 8. he-2-15-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 92 92 8.6 10.3 8.1 12.5 9. he-0-9-0-0-ar01.sfsutro.ca.sfba.comcast.net 0.0% 92 92 12.5 11.6 9.8 13.6 10. be-100-ar01.oakland.ca.sfba.comcast.net 3.3% 92 88 22.0 25.2 21.9 49.6 11. te-0-4-0-2-ur06.santaclara.ca.sfba.comcast. 3.3% 92 89 23.8 23.8 23.0 25.4 12. te-6-0-acr03.santaclara.ca.sfba.comcast.net 9.8% 92 83 23.8 24.4 22.6 46.3 13. c-76-102-14-35.hsd1.ca.comcast.net 6.5% 92 86 36.3 31.7 30.5 38.3
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
From Mountain View i do not see web browsing issues, however traceroutes to different websites (like google.com) break down in Oakland.
-Grant On Wed, Oct 2, 2013 at 5:18 PM, John Neiberger <jneiberger@gmail.com> wrote:
I just checked into this. Looks like some sort of transport issue in sfba. The appropriate teams have been working on it for a little while. I haven't heard what the expected time to repair is yet.
John On Oct 2, 2013 5:57 PM, "Jeremy Chadwick" <jdc@koitsu.org> wrote:
This just happened -- looks like something in Comcast's network within Oakland broke, reconverged (quickly and seamless), but issues remain.
Ren Provo et al, if you're lurking, might want to make someone aware of this (if they aren't already). :-)
src: 76.102.14.35 dst: 208.79.90.130
=== Wed Oct 2 16:14:00 PDT 2013 (1380755640) Start: Wed Oct 2 16:14:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.4 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 9.6 10.8 8.1 73.5 3.|-- 68.86.249.253 0.0% 40 40 8.9 10.2 8.3 62.2 4.|-- 69.139.199.98 0.0% 40 40 10.6 13.1 9.8 47.4 5.|-- 68.85.155.14 0.0% 40 40 12.0 16.8 11.4 87.7 6.|-- 68.86.90.157 0.0% 40 40 14.6 17.8 12.6 87.5 7.|-- 68.86.89.210 0.0% 40 40 23.7 24.7 20.6 92.9 8.|-- 68.86.87.210 0.0% 40 40 25.9 25.0 20.7 99.0 9.|-- 173.167.57.138 0.0% 40 40 22.1 24.7 21.3 89.0 10.|-- 69.31.127.141 0.0% 40 40 20.5 22.0 19.8 67.1 11.|-- 69.31.124.114 0.0% 40 40 20.6 21.6 20.0 45.9 12.|-- 67.199.135.102 0.0% 40 40 39.9 36.5 21.1 231.0 13.|-- 208.79.90.130 0.0% 40 40 21.2 20.7 20.2 21.7 === END
=== Wed Oct 2 16:15:00 PDT 2013 (1380755700) Start: Wed Oct 2 16:15:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 8.1 9.4 8.1 13.6 3.|-- 68.86.249.253 0.0% 40 40 8.7 9.7 8.4 14.1 4.|-- 68.86.143.98 0.0% 40 40 13.1 13.1 10.2 21.1 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 9.|-- 173.167.57.138 0.0% 40 40 33.0 33.9 31.6 37.1 10.|-- 69.31.127.141 5.0% 40 38 30.2 31.4 29.7 39.3 11.|-- 69.31.124.114 7.5% 40 37 31.8 38.5 29.8 104.0 12.|-- 67.199.135.102 7.5% 40 37 31.1 36.6 30.6 119.7 13.|-- 208.79.90.130 7.5% 40 37 30.9 31.9 30.4 38.9 === END
68.86.249.253 = te-0-2-0-5-ur06.santaclara.ca.sfba.comcast.net 69.139.199.98 = te-1-1-0-11-ar01.oakland.ca.sfba.comcast.net 68.85.155.14 = be-90-ar01.sfsutro.ca.sfba.comcast.net 68.86.143.98 = te-1-1-0-5-ar01.oakland.ca.sfba.comcast.net
And packet loss has gradually been getting worse (impacting all forms of packets, not just TCP).
=== Wed Oct 2 16:45:00 PDT 2013 (1380757500) Start: Wed Oct 2 16:45:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 8.6 9.9 8.1 14.3 3.|-- 68.86.249.253 0.0% 40 40 9.2 10.2 8.4 16.7 4.|-- 68.86.143.98 0.0% 40 40 13.9 13.1 10.6 16.6 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 9.|-- 173.167.57.138 15.0% 40 34 34.6 34.8 31.7 41.7 10.|-- 69.31.127.141 25.0% 40 30 31.3 32.6 30.0 41.7 11.|-- 69.31.124.114 5.0% 40 38 31.8 33.5 30.5 58.6 12.|-- 67.199.135.102 12.8% 39 34 30.9 37.1 30.9 77.3 13.|-- 208.79.90.130 7.7% 39 36 32.9 32.3 30.8 37.3 === END
And here's what I see on the return path, for a brief run:
src: 208.79.90.130 dst: 76.102.14.35
Host Loss% Snt Rcv Last Avg Best Wrst 1. 208.79.90.129 0.0% 93 93 1.5 6.1 0.9 163.0 2. s7.lax.arpnetworks.com 0.0% 93 93 0.7 11.2 0.5 150.5 3. ge0-15.as01.lax07.mzima.net 0.0% 93 93 4.1 2.4 0.4 64.6 4. ae3-50g.cr1.lax1.us.nlayer.net 0.0% 93 93 0.6 2.2 0.4 56.0 5. ae1-50g.ar1.lax1.us.nlayer.net 1.1% 93 92 3.1 3.3 1.3 6.4 6. be-102-pe01.600wseventh.ca.ibone.comcast.ne 0.0% 93 93 3.8 3.1 1.1 5.4 7. pos-0-8-0-0-cr01.losangeles.ca.ibone.comcas 0.0% 93 93 3.0 2.9 1.5 6.1 8. he-2-15-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 92 92 8.6 10.3 8.1 12.5 9. he-0-9-0-0-ar01.sfsutro.ca.sfba.comcast.net 0.0% 92 92 12.5 11.6 9.8 13.6 10. be-100-ar01.oakland.ca.sfba.comcast.net 3.3% 92 88 22.0 25.2 21.9 49.6 11. te-0-4-0-2-ur06.santaclara.ca.sfba.comcast. 3.3% 92 89 23.8 23.8 23.0 25.4 12. te-6-0-acr03.santaclara.ca.sfba.comcast.net 9.8% 92 83 23.8 24.4 22.6 46.3 13. c-76-102-14-35.hsd1.ca.comcast.net 6.5% 92 86 36.3 31.7 30.5 38.3
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Follow-up: Since roughly 18:14 PDT things look back to normal. Others who saw issues may want to re-test and follow up as necessary. Thanks everyone! -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | On Wed, Oct 02, 2013 at 04:54:34PM -0700, Jeremy Chadwick wrote:
This just happened -- looks like something in Comcast's network within Oakland broke, reconverged (quickly and seamless), but issues remain.
Ren Provo et al, if you're lurking, might want to make someone aware of this (if they aren't already). :-)
src: 76.102.14.35 dst: 208.79.90.130
=== Wed Oct 2 16:14:00 PDT 2013 (1380755640) Start: Wed Oct 2 16:14:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.4 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 9.6 10.8 8.1 73.5 3.|-- 68.86.249.253 0.0% 40 40 8.9 10.2 8.3 62.2 4.|-- 69.139.199.98 0.0% 40 40 10.6 13.1 9.8 47.4 5.|-- 68.85.155.14 0.0% 40 40 12.0 16.8 11.4 87.7 6.|-- 68.86.90.157 0.0% 40 40 14.6 17.8 12.6 87.5 7.|-- 68.86.89.210 0.0% 40 40 23.7 24.7 20.6 92.9 8.|-- 68.86.87.210 0.0% 40 40 25.9 25.0 20.7 99.0 9.|-- 173.167.57.138 0.0% 40 40 22.1 24.7 21.3 89.0 10.|-- 69.31.127.141 0.0% 40 40 20.5 22.0 19.8 67.1 11.|-- 69.31.124.114 0.0% 40 40 20.6 21.6 20.0 45.9 12.|-- 67.199.135.102 0.0% 40 40 39.9 36.5 21.1 231.0 13.|-- 208.79.90.130 0.0% 40 40 21.2 20.7 20.2 21.7 === END
=== Wed Oct 2 16:15:00 PDT 2013 (1380755700) Start: Wed Oct 2 16:15:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 8.1 9.4 8.1 13.6 3.|-- 68.86.249.253 0.0% 40 40 8.7 9.7 8.4 14.1 4.|-- 68.86.143.98 0.0% 40 40 13.1 13.1 10.2 21.1 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 9.|-- 173.167.57.138 0.0% 40 40 33.0 33.9 31.6 37.1 10.|-- 69.31.127.141 5.0% 40 38 30.2 31.4 29.7 39.3 11.|-- 69.31.124.114 7.5% 40 37 31.8 38.5 29.8 104.0 12.|-- 67.199.135.102 7.5% 40 37 31.1 36.6 30.6 119.7 13.|-- 208.79.90.130 7.5% 40 37 30.9 31.9 30.4 38.9 === END
68.86.249.253 = te-0-2-0-5-ur06.santaclara.ca.sfba.comcast.net 69.139.199.98 = te-1-1-0-11-ar01.oakland.ca.sfba.comcast.net 68.85.155.14 = be-90-ar01.sfsutro.ca.sfba.comcast.net 68.86.143.98 = te-1-1-0-5-ar01.oakland.ca.sfba.comcast.net
And packet loss has gradually been getting worse (impacting all forms of packets, not just TCP).
=== Wed Oct 2 16:45:00 PDT 2013 (1380757500) Start: Wed Oct 2 16:45:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 8.6 9.9 8.1 14.3 3.|-- 68.86.249.253 0.0% 40 40 9.2 10.2 8.4 16.7 4.|-- 68.86.143.98 0.0% 40 40 13.9 13.1 10.6 16.6 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 9.|-- 173.167.57.138 15.0% 40 34 34.6 34.8 31.7 41.7 10.|-- 69.31.127.141 25.0% 40 30 31.3 32.6 30.0 41.7 11.|-- 69.31.124.114 5.0% 40 38 31.8 33.5 30.5 58.6 12.|-- 67.199.135.102 12.8% 39 34 30.9 37.1 30.9 77.3 13.|-- 208.79.90.130 7.7% 39 36 32.9 32.3 30.8 37.3 === END
And here's what I see on the return path, for a brief run:
src: 208.79.90.130 dst: 76.102.14.35
Host Loss% Snt Rcv Last Avg Best Wrst 1. 208.79.90.129 0.0% 93 93 1.5 6.1 0.9 163.0 2. s7.lax.arpnetworks.com 0.0% 93 93 0.7 11.2 0.5 150.5 3. ge0-15.as01.lax07.mzima.net 0.0% 93 93 4.1 2.4 0.4 64.6 4. ae3-50g.cr1.lax1.us.nlayer.net 0.0% 93 93 0.6 2.2 0.4 56.0 5. ae1-50g.ar1.lax1.us.nlayer.net 1.1% 93 92 3.1 3.3 1.3 6.4 6. be-102-pe01.600wseventh.ca.ibone.comcast.ne 0.0% 93 93 3.8 3.1 1.1 5.4 7. pos-0-8-0-0-cr01.losangeles.ca.ibone.comcas 0.0% 93 93 3.0 2.9 1.5 6.1 8. he-2-15-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 92 92 8.6 10.3 8.1 12.5 9. he-0-9-0-0-ar01.sfsutro.ca.sfba.comcast.net 0.0% 92 92 12.5 11.6 9.8 13.6 10. be-100-ar01.oakland.ca.sfba.comcast.net 3.3% 92 88 22.0 25.2 21.9 49.6 11. te-0-4-0-2-ur06.santaclara.ca.sfba.comcast. 3.3% 92 89 23.8 23.8 23.0 25.4 12. te-6-0-acr03.santaclara.ca.sfba.comcast.net 9.8% 92 83 23.8 24.4 22.6 46.3 13. c-76-102-14-35.hsd1.ca.comcast.net 6.5% 92 86 36.3 31.7 30.5 38.3
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Confirmed - working fine from the South Bay, to the same hosts that were seeing problems earlier. Scott On Wed, Oct 2, 2013 at 6:38 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Follow-up:
Since roughly 18:14 PDT things look back to normal. Others who saw issues may want to re-test and follow up as necessary.
Thanks everyone!
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Wed, Oct 02, 2013 at 04:54:34PM -0700, Jeremy Chadwick wrote:
This just happened -- looks like something in Comcast's network within Oakland broke, reconverged (quickly and seamless), but issues remain.
Ren Provo et al, if you're lurking, might want to make someone aware of this (if they aren't already). :-)
src: 76.102.14.35 dst: 208.79.90.130
=== Wed Oct 2 16:14:00 PDT 2013 (1380755640) Start: Wed Oct 2 16:14:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.4 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 9.6 10.8 8.1 73.5 3.|-- 68.86.249.253 0.0% 40 40 8.9 10.2 8.3 62.2 4.|-- 69.139.199.98 0.0% 40 40 10.6 13.1 9.8 47.4 5.|-- 68.85.155.14 0.0% 40 40 12.0 16.8 11.4 87.7 6.|-- 68.86.90.157 0.0% 40 40 14.6 17.8 12.6 87.5 7.|-- 68.86.89.210 0.0% 40 40 23.7 24.7 20.6 92.9 8.|-- 68.86.87.210 0.0% 40 40 25.9 25.0 20.7 99.0 9.|-- 173.167.57.138 0.0% 40 40 22.1 24.7 21.3 89.0 10.|-- 69.31.127.141 0.0% 40 40 20.5 22.0 19.8 67.1 11.|-- 69.31.124.114 0.0% 40 40 20.6 21.6 20.0 45.9 12.|-- 67.199.135.102 0.0% 40 40 39.9 36.5 21.1 231.0 13.|-- 208.79.90.130 0.0% 40 40 21.2 20.7 20.2 21.7 === END
=== Wed Oct 2 16:15:00 PDT 2013 (1380755700) Start: Wed Oct 2 16:15:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 8.1 9.4 8.1 13.6 3.|-- 68.86.249.253 0.0% 40 40 8.7 9.7 8.4 14.1 4.|-- 68.86.143.98 0.0% 40 40 13.1 13.1 10.2 21.1 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 9.|-- 173.167.57.138 0.0% 40 40 33.0 33.9 31.6 37.1 10.|-- 69.31.127.141 5.0% 40 38 30.2 31.4 29.7 39.3 11.|-- 69.31.124.114 7.5% 40 37 31.8 38.5 29.8 104.0 12.|-- 67.199.135.102 7.5% 40 37 31.1 36.6 30.6 119.7 13.|-- 208.79.90.130 7.5% 40 37 30.9 31.9 30.4 38.9 === END
68.86.249.253 = te-0-2-0-5-ur06.santaclara.ca.sfba.comcast.net 69.139.199.98 = te-1-1-0-11-ar01.oakland.ca.sfba.comcast.net 68.85.155.14 = be-90-ar01.sfsutro.ca.sfba.comcast.net 68.86.143.98 = te-1-1-0-5-ar01.oakland.ca.sfba.comcast.net
And packet loss has gradually been getting worse (impacting all forms of packets, not just TCP).
=== Wed Oct 2 16:45:00 PDT 2013 (1380757500) Start: Wed Oct 2 16:45:00 2013 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 76.102.12.1 0.0% 40 40 8.6 9.9 8.1 14.3 3.|-- 68.86.249.253 0.0% 40 40 9.2 10.2 8.4 16.7 4.|-- 68.86.143.98 0.0% 40 40 13.9 13.1 10.6 16.6 5.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 8.|-- ??? 100.0 40 0 0.0 0.0 0.0 0.0 9.|-- 173.167.57.138 15.0% 40 34 34.6 34.8 31.7 41.7 10.|-- 69.31.127.141 25.0% 40 30 31.3 32.6 30.0 41.7 11.|-- 69.31.124.114 5.0% 40 38 31.8 33.5 30.5 58.6 12.|-- 67.199.135.102 12.8% 39 34 30.9 37.1 30.9 77.3 13.|-- 208.79.90.130 7.7% 39 36 32.9 32.3 30.8 37.3 === END
And here's what I see on the return path, for a brief run:
src: 208.79.90.130 dst: 76.102.14.35
Host Loss% Snt Rcv Last Avg Best Wrst 1. 208.79.90.129 0.0% 93 93 1.5 6.1 0.9 163.0 2. s7.lax.arpnetworks.com 0.0% 93 93 0.7 11.2 0.5 150.5 3. ge0-15.as01.lax07.mzima.net 0.0% 93 93 4.1 2.4 0.4 64.6 4. ae3-50g.cr1.lax1.us.nlayer.net 0.0% 93 93 0.6 2.2 0.4 56.0 5. ae1-50g.ar1.lax1.us.nlayer.net 1.1% 93 92 3.1 3.3 1.3 6.4 6. be-102-pe01.600wseventh.ca.ibone.comcast.ne 0.0% 93 93 3.8 3.1 1.1 5.4 7. pos-0-8-0-0-cr01.losangeles.ca.ibone.comcas 0.0% 93 93 3.0 2.9 1.5 6.1 8. he-2-15-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 92 92 8.6 10.3 8.1 12.5 9. he-0-9-0-0-ar01.sfsutro.ca.sfba.comcast.net 0.0% 92 92 12.5 11.6 9.8 13.6 10. be-100-ar01.oakland.ca.sfba.comcast.net 3.3% 92 88 22.0 25.2 21.9 49.6 11. te-0-4-0-2-ur06.santaclara.ca.sfba.comcast. 3.3% 92 89 23.8 23.8 23.0 25.4 12. te-6-0-acr03.santaclara.ca.sfba.comcast.net 9.8% 92 83 23.8 24.4 22.6 46.3 13. c-76-102-14-35.hsd1.ca.comcast.net 6.5% 92 86 36.3 31.7 30.5 38.3
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Wed, Oct 2, 2013 at 4:54 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
This just happened -- looks like something in Comcast's network within Oakland broke, reconverged (quickly and seamless), but issues remain.
Ren Provo et al, if you're lurking, might want to make someone aware of this (if they aren't already). :-)
If LinkedIn is to be believed, she's no longer with Comcast, but is instead firmly ensconced at Apple, working to bring your apple-fu even closer to you, one AS at a time... Matt
On Oct 2, 2013 7:50 PM, "Matthew Petach" <mpetach@netflight.com> wrote:
On Wed, Oct 2, 2013 at 4:54 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
This just happened -- looks like something in Comcast's network within Oakland broke, reconverged (quickly and seamless), but issues remain.
Ren Provo et al, if you're lurking, might want to make someone aware of this (if they aren't already). :-)
If LinkedIn is to be believed, she's no longer with Comcast, but is instead firmly ensconced at Apple, working to bring your apple-fu even closer to you, one AS at a time...
Matt
Sadly for us at Comcast, that is correct.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (5)
-
Grant Ridder -
Jeremy Chadwick -
John Neiberger -
Matthew Petach -
Scott Howard