My previously posted IPv4 and IPv6 traceroutes show a 4 msec incremental hop between atl1 and frc1, too. If the host names are I accurate, I admit, too, that I don't understand how one can get between Atlanta and San Francisco in ~4 msec. My daily IPv6 tcptraceroute6 snapshot shows that I normally get to 2a03:2880:2110:3f01:face:b00c:: via that ATL hop, but Chicago to Atlanta taking almost 30 msec? More likely some asymmetric routing going on here. <snip> 7 sl-st31-chi-bu-20.v6.sprintlink.net (2600:0:2:1239:144:232:1:104) 16.012 ms 15.678 ms 15.624 ms 8 2600:4:2000::f (2600:4:2000::f) 15.428 ms 15.376 ms 15.396 ms 9 ae1.bb02.ord1.tfbnw.net (2620:0:1cff:dead:beee::452) 15.704 ms 15.603 ms 15.625 ms 10 ae12.bb02.atl1.tfbnw.net (2620:0:1cff:dead:beef::1336) 43.805 ms 36.058 ms 48.112 ms 11 ae11.bb04.frc1.tfbnw.net (2620:0:1cff:dead:beef::3be) 40.169 ms 40.171 ms 40.120 ms 12 ae40.dr02.frc1.tfbnw.net (2620:0:1cff:dead:beef::1c7) 39.875 ms 39.934 ms 39.774 ms 13 po1020.csw03d.frc1.tfbnw.net (2620:0:1cff:dead:beef::163f) 40.388 ms 40.266 ms 40.973 ms 14 * * * 15 www6-slb-ecmp-03-frc1.facebook.com (2a03:2880:2110:3f01:face:b00c::) 40.239 ms [open] 40.160 ms 40.212 ms Frank -----Original Message----- From: Jeremy Chadwick [mailto:jdc@koitsu.org] Sent: Monday, January 28, 2013 8:17 PM To: Frank Bulk Cc: 'Josh Luthman'; outages@outages.org Subject: Re: [outages] Facebook is down On Mon, Jan 28, 2013 at 07:44:13PM -0600, Frank Bulk wrote:
Appears to be inside of fb's network - prn1 = Prineville and San Francisco = frc1. {snip}
$ mtr 173.252.100.16 Host Loss% Snt Rcv Last Avg Best Wrst 1. gw.home.lan 0.0% 7 7 0.3 0.3 0.2 0.4 2. c-67-180-84-1.hsd1.ca.comcast.net 0.0% 6 6 21.9 23.8 21.9 27.0 3. te-0-0-0-12-ur05.santaclara.ca.sfba.comcast 0.0% 6 6 10.1 9.9 8.5 11.5 4. te-1-1-0-4-ar01.sfsutro.ca.sfba.comcast.net 0.0% 6 6 12.7 15.8 12.7 20.7 5. he-1-5-0-0-cr01.sanjose.ca.ibone.comcast.ne 0.0% 6 6 18.5 18.9 14.6 21.1 6. pos-2-11-0-0-cr01.losangeles.ca.ibone.comca 0.0% 6 6 22.1 22.4 20.7 24.0 7. xe-0-0-1-0-pe01.onewilshire.ca.ibone.comcas 0.0% 6 6 21.0 21.3 20.9 21.6 8. 173.167.59.58 0.0% 6 6 20.7 21.9 20.1 25.0 9. ae1.bb02.lax1.tfbnw.net 0.0% 6 6 20.9 21.6 20.4 23.4 10. ae13.bb02.atl1.tfbnw.net 0.0% 6 6 84.1 87.5 82.3 99.5 11. ae7.bb02.frc1.tfbnw.net 0.0% 6 6 87.7 90.2 87.1 103.0 12. ae2.dr03.frc1.tfbnw.net 0.0% 6 6 83.5 83.8 83.5 85.2 13. ??? To me that looks like the forward path is: 1. LAN 2. Comcast: Mountain View, CA 3. Comcast: Santa Clara, CA 4. Comcast: San Francisco, CA (always been an assumption on my part) 5. Comcast: San Jose 6. Comcast: Los Angeles 7. Comcast: Los Angeles (One Wilshire) 8. Comcast: possible FB<->Comcast peering point 9. Facebook: Los Angeles 10. Facebook: Atlanta (note 60ms+ latency jump for cross-country) 11. Facebook: ??? 11. Facebook: ??? If it went from LAX->ATL->SFO, I would expect to see the latency jump ~60-80ms at #9/#10, and then again at #10/#11. But again, this is the forward-path only. And I refuse to believe FRC means Franca Airport in Brazil. ;-) -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |