I just came across this on NANOG while troubleshooting something: http://mailman.nanog.org/pipermail/nanog/2013-September/061332.html If the issue described there is what I'm seeing, then it looks like some part of the 'net in the LA area is very very broken -- and worse, *has* been broken since roughly Sep 27 21:16:00 PDT (UTC-7). What I saw (and am still seeing): src IP: 208.79.90.130 (Southern CA, AS 25795 (I think)) dst IP: 67.18.187.25 (Texas, AS 21844 (I think)) === Fri Sep 27 21:14:00 PDT 2013 (1380341640) Start: Fri Sep 27 21:14:00 2013 HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst 1.|-- 208.79.90.129 0.0% 40 40 3.7 11.0 0.9 202.9 2.|-- 206.223.143.131 0.0% 40 40 0.6 0.8 0.5 8.5 3.|-- 173.192.18.140 2.5% 40 39 28.4 30.1 28.2 86.8 4.|-- 173.192.18.225 0.0% 40 40 29.0 29.2 28.7 40.9 5.|-- 70.87.255.66 0.0% 40 40 31.6 31.5 31.3 31.9 6.|-- 70.87.254.74 0.0% 40 40 29.0 31.5 28.6 60.3 7.|-- 67.18.7.90 0.0% 40 40 28.7 28.6 28.4 29.0 8.|-- 67.18.187.25 0.0% 40 40 31.9 31.5 31.3 31.9 === END === Fri Sep 27 21:15:00 PDT 2013 (1380341700) Start: Fri Sep 27 21:15:00 2013 HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst 1.|-- 208.79.90.129 0.0% 40 40 2.2 51.1 0.9 245.7 2.|-- 206.223.143.131 2.5% 40 39 0.9 12.3 0.5 183.2 3.|-- 173.192.18.140 5.0% 40 38 0.8 29.4 0.7 284.1 4.|-- 173.192.18.225 0.0% 40 40 0.8 17.1 0.5 177.8 5.|-- 70.87.255.66 30.0% 40 28 11.2 39.1 3.4 199.4 6.|-- 70.87.254.74 0.0% 40 40 11.0 21.1 0.5 121.5 7.|-- 67.18.7.90 0.0% 40 40 163.8 122.6 4.6 347.0 8.|-- 67.18.187.25 22.5% 40 31 171.6 81.4 0.5 173.8 === END === Fri Sep 27 21:16:00 PDT 2013 (1380341760) Start: Fri Sep 27 21:16:00 2013 HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst 1.|-- 208.79.90.129 0.0% 40 40 2.9 7.4 0.9 95.8 2.|-- 208.79.88.135 0.0% 40 40 0.6 15.8 0.4 182.3 3.|-- 129.250.198.185 0.0% 40 40 0.7 0.8 0.7 1.2 4.|-- 129.250.5.69 0.0% 40 40 0.7 5.1 0.6 34.1 5.|-- 129.250.6.11 45.0% 40 22 17.8 14.6 9.6 47.1 6.|-- 129.250.5.53 0.0% 40 40 11.6 10.8 10.0 12.2 7.|-- 128.241.219.234 0.0% 40 40 162.7 163.1 161.7 178.3 8.|-- 173.192.18.151 2.5% 40 39 172.3 172.3 171.2 176.3 9.|-- 173.192.18.166 12.5% 40 35 157.2 156.3 155.6 157.9 10.|-- 173.192.18.140 7.5% 40 37 198.5 201.1 198.4 234.7 11.|-- 173.192.18.225 0.0% 40 40 183.0 190.1 182.9 355.2 12.|-- 70.87.255.66 0.0% 40 40 183.0 183.8 183.0 185.9 13.|-- 70.87.254.74 0.0% 40 40 193.6 195.8 192.0 284.1 14.|-- 67.18.7.90 2.5% 40 39 197.0 197.4 196.9 198.8 15.|-- 67.18.187.25 2.5% 40 39 192.8 192.8 192.2 194.0 === END FYI: 206.223.143.131 resolves to te2-6.bbr01.cs01.lax01.networklayer.com.any2ix.coresite.com. ARIN states 206.223.143.0/24 is CoreSite. The packet path for the above is: LAX (ARP Networks) -> SFO (NTT/Verio) -> San Jose (NTT/Verio) -> LAX (NetworkLayer) -> Dallas (NetworkLayer and Linode) -> 67.18.187.25 And now for the amusing part -- the return path: src IP: 67.18.187.25 (Texas, AS 21844 (I think)) dst IP: 208.79.90.130 (Southern CA, AS 25795 (I think)) traceroute to omake.koitsu.org (208.79.90.130), 30 hops max, 60 byte packets 1 router2-dal.linode.com (67.18.7.162) 0.604 ms 0.812 ms 0.815 ms 2 xe-2-0-0.car04.dllstx2.networklayer.com (67.18.7.93) 0.415 ms 0.428 ms 0.426 ms 3 po102.dsr01.dllstx2.networklayer.com (70.87.254.81) 0.756 ms 0.764 ms 0.963 ms 4 po21.dsr01.dllstx3.networklayer.com (70.87.255.65) 0.702 ms 0.759 ms 1.103 ms 5 ae16.bbr02.eq01.dal03.networklayer.com (173.192.18.228) 0.438 ms 0.446 ms 0.441 ms 6 ae7.bbr01.eq01.dal03.networklayer.com (173.192.18.208) 1.271 ms 0.709 ms 0.612 ms 7 ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 28.374 ms 28.431 ms 28.411 ms 8 ae7.bbr02.cs01.lax01.networklayer.com (173.192.18.167) 28.773 ms 28.747 ms 28.727 ms 9 * * * 10 ae0.bbr01.eq01.tok01.networklayer.com (50.97.18.161) 137.992 ms 138.001 ms 137.967 ms 11 ae7.bbr02.eq01.tok01.networklayer.com (50.97.18.163) 138.084 ms 138.068 ms 138.064 ms 12 ae0.bbr01.pn01.hkg01.networklayer.com (50.97.18.167) 181.952 ms 182.026 ms 181.990 ms 13 hutchcity21-10G.hkix.net (202.40.160.193) 180.121 ms 179.677 ms 179.631 ms 14 218.189.5.51 (218.189.5.51) 179.553 ms 179.521 ms 179.505 ms 15 d1-34-224-143-118-on-nets.com (118.143.224.34) 186.883 ms 186.829 ms 186.807 ms 16 * * * 17 omake.koitsu.org (208.79.90.130) 192.086 ms 192.973 ms 192.095 ms The packet path for the above is: Dallas (Linode then NetworkLayer) -> LAX (NetworkLayer) -> Tokyo (NetworkLayer) -> Hong Kong (NetworkLayer then via HKIX) -> Hong Kong (Hutchison Global Communications) (AS 9304) -> ? -> 208.79.90.130 Reminder: 208.79.90.130 is not physically in Hong Kong. I haven't spent the time to look at bgplay.routeviews.org yet, but I'm not sure it'll necessarily have insights into any of this. I have reached out to ARP Networks to have them check things, but this is an interesting situation I thought I'd mention as folks more familiar with BGP than myself could probably assist with. I can only speculate at this point, but I wonder if AS 9304 may have announced some kind of preferencing that certain things have picked up (but never withdrew/changed?). There are lots of explanations though, so that's purely hearsay on my part. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |