LAX routing anomaly: NetworkLayer? CoreSite? Hong Kong ISPs?
I just came across this on NANOG while troubleshooting something: http://mailman.nanog.org/pipermail/nanog/2013-September/061332.html If the issue described there is what I'm seeing, then it looks like some part of the 'net in the LA area is very very broken -- and worse, *has* been broken since roughly Sep 27 21:16:00 PDT (UTC-7). What I saw (and am still seeing): src IP: 208.79.90.130 (Southern CA, AS 25795 (I think)) dst IP: 67.18.187.25 (Texas, AS 21844 (I think)) === Fri Sep 27 21:14:00 PDT 2013 (1380341640) Start: Fri Sep 27 21:14:00 2013 HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst 1.|-- 208.79.90.129 0.0% 40 40 3.7 11.0 0.9 202.9 2.|-- 206.223.143.131 0.0% 40 40 0.6 0.8 0.5 8.5 3.|-- 173.192.18.140 2.5% 40 39 28.4 30.1 28.2 86.8 4.|-- 173.192.18.225 0.0% 40 40 29.0 29.2 28.7 40.9 5.|-- 70.87.255.66 0.0% 40 40 31.6 31.5 31.3 31.9 6.|-- 70.87.254.74 0.0% 40 40 29.0 31.5 28.6 60.3 7.|-- 67.18.7.90 0.0% 40 40 28.7 28.6 28.4 29.0 8.|-- 67.18.187.25 0.0% 40 40 31.9 31.5 31.3 31.9 === END === Fri Sep 27 21:15:00 PDT 2013 (1380341700) Start: Fri Sep 27 21:15:00 2013 HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst 1.|-- 208.79.90.129 0.0% 40 40 2.2 51.1 0.9 245.7 2.|-- 206.223.143.131 2.5% 40 39 0.9 12.3 0.5 183.2 3.|-- 173.192.18.140 5.0% 40 38 0.8 29.4 0.7 284.1 4.|-- 173.192.18.225 0.0% 40 40 0.8 17.1 0.5 177.8 5.|-- 70.87.255.66 30.0% 40 28 11.2 39.1 3.4 199.4 6.|-- 70.87.254.74 0.0% 40 40 11.0 21.1 0.5 121.5 7.|-- 67.18.7.90 0.0% 40 40 163.8 122.6 4.6 347.0 8.|-- 67.18.187.25 22.5% 40 31 171.6 81.4 0.5 173.8 === END === Fri Sep 27 21:16:00 PDT 2013 (1380341760) Start: Fri Sep 27 21:16:00 2013 HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst 1.|-- 208.79.90.129 0.0% 40 40 2.9 7.4 0.9 95.8 2.|-- 208.79.88.135 0.0% 40 40 0.6 15.8 0.4 182.3 3.|-- 129.250.198.185 0.0% 40 40 0.7 0.8 0.7 1.2 4.|-- 129.250.5.69 0.0% 40 40 0.7 5.1 0.6 34.1 5.|-- 129.250.6.11 45.0% 40 22 17.8 14.6 9.6 47.1 6.|-- 129.250.5.53 0.0% 40 40 11.6 10.8 10.0 12.2 7.|-- 128.241.219.234 0.0% 40 40 162.7 163.1 161.7 178.3 8.|-- 173.192.18.151 2.5% 40 39 172.3 172.3 171.2 176.3 9.|-- 173.192.18.166 12.5% 40 35 157.2 156.3 155.6 157.9 10.|-- 173.192.18.140 7.5% 40 37 198.5 201.1 198.4 234.7 11.|-- 173.192.18.225 0.0% 40 40 183.0 190.1 182.9 355.2 12.|-- 70.87.255.66 0.0% 40 40 183.0 183.8 183.0 185.9 13.|-- 70.87.254.74 0.0% 40 40 193.6 195.8 192.0 284.1 14.|-- 67.18.7.90 2.5% 40 39 197.0 197.4 196.9 198.8 15.|-- 67.18.187.25 2.5% 40 39 192.8 192.8 192.2 194.0 === END FYI: 206.223.143.131 resolves to te2-6.bbr01.cs01.lax01.networklayer.com.any2ix.coresite.com. ARIN states 206.223.143.0/24 is CoreSite. The packet path for the above is: LAX (ARP Networks) -> SFO (NTT/Verio) -> San Jose (NTT/Verio) -> LAX (NetworkLayer) -> Dallas (NetworkLayer and Linode) -> 67.18.187.25 And now for the amusing part -- the return path: src IP: 67.18.187.25 (Texas, AS 21844 (I think)) dst IP: 208.79.90.130 (Southern CA, AS 25795 (I think)) traceroute to omake.koitsu.org (208.79.90.130), 30 hops max, 60 byte packets 1 router2-dal.linode.com (67.18.7.162) 0.604 ms 0.812 ms 0.815 ms 2 xe-2-0-0.car04.dllstx2.networklayer.com (67.18.7.93) 0.415 ms 0.428 ms 0.426 ms 3 po102.dsr01.dllstx2.networklayer.com (70.87.254.81) 0.756 ms 0.764 ms 0.963 ms 4 po21.dsr01.dllstx3.networklayer.com (70.87.255.65) 0.702 ms 0.759 ms 1.103 ms 5 ae16.bbr02.eq01.dal03.networklayer.com (173.192.18.228) 0.438 ms 0.446 ms 0.441 ms 6 ae7.bbr01.eq01.dal03.networklayer.com (173.192.18.208) 1.271 ms 0.709 ms 0.612 ms 7 ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 28.374 ms 28.431 ms 28.411 ms 8 ae7.bbr02.cs01.lax01.networklayer.com (173.192.18.167) 28.773 ms 28.747 ms 28.727 ms 9 * * * 10 ae0.bbr01.eq01.tok01.networklayer.com (50.97.18.161) 137.992 ms 138.001 ms 137.967 ms 11 ae7.bbr02.eq01.tok01.networklayer.com (50.97.18.163) 138.084 ms 138.068 ms 138.064 ms 12 ae0.bbr01.pn01.hkg01.networklayer.com (50.97.18.167) 181.952 ms 182.026 ms 181.990 ms 13 hutchcity21-10G.hkix.net (202.40.160.193) 180.121 ms 179.677 ms 179.631 ms 14 218.189.5.51 (218.189.5.51) 179.553 ms 179.521 ms 179.505 ms 15 d1-34-224-143-118-on-nets.com (118.143.224.34) 186.883 ms 186.829 ms 186.807 ms 16 * * * 17 omake.koitsu.org (208.79.90.130) 192.086 ms 192.973 ms 192.095 ms The packet path for the above is: Dallas (Linode then NetworkLayer) -> LAX (NetworkLayer) -> Tokyo (NetworkLayer) -> Hong Kong (NetworkLayer then via HKIX) -> Hong Kong (Hutchison Global Communications) (AS 9304) -> ? -> 208.79.90.130 Reminder: 208.79.90.130 is not physically in Hong Kong. I haven't spent the time to look at bgplay.routeviews.org yet, but I'm not sure it'll necessarily have insights into any of this. I have reached out to ARP Networks to have them check things, but this is an interesting situation I thought I'd mention as folks more familiar with BGP than myself could probably assist with. I can only speculate at this point, but I wonder if AS 9304 may have announced some kind of preferencing that certain things have picked up (but never withdrew/changed?). There are lots of explanations though, so that's purely hearsay on my part. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
On 9/29/2013 22:31, Jeremy Chadwick wrote:
I just came across this on NANOG while troubleshooting something:
http://mailman.nanog.org/pipermail/nanog/2013-September/061332.html
If the issue described there is what I'm seeing, then it looks like some part of the 'net in the LA area is very very broken -- and worse, *has* been broken since roughly Sep 27 21:16:00 PDT (UTC-7).
Not sure if relevant. https://twitter.com/arpnetworks/status/384577083587833856 "CoreSite will be performing emergency maintenance on their Any2Easy route servers during the hours; routing oddities may occur" -- staticsafe O< ascii ribbon campaign - stop html mail - www.asciiribbon.org Please don't top post. It is not logical. Please don't CC me! I'm subscribed to whatever list I just posted on.
Follow-up: I got lots of off-list responses about this, all of which were very informative. Thanks everyone. Wanted to report that after CoreSite did some maintenance this morning, things are back in order: src IP: 208.79.90.130 dst IP: 67.18.187.25 === Mon Sep 30 02:56:00 PDT 2013 (1380534960) Start: Mon Sep 30 02:56:00 2013 HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst 1.|-- 208.79.90.129 0.0% 40 40 0.9 5.6 0.8 128.4 2.|-- 208.79.88.135 0.0% 40 40 7.9 1.0 0.5 9.7 3.|-- 129.250.198.185 0.0% 40 40 0.7 0.8 0.7 1.2 4.|-- 129.250.5.69 0.0% 40 40 0.5 3.9 0.5 28.7 5.|-- 129.250.6.11 0.0% 40 40 10.3 13.1 9.6 35.4 6.|-- 129.250.5.53 0.0% 40 40 11.1 10.7 10.1 11.6 7.|-- 128.241.219.234 0.0% 40 40 162.8 165.4 161.7 206.4 8.|-- 173.192.18.151 0.0% 40 40 172.3 172.0 171.3 174.8 9.|-- 173.192.18.166 10.0% 40 36 156.7 156.6 155.6 160.6 10.|-- 173.192.18.140 10.0% 40 36 199.4 201.2 198.4 246.2 11.|-- 173.192.18.225 0.0% 40 40 184.1 185.5 182.9 222.8 12.|-- 70.87.255.66 0.0% 40 40 192.2 192.5 191.7 203.6 13.|-- 70.87.254.74 0.0% 40 40 193.0 194.8 192.0 229.3 14.|-- 67.18.7.90 2.5% 40 39 197.8 197.5 196.8 198.6 15.|-- 67.18.187.25 2.5% 40 39 200.5 201.4 200.5 204.1 === END === Mon Sep 30 02:57:00 PDT 2013 (1380535020) Start: Mon Sep 30 02:57:00 2013 HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst 1.|-- 208.79.90.129 0.0% 40 40 25.9 42.7 0.9 412.7 2.|-- 208.79.88.135 0.0% 40 40 0.5 77.9 0.4 163.2 3.|-- 129.250.198.185 5.0% 40 38 63.0 98.5 0.7 215.0 4.|-- 129.250.5.69 0.0% 40 40 29.1 95.9 0.6 261.0 5.|-- 129.250.6.11 0.0% 40 40 31.4 100.9 9.6 184.4 6.|-- 129.250.5.53 0.0% 40 40 28.5 100.0 10.1 224.1 7.|-- 128.241.219.234 0.0% 40 40 28.5 170.4 28.5 192.8 8.|-- 173.192.18.151 0.0% 40 40 31.5 175.6 31.4 198.7 9.|-- 173.192.18.166 2.5% 40 39 31.4 170.4 31.4 218.0 10.|-- 173.192.18.140 2.5% 40 39 31.4 188.5 31.4 244.7 11.|-- 173.192.18.225 0.0% 40 40 31.4 180.8 31.4 193.2 12.|-- 70.87.255.66 0.0% 40 40 31.4 184.5 31.4 193.1 13.|-- 70.87.254.74 0.0% 40 40 31.5 184.9 31.4 198.7 14.|-- 67.18.7.90 0.0% 40 40 31.4 187.0 31.4 199.0 15.|-- 67.18.187.25 0.0% 40 40 31.4 188.6 31.4 202.3 === END === Mon Sep 30 02:58:00 PDT 2013 (1380535080) Start: Mon Sep 30 02:58:00 2013 HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst 1.|-- 208.79.90.129 0.0% 40 40 9.3 16.4 0.9 177.8 2.|-- 206.223.143.131 0.0% 40 40 0.5 0.9 0.5 10.9 3.|-- 173.192.18.140 12.5% 40 35 28.1 28.6 28.1 34.2 4.|-- 173.192.18.225 0.0% 40 40 29.0 29.0 28.7 29.8 5.|-- 70.87.255.66 0.0% 40 40 31.5 31.5 31.3 32.2 6.|-- 70.87.254.74 0.0% 40 40 28.6 34.6 28.5 110.6 7.|-- 67.18.7.90 0.0% 40 40 28.6 28.6 28.4 29.0 8.|-- 67.18.187.25 0.0% 40 40 31.4 31.5 31.4 32.2 === END And the return path: src IP: 67.18.187.25 dst IP: 208.79.90.130 traceroute to omake.koitsu.org (208.79.90.130), 30 hops max, 60 byte packets 1 router2-dal.linode.com (67.18.7.162) 0.455 ms 0.601 ms 0.788 ms 2 xe-2-0-0.car04.dllstx2.networklayer.com (67.18.7.93) 0.293 ms 0.313 ms 0.317 ms 3 po102.dsr01.dllstx2.networklayer.com (70.87.254.81) 0.562 ms 0.686 ms 0.760 ms 4 po21.dsr01.dllstx3.networklayer.com (70.87.255.65) 0.654 ms 0.731 ms 0.873 ms 5 ae16.bbr02.eq01.dal03.networklayer.com (173.192.18.228) 0.334 ms 0.334 ms 0.347 ms 6 ae7.bbr01.eq01.dal03.networklayer.com (173.192.18.208) 0.486 ms 0.578 ms 0.481 ms 7 ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 33.662 ms 33.719 ms 33.698 ms 8 any2-ix.la.arpnetworks.com (206.223.143.166) 34.656 ms 34.998 ms 35.247 ms 9 omake.koitsu.org (208.79.90.130) 31.298 ms 31.299 ms 31.285 ms -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | On Sun, Sep 29, 2013 at 07:31:21PM -0700, Jeremy Chadwick wrote:
I just came across this on NANOG while troubleshooting something:
http://mailman.nanog.org/pipermail/nanog/2013-September/061332.html
If the issue described there is what I'm seeing, then it looks like some part of the 'net in the LA area is very very broken -- and worse, *has* been broken since roughly Sep 27 21:16:00 PDT (UTC-7).
What I saw (and am still seeing):
src IP: 208.79.90.130 (Southern CA, AS 25795 (I think)) dst IP: 67.18.187.25 (Texas, AS 21844 (I think))
=== Fri Sep 27 21:14:00 PDT 2013 (1380341640) Start: Fri Sep 27 21:14:00 2013 HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst 1.|-- 208.79.90.129 0.0% 40 40 3.7 11.0 0.9 202.9 2.|-- 206.223.143.131 0.0% 40 40 0.6 0.8 0.5 8.5 3.|-- 173.192.18.140 2.5% 40 39 28.4 30.1 28.2 86.8 4.|-- 173.192.18.225 0.0% 40 40 29.0 29.2 28.7 40.9 5.|-- 70.87.255.66 0.0% 40 40 31.6 31.5 31.3 31.9 6.|-- 70.87.254.74 0.0% 40 40 29.0 31.5 28.6 60.3 7.|-- 67.18.7.90 0.0% 40 40 28.7 28.6 28.4 29.0 8.|-- 67.18.187.25 0.0% 40 40 31.9 31.5 31.3 31.9 === END
=== Fri Sep 27 21:15:00 PDT 2013 (1380341700) Start: Fri Sep 27 21:15:00 2013 HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst 1.|-- 208.79.90.129 0.0% 40 40 2.2 51.1 0.9 245.7 2.|-- 206.223.143.131 2.5% 40 39 0.9 12.3 0.5 183.2 3.|-- 173.192.18.140 5.0% 40 38 0.8 29.4 0.7 284.1 4.|-- 173.192.18.225 0.0% 40 40 0.8 17.1 0.5 177.8 5.|-- 70.87.255.66 30.0% 40 28 11.2 39.1 3.4 199.4 6.|-- 70.87.254.74 0.0% 40 40 11.0 21.1 0.5 121.5 7.|-- 67.18.7.90 0.0% 40 40 163.8 122.6 4.6 347.0 8.|-- 67.18.187.25 22.5% 40 31 171.6 81.4 0.5 173.8 === END
=== Fri Sep 27 21:16:00 PDT 2013 (1380341760) Start: Fri Sep 27 21:16:00 2013 HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst 1.|-- 208.79.90.129 0.0% 40 40 2.9 7.4 0.9 95.8 2.|-- 208.79.88.135 0.0% 40 40 0.6 15.8 0.4 182.3 3.|-- 129.250.198.185 0.0% 40 40 0.7 0.8 0.7 1.2 4.|-- 129.250.5.69 0.0% 40 40 0.7 5.1 0.6 34.1 5.|-- 129.250.6.11 45.0% 40 22 17.8 14.6 9.6 47.1 6.|-- 129.250.5.53 0.0% 40 40 11.6 10.8 10.0 12.2 7.|-- 128.241.219.234 0.0% 40 40 162.7 163.1 161.7 178.3 8.|-- 173.192.18.151 2.5% 40 39 172.3 172.3 171.2 176.3 9.|-- 173.192.18.166 12.5% 40 35 157.2 156.3 155.6 157.9 10.|-- 173.192.18.140 7.5% 40 37 198.5 201.1 198.4 234.7 11.|-- 173.192.18.225 0.0% 40 40 183.0 190.1 182.9 355.2 12.|-- 70.87.255.66 0.0% 40 40 183.0 183.8 183.0 185.9 13.|-- 70.87.254.74 0.0% 40 40 193.6 195.8 192.0 284.1 14.|-- 67.18.7.90 2.5% 40 39 197.0 197.4 196.9 198.8 15.|-- 67.18.187.25 2.5% 40 39 192.8 192.8 192.2 194.0 === END
FYI: 206.223.143.131 resolves to te2-6.bbr01.cs01.lax01.networklayer.com.any2ix.coresite.com.
ARIN states 206.223.143.0/24 is CoreSite.
The packet path for the above is:
LAX (ARP Networks) -> SFO (NTT/Verio) -> San Jose (NTT/Verio) -> LAX (NetworkLayer) -> Dallas (NetworkLayer and Linode) -> 67.18.187.25
And now for the amusing part -- the return path:
src IP: 67.18.187.25 (Texas, AS 21844 (I think)) dst IP: 208.79.90.130 (Southern CA, AS 25795 (I think))
traceroute to omake.koitsu.org (208.79.90.130), 30 hops max, 60 byte packets 1 router2-dal.linode.com (67.18.7.162) 0.604 ms 0.812 ms 0.815 ms 2 xe-2-0-0.car04.dllstx2.networklayer.com (67.18.7.93) 0.415 ms 0.428 ms 0.426 ms 3 po102.dsr01.dllstx2.networklayer.com (70.87.254.81) 0.756 ms 0.764 ms 0.963 ms 4 po21.dsr01.dllstx3.networklayer.com (70.87.255.65) 0.702 ms 0.759 ms 1.103 ms 5 ae16.bbr02.eq01.dal03.networklayer.com (173.192.18.228) 0.438 ms 0.446 ms 0.441 ms 6 ae7.bbr01.eq01.dal03.networklayer.com (173.192.18.208) 1.271 ms 0.709 ms 0.612 ms 7 ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 28.374 ms 28.431 ms 28.411 ms 8 ae7.bbr02.cs01.lax01.networklayer.com (173.192.18.167) 28.773 ms 28.747 ms 28.727 ms 9 * * * 10 ae0.bbr01.eq01.tok01.networklayer.com (50.97.18.161) 137.992 ms 138.001 ms 137.967 ms 11 ae7.bbr02.eq01.tok01.networklayer.com (50.97.18.163) 138.084 ms 138.068 ms 138.064 ms 12 ae0.bbr01.pn01.hkg01.networklayer.com (50.97.18.167) 181.952 ms 182.026 ms 181.990 ms 13 hutchcity21-10G.hkix.net (202.40.160.193) 180.121 ms 179.677 ms 179.631 ms 14 218.189.5.51 (218.189.5.51) 179.553 ms 179.521 ms 179.505 ms 15 d1-34-224-143-118-on-nets.com (118.143.224.34) 186.883 ms 186.829 ms 186.807 ms 16 * * * 17 omake.koitsu.org (208.79.90.130) 192.086 ms 192.973 ms 192.095 ms
The packet path for the above is:
Dallas (Linode then NetworkLayer) -> LAX (NetworkLayer) -> Tokyo (NetworkLayer) -> Hong Kong (NetworkLayer then via HKIX) -> Hong Kong (Hutchison Global Communications) (AS 9304) -> ? -> 208.79.90.130
Reminder: 208.79.90.130 is not physically in Hong Kong.
I haven't spent the time to look at bgplay.routeviews.org yet, but I'm not sure it'll necessarily have insights into any of this.
I have reached out to ARP Networks to have them check things, but this is an interesting situation I thought I'd mention as folks more familiar with BGP than myself could probably assist with.
I can only speculate at this point, but I wonder if AS 9304 may have announced some kind of preferencing that certain things have picked up (but never withdrew/changed?). There are lots of explanations though, so that's purely hearsay on my part.
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (2)
-
Jeremy Chadwick -
staticsafe