Rob, Thanks for getting back to us. You are indeed correct, the problem does appear to be inside the Corporate Colocation network. Their NOC has been insisting this was upstream and the routing issues we're seeing as a result incorrectly implicated peers along the way. I apologize for wasting your time, and again thank you for troubleshooting and responses. -Erik On 04/26/2012 07:40 PM, Rob Mosher wrote:

You appear to be a customer of Corporate Colocation. Their equipment is the next hop in the traceroute, and the apparent point of failure. Perhaps contacting your provider before erroneously posting to an outages list may be prudent, as the point of failure does not seem to lie within HE, Abovenet, or GT-T, all of which have been named in your email.

Also, GT-T should have sourced their trace from a non Equinix IX IP address, and it would have gotten further. This obviously did not help their diagnostics.

Here is an MTR to 205.134.255.113, from an internal HE IP address. Please contact your provider, Corporate Colocation, for further troubleshooting.

Host Loss% Snt Last Avg Best Wrst StDev 1. gige-g3-8.core1.fmt1.he.net 0.0% 100 0.5 3.5 0.4 12.5 3.9 2. 10gigabitethernet1-1.core1.pao1.he.net 0.0% 99 0.8 3.5 0.7 11.5 3.6 3. 10gigabitethernet1-1.core1.lax1.he.net 0.0% 99 9.1 11.3 9.0 21.6 3.3 4. corporate-colocation-inc.gigabitethernet3- 70.4% 99 9.3 21.5 9.1 256.3 49.2 5. ???

-- Rob Mosher Senior Network and Software Engineer Hurricane Electric / AS6939

On 4/26/2012 7:31 PM, Erik S. wrote:

...

Hi Rob,

Thanks for the quick response. Sorry for the confusion, the information I was provided at first was a little cloudy. We also use GT-T/Packet Exchange and they're pointing this at above.net.

Here's the response we received from PKX/GT-T regarding the routing anomaly we're experiencing:

-------- Original Message -------- Subject: GTT TT#(49181), SVC ID: PEX/NATIVE LAYER 3/9014-1 RE: Routing issue preventing our network from reaching * Date: Thu, 26 Apr 2012 21:52:27 +0000 From: GTT NOC <NOC@gt-t.net>

From LAX core device the routes goes to abovenet for both the ip address and dies within the above net network. We will investigate with abovenet and update you shortly.

cr1.lax1#traceroute 108.171.183.75

Type Ctrl-C to abort.

------------------------------------------------------------------- Tracing the route to 108.171.183.75, 30 hops max, 40 byte packets ------------------------------------------------------------------- TTL Hostname Probe1 Probe2 Probe3 1 mpr1.lax2.us.above.net (206.223.123.71) 012.000 ms 001.000 ms 001.000 ms 2 * * * 3 * * * 4 * * * 5 *

-- End forwarded message --

If they have not yet opened a ticket with you, this leads me to believe there's a problem elsewhere and we're being given the run-around.

The particular trace above is to one of our off-site boxes at rackspace.

This is an issue that seems to be impacting multiple networks and regions, so I'm not entirely convinced there's not something more going on.

For example, besides the IP referenced above, our 205.134.0.0 subnets are also failing. I pulled this trace from Above.net's looking glass:

traceroute to 205.134.255.113 (205.134.255.113), 30 hops max, 40 byte packets 1 xe-0-1-0.mpr1.lax12.us.above.net (64.125.31.189) 22.612 ms 0.421 ms 0.368 ms 2 10gigabitethernet1-3.core1.lax1.he.net (206.223.123.37) 0.944 ms 12.178 ms 0.956 ms 3 * * * 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * corporate-colocation-inc.gigabitethernet3-16.core1.lax1.he.net (72.52.77.38) 1.135 ms !H * 28 * * * 29 * * *

Tracing out from 205.134.255.113 anywhere yields this:

root@wczverify1 [~]# traceroute above.net traceroute to above.net (64.125.228.35), 30 hops max, 40 byte packets 1 74.124.198.1 (74.124.198.1) 0.461 ms 0.465 ms 0.530 ms 2 gige-g2-19.core1.lax2.he.net (64.62.194.161) 0.363 ms 0.433 ms 0.509 ms 3 * * * 4 * * * 5 * * * 6 * * *

If you could shed any additional light on this issue for us I'd be most appreciative.

Thanks.

Best Regards,

Erik Soroka Tier III System Administrator InMotion Hosting, Inc. eriks@inmotionhosting.com 888-321-4678 (ext 834) 757-416-6575 (int'l)

On 04/26/2012 07:20 PM, Rob Mosher wrote:

...

Hi Erik,

Where did you get the information that we were blaming above.net for anything? We are not aware of any issue between above.net and he.net, nor have we seen any tickets opened regarding this. Can you please include the source and destination IP addresses of the traceroute quoted below, as well as a traceroute in the opposite direction if possible?

As far as I can tell, everything looks clean here.

root@tserv15:~# tcptraceroute lg.above.net traceroute to lg.above.net (64.124.253.67), 30 hops max, 60 byte packets 1 gige-g4-6.core1.lax1.he.net (66.220.18.41) 7.683 ms 7.725 ms 7.821 ms 2 mpr1.lax2.us.above.net (206.223.123.71) 0.781 ms 0.766 ms 0.767 ms 3 xe-3-0-0.cr1.lax112.us.above.net (64.125.30.17) 1.055 ms 1.057 ms 1.293 ms 4 xe-2-1-0.cr1.sjc2.us.above.net (64.125.24.18) 10.125 ms 10.129 ms 10.120 ms 5 xe-1-1-0.er1.sjc2.us.above.net (64.125.26.198) 9.815 ms 9.800 ms 9.800 ms 6 f1-1.itr3.sjc2.us.corp.above.net (64.124.248.122) 10.239 ms 10.002 ms 9.995 ms 7 lg.above.net (64.124.253.67) 10.227 ms 10.338 ms 10.267 ms

-- Rob Mosher Senior Network and Software Engineer Hurricane Electric / AS6939

...

---------- Forwarded message ---------- From: "Erik S." <eriks@inmotionhosting.com <mailto:eriks@inmotionhosting.com>> Date: Apr 26, 2012 6:06 PM Subject: [outages] above.net <http://above.net> / he.net <http://he.net> routing problems in LAX? To: "outages@outages.org <mailto:outages@outages.org>" <outages@outages.org <mailto:outages@outages.org>>

We're seeing major routing issues in Los Angeles with peering from HE.net <-> Above.net, specifically after mpr1.lax2.us.above.net <http://mpr1.lax2.us.above.net>.

Above.net's looking glass in LA shows traces dying right after he.net <http://he.net>,

1 xe-0-1-0.cr2.lax112.us.above.net <http://xe-0-1-0.cr2.lax112.us.above.net> (64.125.30.250 <tel:%2864.125.30.250>) 0.780 ms 0.571 ms 0.479 ms 2 xe-8-0-0.mpr1.lax12.us.above.net <http://xe-8-0-0.mpr1.lax12.us.above.net> (64.125.30.30) 0.393 ms 0.483 ms 0.404 ms 3 10gigabitethernet1-3.core1.lax1.he.net <http://10gigabitethernet1-3.core1.lax1.he.net> (206.223.123.37) 0.884 ms 4.619 ms 1.100 ms 4 * * *

Both sides seem to be blaming the other. Is anyone routing via above.net <http://above.net> or he.net <http://he.net> seeing similar issues out west?

Best Regards,

Erik Soroka Tier III System Administrator InMotion Hosting, Inc. eriks@inmotionhosting.com <mailto:eriks@inmotionhosting.com>

_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages