Seeing massive packet loss and high latency from 360.net (which appears to be owned by zayo/abovenet/whatever) in denver.... Here is a trace snip from our server @ softlayer in dallas to one of our servers in denver 5: ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 29.698ms 6: no reply 7: 66.62.2.213 (66.62.2.213) 4588.320ms 8: den1-core-01-xe-1-1-0.360.net (66.62.2.166) 22583.831ms I only have one one server in denver that I can reproduce this with hoping someone can else can test and reproduce similar issues thanks chris
Chris, You need to provide traceroutes from both directions. The below is for Dallas --> Denver, but the return path (Denver --> Dallas) needs to be provided too. You might find that the return path goes through some provider other than Zayo/360/Abovenet/whateverthey'recalledtoday. I know that's hard to do when the path has latency or packet loss (which you don't show in your results -- you only show latency), but this is exactly what a cronjob traceroute writing to a log file is for. :-) My point: remember that routing on the Internet most of the time is asymmetric. Reference material (read, do not skim): http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N4... Finally, you didn't provide IP addresses of either server (in Denver or Dallas), so when you ask for "someone else to test", that's not easily doable, at least not to the endpoints involved (pinging routers is not sufficient evidence, sadly). -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Thu, Dec 13, 2012 at 11:10:32PM -0500, chris wrote:
Seeing massive packet loss and high latency from 360.net (which appears to be owned by zayo/abovenet/whatever) in denver....
Here is a trace snip from our server @ softlayer in dallas to one of our servers in denver
5: ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 29.698ms 6: no reply 7: 66.62.2.213 (66.62.2.213) 4588.320ms 8: den1-core-01-xe-1-1-0.360.net (66.62.2.166) 22583.831ms
I only have one one server in denver that I can reproduce this with hoping someone can else can test and reproduce similar issues
thanks chris
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Ok sorry, the denver side for us is 208.89.160.11 and the dallas softlayer side for us is 70.87.254.1 You are right about different paths it appears that the route back is direct via level3 instead of 360/zayo and same latency is present so looks like more likely a softlayer issue? trace from our dallas box(softlayer) to our denver box(greenhouse): 2: po101.dsr01.dllstx5.networklayer.com (70.87.254.1) 1.941ms 3: po51.dsr01.dllstx3.networklayer.com (70.85.127.105) 2.217ms 4: ae16.bbr01.eq01.dal03.networklayer.com (173.192.18.224) 1.719ms 5: ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 32.448ms 6: 360.net.any2ix.coresite.com (206.223.143.201) 657.255ms 7: lax1-core-01-xe-0-0-0.360.net (66.62.2.213) 36.196ms 8: den1-core-01-ae0.360.net (66.62.2.169) 246.045ms 9: den1-edge-01-lag2.360.net (66.62.2.194) asymm 8 271.648ms 10: 66.62.160.30 (66.62.160.30) asymm 9 543.816ms 11: CYSWYDC01ESW1-001-1-1.GREENHOUSEDATA.NET (208.89.160.11) asymm 10 793.839ms reached from denver box to dallas (mtr is all thats installed and cant install anything else connectivity is too horrible) Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 2. CYSWYDC01RTR1-001-0-1.GREENHOUSEDATA.NET 0.0% 17 0.3 0.3 0.3 0.4 0.0 3. ge-6-13.car2.Denver1.Level3.net 5.9% 17 4.6 121.9 4.6 417.4 116.1 4. ae-21-52.car1.Denver1.Level3.net 0.0% 17 101.9 25.2 2.9 140.6 40.6 5. te1-5.bbr01.cf01.den01.networklayer.com 43.8% 17 565.9 479.9 169.5 919.2 214.1 6. ae7.bbr01.cs01.den01.networklayer.com 37.5% 17 642.3 665.4 170.0 928.8 261.4 7. ae12.bbr02.eq01.dal03.networklayer.com 50.0% 17 1168. 650.2 95.4 1168. 391.4 8. po32.dsr01.dllstx3.networklayer.com 31.2% 17 423.9 643.3 80.0 1135. 368.2 9. po101.dsr01.dllstx5.networklayer.com 50.0% 16 347.1 814.1 347.1 1874. 508.6 thanks for pointing it out chris On Fri, Dec 14, 2012 at 12:00 AM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Chris,
You need to provide traceroutes from both directions. The below is for Dallas --> Denver, but the return path (Denver --> Dallas) needs to be provided too. You might find that the return path goes through some provider other than Zayo/360/Abovenet/whateverthey'recalledtoday.
I know that's hard to do when the path has latency or packet loss (which you don't show in your results -- you only show latency), but this is exactly what a cronjob traceroute writing to a log file is for. :-)
My point: remember that routing on the Internet most of the time is asymmetric. Reference material (read, do not skim):
http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N4...
Finally, you didn't provide IP addresses of either server (in Denver or Dallas), so when you ask for "someone else to test", that's not easily doable, at least not to the endpoints involved (pinging routers is not sufficient evidence, sadly).
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Thu, Dec 13, 2012 at 11:10:32PM -0500, chris wrote:
Seeing massive packet loss and high latency from 360.net (which appears to be owned by zayo/abovenet/whatever) in denver....
Here is a trace snip from our server @ softlayer in dallas to one of our servers in denver
5: ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 29.698ms 6: no reply 7: 66.62.2.213 (66.62.2.213) 4588.320ms 8: den1-core-01-xe-1-1-0.360.net (66.62.2.166) 22583.831ms
I only have one one server in denver that I can reproduce this with hoping someone can else can test and reproduce similar issues
thanks chris
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
For Dallas->Denver, the latency begins between hop #7 (360.net in LAX/Los Angeles) and hop #8 (360.net in DEN/Denver). The latency at hop #6 is probably the result of ICMP prioritisation (indicated by lower latency at hop #7). For Denver->Dallas, latency and packet loss begins between hop #4 (Level3 in Denver) and hop #5 (SoftLayer in Denver). Thus, to solve this issue, you would need to have the following 3 providers all on the phone at the same time talking to one another simultaneously, and give all 3 of them the below traceroutes/mtrs with evidence (because they need to know what physical interfaces the packets are flowing across): - 360.net / Zayo / Abovenet - Level3 - SoftLayer The most you can do yourself is contact the providers who you have service with (i.e. SoftLayer and GreenhouseData.net(?)) and ask if there is anything they can do or contact their peering providers. Contact them and put pressure on them. Because the issue is so far "in" (i.e. not at hop #2 or #3 within traceroutes), there is very little you can do. Even if you had peering arrangements with, say, L3 and 360/Zayo/AboveNet, its not evident from the traceroutes and you'll usually be ignored (or more specifically you will be asked repeatedly "What's your customer ID?" and if you don't have one, basically told to sod off). Remember: the Internet is always broken, 24x7x365, in some regard. It sucks when your packets suffer from it, but there's very little you can do when you don't have contractual arrangements with the providers who are experiencing issues. P.S. -- From my home Comcast connection in Mountain View, I see no issues to either 208.89.160.11 nor 70.87.254.1 -- but my packets go through completely different interfaces and/or routers than whats shown in your traceroutes. The same goes from tests performed from my VPS in Los Angeles (who has peering with SoftLayer directly) -- again, packets going through completely different interfaces. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Fri, Dec 14, 2012 at 12:11:45AM -0500, chris wrote:
Ok sorry, the denver side for us is 208.89.160.11 and the dallas softlayer side for us is 70.87.254.1
You are right about different paths it appears that the route back is direct via level3 instead of 360/zayo and same latency is present so looks like more likely a softlayer issue?
trace from our dallas box(softlayer) to our denver box(greenhouse): 2: po101.dsr01.dllstx5.networklayer.com (70.87.254.1) 1.941ms 3: po51.dsr01.dllstx3.networklayer.com (70.85.127.105) 2.217ms 4: ae16.bbr01.eq01.dal03.networklayer.com (173.192.18.224) 1.719ms 5: ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 32.448ms 6: 360.net.any2ix.coresite.com (206.223.143.201) 657.255ms 7: lax1-core-01-xe-0-0-0.360.net (66.62.2.213) 36.196ms 8: den1-core-01-ae0.360.net (66.62.2.169) 246.045ms 9: den1-edge-01-lag2.360.net (66.62.2.194) asymm 8 271.648ms 10: 66.62.160.30 (66.62.160.30) asymm 9 543.816ms 11: CYSWYDC01ESW1-001-1-1.GREENHOUSEDATA.NET (208.89.160.11) asymm 10 793.839ms reached
from denver box to dallas (mtr is all thats installed and cant install anything else connectivity is too horrible)
Packets Pings Host
Loss% Snt Last Avg Best Wrst StDev
2. CYSWYDC01RTR1-001-0-1.GREENHOUSEDATA.NET > > 0.0% 17 0.3 0.3 0.3 0.4 0.0 3. ge-6-13.car2.Denver1.Level3.net > > 5.9% 17 4.6 121.9 4.6 417.4 116.1 4. ae-21-52.car1.Denver1.Level3.net > > 0.0% 17 101.9 25.2 2.9 140.6 40.6 5. te1-5.bbr01.cf01.den01.networklayer.com > > 43.8% 17 565.9 479.9 169.5 919.2 214.1 6. ae7.bbr01.cs01.den01.networklayer.com > > 37.5% 17 642.3 665.4 170.0 928.8 261.4 7. ae12.bbr02.eq01.dal03.networklayer.com > > 50.0% 17 1168. 650.2 95.4 1168. 391.4 8. po32.dsr01.dllstx3.networklayer.com > > 31.2% 17 423.9 643.3 80.0 1135. 368.2 9. po101.dsr01.dllstx5.networklayer.com > > 50.0% 16 347.1 814.1 347.1 1874. 508.6
thanks for pointing it out chris
On Fri, Dec 14, 2012 at 12:00 AM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Chris,
You need to provide traceroutes from both directions. The below is for Dallas --> Denver, but the return path (Denver --> Dallas) needs to be provided too. You might find that the return path goes through some provider other than Zayo/360/Abovenet/whateverthey'recalledtoday.
I know that's hard to do when the path has latency or packet loss (which you don't show in your results -- you only show latency), but this is exactly what a cronjob traceroute writing to a log file is for. :-)
My point: remember that routing on the Internet most of the time is asymmetric. Reference material (read, do not skim):
http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N4...
Finally, you didn't provide IP addresses of either server (in Denver or Dallas), so when you ask for "someone else to test", that's not easily doable, at least not to the endpoints involved (pinging routers is not sufficient evidence, sadly).
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Thu, Dec 13, 2012 at 11:10:32PM -0500, chris wrote:
Seeing massive packet loss and high latency from 360.net (which appears to be owned by zayo/abovenet/whatever) in denver....
Here is a trace snip from our server @ softlayer in dallas to one of our servers in denver
5: ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 29.698ms 6: no reply 7: 66.62.2.213 (66.62.2.213) 4588.320ms 8: den1-core-01-xe-1-1-0.360.net (66.62.2.166) 22583.831ms
I only have one one server in denver that I can reproduce this with hoping someone can else can test and reproduce similar issues
thanks chris
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Yeah I have a ticket open with softlayer as it seems they are the most common factor and the only one i have a direct relationship in this situation, no response so far in about an hour but hopefully before the workforce starts ringing phones off the hook :) On Fri, Dec 14, 2012 at 12:28 AM, Jeremy Chadwick <jdc@koitsu.org> wrote:
For Dallas->Denver, the latency begins between hop #7 (360.net in LAX/Los Angeles) and hop #8 (360.net in DEN/Denver). The latency at hop #6 is probably the result of ICMP prioritisation (indicated by lower latency at hop #7).
For Denver->Dallas, latency and packet loss begins between hop #4 (Level3 in Denver) and hop #5 (SoftLayer in Denver).
Thus, to solve this issue, you would need to have the following 3 providers all on the phone at the same time talking to one another simultaneously, and give all 3 of them the below traceroutes/mtrs with evidence (because they need to know what physical interfaces the packets are flowing across):
- 360.net / Zayo / Abovenet - Level3 - SoftLayer
The most you can do yourself is contact the providers who you have service with (i.e. SoftLayer and GreenhouseData.net(?)) and ask if there is anything they can do or contact their peering providers. Contact them and put pressure on them.
Because the issue is so far "in" (i.e. not at hop #2 or #3 within traceroutes), there is very little you can do. Even if you had peering arrangements with, say, L3 and 360/Zayo/AboveNet, its not evident from the traceroutes and you'll usually be ignored (or more specifically you will be asked repeatedly "What's your customer ID?" and if you don't have one, basically told to sod off).
Remember: the Internet is always broken, 24x7x365, in some regard. It sucks when your packets suffer from it, but there's very little you can do when you don't have contractual arrangements with the providers who are experiencing issues.
P.S. -- From my home Comcast connection in Mountain View, I see no issues to either 208.89.160.11 nor 70.87.254.1 -- but my packets go through completely different interfaces and/or routers than whats shown in your traceroutes. The same goes from tests performed from my VPS in Los Angeles (who has peering with SoftLayer directly) -- again, packets going through completely different interfaces.
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Fri, Dec 14, 2012 at 12:11:45AM -0500, chris wrote:
Ok sorry, the denver side for us is 208.89.160.11 and the dallas softlayer side for us is 70.87.254.1
You are right about different paths it appears that the route back is direct via level3 instead of 360/zayo and same latency is present so looks like more likely a softlayer issue?
trace from our dallas box(softlayer) to our denver box(greenhouse): 2: po101.dsr01.dllstx5.networklayer.com (70.87.254.1) 1.941ms 3: po51.dsr01.dllstx3.networklayer.com (70.85.127.105) 2.217ms 4: ae16.bbr01.eq01.dal03.networklayer.com (173.192.18.224) 1.719ms 5: ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 32.448ms 6: 360.net.any2ix.coresite.com (206.223.143.201) 657.255ms 7: lax1-core-01-xe-0-0-0.360.net (66.62.2.213) 36.196ms 8: den1-core-01-ae0.360.net (66.62.2.169) 246.045ms 9: den1-edge-01-lag2.360.net (66.62.2.194) asymm 8 271.648ms 10: 66.62.160.30 (66.62.160.30) asymm 9 543.816ms 11: CYSWYDC01ESW1-001-1-1.GREENHOUSEDATA.NET (208.89.160.11) asymm 10 793.839ms reached
from denver box to dallas (mtr is all thats installed and cant install anything else connectivity is too horrible)
Packets Pings Host
Loss% Snt Last Avg Best Wrst StDev
2. CYSWYDC01RTR1-001-0-1.GREENHOUSEDATA.NET > > 0.0% 17 0.3 0.3 0.3 0.4 0.0 3. ge-6-13.car2.Denver1.Level3.net > > 5.9% 17 4.6 121.9 4.6 417.4 116.1 4. ae-21-52.car1.Denver1.Level3.net > > 0.0% 17 101.9 25.2 2.9 140.6 40.6 5. te1-5.bbr01.cf01.den01.networklayer.com > > 43.8% 17 565.9 479.9 169.5 919.2 214.1 6. ae7.bbr01.cs01.den01.networklayer.com > > 37.5% 17 642.3 665.4 170.0 928.8 261.4 7. ae12.bbr02.eq01.dal03.networklayer.com > > 50.0% 17 1168. 650.2 95.4 1168. 391.4 8. po32.dsr01.dllstx3.networklayer.com > > 31.2% 17 423.9 643.3 80.0 1135. 368.2 9. po101.dsr01.dllstx5.networklayer.com > > 50.0% 16 347.1 814.1 347.1 1874. 508.6
thanks for pointing it out chris
On Fri, Dec 14, 2012 at 12:00 AM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Chris,
You need to provide traceroutes from both directions. The below is for Dallas --> Denver, but the return path (Denver --> Dallas) needs to be provided too. You might find that the return path goes through some provider other than Zayo/360/Abovenet/whateverthey'recalledtoday.
I know that's hard to do when the path has latency or packet loss (which you don't show in your results -- you only show latency), but this is exactly what a cronjob traceroute writing to a log file is for. :-)
My point: remember that routing on the Internet most of the time is asymmetric. Reference material (read, do not skim):
http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N4...
Finally, you didn't provide IP addresses of either server (in Denver or Dallas), so when you ask for "someone else to test", that's not easily doable, at least not to the endpoints involved (pinging routers is not sufficient evidence, sadly).
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Thu, Dec 13, 2012 at 11:10:32PM -0500, chris wrote:
Seeing massive packet loss and high latency from 360.net (which appears to be owned by zayo/abovenet/whatever) in denver....
Here is a trace snip from our server @ softlayer in dallas to one of our servers in denver
5: ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 29.698ms 6: no reply 7: 66.62.2.213 (66.62.2.213) 4588.320ms 8: den1-core-01-xe-1-1-0.360.net (66.62.2.166) 22583.831ms
I only have one one server in denver that I can reproduce this with hoping someone can else can test and reproduce similar issues
thanks chris
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Hi Chris, Not entirely what we saw tonight, but we had issues reaching some places via a very similar path, and we peer with 360 and level3. I've localprefed around it for the time being, but only just saw this email. Can't confirm our issue was what you were seeing, but we were definitely seeing packet loss, latency, and what appeared to be occasional loops on networklayer. Not particularly useful, I realise, but I guess count me in as a tentative "me too." TD On 12/13/2012 10:11 PM, chris wrote:
Ok sorry, the denver side for us is 208.89.160.11 and the dallas softlayer side for us is 70.87.254.1
You are right about different paths it appears that the route back is direct via level3 instead of 360/zayo and same latency is present so looks like more likely a softlayer issue?
trace from our dallas box(softlayer) to our denver box(greenhouse): 2: po101.dsr01.dllstx5.networklayer.com (70.87.254.1) 1.941ms 3: po51.dsr01.dllstx3.networklayer.com (70.85.127.105) 2.217ms 4: ae16.bbr01.eq01.dal03.networklayer.com (173.192.18.224) 1.719ms 5: ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 32.448ms 6: 360.net.any2ix.coresite.com (206.223.143.201) 657.255ms 7: lax1-core-01-xe-0-0-0.360.net (66.62.2.213) 36.196ms 8: den1-core-01-ae0.360.net (66.62.2.169) 246.045ms 9: den1-edge-01-lag2.360.net (66.62.2.194) asymm 8 271.648ms 10: 66.62.160.30 (66.62.160.30) asymm 9 543.816ms 11: CYSWYDC01ESW1-001-1-1.GREENHOUSEDATA.NET (208.89.160.11) asymm 10 793.839ms reached
from denver box to dallas (mtr is all thats installed and cant install anything else connectivity is too horrible)
Packets Pings Host
Loss% Snt Last Avg Best Wrst StDev
2. CYSWYDC01RTR1-001-0-1.GREENHOUSEDATA.NET
0.0% 17 0.3 0.3 0.3 0.4 0.0 3. ge-6-13.car2.Denver1.Level3.net
5.9% 17 4.6 121.9 4.6 417.4 116.1 4. ae-21-52.car1.Denver1.Level3.net
0.0% 17 101.9 25.2 2.9 140.6 40.6 5. te1-5.bbr01.cf01.den01.networklayer.com
43.8% 17 565.9 479.9 169.5 919.2 214.1 6. ae7.bbr01.cs01.den01.networklayer.com
37.5% 17 642.3 665.4 170.0 928.8 261.4 7. ae12.bbr02.eq01.dal03.networklayer.com
50.0% 17 1168. 650.2 95.4 1168. 391.4 8. po32.dsr01.dllstx3.networklayer.com
31.2% 17 423.9 643.3 80.0 1135. 368.2 9. po101.dsr01.dllstx5.networklayer.com
50.0% 16 347.1 814.1 347.1 1874. 508.6
thanks for pointing it out chris
On Fri, Dec 14, 2012 at 12:00 AM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Chris,
You need to provide traceroutes from both directions. The below is for Dallas --> Denver, but the return path (Denver --> Dallas) needs to be provided too. You might find that the return path goes through some provider other than Zayo/360/Abovenet/whateverthey'recalledtoday.
I know that's hard to do when the path has latency or packet loss (which you don't show in your results -- you only show latency), but this is exactly what a cronjob traceroute writing to a log file is for. :-)
My point: remember that routing on the Internet most of the time is asymmetric. Reference material (read, do not skim):
http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N4...
Finally, you didn't provide IP addresses of either server (in Denver or Dallas), so when you ask for "someone else to test", that's not easily doable, at least not to the endpoints involved (pinging routers is not sufficient evidence, sadly).
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Thu, Dec 13, 2012 at 11:10:32PM -0500, chris wrote:
Seeing massive packet loss and high latency from 360.net (which appears to be owned by zayo/abovenet/whatever) in denver....
Here is a trace snip from our server @ softlayer in dallas to one of our servers in denver
5: ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 29.698ms 6: no reply 7: 66.62.2.213 (66.62.2.213) 4588.320ms 8: den1-core-01-xe-1-1-0.360.net (66.62.2.166) 22583.831ms
I only have one one server in denver that I can reproduce this with hoping someone can else can test and reproduce similar issues
thanks chris _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Hi Tim, Just tracerouted your ip and saw a hop in the same location: 11. xe-0-0-1.mpr1.den1.above.net 0.0% 32 59.6 60.0 59.3 70.6 1.9 12. 208.185.20.58.t01657-09.above.ne 0.0% 32 59.7 60.3 59.4 76.0 2.9 13. den1-edge-01-lag2.360.net 46.9% 32 60.6 63.0 60.3 95.1 8.3 14. 66.62.160.42 0.0% 32 200.1 79.1 65.9 200.1 37.1 15. static-65-19-63-33.cybermesa.com 0.0% 32 248.4 80.9 67.8 248.4 41.3 Hop 13 was one of the links we were hitting and seeing alot of latency and loss, also noticed its acronym LAG maybe this was some kind of weird LACP failure Anyway after checking your side and seeing it was normal, I retested and the issue appears to be resolved for me in both directions So either someone on here silently bumped the issue and got it fixed or softlayer was able to resolve this after my ticket Either way, thanks! chris On Fri, Dec 14, 2012 at 12:41 AM, Tim Densmore <tdensmore@tarpit.cybermesa.com> wrote:
Hi Chris,
Not entirely what we saw tonight, but we had issues reaching some places via a very similar path, and we peer with 360 and level3. I've localprefed around it for the time being, but only just saw this email. Can't confirm our issue was what you were seeing, but we were definitely seeing packet loss, latency, and what appeared to be occasional loops on networklayer. Not particularly useful, I realise, but I guess count me in as a tentative "me too."
TD
On 12/13/2012 10:11 PM, chris wrote:
Ok sorry, the denver side for us is 208.89.160.11 and the dallas softlayer side for us is 70.87.254.1
You are right about different paths it appears that the route back is direct via level3 instead of 360/zayo and same latency is present so looks like more likely a softlayer issue?
trace from our dallas box(softlayer) to our denver box(greenhouse): 2: po101.dsr01.dllstx5.networklayer.com (70.87.254.1) 1.941ms 3: po51.dsr01.dllstx3.networklayer.com (70.85.127.105) 2.217ms 4: ae16.bbr01.eq01.dal03.networklayer.com (173.192.18.224) 1.719ms 5: ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 32.448ms 6: 360.net.any2ix.coresite.com (206.223.143.201) 657.255ms 7: lax1-core-01-xe-0-0-0.360.net (66.62.2.213) 36.196ms 8: den1-core-01-ae0.360.net (66.62.2.169) 246.045ms 9: den1-edge-01-lag2.360.net (66.62.2.194) asymm 8 271.648ms 10: 66.62.160.30 (66.62.160.30) asymm 9 543.816ms 11: CYSWYDC01ESW1-001-1-1.GREENHOUSEDATA.NET (208.89.160.11) asymm 10 793.839ms reached
from denver box to dallas (mtr is all thats installed and cant install anything else connectivity is too horrible)
Packets Pings Host
Loss% Snt Last Avg Best Wrst StDev
2. CYSWYDC01RTR1-001-0-1.GREENHOUSEDATA.NET
0.0% 17 0.3 0.3 0.3 0.4 0.0 3. ge-6-13.car2.Denver1.Level3.net
5.9% 17 4.6 121.9 4.6 417.4 116.1 4. ae-21-52.car1.Denver1.Level3.net
0.0% 17 101.9 25.2 2.9 140.6 40.6 5. te1-5.bbr01.cf01.den01.networklayer.com
43.8% 17 565.9 479.9 169.5 919.2 214.1 6. ae7.bbr01.cs01.den01.networklayer.com
37.5% 17 642.3 665.4 170.0 928.8 261.4 7. ae12.bbr02.eq01.dal03.networklayer.com
50.0% 17 1168. 650.2 95.4 1168. 391.4 8. po32.dsr01.dllstx3.networklayer.com
31.2% 17 423.9 643.3 80.0 1135. 368.2 9. po101.dsr01.dllstx5.networklayer.com
50.0% 16 347.1 814.1 347.1 1874. 508.6
thanks for pointing it out chris
On Fri, Dec 14, 2012 at 12:00 AM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Chris,
You need to provide traceroutes from both directions. The below is for Dallas --> Denver, but the return path (Denver --> Dallas) needs to be provided too. You might find that the return path goes through some provider other than Zayo/360/Abovenet/whateverthey'recalledtoday.
I know that's hard to do when the path has latency or packet loss (which you don't show in your results -- you only show latency), but this is exactly what a cronjob traceroute writing to a log file is for. :-)
My point: remember that routing on the Internet most of the time is asymmetric. Reference material (read, do not skim):
http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N4...
Finally, you didn't provide IP addresses of either server (in Denver or Dallas), so when you ask for "someone else to test", that's not easily doable, at least not to the endpoints involved (pinging routers is not sufficient evidence, sadly).
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Thu, Dec 13, 2012 at 11:10:32PM -0500, chris wrote:
Seeing massive packet loss and high latency from 360.net (which appears to be owned by zayo/abovenet/whatever) in denver....
Here is a trace snip from our server @ softlayer in dallas to one of our servers in denver
5: ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 29.698ms 6: no reply 7: 66.62.2.213 (66.62.2.213) 4588.320ms 8: den1-core-01-xe-1-1-0.360.net (66.62.2.166) 22583.831ms
I only have one one server in denver that I can reproduce this with hoping someone can else can test and reproduce similar issues
thanks chris _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (3)
-
chris -
Jeremy Chadwick -
Tim Densmore