Seeing intermittent issues with level 3 router at 64.208.27.65 Hop (ms) (ms) (ms) IP Address Host name 1 0 0 0 8.9.232.73 xe-5-3-0.edge3.dallas1.level3.net 2 0 0 0 4.69.145.141 ae-3-80.edge4.dallas3.level3.net 3 8 0 0 4.68.63.146 gblx-level3-40g.dallas3.level3.net 4 0 0 0 67.16.150.190 ae2-80g.csr1.dal2.gblx.net 5 0 0 0 64.208.27.65 te1-3-10g.ar3.dal2.gblx.net Hop (ms) (ms) (ms) IP Address Host name 1 1 0 0 8.9.232.73 xe-5-3-0.edge3.dallas1.level3.net 2 0 0 0 4.69.145.141 ae-3-80.edge4.dallas3.level3.net 3 0 0 0 4.68.63.146 gblx-level3-40g.dallas3.level3.net 4 0 0 0 67.16.150.190 ae2-80g.csr1.dal2.gblx.net 5 7 1 208 64.208.27.65 te1-3-10g.ar3.dal2.gblx.net Trace complete Ping times from 8ms to 330ms. -- Jeremie Chism Triton Communications
1. Usually Level 3-owned routers use ICMP prioritisation; routing (forwarding) has priority over this. 2. Many traceroute implementations use UDP for their probe packets (in attempt to solicit ICMP TTL exceeded messages to learn the network topology); please make sure you're using pure ICMP all the way through if possible. 3. Your below traces indicate a destination IP of 64.208.27.65, which is not sufficient (ICMP prioritisation will be used to explain what you're seeing, even if there is a real issue). Please pick a destination IP of an actual host that's somewhere behind that router which will respond to ICMP packets reliably. You need to see if the packet loss "trickles downward" past that hop to every subsequent hop after that point (including the destination host), or if high latency, if the latency "trickles downward" as well. Here's an example of a bogus claim as a result of ICMP prio: === Mon Jan 21 01:39:00 PST 2013 (1358761140) HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.3 0.3 0.2 0.4 2.|-- 67.180.84.1 0.0% 40 40 21.8 23.0 11.2 35.4 3.|-- 68.85.191.249 0.0% 40 40 10.1 9.8 8.1 14.2 4.|-- 69.139.198.82 0.0% 40 40 21.0 17.2 9.9 22.8 5.|-- 68.86.90.93 0.0% 40 40 23.1 19.2 11.8 30.3 6.|-- 68.86.87.190 15.0% 40 34 22.3 25.9 19.9 122.6 7.|-- 75.149.231.238 37.5% 40 25 56.3 26.5 20.0 56.3 8.|-- 67.199.135.102 0.0% 40 40 21.1 25.8 20.9 49.0 9.|-- 206.125.172.42 0.0% 40 40 20.7 21.3 19.8 22.5 === END While here's a real issue: === Fri Jan 4 07:27:00 PST 2013 (1357313220) HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 67.180.84.1 92.5% 40 3 16.0 16.7 11.5 22.7 3.|-- 68.85.191.249 92.5% 40 3 9.4 9.3 8.7 9.7 4.|-- 69.139.198.82 90.0% 40 4 19.1 23.0 12.6 39.2 5.|-- 68.86.90.93 90.0% 40 4 12.4 14.8 12.4 17.5 6.|-- 68.86.87.190 90.0% 40 4 48.3 28.6 21.6 48.3 7.|-- 75.149.231.238 90.0% 40 4 32.7 23.9 20.4 32.7 8.|-- 67.199.135.102 90.0% 40 4 145.7 52.5 21.3 145.7 9.|-- 206.125.172.42 90.0% 40 4 20.8 21.3 19.9 22.4 === END All of this is covered here (read do not skim): http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N4... -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Tue, Jan 22, 2013 at 11:40:39AM -0600, Jeremie Chism wrote:
Seeing intermittent issues with level 3 router at 64.208.27.65
Hop (ms) (ms) (ms) IP Address Host name 1 0 0 0 8.9.232.73 xe-5-3-0.edge3.dallas1.level3.net 2 0 0 0 4.69.145.141 ae-3-80.edge4.dallas3.level3.net 3 8 0 0 4.68.63.146 gblx-level3-40g.dallas3.level3.net 4 0 0 0 67.16.150.190 ae2-80g.csr1.dal2.gblx.net 5 0 0 0 64.208.27.65 te1-3-10g.ar3.dal2.gblx.net
Hop (ms) (ms) (ms) IP Address Host name 1 1 0 0 8.9.232.73 xe-5-3-0.edge3.dallas1.level3.net 2 0 0 0 4.69.145.141 ae-3-80.edge4.dallas3.level3.net 3 0 0 0 4.68.63.146 gblx-level3-40g.dallas3.level3.net 4 0 0 0 67.16.150.190 ae2-80g.csr1.dal2.gblx.net 5 7 1 208 64.208.27.65 te1-3-10g.ar3.dal2.gblx.net Trace complete
Ping times from 8ms to 330ms.
-- Jeremie Chism Triton Communications
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Tue, Jan 22, 2013 at 1:45 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
You need to see if the packet loss "trickles downward" past that hop to every subsequent hop after that point (including the destination host), or if high latency, if the latency "trickles downward" as well.
While is still anecdotal, I was having L3 issues yesterday and this was a posted few minutes ago to a VoiceOps list that I follow:
Any one seeing packet loss or other issue on the west coast with Level 3?
Let us know what you turn up! Gabe
We are getting multiple reports of call quality when tracing over these links they are egressing and ingressing level3 I have a ticket opened for our circuit to them to get more information on the issue --- Respectfully, Daniel L. Green II Network Engineer Momentum | www.gomomentum.com O: 702.933.1545 Ext: 1337 W: 702.403.1234 F: 702.365.8647 Facebook | Twitter | LinkedIn 8350 S. Durango | Suite 220 | Las Vegas | Nevada | 89113 The information contained in this E-mail message belongs to Momentum Telecom, Inc., is privileged, confidential, and may be protected from disclosure; please be aware that any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. If you think that you have received this E-mail message in error, please reply to the sender. This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. It is the responsibility, however, of the recipient to ensure that it is virus free. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Gabriel Gunderson Sent: Tuesday, January 22, 2013 1:46 PM To: Jeremy Chadwick Cc: outages@outages.org Subject: Re: [outages] Level 3 issues On Tue, Jan 22, 2013 at 1:45 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
You need to see if the packet loss "trickles downward" past that hop to every subsequent hop after that point (including the destination host), or if high latency, if the latency "trickles downward" as well.
While is still anecdotal, I was having L3 issues yesterday and this was a posted few minutes ago to a VoiceOps list that I follow:
Any one seeing packet loss or other issue on the west coast with Level 3?
Let us know what you turn up! Gabe _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
That might explain why I seeing IP SLA time-outs to three disparate dst over these links this morning. Matt On Jan 22, 2013, at 12:45 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
1. Usually Level 3-owned routers use ICMP prioritisation; routing (forwarding) has priority over this.
2. Many traceroute implementations use UDP for their probe packets (in attempt to solicit ICMP TTL exceeded messages to learn the network topology); please make sure you're using pure ICMP all the way through if possible.
3. Your below traces indicate a destination IP of 64.208.27.65, which is not sufficient (ICMP prioritisation will be used to explain what you're seeing, even if there is a real issue). Please pick a destination IP of an actual host that's somewhere behind that router which will respond to ICMP packets reliably.
You need to see if the packet loss "trickles downward" past that hop to every subsequent hop after that point (including the destination host), or if high latency, if the latency "trickles downward" as well.
Here's an example of a bogus claim as a result of ICMP prio:
=== Mon Jan 21 01:39:00 PST 2013 (1358761140) HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.3 0.3 0.2 0.4 2.|-- 67.180.84.1 0.0% 40 40 21.8 23.0 11.2 35.4 3.|-- 68.85.191.249 0.0% 40 40 10.1 9.8 8.1 14.2 4.|-- 69.139.198.82 0.0% 40 40 21.0 17.2 9.9 22.8 5.|-- 68.86.90.93 0.0% 40 40 23.1 19.2 11.8 30.3 6.|-- 68.86.87.190 15.0% 40 34 22.3 25.9 19.9 122.6 7.|-- 75.149.231.238 37.5% 40 25 56.3 26.5 20.0 56.3 8.|-- 67.199.135.102 0.0% 40 40 21.1 25.8 20.9 49.0 9.|-- 206.125.172.42 0.0% 40 40 20.7 21.3 19.8 22.5 === END
While here's a real issue:
=== Fri Jan 4 07:27:00 PST 2013 (1357313220) HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 67.180.84.1 92.5% 40 3 16.0 16.7 11.5 22.7 3.|-- 68.85.191.249 92.5% 40 3 9.4 9.3 8.7 9.7 4.|-- 69.139.198.82 90.0% 40 4 19.1 23.0 12.6 39.2 5.|-- 68.86.90.93 90.0% 40 4 12.4 14.8 12.4 17.5 6.|-- 68.86.87.190 90.0% 40 4 48.3 28.6 21.6 48.3 7.|-- 75.149.231.238 90.0% 40 4 32.7 23.9 20.4 32.7 8.|-- 67.199.135.102 90.0% 40 4 145.7 52.5 21.3 145.7 9.|-- 206.125.172.42 90.0% 40 4 20.8 21.3 19.9 22.4 === END
All of this is covered here (read do not skim):
http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N4...
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Tue, Jan 22, 2013 at 11:40:39AM -0600, Jeremie Chism wrote:
Seeing intermittent issues with level 3 router at 64.208.27.65
Hop (ms) (ms) (ms) IP Address Host name 1 0 0 0 8.9.232.73 xe-5-3-0.edge3.dallas1.level3.net 2 0 0 0 4.69.145.141 ae-3-80.edge4.dallas3.level3.net 3 8 0 0 4.68.63.146 gblx-level3-40g.dallas3.level3.net 4 0 0 0 67.16.150.190 ae2-80g.csr1.dal2.gblx.net 5 0 0 0 64.208.27.65 te1-3-10g.ar3.dal2.gblx.net
Hop (ms) (ms) (ms) IP Address Host name 1 1 0 0 8.9.232.73 xe-5-3-0.edge3.dallas1.level3.net 2 0 0 0 4.69.145.141 ae-3-80.edge4.dallas3.level3.net 3 0 0 0 4.68.63.146 gblx-level3-40g.dallas3.level3.net 4 0 0 0 67.16.150.190 ae2-80g.csr1.dal2.gblx.net 5 7 1 208 64.208.27.65 te1-3-10g.ar3.dal2.gblx.net Trace complete
Ping times from 8ms to 330ms.
-- Jeremie Chism Triton Communications
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Did anyone ever get resolution on this issue? -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Matthew Laughlin Sent: Tuesday, January 22, 2013 4:17 PM To: Jeremy Chadwick Cc: outages@outages.org Subject: Re: [outages] Level 3 issues That might explain why I seeing IP SLA time-outs to three disparate dst over these links this morning. Matt On Jan 22, 2013, at 12:45 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
1. Usually Level 3-owned routers use ICMP prioritisation; routing (forwarding) has priority over this.
2. Many traceroute implementations use UDP for their probe packets (in attempt to solicit ICMP TTL exceeded messages to learn the network topology); please make sure you're using pure ICMP all the way through if possible.
3. Your below traces indicate a destination IP of 64.208.27.65, which is not sufficient (ICMP prioritisation will be used to explain what you're seeing, even if there is a real issue). Please pick a destination IP of an actual host that's somewhere behind that router which will respond to ICMP packets reliably.
You need to see if the packet loss "trickles downward" past that hop to every subsequent hop after that point (including the destination host), or if high latency, if the latency "trickles downward" as well.
Here's an example of a bogus claim as a result of ICMP prio:
=== Mon Jan 21 01:39:00 PST 2013 (1358761140) HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.3 0.3 0.2 0.4 2.|-- 67.180.84.1 0.0% 40 40 21.8 23.0 11.2 35.4 3.|-- 68.85.191.249 0.0% 40 40 10.1 9.8 8.1 14.2 4.|-- 69.139.198.82 0.0% 40 40 21.0 17.2 9.9 22.8 5.|-- 68.86.90.93 0.0% 40 40 23.1 19.2 11.8 30.3 6.|-- 68.86.87.190 15.0% 40 34 22.3 25.9 19.9 122.6 7.|-- 75.149.231.238 37.5% 40 25 56.3 26.5 20.0 56.3 8.|-- 67.199.135.102 0.0% 40 40 21.1 25.8 20.9 49.0 9.|-- 206.125.172.42 0.0% 40 40 20.7 21.3 19.8 22.5 === END
While here's a real issue:
=== Fri Jan 4 07:27:00 PST 2013 (1357313220) HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- 192.168.1.1 0.0% 40 40 0.2 0.3 0.2 0.4 2.|-- 67.180.84.1 92.5% 40 3 16.0 16.7 11.5 22.7 3.|-- 68.85.191.249 92.5% 40 3 9.4 9.3 8.7 9.7 4.|-- 69.139.198.82 90.0% 40 4 19.1 23.0 12.6 39.2 5.|-- 68.86.90.93 90.0% 40 4 12.4 14.8 12.4 17.5 6.|-- 68.86.87.190 90.0% 40 4 48.3 28.6 21.6 48.3 7.|-- 75.149.231.238 90.0% 40 4 32.7 23.9 20.4 32.7 8.|-- 67.199.135.102 90.0% 40 4 145.7 52.5 21.3 145.7 9.|-- 206.125.172.42 90.0% 40 4 20.8 21.3 19.9 22.4 === END
All of this is covered here (read do not skim):
http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Tracero ute_N47_Sun.pdf
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Tue, Jan 22, 2013 at 11:40:39AM -0600, Jeremie Chism wrote:
Seeing intermittent issues with level 3 router at 64.208.27.65
Hop (ms) (ms) (ms) IP Address Host name 1 0 0 0 8.9.232.73 xe-5-3-0.edge3.dallas1.level3.net 2 0 0 0 4.69.145.141 ae-3-80.edge4.dallas3.level3.net 3 8 0 0 4.68.63.146 gblx-level3-40g.dallas3.level3.net 4 0 0 0 67.16.150.190 ae2-80g.csr1.dal2.gblx.net 5 0 0 0 64.208.27.65 te1-3-10g.ar3.dal2.gblx.net
Hop (ms) (ms) (ms) IP Address Host name 1 1 0 0 8.9.232.73 xe-5-3-0.edge3.dallas1.level3.net 2 0 0 0 4.69.145.141 ae-3-80.edge4.dallas3.level3.net 3 0 0 0 4.68.63.146 gblx-level3-40g.dallas3.level3.net 4 0 0 0 67.16.150.190 ae2-80g.csr1.dal2.gblx.net 5 7 1 208 64.208.27.65 te1-3-10g.ar3.dal2.gblx.net Trace complete
Ping times from 8ms to 330ms.
-- Jeremie Chism Triton Communications
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
All I got from Level3 was that the problem went away before it could be isolated. On 01/22/2013 10:34 PM, Blake Pfankuch wrote:
Did anyone ever get resolution on this issue?
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Matthew Laughlin Sent: Tuesday, January 22, 2013 4:17 PM To: Jeremy Chadwick Cc: outages@outages.org Subject: Re: [outages] Level 3 issues
That might explain why I seeing IP SLA time-outs to three disparate dst over these links this morning.
Matt
On Jan 22, 2013, at 12:45 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
participants (7)
-
Blake Pfankuch -
Daniel Green -
Gabriel Gunderson -
Jeremie Chism -
Jeremy Chadwick -
John Barbieri -
Matthew Laughlin