Level3 ICMP Filtering in Chicago?
At about 3:30PM central time today we started seeing ICMP packet loss going into Level3 networks, we believe it is likely some sort of Level3 filtering of some sort. Two traces below from the same origin to the same destination at the same time, one is ICMP and one is UDP. It is not just traceroute ICMP packets, we cannot ping our own equipment when it takes a Level3 path. Other UDP/TCP traffic so far seems unaffected. Is anyone else seeing this? This appears to be a Chicago-area connection. Using ICMP to trace to: 204.11.209.66 Tracing to host 204.11.209.66 traceroute to 204.11.209.66 (204.11.209.66), 30 hops max, 60 byte packets 1 216.17.34.1 (216.17.34.1) [as10242/AS10242] 0.558 ms 0.554 ms 0.552 ms 2 v103.usi-cr02-mpls.usinternet.com (216.17.36.25) [as10242/AS10242] 1.094 ms 1.199 ms 1.555 ms 3 10gigabitethernet2-6.core1.msp1.he.net (216.66.73.173) [AS6939] 1.203 ms 1.530 ms 1.533 ms 4 100gigabitethernet7-1.core1.chi1.he.net (184.105.223.177) [AS46841] 9.076 ms 9.084 ms 9.154 ms 5 chi-bb1-link.telia.net (213.248.104.213) [AS1299] 9.155 ms 9.163 ms 9.163 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * Using UDP to trace to: 204.11.209.66 Tracing to host 204.11.209.66 traceroute to 204.11.209.66 (204.11.209.66), 30 hops max, 60 byte packets 1 216.17.34.1 (216.17.34.1) [as10242/AS10242] 0.438 ms 0.480 ms 0.495 ms 2 v103.usi-cr02-mpls.usinternet.com (216.17.36.25) [as10242/AS10242] 1.106 ms 1.166 ms 1.235 ms 3 10gigabitethernet2-6.core1.msp1.he.net (216.66.73.173) [AS6939] 4.447 ms 4.445 ms 4.440 ms 4 100gigabitethernet7-1.core1.chi1.he.net (184.105.223.177) [AS46841] 12.821 ms 12.898 ms 12.893 ms 5 chi-bb1-link.telia.net (213.248.104.213) [AS1299] 9.056 ms 9.052 ms 9.054 ms 6 level3-ic-300135-chi-bb1.c.telia.net (213.248.87.238) [AS1299] 23.943 ms 24.000 ms 18.147 ms 7 ae-32-52.ebr2.Chicago1.Level3.net (4.69.138.62) [AS3356] 20.890 ms 18.162 ms 18.189 ms 8 ae-5-5.ebr2.Chicago2.Level3.net (4.69.140.194) [AS3356] 20.993 ms 20.982 ms 22.160 ms 9 ae-22-52.car2.Chicago2.Level3.net (4.69.138.165) [AS3356] 18.164 ms 22.165 ms 18.231 ms 10 RED-ANVIL-L.car2.Chicago2.Level3.net (4.30.14.162) [AS3356] 44.853 ms 44.822 ms 44.810 ms 11 ge-1-0-1-0.cr1.noc.ip.redanvil.net (204.15.100.173) [AS33693] 19.849 ms 19.888 ms 19.797 ms
On Mon, 30 Sep 2013, Justin Krejci wrote:
At about 3:30PM central time today we started seeing ICMP packet loss going into Level3 networks, we believe it is likely some sort of Level3 filtering of some sort.
Using ICMP to trace to:
204.11.209.66
With direct connectivity to Level3 in Minneapolis (which transits Chicago for, well, everything), it appears to be working.. $ traceroute -I 204.11.209.66 traceroute to 204.11.209.66 (204.11.209.66), 30 hops max, 60 byte packets 4 ae-11-11.car1.Minneapolis1.Level3.net (4.69.136.101) 38.164 ms 38.218 ms 38.289 ms 5 4.69.201.82 (4.69.201.82) 8.940 ms 8.939 ms 8.936 ms 6 ae-12-51.car2.Chicago2.Level3.net (4.69.138.133) 9.063 ms 9.005 ms 9.054 ms 7 RED-ANVIL-L.car2.Chicago2.Level3.net (4.30.14.162) 9.311 ms 9.281 ms 9.277 ms 8 ge-1-0-1-0.cr1.noc.ip.redanvil.net (204.15.100.173) 11.838 ms 11.917 ms 11.908 ms 9 204.11.209.66 (204.11.209.66) 11.894 ms 11.944 ms 11.933 ms
Thank you. It looks like it may just be that Telia hop into Level3 possibly. ________________________________________ From: Nate Carlson [nanog@natecarlson.com] Sent: Monday, September 30, 2013 4:35 PM To: Justin Krejci Cc: outages@outages.org Subject: Re: [outages] Level3 ICMP Filtering in Chicago? On Mon, 30 Sep 2013, Justin Krejci wrote:
At about 3:30PM central time today we started seeing ICMP packet loss going into Level3 networks, we believe it is likely some sort of Level3 filtering of some sort.
Using ICMP to trace to:
204.11.209.66
With direct connectivity to Level3 in Minneapolis (which transits Chicago for, well, everything), it appears to be working.. $ traceroute -I 204.11.209.66 traceroute to 204.11.209.66 (204.11.209.66), 30 hops max, 60 byte packets 4 ae-11-11.car1.Minneapolis1.Level3.net (4.69.136.101) 38.164 ms 38.218 ms 38.289 ms 5 4.69.201.82 (4.69.201.82) 8.940 ms 8.939 ms 8.936 ms 6 ae-12-51.car2.Chicago2.Level3.net (4.69.138.133) 9.063 ms 9.005 ms 9.054 ms 7 RED-ANVIL-L.car2.Chicago2.Level3.net (4.30.14.162) 9.311 ms 9.281 ms 9.277 ms 8 ge-1-0-1-0.cr1.noc.ip.redanvil.net (204.15.100.173) 11.838 ms 11.917 ms 11.908 ms 9 204.11.209.66 (204.11.209.66) 11.894 ms 11.944 ms 11.933 ms
It could be the return path that is blocking ICMP echo-reply (i.e. destination receives ICMP echo, responds with ICMP echo-reply, but goes out a path/interface that has it filtered). If you can get on to 204.11.209.66 and capture ICMP (both directions) you should be able to determine if it's even seeing the echo. L3 and Telia both have looking glasses: http://lookingglass.level3.net/ http://looking-glass.telia.net/ -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | On Mon, Sep 30, 2013 at 09:07:00PM +0000, Justin Krejci wrote:
At about 3:30PM central time today we started seeing ICMP packet loss going into Level3 networks, we believe it is likely some sort of Level3 filtering of some sort.
Two traces below from the same origin to the same destination at the same time, one is ICMP and one is UDP.
It is not just traceroute ICMP packets, we cannot ping our own equipment when it takes a Level3 path. Other UDP/TCP traffic so far seems unaffected.
Is anyone else seeing this? This appears to be a Chicago-area connection.
Using ICMP to trace to:
204.11.209.66
Tracing to host 204.11.209.66
traceroute to 204.11.209.66 (204.11.209.66), 30 hops max, 60 byte packets 1 216.17.34.1 (216.17.34.1) [as10242/AS10242] 0.558 ms 0.554 ms 0.552 ms 2 v103.usi-cr02-mpls.usinternet.com (216.17.36.25) [as10242/AS10242] 1.094 ms 1.199 ms 1.555 ms 3 10gigabitethernet2-6.core1.msp1.he.net (216.66.73.173) [AS6939] 1.203 ms 1.530 ms 1.533 ms 4 100gigabitethernet7-1.core1.chi1.he.net (184.105.223.177) [AS46841] 9.076 ms 9.084 ms 9.154 ms 5 chi-bb1-link.telia.net (213.248.104.213) [AS1299] 9.155 ms 9.163 ms 9.163 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * *
Using UDP to trace to:
204.11.209.66
Tracing to host 204.11.209.66
traceroute to 204.11.209.66 (204.11.209.66), 30 hops max, 60 byte packets 1 216.17.34.1 (216.17.34.1) [as10242/AS10242] 0.438 ms 0.480 ms 0.495 ms 2 v103.usi-cr02-mpls.usinternet.com (216.17.36.25) [as10242/AS10242] 1.106 ms 1.166 ms 1.235 ms 3 10gigabitethernet2-6.core1.msp1.he.net (216.66.73.173) [AS6939] 4.447 ms 4.445 ms 4.440 ms 4 100gigabitethernet7-1.core1.chi1.he.net (184.105.223.177) [AS46841] 12.821 ms 12.898 ms 12.893 ms 5 chi-bb1-link.telia.net (213.248.104.213) [AS1299] 9.056 ms 9.052 ms 9.054 ms 6 level3-ic-300135-chi-bb1.c.telia.net (213.248.87.238) [AS1299] 23.943 ms 24.000 ms 18.147 ms 7 ae-32-52.ebr2.Chicago1.Level3.net (4.69.138.62) [AS3356] 20.890 ms 18.162 ms 18.189 ms 8 ae-5-5.ebr2.Chicago2.Level3.net (4.69.140.194) [AS3356] 20.993 ms 20.982 ms 22.160 ms 9 ae-22-52.car2.Chicago2.Level3.net (4.69.138.165) [AS3356] 18.164 ms 22.165 ms 18.231 ms 10 RED-ANVIL-L.car2.Chicago2.Level3.net (4.30.14.162) [AS3356] 44.853 ms 44.822 ms 44.810 ms 11 ge-1-0-1-0.cr1.noc.ip.redanvil.net (204.15.100.173) [AS33693] 19.849 ms 19.888 ms 19.797 ms
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
True. We had verified that the ICMP echo is not arriving to the destination, I forgot to mention that already. Thanks for the pointers to the LG's. I can't seem to find the correct combo on their LG's to replicate this behavior. However if you use http://lg.he.net and choose the origin of Minneapolis the trace goes through this same Telia link in Chicago. I will try and reach out more to the parties involved. Thank you! ________________________________________ From: Jeremy Chadwick [jdc@koitsu.org] Sent: Monday, September 30, 2013 4:47 PM To: Justin Krejci Cc: outages@outages.org Subject: Re: [outages] Level3 ICMP Filtering in Chicago? It could be the return path that is blocking ICMP echo-reply (i.e. destination receives ICMP echo, responds with ICMP echo-reply, but goes out a path/interface that has it filtered). If you can get on to 204.11.209.66 and capture ICMP (both directions) you should be able to determine if it's even seeing the echo. L3 and Telia both have looking glasses: http://lookingglass.level3.net/ http://looking-glass.telia.net/ -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | On Mon, Sep 30, 2013 at 09:07:00PM +0000, Justin Krejci wrote:
At about 3:30PM central time today we started seeing ICMP packet loss going into Level3 networks, we believe it is likely some sort of Level3 filtering of some sort.
Two traces below from the same origin to the same destination at the same time, one is ICMP and one is UDP.
It is not just traceroute ICMP packets, we cannot ping our own equipment when it takes a Level3 path. Other UDP/TCP traffic so far seems unaffected.
Is anyone else seeing this? This appears to be a Chicago-area connection.
Using ICMP to trace to:
204.11.209.66
Tracing to host 204.11.209.66
traceroute to 204.11.209.66 (204.11.209.66), 30 hops max, 60 byte packets 1 216.17.34.1 (216.17.34.1) [as10242/AS10242] 0.558 ms 0.554 ms 0.552 ms 2 v103.usi-cr02-mpls.usinternet.com (216.17.36.25) [as10242/AS10242] 1.094 ms 1.199 ms 1.555 ms 3 10gigabitethernet2-6.core1.msp1.he.net (216.66.73.173) [AS6939] 1.203 ms 1.530 ms 1.533 ms 4 100gigabitethernet7-1.core1.chi1.he.net (184.105.223.177) [AS46841] 9.076 ms 9.084 ms 9.154 ms 5 chi-bb1-link.telia.net (213.248.104.213) [AS1299] 9.155 ms 9.163 ms 9.163 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * *
Using UDP to trace to:
204.11.209.66
Tracing to host 204.11.209.66
traceroute to 204.11.209.66 (204.11.209.66), 30 hops max, 60 byte packets 1 216.17.34.1 (216.17.34.1) [as10242/AS10242] 0.438 ms 0.480 ms 0.495 ms 2 v103.usi-cr02-mpls.usinternet.com (216.17.36.25) [as10242/AS10242] 1.106 ms 1.166 ms 1.235 ms 3 10gigabitethernet2-6.core1.msp1.he.net (216.66.73.173) [AS6939] 4.447 ms 4.445 ms 4.440 ms 4 100gigabitethernet7-1.core1.chi1.he.net (184.105.223.177) [AS46841] 12.821 ms 12.898 ms 12.893 ms 5 chi-bb1-link.telia.net (213.248.104.213) [AS1299] 9.056 ms 9.052 ms 9.054 ms 6 level3-ic-300135-chi-bb1.c.telia.net (213.248.87.238) [AS1299] 23.943 ms 24.000 ms 18.147 ms 7 ae-32-52.ebr2.Chicago1.Level3.net (4.69.138.62) [AS3356] 20.890 ms 18.162 ms 18.189 ms 8 ae-5-5.ebr2.Chicago2.Level3.net (4.69.140.194) [AS3356] 20.993 ms 20.982 ms 22.160 ms 9 ae-22-52.car2.Chicago2.Level3.net (4.69.138.165) [AS3356] 18.164 ms 22.165 ms 18.231 ms 10 RED-ANVIL-L.car2.Chicago2.Level3.net (4.30.14.162) [AS3356] 44.853 ms 44.822 ms 44.810 ms 11 ge-1-0-1-0.cr1.noc.ip.redanvil.net (204.15.100.173) [AS33693] 19.849 ms 19.888 ms 19.797 ms
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (3)
-
Jeremy Chadwick -
Justin Krejci -
Nate Carlson