FWIW: I've been grepping through the last 4 years of my mail spool archives to try and find the name/Email of the individual at TeliaSonera who I communicated with about issues with Telia in the US the last time I reported issues. So far I haven't had any luck, which means it might have been something older than 4 years ago. Grumble. :-( -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | On Mon, Sep 30, 2013 at 07:09:44PM -0700, Ryan Bonnell wrote:

Also seeing the same thing for Telia Los Angeles to Level3. ICMP fails, UDP/TCP/GRE function properly. Re-routing to a non Telia path resolves the problem to Level3. Here is a smokeping graph for extra fun. Timestamps are Pacific Time.

-Ryan

$ traceroute www.level3.com traceroute to www.level3.com (4.68.80.110), 30 hops max, 60 byte packets

2 las-b3-link.telia.net (213.248.x.x) 0.605 ms 0.610 ms 0.594 ms 3 las-bb1-link.telia.net (213.155.130.126) 1.884 ms las-bb1-link.telia.net (213.155.134.252) 1.883 ms las-bb1-link.telia.net(213.155.137.58) 0.608 ms 4 ae8.edge1.LosAngeles.Level3.net (4.68.70.129) 15.943 ms 13.126 ms 11.956 ms

$ traceroute www.level3.com -I traceroute to www.level3.com (4.68.80.110), 30 hops max, 60 byte packets

2 las-b3-link.telia.net (213.248.x.x) 0.318 ms 0.328 ms 0.327 ms 3 las-bb1-link.telia.net (213.155.137.58) 0.429 ms 0.429 ms 0.428 ms 4 * * * 5 * * * 6 * * *

On Mon, Sep 30, 2013 at 3:42 PM, Justin Krejci <JKrejci@usinternet.com>wrote:

...

True. We had verified that the ICMP echo is not arriving to the destination, I forgot to mention that already. Thanks for the pointers to the LG's. I can't seem to find the correct combo on their LG's to replicate this behavior. However if you use http://lg.he.net and choose the origin of Minneapolis the trace goes through this same Telia link in Chicago.

I will try and reach out more to the parties involved.

Thank you!

________________________________________ From: Jeremy Chadwick [jdc@koitsu.org] Sent: Monday, September 30, 2013 4:47 PM To: Justin Krejci Cc: outages@outages.org Subject: Re: [outages] Level3 ICMP Filtering in Chicago?

It could be the return path that is blocking ICMP echo-reply (i.e. destination receives ICMP echo, responds with ICMP echo-reply, but goes out a path/interface that has it filtered).

If you can get on to 204.11.209.66 and capture ICMP (both directions) you should be able to determine if it's even seeing the echo.

L3 and Telia both have looking glasses:

http://lookingglass.level3.net/ http://looking-glass.telia.net/

-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |

On Mon, Sep 30, 2013 at 09:07:00PM +0000, Justin Krejci wrote:

...

At about 3:30PM central time today we started seeing ICMP packet loss going into Level3 networks, we believe it is likely some sort of Level3 filtering of some sort.

Two traces below from the same origin to the same destination at the same time, one is ICMP and one is UDP.

It is not just traceroute ICMP packets, we cannot ping our own equipment when it takes a Level3 path. Other UDP/TCP traffic so far seems unaffected.

Is anyone else seeing this? This appears to be a Chicago-area connection.

Using ICMP to trace to:

204.11.209.66

Tracing to host 204.11.209.66

traceroute to 204.11.209.66 (204.11.209.66), 30 hops max, 60 byte packets 1 216.17.34.1 (216.17.34.1) [as10242/AS10242] 0.558 ms 0.554 ms 0.552 ms 2 v103.usi-cr02-mpls.usinternet.com (216.17.36.25) [as10242/AS10242] 1.094 ms 1.199 ms 1.555 ms 3 10gigabitethernet2-6.core1.msp1.he.net (216.66.73.173) [AS6939] 1.203 ms 1.530 ms 1.533 ms 4 100gigabitethernet7-1.core1.chi1.he.net (184.105.223.177) [AS46841] 9.076 ms 9.084 ms 9.154 ms 5 chi-bb1-link.telia.net (213.248.104.213) [AS1299] 9.155 ms 9.163 ms 9.163 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * *

Using UDP to trace to:

204.11.209.66

Tracing to host 204.11.209.66

traceroute to 204.11.209.66 (204.11.209.66), 30 hops max, 60 byte packets 1 216.17.34.1 (216.17.34.1) [as10242/AS10242] 0.438 ms 0.480 ms 0.495 ms 2 v103.usi-cr02-mpls.usinternet.com (216.17.36.25) [as10242/AS10242] 1.106 ms 1.166 ms 1.235 ms 3 10gigabitethernet2-6.core1.msp1.he.net (216.66.73.173) [AS6939] 4.447 ms 4.445 ms 4.440 ms 4 100gigabitethernet7-1.core1.chi1.he.net (184.105.223.177) [AS46841] 12.821 ms 12.898 ms 12.893 ms 5 chi-bb1-link.telia.net (213.248.104.213) [AS1299] 9.056 ms 9.052 ms 9.054 ms 6 level3-ic-300135-chi-bb1.c.telia.net (213.248.87.238) [AS1299] 23.943 ms 24.000 ms 18.147 ms 7 ae-32-52.ebr2.Chicago1.Level3.net (4.69.138.62) [AS3356] 20.890 ms 18.162 ms 18.189 ms 8 ae-5-5.ebr2.Chicago2.Level3.net (4.69.140.194) [AS3356] 20.993 ms 20.982 ms 22.160 ms 9 ae-22-52.car2.Chicago2.Level3.net (4.69.138.165) [AS3356] 18.164 ms 22.165 ms 18.231 ms 10 RED-ANVIL-L.car2.Chicago2.Level3.net (4.30.14.162) [AS3356] 44.853 ms 44.822 ms 44.810 ms 11 ge-1-0-1-0.cr1.noc.ip.redanvil.net (204.15.100.173) [AS33693] 19.849 ms 19.888 ms 19.797 ms

...

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages