seeing large amounts of packet loss between from Level 3 in Washington, DC. Anyone else? Snippet of an MTR: 9.|-- 9-1-21.bear2.Washington111.Level3.net 0.0% 10 1.5 1.9 1.3 4.4 0.8 10.|-- ae-1-3502.ear2.Washington12.Level3.net 90.0% 10 1.9 1.9 1.9 1.9 0.0 11.|-- ix-ae-36-0.tcore1.aeq-ashburn.as6453.net 0.0% 10 15.4 3.5 1.5 15.4 4.4 12.|-- if-ae-2-2.tcore2.aeq-ashburn.as6453.net 0.0% 10 209.4 209.6 209.4 210.2 0.0 13.|-- if-ae-12-4.tcore4.njy-newark.as6453.net 0.0% 10 210.1 210.2 209.9 210.7 0.0 14.|-- if-ae-1-3.tcore3.njy-newark.as6453.net 0.0% 10 209.0 209.0 208.9 209.3 0.0 15.|-- if-ae-15-2.tcore1.l78-london.as6453.net 0.0% 10 208.9 209.1 208.9 209.5 0.0 16.|-- if-ae-3-2.tcore1.pye-paris.as6453.net 0.0% 10 209.3 209.5 209.3 210.4 0.0 17.|-- if-ae-8-1600.tcore1.wyn-marseille.as6453.net 0.0% 10 200.6 201.9 200.4 213.4 4.0 18.|-- if-ae-5-6.tcore1.mlv-mumbai.as6453.net 90.0% 10 211.0 211.0 211.0 211.0 0.0 19.|-- if-ae-2-2.tcore2.mlv-mumbai.as6453.net 0.0% 10 203.6 203.8 203.5 204.1 0.0 20.|-- if-ae-9-2.tcore2.cxr-chennai.as6453.net 0.0% 10 203.6 205.6 203.1 225.6 7.0
Read this https://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N... On Fri, May 10, 2019 at 8:36 PM Justin Kinney via Outages <outages@outages.org> wrote:
seeing large amounts of packet loss between from Level 3 in Washington, DC. Anyone else?
Snippet of an MTR:
9.|-- 9-1-21.bear2.Washington111.Level3.net 0.0% 10 1.5 1.9 1.3 4.4 0.8 10.|-- ae-1-3502.ear2.Washington12.Level3.net 90.0% 10 1.9 1.9 1.9 1.9 0.0 11.|-- ix-ae-36-0.tcore1.aeq-ashburn.as6453.net 0.0% 10 15.4 3.5 1.5 15.4 4.4 12.|-- if-ae-2-2.tcore2.aeq-ashburn.as6453.net 0.0% 10 209.4 209.6 209.4 210.2 0.0 13.|-- if-ae-12-4.tcore4.njy-newark.as6453.net 0.0% 10 210.1 210.2 209.9 210.7 0.0 14.|-- if-ae-1-3.tcore3.njy-newark.as6453.net 0.0% 10 209.0 209.0 208.9 209.3 0.0 15.|-- if-ae-15-2.tcore1.l78-london.as6453.net 0.0% 10 208.9 209.1 208.9 209.5 0.0 16.|-- if-ae-3-2.tcore1.pye-paris.as6453.net 0.0% 10 209.3 209.5 209.3 210.4 0.0 17.|-- if-ae-8-1600.tcore1.wyn-marseille.as6453.net 0.0% 10 200.6 201.9 200.4 213.4 4.0 18.|-- if-ae-5-6.tcore1.mlv-mumbai.as6453.net 90.0% 10 211.0 211.0 211.0 211.0 0.0 19.|-- if-ae-2-2.tcore2.mlv-mumbai.as6453.net 0.0% 10 203.6 203.8 203.5 204.1 0.0 20.|-- if-ae-9-2.tcore2.cxr-chennai.as6453.net 0.0% 10 203.6 205.6 203.1 225.6 7.0 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Hi Justin, What I think Ryan is getting at is that packet loss at a single hop just means that hop is rate-limiting ICMP. It's not a cause for concern. If you see cascading loss (loss starting at one hop and persisting through all subsequent hops), then that *is* a cause for concern. FWIW I checked my monitoring to a target in Level3's network in DC and I don't see any issues. -Ross On Fri, May 10, 2019 at 9:47 PM Ryan Kearney via Outages < outages@outages.org> wrote:
Read this
https://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N...
On Fri, May 10, 2019 at 8:36 PM Justin Kinney via Outages <outages@outages.org> wrote:
seeing large amounts of packet loss between from Level 3 in Washington,
DC. Anyone else?
Snippet of an MTR:
9.|-- 9-1-21.bear2.Washington111.Level3.net 0.0% 10
1.5 1.9 1.3 4.4 0.8
10.|-- ae-1-3502.ear2.Washington12.Level3.net 90.0% 10 1.9 1.9 1.9 1.9 0.0 11.|-- ix-ae-36-0.tcore1.aeq-ashburn.as6453.net 0.0% 10 15.4 3.5 1.5 15.4 4.4 12.|-- if-ae-2-2.tcore2.aeq-ashburn.as6453.net 0.0% 10 209.4 209.6 209.4 210.2 0.0 13.|-- if-ae-12-4.tcore4.njy-newark.as6453.net 0.0% 10 210.1 210.2 209.9 210.7 0.0 14.|-- if-ae-1-3.tcore3.njy-newark.as6453.net 0.0% 10 209.0 209.0 208.9 209.3 0.0 15.|-- if-ae-15-2.tcore1.l78-london.as6453.net 0.0% 10 208.9 209.1 208.9 209.5 0.0 16.|-- if-ae-3-2.tcore1.pye-paris.as6453.net 0.0% 10 209.3 209.5 209.3 210.4 0.0 17.|-- if-ae-8-1600.tcore1.wyn-marseille.as6453.net 0.0% 10 200.6 201.9 200.4 213.4 4.0 18.|-- if-ae-5-6.tcore1.mlv-mumbai.as6453.net 90.0% 10 211.0 211.0 211.0 211.0 0.0 19.|-- if-ae-2-2.tcore2.mlv-mumbai.as6453.net 0.0% 10 203.6 203.8 203.5 204.1 0.0 20.|-- if-ae-9-2.tcore2.cxr-chennai.as6453.net 0.0% 10 203.6 205.6 203.1 225.6 7.0 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
thank you both! the pdf Ryan linked is excellent, and the extra detail is much appreciated. On Fri, May 10, 2019 at 6:51 PM Ross Tajvar <ross@tajvar.io> wrote:
Hi Justin,
What I think Ryan is getting at is that packet loss at a single hop just means that hop is rate-limiting ICMP. It's not a cause for concern. If you see cascading loss (loss starting at one hop and persisting through all subsequent hops), then that *is* a cause for concern.
FWIW I checked my monitoring to a target in Level3's network in DC and I don't see any issues.
-Ross
On Fri, May 10, 2019 at 9:47 PM Ryan Kearney via Outages < outages@outages.org> wrote:
Read this
https://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N...
On Fri, May 10, 2019 at 8:36 PM Justin Kinney via Outages <outages@outages.org> wrote:
seeing large amounts of packet loss between from Level 3 in Washington,
DC. Anyone else?
Snippet of an MTR:
9.|-- 9-1-21.bear2.Washington111.Level3.net 0.0% 10
1.5 1.9 1.3 4.4 0.8
10.|-- ae-1-3502.ear2.Washington12.Level3.net 90.0% 10 1.9 1.9 1.9 1.9 0.0 11.|-- ix-ae-36-0.tcore1.aeq-ashburn.as6453.net 0.0% 10 15.4 3.5 1.5 15.4 4.4 12.|-- if-ae-2-2.tcore2.aeq-ashburn.as6453.net 0.0% 10 209.4 209.6 209.4 210.2 0.0 13.|-- if-ae-12-4.tcore4.njy-newark.as6453.net 0.0% 10 210.1 210.2 209.9 210.7 0.0 14.|-- if-ae-1-3.tcore3.njy-newark.as6453.net 0.0% 10 209.0 209.0 208.9 209.3 0.0 15.|-- if-ae-15-2.tcore1.l78-london.as6453.net 0.0% 10 208.9 209.1 208.9 209.5 0.0 16.|-- if-ae-3-2.tcore1.pye-paris.as6453.net 0.0% 10 209.3 209.5 209.3 210.4 0.0 17.|-- if-ae-8-1600.tcore1.wyn-marseille.as6453.net 0.0% 10 200.6 201.9 200.4 213.4 4.0 18.|-- if-ae-5-6.tcore1.mlv-mumbai.as6453.net 90.0% 10 211.0 211.0 211.0 211.0 0.0 19.|-- if-ae-2-2.tcore2.mlv-mumbai.as6453.net 0.0% 10 203.6 203.8 203.5 204.1 0.0 20.|-- if-ae-9-2.tcore2.cxr-chennai.as6453.net 0.0% 10 203.6 205.6 203.1 225.6 7.0 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Anyone on this list who hasn't read this needs to. I used to work break/fix at Level 3. I cannot tell you how many times otherwise smart engineers misunderstood or misread traceroute results. Seems like PingPlotter results are generally used as the "gotcha" evidence. Most ISPs rate limit and full on block ICMP. Read this
https://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N...
It's not so much that they block ICMP (tho some do), it's more that the router does not prioritize processing packets with an expired TTL. Doing so requires generating an ICMP TTL Expired packet in reply. The original packet could be anything. Many traceroutes use UDP or ICMP and have options for TCP as the initial packet. My favorite traceroutes as problem reports are the ones in a word document where they've pasted a screenshot of a command shell. On Sat, May 11, 2019 at 6:47 AM Mike Bolitho via Outages < outages@outages.org> wrote:
Anyone on this list who hasn't read this needs to. I used to work break/fix at Level 3. I cannot tell you how many times otherwise smart engineers misunderstood or misread traceroute results. Seems like PingPlotter results are generally used as the "gotcha" evidence. Most ISPs rate limit and full on block ICMP.
Read this
https://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N...
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (5)
-
Justin Kinney -
Mike Bolitho -
Nimrod Levy -
Ross Tajvar -
Ryan Kearney