HE IPv6 tunnel PMTU issues with juniper.net
While on my Hurricane Electric IPv6 tunnel, I cannot access juniper.net unless I change my local interface MTU. 1500 fails, but 1280 works. I noticed this a few days ago. Before that I had no problems with a 1500 MTU. Is anyone else seeing this issue?
On 10/01/2014 01:50 PM, Chuck Anderson via Outages wrote:
While on my Hurricane Electric IPv6 tunnel, I cannot access juniper.net unless I change my local interface MTU. 1500 fails, but 1280 works. I noticed this a few days ago. Before that I had no problems with a 1500 MTU. Is anyone else seeing this issue?
No, but if your are using a 6in4 tunnel, the MTU should be 1480 (not 1500). (I just successfully went to www.juniper.net via IPv6 with that MTU 1480.)
On Wed, Oct 01, 2014 at 02:17:01PM +0000, Gary Gapinski via Outages wrote:
On 10/01/2014 01:50 PM, Chuck Anderson via Outages wrote:
While on my Hurricane Electric IPv6 tunnel, I cannot access juniper.net unless I change my local interface MTU. 1500 fails, but 1280 works. I noticed this a few days ago. Before that I had no problems with a 1500 MTU. Is anyone else seeing this issue?
No, but if your are using a 6in4 tunnel, the MTU should be 1480 (not 1500).
(I just successfully went to www.juniper.net via IPv6 with that MTU 1480.)
My tunnel router has a 1280 MTU on the henet interface: 6in4-henet Link encap:IPv6-in-IPv4 inet6 addr: 2001:470:xxxx:xxxx::2/64 Scope:Global inet6 addr: fe80::xxxx:xxxx/128 Scope:Link UP POINTOPOINT RUNNING NOARP MTU:1280 Metric:1 RX packets:17148418 errors:0 dropped:0 overruns:0 frame:0 TX packets:12347808 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2660258163 (2.4 GiB) TX bytes:2833651623 (2.6 GiB) But the LAN interface of that router has an MTU of 1500, as does my desktop system. I believe the issue is that the juniper.net web server has an MTU of 1500 and their network or somewhere along the path is blocking ICMP Packet Too Big messages that would be sent by the HE.net tunnel router. Like I said, I changed nothing on my end, and it was working before. I don't know if juniper.net just added IPv6 to their website, or if something else changed in the path.
On Wed, Oct 1, 2014, at 09:37, Chuck Anderson via Outages wrote:
On Wed, Oct 01, 2014 at 02:17:01PM +0000, Gary Gapinski via Outages wrote:
On 10/01/2014 01:50 PM, Chuck Anderson via Outages wrote:
While on my Hurricane Electric IPv6 tunnel, I cannot access juniper.net unless I change my local interface MTU. 1500 fails, but 1280 works. I noticed this a few days ago. Before that I had no problems with a 1500 MTU. Is anyone else seeing this issue?
No, but if your are using a 6in4 tunnel, the MTU should be 1480 (not 1500).
(I just successfully went to www.juniper.net via IPv6 with that MTU 1480.)
My tunnel router has a 1280 MTU on the henet interface:
6in4-henet Link encap:IPv6-in-IPv4 inet6 addr: 2001:470:xxxx:xxxx::2/64 Scope:Global inet6 addr: fe80::xxxx:xxxx/128 Scope:Link UP POINTOPOINT RUNNING NOARP MTU:1280 Metric:1 RX packets:17148418 errors:0 dropped:0 overruns:0 frame:0 TX packets:12347808 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2660258163 (2.4 GiB) TX bytes:2833651623 (2.6 GiB)
But the LAN interface of that router has an MTU of 1500, as does my desktop system. I believe the issue is that the juniper.net web server has an MTU of 1500 and their network or somewhere along the path is blocking ICMP Packet Too Big messages that would be sent by the HE.net tunnel router.
Like I said, I changed nothing on my end, and it was working before. I don't know if juniper.net just added IPv6 to their website, or if something else changed in the path.
It's nearly a requirement to lower your MTU / enable mss-clamping when doing ipv6 tunnels. It's possible some connectivity of yours was broken and you just didn't notice it until now. I had to do this on my J series and I also have to do it on my OpenBSD firewall -- # mss clamping down to 1280. 1220 + 60 for ipv6 header match on egress all scrub (random-id no-df max-mss 1220) The whole fragmentation situation with IPv6 is kind of a joke http://en.wikipedia.org/wiki/IPv6_packet#Fragmentation
On Wed, Oct 01, 2014 at 10:09:49AM -0500, Mark Felder via Outages wrote:
On Wed, Oct 1, 2014, at 09:37, Chuck Anderson via Outages wrote:
On Wed, Oct 01, 2014 at 02:17:01PM +0000, Gary Gapinski via Outages wrote:
On 10/01/2014 01:50 PM, Chuck Anderson via Outages wrote:
While on my Hurricane Electric IPv6 tunnel, I cannot access juniper.net unless I change my local interface MTU. 1500 fails, but 1280 works. I noticed this a few days ago. Before that I had no problems with a 1500 MTU. Is anyone else seeing this issue?
No, but if your are using a 6in4 tunnel, the MTU should be 1480 (not 1500).
(I just successfully went to www.juniper.net via IPv6 with that MTU 1480.)
My tunnel router has a 1280 MTU on the henet interface:
6in4-henet Link encap:IPv6-in-IPv4 inet6 addr: 2001:470:xxxx:xxxx::2/64 Scope:Global inet6 addr: fe80::xxxx:xxxx/128 Scope:Link UP POINTOPOINT RUNNING NOARP MTU:1280 Metric:1 RX packets:17148418 errors:0 dropped:0 overruns:0 frame:0 TX packets:12347808 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2660258163 (2.4 GiB) TX bytes:2833651623 (2.6 GiB)
But the LAN interface of that router has an MTU of 1500, as does my desktop system. I believe the issue is that the juniper.net web server has an MTU of 1500 and their network or somewhere along the path is blocking ICMP Packet Too Big messages that would be sent by the HE.net tunnel router.
Like I said, I changed nothing on my end, and it was working before. I don't know if juniper.net just added IPv6 to their website, or if something else changed in the path.
It's nearly a requirement to lower your MTU / enable mss-clamping when doing ipv6 tunnels. It's possible some connectivity of yours was broken and you just didn't notice it until now. I had to do this on my J series and I also have to do it on my OpenBSD firewall --
# mss clamping down to 1280. 1220 + 60 for ipv6 header match on egress all scrub (random-id no-df max-mss 1220)
The whole fragmentation situation with IPv6 is kind of a joke
I know. But I'm reporting this here on outages in the hopes that a responsible party would see it and can fix the root cause of this particular issue. E.g. stop dropping ICMP Packet Too Big.
Once upon a time, Chuck Anderson via Outages <outages@outages.org> said:
I know. But I'm reporting this here on outages in the hopes that a responsible party would see it and can fix the root cause of this particular issue. E.g. stop dropping ICMP Packet Too Big.
For me, www.juniper.net is on Akamai, so you might try there. You might also see what happens with other Akamized IPv6 sites. -- Chris Adams <cma@cmadams.net>
Hi, On Wed, Oct 01, 2014 at 10:09:49AM -0500, Mark Felder via Outages wrote:
It's nearly a requirement to lower your MTU / enable mss-clamping when doing ipv6 tunnels. It's possible some connectivity of yours was broken and you just didn't notice it until now. I had to do this on my J series and I also have to do it on my OpenBSD firewall --
Actually this is the wrong way to go about it. It's hiding the symptoms instead of fixing the problem.
The whole fragmentation situation with IPv6 is kind of a joke
This is not about fragmentation. (Indeed, the whole story about IPv6 frag handling in *BSD firewalls is too sad to think about it) gert -- USENET is *not* the non-clickable part of WWW! //www.muc.de/~gert/ Gert Doering - Munich, Germany gert@greenie.muc.de fax: +49-89-35655025 gert@net.informatik.tu-muenchen.de
On Wed, Oct 01, 2014 at 03:16:10PM +0000, Gary Gapinski via Outages wrote:
On 10/01/2014 02:37 PM, Chuck Anderson wrote:
But the LAN interface of that router has an MTU of 1500, as does my desktop system. I believe the issue is that the juniper.net web server has an MTU of 1500 and their network or somewhere along the path is blocking ICMP Packet Too Big messages that would be sent by the HE.net tunnel router.
Perhaps, though IMO unlikely.
I too have a LAN MTU of 1500, and have a (HE.net) tunnel MTU of 1480. I do not see any PMTU problems to www.juniper.net or elsewhere.
However, I am explicitly advertising MTU 1480 in LAN RAs.
Interesting. Does the 1480 MTU setting from RA show up on your interface configuration, or in a routing table entry? Is your TCP connection to www.juniper.net using an MSS of 1220? If so, then of course you wouldn't see the issue. I'm using OpenWRT for the HE.net tunnel. I'll have to see how to send the MTU in the LAN RA.
Here is a partial outbound route from my origin:
gapinski@eins:~$ traceroute -6 -A www.juniper.net traceroute to www.juniper.net (2600:1408:17:293::720), 30 hops max, 80 byte packets � 3 ge3-4.core1.chi1.he.net (2001:470:0:6e::1) [AS6939] 43.778 ms 43.892 ms 43.876 ms 4 twcable-backbone-as7843.10gigabitethernet1-1-6.switch3.chi1.he.net (2001:470:0:2bb::2) [AS6939] 55.905 ms 55.988 ms 55.983 ms 5 2001:1998:0:4::171 (2001:1998:0:4::171) [AS7843] 39.962 ms 2001:1998:0:4::ed (2001:1998:0:4::ed) [AS7843] 40.635 ms 2001:1998:0:4::171 (2001:1998:0:4::171) [AS7843] 42.125 ms 6 2001:1998:0:4::a3 (2001:1998:0:4::a3) [AS7843] 63.789 ms 49.818 ms 2001:1998:0:4::b1 (2001:1998:0:4::b1) [AS7843] 50.215 ms 7 2001:1998:0:4::80 (2001:1998:0:4::80) [AS7843] 54.505 ms 2001:1998:0:4::a4 (2001:1998:0:4::a4) [AS7843] 55.892 ms 2001:1998:0:4::80 (2001:1998:0:4::80) [AS7843] 55.006 ms 8 2001:1998:0:4::5c (2001:1998:0:4::5c) [AS7843] 57.100 ms 57.060 ms 58.693 ms 9 2001:1998:0:4::90 (2001:1998:0:4::90) [AS7843] 56.811 ms 57.152 ms 2001:1998:0:4::86 (2001:1998:0:4::86) [AS7843] 57.956 ms 10 2001:1998:0:4::f4 (2001:1998:0:4::f4) [AS7843] 55.055 ms 2001:1998:0:4::1a1 (2001:1998:0:4::1a1) [AS7843] 55.153 ms 2001:1998:0:4::f4 (2001:1998:0:4::f4) [AS7843] 55.107 ms 11 2001:1998:0:8::2c6 (2001:1998:0:8::2c6) [AS7843] 56.198 ms 56.771 ms 55.622 ms 12 2600:1408:17::17dc:947c (2600:1408:17::17dc:947c) [AS20940] 49.388 ms 59.963 ms 60.354 ms gapinski@eins:~$
Traceroute from here: traceroute to www.juniper.net (2600:141b:4:2a2::720), 30 hops max, 80 byte packets 1 2001:470:xxxx:1::1 (2001:470:xxxx:1::1) 0.327 ms 0.328 ms 0.352 ms 2 2001:470:xxxx:xxxx::1 (2001:470:xxxx:xxxx::1) 19.155 ms 22.996 ms 26.192 ms 3 ge3-8.core1.nyc4.he.net (2001:470:0:5d::1) 27.735 ms 28.878 ms 28.996 ms 4 v6-paix-ny.3-2.r1.ny.hwng.net (2001:504:f::4c) 29.094 ms 28.999 ms 29.070 ms 5 2001:4de0:1000:64::2 (2001:4de0:1000:64::2) 29.069 ms * * 6 * 2001:4de0:2001:8::2 (2001:4de0:2001:8::2) 23.048 ms 23.912 ms 7 2600:141b:4::48f7:acf (2600:141b:4::48f7:acf) 23.960 ms 16.420 ms *
On Wed, Oct 01, 2014 at 05:04:26PM +0000, Gary Gapinski via Outages wrote:
On 10/01/2014 03:27 PM, Chuck Anderson via Outages wrote:
Interesting. Does the 1480 MTU setting from RA show up on your interface configuration, or in a routing table entry?
No, and no (found no easy way to check).
The Linux "/sbin/ip" command apparently can show mtu and advmss (as seen in Google search results), but on my system it doesn't show them. cat /proc/sys/net/ipv6/conf/<interface>/mtu does show it though.
Is your TCP connection to www.juniper.net using an MSS of 1220? If so, then of course you wouldn't see the issue.
Found no easy way to check.
Given the lack of "advmss" output in "/sbin/ip -6 route show", I'm using tcpdump to verify the initial SYN packet.
However, if I were in your place, I would be watching for ICMPv6 packets coming in your direction (including from your edge router) while capturing the TCP connection startup (namely, watching for PMTU to be determined).
I am running Ubiquiti (EdgeOS), and it uses radvd. The following is what I see in /etc/radvd.conf:
interface eth2 { # This section was automatically generated by the Vyatta # configuration sub-system. Do not edit it. # # Generated by root on Sun Sep 7 13:01:05 2014 # IgnoreIfMissing on; AdvSendAdvert on; AdvOtherConfigFlag off; AdvDefaultLifetime 180; AdvLinkMTU 1480;
I did just set this on my OpenWrt router. I see from "rdisc6" (and tcpdump) that I'm now getting a Link MTU option. But "/sbin/ip" still doesn't show it anywhere. /proc/sys/net/ipv6/conf/<interface>/mtu is showing the change to 1480.
While watching a web browser conversation with www.juniper.net , I see the initial SYN with MSS of 1420 and return SYN ACK with MSS 1440.
Right. If you turn off the AdvLinkMTU option (or set it back to 1500) such that the initial SYN sets an MSS of 1440, do you then have problems reaching www.juniper.net?
On 10/01/2014 05:58 PM, Chuck Anderson via Outages wrote:
The Linux "/sbin/ip" command apparently can show mtu and advmss (as seen in Google search results), but on my system it doesn't show them. cat /proc/sys/net/ipv6/conf/<interface>/mtu does show it though.
Ditto mine (proc/sys/net/ipv6/conf/<interface>/mtu is 1480). ip does not cough up MTU.
Right. If you turn off the AdvLinkMTU option (or set it back to 1500) such that the initial SYN sets an MSS of 1440, do you then have problems reaching www.juniper.net?
No, which is a bit puzzling. WIth MTU at 1500 on LAN interface, TCP connect had outbound SYN with MSS 1420, return SYN ACK with MSS 1420. No related ICMPv6 seen. This was tried from a LAN device, not the edge router (i.e., the one with the tunnel).
On Wed, Oct 01, 2014 at 07:01:59PM +0000, Gary Gapinski via Outages wrote:
On 10/01/2014 05:58 PM, Chuck Anderson via Outages wrote:
The Linux "/sbin/ip" command apparently can show mtu and advmss (as seen in Google search results), but on my system it doesn't show them. cat /proc/sys/net/ipv6/conf/<interface>/mtu does show it though.
Ditto mine (proc/sys/net/ipv6/conf/<interface>/mtu is 1480). ip does not cough up MTU.
Right. If you turn off the AdvLinkMTU option (or set it back to 1500) such that the initial SYN sets an MSS of 1440, do you then have problems reaching www.juniper.net?
No, which is a bit puzzling. WIth MTU at 1500 on LAN interface, TCP connect had outbound SYN with MSS 1420, return SYN ACK with MSS 1420. No related ICMPv6 seen. This was tried from a LAN device, not the edge router (i.e., the one with the tunnel).
I /think/ you may have to flush your routing table after changing MTU--it seems the route cache maintains the per-destination MTU/MSS. At one point I saw some extra routes with MTU 1280 in "/sbin/ip -6 route show".
On 10/1/14 6:50 AM, Chuck Anderson via Outages wrote:
While on my Hurricane Electric IPv6 tunnel, I cannot access juniper.net unless I change my local interface MTU. 1500 fails, but 1280 works. I noticed this a few days ago. Before that I had no problems with a 1500 MTU. Is anyone else seeing this issue?
The MTU of your tunnel is lower than 1500. Chances are the service on the other end isn't able to recieve pmtud messages because it's load-balanced... and the first packet that triggers that (ptb) is sent towards your tunnel ingress. _______________________________________________
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Wed, Oct 1, 2014 at 11:14 AM, joel jaeggli via Outages <outages@outages.org> wrote:
On 10/1/14 6:50 AM, Chuck Anderson via Outages wrote:
While on my Hurricane Electric IPv6 tunnel, I cannot access juniper.net unless I change my local interface MTU. 1500 fails, but 1280 works. I noticed this a few days ago. Before that I had no problems with a 1500 MTU. Is anyone else seeing this issue?
The MTU of your tunnel is lower than 1500.
Chances are the service on the other end isn't able to recieve pmtud messages because it's load-balanced... and the first packet that triggers that (ptb) is sent towards your tunnel ingress.
Welcome to the wonders of IPv6, where fragmentation in the network isn't possible. It's better this way, trust us... W
_______________________________________________
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf
On Wed, Oct 01, 2014 at 08:14:52AM -0700, joel jaeggli via Outages wrote:
On 10/1/14 6:50 AM, Chuck Anderson via Outages wrote:
While on my Hurricane Electric IPv6 tunnel, I cannot access juniper.net unless I change my local interface MTU. 1500 fails, but 1280 works. I noticed this a few days ago. Before that I had no problems with a 1500 MTU. Is anyone else seeing this issue?
The MTU of your tunnel is lower than 1500.
Chances are the service on the other end isn't able to recieve pmtud messages because it's load-balanced... and the first packet that triggers that (ptb) is sent towards your tunnel ingress.
Chances are the first large packet is from the server end, not my client end. So if www.juniper.net sends a 1500-byte packet, it should arrive at the HE.net tunnel router and that router should send a PTB back to the server. Are you saying the PTB might not be received by the server because it is behind a load-balancer? Shouldn't the load-balancer handle the PTB directly in that case since it terminates the TCP connection?
Actually, given the nature of browsers these days, it is not at all unlikely that your request to the Juniper web server contains 1280+ octets of HTTP header data and thus is the largest packet. I think if you investigate further you will find that most flows stall at the point of a TCP packet larger than N octets payload hitting the Juniper from the LAN side and being small enough that it doesn't generate a PTB message, so it gets silently dropped. At least that's what happened in my environment before I reduced the tcp-mss as described in my earlier message. And to Warren, yes, not fragmenting in the network _IS_ actually better. Owen On Oct 1, 2014, at 09:30 , Chuck Anderson via Outages <outages@outages.org> wrote:
On Wed, Oct 01, 2014 at 08:14:52AM -0700, joel jaeggli via Outages wrote:
On 10/1/14 6:50 AM, Chuck Anderson via Outages wrote:
While on my Hurricane Electric IPv6 tunnel, I cannot access juniper.net unless I change my local interface MTU. 1500 fails, but 1280 works. I noticed this a few days ago. Before that I had no problems with a 1500 MTU. Is anyone else seeing this issue?
The MTU of your tunnel is lower than 1500.
Chances are the service on the other end isn't able to recieve pmtud messages because it's load-balanced... and the first packet that triggers that (ptb) is sent towards your tunnel ingress.
Chances are the first large packet is from the server end, not my client end. So if www.juniper.net sends a 1500-byte packet, it should arrive at the HE.net tunnel router and that router should send a PTB back to the server. Are you saying the PTB might not be received by the server because it is behind a load-balancer? Shouldn't the load-balancer handle the PTB directly in that case since it terminates the TCP connection? _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On 10/1/14 10:21 AM, Owen DeLong wrote:
Actually, given the nature of browsers these days, it is not at all unlikely that your request to the Juniper web server contains 1280+ octets of HTTP header data and thus is the largest packet. I think if you investigate further you will find that most flows stall at the point of a TCP packet larger than N octets payload hitting the Juniper from the LAN side and being small enough that it doesn't generate a PTB message, so it gets silently dropped.
if it's not generating a ptb on the tunnel ingress it's kinda broken.
At least that's what happened in my environment before I reduced the tcp-mss as described in my earlier message.
And to Warren, yes, not fragmenting in the network _IS_ actually better.
Owen
On Oct 1, 2014, at 09:30 , Chuck Anderson via Outages <outages@outages.org> wrote:
On Wed, Oct 01, 2014 at 08:14:52AM -0700, joel jaeggli via Outages wrote:
On 10/1/14 6:50 AM, Chuck Anderson via Outages wrote:
While on my Hurricane Electric IPv6 tunnel, I cannot access juniper.net unless I change my local interface MTU. 1500 fails, but 1280 works. I noticed this a few days ago. Before that I had no problems with a 1500 MTU. Is anyone else seeing this issue?
The MTU of your tunnel is lower than 1500.
Chances are the service on the other end isn't able to recieve pmtud messages because it's load-balanced... and the first packet that triggers that (ptb) is sent towards your tunnel ingress.
Chances are the first large packet is from the server end, not my client end.
yes ssl server for example is a good way to stimulate that.
So if www.juniper.net sends a 1500-byte packet, it
should arrive at the HE.net tunnel router and that router should send a PTB back to the server. Are you saying the PTB might not be received by the server because it is behind a load-balancer?
or behind a layer-3 ecmp hash in which case it may not be getting back to the machine which has the layer 3 flow terminated.
Shouldn't the load-balancer handle the PTB directly in that case since it terminates the TCP connection?
if the right device receives it then yes. https://tools.ietf.org/html/draft-jaeggli-v6ops-pmtud-ecmp-problem-01
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
This problem can be solved by setting the tcp-mss under internet-options on the juniper. I used 1280, but there may be a larger more optimal value. Owen On Oct 1, 2014, at 06:50 , Chuck Anderson via Outages <outages@outages.org> wrote:
While on my Hurricane Electric IPv6 tunnel, I cannot access juniper.net unless I change my local interface MTU. 1500 fails, but 1280 works. I noticed this a few days ago. Before that I had no problems with a 1500 MTU. Is anyone else seeing this issue? _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (8)
-
Chris Adams -
Chuck Anderson -
Gary Gapinski -
Gert Doering -
joel jaeggli -
Mark Felder -
Owen DeLong -
Warren Kumari