VPN issues over Spectrum L3 boundaries
We have several offices over the Ohio and Pennsylvania area that are experiencing issues passing traffic over VPN tunnels (specifically, there is always a Spectrum >< Level 3 interconnect). It is a very strange issue. The VPN tunnel will actually establish, and if you source your ping from inside the internal network across the VPN tunnel to the destination, the traffic gets there and replies, but the replies never make it back to the original sending point. Anyone else experiencing any similar issues like this? Best Regards, Josh This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com immediately and permanently delete this email and any attachments.
Found a thread in the Spectrum forums talking about the issue finally - it was marked as resolved so I started a new one. https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-... From: Outages <outages-bounces@outages.org> On Behalf Of Biddle, Josh via Outages Sent: Sunday, October 13, 2019 12:00 PM To: outages@outages.org Subject: [outages] VPN issues over Spectrum L3 boundaries We have several offices over the Ohio and Pennsylvania area that are experiencing issues passing traffic over VPN tunnels (specifically, there is always a Spectrum >< Level 3 interconnect). It is a very strange issue. The VPN tunnel will actually establish, and if you source your ping from inside the internal network across the VPN tunnel to the destination, the traffic gets there and replies, but the replies never make it back to the original sending point. Anyone else experiencing any similar issues like this? Best Regards, Josh This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com<mailto:compliance@NTST.com> immediately and permanently delete this email and any attachments. This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com immediately and permanently delete this email and any attachments.
We had a similar issue last week that we chalked up to a Spectrum outage. Because this was all new install we have not gone back and tested again yet but very similar to you - Multiple sites over Ohio. VPN would establish and one side would send traffic and it would be received on the other end. The other side would send traffic and it would not be received. Justin
On Oct 15, 2019, at 8:04 AM, Biddle, Josh via Outages <outages@outages.org> wrote:
Found a thread in the Spectrum forums talking about the issue finally – it was marked as resolved so I started a new one.
https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-... <https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-7-162/m-p/164091#M53497>
From: Outages <outages-bounces@outages.org <mailto:outages-bounces@outages.org>> On Behalf Of Biddle, Josh via Outages Sent: Sunday, October 13, 2019 12:00 PM To: outages@outages.org <mailto:outages@outages.org> Subject: [outages] VPN issues over Spectrum L3 boundaries
We have several offices over the Ohio and Pennsylvania area that are experiencing issues passing traffic over VPN tunnels (specifically, there is always a Spectrum >< Level 3 interconnect). It is a very strange issue. The VPN tunnel will actually establish, and if you source your ping from inside the internal network across the VPN tunnel to the destination, the traffic gets there and replies, but the replies never make it back to the original sending point.
Anyone else experiencing any similar issues like this?
Best Regards, Josh
This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com <mailto:compliance@NTST.com> immediately and permanently delete this email and any attachments. This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com <mailto:compliance@NTST.com> immediately and permanently delete this email and any attachments. _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
See if you can get proof with traceroutes and post on the thread that I made to see if we can get some type of answer out of someone. My best guess at this time is that it is some business squabble at the transit provider where these two Internet providers interconnect. From: Justin Oeder <justin.oeder@beyondhosting.net> Sent: Tuesday, October 15, 2019 8:51 AM To: Biddle, Josh <JBiddle@ntst.com> Cc: outages@outages.org Subject: Re: [outages] VPN issues over Spectrum L3 boundaries We had a similar issue last week that we chalked up to a Spectrum outage. Because this was all new install we have not gone back and tested again yet but very similar to you - Multiple sites over Ohio. VPN would establish and one side would send traffic and it would be received on the other end. The other side would send traffic and it would not be received. Justin On Oct 15, 2019, at 8:04 AM, Biddle, Josh via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Found a thread in the Spectrum forums talking about the issue finally – it was marked as resolved so I started a new one. https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-7-162/m-p/164091#M53497<https://urldefense.proofpoint.com/v2/url?u=https-3A__forums.timewarnercable.com_t5_Connectivity_Traffic-2Dissues-2Dat-2D66-2D109-2D7-2D162_m-2Dp_164091-23M53497&d=DwMFaQ&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=iQLepJM5vHmCTqpsc5_QOKPNiy5wXujKdcHvm7JWg0k&s=RUIXDoc0T_bmNFkrtsGtYl5C5cVvjOWGsqTFkuO39dE&e=> From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Biddle, Josh via Outages Sent: Sunday, October 13, 2019 12:00 PM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] VPN issues over Spectrum L3 boundaries We have several offices over the Ohio and Pennsylvania area that are experiencing issues passing traffic over VPN tunnels (specifically, there is always a Spectrum >< Level 3 interconnect). It is a very strange issue. The VPN tunnel will actually establish, and if you source your ping from inside the internal network across the VPN tunnel to the destination, the traffic gets there and replies, but the replies never make it back to the original sending point. Anyone else experiencing any similar issues like this? Best Regards, Josh This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com<mailto:compliance@NTST.com> immediately and permanently delete this email and any attachments. This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com<mailto:compliance@NTST.com> immediately and permanently delete this email and any attachments. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages<https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwMFaQ&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=iQLepJM5vHmCTqpsc5_QOKPNiy5wXujKdcHvm7JWg0k&s=2be7j5W6TaOMjgT-Wm0C5ThoYvwYYbN7BHQUklMyQdc&e=> This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com immediately and permanently delete this email and any attachments.
Sounds more like a technical issue such as a tunnel with lower MTU. So your signalling works but the tunnel and data doest get established. I would very much doubt a commercial problem caused someone to randomly implement filters. You ought to be able to test it tho by seeing if the relevant ports are open in either direction and if ping/no fragment works at the maximum tunnel MTU. Hth On Wed, 16 Oct 2019, 00:33 Biddle, Josh via Outages, <outages@outages.org> wrote:
See if you can get proof with traceroutes and post on the thread that I made to see if we can get some type of answer out of someone. My best guess at this time is that it is some business squabble at the transit provider where these two Internet providers interconnect.
*From:* Justin Oeder <justin.oeder@beyondhosting.net> *Sent:* Tuesday, October 15, 2019 8:51 AM *To:* Biddle, Josh <JBiddle@ntst.com> *Cc:* outages@outages.org *Subject:* Re: [outages] VPN issues over Spectrum L3 boundaries
We had a similar issue last week that we chalked up to a Spectrum outage.
Because this was all new install we have not gone back and tested again yet but very similar to you - Multiple sites over Ohio. VPN would establish and one side would send traffic and it would be received on the other end. The other side would send traffic and it would not be received.
Justin
On Oct 15, 2019, at 8:04 AM, Biddle, Josh via Outages <outages@outages.org> wrote:
Found a thread in the Spectrum forums talking about the issue finally – it was marked as resolved so I started a new one.
https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-... <https://urldefense.proofpoint.com/v2/url?u=https-3A__forums.timewarnercable.com_t5_Connectivity_Traffic-2Dissues-2Dat-2D66-2D109-2D7-2D162_m-2Dp_164091-23M53497&d=DwMFaQ&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=iQLepJM5vHmCTqpsc5_QOKPNiy5wXujKdcHvm7JWg0k&s=RUIXDoc0T_bmNFkrtsGtYl5C5cVvjOWGsqTFkuO39dE&e=>
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Biddle, Josh via Outages *Sent:* Sunday, October 13, 2019 12:00 PM *To:* outages@outages.org *Subject:* [outages] VPN issues over Spectrum L3 boundaries
We have several offices over the Ohio and Pennsylvania area that are experiencing issues passing traffic over VPN tunnels (specifically, there is always a Spectrum >< Level 3 interconnect). It is a very strange issue. The VPN tunnel will actually establish, and if you source your ping from inside the internal network across the VPN tunnel to the destination, the traffic gets there and replies, but the replies never make it back to the original sending point.
Anyone else experiencing any similar issues like this?
Best Regards,
*Josh*
This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com immediately and permanently delete this email and any attachments.
This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com immediately and permanently delete this email and any attachments. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages <https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwMFaQ&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=iQLepJM5vHmCTqpsc5_QOKPNiy5wXujKdcHvm7JWg0k&s=2be7j5W6TaOMjgT-Wm0C5ThoYvwYYbN7BHQUklMyQdc&e=>
This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com immediately and permanently delete this email and any attachments. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
The original issue began 2-3 months ago at one client office. This past week it has spread to two clients over at least 6 different offices. Any other thoughts? From: Stephen Wilcox <steve.wilcox@ixreach.com> Sent: Tuesday, October 15, 2019 4:38 PM To: Biddle, Josh <JBiddle@ntst.com> Cc: Justin Oeder <justin.oeder@beyondhosting.net>; outages@outages.org Subject: Re: [outages] VPN issues over Spectrum L3 boundaries Sounds more like a technical issue such as a tunnel with lower MTU. So your signalling works but the tunnel and data doest get established. I would very much doubt a commercial problem caused someone to randomly implement filters. You ought to be able to test it tho by seeing if the relevant ports are open in either direction and if ping/no fragment works at the maximum tunnel MTU. Hth On Wed, 16 Oct 2019, 00:33 Biddle, Josh via Outages, <outages@outages.org<mailto:outages@outages.org>> wrote: See if you can get proof with traceroutes and post on the thread that I made to see if we can get some type of answer out of someone. My best guess at this time is that it is some business squabble at the transit provider where these two Internet providers interconnect. From: Justin Oeder <justin.oeder@beyondhosting.net<mailto:justin.oeder@beyondhosting.net>> Sent: Tuesday, October 15, 2019 8:51 AM To: Biddle, Josh <JBiddle@ntst.com<mailto:JBiddle@ntst.com>> Cc: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] VPN issues over Spectrum L3 boundaries We had a similar issue last week that we chalked up to a Spectrum outage. Because this was all new install we have not gone back and tested again yet but very similar to you - Multiple sites over Ohio. VPN would establish and one side would send traffic and it would be received on the other end. The other side would send traffic and it would not be received. Justin On Oct 15, 2019, at 8:04 AM, Biddle, Josh via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Found a thread in the Spectrum forums talking about the issue finally – it was marked as resolved so I started a new one. https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-7-162/m-p/164091#M53497<https://urldefense.proofpoint.com/v2/url?u=https-3A__forums.timewarnercable.com_t5_Connectivity_Traffic-2Dissues-2Dat-2D66-2D109-2D7-2D162_m-2Dp_164091-23M53497&d=DwMFaQ&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=iQLepJM5vHmCTqpsc5_QOKPNiy5wXujKdcHvm7JWg0k&s=RUIXDoc0T_bmNFkrtsGtYl5C5cVvjOWGsqTFkuO39dE&e=> From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Biddle, Josh via Outages Sent: Sunday, October 13, 2019 12:00 PM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] VPN issues over Spectrum L3 boundaries We have several offices over the Ohio and Pennsylvania area that are experiencing issues passing traffic over VPN tunnels (specifically, there is always a Spectrum >< Level 3 interconnect). It is a very strange issue. The VPN tunnel will actually establish, and if you source your ping from inside the internal network across the VPN tunnel to the destination, the traffic gets there and replies, but the replies never make it back to the original sending point. Anyone else experiencing any similar issues like this? Best Regards, Josh This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com<mailto:compliance@NTST.com> immediately and permanently delete this email and any attachments. This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com<mailto:compliance@NTST.com> immediately and permanently delete this email and any attachments. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages<https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwMFaQ&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=iQLepJM5vHmCTqpsc5_QOKPNiy5wXujKdcHvm7JWg0k&s=2be7j5W6TaOMjgT-Wm0C5ThoYvwYYbN7BHQUklMyQdc&e=> This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com<mailto:compliance@NTST.com> immediately and permanently delete this email and any attachments. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages<https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwMFaQ&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=p9pDGllAuH7XysxTdZgskh5NId7UuP2Nsa9hbEZh-rQ&s=6BXmu4K5MpMz8Qk1HOWepbDeauFt-Fq3YUQa4qOYHN0&e=> This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com immediately and permanently delete this email and any attachments.
A customer of mine has had the same issue with a TW-connected site in Ohio and another in PA. One VPN tunnel works fine, the other has one-way ISAKMP traffic to the other head-end, which connects to Level3. A traceroute shows the failing path includes 66.109.7.162. The failing direction is from the PA/OH sites toward the L3 head end. Full-size pings work fine. It's the UDP/500 that vanishes. I.E., it has nothing to do with MTU. Both started having the issue around Aug 26. Mysteriously, every week to 10 days, the broken path will start working for a while. This is usually shortly after midnight EDT; they go back down 1-3 hours later and stay down. The log entries for the two sites match within seconds. The customer's contract is with Comcast Business so it's been difficult to get to someone clueful about this symptom in TW. -Marty On 10/15/2019 8:04 AM, Biddle, Josh via Outages wrote:
Found a thread in the Spectrum forums talking about the issue finally – it was marked as resolved so I started a new one.
https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-...
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Biddle, Josh via Outages *Sent:* Sunday, October 13, 2019 12:00 PM *To:* outages@outages.org *Subject:* [outages] VPN issues over Spectrum L3 boundaries
We have several offices over the Ohio and Pennsylvania area that are experiencing issues passing traffic over VPN tunnels (specifically, there is always a Spectrum >< Level 3 interconnect). It is a very strange issue. The VPN tunnel will actually establish, and if you source your ping from inside the internal network across the VPN tunnel to the destination, the traffic gets there and replies, but the replies never make it back to the original sending point.
Anyone else experiencing any similar issues like this?
Best Regards,
*Josh*
Our issues have recently magically resolved (last Thursday 10/17) due to L3 vanishing from the hop list. Our traffic now disappears into ntt.net and we are seeing two way IPsec traffic without any issues. Marty, did your issues resolve? [cid:image001.png@01D58815.06155610] From: Marty Adkins <marty@martyadkins.com> Sent: Friday, October 18, 2019 4:01 PM To: Biddle, Josh <JBiddle@ntst.com>; outages@outages.org Subject: Re: [outages] VPN issues over Spectrum L3 boundaries A customer of mine has had the same issue with a TW-connected site in Ohio and another in PA. One VPN tunnel works fine, the other has one-way ISAKMP traffic to the other head-end, which connects to Level3. A traceroute shows the failing path includes 66.109.7.162. The failing direction is from the PA/OH sites toward the L3 head end. Full-size pings work fine. It's the UDP/500 that vanishes. I.E., it has nothing to do with MTU. Both started having the issue around Aug 26. Mysteriously, every week to 10 days, the broken path will start working for a while. This is usually shortly after midnight EDT; they go back down 1-3 hours later and stay down. The log entries for the two sites match within seconds. The customer's contract is with Comcast Business so it's been difficult to get to someone clueful about this symptom in TW. -Marty On 10/15/2019 8:04 AM, Biddle, Josh via Outages wrote: Found a thread in the Spectrum forums talking about the issue finally - it was marked as resolved so I started a new one. https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-7-162/m-p/164091#M53497<https://urldefense.proofpoint.com/v2/url?u=https-3A__forums.timewarnercable.com_t5_Connectivity_Traffic-2Dissues-2Dat-2D66-2D109-2D7-2D162_m-2Dp_164091-23M53497&d=DwMD-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=pXHvd5iI_J5DVYGMDKBSdUZl1iyilZUvR3oyvE3BJ0E&s=_90gCNY2Ln1XdqWtMNguWIRejhsdUxoLyvgTpo-R5jM&e=> From: Outages <outages-bounces@outages.org><mailto:outages-bounces@outages.org> On Behalf Of Biddle, Josh via Outages Sent: Sunday, October 13, 2019 12:00 PM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] VPN issues over Spectrum L3 boundaries We have several offices over the Ohio and Pennsylvania area that are experiencing issues passing traffic over VPN tunnels (specifically, there is always a Spectrum >< Level 3 interconnect). It is a very strange issue. The VPN tunnel will actually establish, and if you source your ping from inside the internal network across the VPN tunnel to the destination, the traffic gets there and replies, but the replies never make it back to the original sending point. Anyone else experiencing any similar issues like this? Best Regards, Josh This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com immediately and permanently delete this email and any attachments.
I had a recent issue that was similar to this. In that case it was a DDOS signature update on a specific vendors' DDOS scrubber at the host site that was the problem. The specific tunnel src/dst flow would be dropped, however all other traffic between endpoints was allowed. Because all the traffic for that specific flow hit a threshold above the rule and the IP's in question were flagged low enough in reputation to be fully inspected, it was blocked. After whitelisting the IP's in the DDOS solution, all VPN traffic worked fine. Just a thought. ________________________________ From: Outages <outages-bounces@outages.org> on behalf of Biddle, Josh via Outages <outages@outages.org> Sent: Monday, October 21, 2019 12:39 PM To: Marty Adkins <marty@martyadkins.com>; outages@outages.org <outages@outages.org> Cc: D L <route2null0@yahoo.com>; Cullis, Ben <BCullis@ntst.com>; Cochran, Brian <BCochran@ntst.com> Subject: Re: [outages] VPN issues over Spectrum L3 boundaries Our issues have recently magically resolved (last Thursday 10/17) due to L3 vanishing from the hop list. Our traffic now disappears into ntt.net and we are seeing two way IPsec traffic without any issues. Marty, did your issues resolve? [cid:image001.png@01D58815.06155610] From: Marty Adkins <marty@martyadkins.com> Sent: Friday, October 18, 2019 4:01 PM To: Biddle, Josh <JBiddle@ntst.com>; outages@outages.org Subject: Re: [outages] VPN issues over Spectrum L3 boundaries A customer of mine has had the same issue with a TW-connected site in Ohio and another in PA. One VPN tunnel works fine, the other has one-way ISAKMP traffic to the other head-end, which connects to Level3. A traceroute shows the failing path includes 66.109.7.162. The failing direction is from the PA/OH sites toward the L3 head end. Full-size pings work fine. It's the UDP/500 that vanishes. I.E., it has nothing to do with MTU. Both started having the issue around Aug 26. Mysteriously, every week to 10 days, the broken path will start working for a while. This is usually shortly after midnight EDT; they go back down 1-3 hours later and stay down. The log entries for the two sites match within seconds. The customer's contract is with Comcast Business so it's been difficult to get to someone clueful about this symptom in TW. -Marty On 10/15/2019 8:04 AM, Biddle, Josh via Outages wrote: Found a thread in the Spectrum forums talking about the issue finally – it was marked as resolved so I started a new one. https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-7-162/m-p/164091#M53497<https://urldefense.proofpoint.com/v2/url?u=https-3A__forums.timewarnercable.com_t5_Connectivity_Traffic-2Dissues-2Dat-2D66-2D109-2D7-2D162_m-2Dp_164091-23M53497&d=DwMD-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=pXHvd5iI_J5DVYGMDKBSdUZl1iyilZUvR3oyvE3BJ0E&s=_90gCNY2Ln1XdqWtMNguWIRejhsdUxoLyvgTpo-R5jM&e=> From: Outages <outages-bounces@outages.org><mailto:outages-bounces@outages.org> On Behalf Of Biddle, Josh via Outages Sent: Sunday, October 13, 2019 12:00 PM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] VPN issues over Spectrum L3 boundaries We have several offices over the Ohio and Pennsylvania area that are experiencing issues passing traffic over VPN tunnels (specifically, there is always a Spectrum >< Level 3 interconnect). It is a very strange issue. The VPN tunnel will actually establish, and if you source your ping from inside the internal network across the VPN tunnel to the destination, the traffic gets there and replies, but the replies never make it back to the original sending point. Anyone else experiencing any similar issues like this? Best Regards, Josh This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com immediately and permanently delete this email and any attachments.
No change for them because the one head-end is fed by L3, so there's no way it can vanish from the path. :) On 10/21/2019 1:39 PM, Biddle, Josh wrote:
Our issues have recently magically resolved (last Thursday 10/17) due to L3 vanishing from the hop list. Our traffic now disappears into ntt.net and we are seeing two way IPsec traffic without any issues.
Marty, did your issues resolve?
*From:* Marty Adkins <marty@martyadkins.com> *Sent:* Friday, October 18, 2019 4:01 PM *To:* Biddle, Josh <JBiddle@ntst.com>; outages@outages.org *Subject:* Re: [outages] VPN issues over Spectrum L3 boundaries
A customer of mine has had the same issue with a TW-connected site in Ohio and another in PA. One VPN tunnel works fine, the other has one-way ISAKMP traffic to the other head-end, which connects to Level3. A traceroute shows the failing path includes 66.109.7.162. The failing direction is from the PA/OH sites toward the L3 head end. Full-size pings work fine. It's the UDP/500 that vanishes. I.E., it has nothing to do with MTU.
Both started having the issue around Aug 26. Mysteriously, every week to 10 days, the broken path will start working for a while. This is usually shortly after midnight EDT; they go back down 1-3 hours later and stay down. The log entries for the two sites match within seconds.
The customer's contract is with Comcast Business so it's been difficult to get to someone clueful about this symptom in TW.
-Marty
On 10/15/2019 8:04 AM, Biddle, Josh via Outages wrote:
Found a thread in the Spectrum forums talking about the issue finally – it was marked as resolved so I started a new one.
https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-... <https://urldefense.proofpoint.com/v2/url?u=https-3A__forums.timewarnercable.com_t5_Connectivity_Traffic-2Dissues-2Dat-2D66-2D109-2D7-2D162_m-2Dp_164091-23M53497&d=DwMD-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=pXHvd5iI_J5DVYGMDKBSdUZl1iyilZUvR3oyvE3BJ0E&s=_90gCNY2Ln1XdqWtMNguWIRejhsdUxoLyvgTpo-R5jM&e=>
*From:* Outages <outages-bounces@outages.org> <mailto:outages-bounces@outages.org> *On Behalf Of *Biddle, Josh via Outages *Sent:* Sunday, October 13, 2019 12:00 PM *To:* outages@outages.org <mailto:outages@outages.org> *Subject:* [outages] VPN issues over Spectrum L3 boundaries
We have several offices over the Ohio and Pennsylvania area that are experiencing issues passing traffic over VPN tunnels (specifically, there is always a Spectrum >< Level 3 interconnect). It is a very strange issue. The VPN tunnel will actually establish, and if you source your ping from inside the internal network across the VPN tunnel to the destination, the traffic gets there and replies, but the replies never make it back to the original sending point.
Anyone else experiencing any similar issues like this?
Best Regards,
*Josh*
After getting the right TW/Spectrum folks involved, this was solved by a routing change. Traffic from OH and PA sites now traverse an L3 peering point at Newark, and the NC site now goes via Atlanta. The problematic one-way path peered in D.C. and a TW team is still working to resolve whether that's their issue or within L3. Supposedly the routing changes that were made on Oct 22 would affect other customers as well. -Marty On 10/21/2019 2:19 PM, Marty Adkins wrote:
No change for them because the one head-end is fed by L3, so there's no way it can vanish from the path. :)
On 10/21/2019 1:39 PM, Biddle, Josh wrote:
Our issues have recently magically resolved (last Thursday 10/17) due to L3 vanishing from the hop list. Our traffic now disappears into ntt.net and we are seeing two way IPsec traffic without any issues.
Marty, did your issues resolve?
participants (5)
-
Biddle, Josh -
Corey Davelaar -
Justin Oeder -
Marty Adkins -
Stephen Wilcox