Spectrum/Charter IPv6 connectivity to CloudFlare down in Texas
Things have been rather slow the last couple of days, just figured out why; No IPv6 connectivity to any CloudFlare services.
mtr -6 --report-wide cloudflare.com Start: 2023-04-19T23:20:42+0000 HOST: defiant Loss% Snt Last Avg Best Wrst StDev 1.|-- 2603-8081-2340-2868-0215-17ff-fe94-6457.res6.spectrum.com 0.0% 10 3.6 19.1 1.5 93.2 36.1 2.|-- 2603-90c5-0003-0180-0000-0000-0000-0001.inf6.spectrum.com 0.0% 10 14.5 36.6 11.3 171.2 50.0 3.|-- lag-63.hcr02rdrktxaz.netops.charter.com 0.0% 10 18.8 37.3 12.0 143.9 45.3 4.|-- lag-19.ausxtxir02r.netops.charter.com 60.0% 10 65.8 30.2 11.7 65.8 24.9 5.|-- lag-22.rcr01hstqtx02.netops.charter.com 80.0% 10 26.5 25.5 24.5 26.5 1.4 6.|-- lag-16.hstqtx0209w-bcr00.netops.charter.com 90.0% 10 25.7 25.7 25.7 25.7 0.0 7.|-- lag-12.dllstx976iw-bcr00.netops.charter.com 40.0% 10 25.4 26.3 18.2 44.4 9.5 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
IPv4 works however:
mtr -4 --report-wide cloudflare.com Start: 2023-04-19T23:20:21+0000 HOST: defiant Loss% Snt Last Avg Best Wrst StDev 1.|-- _gateway 0.0% 10 2.1 1.8 1.4 2.1 0.2 2.|-- cpe-70-114-160-1.austin.res.rr.com 0.0% 10 51.9 34.4 14.3 149.7 42.1 3.|-- lag-63.hcr01rdrktxaz.netops.charter.com 0.0% 10 13.1 38.5 13.1 159.1 44.6 4.|-- lag-19.ausutxla01r.netops.charter.com 0.0% 10 18.1 21.0 14.3 34.8 6.0 5.|-- lag-22.rcr01dllatx37.netops.charter.com 0.0% 10 23.4 23.3 18.3 33.3 4.3 6.|-- lag-14.dllstx976iw-bcr00.netops.charter.com 0.0% 10 19.8 27.0 16.1 51.2 11.9 7.|-- lag-0.pr3.dfw10.netops.charter.com 0.0% 10 39.3 28.5 14.1 62.3 16.7 8.|-- 141.101.74.104 40.0% 10 53.2 44.3 19.4 120.2 39.2 9.|-- 172.71.168.2 0.0% 10 28.3 28.8 10.8 79.4 19.5 10.|-- 104.16.133.229 0.0% 10 122.0 42.8 16.9 122.0 34.4
It's kind of wild how many things don't fall back to IPv4 quickly (lots of Android TV apps) or at all (.NET). I tried reaching out to Spectrum support, but being a lowly consumer I was only told that "The modem doesn't have any filters" and that "the problem may be on the CloudFlare side", which was sadly what I predicted. So, I figured I would raise the issue here in the hopes that actual network people would see it. -Carl
Have been dealing with this for 2 days, nobody at Spectrum can figure this out. Daniel Marks May Mobility, Inc
On Apr 19, 2023, at 19:26, Carl Perry via Outages <outages@outages.org> wrote:
Things have been rather slow the last couple of days, just figured out why; No IPv6 connectivity to any CloudFlare services.
mtr -6 --report-wide cloudflare.com Start: 2023-04-19T23:20:42+0000 HOST: defiant Loss% Snt Last Avg Best Wrst StDev 1.|-- 2603-8081-2340-2868-0215-17ff-fe94-6457.res6.spectrum.com 0.0% 10 3.6 19.1 1.5 93.2 36.1 2.|-- 2603-90c5-0003-0180-0000-0000-0000-0001.inf6.spectrum.com 0.0% 10 14.5 36.6 11.3 171.2 50.0 3.|-- lag-63.hcr02rdrktxaz.netops.charter.com 0.0% 10 18.8 37.3 12.0 143.9 45.3 4.|-- lag-19.ausxtxir02r.netops.charter.com 60.0% 10 65.8 30.2 11.7 65.8 24.9 5.|-- lag-22.rcr01hstqtx02.netops.charter.com 80.0% 10 26.5 25.5 24.5 26.5 1.4 6.|-- lag-16.hstqtx0209w-bcr00.netops.charter.com 90.0% 10 25.7 25.7 25.7 25.7 0.0 7.|-- lag-12.dllstx976iw-bcr00.netops.charter.com 40.0% 10 25.4 26.3 18.2 44.4 9.5 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
IPv4 works however:
mtr -4 --report-wide cloudflare.com Start: 2023-04-19T23:20:21+0000 HOST: defiant Loss% Snt Last Avg Best Wrst StDev 1.|-- _gateway 0.0% 10 2.1 1.8 1.4 2.1 0.2 2.|-- cpe-70-114-160-1.austin.res.rr.com 0.0% 10 51.9 34.4 14.3 149.7 42.1 3.|-- lag-63.hcr01rdrktxaz.netops.charter.com 0.0% 10 13.1 38.5 13.1 159.1 44.6 4.|-- lag-19.ausutxla01r.netops.charter.com 0.0% 10 18.1 21.0 14.3 34.8 6.0 5.|-- lag-22.rcr01dllatx37.netops.charter.com 0.0% 10 23.4 23.3 18.3 33.3 4.3 6.|-- lag-14.dllstx976iw-bcr00.netops.charter.com 0.0% 10 19.8 27.0 16.1 51.2 11.9 7.|-- lag-0.pr3.dfw10.netops.charter.com 0.0% 10 39.3 28.5 14.1 62.3 16.7 8.|-- 141.101.74.104 40.0% 10 53.2 44.3 19.4 120.2 39.2 9.|-- 172.71.168.2 0.0% 10 28.3 28.8 10.8 79.4 19.5 10.|-- 104.16.133.229 0.0% 10 122.0 42.8 16.9 122.0 34.4
It's kind of wild how many things don't fall back to IPv4 quickly (lots of Android TV apps) or at all (.NET). I tried reaching out to Spectrum support, but being a lowly consumer I was only told that "The modem doesn't have any filters" and that "the problem may be on the CloudFlare side", which was sadly what I predicted. So, I figured I would raise the issue here in the hopes that actual network people would see it.
-Carl
_______________________________________________ Outages mailing list Outages@outages.org https://www.google.com/url?q=https://puck.nether.net/mailman/listinfo/outage...
Hey folks, Could either of you contact me off-list so I can investigate? A specific start-of-problem would be super helpful. -- Tom Strickx Principal Network Engineer AS13335 - Cloudflare On Thu, Apr 20, 2023 at 4:38 AM Daniel Marks via Outages < outages@outages.org> wrote:
Have been dealing with this for 2 days, nobody at Spectrum can figure this out.
Daniel Marks May Mobility, Inc
On Apr 19, 2023, at 19:26, Carl Perry via Outages <outages@outages.org>
wrote:
Things have been rather slow the last couple of days, just figured out
why; No IPv6 connectivity to any CloudFlare services.
mtr -6 --report-wide cloudflare.com Start: 2023-04-19T23:20:42+0000 HOST: defiant Loss% Snt Last Avg Best Wrst StDev 1.|-- 2603-8081-2340-2868-0215-17ff-fe94-6457.res6.spectrum.com 0.0%
10 3.6 19.1 1.5 93.2 36.1
2.|-- 2603-90c5-0003-0180-0000-0000-0000-0001.inf6.spectrum.com 0.0% 10 14.5 36.6 11.3 171.2 50.0 3.|-- lag-63.hcr02rdrktxaz.netops.charter.com 0.0% 10 18.8 37.3 12.0 143.9 45.3 4.|-- lag-19.ausxtxir02r.netops.charter.com 60.0% 10 65.8 30.2 11.7 65.8 24.9 5.|-- lag-22.rcr01hstqtx02.netops.charter.com 80.0% 10 26.5 25.5 24.5 26.5 1.4 6.|-- lag-16.hstqtx0209w-bcr00.netops.charter.com 90.0% 10 25.7 25.7 25.7 25.7 0.0 7.|-- lag-12.dllstx976iw-bcr00.netops.charter.com 40.0% 10 25.4 26.3 18.2 44.4 9.5 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
IPv4 works however:
mtr -4 --report-wide cloudflare.com Start: 2023-04-19T23:20:21+0000 HOST: defiant Loss% Snt Last Avg Best Wrst StDev 1.|-- _gateway 0.0% 10 2.1 1.8 1.4 2.1 0.2 2.|-- cpe-70-114-160-1.austin.res.rr.com 0.0% 10 51.9 34.4 14.3 149.7 42.1 3.|-- lag-63.hcr01rdrktxaz.netops.charter.com 0.0% 10 13.1 38.5 13.1 159.1 44.6 4.|-- lag-19.ausutxla01r.netops.charter.com 0.0% 10 18.1 21.0 14.3 34.8 6.0 5.|-- lag-22.rcr01dllatx37.netops.charter.com 0.0% 10 23.4 23.3 18.3 33.3 4.3 6.|-- lag-14.dllstx976iw-bcr00.netops.charter.com 0.0% 10 19.8 27.0 16.1 51.2 11.9 7.|-- lag-0.pr3.dfw10.netops.charter.com 0.0% 10 39.3 28.5 14.1 62.3 16.7 8.|-- 141.101.74.104 40.0% 10 53.2 44.3 19.4 120.2 39.2 9.|-- 172.71.168.2 0.0% 10 28.3 28.8 10.8 79.4 19.5 10.|-- 104.16.133.229 0.0% 10 122.0 42.8 16.9 122.0 34.4
It's kind of wild how many things don't fall back to IPv4 quickly (lots of Android TV apps) or at all (.NET). I tried reaching out to Spectrum support, but being a lowly consumer I was only told that "The modem doesn't have any filters" and that "the problem may be on the CloudFlare side", which was sadly what I predicted. So, I figured I would raise the issue here in the hopes that actual network people would see it.
-Carl
_______________________________________________ Outages mailing list Outages@outages.org
https://www.google.com/url?q=https://puck.nether.net/mailman/listinfo/outage... _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Charter has acknowledged the issue, and is tracking this internally as INC000026440632. Appears to be an issue with a linecard in Houston. We've rerouted IPv6 out of Dallas for now to mitigate impact. Sorry for the trouble! -- Tom Strickx Principal Network Engineer AS13335 - Cloudflare On Thu, Apr 20, 2023 at 2:05 PM Tom Strickx <tstrickx@cloudflare.com> wrote:
Hey folks,
Could either of you contact me off-list so I can investigate? A specific start-of-problem would be super helpful. -- Tom Strickx Principal Network Engineer AS13335 - Cloudflare
On Thu, Apr 20, 2023 at 4:38 AM Daniel Marks via Outages < outages@outages.org> wrote:
Have been dealing with this for 2 days, nobody at Spectrum can figure this out.
Daniel Marks May Mobility, Inc
On Apr 19, 2023, at 19:26, Carl Perry via Outages <outages@outages.org>
wrote:
Things have been rather slow the last couple of days, just figured out
why; No IPv6 connectivity to any CloudFlare services.
mtr -6 --report-wide cloudflare.com Start: 2023-04-19T23:20:42+0000 HOST: defiant Loss% Snt Last Avg Best Wrst StDev 1.|-- 2603-8081-2340-2868-0215-17ff-fe94-6457.res6.spectrum.com
0.0% 10 3.6 19.1 1.5 93.2 36.1
2.|-- 2603-90c5-0003-0180-0000-0000-0000-0001.inf6.spectrum.com 0.0% 10 14.5 36.6 11.3 171.2 50.0 3.|-- lag-63.hcr02rdrktxaz.netops.charter.com 0.0% 10 18.8 37.3 12.0 143.9 45.3 4.|-- lag-19.ausxtxir02r.netops.charter.com 60.0% 10 65.8 30.2 11.7 65.8 24.9 5.|-- lag-22.rcr01hstqtx02.netops.charter.com 80.0% 10 26.5 25.5 24.5 26.5 1.4 6.|-- lag-16.hstqtx0209w-bcr00.netops.charter.com 90.0% 10 25.7 25.7 25.7 25.7 0.0 7.|-- lag-12.dllstx976iw-bcr00.netops.charter.com 40.0% 10 25.4 26.3 18.2 44.4 9.5 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
IPv4 works however:
mtr -4 --report-wide cloudflare.com Start: 2023-04-19T23:20:21+0000 HOST: defiant Loss% Snt Last Avg Best Wrst StDev 1.|-- _gateway 0.0% 10 2.1 1.8 1.4 2.1 0.2 2.|-- cpe-70-114-160-1.austin.res.rr.com 0.0% 10 51.9 34.4 14.3 149.7 42.1 3.|-- lag-63.hcr01rdrktxaz.netops.charter.com 0.0% 10 13.1 38.5 13.1 159.1 44.6 4.|-- lag-19.ausutxla01r.netops.charter.com 0.0% 10 18.1 21.0 14.3 34.8 6.0 5.|-- lag-22.rcr01dllatx37.netops.charter.com 0.0% 10 23.4 23.3 18.3 33.3 4.3 6.|-- lag-14.dllstx976iw-bcr00.netops.charter.com 0.0% 10 19.8 27.0 16.1 51.2 11.9 7.|-- lag-0.pr3.dfw10.netops.charter.com 0.0% 10 39.3 28.5 14.1 62.3 16.7 8.|-- 141.101.74.104 40.0% 10 53.2 44.3 19.4 120.2 39.2 9.|-- 172.71.168.2 0.0% 10 28.3 28.8 10.8 79.4 19.5 10.|-- 104.16.133.229 0.0% 10 122.0 42.8 16.9 122.0 34.4
It's kind of wild how many things don't fall back to IPv4 quickly (lots of Android TV apps) or at all (.NET). I tried reaching out to Spectrum support, but being a lowly consumer I was only told that "The modem doesn't have any filters" and that "the problem may be on the CloudFlare side", which was sadly what I predicted. So, I figured I would raise the issue here in the hopes that actual network people would see it.
-Carl
_______________________________________________ Outages mailing list Outages@outages.org
https://www.google.com/url?q=https://puck.nether.net/mailman/listinfo/outage... _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
As a follow up, we're still working with Spectrum engineers on a workaround, IPv6 is still mostly down in the DFW area on Spectrum outside of the large CDNs (i.e. Quad9 and Telia-bound traffic is still offline) - their ETA is another 24-48 hours as a line card for Houston was still on order, according to an engineer I spoke to. Not a great time for us IPv6-only networks. On Thu, Apr 20, 2023 at 11:18 AM Tom Strickx <tstrickx@cloudflare.com> wrote:
Charter has acknowledged the issue, and is tracking this internally as INC000026440632. Appears to be an issue with a linecard in Houston.
We've rerouted IPv6 out of Dallas for now to mitigate impact. Sorry for the trouble! -- Tom Strickx Principal Network Engineer AS13335 - Cloudflare
On Thu, Apr 20, 2023 at 2:05 PM Tom Strickx <tstrickx@cloudflare.com> wrote:
Hey folks,
Could either of you contact me off-list so I can investigate? A specific start-of-problem would be super helpful. -- Tom Strickx Principal Network Engineer AS13335 - Cloudflare
On Thu, Apr 20, 2023 at 4:38 AM Daniel Marks via Outages <outages@outages.org> wrote:
Have been dealing with this for 2 days, nobody at Spectrum can figure this out.
Daniel Marks May Mobility, Inc
On Apr 19, 2023, at 19:26, Carl Perry via Outages <outages@outages.org> wrote:
Things have been rather slow the last couple of days, just figured out why; No IPv6 connectivity to any CloudFlare services.
mtr -6 --report-wide cloudflare.com Start: 2023-04-19T23:20:42+0000 HOST: defiant Loss% Snt Last Avg Best Wrst StDev 1.|-- 2603-8081-2340-2868-0215-17ff-fe94-6457.res6.spectrum.com 0.0% 10 3.6 19.1 1.5 93.2 36.1 2.|-- 2603-90c5-0003-0180-0000-0000-0000-0001.inf6.spectrum.com 0.0% 10 14.5 36.6 11.3 171.2 50.0 3.|-- lag-63.hcr02rdrktxaz.netops.charter.com 0.0% 10 18.8 37.3 12.0 143.9 45.3 4.|-- lag-19.ausxtxir02r.netops.charter.com 60.0% 10 65.8 30.2 11.7 65.8 24.9 5.|-- lag-22.rcr01hstqtx02.netops.charter.com 80.0% 10 26.5 25.5 24.5 26.5 1.4 6.|-- lag-16.hstqtx0209w-bcr00.netops.charter.com 90.0% 10 25.7 25.7 25.7 25.7 0.0 7.|-- lag-12.dllstx976iw-bcr00.netops.charter.com 40.0% 10 25.4 26.3 18.2 44.4 9.5 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
IPv4 works however:
mtr -4 --report-wide cloudflare.com Start: 2023-04-19T23:20:21+0000 HOST: defiant Loss% Snt Last Avg Best Wrst StDev 1.|-- _gateway 0.0% 10 2.1 1.8 1.4 2.1 0.2 2.|-- cpe-70-114-160-1.austin.res.rr.com 0.0% 10 51.9 34.4 14.3 149.7 42.1 3.|-- lag-63.hcr01rdrktxaz.netops.charter.com 0.0% 10 13.1 38.5 13.1 159.1 44.6 4.|-- lag-19.ausutxla01r.netops.charter.com 0.0% 10 18.1 21.0 14.3 34.8 6.0 5.|-- lag-22.rcr01dllatx37.netops.charter.com 0.0% 10 23.4 23.3 18.3 33.3 4.3 6.|-- lag-14.dllstx976iw-bcr00.netops.charter.com 0.0% 10 19.8 27.0 16.1 51.2 11.9 7.|-- lag-0.pr3.dfw10.netops.charter.com 0.0% 10 39.3 28.5 14.1 62.3 16.7 8.|-- 141.101.74.104 40.0% 10 53.2 44.3 19.4 120.2 39.2 9.|-- 172.71.168.2 0.0% 10 28.3 28.8 10.8 79.4 19.5 10.|-- 104.16.133.229 0.0% 10 122.0 42.8 16.9 122.0 34.4
It's kind of wild how many things don't fall back to IPv4 quickly (lots of Android TV apps) or at all (.NET). I tried reaching out to Spectrum support, but being a lowly consumer I was only told that "The modem doesn't have any filters" and that "the problem may be on the CloudFlare side", which was sadly what I predicted. So, I figured I would raise the issue here in the hopes that actual network people would see it.
-Carl
_______________________________________________ Outages mailing list Outages@outages.org https://www.google.com/url?q=https://puck.nether.net/mailman/listinfo/outage...
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- -Dan
participants (3)
-
Carl Perry -
Daniel Marks -
Tom Strickx