It seems like the Charter issue this morning was "regional" (perhaps a rather small "region"). It sounds like it primarily affected the Madison, WI area, which we hang off of (we're in Verona, WI). We re-enabled our BGP advertisements via Charter about 15 minutes ago and everything seems to be operating normally. Yes, we like to hold off for a while after the up-stream gives the all-clear, just in case there are some lingering issues that still need to shake out. Thanks to those who replied with observations and offers of assistance, both on and off-list. From: Jeremy Charles Sent: Friday, February 24, 2017 10:01 AM To: 'outages@outages.org' <outages@outages.org> Subject: RE: Charter issues this morning? Update received from Charter just a moment ago.... I think we'll give it a bit for things to settle down before we put traffic back on them. STATUS: The SE NOC has been advised that the bad fiber has been re-spliced at this time. Following the completion of these actions, both fibers have been verified and the original links have restored. If you have any questions or concerns please call 866-603-3199 and reference your ticket number. From: Jeremy Charles Sent: Friday, February 24, 2017 8:45 AM To: 'outages@outages.org' <outages@outages.org<mailto:outages@outages.org>> Subject: Charter issues this morning? Anyone else using Charter Internet have issues with connectivity this morning? We (AS 10359) have site-to-site VPN tunnels to about 150 customers across the US. We started seeing loss of connectivity to about 20% of them at about 06:25 CST. When we filtered out all advertisements sent/received via Charter, we regained connectivity to everyone. (We have two ISPs in addition to Charter.) Additionally, our total aggregate Internet consumption jumped from about 200 Mbps to 500 Mbps as soon as Charter was taken out of the mix. It looks like a lot of traffic was preferring to come to us via Charter (which is normal) but just wasn't making it here. One of our engineers waited on hold for Charter for about 20 minutes and then gave up to go get coffee. :-) [https://mrtg1.epic.com/mrtg/internet/inbound-borders.png] == Jeremy Charles Epic's Computer and Technology Services Division jcharles@epic.com<mailto:jcharles@epic.com> 608-271-9000
