Starting around 6:15 Eastern or so noticed that BGP was still up with them, but no traffic between a home fios address in NJ and our L3 Transit. We have since turned down transit and regained connectivity. Initially I was able to reach some IPs (eg: 4.2.2.1) from Choopa and Cologix, but now it’s not reachable anywhere I have access. Lots of chatter on Twitter from geographically diverse accounts... Additionally, Cloudflare is reporting a problem, but it’s not clear if that’s related to CenturyLink customers reporting issues or not: +++++ Update - Cloudflare is investigating an increased level of HTTP 5xx class errors. We are working to understand the full impact and mitigate this problem. More updates to follow shortly. Aug 30, 10:45 UTC Investigating - Cloudflare is aware of, and investigating an issue which potentially impacts multiple customers. Further detail will be provided as more information becomes available. Aug 30, 10:21 UTC +++++ Charles
Confirm for Europe, too. On 30 Aug 2020, at 12:56, Charles Sprickman via Outages wrote:
Starting around 6:15 Eastern or so noticed that BGP was still up with them, but no traffic between a home fios address in NJ and our L3 Transit. We have since turned down transit and regained connectivity.
Initially I was able to reach some IPs (eg: 4.2.2.1) from Choopa and Cologix, but now it’s not reachable anywhere I have access.
Lots of chatter on Twitter from geographically diverse accounts...
Additionally, Cloudflare is reporting a problem, but it’s not clear if that’s related to CenturyLink customers reporting issues or not:
+++++ Update - Cloudflare is investigating an increased level of HTTP 5xx class errors.
We are working to understand the full impact and mitigate this problem. More updates to follow shortly.
Aug 30, 10:45 UTC
Investigating - Cloudflare is aware of, and investigating an issue which potentially impacts multiple customers. Further detail will be provided as more information becomes available.
Aug 30, 10:21 UTC +++++
Charles _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I can confirm issues seen from US and EU continues for our ip traffic destined to US market, some traffic is dies on their Miami router even though we're no longer announcing on any of their routers our ip space, it appears as if the routes are stuck on their routers in certain locations. On Sun, Aug 30, 2020 at 7:43 AM Jörg Kost via Outages <outages@outages.org> wrote:
Confirm for Europe, too.
On 30 Aug 2020, at 12:56, Charles Sprickman via Outages wrote:
Starting around 6:15 Eastern or so noticed that BGP was still up with them, but no traffic between a home fios address in NJ and our L3 Transit. We have since turned down transit and regained connectivity.
Initially I was able to reach some IPs (eg: 4.2.2.1) from Choopa and Cologix, but now it’s not reachable anywhere I have access.
Lots of chatter on Twitter from geographically diverse accounts...
Additionally, Cloudflare is reporting a problem, but it’s not clear if that’s related to CenturyLink customers reporting issues or not:
+++++ Update - Cloudflare is investigating an increased level of HTTP 5xx class errors.
We are working to understand the full impact and mitigate this problem. More updates to follow shortly.
Aug 30, 10:45 UTC
Investigating - Cloudflare is aware of, and investigating an issue which potentially impacts multiple customers. Further detail will be provided as more information becomes available.
Aug 30, 10:21 UTC +++++
Charles _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Got a trace? On Sun, Aug 30, 2020 at 8:01 AM Dejan Protich via Outages < outages@outages.org> wrote:
I can confirm issues seen from US and EU continues for our ip traffic destined to US market, some traffic is dies on their Miami router even though we're no longer announcing on any of their routers our ip space, it appears as if the routes are stuck on their routers in certain locations.
On Sun, Aug 30, 2020 at 7:43 AM Jörg Kost via Outages <outages@outages.org> wrote:
Confirm for Europe, too.
On 30 Aug 2020, at 12:56, Charles Sprickman via Outages wrote:
Starting around 6:15 Eastern or so noticed that BGP was still up with them, but no traffic between a home fios address in NJ and our L3 Transit. We have since turned down transit and regained connectivity.
Initially I was able to reach some IPs (eg: 4.2.2.1) from Choopa and Cologix, but now it’s not reachable anywhere I have access.
Lots of chatter on Twitter from geographically diverse accounts...
Additionally, Cloudflare is reporting a problem, but it’s not clear if that’s related to CenturyLink customers reporting issues or not:
+++++ Update - Cloudflare is investigating an increased level of HTTP 5xx class errors.
We are working to understand the full impact and mitigate this problem. More updates to follow shortly.
Aug 30, 10:45 UTC
Investigating - Cloudflare is aware of, and investigating an issue which potentially impacts multiple customers. Further detail will be provided as more information becomes available.
Aug 30, 10:21 UTC +++++
Charles _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com
the RFO is making the rounds http://seele.lamehost.it/~marco/blind/Network_Event_Formal_RFO_Multiple_Mark... it kinda explains the flowspec issue but completely ignores the stuck routes, which imiho was the more damaging problem. randy
participants (5)
-
Charles Sprickman -
Dejan Protich -
Jason Kuehl -
Jörg Kost -
Randy Bush