We have been seeing serious issues with Level3/CenturyLink services over the past hour or so. Everything on-net seems OK, but transit traffic looks to have problems. Thoughts?
Confirmed Level3 carrier issue from the Orlando region --- possibly my other data centers as well (haven't checked them yet, been focusing on my ORL-FL outage). Odd part --- I disconnected my Level3 circuit at the ORL-FL facility so that I could fully failover to my other carrier link. Level3 is still advertising my routes, even though my link and BGP session is down. er-02.orl-fl> show interfaces descriptions | match Level3 xe-2/0/0 down down LEVEL3 | er-02.orl-fl> show bgp summary | match 3356 67.30.142.81 3356 0 0 0 0 27:43 Active 2001:1900:2100::26ad 3356 0 0 0 0 27:43 Idle BGP routing table entry for 209.208.0.0/17 Versions: Process bRIB/RIB SendTblVer Speaker 86180864 86180864 Last Modified: May 16 05:47:52.095 for 15w1d Paths: (25 available, best #25) Advertised IPv4 Unicast paths to update-groups (with more than one peer): 0.1 0.2 0.8 0.9 0.10 Path #1: Received by speaker 0 Not advertised to any peer 3356 6364 129.250.8.174 (metric 9340) from 129.250.0.2 (129.250.0.2) Origin IGP, metric 4294967294, localpref 100, valid, confed-internal Received Path ID 0, Local Path ID 0, version 0 Community: 2914:390 2914:1003 2914:2000 2914:3000 65504:3356 Path #2: Received by speaker 0 Not advertised to any peer 3356 6364 129.250.9.50 (metric 24142) from 129.250.0.6 (129.250.0.6) Origin IGP, metric 4294967294, localpref 100, valid, confed-internal Received Path ID 0, Local Path ID 0, version 0 Community: 2914:390 2914:1214 2914:2213 2914:3200 65504:3356 Path #3: Received by speaker 0 Not advertised to any peer 174 6364 130.117.15.89 (metric 24174) from 129.250.0.13 (129.250.0.13) Origin IGP, metric 4294967294, localpref 100, valid, confed-internal Received Path ID 0, Local Path ID 0, version 0 Community: 2914:390 2914:1204 2914:2205 2914:3200 65504:174 Path #4: Received by speaker 0 Regards, Stephen Flynn Atlantic.Net Direct: (321) 206-1390 sflynn@staff.atlantic.net www.atlantic.net -----Original Message----- From: Outages <outages-bounces@outages.org> On Behalf Of randal k via Outages Sent: Sunday, August 30, 2020 7:10 AM To: outages@outages.org Subject: [outages] CenturyLink peering issues? CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. We have been seeing serious issues with Level3/CenturyLink services over the past hour or so. Everything on-net seems OK, but transit traffic looks to have problems. Thoughts? _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Once upon a time, Stephen Flynn via Outages <outages@outages.org> said:
Odd part --- I disconnected my Level3 circuit at the ORL-FL facility so that I could fully failover to my other carrier link. Level3 is still advertising my routes, even though my link and BGP session is down.
I can confirm this - I shut down IPv4 BGP with Level3 in Chicago, but checking route-views and such, they're still advertising our routes (but don't know how to get to us once packets hit their network).. AS7007 all over again? -- Chris Adams <cma@cmadams.net>
It's weird you say that - I have been attempting to use Level3's well known communities to attempt to prepend, no-export etc and have been having limited luck ... and those communities even appear in route-views, so I know they're being sent! On Sun, Aug 30, 2020 at 6:10 AM Chris Adams via Outages <outages@outages.org> wrote:
Once upon a time, Stephen Flynn via Outages <outages@outages.org> said:
Odd part --- I disconnected my Level3 circuit at the ORL-FL facility so that I could fully failover to my other carrier link. Level3 is still advertising my routes, even though my link and BGP session is down.
I can confirm this - I shut down IPv4 BGP with Level3 in Chicago, but checking route-views and such, they're still advertising our routes (but don't know how to get to us once packets hit their network)..
AS7007 all over again? -- Chris Adams <cma@cmadams.net> _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Once upon a time, randal k <rkohutek+outages@gmail.com> said:
It's weird you say that - I have been attempting to use Level3's well known communities to attempt to prepend, no-export etc and have been having limited luck ... and those communities even appear in route-views, so I know they're being sent!
I brought my session back up, not accepting any routes, and prepending ours (just in case the change propagated)... it took a while, but eventually I do see the prepended routes. But for a provider that said they shut down their link to Level3, I still see their routes... possibly (slowly) propagating changes but not actual withdraws? -- Chris Adams <cma@cmadams.net>
Just attempted a prepend (x4) on my Level3 advertisements --- only noticed a change within the NTT network tables. Other large carriers received no advertisement changes (Telia, Hurricane Electric, AT&T) I then noticed that my BGP session was constantly flapping. I've now shutdown my circuit again. Level3 is still announcing my address space. Thank you Level3! Regards, Stephen Flynn Atlantic.Net Direct: (321) 206-1390 sflynn@staff.atlantic.net www.atlantic.net -----Original Message----- From: Outages <outages-bounces@outages.org> On Behalf Of Chris Adams via Outages Sent: Sunday, August 30, 2020 8:48 AM To: outages@outages.org Subject: Re: [outages] CenturyLink peering issues? CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Once upon a time, randal k <rkohutek+outages@gmail.com> said:
It's weird you say that - I have been attempting to use Level3's well known communities to attempt to prepend, no-export etc and have been having limited luck ... and those communities even appear in route-views, so I know they're being sent!
I brought my session back up, not accepting any routes, and prepending ours (just in case the change propagated)... it took a while, but eventually I do see the prepended routes. But for a provider that said they shut down their link to Level3, I still see their routes... possibly (slowly) propagating changes but not actual withdraws? -- Chris Adams <cma@cmadams.net> _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Telia and Hurricane Electric are now showing cleared routing tables and sending my traffic to the correct carrier circuits. My L3 circuit is still admin down --- it will remain this way for the remainder of the day and until I get an update from CenturyLink that the issue is resolved. Regards, Stephen Flynn Atlantic.Net Direct: (321) 206-1390 sflynn@staff.atlantic.net www.atlantic.net -----Original Message----- From: Outages <outages-bounces@outages.org> On Behalf Of Stephen Flynn via Outages Sent: Sunday, August 30, 2020 9:33 AM To: outages@outages.org Subject: Re: [outages] CenturyLink peering issues? CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Just attempted a prepend (x4) on my Level3 advertisements --- only noticed a change within the NTT network tables. Other large carriers received no advertisement changes (Telia, Hurricane Electric, AT&T) I then noticed that my BGP session was constantly flapping. I've now shutdown my circuit again. Level3 is still announcing my address space. Thank you Level3! Regards, Stephen Flynn Atlantic.Net Direct: (321) 206-1390 sflynn@staff.atlantic.net www.atlantic.net -----Original Message----- From: Outages <outages-bounces@outages.org> On Behalf Of Chris Adams via Outages Sent: Sunday, August 30, 2020 8:48 AM To: outages@outages.org Subject: Re: [outages] CenturyLink peering issues? CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Once upon a time, randal k <rkohutek+outages@gmail.com> said:
It's weird you say that - I have been attempting to use Level3's well known communities to attempt to prepend, no-export etc and have been having limited luck ... and those communities even appear in route-views, so I know they're being sent!
I brought my session back up, not accepting any routes, and prepending ours (just in case the change propagated)... it took a while, but eventually I do see the prepended routes. But for a provider that said they shut down their link to Level3, I still see their routes... possibly (slowly) propagating changes but not actual withdraws? -- Chris Adams <cma@cmadams.net> _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Ugh; seeing same thing. Have had sessions turned down for over two hours and looking glasses are still showing 3356 propagating the advertisements. On 8/30/20, 8:11 AM, "Outages on behalf of Chris Adams via Outages" <outages-bounces@outages.org on behalf of outages@outages.org> wrote: Once upon a time, Stephen Flynn via Outages <outages@outages.org> said: > Odd part --- I disconnected my Level3 circuit at the ORL-FL facility so that I could fully failover to my other carrier link. > Level3 is still advertising my routes, even though my link and BGP session is down. I can confirm this - I shut down IPv4 BGP with Level3 in Chicago, but checking route-views and such, they're still advertising our routes (but don't know how to get to us once packets hit their network).. AS7007 all over again? -- Chris Adams <cma@cmadams.net> _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Some of our customers cannot reach our website in LA from Virginia Beach, and connections are quite intermittent. Not sure related, but experienced small annoying network issue around 2PM PDT from our network and got to this point -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of randal k via Outages Sent: Sunday, August 30, 2020 4:10 AM To: outages@outages.org Subject: [outages] CenturyLink peering issues? We have been seeing serious issues with Level3/CenturyLink services over the past hour or so. Everything on-net seems OK, but transit traffic looks to have problems. Thoughts? _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (5)
-
Chris Adams -
David Hubbard -
randal k -
Stephen Flynn -
Tim Choe