August 2020 - Outages - Outages Lists

FiOS issue in NYC Area?
by Izzy Goldstein - TeleGo 16 Feb '21

16 Feb '21

Many of our clients who are using fios are reporting issues with their voip phones when i switch them to their backup ISP (Optimum etc) it works fine anyone else experiencing any fios issues ? -- Izzy Goldstein Chief Technology Officer Main: (212) 477-1000 x 2085 <(212)%20477-1000> Direct: (929) 477-2085 Website: www.telego.com <http://www.telego.net/> <http://www.telego.com/> Confidentiality Notice: This e-mail may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error please notify us immediately by email reply and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of TeleGo (T). Employees of TeleGo are expressly required not to make defamatory statements and not to infringe or authorize any infringement of copyright or any other legal right by email communications. Any such communication is contrary to TeleGo policy and outside the scope of the employment of the individual concerned. TeleGo will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising. TeleGo Hosted PBX <https://youtu.be/DaT8YAZ4V0w>

7 18

3556 problems
by Jay Ashworth 02 Sep '20

02 Sep '20

Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance. -- Sent from my Android device with K-9 Mail. Please excuse my brevity.

16 18

Major Level3 (CenturyLink) Issues
by Charles Sprickman 02 Sep '20

02 Sep '20

Starting around 6:15 Eastern or so noticed that BGP was still up with them, but no traffic between a home fios address in NJ and our L3 Transit. We have since turned down transit and regained connectivity. Initially I was able to reach some IPs (eg: 4.2.2.1) from Choopa and Cologix, but now it’s not reachable anywhere I have access. Lots of chatter on Twitter from geographically diverse accounts... Additionally, Cloudflare is reporting a problem, but it’s not clear if that’s related to CenturyLink customers reporting issues or not: +++++ Update - Cloudflare is investigating an increased level of HTTP 5xx class errors. We are working to understand the full impact and mitigate this problem. More updates to follow shortly. Aug 30, 10:45 UTC Investigating - Cloudflare is aware of, and investigating an issue which potentially impacts multiple customers. Further detail will be provided as more information becomes available. Aug 30, 10:21 UTC +++++ Charles

5 4

Re: [outages] Outages Digest, Vol 147, Issue 10
by Tino Montemor 30 Aug '20

30 Aug '20

Just want to reflect (pun?) what others are saying, Centurylink wasn't letting go of our routes. As of about 2 minutes ago it seems out AS Prepend has finally began to propagate _____ Tino Montemor | Skechers USA, Inc. O: 310-406-0132 M: 909-721-6673 ________________________________ From: Outages <outages-bounces(a)outages.org> on behalf of outages-request(a)outages.org <outages-request(a)outages.org> Sent: Sunday, August 30, 2020 6:32:35 AM To: outages(a)outages.org <outages(a)outages.org> Subject: Outages Digest, Vol 147, Issue 10 Send Outages mailing list submissions to outages(a)outages.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailma… or, via email, send a message with subject or body 'help' to outages-request(a)outages.org You can reach the person managing the list at outages-owner(a)outages.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..." Today's Topics: 1. Re: CenturyLink peering issues? (David Hubbard) 2. Re: 3356 does not WITHDRAW bgp routes (randal k) 3. Re: 3356 does not WITHDRAW bgp routes (David Hubbard) 4. Re: CenturyLink peering issues? (Chris Adams) 5. Re: 3356 does not WITHDRAW bgp routes (randal k) 6. Re: CenturyLink peering issues? (Stephen Flynn) ---------------------------------------------------------------------- Message: 1 Date: Sun, 30 Aug 2020 12:22:27 +0000 From: David Hubbard <dhubbard(a)dino.hostasaurus.com> To: "outages(a)outages.org" <outages(a)outages.org> Subject: Re: [outages] CenturyLink peering issues? Message-ID: <D5BED54B-45C3-4EC8-9DBA-EF1276D52C73(a)dino.hostasaurus.com> Content-Type: text/plain; charset="utf-8" Ugh; seeing same thing. Have had sessions turned down for over two hours and looking glasses are still showing 3356 propagating the advertisements. ?On 8/30/20, 8:11 AM, "Outages on behalf of Chris Adams via Outages" <outages-bounces(a)outages.org on behalf of outages(a)outages.org> wrote: Once upon a time, Stephen Flynn via Outages <outages(a)outages.org> said: > Odd part --- I disconnected my Level3 circuit at the ORL-FL facility so that I could fully failover to my other carrier link. > Level3 is still advertising my routes, even though my link and BGP session is down. I can confirm this - I shut down IPv4 BGP with Level3 in Chicago, but checking route-views and such, they're still advertising our routes (but don't know how to get to us once packets hit their network).. AS7007 all over again? -- Chris Adams <cma(a)cmadams.net> _______________________________________________ Outages mailing list Outages(a)outages.org https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailma… ------------------------------ Message: 2 Date: Sun, 30 Aug 2020 06:33:41 -0600 From: randal k <rkohutek+outages(a)gmail.com> To: outages(a)outages.org Subject: Re: [outages] 3356 does not WITHDRAW bgp routes Message-ID: <CANeLk7RmHnXa=EieXpahpmM966bL8==7qsywR89Yrd0fuZ9tHA(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" Seeing the same thing - shut a peer, still seeing those routes via 3356 across multiple route-servers & looking glasses. So, do we disconnect 3356 and suffer the blackhole in hopes that it will eventually withdraw those routes, or leave it on to prevent blackholing but suffer massive packet loss to other carriers? Wow. On Sun, Aug 30, 2020 at 6:28 AM Lukas Tribus via Outages <outages(a)outages.org> wrote: > > As previously mentioned by Stephen Flynn, 3356 does not WITHDRAW stale > bgp routes, can be confirmed with AT&T's route server at (telnet > route-server.ip.att.net) > > Stale routes from 1 hour + are still announced by 3356. > > > This is causing blackholing. > _______________________________________________ > Outages mailing list > Outages(a)outages.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailma… ------------------------------ Message: 3 Date: Sun, 30 Aug 2020 12:47:57 +0000 From: David Hubbard <dhubbard(a)dino.hostasaurus.com> To: "outages(a)outages.org" <outages(a)outages.org> Subject: Re: [outages] 3356 does not WITHDRAW bgp routes Message-ID: <4BA8D9E3-9A4F-45FB-AFE4-8BBB2E3C6902(a)dino.hostasaurus.com> Content-Type: text/plain; charset="utf-8" I tried bringing two circuits back up hoping to not have blackholing, one never left idle, the other received <2000 routes, which I know from prior outages to mean that the entire region (Tampa Bay) has been BGP isolated from the rest of their network. Good times... ?On 8/30/20, 8:43 AM, "Outages on behalf of randal k via Outages" <outages-bounces(a)outages.org on behalf of outages(a)outages.org> wrote: Seeing the same thing - shut a peer, still seeing those routes via 3356 across multiple route-servers & looking glasses. So, do we disconnect 3356 and suffer the blackhole in hopes that it will eventually withdraw those routes, or leave it on to prevent blackholing but suffer massive packet loss to other carriers? Wow. On Sun, Aug 30, 2020 at 6:28 AM Lukas Tribus via Outages <outages(a)outages.org> wrote: > > As previously mentioned by Stephen Flynn, 3356 does not WITHDRAW stale > bgp routes, can be confirmed with AT&T's route server at (telnet > route-server.ip.att.net) > > Stale routes from 1 hour + are still announced by 3356. > > > This is causing blackholing. > _______________________________________________ > Outages mailing list > Outages(a)outages.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailma… _______________________________________________ Outages mailing list Outages(a)outages.org https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailma… ------------------------------ Message: 4 Date: Sun, 30 Aug 2020 07:48:14 -0500 From: Chris Adams <cma(a)cmadams.net> To: outages(a)outages.org Subject: Re: [outages] CenturyLink peering issues? Message-ID: <20200830124814.GB29578(a)cmadams.net> Content-Type: text/plain; charset=us-ascii Once upon a time, randal k <rkohutek+outages(a)gmail.com> said: > It's weird you say that - I have been attempting to use Level3's well > known communities to attempt to prepend, no-export etc and have been > having limited luck ... and those communities even appear in > route-views, so I know they're being sent! I brought my session back up, not accepting any routes, and prepending ours (just in case the change propagated)... it took a while, but eventually I do see the prepended routes. But for a provider that said they shut down their link to Level3, I still see their routes... possibly (slowly) propagating changes but not actual withdraws? -- Chris Adams <cma(a)cmadams.net> ------------------------------ Message: 5 Date: Sun, 30 Aug 2020 06:52:16 -0600 From: randal k <rkohutek+outages(a)gmail.com> To: outages(a)outages.org Subject: Re: [outages] 3356 does not WITHDRAW bgp routes Message-ID: <CANeLk7RcTGWkQp4SYJQO=j=c04dTpYSGrXTxfK9ovi_QkgyxZQ(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" Watching closely, after applying ^3356$ on our inbound routes, I can see that they are bouncing sessions and slowly adding in prefixes -- I have 17x 3356-originated routes in PA, and 893x in CO. And they have reset the PA BGP session numerous times. On Sun, Aug 30, 2020 at 6:33 AM randal k <rkohutek+outages(a)gmail.com> wrote: > > Seeing the same thing - shut a peer, still seeing those routes via > 3356 across multiple route-servers & looking glasses. > > So, do we disconnect 3356 and suffer the blackhole in hopes that it > will eventually withdraw those routes, or leave it on to prevent > blackholing but suffer massive packet loss to other carriers? Wow. > > On Sun, Aug 30, 2020 at 6:28 AM Lukas Tribus via Outages > <outages(a)outages.org> wrote: > > > > As previously mentioned by Stephen Flynn, 3356 does not WITHDRAW stale > > bgp routes, can be confirmed with AT&T's route server at (telnet > > route-server.ip.att.net) > > > > Stale routes from 1 hour + are still announced by 3356. > > > > > > This is causing blackholing. > > _______________________________________________ > > Outages mailing list > > Outages(a)outages.org > > https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailma… ------------------------------ Message: 6 Date: Sun, 30 Aug 2020 13:32:31 +0000 From: Stephen Flynn <sflynn(a)staff.atlantic.net> To: "outages(a)outages.org" <outages(a)outages.org> Subject: Re: [outages] CenturyLink peering issues? Message-ID: <BN6PR19MB10269D437047598C160A6724EB500(a)BN6PR19MB1026.namprd19.prod.outlook.com> Content-Type: text/plain; charset="us-ascii" Just attempted a prepend (x4) on my Level3 advertisements --- only noticed a change within the NTT network tables. Other large carriers received no advertisement changes (Telia, Hurricane Electric, AT&T) I then noticed that my BGP session was constantly flapping. I've now shutdown my circuit again. Level3 is still announcing my address space. Thank you Level3! Regards, Stephen Flynn Atlantic.Net Direct: (321) 206-1390 sflynn(a)staff.atlantic.net www.atlantic.net<http://www.atlantic.net> -----Original Message----- From: Outages <outages-bounces(a)outages.org> On Behalf Of Chris Adams via Outages Sent: Sunday, August 30, 2020 8:48 AM To: outages(a)outages.org Subject: Re: [outages] CenturyLink peering issues? CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Once upon a time, randal k <rkohutek+outages(a)gmail.com> said: > It's weird you say that - I have been attempting to use Level3's well > known communities to attempt to prepend, no-export etc and have been > having limited luck ... and those communities even appear in > route-views, so I know they're being sent! I brought my session back up, not accepting any routes, and prepending ours (just in case the change propagated)... it took a while, but eventually I do see the prepended routes. But for a provider that said they shut down their link to Level3, I still see their routes... possibly (slowly) propagating changes but not actual withdraws? -- Chris Adams <cma(a)cmadams.net> _______________________________________________ Outages mailing list Outages(a)outages.org https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailma… ------------------------------ Subject: Digest Footer _______________________________________________ Outages mailing list Outages(a)outages.org https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailma… ------------------------------ End of Outages Digest, Vol 147, Issue 10 ****************************************

3 2

East Coast
by Todd Gregory 30 Aug '20

30 Aug '20

We are seeing all kinds of issues here on the East Coast too. [cid:image003.jpg@01D67EA0.30175170]

5 4

CenturyLink peering issues?
by randal k 30 Aug '20

30 Aug '20

We have been seeing serious issues with Level3/CenturyLink services over the past hour or so. Everything on-net seems OK, but transit traffic looks to have problems. Thoughts?

5 8

3356 does not WITHDRAW bgp routes
by Lukas Tribus 30 Aug '20

30 Aug '20

As previously mentioned by Stephen Flynn, 3356 does not WITHDRAW stale bgp routes, can be confirmed with AT&T's route server at (telnet route-server.ip.att.net) Stale routes from 1 hour + are still announced by 3356. This is causing blackholing.

3 3

US Kentucky Outage
by Jason Kuehl 30 Aug '20

30 Aug '20

We are seeing all kinds of issues over in Kentucky. Other sites are NYC, MA, WI all with the same issues -- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.kuehl(a)gmail.com

1 0

CenturyLink packetloss in Chicago
by Andy Brezinsky 30 Aug '20

30 Aug '20

Seeing 70% packet loss through CenturyLink routes in the Chicago region. Not even able to hit the CL LG to dig in further. Anyone else seeing the same? A lot of other sites (Discord, Cloudflare, Hulu) are reporting trouble too, might be related.

2 1

3356 IPv4 outage (EMEA?)
by Lukas Tribus 30 Aug '20

30 Aug '20

I'm seeing IPv4 flapping and no traffic at all on 3356 transit links since approximately 10:00 UTC at multiple EMEA locations (Italy and Germany). Probably causing issues with Cloudflare (I guess they point a default to 3356). Ticket systems are unreachable, Hotlines busy.

1 0