February 2022 - Outages - Outages Lists

Possible Cox routing issues in Phoenix
by Mike Bolitho 24 Feb '22

24 Feb '22

There are some interesting routing issues impacting at least the Phoenix market for Cox. Haven't done a ton of digging yet but we saw about a 25% drop in ingress traffic on our Cox circuit and a corresponding jump on our Lumen circuit. Cox is still sending us routes to we're egressing there but having intermittent connectivity to some resources because of it. - Mike Bolitho

2 1

Slack?
by Paul Ferguson 22 Feb '22

22 Feb '22

Apparently there's a Slack outage, any details or RFO? Downdetector also suggest AWS and github, as well... Cheers, - ferg -- Paul Ferguson Tacoma, WA USA Illegitimi non carborundum.

12 11

Re: [outages] Outages Digest, Vol 165, Issue 12
by John-Paul Reed 22 Feb '22

22 Feb '22

Our company has employees all over the world. We use Slack and I am in San Antonio, TX and can confirm we are still seeing issues with Slack. We also use AWS and Github and i have had no reports of issues from our end users on those services. Regards, John-Paul On Tue, Feb 22, 2022, 9:45 AM <outages-request(a)outages.org> wrote: > Send Outages mailing list submissions to > outages(a)outages.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://puck.nether.net/mailman/listinfo/outages > or, via email, send a message with subject or body 'help' to > outages-request(a)outages.org > > You can reach the person managing the list at > outages-owner(a)outages.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Outages digest..." > > > Today's Topics: > > 1. Re: Slack? (Stephane Bortzmeyer) > 2. Re: Slack? (Clayton Dukes) > 3. Re: Slack? (Jacob Perkins) > 4. Re: Slack? (Daniel Marks) > 5. Re: Slack? (Jeff Belcher) > 6. Re: Slack? (Jim O'Brien) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 22 Feb 2022 16:03:34 +0100 > From: Stephane Bortzmeyer <bortzmeyer(a)nic.fr> > To: Paul Ferguson <fergdawgster(a)mykolab.com> > Cc: Outages ML <outages(a)outages.org> > Subject: Re: [outages] Slack? > Message-ID: <YhT7Rrppz8+Xcw/V(a)nic.fr> > Content-Type: text/plain; charset=us-ascii > > On Tue, Feb 22, 2022 at 06:49:27AM -0800, > Paul Ferguson via Outages <outages(a)outages.org> wrote > a message of 16 lines which said: > > > Apparently there's a Slack outage, any details or RFO? > > > > Downdetector also suggest AWS and github, as well... > > Slack and Github works for me (AS 2484) > > > > ------------------------------ > > Message: 2 > Date: Tue, 22 Feb 2022 10:04:28 -0500 > From: Clayton Dukes <cdukes(a)gmail.com> > To: Paul Ferguson <fergdawgster(a)mykolab.com> > Cc: Outages ML <outages(a)outages.org> > Subject: Re: [outages] Slack? > Message-ID: > < > CAANP6Qf6HyViTLUadfjW5NYg-RiHDQs5M06c7BLLVQ3s4M9KvQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Same here. Our team also had trouble connecting to github around the same > time. > > ______________________________________________________________ > > Clayton Dukes > ______________________________________________________________ > > > On Tue, Feb 22, 2022 at 9:56 AM Paul Ferguson via Outages < > outages(a)outages.org> wrote: > > > Apparently there's a Slack outage, any details or RFO? > > > > Downdetector also suggest AWS and github, as well... > > > > Cheers, > > > > - ferg > > > > -- > > Paul Ferguson > > Tacoma, WA USA > > Illegitimi non carborundum. > > _______________________________________________ > > Outages mailing list > > Outages(a)outages.org > > https://puck.nether.net/mailman/listinfo/outages > > >

2 1

Coresite LA1 to LA2 fiber cut
by Jay Hennigan 17 Feb '22

17 Feb '22

We lost circuits between One Wilshire and 900 Broadway. Coresite master ticket 02532710 No ETR. -- Jay Hennigan - jay(a)west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV

2 2

Ping to Google 8.8.8.8
by Justin Krejci 10 Feb '22

10 Feb '22

Seeing some hosts and not others get no reply to ICMP pings from the Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2 Minneapolis routers gets no replies. Since many use and assume 8.8.8.8 is invincible, this has now started setting off monitoring alerts. DNS queries so far seem unaffected.

16 21

DNSSEC issues .se
by Jonathan Sélea 04 Feb '22

04 Feb '22

3 6

Re: [outages] Outages Digest, Vol 165, Issue 4
by Douglas Haber 04 Feb '22

04 Feb '22

https://www.dell.com/en-us/work/shop/dell-laptops-and-notebooks/latitude-55… On 2/4/22, 11:48, "Outages on behalf of outages-request(a)outages.org" <outages-bounces(a)outages.org on behalf of outages-request(a)outages.org> wrote: Send Outages mailing list submissions to outages(a)outages.org To subscribe or unsubscribe via the World Wide Web, visit https://linklock.titanhq.com/analyse?url=https%3A%2F%2Fpuck.nether.net%2Fma… or, via email, send a message with subject or body 'help' to outages-request(a)outages.org You can reach the person managing the list at outages-owner(a)outages.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..." Today's Topics: 1. Re: DNSSEC issues .se (Jonathan S?lea) ---------------------------------------------------------------------- Message: 1 Date: Fri, 4 Feb 2022 16:44:16 +0000 From: Jonathan S?lea <jonathan.selea(a)portsgroup.com> To: "Outages(a)outages.org" <outages(a)outages.org> Subject: Re: [outages] DNSSEC issues .se Message-ID: <AM5P195MB019635068C070F7B4F637834E9299(a)AM5P195MB0196.EURP195.PROD.OUTLOOK.COM> Content-Type: text/plain; charset="utf-8" This just came from IIS: We currently have an incorrect .se zone file published, where approximately 9,000 .se domains have incorrect DNSSEC signatures, ie DNSSEC validated resolvers will not approve naming of these 9,000 domains. Work with troubleshooting and measures is in progress. We will inform again as soon as we have new information about this. Kind regards, Registry Services The Swedish Internet Foundation I personally have been investigating this since 14:00 ? and contacted IIS around 15:00, and it took two more hours for them to actually acknowledge the problem. https://linklock.titanhq.com/analyse?url=https%3A%2F%2Ftwitter.com%2Fstifte… Med v?nlig h?lsning / Best Regards? Jonathan S?lea Linux Technician <tel:+46%2070%20726%2000%2050> +46 70 726 00 50 <mailto:jonathan.selea@portsgroup.com> jonathan.selea(a)portsgroup.com G?teborg, Kungsgatan 42 <https://linklock.titanhq.com/analyse?url=https%3A%2F%2Fportsgroup.com%2F&da…> From: Outages <outages-bounces(a)outages.org> On Behalf Of James Lawrie via Outages Sent: den 4 februari 2022 17:31 To: Outages <outages(a)outages.org> Subject: Re: [outages] DNSSEC issues .se Just to give a counter example for investigation (not sure where it is in the zone, replying quickly), my silvermou.se is resolving On 4 Feb 2022, at 16:25, Jonathan S?lea via Outages wrote: Anyone else seeing dnssec issues on unsigned .se domains? Apparently, if a unsigned domain is followed by a signed domain in the .se zone - the domain wont resolve due to NSEC errors. Example: Sportbladet.se Kgkfastigheter.se Deltacity.se Med v?nlig h?lsning / Best Regards? Jonathan S?lea Linux Technician <tel:+46%2070%20726%2000%2050> +46 70 726 00 50 <mailto:jonathan.selea@portsgroup.com> jonathan.selea(a)portsgroup.com G?teborg, Kungsgatan 42 <https://linklock.titanhq.com/analyse?url=https%3A%2F%2Favanan.url-protectio…> <https://linklock.titanhq.com/analyse?url=https%3A%2F%2Favanan.url-protectio…> The General Terms applicable to our services are available on our website, <https://linklock.titanhq.com/analyse?url=https%3A%2F%2Favanan.url-protectio…> here. Please refer to our <https://linklock.titanhq.com/analyse?url=https%3A%2F%2Favanan.url-protectio…> Privacy Policy for information about how we process personal data. This e-mail may contain legally privileged and confidential information. If you are not the intended addressee, you are hereby notified that any reading, distribution, copying or other use of this message or attachments is strictly prohibited. If you have received this message in error, return to us and delete this email. Thank you. _______________________________________________ Outages mailing list Outages(a)outages.org <mailto:Outages@outages.org> <https://linklock.titanhq.com/analyse?url=https%3A%2F%2Favanan.url-protectio…> https://linklock.titanhq.com/analyse?url=https%3A%2F%2Fpuck.nether.net%2Fma… -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://linklock.titanhq.com/analyse?url=https%3A%2F%2Fpuck.nether.net%2Fpi…> -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 272 bytes Desc: not available URL: <https://linklock.titanhq.com/analyse?url=https%3A%2F%2Fpuck.nether.net%2Fpi…> -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 291 bytes Desc: not available URL: <https://linklock.titanhq.com/analyse?url=https%3A%2F%2Fpuck.nether.net%2Fpi…> -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.png Type: image/png Size: 278 bytes Desc: not available URL: <https://linklock.titanhq.com/analyse?url=https%3A%2F%2Fpuck.nether.net%2Fpi…> -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.png Type: image/png Size: 2352 bytes Desc: not available URL: <https://linklock.titanhq.com/analyse?url=https%3A%2F%2Fpuck.nether.net%2Fpi…> -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 26733 bytes Desc: not available URL: <https://linklock.titanhq.com/analyse?url=https%3A%2F%2Fpuck.nether.net%2Fpi…> -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4968 bytes Desc: not available URL: <https://linklock.titanhq.com/analyse?url=https%3A%2F%2Fpuck.nether.net%2Fpi…> ------------------------------ Subject: Digest Footer _______________________________________________ Outages mailing list Outages(a)outages.org https://linklock.titanhq.com/analyse?url=https%3A%2F%2Fpuck.nether.net%2Fma… ------------------------------ End of Outages Digest, Vol 165, Issue 4 ***************************************

1 0

GTT daily congestion in Europe
by Lukas Tribus 04 Feb '22

04 Feb '22

Dear list, There is major congestion for 8 - 10 hours every day on the GTT network from Frankfurt, Germany towards Milan, Italy. Before noon (localtime, that would be 1200~1300Z) things are looking fine but in the afternoon, packet-loss and latency increase steadily (peaking in the evening). We are talking packet-loss in excess of 10% and ten times the expected latency: lukas@htznr2:~$ mtr 62.10.128.1 -wrs1000 -c1000 Start: 2022-02-03T21:33:28+0100 HOST: htznr2 Loss% Snt Last Avg Best Wrst StDev 1.|-- _gateway 0.0% 1000 3.1 4.9 2.3 18.0 1.3 2.|-- 10114.your-cloud.host 0.0% 1000 0.2 0.3 0.2 4.0 0.2 3.|-- ??? 100.0 1000 0.0 0.0 0.0 0.0 0.0 4.|-- 85-10-250-153.clients.your-server.de 0.0% 1000 1.3 4.8 0.8 98.6 11.8 5.|-- spine12.cloud1.nbg1.hetzner.com 0.0% 1000 1.1 4.8 0.8 66.8 11.4 6.|-- 213-239-239-121.clients.your-server.de 0.0% 1000 3.0 2.3 0.5 43.4 4.1 7.|-- core1.fra.hetzner.com 0.0% 1000 3.6 5.7 3.5 65.2 5.0 8.|-- ae18.cr6-fra2.ip4.gtt.net 0.0% 1000 4.8 5.8 3.6 116.7 6.7 9.|-- et-5-1-0.cr0-mil2.ip4.gtt.net 11.1% 1000 114.0 112.3 63.5 172.7 13.4 10.|-- tiscali.ip4.gtt.net 11.5% 1000 111.5 113.4 64.3 148.8 13.2 11.|-- static-94-32-126-81.clienti.tiscali.it 10.6% 1000 121.7 126.4 79.8 162.9 13.7 12.|-- static-94-32-135-162.clienti.tiscali.it 11.2% 1000 119.1 126.8 80.9 176.8 13.9 13.|-- static-94-32-135-182.clienti.tiscali.it 11.3% 1000 119.0 127.6 79.5 171.2 14.2 14.|-- static-213-205-54-210.clienti.tiscali.it 78.7% 1000 130.1 125.3 73.8 228.1 16.2 15.|-- ??? 100.0 1000 0.0 0.0 0.0 0.0 0.0 lukas@htznr2:~$ 2022-02-01T18:10:07Z https://pastebin.com/raw/MSvuSuR0 2022-02-02T17:39:44Z https://pastebin.com/raw/zWD8GHGa 2022-02-03T20:33:28Z (as above) https://pastebin.com/raw/DHWDyi1g 2022-02-04T13:11:51Z https://pastebin.com/raw/qhrWZBQX GTT NOC has been advised Feb 1st, 18:000Z, but apparently they always verify network issues in the morning, which is also when those links are uncongested, so at this point there is no confirmation from them. I'm assuming the network teams are aware of it anyway, but we don't know for sure. If you are using GTT for transit in Europe and traffic to Italy matters to your customers, you probably want to reroute (for example: reducing loc-pref based on GTT's BGP community 3257:53900 for prefixes learned in Italy). cheers, lukas

2 1

Spectrum Cable Internet South Texas
by John-Paul Reed 01 Feb '22

01 Feb '22

At approximately 11:30am CST i had multiple Spectrum Cable Internet customers in South Texas (San Antonio) go down. Anyone else seeing this issur? Feels like a BGP routing issue. Regards, John-Paul Reed TW Enterprises

1 0