November 2025 - Outages - Outages Lists

AWS Monday outage AAR
by Jay R. Ashworth 30 Dec '25

30 Dec '25

Thanks to Brad Chapman for catching and excerpting this; please discuss on his thread in -discuss. """ It looks like the "Post-Event Summary" for this outage has been published. I've excerpted the one-sentence root cause analysis (emphasis mine): https://aws.amazon.com/message/101925/ The root cause of this issue was a latent race condition in the DynamoDB DNS management system that resulted in an incorrect empty DNS record for the service’s regional endpoint (dynamodb.us-east-1.amazonaws.com) that the automation failed to repair. """ Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

3 2

Multiple Atlassian services are seeing outages - 21 November 2025
by Andy Ringsmuth 21 Nov '25

21 Nov '25

Just received this a few minutes ago. Sent to Outages at 8:59 a.m. CST with a Reply-To: pointed to -discussion. ---- Andy Ringsmuth andy(a)andyring.com Love others, encourage others, help others. > Begin forwarded message: > > From: noreply(a)statuspage.io > Subject: Trello Incident - Multiple Atlassian services are seeing outages - 21 November 2025 > Date: November 21, 2025 at 8:56:26 AM CST > To: andy(a)newslink.com > > Trello > Multiple Atlassian services are seeing outages > New incident: Investigating > Multiple Atlassian services are seeing outages and we are investigating the same. We shall keep you posted on the progress in 60 minutes, if not sooner > Time posted > Nov 21, 09:55 EST > Components affected > Trello.com <http://trello.com/> > API > Atlassian Support - Support Portal > Atlassian Support Ticketing > Atlassian Support Knowledge Base > View full incident details <https://stspg.io/yg3lhj1qr390> > You received this email because you are subscribed to Trello's service status notifications. > > Manage your subscription <https://trello.status.atlassian.com/subscriptions/nsdbcxcrx372/edit> > Powered by Atlassian Statuspage <https://www.atlassian.com/software/statuspage?utm_source=email&utm_medium=p…>

1 0

issues with Cloudflare we have Solution
by serversgate.com＠gmail.com 19 Nov '25

19 Nov '25

issues with Cloudflare we have solutions you can change your own custom DNS with servers gate provider https://www.serversgate.com

8 9

Re: issues with Cloudflare?
by char＠char.blog 18 Nov '25

18 Nov '25

Looks like global issues, 500 responses and dashboard/aip issues; https://www.cloudflarestatus.com/incidents/8gmgl950y3h7 -- ++char

3 2

issues with Cloudflare?
by Joly MacFie 18 Nov '25

18 Nov '25

https://challenges.cloudflare.com/ [image: image.png] -- Joly MacFie Whatsapp +12185659365

12 12

Spectrum Los Angeles Fiber Cut
by jd＠boldnetworks.net 13 Nov '25

13 Nov '25

no specific location, master ticket ECT-85260554, estimated repair 7-8 more hours as of 930 pm pacific. can't see much down but one location and it started around 645 pm pacific.

1 0

Re: Issue with AT&T customers reaching at least one of our IPs
by Anthony Hoppe 13 Nov '25

13 Nov '25

Hello Outages List Members, First and foremost, I apologize for my mis-use of the intention of this list. I realize this is not an outage from that perspective. I purposefully chose to post to the main list as to try to enhance my chances of my plea making it to someone who can attempt to act. I have received several private replies, which I appreciate very much, and that was really the intention of my post which I should have clearly stated up front. Because there is no known process that I can find for an AS that does not have a direct relationship with AT&T to reach out when there are problems like this is frustrating (we would really like to know if there is one). I hope you can understand, and forgive me, for my misuse of the intention of the list. No further replies from me will be sent to this thread. If replying, please do it privately moving forward. Thanks! ~ Anthony -- Anthony Hoppe Sr. Network Engineer ahoppe(a)backblaze.com Backblaze.com On Thu, Nov 13, 2025 at 12:20 AM Steve Haering <steve(a)thc-lab.net> wrote: > > Evening, friends, > > Please feel free to reach out if outages.org decides it is time to have a more robust application/stack, instead of a distribution list. I can make a quick, cheap, haphazard webapp that will facilitate handling things like this in...a different way where everyone on the distro list doesn't get the email chiding the initial user about their usage. I can't promise it will be a good webapp - but it might be better than effectively tagging @everyone for what amounts to a triage issue. > > This is a cool list, appreciate all you do. Except for this particular email. 🙂 > > Much love, > THC > > > ________________________________ > From: J. Hellenthal via Outages <outages(a)outages.org> > Sent: Wednesday, November 12, 2025 7:16 PM > To: Outages Reporting <outages(a)outages.org> > Cc: Anthony Hoppe <ahoppe(a)backblaze.com>; Allan Paschall <APaschall(a)bankofsw.com>; noc <noc(a)nitelusa.com>; Outages Reporting <outages(a)outages.org>; J. Hellenthal <jhellenthal(a)dataix.net> > Subject: [Outages] Re: Issue with AT&T customers reaching at least one of our IPs > > This isn't an outage! > > > Take it to -discussion > -- > J. Hellenthal > > The fact that there's a highway to Hell but only a stairway to Heaven says a lot about anticipated traffic volume. > > > On Nov 12, 2025, at 17:52, noc via Outages <outages(a)outages.org> wrote: > > > > Hello, > > > > Thank you for contacting the Nitel NOC. In order to properly open a case, we need to identify the affected service in our system. Please provide us with the following information to locate the service: > > > > > > * NIT/HC Circuit ID: > > * Location Address: > > > > > > Additionally, in order to start the troubleshooting process, please confirm the following information: > > > > > > * Local Contact Name/Phone Number: > > * Site Access Hours: > > > > > > Our team is committed to resolving your issue as efficiently as possible. > > > > If you require immediate assistance, don't hesitate to contact us directly at (866) 892-0915. > > > > Thank you, > > > > > > Jafeth Quesada > > > > NOC Analyst ll > > > > 866.892.0915 > > > > > > > > Comcast Business > > > > Philadelphia, PA > > > > > > > > [A blue text on a black background Description automatically generated] > > > > > > > > Nitel is now part of Comcast Business. > > > > ________________________________ > > From: Allan Paschall via Outages <outages(a)outages.org> > > Sent: Wednesday, November 12, 2025 1:36 PM > > To: Outages (Planned & Unplanned) Reporting <outages(a)outages.org> > > Cc: Anthony Hoppe <ahoppe(a)backblaze.com>; Allan Paschall <APaschall(a)bankofsw.com> > > Subject: [Outages] Re: Issue with AT&T customers reaching at least one of our IPs > > > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. > > > > Anthony, > > > > I see that your IP address is on this blacklist: https://matrix.spfbl.net/ > > > > Below is the message reason given: > > > > A domain is considered non-compliant when the WHOIS search result for that domain does not contain the email address of the domain owner. Update the registration data and remove privacy protection for this domain in WHOIS and wait one hour for the cached result of this WHOIS query to expire. > > > > This IP was flagged due to misconfiguration of the e-mail service or the suspicion that there is no MTA at it. > > > > > > The link below should take you to the message and the instructions to have the blacklist removed. I hope this helps! > > > > https://matrix.spfbl.net/en/149.137.136.16 > > > > Allan Paschall > > > > > > > > > > From: Anthony Hoppe via Outages <outages(a)outages.org> > > Date: Wednesday, November 12, 2025 at 12:21 PM > > To: outages(a)outages.org <outages(a)outages.org> > > Cc: Anthony Hoppe <ahoppe(a)backblaze.com> > > Subject: [Outages] Issue with AT&T customers reaching at least one of our IPs > > > > [This message originated from outside Bank of the Southwest] > > > > Hello List, > > > > I need help getting in touch with someone helpful at AT&T (NetEng > > and/or trust & safety/abuse/compliance/security or preferably all the > > above). Starting on November 6th, customers behind AT&T have been > > unable to reach our IP address 149.137.136.16 which is the API VIP we > > use to serve downloads from Backblaze B2 buckets. > > > > If you go one up (.17) or one down (.15) no issues. > > > > I've tried calling the NOC contact per PeeringDB and they were > > ultimately unhelpful. Because we do not peer directly with AT&T they > > will not talk to us. > > > > I can confirm the reachability issue via AT&T's own public route > > server (https://urldefense.proofpoint.com/v2/url?u=http-3A__route-2Dserver.ip.att.n…) where I can ping 149.137.136.15 and > > .17 successfully, but cannot ping 149.137.136.16. MTRs from customers > > seem to show the path dies well before us. Here are a couple of > > examples that show 149.137.136.15 good and 149.137.136.16 bad: > > > > This one is from Michigan: > > > > $ mtr -w 149.137.136.15 > > Password: > > Start: 2025-11-12T13:39:10-0500 > > HOST: ip-192-168-0-231.ec2.internal Loss% Snt Last > > Avg Best Wrst StDev > > 1.|-- ip-192-168-0-1.ec2.internal 0.0% 10 > > 4.0 2.9 2.5 4.0 0.4 > > 2.|-- ip-192-168-1-254.ec2.internal 0.0% 10 > > 3.3 3.3 2.6 3.5 0.2 > > 3.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__108-2D202-2D236-2D1.lig… 0.0% 10 > > 5.4 5.0 4.2 6.5 0.8 > > 4.|-- 71.151.199.86 0.0% 10 > > 6.5 5.8 4.8 6.9 0.6 > > 5.|-- ??? 100.0 10 > > 0.0 0.0 0.0 0.0 0.0 > > 6.|-- ??? 100.0 10 > > 0.0 0.0 0.0 0.0 0.0 > > 7.|-- ??? 100.0 10 > > 0.0 0.0 0.0 0.0 0.0 > > 8.|-- ??? 100.0 10 > > 0.0 0.0 0.0 0.0 0.0 > > 9.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__ae3.cr12-2Dchi1.ip4.gtt… 0.0% 10 > > 14.7 14.8 13.7 15.9 0.6 > > 10.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__ae4.cr3-2Dwas1.ip4.gtt.… 0.0% 10 > > 34.7 34.9 34.0 35.4 0.4 > > 11.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__ip4.gtt.net&d=DwICAg&c=… 0.0% 10 > > 34.7 34.6 33.7 35.7 0.5 > > 12.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__api005.backblazeb2.com&… 0.0% 10 > > 34.8 35.1 34.1 37.5 0.9 > > > > > > $ mtr -w 149.137.136.16 > > Start: 2025-11-12T13:39:54-0500 > > HOST: ip-192-168-0-231.ec2.internal Loss% Snt Last > > Avg Best Wrst StDev > > 1.|-- ip-192-168-0-1.ec2.internal 0.0% 10 > > 2.5 2.8 2.5 3.0 0.1 > > 2.|-- ip-192-168-1-254.ec2.internal 0.0% 10 > > 3.4 3.5 3.0 3.8 0.2 > > 3.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__108-2D202-2D236-2D1.lig… 0.0% 10 > > 4.0 5.3 3.9 8.9 1.6 > > 4.|-- ??? 100.0 10 > > 0.0 0.0 0.0 0.0 0.0 > > > > This one is from East Bay, California: > > > > $ mtr -w 149.137.136.15 > > Start: 2025-11-11T14:54:50-0800 > > HOST: maureen Loss% Snt Last > > Avg Best Wrst StDev > > 1.|-- router.lan 0.0% 10 > > 0.6 0.6 0.4 0.8 0.1 > > 2.|-- 192.168.1.254 0.0% 10 > > 1.6 1.5 1.4 1.7 0.1 > > 3.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__99-2D189-2D116-2D1.ligh… 0.0% 10 > > 19.7 19.9 19.6 20.6 0.3 > > 4.|-- 71.148.180.125 0.0% 10 > > 20.7 20.2 20.0 20.7 0.3 > > 5.|-- ??? 100.0 10 > > 0.0 0.0 0.0 0.0 0.0 > > 6.|-- 32.130.91.83 40.0% 10 > > 23.1 34.2 22.3 91.9 28.3 > > 7.|-- 192.205.32.98 0.0% 10 > > 29.4 24.9 23.1 30.2 2.6 > > 8.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__ae4.cr3-2Dwas1.ip4.gtt.… 0.0% 10 > > 81.9 81.9 81.6 82.5 0.3 > > 9.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__ip4.gtt.net&d=DwICAg&c=… 0.0% 10 > > 82.0 82.0 81.7 82.5 0.2 > > 10.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__api005.backblazeb2.com&… 0.0% 10 > > 82.4 82.1 81.3 82.9 0.4 > > > > $ mtr -w 149.137.136.16 > > Start: 2025-11-11T13:49:14-0800 > > HOST: maureen Loss% Snt Last > > Avg Best Wrst StDev > > 1.|-- router.lan 0.0% 10 > > 0.7 0.7 0.5 0.7 0.1 > > 2.|-- 192.168.1.254 0.0% 10 > > 1.5 1.5 1.4 1.5 0.1 > > 3.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__99-2D189-2D116-2D1.ligh… 0.0% 10 > > 20.3 20.1 19.6 20.8 0.3 > > 4.|-- ??? 100.0 10 > > 0.0 0.0 0.0 0.0 0.0 > > > > If there is anybody on this list that can throw me a bone, it would be > > very much appreciated. > > > > -- > > > > Anthony Hoppe > > Sr. Network Engineer > > ahoppe(a)backblaze.com > > https://urldefense.proofpoint.com/v2/url?u=http-3A__Backblaze.com&d=DwICAg&… > > > > -- > > This email, including its contents and any attachment(s), may contain > > confidential and/or proprietary information and is solely for the review > > and use of the intended recipient(s). If you have received this email in > > error, please notify the sender and permanently delete this email, its > > content, and any attachment(s). Any disclosure, copying, or taking of any > > action in reliance on an email received in error is strictly prohibited. > > ______________________________________________ > > Outages mailing list outages(a)outages.org > > Sign up for an account https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.outages.org_acco… > > To subscribe send an email to outages-join(a)outages.org > > To unsubscribe send an email to outages-leave(a)outages.org > > To contact the list owners outages-owner(a)outages.org > > Archives https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.outages.org_arch… > > > > Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion(a)outages.org > > NOTICE: The information contained in this message is confidential and for the sole use of the intended recipient/s. If you are not the intended recipient, please (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. > > ______________________________________________ > > Outages mailing list outages(a)outages.org > > Sign up for an account https://lists.outages.org/accounts/signup/ > > To subscribe send an email to outages-join(a)outages.org > > To unsubscribe send an email to outages-leave(a)outages.org > > To contact the list owners outages-owner(a)outages.org > > Archives https://lists.outages.org/archives/list/outages@outages.org/ > > > > Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion(a)outages.org > > > > ________________________________ > > > > CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. > > Thank you. > > ______________________________________________ > > Outages mailing list outages(a)outages.org > > Sign up for an account https://lists.outages.org/accounts/signup/ > > To subscribe send an email to outages-join(a)outages.org > > To unsubscribe send an email to outages-leave(a)outages.org > > To contact the list owners outages-owner(a)outages.org > > Archives https://lists.outages.org/archives/list/outages@outages.org/ > > > > Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion(a)outages.org > ______________________________________________ > Outages mailing list outages(a)outages.org > Sign up for an account https://lists.outages.org/accounts/signup/ > To subscribe send an email to outages-join(a)outages.org > To unsubscribe send an email to outages-leave(a)outages.org > To contact the list owners outages-owner(a)outages.org > Archives https://lists.outages.org/archives/list/outages@outages.org/ > > Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion(a)outages.org -- This email, including its contents and any attachment(s), may contain confidential and/or proprietary information and is solely for the review and use of the intended recipient(s). If you have received this email in error, please notify the sender and permanently delete this email, its content, and any attachment(s). Any disclosure, copying, or taking of any action in reliance on an email received in error is strictly prohibited.

1 0

Issue with AT&T customers reaching at least one of our IPs
by Anthony Hoppe 12 Nov '25

12 Nov '25

Hello List, I need help getting in touch with someone helpful at AT&T (NetEng and/or trust & safety/abuse/compliance/security or preferably all the above). Starting on November 6th, customers behind AT&T have been unable to reach our IP address 149.137.136.16 which is the API VIP we use to serve downloads from Backblaze B2 buckets. If you go one up (.17) or one down (.15) no issues. I've tried calling the NOC contact per PeeringDB and they were ultimately unhelpful. Because we do not peer directly with AT&T they will not talk to us. I can confirm the reachability issue via AT&T's own public route server (route-server.ip.att.net) where I can ping 149.137.136.15 and .17 successfully, but cannot ping 149.137.136.16. MTRs from customers seem to show the path dies well before us. Here are a couple of examples that show 149.137.136.15 good and 149.137.136.16 bad: This one is from Michigan: $ mtr -w 149.137.136.15 Password: Start: 2025-11-12T13:39:10-0500 HOST: ip-192-168-0-231.ec2.internal Loss% Snt Last Avg Best Wrst StDev 1.|-- ip-192-168-0-1.ec2.internal 0.0% 10 4.0 2.9 2.5 4.0 0.4 2.|-- ip-192-168-1-254.ec2.internal 0.0% 10 3.3 3.3 2.6 3.5 0.2 3.|-- 108-202-236-1.lightspeed.livnmi.sbcglobal.net 0.0% 10 5.4 5.0 4.2 6.5 0.8 4.|-- 71.151.199.86 0.0% 10 6.5 5.8 4.8 6.9 0.6 5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 9.|-- ae3.cr12-chi1.ip4.gtt.net 0.0% 10 14.7 14.8 13.7 15.9 0.6 10.|-- ae4.cr3-was1.ip4.gtt.net 0.0% 10 34.7 34.9 34.0 35.4 0.4 11.|-- ip4.gtt.net 0.0% 10 34.7 34.6 33.7 35.7 0.5 12.|-- api005.backblazeb2.com 0.0% 10 34.8 35.1 34.1 37.5 0.9 $ mtr -w 149.137.136.16 Start: 2025-11-12T13:39:54-0500 HOST: ip-192-168-0-231.ec2.internal Loss% Snt Last Avg Best Wrst StDev 1.|-- ip-192-168-0-1.ec2.internal 0.0% 10 2.5 2.8 2.5 3.0 0.1 2.|-- ip-192-168-1-254.ec2.internal 0.0% 10 3.4 3.5 3.0 3.8 0.2 3.|-- 108-202-236-1.lightspeed.livnmi.sbcglobal.net 0.0% 10 4.0 5.3 3.9 8.9 1.6 4.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 This one is from East Bay, California: $ mtr -w 149.137.136.15 Start: 2025-11-11T14:54:50-0800 HOST: maureen Loss% Snt Last Avg Best Wrst StDev 1.|-- router.lan 0.0% 10 0.6 0.6 0.4 0.8 0.1 2.|-- 192.168.1.254 0.0% 10 1.6 1.5 1.4 1.7 0.1 3.|-- 99-189-116-1.lightspeed.snmtca.sbcglobal.net 0.0% 10 19.7 19.9 19.6 20.6 0.3 4.|-- 71.148.180.125 0.0% 10 20.7 20.2 20.0 20.7 0.3 5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 6.|-- 32.130.91.83 40.0% 10 23.1 34.2 22.3 91.9 28.3 7.|-- 192.205.32.98 0.0% 10 29.4 24.9 23.1 30.2 2.6 8.|-- ae4.cr3-was1.ip4.gtt.net 0.0% 10 81.9 81.9 81.6 82.5 0.3 9.|-- ip4.gtt.net 0.0% 10 82.0 82.0 81.7 82.5 0.2 10.|-- api005.backblazeb2.com 0.0% 10 82.4 82.1 81.3 82.9 0.4 $ mtr -w 149.137.136.16 Start: 2025-11-11T13:49:14-0800 HOST: maureen Loss% Snt Last Avg Best Wrst StDev 1.|-- router.lan 0.0% 10 0.7 0.7 0.5 0.7 0.1 2.|-- 192.168.1.254 0.0% 10 1.5 1.5 1.4 1.5 0.1 3.|-- 99-189-116-1.lightspeed.snmtca.sbcglobal.net 0.0% 10 20.3 20.1 19.6 20.8 0.3 4.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 If there is anybody on this list that can throw me a bone, it would be very much appreciated. -- Anthony Hoppe Sr. Network Engineer ahoppe(a)backblaze.com Backblaze.com -- This email, including its contents and any attachment(s), may contain confidential and/or proprietary information and is solely for the review and use of the intended recipient(s). If you have received this email in error, please notify the sender and permanently delete this email, its content, and any attachment(s). Any disclosure, copying, or taking of any action in reliance on an email received in error is strictly prohibited.

7 6

ATT wireless down in NW Ohio
by Tim Reichhart 01 Nov '25

01 Nov '25

Is anybody else seeing ATT wireless in NW ohio having issues with service being connected -- Tim Reichhart Owner of Northwest Ohio Broadband LLC Phone number: 419-678-0178 Website: http://nwohiobb.com

1 0