- Outages-discussion - Outages Lists

Big fiber cut near 111 8th?
by Bryan Holloway 02 Mar '26

02 Mar '26

Took out Cogent and Windstream for us; one going to Chicago; the other to Atlanta. Cogent is saying Verizon has a fiber-cut. Curious if others are seeing the same or have heard anything else of note ... Thanks! - bryan

1 0

Re: [Outages] Re: AWS recovering a bit after losing AZ in me-central-1
by George Herbert 01 Mar '26

01 Mar '26

I believe this is the first time my day job in cloud stuff and second job analyzing foreign missile programs have ever overlapped directly. Wheee. Going to be a bad few weeks at this rate. -george Sent from my iPhone > On Mar 1, 2026, at 2:09 PM, Jason L. Sparks via Outages <outages(a)outages.org> wrote: > > As always, AWS status is available here: > https://health.aws.amazon.com/health/status > > Jason > >> On Sun, Mar 1, 2026 at 14:34 Mel Beckman via Outages <outages(a)outages.org> >> wrote: >> >> The other ISP was kind enough to forward to me the AWS event history. I >> don’t know if this is published publicly anywhere online, but I haven’t >> been able to locate it. I would like to know if anyone else finds it so I >> can follow for updates. >> >> >> Affected AWS services >> >> The following AWS services have been affected by this issue. >> >> Disrupted (1 service) >> >> Amazon Elastic Compute Cloud >> >> >> History >> >> [09:41 AM PST] We want to provide some additional information on the power >> issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around >> 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by >> objects that struck the data center, creating sparks and fire. The fire >> department shut off power to the facility and generators as they worked to >> put out the fire. We are still awaiting permission to turn the power back >> on, and once we have, we will ensure we restore power and connectivity >> safely. It will take several hours to restore connectivity to the impacted >> AZ. The other AZs in the region are functioning normally. Customers who >> were running their applications redundantly across the AZs are not impacted >> by this event. EC2 Instance launches will continue to be impaired in the >> impacted AZ. We recommend that customers continue to retry any failed API >> requests. If immediate recovery of an affected resource (EC2 Instance, EBS >> Volume, RDS DB Instance, etc.) is required, we recommend restoring from >> your most recent backup, by launching replacement resources in one of the >> unaffected zones, or an alternate AWS Region. We will provide an update by >> 12:30 PM PST, or sooner if we have additional information to share. >> >> >> [08:59 AM PST] We continue to work toward restoring power in the affected >> Availability Zone in the ME-CENTRAL-1 Region (mec1-az2). In parallel, we >> are actively working on improving error rates and latencies that some >> customers are observing for EC2 Networking and EC2 Describe APIs. Due to >> increased demand in the unaffected Availability Zones, customers may >> experience longer than usual provisioning times or may need to retry >> requests for certain instance types, or pick an alternative instance type. >> We will provide an update by 10:30 AM PST, or sooner if we have additional >> information to share. >> >> >> [07:09 AM PST] We wanted to provide some additional information on the >> isolated power issue. At this time, most AWS Services have weighted away >> from the affected Availability Zone (mec1-az2) and are seeing recovery for >> their affected operations and workflows. For EC2 Instances, EBS Volumes, >> and other resources that are impacted in the affected Zone, we will have a >> longer tail of recovery. At this time, power has not yet been restored to >> the affected AZ. For now, we recommend continuing to retry any failed API >> requests. If immediate recovery is required, we recommend customers restore >> from EBS Snapshots and/or replace affected resources by launching >> replacement resources in one of the unaffected zones, or an alternate >> region. As of this time, recovery is still several hours away. We will >> provide an update by 8:30 AM PST, or sooner if we have additional >> information to share. >> >> >> [06:09 AM PST] We can confirm that a localized power issue has affected a >> single Availability Zone in the ME-CENTRAL-1 Region (mec1-az2). EC2 >> Instances, DB Instances, EBS Volumes, and others resources are currently >> unavailable and will experience connectivity issues at this time. Other AWS >> Services are also experiencing error rates and latencies for some >> workflows. We have weighed away traffic for most services at this time. We >> recommend customers utilize one of the other Availability Zones in the >> ME-CENTRAL-1 Region at this time, as existing instances in other AZ's >> remain unaffected by this issue. We are actively working to restore power >> and connectivity, at which time we will begin to work to recover affected >> resources. As of this time, we expect recovery is multiple hours away. We >> will provide an update by 7:15 AM PST, or sooner if we have additional >> information to share. >> >> >> [05:19 AM PST] We are investigating connectivity and power issues >> affecting APIs and instances in a single Availability Zone (mec1-az2) in >> the ME-CENTRAL-1 Region due to a localized power issue. Existing instances >> in this zone will also be affected. Other AWS Services may also be >> experiencing increased errors and latencies for their workflows, and we are >> working to route requests away from this affected Availability Zone. We >> recommend customers make use of other Availability Zones at this time. >> Targeting new launches using RunInstances in the remaining AZs should >> succeed. Existing instances in the other AZs are not affected. [04:51 AM >> PST] We are investigating issues with AWS services in the ME-CENTRAL-1 >> Region. >> >> >> -mel beckman >> >> On Mar 1, 2026, at 9:59 AM, Mel Beckman <mel(a)beckman.org> wrote: >> >> And I also note that my instances are offline. A customer had called >> their ISP thinking it was their problem, and their ISP told him that since >> 4:30 AM PST AWS had a major outage due to an object that struck a data >> center creating sparks and fire. As a result, the power was shut off to the >> facility, including backup power. They had no ETA on the outage. I’m in the >> AZ2 zone. >> >> -mel via cell >> >> On Mar 1, 2026, at 9:52 AM, Mel Beckman <mel(a)beckman.org> wrote: >> >> I can’t even sign into my AWS Console. It says “an AWS account with that >> sign in information does not exist” when I’ve only put in the email >> address. Not only is this a fundamental security violation, because you >> should never disclose that an account doesn’t exist when only the account >> ID is entered, but it’s plain scary that I can’t get to any of my AWS >> infrastructure to see what’s going on. >> >> -mel via cell >> >> On Mar 1, 2026, at 9:21 AM, Paul M - outages via Outages < >> outages(a)outages.org> wrote: >> >> this is affecting managed services like OpenSearch, as well as basic >> services like EC2. >> >> we are finding that AWS CLI will tell us a resource exists, such as >> EC2, but we cannot access the resources, e.g. ssh to the instance. >> ______________________________________________ >> Outages mailing list outages(a)outages.org >> Sign up for an account https://lists.outages.org/accounts/signup/ >> To subscribe send an email to outages-join(a)outages.org >> To unsubscribe send an email to outages-leave(a)outages.org >> To contact the list owners outages-owner(a)outages.org >> Archives https://lists.outages.org/archives/list/outages@outages.org/ >> >> Thank you for using Outages Lists! For post outage discussion, please >> redirect to outages-discussion(a)outages.org >> >> ______________________________________________ >> Outages mailing list outages(a)outages.org >> Sign up for an account https://lists.outages.org/accounts/signup/ >> To subscribe send an email to outages-join(a)outages.org >> To unsubscribe send an email to outages-leave(a)outages.org >> To contact the list owners outages-owner(a)outages.org >> Archives https://lists.outages.org/archives/list/outages@outages.org/ >> >> Thank you for using Outages Lists! For post outage discussion, please >> redirect to outages-discussion(a)outages.org >> ______________________________________________ >> Outages mailing list outages(a)outages.org >> Sign up for an account https://lists.outages.org/accounts/signup/ >> To subscribe send an email to outages-join(a)outages.org >> To unsubscribe send an email to outages-leave(a)outages.org >> To contact the list owners outages-owner(a)outages.org >> Archives https://lists.outages.org/archives/list/outages@outages.org/ >> >> Thank you for using Outages Lists! For post outage discussion, please >> redirect to outages-discussion(a)outages.org > ______________________________________________ > Outages mailing list outages(a)outages.org > Sign up for an account https://lists.outages.org/accounts/signup/ > To subscribe send an email to outages-join(a)outages.org > To unsubscribe send an email to outages-leave(a)outages.org > To contact the list owners outages-owner(a)outages.org > Archives https://lists.outages.org/archives/list/outages@outages.org/ > > Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion(a)outages.org

1 0

Re: [Outages] Re: AWS recovering a bit after losing AZ in me-central-1
by Brad Chapman 01 Mar '26

01 Mar '26

I think it's fair to assume that any service hosted in the Middle East is going to experience moderate to severe colitis disruption over the next several days weeks months years, depending on how long this "special military operation" drags on. -Brad —Sent from my iPhone > On Mar 1, 2026, at 10:23 AM, Paul M - outages via Outages <outages(a)outages.org> wrote: > > at least one of the Dubai airports was attached; the Dubai World > Central Al Maktoum International airport is not far from the AWS > datacentre, so power lines and other infrastructure could be > collateral damage > ______________________________________________ > Outages mailing list outages(a)outages.org > Sign up for an account https://lists.outages.org/accounts/signup/ > To subscribe send an email to outages-join(a)outages.org > To unsubscribe send an email to outages-leave(a)outages.org > To contact the list owners outages-owner(a)outages.org > Archives https://lists.outages.org/archives/list/outages@outages.org/ > > Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion(a)outages.org

2 1

Re: AT&T outage escalation
by Eric C. Miller 28 Feb '26

28 Feb '26

The chat is saying that no agents are available. We're trying additional chats in parallel. Also trying to check through business center. Eric ________________________________ From: virendra rode <vrode(a)outages.org> Sent: Saturday, February 28, 2026 6:28:54 PM To: Post outages meta-discussion (troubleshooting, analysis, post-mortem, etc) <outages-discussion(a)outages.org> Cc: Eric C. Miller <eric(a)ericheather.com> Subject: Re: [Outages-discussion] AT&T outage escalation On Sat, 2026-02-28 at 22:48 +0000, Eric C. Miller via Outages- discussion wrote: > Hello, does anybody have a working outage escalation list? Our > contacts aren't answering on the weekend and express ticketing isn't > letting us create tickets. It seems that nobody staffs their call > center numbers anymore either. > > Thank you! > > Eric ------- Are you not able to get through to the duty manager? Try using the chat feature. /vrode > > > > > ______________________________________________ > Outages-discussion mailing list outages-discussion(a)outages.org > Sign up for an account https://lists.outages.org/accounts/signup/ > To subscribe send an email to outages-discussion-join(a)outages.org > To unsubscribe send an email to outages-discussion-leave(a)outages.org > To contact the list owners outages-owner(a)outages.org > Archives > https://lists.outages.org/archives/list/outages-discussion@outages.org/ > > Thank you for using outages-discussion Lists!

1 0

AT&T outage escalation
by Eric C. Miller 28 Feb '26

28 Feb '26

Hello, does anybody have a working outage escalation list? Our contacts aren't answering on the weekend and express ticketing isn't letting us create tickets. It seems that nobody staffs their call center numbers anymore either. Thank you! Eric

2 1

Re: [Outages] Re: Cloudflare down again?
by jay＠west.net 20 Feb '26

20 Feb '26

On 2/20/26 10:47, Eric C. Miller via Outages wrote: > We just experienced a huge BGP churn in LAX. Moving to -discussion. It looks like the main mailing list is very laggy about now and we'll see the fallout in the next couple of hours. Coresite acknowledged a bridging loop on Any2 Exchange at One Wilshire. -- Jay Hennigan - jay(a)west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV

1 0

YouTube Auth Outage
by Andrew Latham 18 Feb '26

18 Feb '26

So me and apparently others can't log in at YouTube for the past 40ish minutes. Location is Colorado - Andrew "lathama" Latham -

5 4

Re: [Outages] Cloudflare/Charter issues in Cleveland?
by Gert Doering 10 Feb '26

10 Feb '26

Hi, On Tue, Feb 10, 2026 at 02:32:44PM -0500, Josh Luthman wrote: > And yet here I am doing it. Why you would consider a DNS timeout to be > meaningless is beyond me. Are you a troll or are you completely ignorant? Something in your measurement is physically impossible. How can the DNS query return in 10ms if your network packets need 20ms (without the time to do the actual DNS), as shown by your own "mtr -u"? *shrug* gert -- "If was one thing all people took for granted, was conviction that if you feed honest figures into a computer, honest figures come out. Never doubted it myself till I met a computer with a sense of humor." Robert A. Heinlein, The Moon is a Harsh Mistress Gert Doering - Munich, Germany gert(a)greenie.muc.de

1 0

Re: [Outages] Cloudflare/Charter issues in Cleveland?
by Gert Doering 10 Feb '26

10 Feb '26

Hi, On Tue, Feb 10, 2026 at 01:59:48PM -0500, Josh Luthman wrote: > That is not what I said. You completely changed my words in your quote. > What I actually said was: *DNS is timing out* ***OR*** *~10ms response > times* (using dig google.com @1.1.1.1 a few time manually) "10ms response times" are hardly achievable if your network latency is well beyond that. So the data you provided is sort of... meaningless. gert -- "If was one thing all people took for granted, was conviction that if you feed honest figures into a computer, honest figures come out. Never doubted it myself till I met a computer with a sense of humor." Robert A. Heinlein, The Moon is a Harsh Mistress Gert Doering - Munich, Germany gert(a)greenie.muc.de

1 0

Re: [Outages] Cloudflare/Charter issues in Cleveland?
by Gert Doering 10 Feb '26

10 Feb '26

Hi, On Tue, Feb 10, 2026 at 12:13:52PM -0500, Josh Luthman via Outages wrote: > Nothing on https://www.cloudflarestatus.com/ > > My dns alert for 1.1.1.1 has been red since Tue Feb 10 10:24:06 2026 > (Eastern). > > DNS is timing out or ~10ms response times (using dig google.com @1.1.1.1 a > few time manually) Your network latency to 1.1.1.1 is 15-20ms, so having a threshold on "if it's slower than 10ms, it's broken" sounds like "well, yes, it will be broken forever, but that's self-inflicted pain". Not sure where we should see an outage with sufficient impact for the list members here. gert -- "If was one thing all people took for granted, was conviction that if you feed honest figures into a computer, honest figures come out. Never doubted it myself till I met a computer with a sense of humor." Robert A. Heinlein, The Moon is a Harsh Mistress Gert Doering - Munich, Germany gert(a)greenie.muc.de

1 0