Re: [Outages] Re: AWS recovering a bit after losing AZ in me-central-1
I believe this is the first time my day job in cloud stuff and second job analyzing foreign missile programs have ever overlapped directly. Wheee. Going to be a bad few weeks at this rate. -george Sent from my iPhone
On Mar 1, 2026, at 2:09 PM, Jason L. Sparks via Outages <outages@outages.org> wrote:
As always, AWS status is available here: https://health.aws.amazon.com/health/status
Jason
On Sun, Mar 1, 2026 at 14:34 Mel Beckman via Outages <outages@outages.org> wrote:
The other ISP was kind enough to forward to me the AWS event history. I don’t know if this is published publicly anywhere online, but I haven’t been able to locate it. I would like to know if anyone else finds it so I can follow for updates.
Affected AWS services
The following AWS services have been affected by this issue.
Disrupted (1 service)
Amazon Elastic Compute Cloud
History
[09:41 AM PST] We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the fire. We are still awaiting permission to turn the power back on, and once we have, we will ensure we restore power and connectivity safely. It will take several hours to restore connectivity to the impacted AZ. The other AZs in the region are functioning normally. Customers who were running their applications redundantly across the AZs are not impacted by this event. EC2 Instance launches will continue to be impaired in the impacted AZ. We recommend that customers continue to retry any failed API requests. If immediate recovery of an affected resource (EC2 Instance, EBS Volume, RDS DB Instance, etc.) is required, we recommend restoring from your most recent backup, by launching replacement resources in one of the unaffected zones, or an alternate AWS Region. We will provide an update by 12:30 PM PST, or sooner if we have additional information to share.
[08:59 AM PST] We continue to work toward restoring power in the affected Availability Zone in the ME-CENTRAL-1 Region (mec1-az2). In parallel, we are actively working on improving error rates and latencies that some customers are observing for EC2 Networking and EC2 Describe APIs. Due to increased demand in the unaffected Availability Zones, customers may experience longer than usual provisioning times or may need to retry requests for certain instance types, or pick an alternative instance type. We will provide an update by 10:30 AM PST, or sooner if we have additional information to share.
[07:09 AM PST] We wanted to provide some additional information on the isolated power issue. At this time, most AWS Services have weighted away from the affected Availability Zone (mec1-az2) and are seeing recovery for their affected operations and workflows. For EC2 Instances, EBS Volumes, and other resources that are impacted in the affected Zone, we will have a longer tail of recovery. At this time, power has not yet been restored to the affected AZ. For now, we recommend continuing to retry any failed API requests. If immediate recovery is required, we recommend customers restore from EBS Snapshots and/or replace affected resources by launching replacement resources in one of the unaffected zones, or an alternate region. As of this time, recovery is still several hours away. We will provide an update by 8:30 AM PST, or sooner if we have additional information to share.
[06:09 AM PST] We can confirm that a localized power issue has affected a single Availability Zone in the ME-CENTRAL-1 Region (mec1-az2). EC2 Instances, DB Instances, EBS Volumes, and others resources are currently unavailable and will experience connectivity issues at this time. Other AWS Services are also experiencing error rates and latencies for some workflows. We have weighed away traffic for most services at this time. We recommend customers utilize one of the other Availability Zones in the ME-CENTRAL-1 Region at this time, as existing instances in other AZ's remain unaffected by this issue. We are actively working to restore power and connectivity, at which time we will begin to work to recover affected resources. As of this time, we expect recovery is multiple hours away. We will provide an update by 7:15 AM PST, or sooner if we have additional information to share.
[05:19 AM PST] We are investigating connectivity and power issues affecting APIs and instances in a single Availability Zone (mec1-az2) in the ME-CENTRAL-1 Region due to a localized power issue. Existing instances in this zone will also be affected. Other AWS Services may also be experiencing increased errors and latencies for their workflows, and we are working to route requests away from this affected Availability Zone. We recommend customers make use of other Availability Zones at this time. Targeting new launches using RunInstances in the remaining AZs should succeed. Existing instances in the other AZs are not affected. [04:51 AM PST] We are investigating issues with AWS services in the ME-CENTRAL-1 Region.
-mel beckman
On Mar 1, 2026, at 9:59 AM, Mel Beckman <mel@beckman.org> wrote:
And I also note that my instances are offline. A customer had called their ISP thinking it was their problem, and their ISP told him that since 4:30 AM PST AWS had a major outage due to an object that struck a data center creating sparks and fire. As a result, the power was shut off to the facility, including backup power. They had no ETA on the outage. I’m in the AZ2 zone.
-mel via cell
On Mar 1, 2026, at 9:52 AM, Mel Beckman <mel@beckman.org> wrote:
I can’t even sign into my AWS Console. It says “an AWS account with that sign in information does not exist” when I’ve only put in the email address. Not only is this a fundamental security violation, because you should never disclose that an account doesn’t exist when only the account ID is entered, but it’s plain scary that I can’t get to any of my AWS infrastructure to see what’s going on.
-mel via cell
On Mar 1, 2026, at 9:21 AM, Paul M - outages via Outages < outages@outages.org> wrote:
this is affecting managed services like OpenSearch, as well as basic services like EC2.
we are finding that AWS CLI will tell us a resource exists, such as EC2, but we cannot access the resources, e.g. ssh to the instance. ______________________________________________ Outages mailing list outages@outages.org Sign up for an account https://lists.outages.org/accounts/signup/ To subscribe send an email to outages-join@outages.org To unsubscribe send an email to outages-leave@outages.org To contact the list owners outages-owner@outages.org Archives https://lists.outages.org/archives/list/outages@outages.org/
Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion@outages.org
______________________________________________ Outages mailing list outages@outages.org Sign up for an account https://lists.outages.org/accounts/signup/ To subscribe send an email to outages-join@outages.org To unsubscribe send an email to outages-leave@outages.org To contact the list owners outages-owner@outages.org Archives https://lists.outages.org/archives/list/outages@outages.org/
Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion@outages.org ______________________________________________ Outages mailing list outages@outages.org Sign up for an account https://lists.outages.org/accounts/signup/ To subscribe send an email to outages-join@outages.org To unsubscribe send an email to outages-leave@outages.org To contact the list owners outages-owner@outages.org Archives https://lists.outages.org/archives/list/outages@outages.org/
Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion@outages.org
Outages mailing list outages@outages.org Sign up for an account https://lists.outages.org/accounts/signup/ To subscribe send an email to outages-join@outages.org To unsubscribe send an email to outages-leave@outages.org To contact the list owners outages-owner@outages.org Archives https://lists.outages.org/archives/list/outages@outages.org/
Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion@outages.org
participants (1)
-
George Herbert