March 2026 - Outages-discussion

Big fiber cut near 111 8th?
by Bryan Holloway 02 Mar '26

02 Mar '26

Took out Cogent and Windstream for us; one going to Chicago; the other to Atlanta. Cogent is saying Verizon has a fiber-cut. Curious if others are seeing the same or have heard anything else of note ... Thanks! - bryan

1 0

Re: [Outages] Re: AWS recovering a bit after losing AZ in me-central-1
by George Herbert 01 Mar '26

01 Mar '26

I believe this is the first time my day job in cloud stuff and second job analyzing foreign missile programs have ever overlapped directly. Wheee. Going to be a bad few weeks at this rate. -george Sent from my iPhone > On Mar 1, 2026, at 2:09 PM, Jason L. Sparks via Outages <outages(a)outages.org> wrote: > > As always, AWS status is available here: > https://health.aws.amazon.com/health/status > > Jason > >> On Sun, Mar 1, 2026 at 14:34 Mel Beckman via Outages <outages(a)outages.org> >> wrote: >> >> The other ISP was kind enough to forward to me the AWS event history. I >> don’t know if this is published publicly anywhere online, but I haven’t >> been able to locate it. I would like to know if anyone else finds it so I >> can follow for updates. >> >> >> Affected AWS services >> >> The following AWS services have been affected by this issue. >> >> Disrupted (1 service) >> >> Amazon Elastic Compute Cloud >> >> >> History >> >> [09:41 AM PST] We want to provide some additional information on the power >> issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around >> 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by >> objects that struck the data center, creating sparks and fire. The fire >> department shut off power to the facility and generators as they worked to >> put out the fire. We are still awaiting permission to turn the power back >> on, and once we have, we will ensure we restore power and connectivity >> safely. It will take several hours to restore connectivity to the impacted >> AZ. The other AZs in the region are functioning normally. Customers who >> were running their applications redundantly across the AZs are not impacted >> by this event. EC2 Instance launches will continue to be impaired in the >> impacted AZ. We recommend that customers continue to retry any failed API >> requests. If immediate recovery of an affected resource (EC2 Instance, EBS >> Volume, RDS DB Instance, etc.) is required, we recommend restoring from >> your most recent backup, by launching replacement resources in one of the >> unaffected zones, or an alternate AWS Region. We will provide an update by >> 12:30 PM PST, or sooner if we have additional information to share. >> >> >> [08:59 AM PST] We continue to work toward restoring power in the affected >> Availability Zone in the ME-CENTRAL-1 Region (mec1-az2). In parallel, we >> are actively working on improving error rates and latencies that some >> customers are observing for EC2 Networking and EC2 Describe APIs. Due to >> increased demand in the unaffected Availability Zones, customers may >> experience longer than usual provisioning times or may need to retry >> requests for certain instance types, or pick an alternative instance type. >> We will provide an update by 10:30 AM PST, or sooner if we have additional >> information to share. >> >> >> [07:09 AM PST] We wanted to provide some additional information on the >> isolated power issue. At this time, most AWS Services have weighted away >> from the affected Availability Zone (mec1-az2) and are seeing recovery for >> their affected operations and workflows. For EC2 Instances, EBS Volumes, >> and other resources that are impacted in the affected Zone, we will have a >> longer tail of recovery. At this time, power has not yet been restored to >> the affected AZ. For now, we recommend continuing to retry any failed API >> requests. If immediate recovery is required, we recommend customers restore >> from EBS Snapshots and/or replace affected resources by launching >> replacement resources in one of the unaffected zones, or an alternate >> region. As of this time, recovery is still several hours away. We will >> provide an update by 8:30 AM PST, or sooner if we have additional >> information to share. >> >> >> [06:09 AM PST] We can confirm that a localized power issue has affected a >> single Availability Zone in the ME-CENTRAL-1 Region (mec1-az2). EC2 >> Instances, DB Instances, EBS Volumes, and others resources are currently >> unavailable and will experience connectivity issues at this time. Other AWS >> Services are also experiencing error rates and latencies for some >> workflows. We have weighed away traffic for most services at this time. We >> recommend customers utilize one of the other Availability Zones in the >> ME-CENTRAL-1 Region at this time, as existing instances in other AZ's >> remain unaffected by this issue. We are actively working to restore power >> and connectivity, at which time we will begin to work to recover affected >> resources. As of this time, we expect recovery is multiple hours away. We >> will provide an update by 7:15 AM PST, or sooner if we have additional >> information to share. >> >> >> [05:19 AM PST] We are investigating connectivity and power issues >> affecting APIs and instances in a single Availability Zone (mec1-az2) in >> the ME-CENTRAL-1 Region due to a localized power issue. Existing instances >> in this zone will also be affected. Other AWS Services may also be >> experiencing increased errors and latencies for their workflows, and we are >> working to route requests away from this affected Availability Zone. We >> recommend customers make use of other Availability Zones at this time. >> Targeting new launches using RunInstances in the remaining AZs should >> succeed. Existing instances in the other AZs are not affected. [04:51 AM >> PST] We are investigating issues with AWS services in the ME-CENTRAL-1 >> Region. >> >> >> -mel beckman >> >> On Mar 1, 2026, at 9:59 AM, Mel Beckman <mel(a)beckman.org> wrote: >> >> And I also note that my instances are offline. A customer had called >> their ISP thinking it was their problem, and their ISP told him that since >> 4:30 AM PST AWS had a major outage due to an object that struck a data >> center creating sparks and fire. As a result, the power was shut off to the >> facility, including backup power. They had no ETA on the outage. I’m in the >> AZ2 zone. >> >> -mel via cell >> >> On Mar 1, 2026, at 9:52 AM, Mel Beckman <mel(a)beckman.org> wrote: >> >> I can’t even sign into my AWS Console. It says “an AWS account with that >> sign in information does not exist” when I’ve only put in the email >> address. Not only is this a fundamental security violation, because you >> should never disclose that an account doesn’t exist when only the account >> ID is entered, but it’s plain scary that I can’t get to any of my AWS >> infrastructure to see what’s going on. >> >> -mel via cell >> >> On Mar 1, 2026, at 9:21 AM, Paul M - outages via Outages < >> outages(a)outages.org> wrote: >> >> this is affecting managed services like OpenSearch, as well as basic >> services like EC2. >> >> we are finding that AWS CLI will tell us a resource exists, such as >> EC2, but we cannot access the resources, e.g. ssh to the instance. >> ______________________________________________ >> Outages mailing list outages(a)outages.org >> Sign up for an account https://lists.outages.org/accounts/signup/ >> To subscribe send an email to outages-join(a)outages.org >> To unsubscribe send an email to outages-leave(a)outages.org >> To contact the list owners outages-owner(a)outages.org >> Archives https://lists.outages.org/archives/list/outages@outages.org/ >> >> Thank you for using Outages Lists! For post outage discussion, please >> redirect to outages-discussion(a)outages.org >> >> ______________________________________________ >> Outages mailing list outages(a)outages.org >> Sign up for an account https://lists.outages.org/accounts/signup/ >> To subscribe send an email to outages-join(a)outages.org >> To unsubscribe send an email to outages-leave(a)outages.org >> To contact the list owners outages-owner(a)outages.org >> Archives https://lists.outages.org/archives/list/outages@outages.org/ >> >> Thank you for using Outages Lists! For post outage discussion, please >> redirect to outages-discussion(a)outages.org >> ______________________________________________ >> Outages mailing list outages(a)outages.org >> Sign up for an account https://lists.outages.org/accounts/signup/ >> To subscribe send an email to outages-join(a)outages.org >> To unsubscribe send an email to outages-leave(a)outages.org >> To contact the list owners outages-owner(a)outages.org >> Archives https://lists.outages.org/archives/list/outages@outages.org/ >> >> Thank you for using Outages Lists! For post outage discussion, please >> redirect to outages-discussion(a)outages.org > ______________________________________________ > Outages mailing list outages(a)outages.org > Sign up for an account https://lists.outages.org/accounts/signup/ > To subscribe send an email to outages-join(a)outages.org > To unsubscribe send an email to outages-leave(a)outages.org > To contact the list owners outages-owner(a)outages.org > Archives https://lists.outages.org/archives/list/outages@outages.org/ > > Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion(a)outages.org

1 0

Re: [Outages] Re: AWS recovering a bit after losing AZ in me-central-1
by Brad Chapman 01 Mar '26

01 Mar '26

I think it's fair to assume that any service hosted in the Middle East is going to experience moderate to severe colitis disruption over the next several days weeks months years, depending on how long this "special military operation" drags on. -Brad —Sent from my iPhone > On Mar 1, 2026, at 10:23 AM, Paul M - outages via Outages <outages(a)outages.org> wrote: > > at least one of the Dubai airports was attached; the Dubai World > Central Al Maktoum International airport is not far from the AWS > datacentre, so power lines and other infrastructure could be > collateral damage > ______________________________________________ > Outages mailing list outages(a)outages.org > Sign up for an account https://lists.outages.org/accounts/signup/ > To subscribe send an email to outages-join(a)outages.org > To unsubscribe send an email to outages-leave(a)outages.org > To contact the list owners outages-owner(a)outages.org > Archives https://lists.outages.org/archives/list/outages@outages.org/ > > Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion(a)outages.org

2 1