- Outages-discussion - Outages Lists

Verizon issues on the east cost
by Dovid Bender 04 Aug '26

04 Aug '26

Anyone aware of VZ issues on the east coast? I personally had issues with Vz Mobile to Vz Mobile and I am having issues with calling to VZ Mobile from Sinch. From Vz to Sinch does seem to work

1 1

A pitch for dystopian thriller - full analysis of the telstra outage
by virendra rode 29 Jul '26

29 Jul '26

A modern world hit the brick wall and step backward in time. The level of technical forensics in this CommsDay report outlines the post-mortem of a massive communication failure traced to a single, localized point of origin. (Spoiler alert: an NTP stratum server was promoted from Stratum 3, a 'middle manager' meant to take orders from a higher-level Stratum 2 server and pass the time down the chain to generating its own time like a rogue CEO, breaking the chain of trust.) This was followed by a brutal Senate inquiry. In a nutshell, Telstra's internal culture allowed a cowboy engineer to bypass internal controls. A 15-year-old server taking down an entire network sounds eerily similar to the global Facebook outage. https://telsoc.org/news/telsoc-newsletter-commsday-story-of-the-week-17-2-2… An action replay of last Friday’s hearing can be accessed at, https://www.youtube.com/watch?v=p3WE_-C7pzY enjoy, /vrode

1 0

Sleeping time bombs in our networks
by virendra rode 13 Jul '26

13 Jul '26

With the recent Telstra outage last week—reportedly lasting about 12 hours and traced back to a single faulty NTP server that effectively sent parts of the network back 20 years—it was a painful reminder that even highly redundant networks can still harbor significant single points of failure. Ouch... especially for anything relying on security certificates. Anyone who has managed large-scale, mission-critical infrastructure knows that troubleshooting these types of outages is anything but easy. Redundancy doesn't eliminate complexity. When your NOC is fielding thousands of customer calls, executives are demanding status updates, and engineering teams are peeling back layers of overlays, routing protocols, and dependencies, diagnosing the root cause becomes incredibly intense—especially during a customer-facing outage with real revenue impact. It got me thinking: how many of us know there are significant single points of failure lurking in our own environments? I can think of two in a network I support today (which will remain nameless). So why do organizations knowingly leave these risks in place? The reality is that you can spend millions chasing the holy grail of five nines (99.999%) availability, but very few organizations have an unlimited budget. At some point, every engineering team has to make trade-offs based on cost, complexity, operational risk, and return on investment. Perfect networks don't exist. So I'm curious: What factors influence the decision to leave known single points of failure unaddressed?How do you determine when the cost and complexity of eliminating a risk outweigh the likelihood and impact of failure?How do you validate that your architecture is "good enough"?Does your NOC provide meaningful feedback that influences future network designs—such as recurring outages, convergence issues, scalability concerns, or operational pain points? I've always believed one of the biggest contributors to operational stability is standardization (aka deterministic). The fewer "snowflake" designs we build, the easier networks become to operate, troubleshoot, and evolve. Interested to hear how others approach these decisions. /vrode outages anarchist & masochist

1 0

ARTEMIS II SPLASHDOWN
by vrode＠outages.org 10 Apr '26

10 Apr '26

Welcome home! 73 /vrode

1 0

Big fiber cut near 111 8th?
by Bryan Holloway 02 Mar '26

02 Mar '26

Took out Cogent and Windstream for us; one going to Chicago; the other to Atlanta. Cogent is saying Verizon has a fiber-cut. Curious if others are seeing the same or have heard anything else of note ... Thanks! - bryan

1 0

Re: [Outages] Re: AWS recovering a bit after losing AZ in me-central-1
by George Herbert 01 Mar '26

01 Mar '26

I believe this is the first time my day job in cloud stuff and second job analyzing foreign missile programs have ever overlapped directly. Wheee. Going to be a bad few weeks at this rate. -george Sent from my iPhone > On Mar 1, 2026, at 2:09 PM, Jason L. Sparks via Outages <outages(a)outages.org> wrote: > > As always, AWS status is available here: > https://health.aws.amazon.com/health/status > > Jason > >> On Sun, Mar 1, 2026 at 14:34 Mel Beckman via Outages <outages(a)outages.org> >> wrote: >> >> The other ISP was kind enough to forward to me the AWS event history. I >> don’t know if this is published publicly anywhere online, but I haven’t >> been able to locate it. I would like to know if anyone else finds it so I >> can follow for updates. >> >> >> Affected AWS services >> >> The following AWS services have been affected by this issue. >> >> Disrupted (1 service) >> >> Amazon Elastic Compute Cloud >> >> >> History >> >> [09:41 AM PST] We want to provide some additional information on the power >> issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around >> 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by >> objects that struck the data center, creating sparks and fire. The fire >> department shut off power to the facility and generators as they worked to >> put out the fire. We are still awaiting permission to turn the power back >> on, and once we have, we will ensure we restore power and connectivity >> safely. It will take several hours to restore connectivity to the impacted >> AZ. The other AZs in the region are functioning normally. Customers who >> were running their applications redundantly across the AZs are not impacted >> by this event. EC2 Instance launches will continue to be impaired in the >> impacted AZ. We recommend that customers continue to retry any failed API >> requests. If immediate recovery of an affected resource (EC2 Instance, EBS >> Volume, RDS DB Instance, etc.) is required, we recommend restoring from >> your most recent backup, by launching replacement resources in one of the >> unaffected zones, or an alternate AWS Region. We will provide an update by >> 12:30 PM PST, or sooner if we have additional information to share. >> >> >> [08:59 AM PST] We continue to work toward restoring power in the affected >> Availability Zone in the ME-CENTRAL-1 Region (mec1-az2). In parallel, we >> are actively working on improving error rates and latencies that some >> customers are observing for EC2 Networking and EC2 Describe APIs. Due to >> increased demand in the unaffected Availability Zones, customers may >> experience longer than usual provisioning times or may need to retry >> requests for certain instance types, or pick an alternative instance type. >> We will provide an update by 10:30 AM PST, or sooner if we have additional >> information to share. >> >> >> [07:09 AM PST] We wanted to provide some additional information on the >> isolated power issue. At this time, most AWS Services have weighted away >> from the affected Availability Zone (mec1-az2) and are seeing recovery for >> their affected operations and workflows. For EC2 Instances, EBS Volumes, >> and other resources that are impacted in the affected Zone, we will have a >> longer tail of recovery. At this time, power has not yet been restored to >> the affected AZ. For now, we recommend continuing to retry any failed API >> requests. If immediate recovery is required, we recommend customers restore >> from EBS Snapshots and/or replace affected resources by launching >> replacement resources in one of the unaffected zones, or an alternate >> region. As of this time, recovery is still several hours away. We will >> provide an update by 8:30 AM PST, or sooner if we have additional >> information to share. >> >> >> [06:09 AM PST] We can confirm that a localized power issue has affected a >> single Availability Zone in the ME-CENTRAL-1 Region (mec1-az2). EC2 >> Instances, DB Instances, EBS Volumes, and others resources are currently >> unavailable and will experience connectivity issues at this time. Other AWS >> Services are also experiencing error rates and latencies for some >> workflows. We have weighed away traffic for most services at this time. We >> recommend customers utilize one of the other Availability Zones in the >> ME-CENTRAL-1 Region at this time, as existing instances in other AZ's >> remain unaffected by this issue. We are actively working to restore power >> and connectivity, at which time we will begin to work to recover affected >> resources. As of this time, we expect recovery is multiple hours away. We >> will provide an update by 7:15 AM PST, or sooner if we have additional >> information to share. >> >> >> [05:19 AM PST] We are investigating connectivity and power issues >> affecting APIs and instances in a single Availability Zone (mec1-az2) in >> the ME-CENTRAL-1 Region due to a localized power issue. Existing instances >> in this zone will also be affected. Other AWS Services may also be >> experiencing increased errors and latencies for their workflows, and we are >> working to route requests away from this affected Availability Zone. We >> recommend customers make use of other Availability Zones at this time. >> Targeting new launches using RunInstances in the remaining AZs should >> succeed. Existing instances in the other AZs are not affected. [04:51 AM >> PST] We are investigating issues with AWS services in the ME-CENTRAL-1 >> Region. >> >> >> -mel beckman >> >> On Mar 1, 2026, at 9:59 AM, Mel Beckman <mel(a)beckman.org> wrote: >> >> And I also note that my instances are offline. A customer had called >> their ISP thinking it was their problem, and their ISP told him that since >> 4:30 AM PST AWS had a major outage due to an object that struck a data >> center creating sparks and fire. As a result, the power was shut off to the >> facility, including backup power. They had no ETA on the outage. I’m in the >> AZ2 zone. >> >> -mel via cell >> >> On Mar 1, 2026, at 9:52 AM, Mel Beckman <mel(a)beckman.org> wrote: >> >> I can’t even sign into my AWS Console. It says “an AWS account with that >> sign in information does not exist” when I’ve only put in the email >> address. Not only is this a fundamental security violation, because you >> should never disclose that an account doesn’t exist when only the account >> ID is entered, but it’s plain scary that I can’t get to any of my AWS >> infrastructure to see what’s going on. >> >> -mel via cell >> >> On Mar 1, 2026, at 9:21 AM, Paul M - outages via Outages < >> outages(a)outages.org> wrote: >> >> this is affecting managed services like OpenSearch, as well as basic >> services like EC2. >> >> we are finding that AWS CLI will tell us a resource exists, such as >> EC2, but we cannot access the resources, e.g. ssh to the instance. >> ______________________________________________ >> Outages mailing list outages(a)outages.org >> Sign up for an account https://lists.outages.org/accounts/signup/ >> To subscribe send an email to outages-join(a)outages.org >> To unsubscribe send an email to outages-leave(a)outages.org >> To contact the list owners outages-owner(a)outages.org >> Archives https://lists.outages.org/archives/list/outages@outages.org/ >> >> Thank you for using Outages Lists! For post outage discussion, please >> redirect to outages-discussion(a)outages.org >> >> ______________________________________________ >> Outages mailing list outages(a)outages.org >> Sign up for an account https://lists.outages.org/accounts/signup/ >> To subscribe send an email to outages-join(a)outages.org >> To unsubscribe send an email to outages-leave(a)outages.org >> To contact the list owners outages-owner(a)outages.org >> Archives https://lists.outages.org/archives/list/outages@outages.org/ >> >> Thank you for using Outages Lists! For post outage discussion, please >> redirect to outages-discussion(a)outages.org >> ______________________________________________ >> Outages mailing list outages(a)outages.org >> Sign up for an account https://lists.outages.org/accounts/signup/ >> To subscribe send an email to outages-join(a)outages.org >> To unsubscribe send an email to outages-leave(a)outages.org >> To contact the list owners outages-owner(a)outages.org >> Archives https://lists.outages.org/archives/list/outages@outages.org/ >> >> Thank you for using Outages Lists! For post outage discussion, please >> redirect to outages-discussion(a)outages.org > ______________________________________________ > Outages mailing list outages(a)outages.org > Sign up for an account https://lists.outages.org/accounts/signup/ > To subscribe send an email to outages-join(a)outages.org > To unsubscribe send an email to outages-leave(a)outages.org > To contact the list owners outages-owner(a)outages.org > Archives https://lists.outages.org/archives/list/outages@outages.org/ > > Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion(a)outages.org

1 0

Re: [Outages] Re: AWS recovering a bit after losing AZ in me-central-1
by Brad Chapman 01 Mar '26

01 Mar '26

I think it's fair to assume that any service hosted in the Middle East is going to experience moderate to severe colitis disruption over the next several days weeks months years, depending on how long this "special military operation" drags on. -Brad —Sent from my iPhone > On Mar 1, 2026, at 10:23 AM, Paul M - outages via Outages <outages(a)outages.org> wrote: > > at least one of the Dubai airports was attached; the Dubai World > Central Al Maktoum International airport is not far from the AWS > datacentre, so power lines and other infrastructure could be > collateral damage > ______________________________________________ > Outages mailing list outages(a)outages.org > Sign up for an account https://lists.outages.org/accounts/signup/ > To subscribe send an email to outages-join(a)outages.org > To unsubscribe send an email to outages-leave(a)outages.org > To contact the list owners outages-owner(a)outages.org > Archives https://lists.outages.org/archives/list/outages@outages.org/ > > Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion(a)outages.org

2 1

Re: AT&T outage escalation
by Eric C. Miller 28 Feb '26

28 Feb '26

The chat is saying that no agents are available. We're trying additional chats in parallel. Also trying to check through business center. Eric ________________________________ From: virendra rode <vrode(a)outages.org> Sent: Saturday, February 28, 2026 6:28:54 PM To: Post outages meta-discussion (troubleshooting, analysis, post-mortem, etc) <outages-discussion(a)outages.org> Cc: Eric C. Miller <eric(a)ericheather.com> Subject: Re: [Outages-discussion] AT&T outage escalation On Sat, 2026-02-28 at 22:48 +0000, Eric C. Miller via Outages- discussion wrote: > Hello, does anybody have a working outage escalation list? Our > contacts aren't answering on the weekend and express ticketing isn't > letting us create tickets. It seems that nobody staffs their call > center numbers anymore either. > > Thank you! > > Eric ------- Are you not able to get through to the duty manager? Try using the chat feature. /vrode > > > > > ______________________________________________ > Outages-discussion mailing list outages-discussion(a)outages.org > Sign up for an account https://lists.outages.org/accounts/signup/ > To subscribe send an email to outages-discussion-join(a)outages.org > To unsubscribe send an email to outages-discussion-leave(a)outages.org > To contact the list owners outages-owner(a)outages.org > Archives > https://lists.outages.org/archives/list/outages-discussion@outages.org/ > > Thank you for using outages-discussion Lists!

1 0

AT&T outage escalation
by Eric C. Miller 28 Feb '26

28 Feb '26

Hello, does anybody have a working outage escalation list? Our contacts aren't answering on the weekend and express ticketing isn't letting us create tickets. It seems that nobody staffs their call center numbers anymore either. Thank you! Eric

2 1

Re: [Outages] Re: Cloudflare down again?
by jay＠west.net 20 Feb '26

20 Feb '26

On 2/20/26 10:47, Eric C. Miller via Outages wrote: > We just experienced a huge BGP churn in LAX. Moving to -discussion. It looks like the main mailing list is very laggy about now and we'll see the fallout in the next couple of hours. Coresite acknowledged a bridging loop on Any2 Exchange at One Wilshire. -- Jay Hennigan - jay(a)west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV

1 0