It could be something else, but one of our sites is down because MySQL endpoint is unresponsive, and attempting to login to contact them panel.dreamhost.com has the same problem (gives either 503 or times out).
cloudflare is having issues again (https://www.cloudflarestatus.com/) -- probably unrelated but ::shrug:: On Thu, Nov 2, 2023 at 7:16 AM James Lawrie via Outages <outages@outages.org> wrote:
It could be something else, but one of our sites is down because MySQL endpoint is unresponsive, and attempting to login to contact them panel.dreamhost.com has the same problem (gives either 503 or times out). _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
All my sites, hosted on one of their VPSes is down hard. I can't SSH in, and it's not responding to pings. According to their status page, their data center on Portland is down hard: https://www.dreamhoststatus.com/ On Thu, Nov 2, 2023 at 7:27 AM Max Leonard via Outages <outages@outages.org> wrote:
cloudflare is having issues again (https://www.cloudflarestatus.com/) -- probably unrelated but ::shrug::
On Thu, Nov 2, 2023 at 7:16 AM James Lawrie via Outages < outages@outages.org> wrote:
It could be something else, but one of our sites is down because MySQL endpoint is unresponsive, and attempting to login to contact them panel.dreamhost.com has the same problem (gives either 503 or times out). _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Flexential PDX02 is entirely down right now. Full power outage. -- Tom Strickx Principal Network Engineer AS13335 - Cloudflare On Thu, Nov 2, 2023 at 12:31 PM Bob Colon via Outages <outages@outages.org> wrote:
All my sites, hosted on one of their VPSes is down hard. I can't SSH in, and it's not responding to pings. According to their status page, their data center on Portland is down hard: https://www.dreamhoststatus.com/
On Thu, Nov 2, 2023 at 7:27 AM Max Leonard via Outages < outages@outages.org> wrote:
cloudflare is having issues again (https://www.cloudflarestatus.com/) -- probably unrelated but ::shrug::
On Thu, Nov 2, 2023 at 7:16 AM James Lawrie via Outages < outages@outages.org> wrote:
It could be something else, but one of our sites is down because MySQL endpoint is unresponsive, and attempting to login to contact them panel.dreamhost.com has the same problem (gives either 503 or times out). _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We're in Flexential PDX02, I have guys on-site now. Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in progress for some portion of the site. I still have ~70 racks without power....but things are coming back to life slowly. On Thu, Nov 2, 2023 at 6:01 AM Tom Strickx via Outages <outages@outages.org> wrote:
Flexential PDX02 is entirely down right now. Full power outage. -- Tom Strickx Principal Network Engineer AS13335 - Cloudflare
On Thu, Nov 2, 2023 at 12:31 PM Bob Colon via Outages <outages@outages.org> wrote:
All my sites, hosted on one of their VPSes is down hard. I can't SSH in, and it's not responding to pings. According to their status page, their data center on Portland is down hard: https://www.dreamhoststatus.com/
On Thu, Nov 2, 2023 at 7:27 AM Max Leonard via Outages < outages@outages.org> wrote:
cloudflare is having issues again (https://www.cloudflarestatus.com/) -- probably unrelated but ::shrug::
On Thu, Nov 2, 2023 at 7:16 AM James Lawrie via Outages < outages@outages.org> wrote:
It could be something else, but one of our sites is down because MySQL endpoint is unresponsive, and attempting to login to contact them panel.dreamhost.com has the same problem (gives either 503 or times out). _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I can only imagine the series of failures that led to that outcome. That’s wild. Resume Generating Event for someone I’d assume. -- Christopher Conley Systems Administrator | Fors Marsh 4250 N Fairfax Dr, Suite 520 Arlington, VA 22203 From: Outages <outages-bounces@outages.org> On Behalf Of Matt Morgensen via Outages Sent: Thursday, November 2, 2023 9:09 AM To: Outages <outages@outages.org> Subject: [EXTERNAL] Re: [outages] Dreamhost MySQL outage? We're in Flexential PDX02, I have guys on-site now. Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in progress for some portion of the site. I still have ~70 racks without power....but things are coming back to life slowly. On Thu, Nov 2, 2023 at 6:01 AM Tom Strickx via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Flexential PDX02 is entirely down right now. Full power outage. -- Tom Strickx Principal Network Engineer AS13335 - Cloudflare On Thu, Nov 2, 2023 at 12:31 PM Bob Colon via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: All my sites, hosted on one of their VPSes is down hard. I can't SSH in, and it's not responding to pings. According to their status page, their data center on Portland is down hard: https://www.dreamhoststatus.com/ On Thu, Nov 2, 2023 at 7:27 AM Max Leonard via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: cloudflare is having issues again (https://www.cloudflarestatus.com/) -- probably unrelated but ::shrug:: On Thu, Nov 2, 2023 at 7:16 AM James Lawrie via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: It could be something else, but one of our sites is down because MySQL endpoint is unresponsive, and attempting to login to contact them panel.dreamhost.com<http://panel.dreamhost.com> has the same problem (gives either 503 or times out). _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future.
That means a series of parallel gens failed “all together”. The RFO for this is going to be uncomfortable. On Thu, Nov 2, 2023 at 09:10 Matt Morgensen via Outages <outages@outages.org> wrote:
We're in Flexential PDX02, I have guys on-site now.
Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in progress for some portion of the site. I still have ~70 racks without power....but things are coming back to life slowly.
On Thu, Nov 2, 2023 at 6:01 AM Tom Strickx via Outages < outages@outages.org> wrote:
Flexential PDX02 is entirely down right now. Full power outage. -- Tom Strickx Principal Network Engineer AS13335 - Cloudflare
On Thu, Nov 2, 2023 at 12:31 PM Bob Colon via Outages < outages@outages.org> wrote:
All my sites, hosted on one of their VPSes is down hard. I can't SSH in, and it's not responding to pings. According to their status page, their data center on Portland is down hard: https://www.dreamhoststatus.com/
On Thu, Nov 2, 2023 at 7:27 AM Max Leonard via Outages < outages@outages.org> wrote:
cloudflare is having issues again (https://www.cloudflarestatus.com/) -- probably unrelated but ::shrug::
On Thu, Nov 2, 2023 at 7:16 AM James Lawrie via Outages < outages@outages.org> wrote:
It could be something else, but one of our sites is down because MySQL endpoint is unresponsive, and attempting to login to contact them panel.dreamhost.com has the same problem (gives either 503 or times out). _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
The RFO is irrelevant. The damage is done. Maybe there are questions? Most important is when you got back in business. The rest is an almost total waste of time. The public stuff will be image oriented. It has to be. Privately, you all may get more mileage. Contractually? If you know you know. Many can/should/will get compensation for the damage. I would. Whether credits or benefits at renewal time. This may be worth exploiting as a customer. After all, these aren't supposed to go down. Hope it never happens to (me)! If it does, this is the script. - How did all generators fail together? - How did all transfer switches fail to shift from utility to generator together? - How did all the UPS fail all together? - Why did the facility come back in pieces? - Are you sure you're really an N+ facility and not an N facility? Show me. Sign new SLA - Why did it take an hour to restore "any" power? - How will this not happen again? - How do you plan to make me whole? They know at the end of the day this sucks. And they are embarrassed. They really do care. Bunch of good people. However, who's leaving customer wise? That's calculus. Answer? Almost none of you. :) YMMV -M< On Fri, Nov 3, 2023 at 12:45 PM Zach Hanna <chezbut@gmail.com> wrote:
I was told it was the transfer switches, not the gensets.
But waiting for the RFO
Sent from my iPhone
On Nov 2, 2023, at 5:26 PM, Martin Hannigan via Outages < outages@outages.org> wrote:
That means a series of parallel gens failed “all together”. The RFO for this is going to be uncomfortable.
On Thu, Nov 2, 2023 at 09:10 Matt Morgensen via Outages < outages@outages.org> wrote:
We're in Flexential PDX02, I have guys on-site now.
Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in progress for some portion of the site. I still have ~70 racks without power....but things are coming back to life slowly.
On Thu, Nov 2, 2023 at 6:01 AM Tom Strickx via Outages < outages@outages.org> wrote:
Flexential PDX02 is entirely down right now. Full power outage. -- Tom Strickx Principal Network Engineer AS13335 - Cloudflare
On Thu, Nov 2, 2023 at 12:31 PM Bob Colon via Outages < outages@outages.org> wrote:
All my sites, hosted on one of their VPSes is down hard. I can't SSH in, and it's not responding to pings. According to their status page, their data center on Portland is down hard: https://www.dreamhoststatus.com/
On Thu, Nov 2, 2023 at 7:27 AM Max Leonard via Outages < outages@outages.org> wrote:
cloudflare is having issues again (https://www.cloudflarestatus.com/) -- probably unrelated but ::shrug::
On Thu, Nov 2, 2023 at 7:16 AM James Lawrie via Outages < outages@outages.org> wrote:
It could be something else, but one of our sites is down because MySQL endpoint is unresponsive, and attempting to login to contact them panel.dreamhost.com has the same problem (gives either 503 or times out). _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On 11/3/23 5:27 PM, Martin Hannigan via Outages wrote:
Maybe there are questions?
https://blog.cloudflare.com/post-mortem-on-cloudflare-control-plane-and-anal... That has some info. -- Bryan Fields 727-409-1194 - Voice http://bryanfields.net
Counter to best practices, Flexential did not inform Cloudflare that they had failed over to generator power. Off to a good start, then... It is also unusual that Flexential ran both the one remaining utility feed and the generators at the same time... we haven't gotten a clear answer why they ran utility power and generator power. Yeah, there's a reason the power company tells homeowners to not improvise by backfeeding their house from a generator using a "suicide cord" when the linemen are working outside. You're supposed to install a cutover switch, or at least turn off your house main circuit breaker. Some of what follows is informed speculation based on the most likely series of events as well as what individual Flexential employees have shared with us unofficially. Oh boy, this is about to get spicy... One possible reason they may have left the utility line running is because Flexential was part of a program with PGE called DSG ... [which] allows the local utility to run a data center's generators to help supply additional power to the grid. In exchange, the power company helps maintain the generators and supplies fuel. We have been unable to locate any record of Flexential informing us about the DSG program. We've asked if DSG was active at the time and have not received an answer. You can't ask what you don't know, but it seems like power generation is one of those important things that should be told to your single largest customer who is leasing 10% of your entire facility. At approximately 11:40 UTC, there was a ground fault on a PGE transformer at PDX-04... [and] ground faults with high voltage (12,470 volt) power lines are very bad. That's underselling it a bit. Fortunately ... PDX-04 also contains a bank of UPS batteries... [that] are supposedly sufficient to power the facility for approximately 10 minutes... In reality, the batteries started to fail after only 4 minutes ... and it took Flexential far longer than 10 minutes to get the generators restored. Correct me if I'm wrong, but aren't UPS batteries supposed to be exercised with deep-cycling on a regular basis? It sounds like they were extremely worn out when they were needed most. While we haven't gotten official confirmation, we have been told by employees that [the generators] needed to be physically accessed and manually restarted because of the way the ground fault had tripped circuits. Second, Flexential's access control system was not powered by the battery backups, so it was offline. That sounds objectively dumber than what happened at the Meta/Facebook datacenter outage a while ago, where the doors and badge readers were still online, but the badges couldn't be evaluated via the network due to the BGP crash, and the credentials weren't cached locally either. And third, the overnight staffing at the site did not include an experienced operations or electrical expert — the overnight shift consisted of security and an unaccompanied technician who had only been on the job for a week. :picard-facepalm: Throughout this, Flexential never informed Cloudflare that there was any issue at the facility. [We] attempted to contact Flexential and dispatched our local team to physically travel to the facility. Adele: "Hello from the outsiiiiide..." "We have a number of questions that we need answered from Flexential." Understatement of the year. They must be seething. Cloudflare's report here is fairly even-handed and appears to have been fact-checked as well as possible under the circumstances, with corroborated statements from anonymous employees. Having read the technical stack in the document and their plans to beef up disaster recovery, Cloudflare has my utmost respect for quickly acknowledging and apologizing for the fact that they didn't assert and require that new services would be fully capable of active, redundant operation in the event of catastrophic service loss at their primary datacenter—a site which they believed to be reliable and indefatigable. They had planned many disaster exercises to combat the loss of PDX-04, but not in the event of a complete loss of power in excess of 10 minutes, or even 4 with shoddy batteries. To quote Ricky Ricardo, Flexential has some 'splainin' to do. -Brad —Sent from my iPhone On Nov 4, 2023, at 10:30 PM, Bryan Fields via Outages <outages@outages.org> wrote: On 11/3/23 5:27 PM, Martin Hannigan via Outages wrote: Maybe there are questions? https://urldefense.com/v3/__https://blog.cloudflare.com/post-mortem-on-cloud... That has some info. -- Bryan Fields 727-409-1194 - Voice https://urldefense.com/v3/__http://bryanfields.net__;!!PIZeeW5wscynRQ!rIKcGF... _______________________________________________ Outages mailing list Outages@outages.org https://urldefense.com/v3/__https://puck.nether.net/mailman/listinfo/outages...
On 11/5/23 3:24 AM, Chapman, Brad (NBCUniversal) wrote:
Counter to best practices, Flexential did not inform Cloudflare that they had failed over to generator power.
Off to a good start, then...
It's proper to prefix text that's being quoted with a "> "? It's super confusing to read your reply between, trying to separate your content from cloud flair's and then top posting too. Adding [EXTERNAL] to the subject doesn't make sense either.
It is also unusual that Flexential ran both the one remaining utility feed and the generators at the same time... we haven't gotten a clear answer why they ran utility power and generator power.
Yeah, there's a reason the power company tells homeowners to not improvise by backfeeding their house from a generator using a "suicide cord" when the linemen are working outside. You're supposed to install a cutover switch, or at least turn off your house main circuit breaker.
It would appear with DSG there is a bit more to it than a "suicide cord". When you own your own substation, as a datacenter does and have a common bus between them you have all the gear to match phase, detect faults and transfer at full load. It would appear that something in this other substation didn't clear the fault in time and allowed it to come over on the secondary side and fault the generators. I'd probably say DSG is a good idea, but only when there's not a failure on the utility side. And I'm no expert, so take my comments with a grain of salt.
One possible reason they may have left the utility line running is because Flexential was part of a program with PGE called DSG ... [which] allows the local utility to run a data center's generators to help supply additional power to the grid. In exchange, the power company helps maintain the generators and supplies fuel. We have been unable to locate any record of Flexential informing us about the DSG program. We've asked if DSG was active at the time and have not received an answer.
You can't ask what you don't know, but it seems like power generation is one of those important things that should be told to your single largest customer who is leasing 10% of your entire facility
Here's a link to the DSG faq. Looks like it's a nice way to exercise your gensets.
https://assets.ctfassets.net/416ywc1laqmd/6xPDM0LVfZrHyuzUbQAMeD/04f107a741a...
At approximately 11:40 UTC, there was a ground fault on a PGE transformer at PDX-04... [and] ground faults with high voltage (12,470 volt) power lines are very bad.
That's underselling it a bit.
Fortunately ... PDX-04 also contains a bank of UPS batteries... [that] are supposedly sufficient to power the facility for approximately 10 minutes... In reality, the batteries started to fail after only 4 minutes ... and it took Flexential far longer than 10 minutes to get the generators restored.
Correct me if I'm wrong, but aren't UPS batteries supposed to be exercised with deep-cycling on a regular basis? It sounds like they were extremely worn out when they were needed most.
While we haven't gotten official confirmation, we have been told by employees that [the generators] needed to be physically accessed and manually restarted because of the way the ground fault had tripped circuits. Second, Flexential's access control system was not powered by the battery backups, so it was offline.
That sounds objectively dumber than what happened at the Meta/Facebook datacenter outage a while ago, where the doors and badge readers were still online, but the badges couldn't be evaluated via the network due to the BGP crash, and the credentials weren't cached locally either.
It's almost impossible to buy off the shelf access control solutions that don't suck. Windows 10 home edition is a common server platform... Even if the system failed open and you have to go reset breakers on gensets manually, it's a good run to it and then time to put on PPE (440v at thousand of amps is where arc fault becomes real) before you can even reset it. This is assuming you were able to correctly diagnose the reason for the fault. 4-5 min is not long enough here, and I'd argue even 15 is too little. All this assumes the right people are on site too.
And third, the overnight staffing at the site did not include an experienced operations or electrical expert — the overnight shift consisted of security and an unaccompanied technician who had only been on the job for a week.
:picard-facepalm:
The overnight team isn't going to be the senior people, anywhere. -- Bryan Fields 727-409-1194 - Voice http://bryanfields.net
----- Original Message -----
From: "Bryan Fields via Outages" <outages@outages.org>
On 11/5/23 3:24 AM, Chapman, Brad (NBCUniversal) wrote:
Counter to best practices, Flexential did not inform Cloudflare that they had failed over to generator power.
Off to a good start, then...
It's proper to prefix text that's being quoted with a "> "? It's super confusing to read your reply between, trying to separate your content from cloud flair's and then top posting too. Adding [EXTERNAL] to the subject doesn't make sense either.
But *lots* of mail clients don't quote that way anymore -- including the one Brad uses, clearly -- and he *did* mark the quotes; your editor just flattened the indent/italics he used for quoting. I don't like it either, but I was never able to do anything about the kidnapping of the word 'hacker' either... <admin> When possible, though, we do recommend that members of the lists configure their mail clients to treat the list as flat-ASCII or ISO-8859(-1), with no styling, and use visible quote marking. </admin>
It is also unusual that Flexential ran both the one remaining utility feed and the generators at the same time... we haven't gotten a clear answer why they ran utility power and generator power.
Yeah, there's a reason the power company tells homeowners to not improvise by backfeeding their house from a generator using a "suicide cord" when the linemen are working outside. You're supposed to install a cutover switch, or at least turn off your house main circuit breaker.
It would appear with DSG there is a bit more to it than a "suicide cord". When you own your own substation, as a datacenter does and have a common bus between them you have all the gear to match phase, detect faults and transfer at full load. It would appear that something in this other substation didn't clear the fault in time and allowed it to come over on the secondary side and fault the generators.
I'd probably say DSG is a good idea, but only when there's not a failure on the utility side. And I'm no expert, so take my comments with a grain of salt.
I believe this is more commonly called 'cogeneration', and indeed, the controls that the utility permits you to use to do it are pretty smart, able to tell if the utility side drops, and cutting it out.
You can't ask what you don't know, but it seems like power generation is one of those important things that should be told to your single largest customer who is leasing 10% of your entire facility
Here's a link to the DSG faq. Looks like it's a nice way to exercise your gensets.
https://assets.ctfassets.net/416ywc1laqmd/6xPDM0LVfZrHyuzUbQAMeD/04f107a741a...
A smaller case of cogen, yeah. On point, though, I too would expect that if I was 10% of the entire leasing base, that I would get a bit more information; my guy should be on the NOC's call list.
While we haven't gotten official confirmation, we have been told by employees that [the generators] needed to be physically accessed and manually restarted because of the way the ground fault had tripped circuits. Second, Flexential's access control system was not powered by the battery backups, so it was offline.
That sounds objectively dumber than what happened at the Meta/Facebook datacenter outage a while ago, where the doors and badge readers were still online, but the badges couldn't be evaluated via the network due to the BGP crash, and the credentials weren't cached locally either.
It's almost impossible to buy off the shelf access control solutions that don't suck. Windows 10 home edition is a common server platform...
Yeesh...
Even if the system failed open and you have to go reset breakers on gensets manually, it's a good run to it and then time to put on PPE (440v at thousand of amps is where arc fault becomes real) before you can even reset it. This is assuming you were able to correctly diagnose the reason for the fault. 4-5 min is not long enough here, and I'd argue even 15 is too little. All this assumes the right people are on site too.
Sounds about right. I wouldn't plan for less than an hour, myself, from either side of the poker table.
And third, the overnight staffing at the site did not include an experienced operations or electrical expert — the overnight shift consisted of security and an unaccompanied technician who had only been on the job for a week.
:picard-facepalm:
The overnight team isn't going to be the senior people, anywhere.
You bet. :-) Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
The Internet is so fragile. On Sun, Nov 5, 2023 at 01:30 Bryan Fields via Outages <outages@outages.org> wrote:
On 11/3/23 5:27 PM, Martin Hannigan via Outages wrote:
Maybe there are questions?
https://blog.cloudflare.com/post-mortem-on-cloudflare-control-plane-and-anal...
That has some info. -- Bryan Fields
727-409-1194 - Voice http://bryanfields.net
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (11)
-
Bob Colon -
Bryan Fields -
Chapman, Brad (NBCUniversal) -
Christopher Conley -
James Lawrie -
Jay R. Ashworth -
Martin Hannigan -
Matt Morgensen -
Max Leonard -
Tom Strickx -
Zach Hanna