Thanks to Brad Chapman for catching and excerpting this; please discuss on his
thread in -discuss.
"""
It looks like the "Post-Event Summary" for this outage has been published.
I've excerpted the one-sentence root cause analysis (emphasis mine):
https://aws.amazon.com/message/101925/
The root cause of this issue was a latent race condition in the DynamoDB DNS
management system that resulted in an incorrect empty DNS record for the
service’s regional endpoint (dynamodb.us-east-1.amazonaws.com) that the
automation failed to repair.
"""
Cheers,
-- jra
--
Jay R. Ashworth Baylink jra(a)baylink.com
Designer The Things I Think RFC 2100
Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII
St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
Is it just me? I can’t seem to reach any he.net sites. Just timing out.
On Comcast at home. Tried Verizon over the air. Through Zscaler… nothin’
from any of them.
Oh, looks like downdetector.com would agree with me.
Not sure if it extends beyond websites. Guess it’s time to check that too.
Widely reported, but not yet on Outages:
https://groups.google.com/a/list.nist.gov/g/internet-time-service/c/o0dDDcr…
jeff.sherman(a)nist.gov
2:18 AM (17 hours ago)
to Internet-time-service
Dear colleagues,
In short, the atomic ensemble time scale at our Boulder campus has failed due to a prolonged utility power outage. One impact is that the Boulder Internet Time Services no longer have an accurate time reference. At time of writing the Boulder servers are still available due a standby power generator, but I will attempt to disable them to avoid disseminating incorrect time.
The affected servers are:
time-a-b.nist.govtime-b-b.nist.govtime-c-b.nist.govtime-d-b.nist.govtime-e-b.nist.govntp-b.nist.gov (authenticated NTP)
No time to repair estimate is available until we regain staff access and power. Efforts are currently focused on obtaining an alternate source of power so the hydrogen maser clocks survive beyond their battery backups.
More details follow.
Due to prolonged high wind gusts there have been a combination of utility power line damage and preemptive utility shutdowns (in the interest of wildfire prevention) in the Boulder, CO area. NIST's campus lost utility power Wednesday (Dec. 17 2025) around 22:23 UTC. At time of writing utility power is still off to the campus. Facility operators anticipated needing to shutdown the heat-exchange infrastructure providing air cooling to many parts of the building, including some internal networking closets. As a result, many of these too were preemptively shutdown with the result that our group lacks much of the monitoring and control capabilities we ordinarily have. Also, the site has been closed to all but emergency personnel Thursday and Friday, and at time of writing remains closed.
At initial power loss, there was no immediate impact to the NIST atomic time scale or distribution services because the projects are afforded standby power generators. However, we now have strong evidence one of the crucial generators has failed. In the downstream path is the primary signal distribution chain, including to the Boulder Internet Time Service. Another campus building houses additional clocks backed up by a different power generator; if these survive it will allow us to re-align the primary time scale when site stability returns without making use of external clocks or reference signals.
Best wishes,
-Jeff Sherman
