Can’t believe it’s still dead… -Mike
On Mar 26, 2023, at 17:13, T.Suzuki via Outages <outages@outages.org> wrote:
On Sun, 26 Mar 2023 08:35:29 -0700 Hugo Slabbert <hugo@slabnet.com> wrote:
What would be the symptoms here of a "water torture attack" rather than what John had indicated as a firewall failure in their infrastructure:
Initial looks from the firewall team point to an automatic failover event and the secondary failed.
And the symptoms of which lined up with network level info from Paul earlier:
They only seem to have two auth nameservers for faa, both within the faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the servers are in all die just within each block run by the FAA.
Seems like an internal routing meltdown making the only 2 nameservers unreachable reliably.
Are you saying that your open resolvers have a per client rate limit applied, that rate limit got tripped, and shortly thereafter the resolvers became unavailable, suggesting query floods for the domain(s) that knocked the resolvers offline (or from the other discussion, possibly was the thing that overwhelmed that firewall layer, causing the initial failover and possibly also causing the firewall secondary to fail to come online)?
Yes. (limitting per client, and per second for all) Perhaps, large numbers open resolvers including no ratelimit are used. Then massive random subdomain queries caused the firewall symptoms. (It's only my guess.)
On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <outages@outages.org> wrote:
Hi, I'm a researcher of DNS vulnerabilities.
It loos like random subdomain attacks (water tourtue attack).
This is the data of my rate-limitted openresolver as a honeypot. http://www.e-ontap.com/dns/todaydowngov.txt http://www.e-ontap.com/dns/todaydown.txt (You can not view these page if you are using 8.8.8.8, sorry.)
Raw logs of my Unbound (Time is JST) local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head -5 Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL < unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210 unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45 Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN SERVFAIL 15.112813 0 30 Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | head -5 Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. from 2620:74:27::2:30 no server to query nameserver addresses not usable Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. upstream server timeout Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. upstream server timeout local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | tail -5 Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL < eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL < faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail -5 Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210 eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44 Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL < faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210 faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46 Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A IN SERVFAIL 0.000000 0 34 local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc -l 1408
-- T.Suzuki -- T.Suzuki / E.F.シューマッハーとI.イリイチを読もう _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- T.Suzuki / E.F.シューマッハーとI.イリイチを読もう _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages