FAA.gov nameserver outage
Looks like faa.gov's nameservers are all having a bad time, only occasionally responding right now from multiple tests from my home network, datacenter POPs (Seattle, Chicago), and 8.8.8.8 -- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler
Oh my. Seeing the same from the Northeast. Get Outlook for Android<https://aka.ms/AAb9ysg> ________________________________ From: Outages <outages-bounces@outages.org> on behalf of Michael Loftis via Outages <outages@outages.org> Sent: Saturday, March 25, 2023 9:34:32 PM To: outages <outages@outages.org> Subject: [outages] FAA.gov nameserver outage Looks like faa.gov's nameservers are all having a bad time, only occasionally responding right now from multiple tests from my home network, datacenter POPs (Seattle, Chicago), and 8.8.8.8 -- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Seeing the same here across all our POPs. Has anyone notified FAA NOC yet? -- Sent from Gmail Mobile
Was down 60 seconds ago; checked https://nasstatus.faa.gov/ and that was up; seems main faa.gov is operational again. On Sat, Mar 25, 2023 at 9:51 PM Michael B. Williams via Outages < outages@outages.org> wrote:
Seeing the same here across all our POPs.
Has anyone notified FAA NOC yet? -- Sent from Gmail Mobile _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I don't know how one would contact them, but the list of the senior IT officers are on their memorandum to migrate to IPv6 dated September 2021. This memo states they would be migrating in 2023. Perhaps we're seeing evidence of a recent change? https://www.faa.gov/sites/faa.gov/files/about/office_org/headquarters_office... —Sent from my iPhone On Mar 25, 2023, at 6:53 PM, Michael B. Williams via Outages <outages@outages.org> wrote: Seeing the same here across all our POPs. Has anyone notified FAA NOC yet? -- Sent from Gmail Mobile _______________________________________________ Outages mailing list Outages@outages.org https://urldefense.com/v3/__https://puck.nether.net/mailman/listinfo/outages...
On Sun, 2023-03-26 at 02:00 +0000, Chapman, Brad (NBCUniversal) via Outages wrote:
I don't know how one would contact them, but the list of the senior IT officers are on their memorandum to migrate to IPv6 dated September 2021. This memo states they would be migrating in 2023. Perhaps we're seeing evidence of a recent change?
https://www.faa.gov/sites/faa.gov/files/about/office_org/headquarters_office...
—Sent from my iPhone
On Mar 25, 2023, at 6:53 PM, Michael B. Williams via Outages <outages@outages.org> wrote:
Seeing the same here across all our POPs.
Has anyone notified FAA NOC yet?
I did notice ipv6 path traversal...but now its totally dead
/vrode
On Sat, 2023-03-25 at 20:06 -0700, virendra rode via Outages wrote:
On Sun, 2023-03-26 at 02:00 +0000, Chapman, Brad (NBCUniversal) via Outages wrote:
I don't know how one would contact them, but the list of the senior IT officers are on their memorandum to migrate to IPv6 dated September 2021. This memo states they would be migrating in 2023. Perhaps we're seeing evidence of a recent change?
https://www.faa.gov/sites/faa.gov/files/about/office_org/headquarters_office...
—Sent from my iPhone
On Mar 25, 2023, at 6:53 PM, Michael B. Williams via Outages <outages@outages.org> wrote:
Seeing the same here across all our POPs.
Has anyone notified FAA NOC yet?
I did notice ipv6 path traversal...but now its totally dead
/vrode _______________________________________________ Outages mailing list Outages@outages.org
More proof, just so I'm not blowing smoke, traceroute to www.faa.gov (2600:1406:6c00:192::1256), 30 hops max, 80 byte packets 1 2603-8001-3b00-2dc7-0000-0000-0000-0001.res6.spectrum.com (2603:8001:3b00:2dc7::1) 15.618 ms 15.521 ms 15.473 ms 2 2603-90c5-0001-0927-0000-0000-0000-0001.inf6.spectrum.com (2603:90c5:1:927::1) 31.726 ms 19.029 ms 20.498 ms 3 lag-61.snaucaoszh1.netops.charter.com (2605:e000:0:4::7:37d) 21.835 ms 31.496 ms 31.462 ms 4 lag-22.cyprcabw02r.netops.charter.com (2605:e000:0:8::6:e) 31.500 ms 31.464 ms 38.891 ms 5 * * * 6 * * * 7 lag-1.pr2.lax10.netops.charter.com (2001:1998:0:4::559) 25.888 ms 24.790 ms lag-2.pr2.lax10.netops.charter.com (2001:1998:56::27) 32.321 ms 8 2001:1998:0:8::667 (2001:1998:0:8::667) 24.557 ms 32.133 ms 32.091 ms 9 * * ae2.ctl-lax8.netarch.akamai.com (2600:1488:a080:114::b) 49.474 ms /vrode
All - I finally got a hold of someone at FAA and they said they aren’t seeing any issues. I tried to explain but didn’t get anywhere. Michael On Sun, Mar 26, 2023 at 14:21 virendra rode <virendra.rode@outages.org> wrote:
On Sat, 2023-03-25 at 20:06 -0700, virendra rode via Outages wrote:
On Sun, 2023-03-26 at 02:00 +0000, Chapman, Brad (NBCUniversal) via Outages wrote:
I don't know how one would contact them, but the list of the senior IT officers are on their memorandum to migrate to IPv6 dated September 2021. This memo states they would be migrating in 2023. Perhaps we're seeing evidence of a recent change?
https://www.faa.gov/sites/faa.gov/files/about/office_org/headquarters_office...
—Sent from my iPhone
On Mar 25, 2023, at 6:53 PM, Michael B. Williams via Outages < outages@outages.org> wrote:
Seeing the same here across all our POPs.
Has anyone notified FAA NOC yet?
--------------- I did notice ipv6 path traversal...but now its totally dead
/vrode
_______________________________________________ Outages mailing list Outages@outages.org
------------------- More proof, just so I'm not blowing smoke,
traceroute to www.faa.gov (2600:1406:6c00:192::1256), 30 hops max, 80 byte packets 1 2603-8001-3b00-2dc7-0000-0000-0000-0001.res6.spectrum.com (2603:8001:3b00:2dc7::1) 15.618 ms 15.521 ms 15.473 ms 2 2603-90c5-0001-0927-0000-0000-0000-0001.inf6.spectrum.com (2603:90c5:1:927::1) 31.726 ms 19.029 ms 20.498 ms 3 lag-61.snaucaoszh1.netops.charter.com (2605:e000:0:4::7:37d) 21.835 ms 31.496 ms 31.462 ms 4 lag-22.cyprcabw02r.netops.charter.com (2605:e000:0:8::6:e) 31.500 ms 31.464 ms 38.891 ms 5 * * * 6 * * * 7 lag-1.pr2.lax10.netops.charter.com (2001:1998:0:4::559) 25.888 ms 24.790 ms lag-2.pr2.lax10.netops.charter.com (2001:1998:56::27) 32.321 ms 8 2001:1998:0:8::667 (2001:1998:0:8::667) 24.557 ms 32.133 ms 32.091 ms 9 * * ae2.ctl-lax8.netarch.akamai.com (2600:1488:a080:114::b) 49.474 ms
/vrode
https://puck.nether.net/mailman/listinfo/outages
-- Sent from Gmail Mobile
This image says otherwise. [image0.jpeg] —Sent from my iPhone On Mar 25, 2023, at 8:29 PM, Michael B. Williams <Michael.Williams@glexia.com> wrote: All - I finally got a hold of someone at FAA and they said they aren’t seeing any issues. I tried to explain but didn’t get anywhere. Michael On Sun, Mar 26, 2023 at 14:21 virendra rode <virendra.rode@outages.org<mailto:virendra.rode@outages.org>> wrote: On Sat, 2023-03-25 at 20:06 -0700, virendra rode via Outages wrote: On Sun, 2023-03-26 at 02:00 +0000, Chapman, Brad (NBCUniversal) via Outages wrote: I don't know how one would contact them, but the list of the senior IT officers are on their memorandum to migrate to IPv6 dated September 2021. This memo states they would be migrating in 2023. Perhaps we're seeing evidence of a recent change? https://www.faa.gov/sites/faa.gov/files/about/office_org/headquarters_offices/afn/transition-ipv6.pdf<https://urldefense.com/v3/__https://www.faa.gov/sites/faa.gov/files/about/office_org/headquarters_offices/afn/transition-ipv6.pdf__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhWPAxlYWQ$> —Sent from my iPhone On Mar 25, 2023, at 6:53 PM, Michael B. Williams via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Seeing the same here across all our POPs. Has anyone notified FAA NOC yet? --------------- I did notice ipv6 path traversal...but now its totally dead /vrode _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> ------------------- More proof, just so I'm not blowing smoke, traceroute to www.faa.gov<https://urldefense.com/v3/__https://www.faa.gov__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhWTxg6fRg$> (2600:1406:6c00:192::1256), 30 hops max, 80 byte packets 1 2603-8001-3b00-2dc7-0000-0000-0000-0001.res6.spectrum.com<https://urldefense.com/v3/__http://2603-8001-3b00-2dc7-0000-0000-0000-0001.res6.spectrum.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhVNye0yxA$> (2603:8001:3b00:2dc7::1) 15.618 ms 15.521 ms 15.473 ms 2 2603-90c5-0001-0927-0000-0000-0000-0001.inf6.spectrum.com<https://urldefense.com/v3/__http://2603-90c5-0001-0927-0000-0000-0000-0001.inf6.spectrum.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhVOufjceA$> (2603:90c5:1:927::1) 31.726 ms 19.029 ms 20.498 ms 3 lag-61.snaucaoszh1.netops.charter.com<https://urldefense.com/v3/__http://lag-61.snaucaoszh1.netops.charter.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhXcPtSajw$> (2605:e000:0:4::7:37d) 21.835 ms 31.496 ms 31.462 ms 4 lag-22.cyprcabw02r.netops.charter.com<https://urldefense.com/v3/__http://lag-22.cyprcabw02r.netops.charter.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhUfU3ZmIQ$> (2605:e000:0:8::6:e) 31.500 ms 31.464 ms 38.891 ms 5 * * * 6 * * * 7 lag-1.pr2.lax10.netops.charter.com<https://urldefense.com/v3/__http://lag-1.pr2.lax10.netops.charter.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhUMcgbOHg$> (2001:1998:0:4::559) 25.888 ms 24.790 ms lag-2.pr2.lax10.netops.charter.com<https://urldefense.com/v3/__http://lag-2.pr2.lax10.netops.charter.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhWIIaxxoQ$> (2001:1998:56::27) 32.321 ms 8 2001:1998:0:8::667 (2001:1998:0:8::667) 24.557 ms 32.133 ms 32.091 ms 9 * * ae2.ctl-lax8.netarch.akamai.com<https://urldefense.com/v3/__http://ae2.ctl-lax8.netarch.akamai.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhW5tAaIhg$> (2600:1488:a080:114::b) 49.474 ms /vrode https://puck.nether.net/mailman/listinfo/outages<https://urldefense.com/v3/__https://puck.nether.net/mailman/listinfo/outages__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhXeGYiCdw$> -- Sent from Gmail Mobile
This screenshot of the FAA status page says otherwise. [image0.jpeg] —Sent from my iPhone On Mar 25, 2023, at 8:29 PM, Michael B. Williams <Michael.Williams@glexia.com> wrote: All - I finally got a hold of someone at FAA and they said they aren’t seeing any issues. I tried to explain but didn’t get anywhere. Michael On Sun, Mar 26, 2023 at 14:21 virendra rode <virendra.rode@outages.org<mailto:virendra.rode@outages.org>> wrote: On Sat, 2023-03-25 at 20:06 -0700, virendra rode via Outages wrote: On Sun, 2023-03-26 at 02:00 +0000, Chapman, Brad (NBCUniversal) via Outages wrote: I don't know how one would contact them, but the list of the senior IT officers are on their memorandum to migrate to IPv6 dated September 2021. This memo states they would be migrating in 2023. Perhaps we're seeing evidence of a recent change? https://www.faa.gov/sites/faa.gov/files/about/office_org/headquarters_offices/afn/transition-ipv6.pdf<https://urldefense.com/v3/__https://www.faa.gov/sites/faa.gov/files/about/office_org/headquarters_offices/afn/transition-ipv6.pdf__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhWPAxlYWQ$> —Sent from my iPhone On Mar 25, 2023, at 6:53 PM, Michael B. Williams via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Seeing the same here across all our POPs. Has anyone notified FAA NOC yet? --------------- I did notice ipv6 path traversal...but now its totally dead /vrode _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> ------------------- More proof, just so I'm not blowing smoke, traceroute to www.faa.gov<https://urldefense.com/v3/__https://www.faa.gov__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhWTxg6fRg$> (2600:1406:6c00:192::1256), 30 hops max, 80 byte packets 1 2603-8001-3b00-2dc7-0000-0000-0000-0001.res6.spectrum.com<https://urldefense.com/v3/__http://2603-8001-3b00-2dc7-0000-0000-0000-0001.res6.spectrum.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhVNye0yxA$> (2603:8001:3b00:2dc7::1) 15.618 ms 15.521 ms 15.473 ms 2 2603-90c5-0001-0927-0000-0000-0000-0001.inf6.spectrum.com<https://urldefense.com/v3/__http://2603-90c5-0001-0927-0000-0000-0000-0001.inf6.spectrum.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhVOufjceA$> (2603:90c5:1:927::1) 31.726 ms 19.029 ms 20.498 ms 3 lag-61.snaucaoszh1.netops.charter.com<https://urldefense.com/v3/__http://lag-61.snaucaoszh1.netops.charter.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhXcPtSajw$> (2605:e000:0:4::7:37d) 21.835 ms 31.496 ms 31.462 ms 4 lag-22.cyprcabw02r.netops.charter.com<https://urldefense.com/v3/__http://lag-22.cyprcabw02r.netops.charter.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhUfU3ZmIQ$> (2605:e000:0:8::6:e) 31.500 ms 31.464 ms 38.891 ms 5 * * * 6 * * * 7 lag-1.pr2.lax10.netops.charter.com<https://urldefense.com/v3/__http://lag-1.pr2.lax10.netops.charter.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhUMcgbOHg$> (2001:1998:0:4::559) 25.888 ms 24.790 ms lag-2.pr2.lax10.netops.charter.com<https://urldefense.com/v3/__http://lag-2.pr2.lax10.netops.charter.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhWIIaxxoQ$> (2001:1998:56::27) 32.321 ms 8 2001:1998:0:8::667 (2001:1998:0:8::667) 24.557 ms 32.133 ms 32.091 ms 9 * * ae2.ctl-lax8.netarch.akamai.com<https://urldefense.com/v3/__http://ae2.ctl-lax8.netarch.akamai.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhW5tAaIhg$> (2600:1488:a080:114::b) 49.474 ms /vrode https://puck.nether.net/mailman/listinfo/outages<https://urldefense.com/v3/__https://puck.nether.net/mailman/listinfo/outages__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhXeGYiCdw$> -- Sent from Gmail Mobile
+I'm adding John Blue (cc'd) to see if he has any insight into this matter. /vrode On Sun, 2023-03-26 at 03:41 +0000, Chapman, Brad (NBCUniversal) wrote:
This screenshot of the FAA status page says otherwise.
image0.jpeg
—Sent from my iPhone
On Mar 25, 2023, at 8:29 PM, Michael B. Williams <Michael.Williams@glexia.com> wrote:
All -
I finally got a hold of someone at FAA and they said they aren’t seeing any issues.
I tried to explain but didn’t get anywhere.
Michael
On Sun, Mar 26, 2023 at 14:21 virendra rode <virendra.rode@outages.org> wrote:
On Sat, 2023-03-25 at 20:06 -0700, virendra rode via Outages wrote:
On Sun, 2023-03-26 at 02:00 +0000, Chapman, Brad (NBCUniversal) via Outages wrote:
I don't know how one would contact them, but the list of the senior IT officers are on their memorandum to migrate to IPv6 dated September 2021. This memo states they would be migrating in 2023. Perhaps we're seeing evidence of a recent change?
https://www.faa.gov/sites/faa.gov/files/about/office_org/headquarters_office...
—Sent from my iPhone
On Mar 25, 2023, at 6:53 PM, Michael B. Williams via Outages <outages@outages.org> wrote:
Seeing the same here across all our POPs.
Has anyone notified FAA NOC yet?
--------------- I did notice ipv6 path traversal...but now its totally dead
/vrode _______________________________________________ Outages mailing list Outages@outages.org
More proof, just so I'm not blowing smoke,
traceroute to www.faa.gov (2600:1406:6c00:192::1256), 30 hops max, 80 byte packets 1 2603-8001-3b00-2dc7-0000-0000-0000-0001.res6.spectrum.com (2603:8001:3b00:2dc7::1) 15.618 ms 15.521 ms 15.473 ms 2 2603-90c5-0001-0927-0000-0000-0000-0001.inf6.spectrum.com (2603:90c5:1:927::1) 31.726 ms 19.029 ms 20.498 ms 3 lag-61.snaucaoszh1.netops.charter.com (2605:e000:0:4::7:37d) 21.835 ms 31.496 ms 31.462 ms 4 lag-22.cyprcabw02r.netops.charter.com (2605:e000:0:8::6:e) 31.500 ms 31.464 ms 38.891 ms 5 * * * 6 * * * 7 lag-1.pr2.lax10.netops.charter.com (2001:1998:0:4::559) 25.888 ms 24.790 ms lag-2.pr2.lax10.netops.charter.com (2001:1998:56::27) 32.321 ms 8 2001:1998:0:8::667 (2001:1998:0:8::667) 24.557 ms 32.133 ms 32.091 ms 9 * * ae2.ctl-lax8.netarch.akamai.com (2600:1488:a080:114::b) 49.474 ms
/vrode
Initial looks from the firewall team point to an automatic failover event and the secondary failed. John W Blue Infrastructure Services - AIF-330 d: 405-954-4177 c: 405-443-8445 john.w.blue@faa.gov<mailto:john.w.blue@faa.gov> From: virendra rode <virendra.rode@outages.org> Sent: Saturday, March 25, 2023 11:05 PM To: Chapman, Brad (NBCUniversal) <Brad.Chapman@nbcuni.com>; Michael B. Williams <Michael.Williams@glexia.com> Cc: outages <outages@outages.org>; Blue, John W (FAA) <john.w.blue@faa.gov> Subject: Re: [outages] [EXTERNAL] Re: FAA.gov nameserver outage +I'm adding John Blue (cc'd) to see if he has any insight into this matter. /vrode On Sun, 2023-03-26 at 03:41 +0000, Chapman, Brad (NBCUniversal) wrote: This screenshot of the FAA status page says otherwise. [image0.jpeg] —Sent from my iPhone On Mar 25, 2023, at 8:29 PM, Michael B. Williams <Michael.Williams@glexia.com<mailto:Michael.Williams@glexia.com>> wrote: All - I finally got a hold of someone at FAA and they said they aren’t seeing any issues. I tried to explain but didn’t get anywhere. Michael On Sun, Mar 26, 2023 at 14:21 virendra rode <virendra.rode@outages.org<mailto:virendra.rode@outages.org>> wrote: On Sat, 2023-03-25 at 20:06 -0700, virendra rode via Outages wrote: On Sun, 2023-03-26 at 02:00 +0000, Chapman, Brad (NBCUniversal) via Outages wrote: I don't know how one would contact them, but the list of the senior IT officers are on their memorandum to migrate to IPv6 dated September 2021. This memo states they would be migrating in 2023. Perhaps we're seeing evidence of a recent change? https://www.faa.gov/sites/faa.gov/files/about/office_org/headquarters_offices/afn/transition-ipv6.pdf<https://urldefense.com/v3/__https:/www.faa.gov/sites/faa.gov/files/about/office_org/headquarters_offices/afn/transition-ipv6.pdf__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhWPAxlYWQ$> —Sent from my iPhone On Mar 25, 2023, at 6:53 PM, Michael B. Williams via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Seeing the same here across all our POPs. Has anyone notified FAA NOC yet? --------------- I did notice ipv6 path traversal...but now its totally dead /vrode _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> ------------------- More proof, just so I'm not blowing smoke, traceroute to www.faa.gov<https://urldefense.com/v3/__https:/www.faa.gov__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhWTxg6fRg$> (2600:1406:6c00:192::1256), 30 hops max, 80 byte packets 1 2603-8001-3b00-2dc7-0000-0000-0000-0001.res6.spectrum.com<https://urldefense.com/v3/__http:/2603-8001-3b00-2dc7-0000-0000-0000-0001.res6.spectrum.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhVNye0yxA$> (2603:8001:3b00:2dc7::1) 15.618 ms 15.521 ms 15.473 ms 2 2603-90c5-0001-0927-0000-0000-0000-0001.inf6.spectrum.com<https://urldefense.com/v3/__http:/2603-90c5-0001-0927-0000-0000-0000-0001.inf6.spectrum.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhVOufjceA$> (2603:90c5:1:927::1) 31.726 ms 19.029 ms 20.498 ms 3 lag-61.snaucaoszh1.netops.charter.com<https://urldefense.com/v3/__http:/lag-61.snaucaoszh1.netops.charter.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhXcPtSajw$> (2605:e000:0:4::7:37d) 21.835 ms 31.496 ms 31.462 ms 4 lag-22.cyprcabw02r.netops.charter.com<https://urldefense.com/v3/__http:/lag-22.cyprcabw02r.netops.charter.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhUfU3ZmIQ$> (2605:e000:0:8::6:e) 31.500 ms 31.464 ms 38.891 ms 5 * * * 6 * * * 7 lag-1.pr2.lax10.netops.charter.com<https://urldefense.com/v3/__http:/lag-1.pr2.lax10.netops.charter.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhUMcgbOHg$> (2001:1998:0:4::559) 25.888 ms 24.790 ms lag-2.pr2.lax10.netops.charter.com<https://urldefense.com/v3/__http:/lag-2.pr2.lax10.netops.charter.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhWIIaxxoQ$> (2001:1998:56::27) 32.321 ms 8 2001:1998:0:8::667 (2001:1998:0:8::667) 24.557 ms 32.133 ms 32.091 ms 9 * * ae2.ctl-lax8.netarch.akamai.com<https://urldefense.com/v3/__http:/ae2.ctl-lax8.netarch.akamai.com__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhW5tAaIhg$> (2600:1488:a080:114::b) 49.474 ms /vrode https://puck.nether.net/mailman/listinfo/outages<https://urldefense.com/v3/__https:/puck.nether.net/mailman/listinfo/outages__;!!PIZeeW5wscynRQ!qznGJHQ7J3qeiuEgKIkWMakaGx4TN4Cq8BxLL-K0TVQAYbHmNgMKD1DffUYQu7QG6zFyNhv4Fob9UnWNVe0_RhXeGYiCdw$>
Thanks, John; we appreciate the transparency. /vrode On Sun, 2023-03-26 at 04:38 +0000, Blue, John W (FAA) wrote:
Initial looks from the firewall team point to an automatic failover event and the secondary failed. John W Blue Infrastructure Services - AIF-330 d: 405-954-4177 c: 405-443-8445 john.w.blue@faa.gov From: virendra rode <virendra.rode@outages.org> Sent: Saturday, March 25, 2023 11:05 PM To: Chapman, Brad (NBCUniversal) <Brad.Chapman@nbcuni.com>; Michael B. Williams <Michael.Williams@glexia.com> Cc: outages <outages@outages.org>; Blue, John W (FAA) <john.w.blue@faa.gov> Subject: Re: [outages] [EXTERNAL] Re: FAA.gov nameserver outage +I'm adding John Blue (cc'd) to see if he has any insight into this matter. /vrode On Sun, 2023-03-26 at 03:41 +0000, Chapman, Brad (NBCUniversal) wrote:
This screenshot of the FAA status page says otherwise. image0.jpeg —Sent from my iPhone
On Mar 25, 2023, at 8:29 PM, Michael B. Williams <Michael.Williams@glexia.com> wrote: All - I finally got a hold of someone at FAA and they said they aren’t seeing any issues. I tried to explain but didn’t get anywhere. Michael On Sun, Mar 26, 2023 at 14:21 virendra rode <virendra.rode@outages.org> wrote:
On Sat, 2023-03-25 at 20:06 -0700, virendra rode via Outages wrote:
On Sun, 2023-03-26 at 02:00 +0000, Chapman, Brad (NBCUniversal) via Outages wrote:
I don't know how one would contact them, but the list of the senior IT officers are on their memorandum to migrate to IPv6 dated September 2021. This memo states they would be migrating in 2023. Perhaps we're seeing evidence of a recent change?
https://www.faa.gov/sites/faa.gov/files/about/office_org/headquarters_office...
—Sent from my iPhone
> On Mar 25, 2023, at 6:53 PM, Michael B. Williams via > Outages <outages@outages.org> wrote: > > Seeing the same here across all our POPs. > > Has anyone notified FAA NOC yet?
I did notice ipv6 path traversal...but now its totally dead /vrode _______________________________________________ Outages mailing list Outages@outages.org
More proof, just so I'm not blowing smoke, traceroute to www.faa.gov (2600:1406:6c00:192::1256), 30 hops max, 80 byte packets 1 2603-8001-3b00-2dc7-0000-0000-0000-0001.res6.spectrum.com (2603:8001:3b00:2dc7::1) 15.618 ms 15.521 ms 15.473 ms 2 2603-90c5-0001-0927-0000-0000-0000-0001.inf6.spectrum.com (2603:90c5:1:927::1) 31.726 ms 19.029 ms 20.498 ms 3 lag-61.snaucaoszh1.netops.charter.com (2605:e000:0:4::7:37d) 21.835 ms 31.496 ms 31.462 ms 4 lag-22.cyprcabw02r.netops.charter.com (2605:e000:0:8::6:e) 31.500 ms 31.464 ms 38.891 ms 5 * * * 6 * * * 7 lag-1.pr2.lax10.netops.charter.com (2001:1998:0:4::559) 25.888 ms 24.790 ms lag-2.pr2.lax10.netops.charter.com (2001:1998:56::27) 32.321 ms 8 2001:1998:0:8::667 (2001:1998:0:8::667) 24.557 ms 32.133 ms 32.091 ms 9 * * ae2.ctl-lax8.netarch.akamai.com (2600:1488:a080:114::b) 49.474 ms /vrode
mloftis> Looks like faa.gov's nameservers are all having a bad time, mloftis> only occasionally responding right now from multiple tests from mloftis> my home network, datacenter POPs (Seattle, Chicago), and mloftis> 8.8.8.8 They only seem to have two auth nameservers for faa, both within the faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the servers are in all die just within each block run by the FAA. Seems like an internal routing meltdown making the only 2 nameservers unreachbable reliably.
On Sat, 2023-03-25 at 19:34 -0600, Michael Loftis via Outages wrote:
Looks like faa.gov's nameservers are all having a bad time, only occasionally responding right now from multiple tests from my home network, datacenter POPs (Seattle, Chicago), and 8.8.8.8
----- faa.gov is CDN (akamaiedge) supported. Latency appears to be off the roof. /vrode
Hi, I'm a researcher of DNS vulnerabilities. It loos like random subdomain attacks (water tourtue attack). This is the data of my rate-limitted openresolver as a honeypot. http://www.e-ontap.com/dns/todaydowngov.txt http://www.e-ontap.com/dns/todaydown.txt (You can not view these page if you are using 8.8.8.8, sorry.) Raw logs of my Unbound (Time is JST) local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head -5 Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL <unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210 unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45 Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN SERVFAIL 15.112813 0 30 Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | head -5 Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. from 2620:74:27::2:30 no server to query nameserver addresses not usable Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. upstream server timeout Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. upstream server timeout local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | tail -5 Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL <eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail -5 Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210 eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44 Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210 faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46 Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A IN SERVFAIL 0.000000 0 34 local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc -l 1408 -- T.Suzuki -- T.Suzuki / E.F.シューマッハーとI.イリイチを読もう
What would be the symptoms here of a "water torture attack" rather than what John had indicated as a firewall failure in their infrastructure:
Initial looks from the firewall team point to an automatic failover event and the secondary failed.
And the symptoms of which lined up with network level info from Paul earlier:
They only seem to have two auth nameservers for faa, both within the faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the servers are in all die just within each block run by the FAA.
Seems like an internal routing meltdown making the only 2 nameservers unreachable reliably.
Are you saying that your open resolvers have a per client rate limit applied, that rate limit got tripped, and shortly thereafter the resolvers became unavailable, suggesting query floods for the domain(s) that knocked the resolvers offline (or from the other discussion, possibly was the thing that overwhelmed that firewall layer, causing the initial failover and possibly also causing the firewall secondary to fail to come online)? On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <outages@outages.org> wrote:
Hi, I'm a researcher of DNS vulnerabilities.
It loos like random subdomain attacks (water tourtue attack).
This is the data of my rate-limitted openresolver as a honeypot. http://www.e-ontap.com/dns/todaydowngov.txt http://www.e-ontap.com/dns/todaydown.txt (You can not view these page if you are using 8.8.8.8, sorry.)
Raw logs of my Unbound (Time is JST) local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head -5 Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL < unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210 unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45 Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN SERVFAIL 15.112813 0 30 Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | head -5 Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. from 2620:74:27::2:30 no server to query nameserver addresses not usable Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. upstream server timeout Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. upstream server timeout local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | tail -5 Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL < eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL < faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail -5 Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210 eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44 Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL < faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210 faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46 Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A IN SERVFAIL 0.000000 0 34 local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc -l 1408
-- T.Suzuki -- T.Suzuki / E.F.シューマッハーとI.イリイチを読もう _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Sun, 26 Mar 2023 08:35:29 -0700 Hugo Slabbert <hugo@slabnet.com> wrote:
What would be the symptoms here of a "water torture attack" rather than what John had indicated as a firewall failure in their infrastructure:
Initial looks from the firewall team point to an automatic failover event and the secondary failed.
And the symptoms of which lined up with network level info from Paul earlier:
They only seem to have two auth nameservers for faa, both within the faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the servers are in all die just within each block run by the FAA.
Seems like an internal routing meltdown making the only 2 nameservers unreachable reliably.
Are you saying that your open resolvers have a per client rate limit applied, that rate limit got tripped, and shortly thereafter the resolvers became unavailable, suggesting query floods for the domain(s) that knocked the resolvers offline (or from the other discussion, possibly was the thing that overwhelmed that firewall layer, causing the initial failover and possibly also causing the firewall secondary to fail to come online)?
Yes. (limitting per client, and per second for all) Perhaps, large numbers open resolvers including no ratelimit are used. Then massive random subdomain queries caused the firewall symptoms. (It's only my guess.)
On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <outages@outages.org> wrote:
Hi, I'm a researcher of DNS vulnerabilities.
It loos like random subdomain attacks (water tourtue attack).
This is the data of my rate-limitted openresolver as a honeypot. http://www.e-ontap.com/dns/todaydowngov.txt http://www.e-ontap.com/dns/todaydown.txt (You can not view these page if you are using 8.8.8.8, sorry.)
Raw logs of my Unbound (Time is JST) local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head -5 Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL < unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210 unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45 Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN SERVFAIL 15.112813 0 30 Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | head -5 Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. from 2620:74:27::2:30 no server to query nameserver addresses not usable Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. upstream server timeout Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. upstream server timeout local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | tail -5 Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL < eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL < faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail -5 Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210 eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44 Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL < faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210 faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46 Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A IN SERVFAIL 0.000000 0 34 local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc -l 1408
-- T.Suzuki -- T.Suzuki / E.F.シューマッハーとI.イリイチを読もう _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- T.Suzuki / E.F.シューマッハーとI.イリイチを読もう
This belongs on the outages discussion list, NOT here. This is only for outages and the immediate outage info. If you don’t understand why, go ask on THAT list. -- Carlos Alvarez 602-368-6403 On Mar 26, 2023 at 5:13 PM -0700, T.Suzuki via Outages <outages@outages.org>, wrote:
On Sun, 26 Mar 2023 08:35:29 -0700 Hugo Slabbert <hugo@slabnet.com> wrote:
What would be the symptoms here of a "water torture attack" rather than what John had indicated as a firewall failure in their infrastructure:
Initial looks from the firewall team point to an automatic failover event and the secondary failed.
And the symptoms of which lined up with network level info from Paul earlier:
They only seem to have two auth nameservers for faa, both within the faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the servers are in all die just within each block run by the FAA.
Seems like an internal routing meltdown making the only 2 nameservers unreachable reliably.
Are you saying that your open resolvers have a per client rate limit applied, that rate limit got tripped, and shortly thereafter the resolvers became unavailable, suggesting query floods for the domain(s) that knocked the resolvers offline (or from the other discussion, possibly was the thing that overwhelmed that firewall layer, causing the initial failover and possibly also causing the firewall secondary to fail to come online)?
Yes. (limitting per client, and per second for all) Perhaps, large numbers open resolvers including no ratelimit are used. Then massive random subdomain queries caused the firewall symptoms. (It's only my guess.)
On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <outages@outages.org> wrote:
Hi, I'm a researcher of DNS vulnerabilities.
It loos like random subdomain attacks (water tourtue attack).
This is the data of my rate-limitted openresolver as a honeypot. http://www.e-ontap.com/dns/todaydowngov.txt http://www.e-ontap.com/dns/todaydown.txt (You can not view these page if you are using 8.8.8.8, sorry.)
Raw logs of my Unbound (Time is JST) local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head -5 Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL < unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210 unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45 Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN SERVFAIL 15.112813 0 30 Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | head -5 Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. from 2620:74:27::2:30 no server to query nameserver addresses not usable Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. upstream server timeout Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. upstream server timeout local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | tail -5 Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL < eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL < faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail -5 Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210 eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44 Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL < faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210 faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46 Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A IN SERVFAIL 0.000000 0 34 local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc -l 1408
-- T.Suzuki -- T.Suzuki / E.F.シューマッハーとI.イリイチを読もう _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- T.Suzuki / E.F.シューマッハーとI.イリイチを読もう _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Can’t believe it’s still dead… -Mike
On Mar 26, 2023, at 17:13, T.Suzuki via Outages <outages@outages.org> wrote:
On Sun, 26 Mar 2023 08:35:29 -0700 Hugo Slabbert <hugo@slabnet.com> wrote:
What would be the symptoms here of a "water torture attack" rather than what John had indicated as a firewall failure in their infrastructure:
Initial looks from the firewall team point to an automatic failover event and the secondary failed.
And the symptoms of which lined up with network level info from Paul earlier:
They only seem to have two auth nameservers for faa, both within the faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the servers are in all die just within each block run by the FAA.
Seems like an internal routing meltdown making the only 2 nameservers unreachable reliably.
Are you saying that your open resolvers have a per client rate limit applied, that rate limit got tripped, and shortly thereafter the resolvers became unavailable, suggesting query floods for the domain(s) that knocked the resolvers offline (or from the other discussion, possibly was the thing that overwhelmed that firewall layer, causing the initial failover and possibly also causing the firewall secondary to fail to come online)?
Yes. (limitting per client, and per second for all) Perhaps, large numbers open resolvers including no ratelimit are used. Then massive random subdomain queries caused the firewall symptoms. (It's only my guess.)
On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <outages@outages.org> wrote:
Hi, I'm a researcher of DNS vulnerabilities.
It loos like random subdomain attacks (water tourtue attack).
This is the data of my rate-limitted openresolver as a honeypot. http://www.e-ontap.com/dns/todaydowngov.txt http://www.e-ontap.com/dns/todaydown.txt (You can not view these page if you are using 8.8.8.8, sorry.)
Raw logs of my Unbound (Time is JST) local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head -5 Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL < unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210 unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45 Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN SERVFAIL 15.112813 0 30 Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | head -5 Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. from 2620:74:27::2:30 no server to query nameserver addresses not usable Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. upstream server timeout Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. upstream server timeout local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | tail -5 Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL < eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL < faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail -5 Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210 eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44 Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL < faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210 faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46 Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A IN SERVFAIL 0.000000 0 34 local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc -l 1408
-- T.Suzuki -- T.Suzuki / E.F.シューマッハーとI.イリイチを読もう _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- T.Suzuki / E.F.シューマッハーとI.イリイチを読もう _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Sun, 26 Mar 2023 17:17:25 -0700 Mike Lyon <mike.lyon@gmail.com> wrote:
Can’t believe it’s still dead…
-Mike
The attack appears to be over, at Mar 26 13:41:28 JST (GMT +0900) (This may be specific to my server). Maybe the cause is something else. Or the person in charge of manual recovery is on holiday. Mar 26 13:41:08 unbound[48103:0] reply: 24.199.82.210 asm.faa.gov. A IN SERVFAIL 0.000000 0 29 Mar 26 13:41:15 unbound[48103:0] query: 24.199.82.210 sas-uss.edc.nas.faa.gov. A IN Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:15 unbound[48103:0] reply: 24.199.82.210 sas-uss.edc.nas.faa.gov. A IN SERVFAIL 0.000000 0 41 Mar 26 13:41:22 unbound[48103:0] query: 24.199.82.210 eforms-stagedev.hq.faa.gov. A IN Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL <eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210 eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44 Mar 26 13:41:23 unbound[48103:0] query: 24.199.82.210 faardm-mceast2.idrac.faa.gov. A IN Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210 faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46 Mar 26 13:41:28 unbound[48103:0] query: 24.199.82.210 chronos3.faa.gov. A IN Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A IN SERVFAIL 0.000000 0 34
On Mar 26, 2023, at 17:13, T.Suzuki via Outages <outages@outages.org> wrote:
〓On Sun, 26 Mar 2023 08:35:29 -0700 Hugo Slabbert <hugo@slabnet.com> wrote:
What would be the symptoms here of a "water torture attack" rather than what John had indicated as a firewall failure in their infrastructure:
Initial looks from the firewall team point to an automatic failover event and the secondary failed.
And the symptoms of which lined up with network level info from Paul earlier:
They only seem to have two auth nameservers for faa, both within the faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the servers are in all die just within each block run by the FAA.
Seems like an internal routing meltdown making the only 2 nameservers unreachable reliably.
Are you saying that your open resolvers have a per client rate limit applied, that rate limit got tripped, and shortly thereafter the resolvers became unavailable, suggesting query floods for the domain(s) that knocked the resolvers offline (or from the other discussion, possibly was the thing that overwhelmed that firewall layer, causing the initial failover and possibly also causing the firewall secondary to fail to come online)?
Yes. (limitting per client, and per second for all) Perhaps, large numbers open resolvers including no ratelimit are used. Then massive random subdomain queries caused the firewall symptoms. (It's only my guess.)
On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <outages@outages.org> wrote:
Hi, I'm a researcher of DNS vulnerabilities.
It loos like random subdomain attacks (water tourtue attack).
This is the data of my rate-limitted openresolver as a honeypot. http://www.e-ontap.com/dns/todaydowngov.txt http://www.e-ontap.com/dns/todaydown.txt (You can not view these page if you are using 8.8.8.8, sorry.)
Raw logs of my Unbound (Time is JST) local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head -5 Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL < unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210 unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45 Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN SERVFAIL 15.112813 0 30 Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | head -5 Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. from 2620:74:27::2:30 no server to query nameserver addresses not usable Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. upstream server timeout Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. upstream server timeout local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | tail -5 Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL < eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL < faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail -5 Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210 eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44 Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL < faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210 faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46 Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A IN SERVFAIL 0.000000 0 34 local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc -l 1408
-- T.Suzuki -- T.Suzuki / E.F.シューマッハーとI.イリイチを読もう _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- T.Suzuki / E.F.シューマッハーとI.イリイチを読もう _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- T.Suzuki / E.F.シューマッハーとI.イリイチを読もう
As is Generalissimo Francisco Franco. Indeed, folks; please move these meta conversations to the -discuss list; they are off topic for the main notification list. Cheers, -- jr '<admin/>' a ----- Original Message -----
From: "Mike Lyon via Outages" <outages@outages.org> To: "T.Suzuki" <tss-outage@e-ontap.com> Cc: "Michael Loftis via Outages" <outages@outages.org> Sent: Sunday, March 26, 2023 8:17:25 PM Subject: Re: [outages] FAA.gov nameserver outage
Can’t believe it’s still dead…
-Mike
On Mar 26, 2023, at 17:13, T.Suzuki via Outages <outages@outages.org> wrote:
On Sun, 26 Mar 2023 08:35:29 -0700 Hugo Slabbert <hugo@slabnet.com> wrote:
What would be the symptoms here of a "water torture attack" rather than what John had indicated as a firewall failure in their infrastructure:
Initial looks from the firewall team point to an automatic failover event and the secondary failed.
And the symptoms of which lined up with network level info from Paul earlier:
They only seem to have two auth nameservers for faa, both within the faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the servers are in all die just within each block run by the FAA.
Seems like an internal routing meltdown making the only 2 nameservers unreachable reliably.
Are you saying that your open resolvers have a per client rate limit applied, that rate limit got tripped, and shortly thereafter the resolvers became unavailable, suggesting query floods for the domain(s) that knocked the resolvers offline (or from the other discussion, possibly was the thing that overwhelmed that firewall layer, causing the initial failover and possibly also causing the firewall secondary to fail to come online)?
Yes. (limitting per client, and per second for all) Perhaps, large numbers open resolvers including no ratelimit are used. Then massive random subdomain queries caused the firewall symptoms. (It's only my guess.)
On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <outages@outages.org> wrote:
Hi, I'm a researcher of DNS vulnerabilities.
It loos like random subdomain attacks (water tourtue attack).
This is the data of my rate-limitted openresolver as a honeypot. http://www.e-ontap.com/dns/todaydowngov.txt http://www.e-ontap.com/dns/todaydown.txt (You can not view these page if you are using 8.8.8.8, sorry.)
Raw logs of my Unbound (Time is JST) local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head -5 Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL < unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210 unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45 Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN SERVFAIL 15.112813 0 30 Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | head -5 Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. from 2620:74:27::2:30 no server to query nameserver addresses not usable Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. upstream server timeout Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. upstream server timeout local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | tail -5 Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL < eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL < faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail -5 Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210 eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44 Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL < faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210 faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46 Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A IN SERVFAIL 0.000000 0 34 local/etc/unbound% local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc -l 1408
-- T.Suzuki -- T.Suzuki / E.F.シューマッハーとI.イリイチを読もう _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- T.Suzuki / E.F.シューマッハーとI.イリイチを読もう _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
No worries. Phil Washington has the ball and will fix this. https://twitter.com/CitizenFreePres/status/1640243188395831297 -Pete On 3/25/23 21:34, Michael Loftis via Outages wrote:
Looks like faa.gov's nameservers are all having a bad time, only occasionally responding right now from multiple tests from my home network, datacenter POPs (Seattle, Chicago), and 8.8.8.8
participants (14)
-
Blue, John W (FAA) -
Carlos Alvarez -
Chapman, Brad (NBCUniversal) -
Hugo Slabbert -
Jay R. Ashworth -
Michael B. Williams -
Michael Loftis -
Mike Lyon -
N DePasquale -
Nate Howe -
Paul Ebersman -
Pete Rohrman -
T.Suzuki -
virendra rode