Google 8.8.8.8 Resolution of Route53 domains

Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve with Google DNS?

Yeh. Still digging into it. From: Outages <outages-bounces@outages.org> On Behalf Of Zach Hanna via Outages Sent: 24 April 2018 12:54 To: outages@outages.org Subject: [outages] Google 8.8.8.8 Resolution of Route53 domains Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve with Google DNS?

Looks more specific to AWS than it does to Google+AWS. Can’t resolve against some of AWS’s NS directly: phil@phil-debian:~$ dig cloudcall.com IN A @ns-163.awsdns-20.com ; <<>> DiG 9.10.3-P4-Debian <<>> cloudcall.com IN A @ns-163.awsdns-20.com ;; global options: +cmd ;; connection timed out; no servers could be reached From: Outages <outages-bounces@outages.org> On Behalf Of Phil Lavin via Outages Sent: 24 April 2018 12:56 To: outages@outages.org Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains Yeh. Still digging into it. From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Zach Hanna via Outages Sent: 24 April 2018 12:54 To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] Google 8.8.8.8 Resolution of Route53 domains Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve with Google DNS?

This doesn’t feel right, though I’ll admit I’ve never checked before. Our only route to ns-163.awsdns-20.com (205.251.192.163) is through HE: inet.0: 757581 destinations, 2107440 routes (757301 active, 0 holddown, 522 hidden) + = Active Route, - = Last Active, * = Both 205.251.192.0/24 *[BGP/170] 01:12:08, localpref 70 AS path: 6939 10297 I, validation-state: unverified > to 216.66.90.21 via ge-1/0/5.0 AS10297 is eNET inc. Is this expected? From: Outages <outages-bounces@outages.org> On Behalf Of Phil Lavin via Outages Sent: 24 April 2018 13:04 To: outages@outages.org Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains Looks more specific to AWS than it does to Google+AWS. Can’t resolve against some of AWS’s NS directly: phil@phil-debian:~$ dig cloudcall.com IN A @ns-163.awsdns-20.com ; <<>> DiG 9.10.3-P4-Debian <<>> cloudcall.com IN A @ns-163.awsdns-20.com ;; global options: +cmd ;; connection timed out; no servers could be reached From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Phil Lavin via Outages Sent: 24 April 2018 12:56 To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains Yeh. Still digging into it. From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Zach Hanna via Outages Sent: 24 April 2018 12:54 To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] Google 8.8.8.8 Resolution of Route53 domains Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve with Google DNS?

instagram.com Server: 8.8.8.8 Address: 8.8.8.8#53 ** server can't find instagram.com: SERVFAIL
On Tue, Apr 24, 2018 at 6:20 AM, Phil Lavin via Outages <outages@outages.org
wrote:
This doesn’t feel right, though I’ll admit I’ve never checked before. Our only route to ns-163.awsdns-20.com (205.251.192.163) is through HE:
inet.0: 757581 destinations, 2107440 routes (757301 active, 0 holddown, 522 hidden)
+ = Active Route, - = Last Active, * = Both
205.251.192.0/24 *[BGP/170] 01:12:08, localpref 70
AS path: 6939 10297 I, validation-state: unverified
> to 216.66.90.21 via ge-1/0/5.0
AS10297 is eNET inc. Is this expected?
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Phil Lavin via Outages *Sent:* 24 April 2018 13:04 *To:* outages@outages.org *Subject:* Re: [outages] Google 8.8.8.8 Resolution of Route53 domains
Looks more specific to AWS than it does to Google+AWS. Can’t resolve against some of AWS’s NS directly:
phil@phil-debian:~$ dig cloudcall.com IN A @ns-163.awsdns-20.com
; <<>> DiG 9.10.3-P4-Debian <<>> cloudcall.com IN A @ns-163.awsdns-20.com
;; global options: +cmd
;; connection timed out; no servers could be reached
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Phil Lavin via Outages *Sent:* 24 April 2018 12:56 *To:* outages@outages.org *Subject:* Re: [outages] Google 8.8.8.8 Resolution of Route53 domains
Yeh. Still digging into it.
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Zach Hanna via Outages *Sent:* 24 April 2018 12:54 *To:* outages@outages.org *Subject:* [outages] Google 8.8.8.8 Resolution of Route53 domains
Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve with Google DNS?
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Yeah that looks wrong to me. It appears if I'm reading it correctly that route was published about 90 minutes ago? Timeline sorta matches... On Apr 24, 2018, 08:27, at 08:27, Phil Lavin via Outages <outages@outages.org> wrote:
This doesn’t feel right, though I’ll admit I’ve never checked before. Our only route to ns-163.awsdns-20.com (205.251.192.163) is through HE: inet.0: 757581 destinations, 2107440 routes (757301 active, 0 holddown, 522 hidden) + = Active Route, - = Last Active, * = Both 205.251.192.0/24 *[BGP/170] 01:12:08, localpref 70 AS path: 6939 10297 I, validation-state: unverified > to 216.66.90.21 via ge-1/0/5.0 AS10297 is eNET inc. Is this expected?
From: Outages <outages-bounces@outages.org> On Behalf Of Phil Lavin via Outages Sent: 24 April 2018 13:04 To: outages@outages.org Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains
Looks more specific to AWS than it does to Google+AWS. Can’t resolve against some of AWS’s NS directly: phil@phil-debian:~$ dig cloudcall.com IN A @ns-163.awsdns-20.com ; <<>> DiG 9.10.3-P4-Debian <<>> cloudcall.com IN A @ns-163.awsdns-20.com ;; global options: +cmd ;; connection timed out; no servers could be reached
From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Phil Lavin via Outages Sent: 24 April 2018 12:56 To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains
Yeh. Still digging into it.
From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Zach Hanna via Outages Sent: 24 April 2018 12:54 To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] Google 8.8.8.8 Resolution of Route53 domains
Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve with Google DNS?
------------------------------------------------------------------------
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

In fact, 3 of 4 nameservers on our main domain are advertised as different /24s by AS 10297. None of these can be queried. The other is advertised by AS 16509 (AWS) – this works. From: Outages <outages-bounces@outages.org> On Behalf Of Phil Lavin via Outages Sent: 24 April 2018 13:20 To: outages@outages.org Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains This doesn’t feel right, though I’ll admit I’ve never checked before. Our only route to ns-163.awsdns-20.com (205.251.192.163) is through HE: inet.0: 757581 destinations, 2107440 routes (757301 active, 0 holddown, 522 hidden) + = Active Route, - = Last Active, * = Both 205.251.192.0/24 *[BGP/170] 01:12:08, localpref 70 AS path: 6939 10297 I, validation-state: unverified > to 216.66.90.21 via ge-1/0/5.0 AS10297 is eNET inc. Is this expected? From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Phil Lavin via Outages Sent: 24 April 2018 13:04 To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains Looks more specific to AWS than it does to Google+AWS. Can’t resolve against some of AWS’s NS directly: phil@phil-debian:~$ dig cloudcall.com IN A @ns-163.awsdns-20.com ; <<>> DiG 9.10.3-P4-Debian <<>> cloudcall.com IN A @ns-163.awsdns-20.com ;; global options: +cmd ;; connection timed out; no servers could be reached From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Phil Lavin via Outages Sent: 24 April 2018 12:56 To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains Yeh. Still digging into it. From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Zach Hanna via Outages Sent: 24 April 2018 12:54 To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] Google 8.8.8.8 Resolution of Route53 domains Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve with Google DNS?

Tue Apr 24 11:05:41 UTC onwards one of Hurricane Electric's peers AS10297 started advertising the following subnets via HE. 205.251.192.0 205.251.193.0 205.251.195.0 205.251.197.0 205.251.199.0 These are all Amazon subnets, usually originated as part of /23s and seemingly host a fair bit of AWS Route53. If you (or your DNS resolver) are a HE transit customer you will be impacted the most. Cheers, Joseph On Tue, Apr 24, 2018, at 9:50 PM, Phil Lavin via Outages wrote:
This doesn’t feel right, though I’ll admit I’ve never checked before. Our only route to ns-163.awsdns-20.com (205.251.192.163) is through HE:> inet.0: 757581 destinations, 2107440 routes (757301 active, 0 holddown, 522 hidden)> + = Active Route, - = Last Active, * = Both
205.251.192.0/24 *[BGP/170] 01:12:08, localpref 70
AS path: 6939 10297 I, validation-state: unverified> > to 216.66.90.21 via ge-1/0/5.0
AS10297 is eNET inc. Is this expected?
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Phil Lavin via Outages *Sent:* 24 April 2018 13:04 *To:* outages@outages.org *Subject:* Re: [outages] Google 8.8.8.8 Resolution of Route53 domains>
Looks more specific to AWS than it does to Google+AWS. Can’t resolve against some of AWS’s NS directly:> phil@phil-debian:~$ dig cloudcall.com IN A @ns-163.awsdns-20.com
; <<>> DiG 9.10.3-P4-Debian <<>> cloudcall.com IN A @ns-163.awsdns- ; 20.com> ;; global options: +cmd
;; connection timed out; no servers could be reached
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Phil Lavin via Outages *Sent:* 24 April 2018 12:56 *To:* outages@outages.org *Subject:* Re: [outages] Google 8.8.8.8 Resolution of Route53 domains>
Yeh. Still digging into it.
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Zach Hanna via Outages *Sent:* 24 April 2018 12:54 *To:* outages@outages.org *Subject:* [outages] Google 8.8.8.8 Resolution of Route53 domains>
Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve with Google DNS?> _________________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Thanks for confirming. I’ve raised a ticket with HE on our circuit. I presume nobody has a contact at 10297? From: Outages <outages-bounces@outages.org> On Behalf Of Joseph B via Outages Sent: 24 April 2018 13:56 To: outages@outages.org Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains Tue Apr 24 11:05:41 UTC onwards one of Hurricane Electric's peers AS10297 started advertising the following subnets via HE. 205.251.192.0 205.251.193.0 205.251.195.0 205.251.197.0 205.251.199.0 These are all Amazon subnets, usually originated as part of /23s and seemingly host a fair bit of AWS Route53. If you (or your DNS resolver) are a HE transit customer you will be impacted the most. Cheers, Joseph On Tue, Apr 24, 2018, at 9:50 PM, Phil Lavin via Outages wrote: This doesn’t feel right, though I’ll admit I’ve never checked before. Our only route to ns-163.awsdns-20.com (205.251.192.163) is through HE: inet.0: 757581 destinations, 2107440 routes (757301 active, 0 holddown, 522 hidden) + = Active Route, - = Last Active, * = Both 205.251.192.0/24 *[BGP/170] 01:12:08, localpref 70 AS path: 6939 10297 I, validation-state: unverified > to 216.66.90.21 via ge-1/0/5.0 AS10297 is eNET inc. Is this expected? From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Phil Lavin via Outages Sent: 24 April 2018 13:04 To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains Looks more specific to AWS than it does to Google+AWS. Can’t resolve against some of AWS’s NS directly: phil@phil-debian:~$ dig cloudcall.com IN A @ns-163.awsdns-20.com ; <<>> DiG 9.10.3-P4-Debian <<>> cloudcall.com IN A @ns-163.awsdns-20.com ;; global options: +cmd ;; connection timed out; no servers could be reached From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Phil Lavin via Outages Sent: 24 April 2018 12:56 To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains Yeh. Still digging into it. From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Zach Hanna via Outages Sent: 24 April 2018 12:54 To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] Google 8.8.8.8 Resolution of Route53 domains Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve with Google DNS? _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages

Those prefixes had been withdrawn now – traffic is flowing correctly again for us. If that was the cause, I suspect things are back to rights for everyone now? One wonders why HE doesn’t apply filters on a peer with 20 legit prefixes… From: Outages <outages-bounces@outages.org> On Behalf Of Joseph B via Outages Sent: 24 April 2018 13:56 To: outages@outages.org Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains Tue Apr 24 11:05:41 UTC onwards one of Hurricane Electric's peers AS10297 started advertising the following subnets via HE. 205.251.192.0 205.251.193.0 205.251.195.0 205.251.197.0 205.251.199.0 These are all Amazon subnets, usually originated as part of /23s and seemingly host a fair bit of AWS Route53. If you (or your DNS resolver) are a HE transit customer you will be impacted the most. Cheers, Joseph On Tue, Apr 24, 2018, at 9:50 PM, Phil Lavin via Outages wrote: This doesn’t feel right, though I’ll admit I’ve never checked before. Our only route to ns-163.awsdns-20.com (205.251.192.163) is through HE: inet.0: 757581 destinations, 2107440 routes (757301 active, 0 holddown, 522 hidden) + = Active Route, - = Last Active, * = Both 205.251.192.0/24 *[BGP/170] 01:12:08, localpref 70 AS path: 6939 10297 I, validation-state: unverified > to 216.66.90.21 via ge-1/0/5.0 AS10297 is eNET inc. Is this expected? From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Phil Lavin via Outages Sent: 24 April 2018 13:04 To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains Looks more specific to AWS than it does to Google+AWS. Can’t resolve against some of AWS’s NS directly: phil@phil-debian:~$ dig cloudcall.com IN A @ns-163.awsdns-20.com ; <<>> DiG 9.10.3-P4-Debian <<>> cloudcall.com IN A @ns-163.awsdns-20.com ;; global options: +cmd ;; connection timed out; no servers could be reached From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Phil Lavin via Outages Sent: 24 April 2018 12:56 To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains Yeh. Still digging into it. From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Zach Hanna via Outages Sent: 24 April 2018 12:54 To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] Google 8.8.8.8 Resolution of Route53 domains Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve with Google DNS? _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages

Resolved here too.. On Tue, Apr 24, 2018 at 7:30 AM Phil Lavin via Outages <outages@outages.org> wrote:
Those prefixes had been withdrawn now – traffic is flowing correctly again for us. If that was the cause, I suspect things are back to rights for everyone now?
One wonders why HE doesn’t apply filters on a peer with 20 legit prefixes…
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Joseph B via Outages *Sent:* 24 April 2018 13:56
*To:* outages@outages.org *Subject:* Re: [outages] Google 8.8.8.8 Resolution of Route53 domains
Tue Apr 24 11:05:41 UTC onwards one of Hurricane Electric's peers AS10297 started advertising the following subnets via HE.
205.251.192.0
205.251.193.0
205.251.195.0
205.251.197.0
205.251.199.0
These are all Amazon subnets, usually originated as part of /23s and seemingly host a fair bit of AWS Route53.
If you (or your DNS resolver) are a HE transit customer you will be impacted the most.
Cheers,
Joseph
On Tue, Apr 24, 2018, at 9:50 PM, Phil Lavin via Outages wrote:
This doesn’t feel right, though I’ll admit I’ve never checked before. Our only route to ns-163.awsdns-20.com (205.251.192.163) is through HE:
inet.0: 757581 destinations, 2107440 routes (757301 active, 0 holddown, 522 hidden)
+ = Active Route, - = Last Active, * = Both
205.251.192.0/24 *[BGP/170] 01:12:08, localpref 70
AS path: 6939 10297 I, validation-state: unverified
> to 216.66.90.21 via ge-1/0/5.0
AS10297 is eNET inc. Is this expected?
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Phil Lavin via Outages *Sent:* 24 April 2018 13:04 *To:* outages@outages.org *Subject:* Re: [outages] Google 8.8.8.8 Resolution of Route53 domains
Looks more specific to AWS than it does to Google+AWS. Can’t resolve against some of AWS’s NS directly:
phil@phil-debian:~$ dig cloudcall.com IN A @ns-163.awsdns-20.com
; <<>> DiG 9.10.3-P4-Debian <<>> cloudcall.com IN A @ns-163.awsdns-20.com
;; global options: +cmd
;; connection timed out; no servers could be reached
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Phil Lavin via Outages *Sent:* 24 April 2018 12:56 *To:* outages@outages.org *Subject:* Re: [outages] Google 8.8.8.8 Resolution of Route53 domains
Yeh. Still digging into it.
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Zach Hanna via Outages *Sent:* 24 April 2018 12:54 *To:* outages@outages.org *Subject:* [outages] Google 8.8.8.8 Resolution of Route53 domains
Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve with Google DNS?
*_______________________________________________*
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

I suspect this was related to this issue (via ycombinator hacker news): https://doublepulsar.com/hijack-of-amazons-internet-domain-service-used-to- reroute-web-traffic-for-two-hours-unnoticed-3a6f0dda6a6f On Tue, Apr 24, 2018 at 8:51 AM, Zach Hanna via Outages <outages@outages.org
wrote:
Resolved here too..
On Tue, Apr 24, 2018 at 7:30 AM Phil Lavin via Outages < outages@outages.org> wrote:
Those prefixes had been withdrawn now – traffic is flowing correctly again for us. If that was the cause, I suspect things are back to rights for everyone now?
One wonders why HE doesn’t apply filters on a peer with 20 legit prefixes…
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Joseph B via Outages *Sent:* 24 April 2018 13:56
*To:* outages@outages.org *Subject:* Re: [outages] Google 8.8.8.8 Resolution of Route53 domains
Tue Apr 24 11:05:41 UTC onwards one of Hurricane Electric's peers AS10297 started advertising the following subnets via HE.
205.251.192.0
205.251.193.0
205.251.195.0
205.251.197.0
205.251.199.0
These are all Amazon subnets, usually originated as part of /23s and seemingly host a fair bit of AWS Route53.
If you (or your DNS resolver) are a HE transit customer you will be impacted the most.
Cheers,
Joseph
On Tue, Apr 24, 2018, at 9:50 PM, Phil Lavin via Outages wrote:
This doesn’t feel right, though I’ll admit I’ve never checked before. Our only route to ns-163.awsdns-20.com (205.251.192.163) is through HE:
inet.0: 757581 destinations, 2107440 routes (757301 active, 0 holddown, 522 hidden)
+ = Active Route, - = Last Active, * = Both
205.251.192.0/24 *[BGP/170] 01:12:08, localpref 70
AS path: 6939 10297 I, validation-state: unverified
> to 216.66.90.21 via ge-1/0/5.0
AS10297 is eNET inc. Is this expected?
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Phil Lavin via Outages *Sent:* 24 April 2018 13:04 *To:* outages@outages.org *Subject:* Re: [outages] Google 8.8.8.8 Resolution of Route53 domains
Looks more specific to AWS than it does to Google+AWS. Can’t resolve against some of AWS’s NS directly:
phil@phil-debian:~$ dig cloudcall.com IN A @ns-163.awsdns-20.com
; <<>> DiG 9.10.3-P4-Debian <<>> cloudcall.com IN A @ns-163.awsdns-20.com
;; global options: +cmd
;; connection timed out; no servers could be reached
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Phil Lavin via Outages *Sent:* 24 April 2018 12:56 *To:* outages@outages.org *Subject:* Re: [outages] Google 8.8.8.8 Resolution of Route53 domains
Yeh. Still digging into it.
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Zach Hanna via Outages *Sent:* 24 April 2018 12:54 *To:* outages@outages.org *Subject:* [outages] Google 8.8.8.8 Resolution of Route53 domains
Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve with Google DNS?
*_______________________________________________*
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- -Ryan McGinnis Platte Valley Communications 308-237-9512 PGP: 62E39BC1

Here is a more detailed analysis of what happened: https://arstechnica.com/information-technology/2018/04/suspicious-event-hija... On Apr 24, 2018, at 14:19 , Ryan McGinnis via Outages wrote: I suspect this was related to this issue (via ycombinator hacker news): https://doublepulsar.com/hijack-of-amazons-internet-domain-service-used-to-r... On Tue, Apr 24, 2018 at 8:51 AM, Zach Hanna via Outages wrote: Resolved here too.. On Tue, Apr 24, 2018 at 7:30 AM Phil Lavin via Outages wrote: Those prefixes had been withdrawn now – traffic is flowing correctly again for us. If that was the cause, I suspect things are back to rights for everyone now? One wonders why HE doesn’t apply filters on a peer with 20 legit prefixes… From: Outages On Behalf Of Joseph B via Outages Sent: 24 April 2018 13:56 To: outages@outages.org Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains Tue Apr 24 11:05:41 UTC onwards one of Hurricane Electric's peers AS10297 started advertising the following subnets via HE. 205.251.192.0 205.251.193.0 205.251.195.0 205.251.197.0 205.251.199.0 These are all Amazon subnets, usually originated as part of /23s and seemingly host a fair bit of AWS Route53. If you (or your DNS resolver) are a HE transit customer you will be impacted the most. Cheers, Joseph On Tue, Apr 24, 2018, at 9:50 PM, Phil Lavin via Outages wrote: This doesn’t feel right, though I’ll admit I’ve never checked before. Our only route to ns-163.awsdns-20.com (205.251.192.163) is through HE: inet.0: 757581 destinations, 2107440 routes (757301 active, 0 holddown, 522 hidden) + = Active Route, - = Last Active, * = Both 205.251.192.0/24 *[BGP/170] 01:12:08, localpref 70 AS path: 6939 10297 I, validation-state: unverified > to 216.66.90.21 via ge-1/0/5.0 AS10297 is eNET inc. Is this expected? From: Outages On Behalf Of Phil Lavin via Outages Sent: 24 April 2018 13:04 To: outages@outages.org Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains Looks more specific to AWS than it does to Google+AWS. Can’t resolve against some of AWS’s NS directly: phil@phil-debian:~$ dig cloudcall.com IN A @ns-163.awsdns-20.com ; DiG 9.10.3-P4-Debian cloudcall.com IN A @ns-163.awsdns-20.com ;; global options: +cmd ;; connection timed out; no servers could be reached From: Outages On Behalf Of Phil Lavin via Outages Sent: 24 April 2018 12:56 To: outages@outages.org Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains Yeh. Still digging into it. From: Outages On Behalf Of Zach Hanna via Outages Sent: 24 April 2018 12:54 To: outages@outages.org Subject: [outages] Google 8.8.8.8 Resolution of Route53 domains Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve with Google DNS? _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages -- -Ryan McGinnis Platte Valley Communications 308-237-9512 PGP: 62E39BC1 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Hi, On Tue, Apr 24, 2018 at 07:47:36PM -0400, J Kibler via Outages wrote:
Here is a more detailed analysis of what happened: https://arstechnica.com/information-technology/2018/04/suspicious-event-hija...
Maybe that is actually good news. Financial damages have been done, by a US company, to a US company, due to neglicience in BGP filtering. This is going to be an expensive lawsuit, and hopefully people will start proper BGP filtering afterwards... gert -- "If was one thing all people took for granted, was conviction that if you feed honest figures into a computer, honest figures come out. Never doubted it myself till I met a computer with a sense of humor." Robert A. Heinlein, The Moon is a Harsh Mistress Gert Doering - Munich, Germany gert@greenie.muc.de

That would require 1) the wallet company understanding the technical details of the outage (a stretch), and 2) understanding that it is preventable, and 3) how and by whom. On Wed, Apr 25, 2018 at 12:04 AM Gert Doering via Outages < outages@outages.org> wrote:
Hi,
On Tue, Apr 24, 2018 at 07:47:36PM -0400, J Kibler via Outages wrote:
Here is a more detailed analysis of what happened:
https://arstechnica.com/information-technology/2018/04/suspicious-event-hija...
Maybe that is actually good news.
Financial damages have been done, by a US company, to a US company, due to neglicience in BGP filtering. This is going to be an expensive lawsuit, and hopefully people will start proper BGP filtering afterwards...
gert -- "If was one thing all people took for granted, was conviction that if you feed honest figures into a computer, honest figures come out. Never doubted it myself till I met a computer with a sense of humor." Robert A. Heinlein, The Moon is a Harsh Mistress
Gert Doering - Munich, Germany gert@greenie.muc.de _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Since the outage is over we should probably take further discussions off-list. On Wed, Apr 25, 2018 at 8:35 AM Zach Hanna <chezbut@gmail.com> wrote:
That would require 1) the wallet company understanding the technical details of the outage (a stretch), and 2) understanding that it is preventable, and 3) how and by whom.
On Wed, Apr 25, 2018 at 12:04 AM Gert Doering via Outages < outages@outages.org> wrote:
Hi,
On Tue, Apr 24, 2018 at 07:47:36PM -0400, J Kibler via Outages wrote:
Here is a more detailed analysis of what happened:
https://arstechnica.com/information-technology/2018/04/suspicious-event-hija...
Maybe that is actually good news.
Financial damages have been done, by a US company, to a US company, due to neglicience in BGP filtering. This is going to be an expensive lawsuit, and hopefully people will start proper BGP filtering afterwards...
gert -- "If was one thing all people took for granted, was conviction that if you feed honest figures into a computer, honest figures come out. Never doubted it myself till I met a computer with a sense of humor." Robert A. Heinlein, The Moon is a Harsh Mistress
Gert Doering - Munich, Germany gert@greenie.muc.de _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

UPDATE: Amazon confirms a widespread issue with Route53 in us-east. On Tue, Apr 24, 2018 at 5:54 AM, Zach Hanna <chezbut@gmail.com> wrote:
Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve with Google DNS?

Where are you seeing them confirm an issue in us-east w/ r53? On Tue, Apr 24, 2018 at 7:57 AM, Zach Hanna via Outages <outages@outages.org
wrote:
UPDATE: Amazon confirms a widespread issue with Route53 in us-east.
On Tue, Apr 24, 2018 at 5:54 AM, Zach Hanna <chezbut@gmail.com> wrote:
Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve with Google DNS?
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Neil Hanlon DevOps Engineer +1 978 451 0707 K A Y A K SWOODOO ▪ checkfelix ▪ momondo ▪ Cheapflights ▪ Mundi

On Tue, Apr 24, 2018 at 02:01:09PM +0200, Stephane Bortzmeyer via Outages <outages@outages.org> wrote a message of 11 lines which said:
Can you provide a few names?
meetup.com instagram.com Seen by the RIPE Atlas probes: % blaeu-resolve -r 100 -q A --nameserver 8.8.8.8 instagram.com Nameserver 8.8.8.8 [ERROR: SERVFAIL] : 54 occurrences [52.44.104.241 52.54.142.35 52.70.114.30 52.70.238.183 54.173.69.129 54.208.115.50 54.208.121.7 54.209.17.35] : 1 occurrences [8.7.198.45] : 1 occurrences [TIMEOUT(S)] : 44 occurrences Test #12293792 done at 2018-04-24T12:12:03Z

Confirmed Route53 affected on AWS status dashboard. Get Outlook for Android<https://aka.ms/ghei36> ________________________________ From: Outages <outages-bounces@outages.org> on behalf of Stephane Bortzmeyer via Outages <outages@outages.org> Sent: Tuesday, April 24, 2018 5:22:49 AM To: outages@outages.org Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains On Tue, Apr 24, 2018 at 02:01:09PM +0200, Stephane Bortzmeyer via Outages <outages@outages.org> wrote a message of 11 lines which said:
Can you provide a few names?
meetup.com instagram.com Seen by the RIPE Atlas probes: % blaeu-resolve -r 100 -q A --nameserver 8.8.8.8 instagram.com Nameserver 8.8.8.8 [ERROR: SERVFAIL] : 54 occurrences [52.44.104.241 52.54.142.35 52.70.114.30 52.70.238.183 54.173.69.129 54.208.115.50 54.208.121.7 54.209.17.35] : 1 occurrences [8.7.198.45] : 1 occurrences [TIMEOUT(S)] : 44 occurrences Test #12293792 done at 2018-04-24T12:12:03Z _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

https://pulse.turbobytes.com/results/5adf25e4ecbe40692e003abb/ 2 out of 4 nameservers for one of my domains hosted on route53 fail consistently from some locations, directly querying route53. Feels like a general outage, not specific to 8.8.8.8 On Tue, Apr 24, 2018 at 7:38 PM Russell Zen via Outages <outages@outages.org> wrote:
Confirmed Route53 affected on AWS status dashboard.
Get Outlook for Android <https://aka.ms/ghei36>
------------------------------ *From:* Outages <outages-bounces@outages.org> on behalf of Stephane Bortzmeyer via Outages <outages@outages.org> *Sent:* Tuesday, April 24, 2018 5:22:49 AM
*To:* outages@outages.org *Subject:* Re: [outages] Google 8.8.8.8 Resolution of Route53 domains On Tue, Apr 24, 2018 at 02:01:09PM +0200, Stephane Bortzmeyer via Outages <outages@outages.org> wrote a message of 11 lines which said:
Can you provide a few names?
meetup.com instagram.com
Seen by the RIPE Atlas probes:
% blaeu-resolve -r 100 -q A --nameserver 8.8.8.8 instagram.com Nameserver 8.8.8.8 [ERROR: SERVFAIL] : 54 occurrences [52.44.104.241 52.54.142.35 52.70.114.30 52.70.238.183 54.173.69.129 54.208.115.50 54.208.121.7 54.209.17.35] : 1 occurrences [8.7.198.45] : 1 occurrences [TIMEOUT(S)] : 44 occurrences Test #12293792 done at 2018-04-24T12:12:03Z _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

On Tue, Apr 24, 2018 at 12:45:06PM +0000, Sajal Kayan <sajal83@gmail.com> wrote a message of 146 lines which said:
Feels like a general outage, not specific to 8.8.8.8
I tend to disagree. See with 25 RIPE Atlas probes: With Google Public DNS : % blaeu-resolve -r 25 --nameserver 8.8.8.8 instagram.com Nameserver 8.8.8.8 [ERROR: SERVFAIL] : 14 occurrences [TIMEOUT(S)] : 9 occurrences Test #12294276 done at 2018-04-24T12:46:17Z With the default resolver (sometimes Google Public DNS, sometimes not): % blaeu-resolve -r 25 instagram.com [2406:da00:ff00::3436:8e23 2406:da00:ff00::3446:721e 2406:da00:ff00::3446:eeb7 2406:da00:ff00::34cc:e891 2406:da00:ff00::36ad:4581 2406:da00:ff00::36d0:7332 2406:da00:ff00::36d0:7907 2406:da00:ff00::36d1:1123] : 1 occurrences [2406:da00:ff00::22e1:dd34 2406:da00:ff00::22e3:8da6 2406:da00:ff00::3414:dcb1 2406:da00:ff00::3415:5ced 2406:da00:ff00::3416:705d 2406:da00:ff00::3416:79b3 2406:da00:ff00::3416:a694 2406:da00:ff00::3448:914f] : 1 occurrences [2406:da00:ff00::22c2:3b71 2406:da00:ff00::22c4:6777 2406:da00:ff00::22c7:3507 2406:da00:ff00::22e0:b7a 2406:da00:ff00::22e1:dd34 2406:da00:ff00::22e3:8da6 2406:da00:ff00::34c8:f395 2406:da00:ff00::369c:f239] : 1 occurrences [2406:da00:ff00::22c0:9c50 2406:da00:ff00::22c1:9fa1 2406:da00:ff00::22c7:bdda 2406:da00:ff00::22e2:135a 2406:da00:ff00::342c:f9cd 2406:da00:ff00::34cb:8741 2406:da00:ff00::36ac:1f08 2406:da00:ff00::36ec:6f50] : 1 occurrences [2406:da00:ff00::22c2:3b71 2406:da00:ff00::22c4:6777 2406:da00:ff00::3416:a694 2406:da00:ff00::3448:914f 2406:da00:ff00::3449:ae77 2406:da00:ff00::3457:418e 2406:da00:ff00::34c8:f395 2406:da00:ff00::369c:f239] : 1 occurrences [ERROR: SERVFAIL] : 2 occurrences [2406:da00:ff00::22c4:8be5 2406:da00:ff00::22c4:926f 2406:da00:ff00::3446:eeb7 2406:da00:ff00::34cc:e891 2406:da00:ff00::36ad:4581 2406:da00:ff00::36d0:7332 2406:da00:ff00::36d0:7907 2406:da00:ff00::36d1:1123] : 1 occurrences [2406:da00:ff00::22c0:dc59 2406:da00:ff00::3406:4b91 2406:da00:ff00::3407:2ad0 2406:da00:ff00::34c8:2550 2406:da00:ff00::34cc:59e0 2406:da00:ff00::34ce:a080 2406:da00:ff00::3656:36bf 2406:da00:ff00::36d1:7380] : 1 occurrences [TIMEOUT(S)] : 3 occurrences [2406:da00:ff00::22c0:9c50 2406:da00:ff00::22c1:9fa1 2406:da00:ff00::22c7:bdda 2406:da00:ff00::22e2:135a 2406:da00:ff00::22e9:ffc5 2406:da00:ff00::23aa:5823 2406:da00:ff00::3404:8bc6 2406:da00:ff00::3416:3580] : 2 occurrences [2406:da00:ff00::22c2:3b71 2406:da00:ff00::22c4:6777 2406:da00:ff00::22c7:3507 2406:da00:ff00::22e0:b7a 2406:da00:ff00::22e1:dd34 2406:da00:ff00::3457:418e 2406:da00:ff00::34c8:f395 2406:da00:ff00::369c:f239] : 2 occurrences [2406:da00:ff00::22c2:3b71 2406:da00:ff00::22c4:6777 2406:da00:ff00::22c7:3507 2406:da00:ff00::3448:914f 2406:da00:ff00::3449:ae77 2406:da00:ff00::3457:418e 2406:da00:ff00::34c8:f395 2406:da00:ff00::369c:f239] : 2 occurrences [2406:da00:ff00::22c0:dc59 2406:da00:ff00::22c4:9e11 2406:da00:ff00::22c6:38da 2406:da00:ff00::22e5:803 2406:da00:ff00::3405:1c5f 2406:da00:ff00::3406:4b91 2406:da00:ff00::3407:2ad0 2406:da00:ff00::34c8:2550] : 1 occurrences [2406:da00:ff00::22c0:dc59 2406:da00:ff00::22c4:9e11 2406:da00:ff00::22c6:38da 2406:da00:ff00::22e5:803 2406:da00:ff00::3405:1c5f 2406:da00:ff00::3406:4b91 2406:da00:ff00::3407:2ad0 2406:da00:ff00::36d1:7380] : 1 occurrences [2406:da00:ff00::3416:705d 2406:da00:ff00::3416:79b3 2406:da00:ff00::3416:a694 2406:da00:ff00::3448:914f 2406:da00:ff00::3449:ae77 2406:da00:ff00::3457:418e 2406:da00:ff00::34c8:f395 2406:da00:ff00::369c:f239] : 1 occurrences [2406:da00:ff00::22c2:3b71 2406:da00:ff00::3416:79b3 2406:da00:ff00::3416:a694 2406:da00:ff00::3448:914f 2406:da00:ff00::3449:ae77 2406:da00:ff00::3457:418e 2406:da00:ff00::34c8:f395 2406:da00:ff00::369c:f239] : 1 occurrences [2406:da00:ff00::22c4:9e11 2406:da00:ff00::22c6:38da 2406:da00:ff00::22e5:803 2406:da00:ff00::3405:1c5f 2406:da00:ff00::3406:4b91 2406:da00:ff00::3407:2ad0 2406:da00:ff00::34c8:2550 2406:da00:ff00::34cc:59e0] : 1 occurrences [2406:da00:ff00::22c4:8be5 2406:da00:ff00::22c4:926f 2406:da00:ff00::22c5:780c 2406:da00:ff00::22c6:25e5 2406:da00:ff00::22cf:2a5 2406:da00:ff00::22e0:b113 2406:da00:ff00::2399:f729 2406:da00:ff00::342c:68f1] : 1 occurrences [2406:da00:ff00::2399:f729 2406:da00:ff00::342c:68f1 2406:da00:ff00::3436:8e23 2406:da00:ff00::3446:721e 2406:da00:ff00::3446:eeb7 2406:da00:ff00::34cc:e891 2406:da00:ff00::36ad:4581 2406:da00:ff00::36d0:7332] : 1 occurrences Test #12294283 done at 2018-04-24T12:48:13Z

On Tue, Apr 24, 2018 at 02:49:17PM +0200, Stephane Bortzmeyer <bortzmeyer@nic.fr> wrote a message of 39 lines which said:
Feels like a general outage, not specific to 8.8.8.8
I tend to disagree. See with 25 RIPE Atlas probes:
With the default resolver (sometimes Google Public DNS, sometimes not):
If you look at the resolvers which SERVFAIL: % blaeu-resolve -r 25 --displayresolvers -4 instagram.com ... [ERROR: SERVFAIL] : 3 occurrences (resolvers ['10.0.0.194', '192.168.12.254', '8.8.4.4']) Test #12294305 done at 2018-04-24T12:50:42Z (Sometimes, RFC 1918 resolvers actually forward to Google Public DNS or another public resolver.)

(Sometimes, RFC 1918 resolvers actually forward to Google Public DNS or another public resolver.)
My point about it being general outage referred to the fact that many users could not get answer from route53, even if they directly contacted aws without involvement of Google. DNS during the outage: https://pulse.turbobytes.com/results/5adf25e4ecbe40692e003abb/ DNS now: https://pulse.turbobytes.com/results/5adf2a6becbe40692e003aee/ MTR during the outage: https://pulse.turbobytes.com/results/5adf2844ecbe40692e003ad2/ MTR now: https://pulse.turbobytes.com/results/5adf2a60ecbe40692e003aec/
205.251.192.0 205.251.193.0 205.251.195.0 205.251.197.0 205.251.199.0
instagram.com simply had the unfortunate misfortune of having all their assigned nameservers in the block Joseph mentioned, and perhaps Google using HE upstream. The domain I included in the results above was lucky to have 2 nameservers (205.251.194.143 and 205.251.196.84) not included in the above list. On Tue, Apr 24, 2018 at 8:04 PM Stephane Bortzmeyer via Outages < outages@outages.org> wrote:
On Tue, Apr 24, 2018 at 02:49:17PM +0200, Stephane Bortzmeyer <bortzmeyer@nic.fr> wrote a message of 39 lines which said:
Feels like a general outage, not specific to 8.8.8.8
I tend to disagree. See with 25 RIPE Atlas probes:
With the default resolver (sometimes Google Public DNS, sometimes not):
If you look at the resolvers which SERVFAIL:
% blaeu-resolve -r 25 --displayresolvers -4 instagram.com ... [ERROR: SERVFAIL] : 3 occurrences (resolvers ['10.0.0.194', '192.168.12.254', '8.8.4.4']) Test #12294305 done at 2018-04-24T12:50:42Z
(Sometimes, RFC 1918 resolvers actually forward to Google Public DNS or another public resolver.) _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Feels like a general outage, not specific to 8.8.8.8
I tend to disagree. See with 25 RIPE Atlas probes:
I see your point. It appears different route53 zones are behaving differently. for example from my local network, instagram.com doesn't resolve, but my own r53 hosted domains do. On Tue, Apr 24, 2018 at 7:49 PM Stephane Bortzmeyer <bortzmeyer@nic.fr> wrote:
On Tue, Apr 24, 2018 at 12:45:06PM +0000, Sajal Kayan <sajal83@gmail.com> wrote a message of 146 lines which said:
Feels like a general outage, not specific to 8.8.8.8
I tend to disagree. See with 25 RIPE Atlas probes:
With Google Public DNS :
% blaeu-resolve -r 25 --nameserver 8.8.8.8 instagram.com Nameserver 8.8.8.8 [ERROR: SERVFAIL] : 14 occurrences [TIMEOUT(S)] : 9 occurrences Test #12294276 done at 2018-04-24T12:46:17Z
With the default resolver (sometimes Google Public DNS, sometimes not):
% blaeu-resolve -r 25 instagram.com [2406:da00:ff00::3436:8e23 2406:da00:ff00::3446:721e 2406:da00:ff00::3446:eeb7 2406:da00:ff00::34cc:e891 2406:da00:ff00::36ad:4581 2406:da00:ff00::36d0:7332 2406:da00:ff00::36d0:7907 2406:da00:ff00::36d1:1123] : 1 occurrences [2406:da00:ff00::22e1:dd34 2406:da00:ff00::22e3:8da6 2406:da00:ff00::3414:dcb1 2406:da00:ff00::3415:5ced 2406:da00:ff00::3416:705d 2406:da00:ff00::3416:79b3 2406:da00:ff00::3416:a694 2406:da00:ff00::3448:914f] : 1 occurrences [2406:da00:ff00::22c2:3b71 2406:da00:ff00::22c4:6777 2406:da00:ff00::22c7:3507 2406:da00:ff00::22e0:b7a 2406:da00:ff00::22e1:dd34 2406:da00:ff00::22e3:8da6 2406:da00:ff00::34c8:f395 2406:da00:ff00::369c:f239] : 1 occurrences [2406:da00:ff00::22c0:9c50 2406:da00:ff00::22c1:9fa1 2406:da00:ff00::22c7:bdda 2406:da00:ff00::22e2:135a 2406:da00:ff00::342c:f9cd 2406:da00:ff00::34cb:8741 2406:da00:ff00::36ac:1f08 2406:da00:ff00::36ec:6f50] : 1 occurrences [2406:da00:ff00::22c2:3b71 2406:da00:ff00::22c4:6777 2406:da00:ff00::3416:a694 2406:da00:ff00::3448:914f 2406:da00:ff00::3449:ae77 2406:da00:ff00::3457:418e 2406:da00:ff00::34c8:f395 2406:da00:ff00::369c:f239] : 1 occurrences [ERROR: SERVFAIL] : 2 occurrences [2406:da00:ff00::22c4:8be5 2406:da00:ff00::22c4:926f 2406:da00:ff00::3446:eeb7 2406:da00:ff00::34cc:e891 2406:da00:ff00::36ad:4581 2406:da00:ff00::36d0:7332 2406:da00:ff00::36d0:7907 2406:da00:ff00::36d1:1123] : 1 occurrences [2406:da00:ff00::22c0:dc59 2406:da00:ff00::3406:4b91 2406:da00:ff00::3407:2ad0 2406:da00:ff00::34c8:2550 2406:da00:ff00::34cc:59e0 2406:da00:ff00::34ce:a080 2406:da00:ff00::3656:36bf 2406:da00:ff00::36d1:7380] : 1 occurrences [TIMEOUT(S)] : 3 occurrences [2406:da00:ff00::22c0:9c50 2406:da00:ff00::22c1:9fa1 2406:da00:ff00::22c7:bdda 2406:da00:ff00::22e2:135a 2406:da00:ff00::22e9:ffc5 2406:da00:ff00::23aa:5823 2406:da00:ff00::3404:8bc6 2406:da00:ff00::3416:3580] : 2 occurrences [2406:da00:ff00::22c2:3b71 2406:da00:ff00::22c4:6777 2406:da00:ff00::22c7:3507 2406:da00:ff00::22e0:b7a 2406:da00:ff00::22e1:dd34 2406:da00:ff00::3457:418e 2406:da00:ff00::34c8:f395 2406:da00:ff00::369c:f239] : 2 occurrences [2406:da00:ff00::22c2:3b71 2406:da00:ff00::22c4:6777 2406:da00:ff00::22c7:3507 2406:da00:ff00::3448:914f 2406:da00:ff00::3449:ae77 2406:da00:ff00::3457:418e 2406:da00:ff00::34c8:f395 2406:da00:ff00::369c:f239] : 2 occurrences [2406:da00:ff00::22c0:dc59 2406:da00:ff00::22c4:9e11 2406:da00:ff00::22c6:38da 2406:da00:ff00::22e5:803 2406:da00:ff00::3405:1c5f 2406:da00:ff00::3406:4b91 2406:da00:ff00::3407:2ad0 2406:da00:ff00::34c8:2550] : 1 occurrences [2406:da00:ff00::22c0:dc59 2406:da00:ff00::22c4:9e11 2406:da00:ff00::22c6:38da 2406:da00:ff00::22e5:803 2406:da00:ff00::3405:1c5f 2406:da00:ff00::3406:4b91 2406:da00:ff00::3407:2ad0 2406:da00:ff00::36d1:7380] : 1 occurrences [2406:da00:ff00::3416:705d 2406:da00:ff00::3416:79b3 2406:da00:ff00::3416:a694 2406:da00:ff00::3448:914f 2406:da00:ff00::3449:ae77 2406:da00:ff00::3457:418e 2406:da00:ff00::34c8:f395 2406:da00:ff00::369c:f239] : 1 occurrences [2406:da00:ff00::22c2:3b71 2406:da00:ff00::3416:79b3 2406:da00:ff00::3416:a694 2406:da00:ff00::3448:914f 2406:da00:ff00::3449:ae77 2406:da00:ff00::3457:418e 2406:da00:ff00::34c8:f395 2406:da00:ff00::369c:f239] : 1 occurrences [2406:da00:ff00::22c4:9e11 2406:da00:ff00::22c6:38da 2406:da00:ff00::22e5:803 2406:da00:ff00::3405:1c5f 2406:da00:ff00::3406:4b91 2406:da00:ff00::3407:2ad0 2406:da00:ff00::34c8:2550 2406:da00:ff00::34cc:59e0] : 1 occurrences [2406:da00:ff00::22c4:8be5 2406:da00:ff00::22c4:926f 2406:da00:ff00::22c5:780c 2406:da00:ff00::22c6:25e5 2406:da00:ff00::22cf:2a5 2406:da00:ff00::22e0:b113 2406:da00:ff00::2399:f729 2406:da00:ff00::342c:68f1] : 1 occurrences [2406:da00:ff00::2399:f729 2406:da00:ff00::342c:68f1 2406:da00:ff00::3436:8e23 2406:da00:ff00::3446:721e 2406:da00:ff00::3446:eeb7 2406:da00:ff00::34cc:e891 2406:da00:ff00::36ad:4581 2406:da00:ff00::36d0:7332] : 1 occurrences Test #12294283 done at 2018-04-24T12:48:13Z

On Tue, Apr 24, 2018 at 05:54:21AM -0600, Zach Hanna via Outages <outages@outages.org> wrote a message of 28 lines which said:
Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve with Google DNS?
It seems now OK. With the RIPE Atlas probes, no more SERVFAIL: % blaeu-resolve -r 25 --nameserver 8.8.8.8 --displayresolvers instagram.com Nameserver 8.8.8.8 [2406:da00:ff00::22c4:8be5 2406:da00:ff00::22c4:926f 2406:da00:ff00::22c5:780c 2406:da00:ff00::22c6:25e5 2406:da00:ff00::22cf:2a5 2406:da00:ff00::36d0:7332 2406:da00:ff00::36d0:7907 2406:da00:ff00::36d1:1123] : 1 occurrences (resolvers ['8.8.8.8']) [2406:da00:ff00::3436:8e23 2406:da00:ff00::3446:721e 2406:da00:ff00::3446:eeb7 2406:da00:ff00::34cc:e891 2406:da00:ff00::36ad:4581 2406:da00:ff00::36d0:7332 2406:da00:ff00::36d0:7907 2406:da00:ff00::36d1:1123] : 1 occurrences (resolvers ['8.8.8.8']) [2406:da00:ff00::22e5:803 2406:da00:ff00::3405:1c5f 2406:da00:ff00::3406:4b91 2406:da00:ff00::3407:2ad0 2406:da00:ff00::34c8:2550 2406:da00:ff00::34cc:59e0 2406:da00:ff00::34ce:a080 2406:da00:ff00::3656:36bf] : 1 occurrences (resolvers ['8.8.8.8']) [2406:da00:ff00::3405:1c5f 2406:da00:ff00::3406:4b91 2406:da00:ff00::3407:2ad0 2406:da00:ff00::34c8:2550 2406:da00:ff00::34cc:59e0 2406:da00:ff00::34ce:a080 2406:da00:ff00::3656:36bf 2406:da00:ff00::36d1:7380] : 1 occurrences (resolvers ['8.8.8.8']) [2406:da00:ff00::22c1:9fa1 2406:da00:ff00::22c7:bdda 2406:da00:ff00::22e2:135a 2406:da00:ff00::22e9:ffc5 2406:da00:ff00::23aa:5823 2406:da00:ff00::3404:8bc6 2406:da00:ff00::3416:3580 2406:da00:ff00::342c:f9cd] : 1 occurrences (resolvers ['8.8.8.8']) [2406:da00:ff00::22c0:9c50 2406:da00:ff00::22c1:9fa1 2406:da00:ff00::22c7:bdda 2406:da00:ff00::22e2:135a 2406:da00:ff00::22e9:ffc5 2406:da00:ff00::23aa:5823 2406:da00:ff00::3404:8bc6 2406:da00:ff00::36ec:6f50] : 1 occurrences (resolvers ['8.8.8.8']) [2406:da00:ff00::22cf:2a5 2406:da00:ff00::22e0:b113 2406:da00:ff00::2399:f729 2406:da00:ff00::342c:68f1 2406:da00:ff00::3436:8e23 2406:da00:ff00::3446:721e 2406:da00:ff00::3446:eeb7 2406:da00:ff00::34cc:e891] : 1 occurrences (resolvers ['8.8.8.8']) [2406:da00:ff00::22c0:dc59 2406:da00:ff00::22c4:9e11 2406:da00:ff00::22c6:38da 2406:da00:ff00::22e5:803 2406:da00:ff00::3405:1c5f 2406:da00:ff00::3406:4b91 2406:da00:ff00::3656:36bf 2406:da00:ff00::36d1:7380] : 4 occurrences (resolvers ['8.8.8.8']) [2406:da00:ff00::22c0:dc59 2406:da00:ff00::22c4:9e11 2406:da00:ff00::22c6:38da 2406:da00:ff00::22e5:803 2406:da00:ff00::34cc:59e0 2406:da00:ff00::34ce:a080 2406:da00:ff00::3656:36bf 2406:da00:ff00::36d1:7380] : 2 occurrences (resolvers ['8.8.8.8']) [2406:da00:ff00::22c4:8be5 2406:da00:ff00::22c4:926f 2406:da00:ff00::22c5:780c 2406:da00:ff00::22c6:25e5 2406:da00:ff00::22cf:2a5 2406:da00:ff00::22e0:b113 2406:da00:ff00::2399:f729 2406:da00:ff00::36d1:1123] : 1 occurrences (resolvers ['8.8.8.8']) [2406:da00:ff00::22c0:9c50 2406:da00:ff00::22c1:9fa1 2406:da00:ff00::3404:8bc6 2406:da00:ff00::3416:3580 2406:da00:ff00::342c:f9cd 2406:da00:ff00::34cb:8741 2406:da00:ff00::36ac:1f08 2406:da00:ff00::36ec:6f50] : 1 occurrences (resolvers ['8.8.8.8']) [2406:da00:ff00::22c2:3b71 2406:da00:ff00::22c4:6777 2406:da00:ff00::22c7:3507 2406:da00:ff00::22e0:b7a 2406:da00:ff00::3449:ae77 2406:da00:ff00::3457:418e 2406:da00:ff00::34c8:f395 2406:da00:ff00::369c:f239] : 3 occurrences (resolvers ['8.8.8.8']) [2406:da00:ff00::22c0:9c50 2406:da00:ff00::23aa:5823 2406:da00:ff00::3404:8bc6 2406:da00:ff00::3416:3580 2406:da00:ff00::342c:f9cd 2406:da00:ff00::34cb:8741 2406:da00:ff00::36ac:1f08 2406:da00:ff00::36ec:6f50] : 1 occurrences (resolvers ['8.8.8.8']) [2406:da00:ff00::22c4:8be5 2406:da00:ff00::22c4:926f 2406:da00:ff00::3446:eeb7 2406:da00:ff00::34cc:e891 2406:da00:ff00::36ad:4581 2406:da00:ff00::36d0:7332 2406:da00:ff00::36d0:7907 2406:da00:ff00::36d1:1123] : 3 occurrences (resolvers ['8.8.8.8']) [2406:da00:ff00::22e0:b7a 2406:da00:ff00::22e1:dd34 2406:da00:ff00::22e3:8da6 2406:da00:ff00::3414:dcb1 2406:da00:ff00::3415:5ced 2406:da00:ff00::3416:705d 2406:da00:ff00::3416:79b3 2406:da00:ff00::3416:a694] : 1 occurrences (resolvers ['8.8.8.8']) [2406:da00:ff00::22c4:8be5 2406:da00:ff00::22c4:926f 2406:da00:ff00::22c5:780c 2406:da00:ff00::22c6:25e5 2406:da00:ff00::36ad:4581 2406:da00:ff00::36d0:7332 2406:da00:ff00::36d0:7907 2406:da00:ff00::36d1:1123] : 1 occurrences (resolvers ['8.8.8.8']) [2406:da00:ff00::22c0:dc59 2406:da00:ff00::22c4:9e11 2406:da00:ff00::22c6:38da 2406:da00:ff00::34c8:2550 2406:da00:ff00::34cc:59e0 2406:da00:ff00::34ce:a080 2406:da00:ff00::3656:36bf 2406:da00:ff00::36d1:7380] : 1 occurrences (resolvers ['8.8.8.8']) Test #12294444 done at 2018-04-24T13:03:09Z
participants (10)
-
Gert Doering
-
Joseph B
-
jrk1231-outml@nym.hush.com
-
Neil Hanlon
-
Phil Lavin
-
Russell Zen
-
Ryan McGinnis
-
Sajal Kayan
-
Stephane Bortzmeyer
-
Zach Hanna