Seeing some hosts and not others get no reply to ICMP pings from the Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2 Minneapolis routers gets no replies. Since many use and assume 8.8.8.8 is invincible, this has now started setting off monitoring alerts. DNS queries so far seem unaffected.
AS786 - not observed any issues from here in last 24h. Best regards, Josh Purcell Me@JoshPurcell.com<mailto:Me@JoshPurcell.com> From: Outages <outages-bounces@outages.org> On Behalf Of Justin Krejci via Outages Sent: 08 February 2022 19:34 To: outages@outages.org Subject: [outages] Ping to Google 8.8.8.8 Seeing some hosts and not others get no reply to ICMP pings from the Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2 Minneapolis routers gets no replies. Since many use and assume 8.8.8.8 is invincible, this has now started setting off monitoring alerts. DNS queries so far seem unaffected.
This happens occasionally on 8.8.8.8 and 8.8.4.4. We used to use these IPs as Internet reachability tests for determining failover to backup circuits and moved away from it after the first occurrence of this happening. -- Christopher Conley Systems Administrator | Fors Marsh Group 1010 N. Glebe Rd, Suite 510 Arlington, VA 22201 From: Outages <outages-bounces@outages.org> On Behalf Of Justin Krejci via Outages Sent: Tuesday, February 8, 2022 2:34 PM To: outages@outages.org Subject: [EXTERNAL] [outages] Ping to Google 8.8.8.8 Seeing some hosts and not others get no reply to ICMP pings from the Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2 Minneapolis routers gets no replies. Since many use and assume 8.8.8.8 is invincible, this has now started setting off monitoring alerts. DNS queries so far seem unaffected.
Is it that atypical for these public-facing services to drop ICMP when they get hot? I'd not consider it an indicator of service down unless you're getting SRVFAILs James Host Architect - CDN Platform E: jhost@vultr.com On Tue, Feb 8, 2022 at 12:01 PM Christopher Conley via Outages < outages@outages.org> wrote:
This happens occasionally on 8.8.8.8 and 8.8.4.4. We used to use these IPs as Internet reachability tests for determining failover to backup circuits and moved away from it after the first occurrence of this happening.
--
Christopher Conley
Systems Administrator | Fors Marsh Group
1010 N. Glebe Rd, Suite 510
Arlington, VA 22201
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Justin Krejci via Outages *Sent:* Tuesday, February 8, 2022 2:34 PM *To:* outages@outages.org *Subject:* [EXTERNAL] [outages] Ping to Google 8.8.8.8
Seeing some hosts and not others get no reply to ICMP pings from the Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2 Minneapolis routers gets no replies. Since many use and assume 8.8.8.8 is invincible, this has now started setting off monitoring alerts. DNS queries so far seem unaffected. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I didn’t set it up, but I did tear it down. From: Outages <outages-bounces@outages.org> On Behalf Of James Host via Outages Sent: Tuesday, February 8, 2022 3:12 PM To: outages@outages.org Subject: Re: [outages] [EXTERNAL] Ping to Google 8.8.8.8 Is it that atypical for these public-facing services to drop ICMP when they get hot? I'd not consider it an indicator of service down unless you're getting SRVFAILs James Host Architect - CDN Platform E: jhost@vultr.com<mailto:jhost@vultr.com> On Tue, Feb 8, 2022 at 12:01 PM Christopher Conley via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: This happens occasionally on 8.8.8.8 and 8.8.4.4. We used to use these IPs as Internet reachability tests for determining failover to backup circuits and moved away from it after the first occurrence of this happening. -- Christopher Conley Systems Administrator | Fors Marsh Group 1010 N. Glebe Rd, Suite 510 Arlington, VA 22201 From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Justin Krejci via Outages Sent: Tuesday, February 8, 2022 2:34 PM To: outages@outages.org<mailto:outages@outages.org> Subject: [EXTERNAL] [outages] Ping to Google 8.8.8.8 Seeing some hosts and not others get no reply to ICMP pings from the Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2 Minneapolis routers gets no replies. Since many use and assume 8.8.8.8 is invincible, this has now started setting off monitoring alerts. DNS queries so far seem unaffected. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Yes, it has been this way for at least a decade. DNS are not polling servers. - Mike Bolitho On Tue, Feb 8, 2022, 1:29 PM James Host via Outages <outages@outages.org> wrote:
Is it that atypical for these public-facing services to drop ICMP when they get hot? I'd not consider it an indicator of service down unless you're getting SRVFAILs
James Host Architect - CDN Platform E: jhost@vultr.com
On Tue, Feb 8, 2022 at 12:01 PM Christopher Conley via Outages < outages@outages.org> wrote:
This happens occasionally on 8.8.8.8 and 8.8.4.4. We used to use these IPs as Internet reachability tests for determining failover to backup circuits and moved away from it after the first occurrence of this happening.
--
Christopher Conley
Systems Administrator | Fors Marsh Group
1010 N. Glebe Rd, Suite 510
Arlington, VA 22201
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Justin Krejci via Outages *Sent:* Tuesday, February 8, 2022 2:34 PM *To:* outages@outages.org *Subject:* [EXTERNAL] [outages] Ping to Google 8.8.8.8
Seeing some hosts and not others get no reply to ICMP pings from the Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2 Minneapolis routers gets no replies. Since many use and assume 8.8.8.8 is invincible, this has now started setting off monitoring alerts. DNS queries so far seem unaffected. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I'm starting to get authentication failures when connecting to IMAP.Gmail.com Payment gateways and things have been really sluggish, jit emails are not coming through in time.. On Tue., Feb. 8, 2022, 3:58 p.m. Mike Bolitho via Outages, < outages@outages.org> wrote:
Yes, it has been this way for at least a decade. DNS are not polling servers.
- Mike Bolitho
On Tue, Feb 8, 2022, 1:29 PM James Host via Outages <outages@outages.org> wrote:
Is it that atypical for these public-facing services to drop ICMP when they get hot? I'd not consider it an indicator of service down unless you're getting SRVFAILs
James Host Architect - CDN Platform E: jhost@vultr.com
On Tue, Feb 8, 2022 at 12:01 PM Christopher Conley via Outages < outages@outages.org> wrote:
This happens occasionally on 8.8.8.8 and 8.8.4.4. We used to use these IPs as Internet reachability tests for determining failover to backup circuits and moved away from it after the first occurrence of this happening.
--
Christopher Conley
Systems Administrator | Fors Marsh Group
1010 N. Glebe Rd, Suite 510
Arlington, VA 22201
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Justin Krejci via Outages *Sent:* Tuesday, February 8, 2022 2:34 PM *To:* outages@outages.org *Subject:* [EXTERNAL] [outages] Ping to Google 8.8.8.8
Seeing some hosts and not others get no reply to ICMP pings from the Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2 Minneapolis routers gets no replies. Since many use and assume 8.8.8.8 is invincible, this has now started setting off monitoring alerts. DNS queries so far seem unaffected. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Tue, Feb 08, 2022 at 07:33:52PM +0000, Justin Krejci via Outages <outages@outages.org> wrote a message of 52 lines which said:
Seeing some hosts and not others get no reply to ICMP pings from the Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2 Minneapolis routers gets no replies. Since many use and assume 8.8.8.8 is invincible, this has now started setting off monitoring alerts. DNS queries so far seem unaffected.
Replying to ICMP echoes is not part of the service Google Public DNS officially does, so it is possible that they sometimes filter and/or rate-limit ICMP echo. It's not an outage.
On Feb 8, 2022, at 2:49 PM, Stephane Bortzmeyer via Outages <outages@outages.org> wrote:
Replying to ICMP echoes is not part of the service Google Public DNS officially does, so it is possible that they sometimes filter and/or rate-limit ICMP echo. It's not an outage.
If this goes on much further, we should probably move to -discuss, but I digress. :) I can confirm I’ve seen this behavior before with 8.8.8.8 - I’ve had times where ICMP echo indicates 100+ms latency to it (whereas 8.8.4.4 is fine), but both will reply to DNS queries with much, much less latency than ICMP echo was indicating. Definitely seems like they’re rate-limiting or de-prioritizing ICMP echo at least on occasion. One should definitely not use ICMP echo against 8.8.8.8 / 8.8.4.4 in monitoring systems (although I know it happens.) This is a good reminder as to why. :) Jeremy
On Tue, 8 Feb 2022 at 19:49, Stephane Bortzmeyer via Outages < outages@outages.org> wrote:
Replying to ICMP echoes is not part of the service Google Public DNS officially does, so it is possible that they sometimes filter and/or rate-limit ICMP echo. It's not an outage.
https://peering.google.com/#/learn-more/faq for more info. M
On 2/8/22 11:33, Justin Krejci via Outages wrote:
Seeing some hosts and not others get no reply to ICMP pings from the Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2 Minneapolis routers gets no replies. Since many use and assume 8.8.8.8 is invincible, this has now started setting off monitoring alerts. DNS queries so far seem unaffected.
Don't do that then. Seriously, public DNS servers were never intended to be used as ping targets used to measure Internet connectivity. -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
On 2/8/22 21:51, Jay Hennigan via Outages wrote:
Don't do that then.
Seriously, public DNS servers were never intended to be used as ping targets used to measure Internet connectivity.
That ship long sailed. If we want to stop this behaviour, we'll have to do something about it, specifically, offering an alternative to Internet users that is regarded as official, e.g., like we do with other public services such as NTP. Mark.
On Wed, 9 Feb 2022, 04:45 Mark Tinka via Outages, <outages@outages.org> wrote:
If we want to stop this behaviour, we'll have to do something about it, specifically, offering an alternative to Internet users that is regarded as official, e.g., like we do with other public services such as NTP.
Do a DNS query. You don't even have to randomise the id number, just query for something that will have a small set of results (so, not the root) and ensure checking is disabled. For 8.8.8.8, I'm guessing "dns.google" is probably an excellent target. If you wanted something generic, what about a PTR query for something in 10/8, directed at the AS112 project? That's pretty much the sinkhole that expects that kind of unwanted traffic... M
On 2/9/22 07:29, Matthew Walster wrote:
Do a DNS query. You don't even have to randomise the id number, just query for something that will have a small set of results (so, not the root) and ensure checking is disabled. For 8.8.8.8, I'm guessing "dns.google" is probably an excellent target.
If you wanted something generic, what about a PTR query for something in 10/8, directed at the AS112 project? That's pretty much the sinkhole that expects that kind of unwanted traffic...
Not a serious problem for you and me, who are operators that know how to do this and understand it well. For Jane, down the road, who doesn't care and just wants her MTV, she's not interested in what a DNS query is. All she is know is the tech-support on the other end of the horn asked her to "ping 8.8.8.8". Mark.
On 2/8/22 9:45 PM, Mark Tinka via Outages wrote:
That ship long sailed.
So. Many ships have sailed, as in left the dock, but never reached their destination.
If we want to stop this behaviour, we'll have to do something about it,
Yes.
specifically, offering an alternative to Internet users that is regarded as official, e.g., like we do with other public services such as NTP.
That's /one/ option. Another option is for Google, et al., to simply drop ICMP packets to their DNS servers. Is this heavy handed? Probably. Is it a viable option to persuade people to not ping said DNS servers? yes. Will it achieve the desired result of having people stop pinging Google's DNS servers? Quite likely. Admittedly with a long tail. -- Grant. . . . unix || die
On 2/9/22 07:30, Grant Taylor via Outages wrote:
Many ships have sailed, as in left the dock, but never reached their destination.
You mean like PMTUd, and such :-)? We probably won't get that one back, and unless we do something, inability to ping 8.8.8.8 will result in unnecessary NOC tickets claiming "the Internet is down".
That's /one/ option.
Another option is for Google, et al., to simply drop ICMP packets to their DNS servers.
Is this heavy handed? Probably.
Is it a viable option to persuade people to not ping said DNS servers? yes.
Will it achieve the desired result of having people stop pinging Google's DNS servers? Quite likely. Admittedly with a long tail.
Yes, this helps Google not having to deal with the problem, but it passes the burden both to the ISP who has to explain why the Internet is now down, and to some other online service who now has to sink ping traffic. Perhaps Yahoo will pick that priviledge up again, like they did back in the day :-(... Mark.
On 2/8/22 11:46 PM, Mark Tinka via Outages wrote:
You mean like PMTUd, and such :-)?
Not what I originally meant, but sure.
We probably won't get that one back, and unless we do something, inability to ping 8.8.8.8 will result in unnecessary NOC tickets claiming "the Internet is down".
Probably some, for a while. (See more below.)
Yes, this helps Google not having to deal with the problem, but it passes the burden both to the ISP who has to explain why the Internet is now down, and to some other online service who now has to sink ping traffic. Perhaps Yahoo will pick that priviledge up again, like they did back in the day :-(...
So ... an end user education issue. - No, the Internet is not down. - The specific test you are doing is (now) bad (for reasons). - See how your $StreamingServiceVideo is still playing? -- Did you receive the test email I just sent you? The Internet is /up/. Who should be responsible for an ISP's user base? I'd naively think that the ISP should be responsible for their own user base. Why should we foist this responsibility off onto another organization? -- Grant. . . . unix || die
This belongs on the discussion list and not on the Outages list. On Feb 9, 2022, 11:38 AM -0700, Grant Taylor via Outages <outages@outages.org>, wrote:
On 2/8/22 11:46 PM, Mark Tinka via Outages wrote:
You mean like PMTUd, and such :-)?
Not what I originally meant, but sure.
We probably won't get that one back, and unless we do something, inability to ping 8.8.8.8 will result in unnecessary NOC tickets claiming "the Internet is down".
Probably some, for a while. (See more below.)
Yes, this helps Google not having to deal with the problem, but it passes the burden both to the ISP who has to explain why the Internet is now down, and to some other online service who now has to sink ping traffic. Perhaps Yahoo will pick that priviledge up again, like they did back in the day :-(...
So ... an end user education issue.
- No, the Internet is not down. - The specific test you are doing is (now) bad (for reasons). - See how your $StreamingServiceVideo is still playing? -- Did you receive the test email I just sent you?
The Internet is /up/.
Who should be responsible for an ISP's user base? I'd naively think that the ISP should be responsible for their own user base. Why should we foist this responsibility off onto another organization?
-- Grant. . . . unix || die
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
AS15169 in Chicago did have issues Tuesday 2022-02-08, which would have affected ICMP traffic to the DNS server IPs. We believe the impact time was roughly 2022-02-08 09:30 - 22:30 CST. I'd be interested in a unicast followup if anyone's still seeing issues today. While indeed we have no SLA for ICMP to Google public DNS, we're not currently *intending* for it to suddenly stop working or otherwise have dramatic behavior shifts. Phil, for AS15169 On Wed, Feb 9, 2022 at 6:51 PM Carlos Alvarez via Outages < outages@outages.org> wrote:
This belongs on the discussion list and not on the Outages list. On Feb 9, 2022, 11:38 AM -0700, Grant Taylor via Outages < outages@outages.org>, wrote:
On 2/8/22 11:46 PM, Mark Tinka via Outages wrote:
You mean like PMTUd, and such :-)?
Not what I originally meant, but sure.
We probably won't get that one back, and unless we do something, inability to ping 8.8.8.8 will result in unnecessary NOC tickets claiming "the Internet is down".
Probably some, for a while. (See more below.)
Yes, this helps Google not having to deal with the problem, but it passes the burden both to the ISP who has to explain why the Internet is now down, and to some other online service who now has to sink ping traffic. Perhaps Yahoo will pick that priviledge up again, like they did back in the day :-(...
So ... an end user education issue.
- No, the Internet is not down. - The specific test you are doing is (now) bad (for reasons). - See how your $StreamingServiceVideo is still playing? -- Did you receive the test email I just sent you?
The Internet is /up/.
Who should be responsible for an ISP's user base? I'd naively think that the ISP should be responsible for their own user base. Why should we foist this responsibility off onto another organization?
-- Grant. . . . unix || die
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On 2/9/22 20:29, Grant Taylor via Outages wrote:
So ... an end user education issue.
- No, the Internet is not down. - The specific test you are doing is (now) bad (for reasons). - See how your $StreamingServiceVideo is still playing? -- Did you receive the test email I just sent you?
The Internet is /up/.
Who should be responsible for an ISP's user base? I'd naively think that the ISP should be responsible for their own user base. Why should we foist this responsibility off onto another organization?
I'd be fine with that, only if it were feasible. Most ISP's do not have the capacity to educate the thousands, tens-of-thousands or millions of customers they service, about why pinging 8.8.8.8 is a bad thing. In some cases, it's because there are simply too many customers to educate in a formal and structured way that sticks. In other cases, the ISP's simply don't have the resources to execute this sort of thing. This does not mean that Google should be responsible for providing a ping target. It just means that the day they choose to stop doing so, we have a massive problem, one we could have been prepared for, but seem to be too lazy to. In any event, providing an alternative that is globally standard, would be far easier than trying to educate customers who only care about getting their MTV. Mark.
On 10/02/2022 04:53, Mark Tinka via Outages wrote:
In any event, providing an alternative that is globally standard, would be far easier than trying to educate customers who only care about getting their MTV.
I'm not sure an AS112-like alternative would be what we want here: for the average user reaching Google/Facebook/Akamai/CloudFront is far more important than reaching some random AS hosting the closest AS112 instance. Overall, it's probably not fair expect to change the behaviour which leads users to test common destinations for validating whether their internet service is available. If we started pushing "ping 8.8.8.8" out of people's mindset, they will likely move to "curl google.com". Giorgio -- www: grg.pw email: me@grg.pw mobile: +44 7716 604314 / +39 393 1049073
ICMPv6 to ipv6.google.com went down for me sometime before 9:44 am. Also likely going through MICE (which is in the Minneapolis area). root@nagios:~/bin# traceroute6 ipv6.google.com traceroute to ipv6.google.com (2607:f8b0:4009:80a::200e), 30 hops max, 80 byte packets 1 router-core.mtcnet.net (2607:fe28:0:1000::1) 0.285 ms 0.328 ms 0.385 ms 2 2001:5f8:7f06:d::1 (2001:5f8:7f06:d::1) 0.200 ms 0.203 ms 0.197 ms 3 2001:5f8:7f06:7::1 (2001:5f8:7f06:7::1) 1.731 ms 1.892 ms 1.888 ms 4 AS15169.micemn.net (2001:504:27::3b41:0:1) 18.262 ms 18.284 ms 18.337 ms 5 2001:4860:0:1010::1 (2001:4860:0:1010::1) 17.176 ms * 17.204 ms 6 * 2001:4860:0:1::572b (2001:4860:0:1::572b) 17.191 ms * 7 ord37s34-in-x0e.1e100.net (2607:f8b0:4009:80a::200e) 17.158 ms 17.194 ms * root@nagios:~/bin# ping6 ipv6.google.com PING ipv6.google.com(ord30s21-in-x0e.1e100.net) 56 data bytes ^C --- ipv6.google.com ping statistics --- 6 packets transmitted, 0 received, 100% packet loss, time 5000ms root@nagios:~/bin# Frank From: Outages <outages-bounces@outages.org> On Behalf Of Justin Krejci via Outages Sent: Tuesday, February 8, 2022 1:34 PM To: outages@outages.org Subject: [outages] Ping to Google 8.8.8.8 Seeing some hosts and not others get no reply to ICMP pings from the Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2 Minneapolis routers gets no replies. Since many use and assume 8.8.8.8 is invincible, this has now started setting off monitoring alerts. DNS queries so far seem unaffected.
participants (16)
-
Carlos Alvarez -
Christopher Conley -
Frank Bulk -
Giorgio Bonfiglio -
Grant Taylor -
James Host -
Jay Hennigan -
Jeremy Gault (KD4NED) -
Josh Purcell -
Justin Krejci -
Mark Tinka -
Matthew Walster -
Mike Bolitho -
Nathanael Newton -
Phil Sykes -
Stephane Bortzmeyer