Belkin [heartbeat.belkin.com]
On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com from my webservers and from subscriber units on a different netblock. Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page. I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why. -Paul
Funny, Just saw something about Belkin routers and a botched firmware upgrade. I wonder if this is what they meant. That being said, it is working fine from Zito Media in Clanton, AL. --John On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:
On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com from my webservers and from subscriber units on a different netblock.
Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page.
I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why.
-Paul
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Where? I keep reading "bad firmware" problems but everything suggests there is no automatic upgrade and I think we can safely assume resi users didn't upgrade their routers over night. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages < outages@outages.org> wrote:
Funny,
Just saw something about Belkin routers and a botched firmware upgrade.
I wonder if this is what they meant.
That being said, it is working fine from Zito Media in Clanton, AL.
--John
On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:
On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com from my webservers and from subscriber units on a different netblock.
Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page.
I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why.
-Paul
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
DSLR: http://www.dslreports.com/forum/r29585798-Belkin-Router-Owners-Take-Note. And I am thinking the same thing you are. Doubt resi users upgraded their router overnight. Like you said, chances are it has to do with that heartbeat. Asked a few folks around the US to ping it, some successful, some not, and they weren't behind anything Belkin related. --John On 10/07/2014 10:48 AM, Josh Luthman wrote:
Where? I keep reading "bad firmware" problems but everything suggests there is no automatic upgrade and I think we can safely assume resi users didn't upgrade their routers over night.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Funny,
Just saw something about Belkin routers and a botched firmware upgrade.
I wonder if this is what they meant.
That being said, it is working fine from Zito Media in Clanton, AL.
--John
On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:
On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com <http://heartbeat.belkin.com> from my webservers and from subscriber units on a different netblock.
Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page.
I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com <http://heartbeat.belkin.com> to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why.
-Paul
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
I think it was initially suspected to be a firmware upgrade, but I doubt that's the case now. It's interesting to me that there are two seemingly unrelated fixes: 1. Put the IP for heartbeat.belkin.com on a loopback 2. Manually add working DNS entries to affected PCs Reports indicate that either fix works, but why? They seem unrelated to one another unless the failing heartbeat messages mess up DNS handling on the router. John On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages < outages@outages.org> wrote:
Where? I keep reading "bad firmware" problems but everything suggests there is no automatic upgrade and I think we can safely assume resi users didn't upgrade their routers over night.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages < outages@outages.org> wrote:
Funny,
Just saw something about Belkin routers and a botched firmware upgrade.
I wonder if this is what they meant.
That being said, it is working fine from Zito Media in Clanton, AL.
--John
On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:
On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com from my webservers and from subscriber units on a different netblock.
Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page.
I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why.
-Paul
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I just saw a forum post where Belkin recommended manually setting DNS to see if that solves the problem. Putting all of this together, this is my best guess as to what happened. 1. Routers regularly send heartbeat messages to heartbeat.belkin.com 2. Some routing failure causes quite a few netblocks (but not all) to be unreachable from Belkin's heartbeat server 3. If the heartbeat fails, something breaks related to DNS handling on the router that does not resolve after a router reset Very interesting. I don't have enough data yet, but I do wonder if the root cause of this all is a routing problem on Belkin's side. John On Tue, Oct 7, 2014 at 10:03 AM, John Neiberger <jneiberger@gmail.com> wrote:
I think it was initially suspected to be a firmware upgrade, but I doubt that's the case now. It's interesting to me that there are two seemingly unrelated fixes:
1. Put the IP for heartbeat.belkin.com on a loopback 2. Manually add working DNS entries to affected PCs
Reports indicate that either fix works, but why? They seem unrelated to one another unless the failing heartbeat messages mess up DNS handling on the router.
John
On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages < outages@outages.org> wrote:
Where? I keep reading "bad firmware" problems but everything suggests there is no automatic upgrade and I think we can safely assume resi users didn't upgrade their routers over night.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages < outages@outages.org> wrote:
Funny,
Just saw something about Belkin routers and a botched firmware upgrade.
I wonder if this is what they meant.
That being said, it is working fine from Zito Media in Clanton, AL.
--John
On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:
On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com from my webservers and from subscriber units on a different netblock.
Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page.
I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why.
-Paul
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I can ping heartbeat.belkin.com and the DSL Reports link and other connected links are stating it was back up roughly 30 to 45 minutes ago, so I think we’re out of the woods. Frank From: Outages [mailto:outages-bounces@outages.org] On Behalf Of John Neiberger via Outages Sent: Tuesday, October 07, 2014 11:04 AM To: Josh Luthman Cc: outages@outages.org; John Barbieri Subject: Re: [outages] Belkin [heartbeat.belkin.com] I think it was initially suspected to be a firmware upgrade, but I doubt that's the case now. It's interesting to me that there are two seemingly unrelated fixes: 1. Put the IP for heartbeat.belkin.com <http://heartbeat.belkin.com> on a loopback 2. Manually add working DNS entries to affected PCs Reports indicate that either fix works, but why? They seem unrelated to one another unless the failing heartbeat messages mess up DNS handling on the router. John On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: Where? I keep reading "bad firmware" problems but everything suggests there is no automatic upgrade and I think we can safely assume resi users didn't upgrade their routers over night. Josh Luthman Office: 937-552-2340 <tel:937-552-2340> Direct: 937-552-2343 <tel:937-552-2343> 1100 Wayne St Suite 1337 Troy, OH 45373 On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: Funny, Just saw something about Belkin routers and a botched firmware upgrade. I wonder if this is what they meant. That being said, it is working fine from Zito Media in Clanton, AL. --John On 10/07/2014 10:17 AM, Paul Miller via Outages wrote: On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com <http://heartbeat.belkin.com> from my webservers and from subscriber units on a different netblock. Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page. I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com <http://heartbeat.belkin.com> to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why. -Paul _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Has anyone else tried the loopback/heartbeat workaround with success? I've seen a few reports so far that this has temporarily solved the problem. John On Tue, Oct 7, 2014 at 10:28 AM, Frank Bulk <frnkblk@iname.com> wrote:
I can ping heartbeat.belkin.com and the DSL Reports link and other connected links are stating it was back up roughly 30 to 45 minutes ago, so I think we’re out of the woods.
Frank
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *John Neiberger via Outages *Sent:* Tuesday, October 07, 2014 11:04 AM *To:* Josh Luthman *Cc:* outages@outages.org; John Barbieri *Subject:* Re: [outages] Belkin [heartbeat.belkin.com]
I think it was initially suspected to be a firmware upgrade, but I doubt that's the case now. It's interesting to me that there are two seemingly unrelated fixes:
1. Put the IP for heartbeat.belkin.com on a loopback
2. Manually add working DNS entries to affected PCs
Reports indicate that either fix works, but why? They seem unrelated to one another unless the failing heartbeat messages mess up DNS handling on the router.
John
On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages < outages@outages.org> wrote:
Where? I keep reading "bad firmware" problems but everything suggests there is no automatic upgrade and I think we can safely assume resi users didn't upgrade their routers over night.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages < outages@outages.org> wrote:
Funny,
Just saw something about Belkin routers and a botched firmware upgrade.
I wonder if this is what they meant.
That being said, it is working fine from Zito Media in Clanton, AL.
--John
On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:
On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com from my webservers and from subscriber units on a different netblock.
Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page.
I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why.
-Paul
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Why do that if Belkin is up and running again? Or are you suggesting that Belkin doesn’t have it all worked out? Frank From: Outages [mailto:outages-bounces@outages.org] On Behalf Of John Neiberger via Outages Sent: Tuesday, October 07, 2014 12:42 PM To: outages@outages.org Subject: Re: [outages] Belkin [heartbeat.belkin.com] Has anyone else tried the loopback/heartbeat workaround with success? I've seen a few reports so far that this has temporarily solved the problem. John On Tue, Oct 7, 2014 at 10:28 AM, Frank Bulk <frnkblk@iname.com <mailto:frnkblk@iname.com> > wrote: I can ping heartbeat.belkin.com <http://heartbeat.belkin.com> and the DSL Reports link and other connected links are stating it was back up roughly 30 to 45 minutes ago, so I think we’re out of the woods. Frank From: Outages [mailto:outages-bounces@outages.org <mailto:outages-bounces@outages.org> ] On Behalf Of John Neiberger via Outages Sent: Tuesday, October 07, 2014 11:04 AM To: Josh Luthman Cc: outages@outages.org <mailto:outages@outages.org> ; John Barbieri Subject: Re: [outages] Belkin [heartbeat.belkin.com <http://heartbeat.belkin.com> ] I think it was initially suspected to be a firmware upgrade, but I doubt that's the case now. It's interesting to me that there are two seemingly unrelated fixes: 1. Put the IP for heartbeat.belkin.com <http://heartbeat.belkin.com> on a loopback 2. Manually add working DNS entries to affected PCs Reports indicate that either fix works, but why? They seem unrelated to one another unless the failing heartbeat messages mess up DNS handling on the router. John On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: Where? I keep reading "bad firmware" problems but everything suggests there is no automatic upgrade and I think we can safely assume resi users didn't upgrade their routers over night. Josh Luthman Office: 937-552-2340 <tel:937-552-2340> Direct: 937-552-2343 <tel:937-552-2343> 1100 Wayne St Suite 1337 Troy, OH 45373 On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: Funny, Just saw something about Belkin routers and a botched firmware upgrade. I wonder if this is what they meant. That being said, it is working fine from Zito Media in Clanton, AL. --John On 10/07/2014 10:17 AM, Paul Miller via Outages wrote: On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com <http://heartbeat.belkin.com> from my webservers and from subscriber units on a different netblock. Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page. I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com <http://heartbeat.belkin.com> to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why. -Paul _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Given the number of current social media postings, it doesn't appear that they are back up yet, but I suppose it is hard to say for sure given the number of people affected. Twitter is all aflutter with new reports of people still having problems. Yours is the only report that indicated they were working again. It could just be that it's taking a while for the devices to correct themselves, or maybe they even need to be restarted or something like that. On Tue, Oct 7, 2014 at 11:57 AM, Frank Bulk <frnkblk@iname.com> wrote:
Why do that if Belkin is up and running again? Or are you suggesting that Belkin doesn’t have it all worked out?
Frank
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *John Neiberger via Outages *Sent:* Tuesday, October 07, 2014 12:42 PM *To:* outages@outages.org
*Subject:* Re: [outages] Belkin [heartbeat.belkin.com]
Has anyone else tried the loopback/heartbeat workaround with success? I've seen a few reports so far that this has temporarily solved the problem.
John
On Tue, Oct 7, 2014 at 10:28 AM, Frank Bulk <frnkblk@iname.com> wrote:
I can ping heartbeat.belkin.com and the DSL Reports link and other connected links are stating it was back up roughly 30 to 45 minutes ago, so I think we’re out of the woods.
Frank
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *John Neiberger via Outages *Sent:* Tuesday, October 07, 2014 11:04 AM *To:* Josh Luthman *Cc:* outages@outages.org; John Barbieri *Subject:* Re: [outages] Belkin [heartbeat.belkin.com]
I think it was initially suspected to be a firmware upgrade, but I doubt that's the case now. It's interesting to me that there are two seemingly unrelated fixes:
1. Put the IP for heartbeat.belkin.com on a loopback
2. Manually add working DNS entries to affected PCs
Reports indicate that either fix works, but why? They seem unrelated to one another unless the failing heartbeat messages mess up DNS handling on the router.
John
On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages < outages@outages.org> wrote:
Where? I keep reading "bad firmware" problems but everything suggests there is no automatic upgrade and I think we can safely assume resi users didn't upgrade their routers over night.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages < outages@outages.org> wrote:
Funny,
Just saw something about Belkin routers and a botched firmware upgrade.
I wonder if this is what they meant.
That being said, it is working fine from Zito Media in Clanton, AL.
--John
On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:
On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com from my webservers and from subscriber units on a different netblock.
Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page.
I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why.
-Paul
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
No, we’re still getting customer reports about the light staying orange. I was just hoping they were having other issues. I’ve just added 67.20.176.130 as a loopback in our network, too. Frank From: John Neiberger [mailto:jneiberger@gmail.com] Sent: Tuesday, October 07, 2014 1:01 PM To: Frank Bulk Cc: outages@outages.org Subject: Re: [outages] Belkin [heartbeat.belkin.com] Given the number of current social media postings, it doesn't appear that they are back up yet, but I suppose it is hard to say for sure given the number of people affected. Twitter is all aflutter with new reports of people still having problems. Yours is the only report that indicated they were working again. It could just be that it's taking a while for the devices to correct themselves, or maybe they even need to be restarted or something like that. On Tue, Oct 7, 2014 at 11:57 AM, Frank Bulk <frnkblk@iname.com <mailto:frnkblk@iname.com> > wrote: Why do that if Belkin is up and running again? Or are you suggesting that Belkin doesn’t have it all worked out? Frank From: Outages [mailto:outages-bounces@outages.org <mailto:outages-bounces@outages.org> ] On Behalf Of John Neiberger via Outages Sent: Tuesday, October 07, 2014 12:42 PM To: outages@outages.org <mailto:outages@outages.org> Subject: Re: [outages] Belkin [heartbeat.belkin.com <http://heartbeat.belkin.com> ] Has anyone else tried the loopback/heartbeat workaround with success? I've seen a few reports so far that this has temporarily solved the problem. John On Tue, Oct 7, 2014 at 10:28 AM, Frank Bulk <frnkblk@iname.com <mailto:frnkblk@iname.com> > wrote: I can ping heartbeat.belkin.com <http://heartbeat.belkin.com> and the DSL Reports link and other connected links are stating it was back up roughly 30 to 45 minutes ago, so I think we’re out of the woods. Frank From: Outages [mailto:outages-bounces@outages.org <mailto:outages-bounces@outages.org> ] On Behalf Of John Neiberger via Outages Sent: Tuesday, October 07, 2014 11:04 AM To: Josh Luthman Cc: outages@outages.org <mailto:outages@outages.org> ; John Barbieri Subject: Re: [outages] Belkin [heartbeat.belkin.com <http://heartbeat.belkin.com> ] I think it was initially suspected to be a firmware upgrade, but I doubt that's the case now. It's interesting to me that there are two seemingly unrelated fixes: 1. Put the IP for heartbeat.belkin.com <http://heartbeat.belkin.com> on a loopback 2. Manually add working DNS entries to affected PCs Reports indicate that either fix works, but why? They seem unrelated to one another unless the failing heartbeat messages mess up DNS handling on the router. John On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: Where? I keep reading "bad firmware" problems but everything suggests there is no automatic upgrade and I think we can safely assume resi users didn't upgrade their routers over night. Josh Luthman Office: 937-552-2340 <tel:937-552-2340> Direct: 937-552-2343 <tel:937-552-2343> 1100 Wayne St Suite 1337 Troy, OH 45373 On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: Funny, Just saw something about Belkin routers and a botched firmware upgrade. I wonder if this is what they meant. That being said, it is working fine from Zito Media in Clanton, AL. --John On 10/07/2014 10:17 AM, Paul Miller via Outages wrote: On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com <http://heartbeat.belkin.com> from my webservers and from subscriber units on a different netblock. Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page. I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com <http://heartbeat.belkin.com> to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why. -Paul _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
I just saw a report from another ISP who says they had a customer whose VoIP continued to work and they were able to RDP into their computer from the outside. It appears that only web traffic was broken. I wonder if parental controls or web filtering are a component in this fiasco. John On Tue, Oct 7, 2014 at 12:04 PM, Frank Bulk <frnkblk@iname.com> wrote:
No, we’re still getting customer reports about the light staying orange. I was just hoping they were having other issues.
I’ve just added 67.20.176.130 as a loopback in our network, too.
Frank
*From:* John Neiberger [mailto:jneiberger@gmail.com] *Sent:* Tuesday, October 07, 2014 1:01 PM *To:* Frank Bulk *Cc:* outages@outages.org
*Subject:* Re: [outages] Belkin [heartbeat.belkin.com]
Given the number of current social media postings, it doesn't appear that they are back up yet, but I suppose it is hard to say for sure given the number of people affected. Twitter is all aflutter with new reports of people still having problems. Yours is the only report that indicated they were working again. It could just be that it's taking a while for the devices to correct themselves, or maybe they even need to be restarted or something like that.
On Tue, Oct 7, 2014 at 11:57 AM, Frank Bulk <frnkblk@iname.com> wrote:
Why do that if Belkin is up and running again? Or are you suggesting that Belkin doesn’t have it all worked out?
Frank
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *John Neiberger via Outages *Sent:* Tuesday, October 07, 2014 12:42 PM *To:* outages@outages.org
*Subject:* Re: [outages] Belkin [heartbeat.belkin.com]
Has anyone else tried the loopback/heartbeat workaround with success? I've seen a few reports so far that this has temporarily solved the problem.
John
On Tue, Oct 7, 2014 at 10:28 AM, Frank Bulk <frnkblk@iname.com> wrote:
I can ping heartbeat.belkin.com and the DSL Reports link and other connected links are stating it was back up roughly 30 to 45 minutes ago, so I think we’re out of the woods.
Frank
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *John Neiberger via Outages *Sent:* Tuesday, October 07, 2014 11:04 AM *To:* Josh Luthman *Cc:* outages@outages.org; John Barbieri *Subject:* Re: [outages] Belkin [heartbeat.belkin.com]
I think it was initially suspected to be a firmware upgrade, but I doubt that's the case now. It's interesting to me that there are two seemingly unrelated fixes:
1. Put the IP for heartbeat.belkin.com on a loopback
2. Manually add working DNS entries to affected PCs
Reports indicate that either fix works, but why? They seem unrelated to one another unless the failing heartbeat messages mess up DNS handling on the router.
John
On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages < outages@outages.org> wrote:
Where? I keep reading "bad firmware" problems but everything suggests there is no automatic upgrade and I think we can safely assume resi users didn't upgrade their routers over night.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages < outages@outages.org> wrote:
Funny,
Just saw something about Belkin routers and a botched firmware upgrade.
I wonder if this is what they meant.
That being said, it is working fine from Zito Media in Clanton, AL.
--John
On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:
On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com from my webservers and from subscriber units on a different netblock.
Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page.
I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why.
-Paul
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
In my experience. It breaks DNS Masquerading. So any DNS query sent to the router goes unanswered. Verified this with a customer by having them hard-set their Local DNS (On their computer) to my public resolvers. I suspect anything that didn't require DNS worked fine (RDP inbound..SIP Outbound once it resolved the SBC Address. Assuming it had a DNS Hostname). The current thought (Around the NOC here) is that DNS is breaking due to another user friendly feature. Where the router forwards you to it's config page automatically, if it thinks you have no internet.. Nick Olsen Network Operations (855) FLSPEED x106 ---------------------------------------- From: "John Neiberger via Outages" <outages@outages.org> Sent: Tuesday, October 07, 2014 2:32 PM To: "Frank Bulk" <frnkblk@iname.com> Cc: "outages@outages.org" <outages@outages.org> Subject: Re: [outages] Belkin [heartbeat.belkin.com] I just saw a report from another ISP who says they had a customer whose VoIP continued to work and they were able to RDP into their computer from the outside. It appears that only web traffic was broken. I wonder if parental controls or web filtering are a component in this fiasco. John On Tue, Oct 7, 2014 at 12:04 PM, Frank Bulk <frnkblk@iname.com> wrote: No, we're still getting customer reports about the light staying orange. I was just hoping they were having other issues. I've just added 67.20.176.130 as a loopback in our network, too. Frank From: John Neiberger [mailto:jneiberger@gmail.com] Sent: Tuesday, October 07, 2014 1:01 PM To: Frank Bulk Cc: outages@outages.org Subject: Re: [outages] Belkin [heartbeat.belkin.com] Given the number of current social media postings, it doesn't appear that they are back up yet, but I suppose it is hard to say for sure given the number of people affected. Twitter is all aflutter with new reports of people still having problems. Yours is the only report that indicated they were working again. It could just be that it's taking a while for the devices to correct themselves, or maybe they even need to be restarted or something like that. On Tue, Oct 7, 2014 at 11:57 AM, Frank Bulk <frnkblk@iname.com> wrote: Why do that if Belkin is up and running again? Or are you suggesting that Belkin doesn't have it all worked out? Frank From: Outages [mailto:outages-bounces@outages.org] On Behalf Of John Neiberger via Outages Sent: Tuesday, October 07, 2014 12:42 PM To: outages@outages.org Subject: Re: [outages] Belkin [heartbeat.belkin.com] Has anyone else tried the loopback/heartbeat workaround with success? I've seen a few reports so far that this has temporarily solved the problem. John On Tue, Oct 7, 2014 at 10:28 AM, Frank Bulk <frnkblk@iname.com> wrote: I can ping heartbeat.belkin.com and the DSL Reports link and other connected links are stating it was back up roughly 30 to 45 minutes ago, so I think we're out of the woods. Frank From: Outages [mailto:outages-bounces@outages.org] On Behalf Of John Neiberger via Outages Sent: Tuesday, October 07, 2014 11:04 AM To: Josh Luthman Cc: outages@outages.org; John Barbieri Subject: Re: [outages] Belkin [heartbeat.belkin.com] I think it was initially suspected to be a firmware upgrade, but I doubt that's the case now. It's interesting to me that there are two seemingly unrelated fixes: 1. Put the IP for heartbeat.belkin.com on a loopback 2. Manually add working DNS entries to affected PCs Reports indicate that either fix works, but why? They seem unrelated to one another unless the failing heartbeat messages mess up DNS handling on the router. John On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages <outages@outages.org> wrote: Where? I keep reading "bad firmware" problems but everything suggests there is no automatic upgrade and I think we can safely assume resi users didn't upgrade their routers over night. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages <outages@outages.org> wrote: Funny, Just saw something about Belkin routers and a botched firmware upgrade. I wonder if this is what they meant. That being said, it is working fine from Zito Media in Clanton, AL. --John On 10/07/2014 10:17 AM, Paul Miller via Outages wrote: On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com from my webservers and from subscriber units on a different netblock. Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page. I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why. -Paul _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Tue, Oct 07, 2014 at 02:43:13PM -0400, Nick Olsen via Outages wrote:
The current thought (Around the NOC here) is that DNS is breaking due to another user friendly feature. Where the router forwards you to it's config page automatically, if it thinks you have no internet..
I still don't have a clear answer on this at my little ISP here. I think the above quote is exactly the problem though. I had a few people start working after a router reboot with the ping trick in place and others that couldn't get going after the reboot. I suspect the "bad firmware" that may also be (is also?) out there prevents it from connecting — though DHCP seems to complete ACK before it goes dead and you never hear from the IP again. Interestingly, it seems it's not just one IP anymore. host heartbeat.belkin.com heartbeat.belkin.com has address 54.163.87.225 heartbeat.belkin.com has address 54.163.74.132 heartbeat.belkin.com has address 54.87.180.46 heartbeat.belkin.com has address 174.129.92.187 heartbeat.belkin.com has address 54.242.182.61 heartbeat.belkin.com has address 23.20.47.97 heartbeat.belkin.com has address 54.161.217.33 heartbeat.belkin.com has address 54.83.179.150 heartbeat.belkin.com has address 54.163.115.57 heartbeat.belkin.com has address 54.87.220.73 heartbeat.belkin.com has address 54.227.172.225 heartbeat.belkin.com has address 54.196.247.165 heartbeat.belkin.com has address 54.163.72.1 -Paul
I wish they added 127.0.0.1 to that list. =) Frank -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Paul Miller via Outages Sent: Tuesday, October 07, 2014 3:48 PM To: outages@outages.org Subject: Re: [outages] Belkin [heartbeat.belkin.com] On Tue, Oct 07, 2014 at 02:43:13PM -0400, Nick Olsen via Outages wrote:
The current thought (Around the NOC here) is that DNS is breaking due to another user friendly feature. Where the router forwards you to it's config page automatically, if it thinks you have no internet..
I still don't have a clear answer on this at my little ISP here. I think the above quote is exactly the problem though. I had a few people start working after a router reboot with the ping trick in place and others that couldn't get going after the reboot. I suspect the "bad firmware" that may also be (is also?) out there prevents it from connecting — though DHCP seems to complete ACK before it goes dead and you never hear from the IP again. Interestingly, it seems it's not just one IP anymore. host heartbeat.belkin.com heartbeat.belkin.com has address 54.163.87.225 heartbeat.belkin.com has address 54.163.74.132 heartbeat.belkin.com has address 54.87.180.46 heartbeat.belkin.com has address 174.129.92.187 heartbeat.belkin.com has address 54.242.182.61 heartbeat.belkin.com has address 23.20.47.97 heartbeat.belkin.com has address 54.161.217.33 heartbeat.belkin.com has address 54.83.179.150 heartbeat.belkin.com has address 54.163.115.57 heartbeat.belkin.com has address 54.87.220.73 heartbeat.belkin.com has address 54.227.172.225 heartbeat.belkin.com has address 54.196.247.165 heartbeat.belkin.com has address 54.163.72.1 -Paul _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Your large host list looks like it is just your local resolver. host heartbeat.belkin.com 8.8.8.8 Using domain server: Name: 8.8.8.8 Address: 8.8.8.8#53 Aliases: heartbeat.belkin.com has address 67.20.176.130 On Tue, Oct 7, 2014 at 1:48 PM, Paul Miller via Outages <outages@outages.org
wrote:
On Tue, Oct 07, 2014 at 02:43:13PM -0400, Nick Olsen via Outages wrote:
The current thought (Around the NOC here) is that DNS is breaking due to another user friendly feature. Where the router forwards you to it's config page automatically, if it thinks you have no internet..
I still don't have a clear answer on this at my little ISP here. I think the above quote is exactly the problem though. I had a few people start working after a router reboot with the ping trick in place and others that couldn't get going after the reboot.
I suspect the "bad firmware" that may also be (is also?) out there prevents it from connecting — though DHCP seems to complete ACK before it goes dead and you never hear from the IP again.
Interestingly, it seems it's not just one IP anymore.
host heartbeat.belkin.com heartbeat.belkin.com has address 54.163.87.225 heartbeat.belkin.com has address 54.163.74.132 heartbeat.belkin.com has address 54.87.180.46 heartbeat.belkin.com has address 174.129.92.187 heartbeat.belkin.com has address 54.242.182.61 heartbeat.belkin.com has address 23.20.47.97 heartbeat.belkin.com has address 54.161.217.33 heartbeat.belkin.com has address 54.83.179.150 heartbeat.belkin.com has address 54.163.115.57 heartbeat.belkin.com has address 54.87.220.73 heartbeat.belkin.com has address 54.227.172.225 heartbeat.belkin.com has address 54.196.247.165 heartbeat.belkin.com has address 54.163.72.1
-Paul _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Once upon a time, Grant Ridder via Outages <outages@outages.org> said:
Your large host list looks like it is just your local resolver.
DNS records have been changed; it is now hosted on Amazon. $ dig +trace heartbeat.belkin.com ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 <<>> @c.root-servers.net +trace heartbeat.belkin.com ; (2 servers found) ;; global options: +cmd . 518400 IN NS k.root-servers.net. . 518400 IN NS c.root-servers.net. . 518400 IN NS j.root-servers.net. . 518400 IN NS e.root-servers.net. . 518400 IN NS g.root-servers.net. . 518400 IN NS l.root-servers.net. . 518400 IN NS i.root-servers.net. . 518400 IN NS b.root-servers.net. . 518400 IN NS h.root-servers.net. . 518400 IN NS m.root-servers.net. . 518400 IN NS f.root-servers.net. . 518400 IN NS d.root-servers.net. . 518400 IN NS a.root-servers.net. ;; Received 496 bytes from 2001:500:2::c#53(2001:500:2::c) in 7 ms com. 172800 IN NS d.gtld-servers.net. com. 172800 IN NS a.gtld-servers.net. com. 172800 IN NS k.gtld-servers.net. com. 172800 IN NS i.gtld-servers.net. com. 172800 IN NS c.gtld-servers.net. com. 172800 IN NS b.gtld-servers.net. com. 172800 IN NS m.gtld-servers.net. com. 172800 IN NS f.gtld-servers.net. com. 172800 IN NS g.gtld-servers.net. com. 172800 IN NS l.gtld-servers.net. com. 172800 IN NS j.gtld-servers.net. com. 172800 IN NS h.gtld-servers.net. com. 172800 IN NS e.gtld-servers.net. ;; Received 510 bytes from 2001:500:2f::f#53(2001:500:2f::f) in 5 ms belkin.com. 172800 IN NS ns1.p06.dynect.net. belkin.com. 172800 IN NS ns3.p06.dynect.net. belkin.com. 172800 IN NS ns2.p06.dynect.net. belkin.com. 172800 IN NS ns4.p06.dynect.net. ;; Received 188 bytes from 192.43.172.30#53(192.43.172.30) in 75 ms heartbeat.belkin.com. 3600 IN A 23.20.47.97 heartbeat.belkin.com. 3600 IN A 54.196.247.165 heartbeat.belkin.com. 3600 IN A 54.163.115.57 heartbeat.belkin.com. 3600 IN A 54.163.87.225 heartbeat.belkin.com. 3600 IN A 174.129.92.187 heartbeat.belkin.com. 3600 IN A 54.83.179.150 heartbeat.belkin.com. 3600 IN A 54.242.182.61 heartbeat.belkin.com. 3600 IN A 54.87.180.46 heartbeat.belkin.com. 3600 IN A 54.161.217.33 heartbeat.belkin.com. 3600 IN A 54.163.72.1 heartbeat.belkin.com. 3600 IN A 54.227.172.225 heartbeat.belkin.com. 3600 IN A 54.87.220.73 heartbeat.belkin.com. 3600 IN A 54.163.74.132 belkin.com. 86400 IN NS ns1.p06.dynect.net. belkin.com. 86400 IN NS ns2.p06.dynect.net. belkin.com. 86400 IN NS ns3.p06.dynect.net. belkin.com. 86400 IN NS ns4.p06.dynect.net. ;; Received 332 bytes from 204.13.250.6#53(204.13.250.6) in 8 ms -- Chris Adams <cma@cmadams.net>
http://tcrn.ch/1y30B6H On Tue, Oct 7, 2014 at 2:23 PM, Chris Adams via Outages <outages@outages.org
wrote:
Once upon a time, Grant Ridder via Outages <outages@outages.org> said:
Your large host list looks like it is just your local resolver.
DNS records have been changed; it is now hosted on Amazon.
$ dig +trace heartbeat.belkin.com
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 <<>> @c.root-servers.net +trace heartbeat.belkin.com ; (2 servers found) ;; global options: +cmd . 518400 IN NS k.root-servers.net. . 518400 IN NS c.root-servers.net. . 518400 IN NS j.root-servers.net. . 518400 IN NS e.root-servers.net. . 518400 IN NS g.root-servers.net. . 518400 IN NS l.root-servers.net. . 518400 IN NS i.root-servers.net. . 518400 IN NS b.root-servers.net. . 518400 IN NS h.root-servers.net. . 518400 IN NS m.root-servers.net. . 518400 IN NS f.root-servers.net. . 518400 IN NS d.root-servers.net. . 518400 IN NS a.root-servers.net. ;; Received 496 bytes from 2001:500:2::c#53(2001:500:2::c) in 7 ms
com. 172800 IN NS d.gtld-servers.net. com. 172800 IN NS a.gtld-servers.net. com. 172800 IN NS k.gtld-servers.net. com. 172800 IN NS i.gtld-servers.net. com. 172800 IN NS c.gtld-servers.net. com. 172800 IN NS b.gtld-servers.net. com. 172800 IN NS m.gtld-servers.net. com. 172800 IN NS f.gtld-servers.net. com. 172800 IN NS g.gtld-servers.net. com. 172800 IN NS l.gtld-servers.net. com. 172800 IN NS j.gtld-servers.net. com. 172800 IN NS h.gtld-servers.net. com. 172800 IN NS e.gtld-servers.net. ;; Received 510 bytes from 2001:500:2f::f#53(2001:500:2f::f) in 5 ms
belkin.com. 172800 IN NS ns1.p06.dynect.net. belkin.com. 172800 IN NS ns3.p06.dynect.net. belkin.com. 172800 IN NS ns2.p06.dynect.net. belkin.com. 172800 IN NS ns4.p06.dynect.net. ;; Received 188 bytes from 192.43.172.30#53(192.43.172.30) in 75 ms
heartbeat.belkin.com. 3600 IN A 23.20.47.97 heartbeat.belkin.com. 3600 IN A 54.196.247.165 heartbeat.belkin.com. 3600 IN A 54.163.115.57 heartbeat.belkin.com. 3600 IN A 54.163.87.225 heartbeat.belkin.com. 3600 IN A 174.129.92.187 heartbeat.belkin.com. 3600 IN A 54.83.179.150 heartbeat.belkin.com. 3600 IN A 54.242.182.61 heartbeat.belkin.com. 3600 IN A 54.87.180.46 heartbeat.belkin.com. 3600 IN A 54.161.217.33 heartbeat.belkin.com. 3600 IN A 54.163.72.1 heartbeat.belkin.com. 3600 IN A 54.227.172.225 heartbeat.belkin.com. 3600 IN A 54.87.220.73 heartbeat.belkin.com. 3600 IN A 54.163.74.132 belkin.com. 86400 IN NS ns1.p06.dynect.net. belkin.com. 86400 IN NS ns2.p06.dynect.net. belkin.com. 86400 IN NS ns3.p06.dynect.net. belkin.com. 86400 IN NS ns4.p06.dynect.net. ;; Received 332 bytes from 204.13.250.6#53(204.13.250.6) in 8 ms
-- Chris Adams <cma@cmadams.net> _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
It's not just the local. 8.8.8.8 is caching the old entry still. Check the authoritative resolvers. On Tue, Oct 7, 2014 at 2:14 PM, Grant Ridder via Outages <outages@outages.org> wrote:
Your large host list looks like it is just your local resolver.
host heartbeat.belkin.com 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases:
heartbeat.belkin.com has address 67.20.176.130
On Tue, Oct 7, 2014 at 1:48 PM, Paul Miller via Outages <outages@outages.org> wrote:
On Tue, Oct 07, 2014 at 02:43:13PM -0400, Nick Olsen via Outages wrote:
The current thought (Around the NOC here) is that DNS is breaking due to another user friendly feature. Where the router forwards you to it's config page automatically, if it thinks you have no internet..
I still don't have a clear answer on this at my little ISP here. I think the above quote is exactly the problem though. I had a few people start working after a router reboot with the ping trick in place and others that couldn't get going after the reboot.
I suspect the "bad firmware" that may also be (is also?) out there prevents it from connecting — though DHCP seems to complete ACK before it goes dead and you never hear from the IP again.
Interestingly, it seems it's not just one IP anymore.
host heartbeat.belkin.com heartbeat.belkin.com has address 54.163.87.225 heartbeat.belkin.com has address 54.163.74.132 heartbeat.belkin.com has address 54.87.180.46 heartbeat.belkin.com has address 174.129.92.187 heartbeat.belkin.com has address 54.242.182.61 heartbeat.belkin.com has address 23.20.47.97 heartbeat.belkin.com has address 54.161.217.33 heartbeat.belkin.com has address 54.83.179.150 heartbeat.belkin.com has address 54.163.115.57 heartbeat.belkin.com has address 54.87.220.73 heartbeat.belkin.com has address 54.227.172.225 heartbeat.belkin.com has address 54.196.247.165 heartbeat.belkin.com has address 54.163.72.1
-Paul _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler
----- Original Message -----
From: "Michael Loftis via Outages" <outages@outages.org>
It's not just the local. 8.8.8.8 is caching the old entry still. Check the authoritative resolvers.
Point of order: No "resolvers" are "authoritative". "Resolvers" (or, more fully, "Customer resolver servers") are servers operated (generally) by "eyeball carriers" for their end-users to utilize to do DNS lookups, so that the end users are not required to operate a resolver themselves -- which most enduser operating systems have not, historically done. Resolvers do the recursing for you, and return a non-authoritative response. Putting this at the ISP level leverages caching (though sometimes pathologically). Many edge routers perform this task, though not all. "Authoritative" is a term used to describe a "Zone" server; one which has a current local or slaved copy of the actual source data for a zone. Queries directly to a zone server will have the authoritative bit set. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
Most of our resolvers show that same long list. I assume in a few hours they all will, once the TTL expires. Frank From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Grant Ridder via Outages Sent: Tuesday, October 07, 2014 4:15 PM To: Paul Miller Cc: outages@outages.org Subject: Re: [outages] Belkin [heartbeat.belkin.com] Your large host list looks like it is just your local resolver. host heartbeat.belkin.com <http://heartbeat.belkin.com> 8.8.8.8 Using domain server: Name: 8.8.8.8 Address: 8.8.8.8#53 Aliases: heartbeat.belkin.com <http://heartbeat.belkin.com> has address 67.20.176.130 On Tue, Oct 7, 2014 at 1:48 PM, Paul Miller via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: On Tue, Oct 07, 2014 at 02:43:13PM -0400, Nick Olsen via Outages wrote:
The current thought (Around the NOC here) is that DNS is breaking due to another user friendly feature. Where the router forwards you to it's config page automatically, if it thinks you have no internet..
I still don't have a clear answer on this at my little ISP here. I think the above quote is exactly the problem though. I had a few people start working after a router reboot with the ping trick in place and others that couldn't get going after the reboot. I suspect the "bad firmware" that may also be (is also?) out there prevents it from connecting — though DHCP seems to complete ACK before it goes dead and you never hear from the IP again. Interestingly, it seems it's not just one IP anymore. host heartbeat.belkin.com <http://heartbeat.belkin.com> heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.163.87.225 <tel:54.163.87.225> heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.163.74.132 <tel:54.163.74.132> heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.87.180.46 heartbeat.belkin.com <http://heartbeat.belkin.com> has address 174.129.92.187 heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.242.182.61 heartbeat.belkin.com <http://heartbeat.belkin.com> has address 23.20.47.97 heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.161.217.33 <tel:54.161.217.33> heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.83.179.150 heartbeat.belkin.com has address 54.163.115.57 <tel:54.163.115.57> heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.87.220.73 heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.227.172.225 heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.196.247.165 heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.163.72.1 -Paul _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
It looks to me, when querying Belkin's setup directly, they are returning RR A records. (Please pay attention to the "QUESTION SECTION" below, as it's going to become relevant): $ dig @a.gtld-servers.net ns belkin.com. ;; QUESTION SECTION: ;belkin.com. IN NS ;; AUTHORITY SECTION: belkin.com. 172800 IN NS ns1.p06.dynect.net. belkin.com. 172800 IN NS ns3.p06.dynect.net. belkin.com. 172800 IN NS ns2.p06.dynect.net. belkin.com. 172800 IN NS ns4.p06.dynect.net. $ dig @ns1.p06.dynect.net a heartbeat.belkin.com. ;; QUESTION SECTION: ;heartbeat.belkin.com. IN A ;; ANSWER SECTION: heartbeat.belkin.com. 3600 IN A 54.87.220.73 heartbeat.belkin.com. 3600 IN A 54.242.182.61 heartbeat.belkin.com. 3600 IN A 54.161.217.33 heartbeat.belkin.com. 3600 IN A 54.163.72.1 heartbeat.belkin.com. 3600 IN A 54.163.87.225 heartbeat.belkin.com. 3600 IN A 54.163.115.57 heartbeat.belkin.com. 3600 IN A 54.87.180.46 heartbeat.belkin.com. 3600 IN A 54.163.74.132 heartbeat.belkin.com. 3600 IN A 54.227.172.225 heartbeat.belkin.com. 3600 IN A 174.129.92.187 heartbeat.belkin.com. 3600 IN A 54.196.247.165 heartbeat.belkin.com. 3600 IN A 23.20.47.97 heartbeat.belkin.com. 3600 IN A 54.83.179.150 And here's the SOA -- note the serial (it's in YYYYMMDDxx notation): $ dig @ns1.p06.dynect.net soa belkin.com. ;; QUESTION SECTION: ;belkin.com. IN SOA ;; AUTHORITY SECTION: belkin.com. 3600 IN SOA ns1.p06.dynect.net. DNSDomainAdmin.belkin.com. 2010071117 3600 600 604800 3600 However using 8.8.8.8 doing the exact same queries doesn't return RR A, it only returns a single record: $ dig @8.8.8.8 ns belkin.com ;; QUESTION SECTION: ;belkin.com. IN NS ;; ANSWER SECTION: belkin.com. 21010 IN NS ns4.p06.dynect.net. belkin.com. 21010 IN NS ns1.p06.dynect.net. belkin.com. 21010 IN NS ns2.p06.dynect.net. belkin.com. 21010 IN NS ns3.p06.dynect.net. $ dig @8.8.8.8 a heartbeat.belkin.com ;; QUESTION SECTION: ;heartbeat.belkin.com. IN A ;; ANSWER SECTION: heartbeat.belkin.com. 8336 IN A 67.20.176.130 Change the query type to ANY (rather than A) and suddenly: $ dig @8.8.8.8 any heartbeat.belkin.com ;; QUESTION SECTION: ;heartbeat.belkin.com. IN ANY ;; ANSWER SECTION: heartbeat.belkin.com. 3195 IN A 23.20.47.97 heartbeat.belkin.com. 3195 IN A 54.163.72.1 heartbeat.belkin.com. 3195 IN A 54.227.172.225 heartbeat.belkin.com. 3195 IN A 54.161.217.33 heartbeat.belkin.com. 3195 IN A 54.87.180.46 heartbeat.belkin.com. 3195 IN A 54.163.74.132 heartbeat.belkin.com. 3195 IN A 54.196.247.165 heartbeat.belkin.com. 3195 IN A 54.242.182.61 heartbeat.belkin.com. 3195 IN A 54.83.179.150 heartbeat.belkin.com. 3195 IN A 54.163.87.225 heartbeat.belkin.com. 3195 IN A 54.87.220.73 heartbeat.belkin.com. 3195 IN A 54.163.115.57 heartbeat.belkin.com. 3195 IN A 174.129.92.187 $ dig @8.8.8.8 soa belkin.com. ;; QUESTION SECTION: ;belkin.com. IN SOA ;; ANSWER SECTION: belkin.com. 21121 IN SOA ns1.p06.dynect.net. DNSDomainAdmin.belkin.com. 2010071117 3600 600 604800 3600 Who knows what Google is doing under the hood here. Gut feeling is they have something cached -- this would imply Belkin was using a single A record originally, then all this happened, and it forced them to move to RR A records, but something within Google's own DNS infrastructure is still returning the old cached copy. Note that 67.20.176.130 doesn't even appear in the RR list. I wouldn't be surprised if somehow this was tickled by someone at Belkin botching a DNS update (forgetting to increment serial, for example). -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | On Tue, Oct 07, 2014 at 02:14:41PM -0700, Grant Ridder via Outages wrote:
Your large host list looks like it is just your local resolver.
host heartbeat.belkin.com 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases:
heartbeat.belkin.com has address 67.20.176.130
On Tue, Oct 7, 2014 at 1:48 PM, Paul Miller via Outages <outages@outages.org
wrote:
On Tue, Oct 07, 2014 at 02:43:13PM -0400, Nick Olsen via Outages wrote:
The current thought (Around the NOC here) is that DNS is breaking due to another user friendly feature. Where the router forwards you to it's config page automatically, if it thinks you have no internet..
I still don't have a clear answer on this at my little ISP here. I think the above quote is exactly the problem though. I had a few people start working after a router reboot with the ping trick in place and others that couldn't get going after the reboot.
I suspect the "bad firmware" that may also be (is also?) out there prevents it from connecting — though DHCP seems to complete ACK before it goes dead and you never hear from the IP again.
Interestingly, it seems it's not just one IP anymore.
host heartbeat.belkin.com heartbeat.belkin.com has address 54.163.87.225 heartbeat.belkin.com has address 54.163.74.132 heartbeat.belkin.com has address 54.87.180.46 heartbeat.belkin.com has address 174.129.92.187 heartbeat.belkin.com has address 54.242.182.61 heartbeat.belkin.com has address 23.20.47.97 heartbeat.belkin.com has address 54.161.217.33 heartbeat.belkin.com has address 54.83.179.150 heartbeat.belkin.com has address 54.163.115.57 heartbeat.belkin.com has address 54.87.220.73 heartbeat.belkin.com has address 54.227.172.225 heartbeat.belkin.com has address 54.196.247.165 heartbeat.belkin.com has address 54.163.72.1
-Paul _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
In the process of moving, these commands were all run within 5 seconds. jbarbieri@jbarbieri-asus:~$ host heartbeat.belkin.com 8.8.8.8 Using domain server: Name: 8.8.8.8 Address: 8.8.8.8#53 Aliases: heartbeat.belkin.com has address 54.163.74.132 heartbeat.belkin.com has address 54.163.87.225 heartbeat.belkin.com has address 54.87.180.46 heartbeat.belkin.com has address 54.196.247.165 heartbeat.belkin.com has address 54.83.179.150 heartbeat.belkin.com has address 54.161.217.33 heartbeat.belkin.com has address 54.87.220.73 heartbeat.belkin.com has address 174.129.92.187 heartbeat.belkin.com has address 54.242.182.61 heartbeat.belkin.com has address 23.20.47.97 heartbeat.belkin.com has address 54.227.172.225 heartbeat.belkin.com has address 54.163.115.57 heartbeat.belkin.com has address 54.163.72.1 jbarbieri@jbarbieri-asus:~$ host heartbeat.belkin.com 8.8.8.8 Using domain server: Name: 8.8.8.8 Address: 8.8.8.8#53 Aliases: heartbeat.belkin.com has address 67.20.176.130 jbarbieri@jbarbieri-asus:~$ host heartbeat.belkin.com 8.8.8.8 Using domain server: Name: 8.8.8.8 Address: 8.8.8.8#53 Aliases: heartbeat.belkin.com has address 54.163.72.1 heartbeat.belkin.com has address 54.87.220.73 heartbeat.belkin.com has address 54.163.74.132 heartbeat.belkin.com has address 54.161.217.33 heartbeat.belkin.com has address 54.163.115.57 heartbeat.belkin.com has address 54.163.87.225 heartbeat.belkin.com has address 54.83.179.150 heartbeat.belkin.com has address 54.87.180.46 heartbeat.belkin.com has address 23.20.47.97 heartbeat.belkin.com has address 54.196.247.165 heartbeat.belkin.com has address 54.227.172.225 heartbeat.belkin.com has address 174.129.92.187 heartbeat.belkin.com has address 54.242.182.61 jbarbieri@jbarbieri-asus:~$ On 10/07/2014 04:14 PM, Grant Ridder via Outages wrote:
Your large host list looks like it is just your local resolver.
host heartbeat.belkin.com <http://heartbeat.belkin.com> 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases:
heartbeat.belkin.com <http://heartbeat.belkin.com> has address 67.20.176.130
On Tue, Oct 7, 2014 at 1:48 PM, Paul Miller via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
On Tue, Oct 07, 2014 at 02:43:13PM -0400, Nick Olsen via Outages wrote: > The current thought (Around the NOC here) is that DNS is breaking due to > another user friendly feature. Where the router forwards you to it's config > page automatically, if it thinks you have no internet..
I still don't have a clear answer on this at my little ISP here. I think the above quote is exactly the problem though. I had a few people start working after a router reboot with the ping trick in place and others that couldn't get going after the reboot.
I suspect the "bad firmware" that may also be (is also?) out there prevents it from connecting — though DHCP seems to complete ACK before it goes dead and you never hear from the IP again.
Interestingly, it seems it's not just one IP anymore.
host heartbeat.belkin.com <http://heartbeat.belkin.com> heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.163.87.225 <tel:54.163.87.225> heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.163.74.132 <tel:54.163.74.132> heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.87.180.46 heartbeat.belkin.com <http://heartbeat.belkin.com> has address 174.129.92.187 heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.242.182.61 heartbeat.belkin.com <http://heartbeat.belkin.com> has address 23.20.47.97 heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.161.217.33 <tel:54.161.217.33> heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.83.179.150 heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.163.115.57 <tel:54.163.115.57> heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.87.220.73 heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.227.172.225 heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.196.247.165 heartbeat.belkin.com <http://heartbeat.belkin.com> has address 54.163.72.1
-Paul _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
For future reference: https://developers.google.com/speed/public-dns/cache will let you request a cache flush. Damian On Tue, Oct 7, 2014 at 2:46 PM, John Barbieri via Outages < outages@outages.org> wrote:
In the process of moving, these commands were all run within 5 seconds.
jbarbieri@jbarbieri-asus:~$ host heartbeat.belkin.com 8.8.8.8 Using domain server: Name: 8.8.8.8 Address: 8.8.8.8#53 Aliases:
heartbeat.belkin.com has address 54.163.74.132 heartbeat.belkin.com has address 54.163.87.225 heartbeat.belkin.com has address 54.87.180.46 heartbeat.belkin.com has address 54.196.247.165 heartbeat.belkin.com has address 54.83.179.150 heartbeat.belkin.com has address 54.161.217.33 heartbeat.belkin.com has address 54.87.220.73 heartbeat.belkin.com has address 174.129.92.187 heartbeat.belkin.com has address 54.242.182.61 heartbeat.belkin.com has address 23.20.47.97 heartbeat.belkin.com has address 54.227.172.225 heartbeat.belkin.com has address 54.163.115.57 heartbeat.belkin.com has address 54.163.72.1
jbarbieri@jbarbieri-asus:~$ host heartbeat.belkin.com 8.8.8.8 Using domain server: Name: 8.8.8.8 Address: 8.8.8.8#53 Aliases:
heartbeat.belkin.com has address 67.20.176.130
jbarbieri@jbarbieri-asus:~$ host heartbeat.belkin.com 8.8.8.8 Using domain server: Name: 8.8.8.8 Address: 8.8.8.8#53 Aliases:
heartbeat.belkin.com has address 54.163.72.1 heartbeat.belkin.com has address 54.87.220.73 heartbeat.belkin.com has address 54.163.74.132 heartbeat.belkin.com has address 54.161.217.33 heartbeat.belkin.com has address 54.163.115.57 heartbeat.belkin.com has address 54.163.87.225 heartbeat.belkin.com has address 54.83.179.150 heartbeat.belkin.com has address 54.87.180.46 heartbeat.belkin.com has address 23.20.47.97 heartbeat.belkin.com has address 54.196.247.165 heartbeat.belkin.com has address 54.227.172.225 heartbeat.belkin.com has address 174.129.92.187 heartbeat.belkin.com has address 54.242.182.61 jbarbieri@jbarbieri-asus:~$
On 10/07/2014 04:14 PM, Grant Ridder via Outages wrote:
Your large host list looks like it is just your local resolver.
host heartbeat.belkin.com 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases:
heartbeat.belkin.com has address 67.20.176.130
On Tue, Oct 7, 2014 at 1:48 PM, Paul Miller via Outages < outages@outages.org> wrote:
On Tue, Oct 07, 2014 at 02:43:13PM -0400, Nick Olsen via Outages wrote:
The current thought (Around the NOC here) is that DNS is breaking due to another user friendly feature. Where the router forwards you to it's config page automatically, if it thinks you have no internet..
I still don't have a clear answer on this at my little ISP here. I think the above quote is exactly the problem though. I had a few people start working after a router reboot with the ping trick in place and others that couldn't get going after the reboot.
I suspect the "bad firmware" that may also be (is also?) out there prevents it from connecting — though DHCP seems to complete ACK before it goes dead and you never hear from the IP again.
Interestingly, it seems it's not just one IP anymore.
host heartbeat.belkin.com heartbeat.belkin.com has address 54.163.87.225 heartbeat.belkin.com has address 54.163.74.132 heartbeat.belkin.com has address 54.87.180.46 heartbeat.belkin.com has address 174.129.92.187 heartbeat.belkin.com has address 54.242.182.61 heartbeat.belkin.com has address 23.20.47.97 heartbeat.belkin.com has address 54.161.217.33 heartbeat.belkin.com has address 54.83.179.150 heartbeat.belkin.com has address 54.163.115.57 heartbeat.belkin.com has address 54.87.220.73 heartbeat.belkin.com has address 54.227.172.225 heartbeat.belkin.com has address 54.196.247.165 heartbeat.belkin.com has address 54.163.72.1
-Paul _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Frank, I'd be very interested to hear if that helps. We're considering doing the same thing since a couple of people have reported good results. John On Tue, Oct 7, 2014 at 12:04 PM, Frank Bulk <frnkblk@iname.com> wrote:
No, we’re still getting customer reports about the light staying orange. I was just hoping they were having other issues.
I’ve just added 67.20.176.130 as a loopback in our network, too.
Frank
*From:* John Neiberger [mailto:jneiberger@gmail.com] *Sent:* Tuesday, October 07, 2014 1:01 PM *To:* Frank Bulk *Cc:* outages@outages.org
*Subject:* Re: [outages] Belkin [heartbeat.belkin.com]
Given the number of current social media postings, it doesn't appear that they are back up yet, but I suppose it is hard to say for sure given the number of people affected. Twitter is all aflutter with new reports of people still having problems. Yours is the only report that indicated they were working again. It could just be that it's taking a while for the devices to correct themselves, or maybe they even need to be restarted or something like that.
On Tue, Oct 7, 2014 at 11:57 AM, Frank Bulk <frnkblk@iname.com> wrote:
Why do that if Belkin is up and running again? Or are you suggesting that Belkin doesn’t have it all worked out?
Frank
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *John Neiberger via Outages *Sent:* Tuesday, October 07, 2014 12:42 PM *To:* outages@outages.org
*Subject:* Re: [outages] Belkin [heartbeat.belkin.com]
Has anyone else tried the loopback/heartbeat workaround with success? I've seen a few reports so far that this has temporarily solved the problem.
John
On Tue, Oct 7, 2014 at 10:28 AM, Frank Bulk <frnkblk@iname.com> wrote:
I can ping heartbeat.belkin.com and the DSL Reports link and other connected links are stating it was back up roughly 30 to 45 minutes ago, so I think we’re out of the woods.
Frank
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *John Neiberger via Outages *Sent:* Tuesday, October 07, 2014 11:04 AM *To:* Josh Luthman *Cc:* outages@outages.org; John Barbieri *Subject:* Re: [outages] Belkin [heartbeat.belkin.com]
I think it was initially suspected to be a firmware upgrade, but I doubt that's the case now. It's interesting to me that there are two seemingly unrelated fixes:
1. Put the IP for heartbeat.belkin.com on a loopback
2. Manually add working DNS entries to affected PCs
Reports indicate that either fix works, but why? They seem unrelated to one another unless the failing heartbeat messages mess up DNS handling on the router.
John
On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages < outages@outages.org> wrote:
Where? I keep reading "bad firmware" problems but everything suggests there is no automatic upgrade and I think we can safely assume resi users didn't upgrade their routers over night.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages < outages@outages.org> wrote:
Funny,
Just saw something about Belkin routers and a botched firmware upgrade.
I wonder if this is what they meant.
That being said, it is working fine from Zito Media in Clanton, AL.
--John
On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:
On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com from my webservers and from subscriber units on a different netblock.
Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page.
I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why.
-Paul
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Oct 8, 2014, at 12:57 AM, Frank Bulk via Outages <outages@outages.org> wrote:
Why do that if Belkin is up and running again?
So that it doesn't recur if things get dorked up, again? It's a great DDoS vector - take down that IP, wedge every affected Belkin CPE on the Internet. This kind of thing is extremely brittle, and apt to go into Sorcerer's Apprentice Mode without warning. ---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Equo ne credite, Teucri. -- Laocoön
Doesn't appear to be all running, at least their official posting doesn't say so - https://belkininternationalinc.statuspage.io/ Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Tue, Oct 7, 2014 at 1:57 PM, Frank Bulk via Outages <outages@outages.org> wrote:
Why do that if Belkin is up and running again? Or are you suggesting that Belkin doesn’t have it all worked out?
Frank
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *John Neiberger via Outages *Sent:* Tuesday, October 07, 2014 12:42 PM *To:* outages@outages.org
*Subject:* Re: [outages] Belkin [heartbeat.belkin.com]
Has anyone else tried the loopback/heartbeat workaround with success? I've seen a few reports so far that this has temporarily solved the problem.
John
On Tue, Oct 7, 2014 at 10:28 AM, Frank Bulk <frnkblk@iname.com> wrote:
I can ping heartbeat.belkin.com and the DSL Reports link and other connected links are stating it was back up roughly 30 to 45 minutes ago, so I think we’re out of the woods.
Frank
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *John Neiberger via Outages *Sent:* Tuesday, October 07, 2014 11:04 AM *To:* Josh Luthman *Cc:* outages@outages.org; John Barbieri *Subject:* Re: [outages] Belkin [heartbeat.belkin.com]
I think it was initially suspected to be a firmware upgrade, but I doubt that's the case now. It's interesting to me that there are two seemingly unrelated fixes:
1. Put the IP for heartbeat.belkin.com on a loopback
2. Manually add working DNS entries to affected PCs
Reports indicate that either fix works, but why? They seem unrelated to one another unless the failing heartbeat messages mess up DNS handling on the router.
John
On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages < outages@outages.org> wrote:
Where? I keep reading "bad firmware" problems but everything suggests there is no automatic upgrade and I think we can safely assume resi users didn't upgrade their routers over night.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages < outages@outages.org> wrote:
Funny,
Just saw something about Belkin routers and a botched firmware upgrade.
I wonder if this is what they meant.
That being said, it is working fine from Zito Media in Clanton, AL.
--John
On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:
On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com from my webservers and from subscriber units on a different netblock.
Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page.
I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why.
-Paul
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Just saw another report that HTTPS and IRC were still working. It looks like this was only port 80 that was busted, so I think we can safely add web filtering or parental controls somewhere into the chain of busted things today. On Tue, Oct 7, 2014 at 12:14 PM, Josh Luthman <josh@imaginenetworksllc.com> wrote:
Doesn't appear to be all running, at least their official posting doesn't say so -
https://belkininternationalinc.statuspage.io/
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Tue, Oct 7, 2014 at 1:57 PM, Frank Bulk via Outages < outages@outages.org> wrote:
Why do that if Belkin is up and running again? Or are you suggesting that Belkin doesn’t have it all worked out?
Frank
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *John Neiberger via Outages *Sent:* Tuesday, October 07, 2014 12:42 PM *To:* outages@outages.org
*Subject:* Re: [outages] Belkin [heartbeat.belkin.com]
Has anyone else tried the loopback/heartbeat workaround with success? I've seen a few reports so far that this has temporarily solved the problem.
John
On Tue, Oct 7, 2014 at 10:28 AM, Frank Bulk <frnkblk@iname.com> wrote:
I can ping heartbeat.belkin.com and the DSL Reports link and other connected links are stating it was back up roughly 30 to 45 minutes ago, so I think we’re out of the woods.
Frank
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *John Neiberger via Outages *Sent:* Tuesday, October 07, 2014 11:04 AM *To:* Josh Luthman *Cc:* outages@outages.org; John Barbieri *Subject:* Re: [outages] Belkin [heartbeat.belkin.com]
I think it was initially suspected to be a firmware upgrade, but I doubt that's the case now. It's interesting to me that there are two seemingly unrelated fixes:
1. Put the IP for heartbeat.belkin.com on a loopback
2. Manually add working DNS entries to affected PCs
Reports indicate that either fix works, but why? They seem unrelated to one another unless the failing heartbeat messages mess up DNS handling on the router.
John
On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages < outages@outages.org> wrote:
Where? I keep reading "bad firmware" problems but everything suggests there is no automatic upgrade and I think we can safely assume resi users didn't upgrade their routers over night.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages < outages@outages.org> wrote:
Funny,
Just saw something about Belkin routers and a botched firmware upgrade.
I wonder if this is what they meant.
That being said, it is working fine from Zito Media in Clanton, AL.
--John
On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:
On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com from my webservers and from subscriber units on a different netblock.
Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page.
I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why.
-Paul
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On 07 Oct 2014, at 19:57, Frank Bulk via Outages <outages@outages.org> wrote:
Why do that if Belkin is up and running again?
This thing seems to go down and up like a yoyo. (I’m seeing it one hop behind 66.179.229.150 — works this very moment.) Can’t get a reliable OS fingerprint (unless it’s an Airport Extreme…). For extra fun, try nmapping it around the outages. Sometimes it is like: Interesting ports on 67.20.176.130: Not shown: 1689 closed ports PORT STATE SERVICE 22/tcp open ssh 111/tcp open rpcbind 137/tcp filtered netbios-ns 138/tcp filtered netbios-dgm 139/tcp filtered netbios-ssn 199/tcp open smux 445/tcp filtered microsoft-ds 937/tcp open unknown What? Grüße, Carsten
I've had a few companies tell me it fixed their problems. A friend that runs a call center said volume was way down an hour or two ago. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Tue, Oct 7, 2014 at 1:42 PM, John Neiberger via Outages < outages@outages.org> wrote:
Has anyone else tried the loopback/heartbeat workaround with success? I've seen a few reports so far that this has temporarily solved the problem.
John
On Tue, Oct 7, 2014 at 10:28 AM, Frank Bulk <frnkblk@iname.com> wrote:
I can ping heartbeat.belkin.com and the DSL Reports link and other connected links are stating it was back up roughly 30 to 45 minutes ago, so I think we’re out of the woods.
Frank
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *John Neiberger via Outages *Sent:* Tuesday, October 07, 2014 11:04 AM *To:* Josh Luthman *Cc:* outages@outages.org; John Barbieri *Subject:* Re: [outages] Belkin [heartbeat.belkin.com]
I think it was initially suspected to be a firmware upgrade, but I doubt that's the case now. It's interesting to me that there are two seemingly unrelated fixes:
1. Put the IP for heartbeat.belkin.com on a loopback
2. Manually add working DNS entries to affected PCs
Reports indicate that either fix works, but why? They seem unrelated to one another unless the failing heartbeat messages mess up DNS handling on the router.
John
On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages < outages@outages.org> wrote:
Where? I keep reading "bad firmware" problems but everything suggests there is no automatic upgrade and I think we can safely assume resi users didn't upgrade their routers over night.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages < outages@outages.org> wrote:
Funny,
Just saw something about Belkin routers and a botched firmware upgrade.
I wonder if this is what they meant.
That being said, it is working fine from Zito Media in Clanton, AL.
--John
On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:
On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com from my webservers and from subscriber units on a different netblock.
Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page.
I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why.
-Paul
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Just because you can ping it does not mean it will work. I've confirmed with other users this is the case. I believe there is some sort of HTTP request that confirms the connection is working (like Windows, Android, etc). Another resolution is to just loopback 67.20.176.130 (seems to be this IP for everyone so far). Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Tue, Oct 7, 2014 at 11:17 AM, Paul Miller via Outages < outages@outages.org> wrote:
On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com from my webservers and from subscriber units on a different netblock.
Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page.
I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why.
-Paul
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Please provide more information on this – our helpdesk is getting calls about this. Are you saying just route 67.20.176.130 to a pingable IP? Frank From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Josh Luthman via Outages Sent: Tuesday, October 07, 2014 10:42 AM To: Paul Miller Cc: outages@outages.org Subject: Re: [outages] Belkin [heartbeat.belkin.com] Just because you can ping it does not mean it will work. I've confirmed with other users this is the case. I believe there is some sort of HTTP request that confirms the connection is working (like Windows, Android, etc). Another resolution is to just loopback 67.20.176.130 (seems to be this IP for everyone so far). Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Tue, Oct 7, 2014 at 11:17 AM, Paul Miller via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: On one of my netblocks my subscribers can nolonger ping Belkin. I can ping heartbeat.belkin.com <http://heartbeat.belkin.com> from my webservers and from subscriber units on a different netblock. Apparently Belkin routers ping the home base every couple seconds to determine the correct color of the Internet light on the router (orange or blue *sigh*). Worse, when it can't ping the Belkin site the router seems to pack up and redirect my subscribers to their router configuration page. I found this pretty easy to work around. I simply DNATed the pings intended for heartbeat.belkin.com <http://heartbeat.belkin.com> to one of my local nameservers and that seems to get them back online. It's a silly workaround, but if anyone else is seeing all the Belkin routers on their network pack up, this could be why. -Paul _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
participants (15)
-
Carsten Bormann -
Chris Adams -
Damian Menscher -
Frank Bulk -
Grant Ridder -
Jay Ashworth -
Jeremy Chadwick -
John Barbieri -
John Neiberger -
Josh Luthman -
Luke Rockwell -
Michael Loftis -
Nick Olsen -
Paul Miller -
Roland Dobbins