It was posted on NANOG that regions.com and walmart.com are down. Tumblr, another UltraDNS user seems to be down as well. -- staticsafe O< ascii ribbon campaign - stop html mail - www.asciiribbon.org Please don't top post - http://goo.gl/YrmAb
On 12/26/12 22:32, staticsafe wrote:
It was posted on NANOG that regions.com and walmart.com are down. Tumblr, another UltraDNS user seems to be down as well.
Apologies, mistakenly thought regions.com was hosted on UltraDNS as well. walmart.com and tumblr.com are though. -- staticsafe O< ascii ribbon campaign - stop html mail - www.asciiribbon.org Please don't top post - http://goo.gl/YrmAb
Nothing on the neustar twitter as far as an outage, just a bunch of sales stuff. https://twitter.com/ultradns. Anyone have insight as to a reason for outage or ETA? -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of staticsafe Sent: Wednesday, December 26, 2012 8:41 PM To: outages@outages.org Subject: Re: [outages] UltraDNS outage? On 12/26/12 22:32, staticsafe wrote:
It was posted on NANOG that regions.com and walmart.com are down. Tumblr, another UltraDNS user seems to be down as well.
Apologies, mistakenly thought regions.com was hosted on UltraDNS as well. walmart.com and tumblr.com are though. -- staticsafe O< ascii ribbon campaign - stop html mail - www.asciiribbon.org Please don't top post - http://goo.gl/YrmAb _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Thu, Dec 27, 2012 at 04:09:41AM +0000, Blake Pfankuch <blake@pfankuch.me> wrote a message of 26 lines which said:
Anyone have insight as to a reason for outage or ETA?
Tested from several places. Works over IPv4, complete failure over IPv6. % check-soa -i walmart.com pdns193.ultradns.biz. 156.154.66.193: OK: 2012122001 (126 ms) 2610:a1:1015::e5: ERROR: Timeout pdns193.ultradns.co.uk. 156.154.69.193: OK: 2012122001 (122 ms) 2610:a1:1017::e5: ERROR: Timeout pdns193.ultradns.com. 156.154.64.193: OK: 2012122001 (137 ms) 2001:502:f3ff::e5: ERROR: Timeout pdns193.ultradns.info. 156.154.68.193: OK: 2012122001 (114 ms) 2610:a1:1016::e5: ERROR: Timeout pdns193.ultradns.net. 156.154.65.193: OK: 2012122001 (140 ms) 2610:a1:1014::e5: ERROR: Timeout pdns193.ultradns.org. 156.154.67.193: OK: 2012122001 (109 ms) 2001:502:4612::e5: ERROR: Timeout tumblr.com is similar. But it has another issues: the two sets of name servers (UltraDNS and Dynect) serve two different zones, with different serial numbers (573 vs. 2012122891).
From @michaelmcg19 on Twitter: "UltraDNS experienced resolution issues on the legacy UltraDNS network related to database propagation issues."
Frank -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Stephane Bortzmeyer Sent: Thursday, December 27, 2012 2:17 AM To: Blake Pfankuch Cc: outages@outages.org Subject: Re: [outages] UltraDNS outage? On Thu, Dec 27, 2012 at 04:09:41AM +0000, Blake Pfankuch <blake@pfankuch.me> wrote a message of 26 lines which said:
Anyone have insight as to a reason for outage or ETA?
Tested from several places. Works over IPv4, complete failure over IPv6. % check-soa -i walmart.com pdns193.ultradns.biz. 156.154.66.193: OK: 2012122001 (126 ms) 2610:a1:1015::e5: ERROR: Timeout pdns193.ultradns.co.uk. 156.154.69.193: OK: 2012122001 (122 ms) 2610:a1:1017::e5: ERROR: Timeout pdns193.ultradns.com. 156.154.64.193: OK: 2012122001 (137 ms) 2001:502:f3ff::e5: ERROR: Timeout pdns193.ultradns.info. 156.154.68.193: OK: 2012122001 (114 ms) 2610:a1:1016::e5: ERROR: Timeout pdns193.ultradns.net. 156.154.65.193: OK: 2012122001 (140 ms) 2610:a1:1014::e5: ERROR: Timeout pdns193.ultradns.org. 156.154.67.193: OK: 2012122001 (109 ms) 2001:502:4612::e5: ERROR: Timeout tumblr.com is similar. But it has another issues: the two sets of name servers (UltraDNS and Dynect) serve two different zones, with different serial numbers (573 vs. 2012122891). _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I can confirm we saw a loss of service for some IPv4 domains hosted by UltraDNS from the UK between 0245UTC and 0345UTC. Tomo. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Stephane Bortzmeyer Sent: 27 December 2012 08:17 To: Blake Pfankuch Cc: outages@outages.org Subject: Re: [outages] UltraDNS outage? On Thu, Dec 27, 2012 at 04:09:41AM +0000, Blake Pfankuch <blake@pfankuch.me> wrote a message of 26 lines which said:
Anyone have insight as to a reason for outage or ETA?
Tested from several places. Works over IPv4, complete failure over IPv6. % check-soa -i walmart.com pdns193.ultradns.biz. 156.154.66.193: OK: 2012122001 (126 ms) 2610:a1:1015::e5: ERROR: Timeout pdns193.ultradns.co.uk. 156.154.69.193: OK: 2012122001 (122 ms) 2610:a1:1017::e5: ERROR: Timeout pdns193.ultradns.com. 156.154.64.193: OK: 2012122001 (137 ms) 2001:502:f3ff::e5: ERROR: Timeout pdns193.ultradns.info. 156.154.68.193: OK: 2012122001 (114 ms) 2610:a1:1016::e5: ERROR: Timeout pdns193.ultradns.net. 156.154.65.193: OK: 2012122001 (140 ms) 2610:a1:1014::e5: ERROR: Timeout pdns193.ultradns.org. 156.154.67.193: OK: 2012122001 (109 ms) 2001:502:4612::e5: ERROR: Timeout tumblr.com is similar. But it has another issues: the two sets of name servers (UltraDNS and Dynect) serve two different zones, with different serial numbers (573 vs. 2012122891). _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On 2012-12-27, at 3:17 AM, Stephane Bortzmeyer wrote:
tumblr.com is similar. But it has another issues: the two sets of name servers (UltraDNS and Dynect) serve two different zones, with different serial numbers (573 vs. 2012122891).
The Dynect nameservers are a new addition -- as of 11pm EST last night Tumblr only had UltraDNS: Last night: Record last updated on..: 2012-08-17. Domain servers in listed order: pdns3.ultradns.org pdns4.ultradns.org pdns2.ultradns.net pdns6.ultradns.co.uk pdns1.ultradns.net pdns5.ultradns.info Today: Record last updated on..: 2012-12-26. Domain servers in listed order: ns1.p03.dynect.net pdns2.ultradns.net ns3.p03.dynect.net pdns4.ultradns.org pdns6.ultradns.co.uk pdns3.ultradns.org ns2.p03.dynect.net ns4.p03.dynect.net Still has the mismatched serial issue, but I suspect they're just maintaining the zones separately, it's not like they would've been able to set up Dynect as secondaries while UltraDNS was having issues anyhow. -Rich -- Rich Lafferty rich@lafferty.ca
Maintaining zones separately is a reliability defense-in-depth technique anyways. George William Herbert Sent from my iPhone On Dec 27, 2012, at 7:17 AM, Rich Lafferty <rich@lafferty.ca> wrote:
On 2012-12-27, at 3:17 AM, Stephane Bortzmeyer wrote:
tumblr.com is similar. But it has another issues: the two sets of name servers (UltraDNS and Dynect) serve two different zones, with different serial numbers (573 vs. 2012122891).
The Dynect nameservers are a new addition -- as of 11pm EST last night Tumblr only had UltraDNS:
Last night:
Record last updated on..: 2012-08-17.
Domain servers in listed order:
pdns3.ultradns.org pdns4.ultradns.org pdns2.ultradns.net pdns6.ultradns.co.uk pdns1.ultradns.net pdns5.ultradns.info
Today:
Record last updated on..: 2012-12-26.
Domain servers in listed order:
ns1.p03.dynect.net pdns2.ultradns.net ns3.p03.dynect.net pdns4.ultradns.org pdns6.ultradns.co.uk pdns3.ultradns.org ns2.p03.dynect.net ns4.p03.dynect.net
Still has the mismatched serial issue, but I suspect they're just maintaining the zones separately, it's not like they would've been able to set up Dynect as secondaries while UltraDNS was having issues anyhow.
-Rich
-- Rich Lafferty rich@lafferty.ca
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
----- Original Message -----
From: "George Herbert" <george.herbert@gmail.com>
Maintaining zones separately is a reliability defense-in-depth technique anyways.
Well sure, but even if you had two separate sets of DNS zone resolver servers, which is what I think you actually mean, you would still *push them* from zones with the same SN, no? Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274
No, I meant two totally separate chains. Yes, you can screw things up by not keeping them in sync, but too many extended outages have happened because you screw up the one true hidden master, and then it breaks and you can't clean up the public facing slaves fast enough (TTL, timeouts, etc). Defense in depth starts with "assume the worst single thing just happened for system X; now what?". Having two separate independent X is a good start. Certs are similar. The truly paranoid should have 2-3 separate CA's certs available for the service. The CA going south, insane, or having to revoke are all real risks. George William Herbert Sent from my iPhone On Dec 29, 2012, at 8:35 AM, Jay Ashworth <jra@baylink.com> wrote:
----- Original Message -----
From: "George Herbert" <george.herbert@gmail.com>
Maintaining zones separately is a reliability defense-in-depth technique anyways.
Well sure, but even if you had two separate sets of DNS zone resolver servers, which is what I think you actually mean, you would still *push them* from zones with the same SN, no?
Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We use UltraDNS and our sites are working fine. On Dec 26, 2012, at 9:39 PM, "staticsafe" <me@staticsafe.ca> wrote:
It was posted on NANOG that regions.com and walmart.com are down. Tumblr, another UltraDNS user seems to be down as well. -- staticsafe O< ascii ribbon campaign - stop html mail - www.asciiribbon.org Please don't top post - http://goo.gl/YrmAb _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Looks like an operational issue: Frank -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Geoffrey Mina Sent: Wednesday, December 26, 2012 10:06 PM To: staticsafe Cc: outages@outages.org Subject: Re: [outages] UltraDNS outage? We use UltraDNS and our sites are working fine. On Dec 26, 2012, at 9:39 PM, "staticsafe" < <mailto:me@staticsafe.ca> me@staticsafe.ca> wrote:
It was posted on NANOG that regions.com and walmart.com are down.
Tumblr, another UltraDNS user seems to be down as well.
--
staticsafe
O< ascii ribbon campaign - stop html mail - <http://www.asciiribbon.org> www.asciiribbon.org
Please don't top post - <http://goo.gl/YrmAb> http://goo.gl/YrmAb
_______________________________________________
Outages mailing list
<mailto:Outages@outages.org> Outages@outages.org
<https://puck.nether.net/mailman/listinfo/outages> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list <mailto:Outages@outages.org> Outages@outages.org <https://puck.nether.net/mailman/listinfo/outages> https://puck.nether.net/mailman/listinfo/outages
participants (9)
-
Blake Pfankuch -
Frank Bulk (iname.com) -
Geoffrey Mina -
George Herbert -
Jay Ashworth -
Rich Lafferty -
staticsafe -
Stephane Bortzmeyer -
Tomo