Neustar UltraDNS sporadic resolution failure
Seeing repeated failures on both udns1.ultradns.net and udns2.ultradns.net. Getting some initial corroboration from several folks on twitter, but no acknowledgement from Neustar. ; <<>> DiG 9.8.3-P1 <<>> @udns2.ultradns.net www.braintreegateway.com ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reached
In addition to this, we saw a whole variety of "weird" problems at this time. Internetpulse reported packet loss between multiple ISPs (ATT/Verizon/Sprint/L3 in different directions). We saw packet loss on Abovenet, too. Reaching AWS was sporadic on multiple ISPs... Anyone else see this happen too? On Tue, Oct 16, 2012 at 8:35 PM, Paul Hinze <paul.t.hinze@gmail.com> wrote:
Seeing repeated failures on both udns1.ultradns.net and udns2.ultradns.net.
Getting some initial corroboration from several folks on twitter, but no acknowledgement from Neustar.
; <<>> DiG 9.8.3-P1 <<>> @udns2.ultradns.net www.braintreegateway.com ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reached _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Works from Comcast in Boston. Where is it down? Remember, Ultra is massively anycasted. -- TTFN, patrick On Oct 16, 2012, at 22:59 , Avleen Vig <avleen@gmail.com> wrote:
In addition to this, we saw a whole variety of "weird" problems at this time. Internetpulse reported packet loss between multiple ISPs (ATT/Verizon/Sprint/L3 in different directions).
We saw packet loss on Abovenet, too.
Reaching AWS was sporadic on multiple ISPs...
Anyone else see this happen too?
On Tue, Oct 16, 2012 at 8:35 PM, Paul Hinze <paul.t.hinze@gmail.com> wrote:
Seeing repeated failures on both udns1.ultradns.net and udns2.ultradns.net.
Getting some initial corroboration from several folks on twitter, but no acknowledgement from Neustar.
; <<>> DiG 9.8.3-P1 <<>> @udns2.ultradns.net www.braintreegateway.com ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reached _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Looks like it was a large scale DDoS attack:
Dear UltraDNS Customer,
Beginning at 00:47 GMT on 10/17/12 (please note corrected start time from original notification), customers on the Neustar UltraDNS network were subjected to a large scale DDoS attack. During the early portions of this attack, several node locations across the UltraDNS network experienced high latency and saturation leading to DNS response timeouts and increased drop rates.
At approximately 01:58 GMT, Neustar had restored normal service levels to a majority of the network locations, but continued to experience higher than normal latency in the Western US region. This increased latency continued in that region until approximately 03:19 GMT when all remaining impact was cleared.
While we have fully restored service to all locations at this time, we will maintain proactive mitigation on the network and continue to actively monitor the situation to guard against a renewal of attack activities.
At Neustar we are committed to providing the highest levels of performance and reliability through the products and solutions we deliver. Please feel free to contact our 24x7 UltraDNS Support Team at ultrasupport@Neustar.biz with any questions or concerns.
Sincerely,
UltraDNS Support
On Tue, Oct 16, 2012 at 10:03 PM, Patrick W. Gilmore <patrick@ianai.net> wrote:
Works from Comcast in Boston.
Where is it down? Remember, Ultra is massively anycasted.
-- TTFN, patrick
On Oct 16, 2012, at 22:59 , Avleen Vig <avleen@gmail.com> wrote:
In addition to this, we saw a whole variety of "weird" problems at this time. Internetpulse reported packet loss between multiple ISPs (ATT/Verizon/Sprint/L3 in different directions).
We saw packet loss on Abovenet, too.
Reaching AWS was sporadic on multiple ISPs...
Anyone else see this happen too?
On Tue, Oct 16, 2012 at 8:35 PM, Paul Hinze <paul.t.hinze@gmail.com> wrote:
Seeing repeated failures on both udns1.ultradns.net and udns2.ultradns.net.
Getting some initial corroboration from several folks on twitter, but no acknowledgement from Neustar.
; <<>> DiG 9.8.3-P1 <<>> @udns2.ultradns.net www.braintreegateway.com ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reached _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 10/17/2012 07:41 AM, Paul Hinze wrote:
Looks like it was a large scale DDoS attack:
Call it coincidence or just simply bad timing, anytime when ultradns has tweeted about dos mitigation they somehow wake up sleeping gremlins. If my memory serves me correct something similar happened in august of this year to their nodes in HK, LA and NY. regards, /virendra
Dear UltraDNS Customer,
Beginning at 00:47 GMT on 10/17/12 (please note corrected start time from original notification), customers on the Neustar UltraDNS network were subjected to a large scale DDoS attack. During the early portions of this attack, several node locations across the UltraDNS network experienced high latency and saturation leading to DNS response timeouts and increased drop rates.
At approximately 01:58 GMT, Neustar had restored normal service levels to a majority of the network locations, but continued to experience higher than normal latency in the Western US region. This increased latency continued in that region until approximately 03:19 GMT when all remaining impact was cleared.
While we have fully restored service to all locations at this time, we will maintain proactive mitigation on the network and continue to actively monitor the situation to guard against a renewal of attack activities.
At Neustar we are committed to providing the highest levels of performance and reliability through the products and solutions we deliver. Please feel free to contact our 24x7 UltraDNS Support Team at ultrasupport@Neustar.biz with any questions or concerns.
Sincerely,
UltraDNS Support
On Tue, Oct 16, 2012 at 10:03 PM, Patrick W. Gilmore <patrick@ianai.net> wrote:
Works from Comcast in Boston.
Where is it down? Remember, Ultra is massively anycasted.
-- TTFN, patrick
On Oct 16, 2012, at 22:59 , Avleen Vig <avleen@gmail.com> wrote:
In addition to this, we saw a whole variety of "weird" problems at this time. Internetpulse reported packet loss between multiple ISPs (ATT/Verizon/Sprint/L3 in different directions).
We saw packet loss on Abovenet, too.
Reaching AWS was sporadic on multiple ISPs...
Anyone else see this happen too?
On Tue, Oct 16, 2012 at 8:35 PM, Paul Hinze <paul.t.hinze@gmail.com> wrote:
Seeing repeated failures on both udns1.ultradns.net and udns2.ultradns.net.
Getting some initial corroboration from several folks on twitter, but no acknowledgement from Neustar.
; <<>> DiG 9.8.3-P1 <<>> @udns2.ultradns.net www.braintreegateway.com ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reached _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iF4EAREIAAYFAlB+0wgACgkQ3HuimOHfh+Hl8AD+LcTL65dDMMUqs+dSgcTuEVFn fFd4f6timgM8IlN3rEwBAIIpGtqETAJyeXsJfPAlwD/mBseBZ9K3zYF9M/tVHJW8 =JFhJ -----END PGP SIGNATURE-----
participants (4)
-
Avleen Vig -
Patrick W. Gilmore -
Paul Hinze -
virendra rode