October 2016 - Outages - Outages Lists

Fiber cut between US and Asia via TPE
by Mike Castoro 02 Nov '16

02 Nov '16

I have a P2P physical line between Shanghai and NYC via the Tran- Pacific Express submarine cable that is down due to a fiber cut caused by a marine vessel incident. China Telecom Americas gave me an ETR of Nov 8 due to storms in the area. So if anyone is having issues communicating with China, this may be a reason. -Mike

2 2

Brooklyn NY Verizon Fios
by Izzy Goldstein - TeleGo 28 Oct '16

28 Oct '16

Master ticket NYH5012237 Estimated time of restoration is 9:45 AM ? Representatives states outage affects the state of New York ? -- Izzy Goldstein Main: (212) 477.1000 Fax: (212) 477.8900 igoldstein(a)telego.com Website <http://www.telego.com/> | LinkedIn <https://www.linkedin.com/company/telego> | Blog <http://www.telego.com/blog> | Confidentially Notice: This e-mail may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error please notify us immediately by email reply and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of TeleGo (T). Employees of TeleGo are expressly required not to make defamatory statements and not to infringe or authorize any infringement of copyright or any other legal right by email communications. Any such communication is contrary to TeleGo policy and outside the scope of the employment of the individual concerned. TeleGo will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising. Think before you print.

1 1

Turkey shuts off internet service in 11 Kurdish cities
by Bill Woodcock 27 Oct '16

27 Oct '16

https://www.engadget.com/2016/10/27/turkey-internet-shutdown-kurdish-cities/ -Bill

1 0

Bandwidth - Packet Loss
by Phil Lavin 24 Oct '16

24 Oct '16

Hi all, Between 15:07 and 15:58 UTC we observed high packet loss on all inbound calls from the Bandwidth SBC at 67.231.13.113. We did not see loss from other carriers/customers, nor from other Bandwidth SBCs. Did anybody else see packet loss? Bandwidth say they are not aware of any problems, during that time period. Thanks Phil Lavin

1 0

Bandwidth.com DID's
by Michael R. Self 23 Oct '16

23 Oct '16

We are seeing a large scale outage with bandwidth.com local did's around the US. This has been ongoing for over 1/2 hour now. Opened ticket with BW but nothing definitive from them as of yet. ————————— Best regards, Michael Self This email composed on my iPhone 8 while driving with my knees. Please excuse brevity and typos due to Siri's hearing problems, accidents, cops or other mishaps.

2 1

ADMIN: code of conduct
by Outages 23 Oct '16

23 Oct '16

Hello World, The OUTAGES list has over several thousand subscribers so it very easy for a thread to have scores of posts while being off-topic. Kindly take 5 minutes to read the FAQ and AUP policy before posting for more details at, http://wiki.outages.org/index.php/Code_of_Conduct Your adherence to the guidelines is appreciated. You may direct feedback to me personally and I thank you for your support. regards, /vrode

1 0

Dyn outage continuing
by Terry Hardie 21 Oct '16

21 Oct '16

Even though their page says the outage is resolved, I'm still getting DNS failures (return SERVFAIL, not no answer) to their anycast networks from California: $ dig @ns2.p34.dynect.net video.twimg.com ; <<>> DiG 9.3.2 <<>> @ns2.p34.dynect.net video.twimg.com ; (1 server found) ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 170 ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;video.twimg.com. IN A ;; Query time: 3103 msec ;; SERVER: 204.13.250.34#53(204.13.250.34) ;; WHEN: Fri Oct 21 17:11:36 2016 ;; MSG SIZE rcvd: 33

32 41

Re: [outages] Outages Digest, Vol 101, Issue 51
by nick roderique 21 Oct '16

21 Oct '16

I know this isn't exactly work related, but it's my day off and I'm trying to watch some Netflix in the Midwest area and I can't get anything. I'm thinking this is related, ping times out and tracert jumps to random public IP's before timing out. C:\Users\nick>ping www.netflix.com Pinging www.us-east-1.prodaa.netflix.com [52.72.205.255] with 32 bytes of data: Request timed out. Request timed out. Request timed out. Request timed out. Ping statistics for 52.72.205.255: Packets: Sent = 4, Received = 0, Lost = 4 (100% loss), C:\Users\nick>tracert www.netflix.com Tracing route to www.us-east-1.prodaa.netflix.com [54.236.170.9] over a maximum of 30 hops: 1 1 ms 1 ms 1 ms 192.168.1.1 2 * * * Request timed out. 3 10 ms 13 ms 11 ms dtr01sstlmo-tge-0-0-1-2.sstl.mo.charter.com [96.34.49.141] 4 16 ms 15 ms 10 ms crr01blvlil-bue-20.blvl.il.charter.com [96.34.76.198] 5 18 ms 18 ms 14 ms bbr01blvlil-bue-80.blvl.il.charter.com [96.34.2.168] 6 17 ms 21 ms 14 ms bbr01olvemo-bue-3.olve.mo.charter.com [96.34.0.14] 7 22 ms 23 ms 29 ms bbr02chcgil-bue-2.chcg.il.charter.com [96.34.0.12] 8 19 ms 17 ms 21 ms prr01chcgil-bue-4.chcg.il.charter.com [96.34.3.11] 9 20 ms 20 ms 17 ms 96-34-152-11.static.unas.mo.charter.com [96.34.152.11] 10 26 ms 29 ms 30 ms 52.95.62.22 11 20 ms 27 ms 18 ms 52.95.62.31 12 44 ms 37 ms 40 ms 52.95.62.148 13 39 ms 43 ms 37 ms 54.239.42.59 14 39 ms 43 ms 50 ms 54.239.42.198 15 68 ms 62 ms 61 ms 54.239.110.153 16 45 ms 47 ms 37 ms 54.239.108.199 17 62 ms 40 ms 42 ms 52.93.24.10 18 41 ms 39 ms 48 ms 52.93.24.5 19 * * * Request timed out. 20 * * * Request timed out. 21 * * * Request timed out. 22 * * * Request timed out. 23 * * * Request timed out. 24 * * * Request timed out. 25 * * * Request timed out. 26 * * * Request timed out. 27 * * * Request timed out. 28 * * * Request timed out. 29 * * * Request timed out. 30 * * * Request timed out. Trace complete. ________________________________ From: Outages <outages-bounces(a)outages.org> on behalf of outages-request(a)outages.org <outages-request(a)outages.org> Sent: Friday, October 21, 2016 12:53 PM To: outages(a)outages.org Subject: Outages Digest, Vol 101, Issue 51 Send Outages mailing list submissions to outages(a)outages.org To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages Outages Info Page - Welcome to puck.nether.net<https://puck.nether.net/mailman/listinfo/outages> puck.nether.net Welcome to the outages list! The primary goal of this mailing list ("outages") is for outages-reporting that would apply to failures of major communications ... or, via email, send a message with subject or body 'help' to outages-request(a)outages.org You can reach the person managing the list at outages-owner(a)outages.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..." Today's Topics: 1. Re: Dyn outage continuing (David Miller) 2. Re: Dyn outage continuing (Sajal Kayan) 3. Re: Dyn outage continuing (Kun, Mike) ---------------------------------------------------------------------- Message: 1 Date: Fri, 21 Oct 2016 13:39:27 -0400 From: David Miller <dmiller(a)tiggee.com> To: "Patrick W. Gilmore" <patrick(a)ianai.net> Cc: "outages(a)outages.org" <outages(a)outages.org> Subject: Re: [outages] Dyn outage continuing Message-ID: <CAE5R536-sMam2OtgQomsxPK=U858d7RgZNWpvJWeWoqL5DK5gw(a)mail.gmail.com> Content-Type: text/plain; charset=UTF-8 What are you basing your claims on? On Fri, Oct 21, 2016 at 1:09 PM, Patrick W. Gilmore via Outages <outages(a)outages.org> wrote: > It is a shame in that PagerDuty is affected. > > However, Dyn is far, far better positioned to withstand attacks than a > company like PagerDuty could possibly be on their own. So I think PagerDuty > did the right thing in using Dyn. > > Remember, Twitter, Pingdom, github, and lots of other people use Dyn. And > this is the first major outage I have heard of their service. Everyone has issues. Here is one: http://hub.dyn.com/dyn-status/update-managed-dns-issue-july-6-2015 > Let?s all be clear that Dyn has to be in the top 10 DNS infrastructures on > the planet. Possibly the largest that sells DNS as a service. This is a > serious attack, and the entire community should help track this miscreant > down, then crush them like a bug. Let's actually be clear... Do you have some inside information the rest of us are not privy to? Sources? How did you come to your earlier statement re: this being a ransom attack? Do you have detailed information describing the top 10 DNS infrastructures on the planet, such that you could make any sort of comparison of infrastructure or size? I'll grant you Akamai, how about the other 9? What do you know about this attack, such that you can make claims that this is "serious" or that there is a single miscreant that can be tracked down or that the community can assist in any way? Without actual details this looks to be wild conjecture and doesn't provide any helpful direction. -DMM [ I work for DNS Made Easy ] ------------------------------ Message: 2 Date: Fri, 21 Oct 2016 17:40:23 +0000 From: Sajal Kayan <sajal83(a)gmail.com> To: Andrew Bunde <abunde(a)tower-research.com>, "outages(a)outages.org" <outages(a)outages.org> Subject: Re: [outages] Dyn outage continuing Message-ID: <CAPdYNsB4n=81ROM41CXLQzQ13oD8G83N65L17r=7=5ucsQLFkg(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" It appears most of the world cant reach Dyn : https://pulse.turbobytes.com/results/580a5178ecbe402e2201a74c/ TurboBytes Pulse: global DNS, HTTP and Traceroute testing<https://pulse.turbobytes.com/results/580a5178ecbe402e2201a74c/> pulse.turbobytes.com {{metadesc}} But for the most part, its resolving thru Google DNS/OpenDNS : https://pulse.turbobytes.com/results/580a51ceecbe402e2201a74e/ TurboBytes Pulse: global DNS, HTTP and Traceroute testing<https://pulse.turbobytes.com/results/580a51ceecbe402e2201a74e/> pulse.turbobytes.com {{metadesc}} On Sat, Oct 22, 2016 at 12:31 AM Andrew Bunde via Outages < outages(a)outages.org> wrote: > Don?t forget https://bit.namecoin.info/ Dot-Bit: Secure Decentralized DNS<https://bit.namecoin.info/> bit.namecoin.info Dot-Bit. Dot-Bit-enabled websites end with ".bit" instead of ".com" or something similar. Dot-Bit uses Bitcoin technology to decentralize and free website addresses ... > > > > > > *From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Jason > Antman via Outages > *Sent:* Friday, October 21, 2016 1:25 PM > *Cc:* outages(a)outages.org > > > *Subject:* Re: [outages] Dyn outage continuing > > > > This is just a horribly painful example of why we/they should all follow > the route of NetFlix ( > http://techblog.netflix.com/2013/05/denominating-multi-region-sites.html > ) and use multiple DNS providers. Denominator ( > https://github.com/Netflix/denominator ) the tool NetFlix uses for this > and open-sourced, currently supports managing (mirrored) DNS records across > AWS Route53, RackSpace CloudDNS, DynECT and UltraDNS. > > > > -Jason > > > > On Fri, Oct 21, 2016 at 1:09 PM, Patrick W. Gilmore via Outages < > outages(a)outages.org> wrote: > > It is a shame in that PagerDuty is affected. > > > > However, Dyn is far, far better positioned to withstand attacks than a > company like PagerDuty could possibly be on their own. So I think PagerDuty > did the right thing in using Dyn. > > > > Remember, Twitter, Pingdom, github, and lots of other people use Dyn. And > this is the first major outage I have heard of their service. > > > > Let?s all be clear that Dyn has to be in the top 10 DNS infrastructures on > the planet. Possibly the largest that sells DNS as a service. This is a > serious attack, and the entire community should help track this miscreant > down, then crush them like a bug. > > > > -- > > TTFN, > > patrick > > > > On Oct 21, 2016, at 1:00 PM, Terry Hardie via Outages <outages(a)outages.org> > wrote: > > > > It's a shame services like PagerDuty use Dyn. Now they're down too... > > On Fri, Oct 21, 2016 at 9:57 AM, Justin Krejci via Outages > <outages(a)outages.org> wrote: > > Previously it was not affecting upper midwest for me, traces went to > Chicago. > > Now traces to twitter DNS servers in Chicago are failing. > > > ________________________________ > From: Neil Hanlon via Outages [outages(a)outages.org] > Sent: Friday, October 21, 2016 11:16 AM > To: Terry Hardie; outages(a)outages.org > Subject: Re: [outages] Dyn outage continuing > > The attacks appear to be continuing again. We are experiencing issues in > Europe/Asia. > > On Fri, Oct 21, 2016 at 12:13 PM Terry Hardie via Outages > <outages(a)outages.org> wrote: > > > Even though their page says the outage is resolved, I'm still getting > DNS failures (return SERVFAIL, not no answer) to their anycast > networks from California: > > $ dig @ns2.p34.dynect.net video.twimg.com > > ; <<>> DiG 9.3.2 <<>> @ns2.p34.dynect.net video.twimg.com > ; (1 server found) > ;; global options: printcmd > ;; Got answer: > ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 170 > ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 > > ;; QUESTION SECTION: > ;video.twimg.com. IN A > > ;; Query time: 3103 msec > ;; SERVER: 204.13.250.34#53(204.13.250.34) > ;; WHEN: Fri Oct 21 17:11:36 2016 > ;; MSG SIZE rcvd: 33 > _______________________________________________ > Outages mailing list > Outages(a)outages.org > https://puck.nether.net/mailman/listinfo/outages > > > -- > > KAYAK > > Neil Hanlon > > Devops Engineer > > > +1 978 902 8171 > > _______________________________________________ > Outages mailing list > Outages(a)outages.org > https://puck.nether.net/mailman/listinfo/outages > > _______________________________________________ > Outages mailing list > Outages(a)outages.org > https://puck.nether.net/mailman/listinfo/outages > > > > > _______________________________________________ > Outages mailing list > Outages(a)outages.org > https://puck.nether.net/mailman/listinfo/outages > > > _______________________________________________ > Outages mailing list > Outages(a)outages.org > https://puck.nether.net/mailman/listinfo/outages >

1 0

Re: [outages] Major DNS DoS
by Ryan Collins 21 Oct '16

21 Oct '16

Dyn reporting another attack started at 15:52 UTC.

1 0

Major DNS DoS
by Adam Greene 21 Oct '16

21 Oct '16

I'm probably not noticing this being reported on this list, because of the very DNS issues taking place, but just in case: http://gizmodo.com/this-is-probably-why-half-the-internet-shut-down-today-17 88062835 Adam

7 7