August 2017 - Outages - Outages Lists

Google Cloud network outage
by Richard Hesse 30 Aug '17

30 Aug '17

I didn't see a post about it here, but Google Cloud is having a widespread outage with their load balancing stack: https://status.cloud.google.com//incident/cloud-networking/17002?_ga=2.5290…

1 0

Re: [outages] Level 3 outages in Houston area.
by Paul M - outages 28 Aug '17

28 Aug '17

Yowzers, things look tough for you all.. http://www.bbc.co.uk/news/world-us-canada-41070237 Wishing you "good luck" seems a bit lame, but it's all I can do. On 28 Aug 2017 02:03, "Hershberger, Jeff via Outages" <outages(a)outages.org> wrote: We are in the Houston Center downtown and have lost all service with Level 3, voice and data as of 1840 CDT.

2 1

Level 3 outages in Houston area.
by Hershberger, Jeff 27 Aug '17

27 Aug '17

We are in the Houston Center downtown and have lost all service with Level 3, voice and data as of 1840 CDT. Per their NOC on our ticket…. 08/27/2017 00:44:03 GMT - The IP NOC continues to implement configuration adjustments whenever possible to resolve outages without assistance from Field Services personnel. Updates will continue to be relayed as new information becomes available. 08/26/2017 20:37:28 GMT - The IP NOC reports additional nodes have become impacted by commercial power outages. Field Services will dispatch to resolve the troubles once it is deemed safe to do so. Level 3 will continue monitoring Hurricane Harvey as it progresses. Please be advised updates will be provided on a 24-hour basis, or as new information becomes available. 08/26/2017 02:22:59 GMT - The IP NOC reports that the affected nodes have been migrated to alternate paths and client service is not impacted by the commercial power outages at this time. The IP NOC will continue to monitor the situation and update if service becomes impacted once again. 08/25/2017 21:15:22 GMT - Hurricane Harvey is impacting IP services in Houston, TX. The IP NOC has advised services are impacted due to ongoing commercial power issues related to the hurricane. Field Services will not be dispatching until it is deemed safe to do so. Level 3 will continue monitoring Hurricane Harvey as it progresses along the Texas Coast. Please be advised updates will be provided on a 24-hour basis, or as new information becomes available. Thanks, Jeff Hershberger | Telecom Manager 2323 Ross Avenue, Suite 600 | Dallas, TX | 75201 V: (214) 953-5748 | F: (214) 661-6665 | jhershberger(a)jw.com<mailto:jhershberger@jw.com> [cid:CF52621A-DBF7-42CC-8289-0DAD97F4F825@jwllp.com]

1 0

Zayo Houston
by Mike Hammett 26 Aug '17

26 Aug '17

I've heard that Zayo lost a card in one of their core routers in Houston, affecting anyone in Houston. Obviously Hurricane Harvey is dampening their efforts to repair that. -----Mike HammettIntelligent Computing SolutionsMidwest Internet ExchangeThe Brothers WISP

2 1

Looks like facebook is having issues ...
by frnkblk＠iname.com 26 Aug '17

26 Aug '17

http://downdetector.com/status/facebook And their other properties: fb messenger and instagram

3 2

Dreamhost DNS Outage: DDoS Attack
by Anthony Hook 24 Aug '17

24 Aug '17

Dreamhost is having a service outage, which they've now identified as a DDoS attack: https://twitter.com/dhstatus/status/900761746767364098 Also: https://www.dreamhoststatus.com/ -AH

1 0

Google's DNS 8.8.8.8 & 8.8.4.4
by Mr. NPP 24 Aug '17

24 Aug '17

In los angeles, experiencing packet loss, slow resolution, and high ping times. it appears google's any cast dns is not working properly. --- 8.8.8.8 ping statistics --- 21 packets transmitted, 11 received, +1 errors, 47% packet loss, time 20229ms rtt min/avg/max/mdev = 44.635/151.432/258.331/76.230 ms from multiple providers. i would check over any2, but it appears my link to any2 is down (hopefully a separate issue) any confirm that they are seeing this issue? Thanks

10 14

Google DNS --- Figuring out which DNS Cluster you are using
by Erik Sundberg 23 Aug '17

23 Aug '17

A couple months ago had to troubleshoot a google DNS issue with Google’s NOC. Below is some helpful information on how to determine which DNS Cluster you are going to. Let’s remember that Google runs DNS Anycast for DNS queries to 8.8.8.8 and 8.8.4.4. Anycast routes your DNS queries to the closes DNS cluster based on the best route / lowest metric to 8.8.8.8/8.8.4.4. Google has deployed multiple DNS clusters across the world and each DNS Cluster has multiple servers. So a DNS query in Chicago will go to a different DNS clusters than queries from a device in Atlanta or New York. How to get a list of google DNS Cluster’s. dig -t TXT +short locations.publicdns.goog. @8.8.8.8 How to print this list in a table format. Script from: https://developers.google.com/speed/public-dns/faq --------------- #!/bin/bash IFS="\"$IFS" for LOC in $(dig -t TXT +short locations.publicdns.goog. @8.8.8.8) do case $LOC in '') : ;; *.*|*:*) printf '%s ' ${LOC} ;; *) printf '%s\n' ${LOC} ;; esac done --------------- Which will give you a list like below. This is all of the IP network’s that google uses for their DNS Clusters and their associated locations. 74.125.18.0/26 iad 74.125.18.64/26 iad 74.125.18.128/26 syd 74.125.18.192/26 lhr 74.125.19.0/24 mrn 74.125.41.0/24 tpe 74.125.42.0/24 atl 74.125.44.0/24 mrn 74.125.45.0/24 tul 74.125.46.0/24 lpp 74.125.47.0/24 bru 74.125.72.0/24 cbf 74.125.73.0/24 bru 74.125.74.0/24 lpp 74.125.75.0/24 chs 74.125.76.0/24 cbf 74.125.77.0/24 chs 74.125.79.0/24 lpp 74.125.80.0/24 dls 74.125.81.0/24 dub 74.125.92.0/24 mrn 74.125.93.0/24 cbf 74.125.112.0/24 lpp 74.125.113.0/24 cbf 74.125.115.0/24 tul 74.125.176.0/24 mrn 74.125.177.0/24 atl 74.125.179.0/24 cbf 74.125.181.0/24 bru 74.125.182.0/24 cbf 74.125.183.0/24 cbf 74.125.184.0/24 chs 74.125.186.0/24 dls 74.125.187.0/24 dls 74.125.190.0/24 sin 74.125.191.0/24 tul 172.217.32.0/26 lhr 172.217.32.64/26 lhr 172.217.32.128/26 sin 172.217.33.0/26 syd 172.217.33.64/26 syd 172.217.33.128/26 fra 172.217.33.192/26 fra 172.217.34.0/26 fra 172.217.34.64/26 bom 172.217.34.192/26 bom 172.217.35.0/24 gru 172.217.36.0/24 atl 172.217.37.0/24 gru 173.194.90.0/24 cbf 173.194.91.0/24 scl 173.194.93.0/24 tpe 173.194.94.0/24 cbf 173.194.95.0/24 tul 173.194.97.0/24 chs 173.194.98.0/24 lpp 173.194.99.0/24 tul 173.194.100.0/24 mrn 173.194.101.0/24 tul 173.194.102.0/24 atl 173.194.103.0/24 cbf 173.194.168.0/26 nrt 173.194.168.64/26 nrt 173.194.168.128/26 nrt 173.194.168.192/26 iad 173.194.169.0/24 grq 173.194.170.0/24 grq 173.194.171.0/24 tpe 2404:6800:4000::/48 bom 2404:6800:4003::/48 sin 2404:6800:4006::/48 syd 2404:6800:4008::/48 tpe 2404:6800:400b::/48 nrt 2607:f8b0:4001::/48 cbf 2607:f8b0:4002::/48 atl 2607:f8b0:4003::/48 tul 2607:f8b0:4004::/48 iad 2607:f8b0:400c::/48 chs 2607:f8b0:400d::/48 mrn 2607:f8b0:400e::/48 dls 2800:3f0:4001::/48 gru 2800:3f0:4003::/48 scl 2a00:1450:4001::/48 fra 2a00:1450:4009::/48 lhr 2a00:1450:400b::/48 dub 2a00:1450:400c::/48 bru 2a00:1450:4010::/48 lpp 2a00:1450:4013::/48 grq There are IPv4 Networks: 68 IPv6 Networks: 20 DNS Cluster’s Identified by POP Code’s: 20 DNS Clusters identified by POP Code to City, State, or Country. Not all of these are Google’s Core Datacenters, some of them are Edge Points of Presences (POPs). https://peering.google.com/#/infrastructure and https://www.google.com/about/datacenters/inside/locations/ Most of these are airport codes, it did my best to get the location correct. iad Washington, DC syd Sydney, Australia lhr London, UK mrn Lenoir, NC tpe Taiwan atl Altanta, GA tul Tulsa, OK lpp Findland bru Brussels, Belgium cbf Council Bluffs, IA chs Charleston, SC dls The Dalles, Oregon dub Dublin, Ireland sin Singapore fra Frankfort, Germany bom Mumbai, India gru Sao Paulo, Brazil scl Santiago, Chile nrt Tokyo, Japan grq Groningen, Netherlans Which Google DNS Server Cluster am I using. I am testing this from Chicago, IL # dig o-o.myaddr.l.google.com -t txt +short @8.8.8.8 "173.194.94.135" <<<<<<DNS Server IP, reference the list above to get the cluster, Council Bluffs, IA "edns0-client-subnet 207.xxx.xxx.0/24" <<<< Your Source IP Block Side note, the google dns servers will not respond to DNS queries to the Cluster’s Member’s IP, they will only respond to dns queries to 8.8.8.8 and 8.8.4.4. So the following will not work. dig google.com @173.194.94.135 Now to see the DNS Cluster load balancing in action. I am doing a dig query from our Telx\Digital Realty POP in Atlanta, GA. We do peer with google at this location. I dig a dig query about 10 times and received the following unique dns cluster member ip’s as responses. dig o-o.myaddr.l.google.com -t txt +short @8.8.8.8 "74.125.42.138" "173.194.102.132" "74.125.177.5" "74.125.177.74" "74.125.177.71" "74.125.177.4" Which all are Google DNS Networks in Atlanta. 74.125.42.0/24 atl 74.125.177.0/24 atl 172.217.36.0/24 atl 173.194.102.0/24 atl 2607:f8b0:4002::/48 atl Just thought it would be helpful when troubleshooting google DNS issues. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.

1 0

AT&T possible routing loop
by Sahil Ganguly 19 Aug '17

19 Aug '17

Is anyone from AT&T on this mailing list that could help us out in pushing for a fix? We're seeing some issues that look like a routing loop getting from AT&T networks: $ traceroute account.box.com traceroute: Warning: account.box.com has multiple addresses; using 107.152.25.197 traceroute to account.box.com (107.152.25.197), 64 hops max, 52 byte packets 1 172.20.10.1 (172.20.10.1) 23.643 ms 16.233 ms 4.400 ms 2 172.26.96.169 (172.26.96.169) 88.196 ms 49.009 ms 21.336 ms 3 172.16.126.36 (172.16.126.36) 31.343 ms 172.16.126.60 (172.16.126.60) 530.635 ms 201.699 ms 4 12.249.2.49 (12.249.2.49) 39.212 ms 104.090 ms 32.640 ms 5 12.83.180.82 (12.83.180.82) 49.394 ms 50.699 ms 40.146 ms 6 cr1.sl9mo.ip.att.net (12.122.18.9) 41.180 ms 39.546 ms 39.920 ms 7 ggr6.la2ca.ip.att.net (12.122.90.61) 36.784 ms 37.823 ms 41.714 ms 8 * * * 9 12.81.131.9 (12.81.131.9) 57.906 ms 66.188 ms 44.169 ms 10 * * * 11 12.81.131.9 (12.81.131.9) 686.796 ms 163.466 ms 43.789 ms 12 * * * 13 * 12.81.131.9 (12.81.131.9) 160.871 ms 54.204 ms 14 * * * 15 12.81.131.9 (12.81.131.9) 85.982 ms * * 16 * Thanks! -- Sahil Ganguly

1 0

REMINDER: ECLIPSE
by Jay R. Ashworth 18 Aug '17

18 Aug '17

Monday, 21 August, for the first time since 1979, a full solar eclipse will pass across the continental United States. This has caused enough excitement among the general public that an estimated 20 million Americans will be along the path of totality, rather than in their normal market areas, and can be expected to be using *LOTS* of mobile voice and data bandwidth in those areas -- especially since most cellular carriers are on record that they'll be deploying COW and COT units along the path. <moderator> Please remember this when evaluating potential outages from now through Tuesday, and use resources like Downdetector and others on our wiki, https://wiki.outages.org to determine if an outage is 'real' before posting about it on the list. As always: ask yourself "do I need to make several thousand cellphones beep" before posting. Two points: 1) If you have any good tools for this sort of work that aren't on our wiki's Tools page already, either add them, or email them to me and I will. 2) If you have *actual outage notices* from a carrier of which you are a direct or indirect customer, those are always on topic. Everybody have a great weekend and Monday -- and let's be careful out there. </moderator> Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

1 0