January 2017 - Outages - Outages Lists

XO fiber cut in Georgia, USA
by Silvey, David 31 Jan '17

31 Jan '17

Received Trouble Report, Isolated to Network Outage Saw circuit go down at 1:59 PM Eastern time. We opened a ticket with XO support and were told this: "Fiber cut between Bogart and Smyrna GA Alert Update: Local market and Transport Operations are still investigating." The information contained in this E-mail message is privileged, confidential, and may be protected from disclosure; please be aware that any other use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. If you think that you have received this E-mail message in error, please reply to the sender. This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. However, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by Kelley Drye & Warren LLP for any loss or damage arising in any way from its use.

1 1

Level3 Looking Glassdown
by Ca By 31 Jan '17

31 Jan '17

Level3 folks, please fix. wget -T 5 lg.level3.net converted 'http://lg.level3.net' (ANSI_X3.4-1968) -> 'http://lg.level3.net' (UTF-8) --2017-01-31 12:58:57-- http://lg.level3.net/ Resolving lg.level3.net (lg.level3.net) 209.245.18.181 Connecting to lg.level3.net (lg.level3.net)|209.245.18.181|:80... failed: Connection timed out. Retrying. traceroute lg.level3.net traceroute to lg.level3.net (209.245.18.181), 30 hops max, 60 byte packets 1 * * * 2 xe-0-1-0-17.r04.atlnga05.us.bb.gin.ntt.net (129.250.203.221) 1.284 ms 1.277 ms ae7-300.cr2-atl1.ip4.gtt.net (173.205.55.97) 0.241 ms 3 ae-4.r03.atlnga05.us.bb.gin.ntt.net (129.250.5.204) 0.427 ms ae4.edge2.Atlanta4.Level3.net (4.68.63.157) 0.222 ms ae-4.r03.atlnga05.us.bb.gin.ntt.net (129.250.5.204) 0.464 ms 4 * * ae-1-51.edge6.Denver1.Level3.net (4.69.147.74) 34.727 ms 5 ae-1-51.edge6.Denver1.Level3.net (4.69.147.74) 34.655 ms 34.639 ms 34.641 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 * * *

4 3

ServiceNow DNSSEC issues?
by Michael Sinatra 27 Jan '17

27 Jan '17

Looks like ServiceNow may have recently botched a KSK roll. 8.8.8.8 is intermittently giving out SERVFAIL responses, and my own resolvers had SERVFAILs (we are a SN shop), starting around 1545 US/Pacific. Examining the cache (and DNSVIZ) shows that at some point within the last 3 hours, the DS record for service-now.com switched from having keytag 30126 to keytag 31893. The DS record has a TTL of 24 hours. It appears that at the same time, or a *short* (under 3 hours) time later, the DNSKEY record for KSK with tag 30126 was removed from service-now.com. The DNSKEY records have a TTL of 2 hours. This would have caused systems to continue to reference the cached DS record to fail validation after the DNSKEY cache TTL expired. (The DNSKEY would have needed to be retained for at least a full TTL, and probably 2xTTL to be on the safe side.) Again, this is causing intermittent breakage of name resolution in service-now.com. Appendix: repeated queries of the DS record for service-now.com to 8.8.8.8: ;; ANSWER SECTION: service-now.com. 86395 IN DS 31893 7 1 18035BB11DFF4E2B75D677724833E8FDB9FAEC13 ;; Query time: 6 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Sat Jan 28 00:33:58 UTC 2017 ;; ANSWER SECTION: service-now.com. 60788 IN DS 30126 7 1 8EB362449D1500DFFD359200F958471D915F83A8 ;; Query time: 24 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Sat Jan 28 00:33:59 UTC 2017 ;; MSG SIZE rcvd: 80 I'll spare you the dig output, but suffice it to say that the auth servers for service-now.com don't have KSK 30126 in their DNSKEY RRSET. michael

1 0

Yahoo ipv6 dns issues
by Ca By 27 Jan '17

27 Jan '17

I keep seeing traceroutes going from the USA to Asia trying to reach ns1 and ns3 yahoo.com Here is the sprint looking glass from dallas Tracing the route to 2406:8600:b8:fe03::1003 1 2600:0:2:1239:144:232:9:248 [sl-crs1-fw-po0-10-5-2.v6.sprintlink.net] 10 msec 8 msec 9 msec 2 2600:0:2:1239:144:232:13:84 8 msec 2600:0:3:1239:144:232:12:192 5 msec 9 msec 3 2600:0:3:1239:144:232:0:a2 4 msec 7 msec 5 msec 4 2001:418:0:2000::259 5 msec 6 msec 5 msec 5 2001:418:0:2000::1f9 66 msec 64 msec 64 msec 6 2001:218:0:2000::52 148 msec 150 msec 153 msec 7 2001:218:0:2000::e 192 msec 196 msec 203 msec 8 2001:218:0:2000::92 194 msec 194 msec 200 msec 9 2001:218:7000:5000::116 437 msec 442 msec 433 msec 10 2403:0:100::1b2 432 msec 422 msec 444 msec 11 2406:8600:b8:fe03::1003 460 msec 430 msec 418 msec Here is conget looking glass from Dallas traceroute to ns1.yahoo.com(2001:4998:130::1001), 30 hops max, 80 byte packets 1 2001:550:1:314::1 (2001:550:1:314::1) 1.013 ms 1.008 ms 2 te0-6-0-16.ccr22.dfw01.atlas.cogentco.com (2001:550:0:1000::9a36:2d0d) 1.022 ms 1.061 ms 3 * be2441.ccr41.iah01.atlas.cogentco.com(2001:550:0:1000::9a36:2941) 5.688 ms 4 be2927.ccr21.elp01.atlas.cogentco.com(2001:550:0:1000::9a36:1dde) 21.653 ms * 5 be2930.ccr22.phx02.atlas.cogentco.com(2001:550:0:1000::9a36:2a4d) 29.552 ms 29.560 ms 6 be2931.ccr21.lax01.atlas.cogentco.com(2001:550:0:1000::9a36:2c56) 41.594 ms 41.742 ms 7 be3033.ccr41.lax05.atlas.cogentco.com(2001:550:0:1000::9a36:1a2a) 41.520 ms be3032.ccr41.lax05.atlas.cogentco.com(2001:550:0:1000::9a36:1f36) 41.527 ms 8 2001:550:2:67::2 (2001:550:2:67::2) 257.596 ms 257.607 ms 9 2001:cb0:c111:1:e::2 (2001:cb0:c111:1:e::2) 256.675 ms 256.663 ms 10 2001:cb0:41f0:1:1::1 (2001:cb0:41f0:1:1::1) 283.284 ms 283.272 ms 11 2001:cb0:41f0:1:2::2 (2001:cb0:41f0:1:2::2) 281.078 ms 281.083 ms 12 2001:cb0:4103:2:8::2 (2001:cb0:4103:2:8::2) 187.118 ms 187.128 ms 13 2406:2000:f014:212::1 (2406:2000:f014:212::1) 178.460 ms 2406:2000:f014:12::1 (2406:2000:f014:12::1) 189.707 ms 14 so-2-0-0.pat1.jpx.yahoo.com(2406:2000:f014:a::) 172.901 ms so-2-0-0.pat1.jpe.yahoo.com (2406:2000:f014:8::) 191.090 ms 15 2406:2000:f02a:5::1 (2406:2000:f02a:5::1) 173.671 ms 173.659 ms 16 ns1.yahoo.com (2001:4998:130::1001) 173.788 ms 173.740 ms

2 1

Level 3 Packet Loss - Boston Area
by Phil Lavin 27 Jan '17

27 Jan '17

Is anybody else experiencing issues in this area? Seeing loss between us and a couple of customers. MTR on both sides shows loss around Boston inside Level3. Phil

7 7

Akamai DNS issues
by frnkblk＠iname.com 24 Jan '17

24 Jan '17

Starting this afternoon our monitoring system recorded several 5 to 15 minute outages accessing some IPv6-enabled Akamai-delivered sites. In each case, where I could catch the problem while it happened, I could not resolve the AAAA to an IPv6 address. We and our upstream provider have over 10 servers of different flavors, and they are all exhibiting this issue, though not Google or OpenDNS' servers. Just two examples are below, but in each case, I get a CNAME back, but it doesn't show an IPv6 address. And also five from https://www.digwebinterface.com/?type=SOA&colorize=on&useresolver=8.8.4.4&ns =all&nameservers. And also here: https://www.whatsmydns.net/#AAAA/www.apple.com and here http://www.preshweb.co.uk/cgi-bin/dns-propagation-tracker.pl?domain=www.appl e.com&type=AAAA&serversmode=random&servers=&nsdomain= Anyone else seeing the same? I emailed Akamai's tix address, but have not gotten anything back other than the auto-responder. Frank ; <<>> DiG 9.7.3 <<>> AAAA www.apple.com @10.20.0.10 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9110 ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 1, ADDITIONAL: 0 ;; QUESTION SECTION: ;www.apple.com. IN AAAA ;; ANSWER SECTION: www.apple.com. 483 IN CNAME www.apple.com.edgekey.net. www.apple.com.edgekey.net. 3193 IN CNAME www.apple.com.edgekey.net.globalredir.akadns.net. www.apple.com.edgekey.net.globalredir.akadns.net. 19 IN CNAME e6858.dsce9.akamaiedge.net. ;; AUTHORITY SECTION: dsce9.akamaiedge.net. 107 IN SOA n0dsce9.akamaiedge.net. hostmaster.akamai.com. 1485306230 1000 1000 1000 1800 ;; Query time: 0 msec ;; SERVER: 10.20.0.10#53(10.20.0.10) ;; WHEN: Tue Jan 24 19:18:43 2017 ;; MSG SIZE rcvd: 228 ; <<>> DiG 9.7.3 <<>> AAAA www.apple.com @96.31.0.32 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58259 ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 1, ADDITIONAL: 0 ;; QUESTION SECTION: ;www.apple.com. IN AAAA ;; ANSWER SECTION: www.apple.com. 1073 IN CNAME www.apple.com.edgekey.net. www.apple.com.edgekey.net. 3584 IN CNAME www.apple.com.edgekey.net.globalredir.akadns.net. www.apple.com.edgekey.net.globalredir.akadns.net. 446 IN CNAME e6858.dsce9.akamaiedge.net. ;; AUTHORITY SECTION: dsce9.akamaiedge.net. 257 IN SOA n0dsce9.akamaiedge.net. hostmaster.akamai.com. 1485306380 1000 1000 1000 1800 ;; Query time: 0 msec ;; SERVER: 96.31.0.32#53(96.31.0.32) ;; WHEN: Tue Jan 24 19:18:43 2017 ;; MSG SIZE rcvd: 228 =============================== https://www.digwebinterface.com/?hostnames=www.apple.com&type=AAAA&showcomma nd=on&colorize=on&useresolver=8.8.4.4&ns=all&nameservers= www.apple.com@168.95.1.1 (HiNet (TW)): dig AAAA +noadditional +noquestion +nocomments +nocmd +nostats www.apple.com. @168.95.1.1 www.apple.com. 1494 IN CNAME www.apple.com.edgekey.net. www.apple.com.edgekey.net. 6888 IN CNAME www.apple.com.edgekey.net.globalredir.akadns.net. www.apple.com.edgekey.net.globalredir.akadns.net. 3289 IN CNAME e6858.dsce9.akamaiedge.net. dsce9.akamaiedge.net. 191 IN SOA n0dsce9.akamaiedge.net. hostmaster.akamai.com. 1485306682 1000 1000 1000 1800 www.apple.com@195.129.12.122 (UUNET (CH)): dig AAAA +noadditional +noquestion +nocomments +nocmd +nostats www.apple.com. @195.129.12.122 www.apple.com. 387 IN CNAME www.apple.com.edgekey.net. www.apple.com.edgekey.net. 12797 IN CNAME www.apple.com.edgekey.net.globalredir.akadns.net. www.apple.com.edgekey.net.globalredir.akadns.net. 2145 IN CNAME e6858.dsce9.akamaiedge.net. dsce9.akamaiedge.net. 159 IN SOA n0dsce9.akamaiedge.net. hostmaster.akamai.com. 1485306650 1000 1000 1000 1800 www.apple.com@192.76.144.66 (UUNET (DE)): dig AAAA +noadditional +noquestion +nocomments +nocmd +nostats www.apple.com. @192.76.144.66 www.apple.com. 1599 IN CNAME www.apple.com.edgekey.net. www.apple.com.edgekey.net. 11114 IN CNAME www.apple.com.edgekey.net.globalredir.akadns.net. www.apple.com.edgekey.net.globalredir.akadns.net. 588 IN CNAME e6858.dsce9.akamaiedge.net. dsce9.akamaiedge.net. 297 IN SOA n0dsce9.akamaiedge.net. hostmaster.akamai.com. 1485306788 1000 1000 1000 1800 www.apple.com@158.43.240.3 (UUNET (UK)): dig AAAA +noadditional +noquestion +nocomments +nocmd +nostats www.apple.com. @158.43.240.3 www.apple.com. 800 IN CNAME www.apple.com.edgekey.net. www.apple.com.edgekey.net. 9769 IN CNAME www.apple.com.edgekey.net.globalredir.akadns.net. www.apple.com.edgekey.net.globalredir.akadns.net. 2596 IN CNAME e6858.dsce9.akamaiedge.net. dsce9.akamaiedge.net. 129 IN SOA n0dsce9.akamaiedge.net. hostmaster.akamai.com. 1485306620 1000 1000 1000 1800 www.apple.com@64.6.64.6 (Verisign (US)): dig AAAA +noadditional +noquestion +nocomments +nocmd +nostats www.apple.com. @64.6.64.6 www.apple.com. 1205 IN CNAME www.apple.com.edgekey.net. www.apple.com.edgekey.net. 21323 IN CNAME www.apple.com.edgekey.net.globalredir.akadns.net. www.apple.com.edgekey.net.globalredir.akadns.net. 995 IN CNAME e6858.dsce9.akamaiedge.net. dsce9.akamaiedge.net. 681 IN SOA n0dsce9.akamaiedge.net. hostmaster.akamai.com. 1485307172 1000 1000 1000 1800

4 5

Fortinet FortiGuard DNS servers
by Wyn Bryant 24 Jan '17

24 Jan '17

Fortinet appears to be having issues with their FortiGuard DNS servers. Our first reported issue was around 6:30 AM ET and both servers are still timing out on most lookups. 208.91.112.53 208.91.112.52 Thanks, Wyn Bryant

1 0

ntp.org DNS lookups failing
by Douglas K. Rand 18 Jan '17

18 Jan '17

All of our DNS queries in the ntp.org zone seem to be failing. Checked a few public DNS servers, such as Google's 8.8.8.8 and 8.8.4.4, and they are missing it also. This is from North Dakota.

4 5

Possible QueryFoundry outage
by Paul M - outages 18 Jan '17

18 Jan '17

On noticing a some VPS hosted at CloudShards, and Cloudshards main site being down, it seems there might be a wider outage with the provider QueryFoundry https://whois.arin.net/rest/net/NET-162-220-24-0-1 AS12179 AS35916 AS174 AS62638 Traceroute isn't giving me any useful information as to upstream of that netblock. Are there are upstream providers who recognise the names and can indicate what the problem might be? thanks Paul

1 2

Anyone experiencing issues reaching networks through FiberNoire.ca AS22652?
by Eric Tykwinski 12 Jan '17

12 Jan '17

We are seeing some major packet loss through both Level3 and Cogent. Ultimate destination is AS36354 (SherWeb, Inc.) and traffic is https to 199.244.76.149 and 199.244.76.151. ICMP and TCP pings work fine, just packet loss. SherWeb is saying this is upstream on status pages and it seems like their only peer is Fibernoire with a backup only to Allstream (AS15290). Sincerely, Eric Tykwinski TrueNet, Inc. P: 610-429-8300

2 1