November 2014 - Outages - Outages Lists

major verizon outage in Los Angeles
by virendra rode 26 Nov '14

26 Nov '14

Verizon fiber was vandalized and burnt as of 5AM PDT (11/26) in Los Angeles causing major outage. Section of an aerial cable was also found cut, no etr yet. regards, /virendra

1 1

Windstream - Voice
by Jayson Navitsky 26 Nov '14

26 Nov '14

We're seeing some issues with Windstream voice in the Northeast, calls to Windstream customers seem to be either returning a fast busy or generic "network difficulties" message. Their customer support 800 #'s also seem to be down (maybe due to call volume?), so I'd guess this is somewhat widespread. -Jayson

6 6

Anyone else seeing major VoIP outage with Windstream?
by Elijah Savage 25 Nov '14

25 Nov '14

Has to be major because you can't get through to their support center either.

1 0

Craigslist DNS Hijacking last night
by Jay Ashworth 24 Nov '14

24 Nov '14

If you operate customer resolver servers, they may be poisoned, still; read the message below. ----- Forwarded Message ----- > From: "Ted Cooper" <ml-nanog090304q(a)elcsplace.com> > To: nanog(a)nanog.org > Sent: Sunday, November 23, 2014 11:08:51 PM > Subject: Re: Craigslist hacked? > On 24/11/14 13:41, Brian Henson wrote: > > Is anyone else seeing their local craigslist redirected to another > > site > > other than craigslist? I see it loading > > http://digitalgangster.com/5um. > > > > Over on [dns-operations]: > > > On 24/11/14 13:38, Brad Volz wrote:> > >> The craigslist account at one of our registrars was compromised and > >> the > >> NS records migrated away from their rightful home. That issue has > >> since > >> been corrected, but the various caches around the Internet are > >> still > >> holding the old data. > >> > >> If you could take a look at your caches to see if craigslist.org > >> has the following NS records: > >> > >> ns1p.craigslist.org > >> ns2p.craigslist.org > >> ns1f.craigslist.org > >> ns2f.craigslist.org > >> > >> If you see something else there, then you have a poisoned cache. > >> > >> Thank you for your assistance in this matter. > >> > >> Brad Volz > >> Network Engineer -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

1 0

Network Solutions DNS not resolving on Comcast or CenturyLink networks
by Dylan Ebner 24 Nov '14

24 Nov '14

We have multiple network solutions domain names not resolving currently when using comcast or CL dns servers. Google public dns servers do resolve. networksolutions.com also does not resolve on comcast or CL dns servers. It does resolve on google dns servers but it is not accessable. COMCAST DNS SERVERS: DOES NOT WORK > server 75.75.76.76 Default Server: cdns02.comcast.net<http://cdns02.comcast.net/> Address: 75.75.76.76 > dms.crlmed.com<http://dms.crlmed.com/> Server: cdns02.comcast.net<http://cdns02.comcast.net/> Address: 75.75.76.76 *** cdns02.comcast.net<http://cdns02.comcast.net/> can't find dms.crlmed.com<http://dms.crlmed.com/>: Server failed CENTURY LINK SERVERS: DOES NOT WORK > server 205.171.3.65 Default Server: resolver1.qwest.net<http://resolver1.qwest.net/> Address: 205.171.3.65 > dms.crlmed.com<http://dms.crlmed.com/> Server: resolver1.qwest.net<http://resolver1.qwest.net/> Address: 205.171.3.65 DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. *** Request to resolver1.qwest.net<http://resolver1.qwest.net/> timed-out GOOGLE PUBLIC DNS: WORKS > server 8.8.8.8 Default Server: google-public-dns-a.google.com<http://google-public-dns-a.google.com/> Address: 8.8.8.8 > dms.crlmed.com<http://dms.crlmed.com/> Server: google-public-dns-a.google.com<http://google-public-dns-a.google.com/> Address: 8.8.8.8 Non-authoritative answer: Name: dms.crlmed.com<http://dms.crlmed.com/> Address: 67.135.55.235 Networksolutions.com also does not work: COMCAST DNS: NOT WORKING > server 75.75.76.76 Default Server: cdns02.comcast.net<http://cdns02.comcast.net/> Address: 75.75.76.76 > networksolutions.com<http://networksolutions.com/> Server: cdns02.comcast.net<http://cdns02.comcast.net/> Address: 75.75.76.76 DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. *** Request to cdns02.comcast.net<http://cdns02.comcast.net/> timed-out CENTURY LINK NOT WORKING > server 205.171.3.65 Default Server: resolver1.qwest.net<http://resolver1.qwest.net/> Address: 205.171.3.65 > networksolutions.com<http://networksolutions.com/> Server: resolver1.qwest.net<http://resolver1.qwest.net/> Address: 205.171.3.65 DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. *** Request to resolver1.qwest.net<http://resolver1.qwest.net/> timed-out > GOOGLE Public DNS WORKS > server 8.8.8.8 Default Server: google-public-dns-a.google.com<http://google-public-dns-a.google.com/> Address: 8.8.8.8 > networksolutions.com<http://networksolutions.com/> Server: google-public-dns-a.google.com<http://google-public-dns-a.google.com/> Address: 8.8.8.8 Non-authoritative answer: Name: networksolutions.com<http://networksolutions.com/> Address: 205.178.187.13 Tracert from comcast network to networksolutions.com: C:\Users\Larry>tracert 205.178.187.13 Tracing route to 205.178.187.13 over a maximum of 30 hops 1 1 ms <1 ms 1 ms 10.0.1.1 2 * 9 ms 10 ms 96.120.48.221 3 12 ms 9 ms 9 ms te-0-2-0-6-sur01.nempls.mn.minn.comcast.net [68. 85.164.177] 4 11 ms 15 ms 13 ms te-0-4-0-13-ar01.roseville.mn.minn.comcast.net [ 69.139.219.193] 5 24 ms 19 ms 19 ms he-5-3-0-0-cr01.350ecermak.il.ibone.comcast.net [68.86.166.253] 6 * * * Request timed out. 7 * * * Request timed out. 8 * * * Request timed out. 9 * * * Request timed out. 10 * * * Request timed out. 11 * *

3 2

Re: [outages] Level 3 outages - Chicago
by *Network 20 Nov '14

20 Nov '14

Latest update from Level 3 *** CASCADED EXTERNAL NOTES 20-Nov-2014 13:58:18 GMT From CASE: 8611542 - Event The IP NOC reports that during non-service affecting work to groom the aggregation router to a different backbone router, intermediate system to intermediate system (ISIS) connectivity was lost. The IP NOC worked to adjust configurations on the aggregation router and ensured traffic was flowing to the desired backbone router. Connectivity is now restored and traffic is routing appropriately. If additional issues are experienced, please contact the Level 3 Technical Service Center at 1-877-4LEVEL3. From: *Network Sent: Thursday, November 20, 2014 7:57 AM To: outages(a)outages.org Subject: Level 3 outages - Chicago Here is respond from Level 3 "The IP NOC continues to investigate the routing issues for multiple customer after configuration changes to routing protocols on a device restored services for some. Troubleshooting is ongoing with no ETTR available." What the next step is: I am associating this ticket to the parent event for the outage. Timeline for next step: You will receive further updates from our event team as they are available. Please contact me if you have any questions.

1 0

Level 3 outages - Chicago
by *Network 20 Nov '14

20 Nov '14

Here is respond from Level 3 "The IP NOC continues to investigate the routing issues for multiple customer after configuration changes to routing protocols on a device restored services for some. Troubleshooting is ongoing with no ETTR available." What the next step is: I am associating this ticket to the parent event for the outage. Timeline for next step: You will receive further updates from our event team as they are available. Please contact me if you have any questions.

1 0

unsubscribe
by Ronnie Montgomery 19 Nov '14

19 Nov '14

unsubscribe

4 4

Re: [outages] level3 issue in chicago (was: FYI, Level 3 issues in Dallas)
by Cool Hand Luke 19 Nov '14

19 Nov '14

(replying to this thread from nanog on outages...) like david below, we've been getting reports this morning from customers unable to reach various web sites. investigating a number of these reports, the one commonality is level3 in chicago. sites are reachable from level3/cincinnati but not level3/chicago. traceroutes make it one hop past our bgp peer and seem to die there. for one particular remote web server (node1104.follett.com, and possibly others) traceroutes and pings from level3's looking glass (using chicago as the site) complete successfully, while neither does when ran from my gear through the same site. On 11/19/2014 10:46 AM, garya(a)completeweb.net wrote: > Also peering problems in Chicago, had to drop BGP to them until they figure it out. > Reported it 4 hours ago, no good response yet. > Gary > >> We have some customers unable to access their websites, seeing this on >> the way to them: >> >> 4 ae-0-11.bar2.Tampa1.Level3.net (4.69.137.110) 0.343 ms 0.423 ms >> 0.406 ms >> 5 ae-12-12.ebr1.Dallas1.Level3.net (4.69.137.118) 23.940 ms 23.470 >> ms 23.426 ms >> 6 ae-71-71.csw2.Dallas1.Level3.net (4.69.151.137) 24.026 ms >> ae-81-81.csw3.Dallas1.Level3.net (4.69.151.149) 24.007 ms >> ae-61-61.csw1.Dallas1.Level3.net (4.69.151.125) 23.855 ms >> 7 ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146) 23.361 ms >> ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 23.329 ms >> ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 23.936 ms >> 8 vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 23.378 ms >> vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.387 ms >> vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.963 ms >> 9 ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 23.357 ms >> ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16) 22.779 ms >> ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 62.640 ms >> 10 vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 24.179 ms 23.297 ms >> 23.403 ms >> 11 ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 65.617 ms >> ae-2-70.edge2.Dallas1.Level3.net (4.69.145.75) 23.286 ms >> ae-1-60.edge10.Dallas1.Level3.net (4.69.145.18) 23.553 ms >> 12 vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.303 ms >> vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 24.027 ms 24.028 ms >> 13 ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 24.018 ms >> ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146) 23.374 ms >> ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 23.342 ms >> 14 vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 24.068 ms >> vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.361 ms 23.348 ms >> 15 ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 24.023 ms >> ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16) 22.740 ms >> ae-4-90.edge10.Dallas1.Level3.net (4.69.145.210) 23.963 ms >> 16 vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.420 ms >> vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.400 ms >> vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 25.317 ms >> 17 ae-2-70.edge10.Dallas1.Level3.net (4.69.145.82) 23.985 ms >> ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 23.476 ms >> ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144) 51.762 ms >> 18 vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 23.416 ms >> vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 24.062 ms >> vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 24.051 ms >> 19 ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144) 44.798 ms >> ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146) 22.856 ms >> ae-1-60.edge10.Dallas1.Level3.net (4.69.145.18) 22.891 ms >> 20 vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.433 ms >> vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 25.333 ms >> vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.367 ms >> 21 ae-2-70.edge2.Dallas1.Level3.net (4.69.145.75) 23.475 ms >> ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208) 23.476 ms >> ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16) 64.753 ms >> 22 vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 24.008 ms 24.043 ms >> vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 23.508 ms >> 23 ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 23.505 ms >> ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144) 23.958 ms >> ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208) 24.058 ms >> 24 vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.429 ms >> vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 23.623 ms 25.547 ms >> 25 ae-2-70.edge10.Dallas1.Level3.net (4.69.145.82) 23.491 ms >> ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144) 23.551 ms >> ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 24.078 ms >> 26 vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 24.472 ms 24.144 ms >> 24.080 ms >> 27 ae-1-60.edge10.Dallas1.Level3.net (4.69.145.18) 28.121 ms >> ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146) 22.848 ms >> ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208) 24.090 ms >> 28 vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 23.509 ms 24.056 ms >> vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.522 ms >> 29 ae-2-70.edge2.Dallas1.Level3.net (4.69.145.75) 24.161 ms 24.153 ms >> ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208) 24.130 ms >> 30 vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.606 ms 23.482 ms >> vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.580 ms >> > >

1 0

zayo fiber cut in los angeles
by jd 19 Nov '14

19 Nov '14

anyone affected? i think it happened about 60-75 minutes ago. thanks

1 1