February 2015 - Outages - Outages Lists

SoftLayer Latency Jump 4x last 30 minutes
by Peter Beckman 27 Feb '15

27 Feb '15

Anyone else seeing increased latency to SoftLayer destinations? Beginning at 2:33:25pm EST and continuing for the last 26 minutes, Round Trip Time jumped from a fairly constant 36-44ms to 142-225ms. This may not seem like much, but it impacts Voice Quality for VoIP calls. HOST: www.nova.org Loss% Snt Last Avg Best Wrst StDev 1. xl0.tic.nova.org 0.0% 15 0.2 0.2 0.1 0.2 0.0 2. wsip-70-168-111-17.dc.dc.cox 0.0% 15 1.4 16.4 0.9 204.2 52.4 3. ip68-100-3-109.dc.dc.cox.net 0.0% 15 0.9 1.1 0.8 2.3 0.5 4. ashbbprj01-ae2.rd.as.cox.net 0.0% 15 1.6 2.3 1.5 8.7 1.8 5. xe-1-0-2.bbr02.eq01.wdc02.ne 0.0% 15 3.9 2.1 1.7 3.9 0.6 6. ae1.bbr02.eq01.chi01.network 0.0% 15 225.8 225.4 223.3 226.1 0.8 7. ae7.bbr01.eq01.chi01.network 6.7% 15 226.4 226.1 225.3 226.7 0.4 8. ae20.bbr01.eq01.dal03.networ 0.0% 15 142.2 143.0 141.8 151.9 2.5 9. ae0.dar02.sr01.dal01.network 0.0% 15 142.1 142.0 140.6 142.5 0.5 10. po2.fcr03.sr04.dal01.network 6.7% 15 142.3 142.8 140.8 151.4 2.5 11. 67.228.7.104-static.reverse. 13.3% 15 142.5 142.3 141.5 142.5 0.3 --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman(a)angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------

3 3

Vandalism of CenturyLink fibre in Arizona causes large outage footprint.
by Roland Dobbins 26 Feb '15

26 Feb '15

<https://uk.news.yahoo.com/police-probing-outage-cut-internet-phones-arizona…> ----------------------------------- Roland Dobbins <rdobbins(a)arbor.net>

5 5

Windstream Phone Outage - Southeast (TN)
by John Pinkerton 26 Feb '15

26 Feb '15

I'm currently experiencing a total phone outage with Windstream for the 4th day in less than one weeks time and the 3rd time since Monday. Poor weather has caused significant issues in Georgia according to their support staff, with snow and ice in TN being blamed for prior day outages. Typical time to repair has been about 1hr over the past week. -- John Pinkerton | IT Manager <http://goog_487791471> www.trewaudio.com john.pinkerton(a)trewaudio.com 615-256-3542 <http://www.trewaudio.com>

1 0

Re: [outages] Level 3 problems in Miami?
by Blair Trosper 26 Feb '15

26 Feb '15

Level 3 confirms, ticket is open. On Thu, Feb 26, 2015 at 8:59 AM, Blair Trosper <blair.trosper(a)gmail.com> wrote: > It's also failing in reverse from the Level 3 LG...doing a traceroute from > Miami to myself, this is the result: > > 1 ae-1-51.edge4.Miami1.Level3.net (4.69.138.76) 0.591 ms 7.49 ms > 0.540 ms > 2 TWC-level3-40G.Miami.Level3.net (4.68.62.182) 0.668 ms 0.680 ms > 15.2 ms > 3 0.0.0.0 * * * > 4 0.0.0.0 * * * > 5 0.0.0.0 * * * > > Looks like it can't get any further than the interconnect router between > Level 3 and TWC...can someone from Level 3 reach out or look into this > please? > > On Thu, Feb 26, 2015 at 8:34 AM, Blair Trosper <blair.trosper(a)gmail.com> > wrote: > >> Also seeing it after this one: >> po5.ar1.mia2.gblx.net (67.16.148.102) >> >> On Thu, Feb 26, 2015 at 8:32 AM, Blair Trosper <blair.trosper(a)gmail.com> >> wrote: >> >>> Anyone else having massive trouble getting to endpoints beyond core >>> routers in Miami on Level 3? >>> >>> I'm cut off (packets die) from Miami and Tampa after this specific >>> router: >>> >>> po4-20g.ar1.mia2.gblx.net (67.16.134.218) >>> >>> If anyone from Level 3 could reach out, or if anyone knows what's going >>> on and can say, I'd appreciate it. >>> >>> Thanks, >>> Blair >>> >> >> >

1 0

Prescott, AZ Fiber Cut (Centurylink)
by L.T. Easterly 25 Feb '15

25 Feb '15

http://dcourier.com/main.asp?SectionID=1&SubSectionID=1&ArticleID=142238 -- __________________________________________________________________________________________ QTH KMLB/Eau Gallie, Florida::ICOM R75::Alinco DJ-X11::RFSpace SDR-IQ::Pixel Loop RF-1B:: Uniden HP2 SDR Captures/loggings posted to: https://corq.co when audio quality allows. __________________________________________________________________________________________

1 1

NameCheap down, was it hacked?
by Chuck Anderson 20 Feb '15

20 Feb '15

It looks like NameCheap is down. It may have been hacked, because I got a Security Notification from them about a domain I have. https://www.namecheap.com/ Website is offline No cached version of this page is available. Error 522 Ray ID: 1bbc4b52c02609be Connection timed out http://downforeveryoneorjustme.com/www.namecheap.com It's not just you! http://www.namecheap.com looks down from here. Check another site? The e-mail says: Received: from mta.messagebus.com (mba1056.namecheap.messagebus.com [108.175.18.57]) From: "Namecheap.com - Security Notification" <support(a)namecheap.com> Subject: Security Notification ---------------------------------------------------------------------- Namecheap.com Activity Notification ---------------------------------------------------------------------- Dear <name> There was some activity in your namecheap account. Information on what type of change occurred is available below. Notification For : HOSTS Date : 2/20/2015 11:36:09 AM IP Address : 192.168.1.10 Username : <username> Domain (if relevant) : <domain>.org Old Details ----------- Host Information ------------------ New Details ----------- Host Information ------------------ Name: www Type: CNAME Address:parkingpage.namecheap.com. Name: @ Type: URL Address: http://www.<domain>.org/?from=@ Additional Information ---------------------- N/A Thanks, Namecheap.com http://www.namecheap.com >host 108.175.18.57 57.18.175.108.in-addr.arpa domain name pointer mba1056.namecheap.messagebus.com. >whois 108.175.18.57 [Querying whois.arin.net] [Redirected to whois.messagebus.com:4321] [Querying whois.messagebus.com] [whois.messagebus.com] %rwhois V-1.5:003fff:00 rwhois.messagebus.com (by Network Solutions, Inc. V-1.5.10-pre6) network:Class-Name:network network:ID:MESSAGEBUS-BLOCK1.108.175.16.0/20 network:Auth-Area:108.175.16.0/20 network:Network-Name:MESSAGEBUS-BLOCK1.108.175.106.0 network:IP-Network:108.175.16.0/20 network:IP-Network-Block:108.175.16.0-108.175.24.0 network:Organization;I:Message Bus network:Tech-Contact;I:hostmaster@labs.messagebus.com network:Admin-Contact;I:hostmaster@labs.messagebus.com network:Created:20120706 network:Updated:20130313 network:Updated-By:hostmaster@labs.messagebus.com %referral rwhois://whois.messagebus.com:4321/auth-area=108.175.16.0/20

2 2

Cogent home page issues?
by Frank Bulk 20 Feb '15

20 Feb '15

According to the status page (http://status.cogentco.com) there's nothing going on, but from AS53347 our monitoring system hasn't been able to reliably pull www.cogentco.com over IPv4 and IPv6 (since 8:15 pm Central) nor ipv6.cogentco.com (same address as the AAAA for www.cogentco.com) From a jump box that's on NTT's network I was able to load www.cogentco.com over IPv4, but I can't load it over IPv6 (get a "No data received") or it takes a very long time. Frank >From AS53347: root@nagios:/etc# tcptraceroute www.cogentco.com Selected device eth0.3, address 96.31.0.5, port 55934 for outgoing packets Tracing the path to www.cogentco.com (38.100.128.10) on TCP port 80 (www), 30 hops max 1 router-core-inside.mtcnet.net (96.31.0.254) 0.256 ms 0.200 ms 0.200 ms 2 sxct.sxcy.mtcnet.net (167.142.156.197) 0.188 ms 0.124 ms 0.120 ms 3 premier.sxcy-mlx.fbnt.netins.net (173.215.60.5) 1.672 ms 1.667 ms 1.597 ms 4 ins-db4-et-0-7-0-0.desm.netins.net (167.142.67.198) 7.933 ms 7.753 ms 7.659 ms 5 ins-dc2-et-9-4.desm.netins.net (167.142.67.197) 7.899 ms 7.934 ms 7.958 ms 6 te0-0-1-1.rcr11.dsm01.atlas.cogentco.com (38.104.184.25) 7.889 ms 7.965 ms 7.897 ms 7 te0-2-0-0.ccr41.ord01.atlas.cogentco.com (154.54.46.237) 15.095 ms 15.216 ms 15.077 ms 8 be2351.ccr21.cle04.atlas.cogentco.com (154.54.44.86) 23.406 ms 23.469 ms 23.863 ms 9 be2485.mpd21.dca01.atlas.cogentco.com (154.54.30.18) 35.619 ms 36.489 ms 35.699 ms 10 * * * 11 * * * 12 * * cogentco.sg (38.100.128.10) [open] 47.213 ms root@nagios:/etc# tcptraceroute6 www.cogentco.com traceroute to cogentco.com (2001:550:1::cc01) from 2607:fe28:0:1000::5, port 80, from port 40783, 30 hops max, 60 bytes packets 1 router-core.mtcnet.net (2607:fe28:0:1000::1) 0.230 ms 0.216 ms 0.204 ms 2 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 0.201 ms 0.130 ms 0.127 ms 3 v6-premier.sxcy-mlx.fbnt.netins.net (2001:5f8:7f0a:2::1) 1.604 ms 1.587 ms 1.598 ms 4 v6-ins-db4-te-0-6-0-4-219.desm.netins.net (2001:5f8:1:1::1) 8.139 ms 7.777 ms 8.205 ms 5 v6-ins-dc2-lo0.desm.netins.net (2001:5f8::5) 7.992 ms 7.959 ms 7.970 ms 6 2001:550:2:24::9:1 (2001:550:2:24::9:1) 8.053 ms 7.954 ms 8.100 ms 7 te0-2-0-0.ccr41.ord01.atlas.cogentco.com (2001:550:0:1000::9a36:2eed) 15.226 ms 15.371 ms 15.153 ms 8 be2351.ccr21.cle04.atlas.cogentco.com (2001:550:0:1000::9a36:2c56) 22.562 ms 22.777 ms 22.460 ms 9 be2485.mpd21.dca01.atlas.cogentco.com (2001:550:0:1000::9a36:1e12) 35.259 ms 35.377 ms 35.435 ms 10 te2-8.ccr01.iad03.atlas.cogentco.com (2001:550:0:1000::9a36:29fe) 35.191 ms 35.144 ms 35.335 ms 11 te2-1.mag01.iad03.atlas.cogentco.com (2001:550:0:1000::9a36:563e) 35.129 ms 35.171 ms 35.129 ms 12 * cogentco.com (2001:550:1::cc01) 47.093 ms [open] * root@nagios:/etc#

1 1

Comcast LA issue?
by Jeremy Chadwick 19 Feb '15

19 Feb '15

Something weird is going on with Comcast in Los Angeles: small packets appear to be getting discarded with a loss rate of ~6-10% while larger packets (I've tried 1000 and 1200) seem to make it through without any loss. Things like SSH and IMAP+SSL are impacted consistently (particularly during key exchange), while HTTP does not appear to be, so the issue appears to be affecting both TCP and ICMP (have not tested UDP). First sign of this issue, according to my logs, is at 11:51 PT (UTC-8), and has gradually gotten worse over the past few hours. Looks like the issue is with the forward path (Source->Destination listed below), as the return path only shows loss at the final destination. (I currently cannot provide a return-path mtr because the only way I can access my box is through VNC, and I don't like doing things like screenshots of a VNC window, so you'll just have to trust me) Nothing anomalous shows up when using L3's looking glass to ping things, and the only Comcast LG I know of is in NYC. I most likely won't see Email responses from anyone (directly or to the list) until this problem is addressed, since my mail is hosted on shown box and cannot currently be fetched due to the problem. Frustrating. Source IP: 69.181.142.213 Destination IP: 174.136.108.146 ICMP 56 bytes: Packets Pings Host Loss% Snt Rcv Last Avg Best Wrst 1. gw.home.lan (192.168.1.1) 0.0% 185 185 0.2 0.2 0.2 0.3 2. 69.181.136.1 0.0% 185 185 8.9 9.9 7.0 21.1 3. te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 185 185 8.7 9.9 7.9 17.0 4. te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134 0.0% 185 185 10.7 12.9 9.7 20.7 5. he-1-5-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.90.93) 98.4% 184 3 49496 49006 47906 49615 6. he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102) 5.4% 184 174 11.9 14.1 10.9 23.5 7. 50.242.148.34 6.5% 184 172 11.7 13.4 10.4 69.9 8. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142) 66.7% 184 61 44.9 20.6 18.0 44.9 9. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142) 59.0% 184 75 18.7 19.8 18.0 26.0 10. ae-27-27.car2.LosAngeles1.Level3.net (4.69.201.18) 2.2% 184 180 20.8 63.1 18.2 293.3 11. GIGLINX-INC.car2.LosAngeles1.Level3.net (4.30.195.86) 6.0% 184 173 19.2 22.2 19.1 39.3 12. mambo.koitsu.org (174.136.108.146) 6.5% 184 172 18.8 20.6 18.5 36.9 Note: Loss shown at above hops #5, #8, and #9 are normal according to historic data (i.e. ICMP prioritisation is in effect) -- however loss at hop #6 is new. ICMP 1200 bytes: Packets Pings Host Loss% Snt Rcv Last Avg Best Wrst 1. gw.home.lan (192.168.1.1) 0.0% 43 43 0.3 0.3 0.2 0.4 2. 69.181.136.1 0.0% 43 43 9.7 10.4 8.6 14.3 3. te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 42 42 10.0 10.0 8.7 16.0 4. te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134 0.0% 42 42 11.7 13.3 10.3 18.9 5. ??? 6. he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102) 0.0% 42 42 14.4 15.9 12.4 25.1 7. 50.242.148.34 0.0% 42 42 12.7 13.6 11.2 23.1 8. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142) 9.8% 42 37 20.0 20.7 18.8 25.0 9. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142) 0.0% 42 42 19.1 21.1 18.9 29.9 10. ae-27-27.car2.LosAngeles1.Level3.net (4.69.201.18) 0.0% 42 42 64.3 105.7 20.9 412.5 11. GIGLINX-INC.car2.LosAngeles1.Level3.net (4.30.195.86) 0.0% 42 42 20.6 22.5 19.9 31.2 12. mambo.koitsu.org (174.136.108.146) 0.0% 42 42 20.7 22.0 20.1 26.4 -- | Jeremy Chadwick jdc(a)koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |

3 6

Verizon BGP issues?
by Pete E 19 Feb '15

19 Feb '15

Anyone else out there experience a pretty vast routing issue with Verizon, starting around 10:50 eastern? We first saw VPN tunnels and SIP Gateways drop off then (caused by) BGP instability.

3 2

NYIIX / 60 Hudson / TelX blip?
by Eric Dugas 13 Feb '15

13 Feb '15

Any one located in the NYC area seen any events from 15:20:00 and 15:45:00 at NYIIX and/or 60 Hudson and/or TelX? TelX confirmed there was no power outage reported. Will contact NYIIX. Thanks Eric Dugas AS40191

2 1