May 2015 - Outages - Outages Lists

Re: [outages] Linode Fremont outage
by Jim Popovitch 31 May '15

31 May '15

On Sun, May 31, 2015 at 12:16 PM, Chris Vervais <cvervais(a)me.com> wrote: > >> On May 31, 2015, at 08:06, Jim Popovitch via Outages <outages(a)outages.org> wrote: >> >> On Sun, May 31, 2015 at 11:00 AM, Mark Keymer via Outages >> <outages(a)outages.org> wrote: >>> But honestly I am guessing it was a failure with no foul play happening. >> >> Apparently no solid proactive testing plan either. > > For all we know the generator was checked last week and was fine. > > In the absence of all the facts we should probably restrain from assuming one thing or another and give them the benefit of the doubt. And, that they're taking this is a learning experience and delving into where gaps exist and how to close them. Of course, for a new DC or relatively new DC operator. But come on, HE @ FMT2 are where I would expect perfection. -Jim P.

4 3

Re: [outages] Linode Fremont outage
by Alex Rubenstein 31 May '15

31 May '15

So the outage was OK to have since they had staff on site? On May 30, 2015 09:01, Paul WALL via Outages <outages(a)outages.org> wrote: It's a good thing Hurricane Electric had electricians on site to deal with the outage. If it were an Equinix or Telx site, the response would have assuredly been far worse. Paul _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

11 12

Re: [outages] Linode Fremont outage
by Paul WALL 30 May '15

30 May '15

It's a good thing Hurricane Electric had electricians on site to deal with the outage. If it were an Equinix or Telx site, the response would have assuredly been far worse. Paul

1 0

Linode Fremont outage
by Sadiq Saif 30 May '15

30 May '15

No, it is not just you. Datacenter Outage - Fremont http://status.linode.com/incidents/2rm9ty3q8h3x -- Sadiq Saif (AS393949) https://staticsafe.ca

3 2

CenturyLink
by Bob Hancock 29 May '15

29 May '15

Is any one seeing issues with CenturyLink circuits? Getting some reports in the mid-west. -- Bob Hancock

6 7

XO outage in Dallas
by Ryan Gelobter 29 May '15

29 May '15

Anyone else having issues with XO in Dallas? Two DIA circuits with BGP down for 1-2hrs.

1 0

TWC national issue?
by Ian Mock 29 May '15

29 May '15

We're having trouble with any TWC business class link getting back to us on TW Telecom/Level3. So far it seems to be a national issue. I have reports from Cali to North/South Carolina. Our block we're having trouble reaching is 209.163.181.64/27. Anyone know what might be going on? _______________________________________________ Tracing route to [209.163.181.75] over a maximum of 30 hops: 1 9 ms <1 ms <1 ms kcbow [192.168.1.1] 2 1126 ms 41 ms 22 ms cpe-172-72-64-1.carolina.res.rr.com [172.72.64.1 ] 3 29 ms 30 ms 31 ms 24.74.253.225 4 13 ms 10 ms 11 ms cpe-024-074-247-128.carolina.res.rr.com [24.74.2 47.128] 5 18 ms 18 ms 27 ms be32.chrcnctr01r.southeast.rr.com [24.93.64.202] 6 26 ms 22 ms 26 ms bu-ether45.asbnva1611w-bcr00.tbone.rr.com [107.1 4.19.44] 7 31 ms 30 ms 30 ms bu-ether32.vinnva0510w-bcr00.tbone.rr.com [107.1 4.19.165] 8 35 ms 22 ms 24 ms 0.ae0.pr0.dca20.tbone.rr.com [107.14.19.65] 9 22 ms 24 ms 24 ms ge-7-1-6.pr0.chi10.tbone.rr.com [66.109.9.74] 10 92 ms * * ae-4-90.edge5.Dallas3.Level3.net [4.69.145.209] 11 * * * Request timed out. 12 47 ms 54 ms 51 ms dal2-pr1-ae1-0.us.twtelecom.net [64.128.212.65] 13 49 ms 46 ms 50 ms dal1-ar1-ae21-0.us.twtelecom.net [66.192.243.70] 14 47 ms 47 ms 49 ms [209.163.181.75] 15 47 ms 46 ms 46 ms [209.163.181.75] #2 Tracing route to [209.163.181.75] over a maximum of 30 hops: 1 <1 ms <1 ms <1 ms router.Belkin [192.168.2.1] 2 * * * Request timed out. 3 21 ms 33 ms 21 ms 24.74.254.49 4 13 ms 16 ms 14 ms cpe-024-074-247-130.carolina.res.rr.com [24.74.2 47.130] 5 18 ms 14 ms 15 ms be32.drhmncev01r.southeast.rr.com [24.93.64.200] 6 18 ms 18 ms 19 ms bu-ether14.atlngamq46w-bcr00.tbone.rr.com [66.10 9.6.82] 7 29 ms 18 ms 17 ms 0.ae2.pr0.atl20.tbone.rr.com [107.14.19.11] 8 * * * Request timed out. 9 35 ms 38 ms * ae-4-90.edge5.Dallas3.Level3.net [4.69.145.209] 10 36 ms 38 ms 36 ms ae-4-90.edge5.Dallas3.Level3.net [4.69.145.209] 11 36 ms 35 ms 36 ms dal2-pr1-ae1-0.us.twtelecom.net [64.128.212.65] 12 37 ms 35 ms 38 ms dal1-ar1-ae21-0.us.twtelecom.net [66.192.243.70] 13 39 ms 39 ms 37 ms [209.163.181.75] 14 37 ms 42 ms 38 ms [209.163.181.75] Trace complete. Thanks, Ian Mock

1 0

Sprint outage (Toledo, OH)
by Jason Slagle 29 May '15

29 May '15

It appears Sprint cell service is out in at least the Toledo, OH market. No signal at all on the phone so it seems like more than a backhaul issue. Twitter, etc blowing up about it. Second or third outage like this in the past year or so.

1 0

Verizon (Dallas) Routing Issues?
by Justin Head 28 May '15

28 May '15

I'm seeing increased latency and some packet loss to sites over ALTER.NET (Verizon) to NTT and Level 3. I'm not seeing the same thing to GTT or Telia right now. I'm based in Dallas, TX. Routing inside Dallas seems fine. speedtest.net: http://pastie.org/pastes/10212782/text amazon.com: http://pastie.org/pastes/10212783/text yahoo.com: http://pastie.org/pastes/10212785/text Anyone else seeing anything similar? Looks like *.NYC4.ALTER.NET might be to blame? - Justin

6 6

Windstream - Atlanta failures
by Slade Edmonds 28 May '15

28 May '15

We've been battling intermittent outages with our Windstream MPLS network for a long while now. Yes, there are the major outages we've suffered that seem to impact Windstream customers nationwide, but on top of those we also have intermittent outages that seem to only impact our Atlanta location, and for brief periods of time. Just curious if there are any other Atlanta area customers running Windstream MPLS that experience these blips. To be accurate, we are actually about 50 miles west of Atlanta (Carrollton). Anyway, here's two from today 5/28/2015: Outage 1: 2:06 a.m. EST - Carrollton went off the map for about 7 seconds. No loss of BGP peering, no loss of BGP routes. Outage 2: 12:32 p.m. EST - Carrollton went off the map for about 45 seconds. No loss of BGP peering, no loss of BGP routes. Thanks!

1 0