October 2009 - Outages - Outages Lists

Level3 issues in Dallas?
by Adam Greene 29 Oct '09

29 Oct '09

We are seeing consistent issues with connectivity which appear to originate at Level3 in Dallas ... Don't think it's related to our particular route announcements from ASN 20208 .... please see below :traceroute smithbarney.com *from 64.72.86.254* traceroute: Warning: smithbarney.com has multiple addresses; using 192.193.224.16 traceroute to smithbarney.com (192.193.224.16), 64 hops max, 40 byte packets 1 64-72-86-249 (64.72.86.249) 0.550 ms 0.554 ms 0.452 ms 2 204-15-4-241-static (204.15.4.241) 2.598 ms 3.343 ms 2.202 ms 3 webjog-hvdn.customer.hvdata.net (64.72.65.249) 5.099 ms 4.866 ms 5.199 ms 4 nwbrnynwj91-ge-7-1-7.lightower.net (72.22.160.199) 4.725 ms 5.016 ms 4.458 ms 5 hpjtnyxbj21-xe-2-3-0.lightower.net (72.22.160.103) 5.721 ms 3.679 ms 4.323 ms 6 pghknyshj91-xe-7-3-0.lightower.net (72.22.160.104) 5.973 ms 6.148 ms 5.320 ms 7 pghknyshj41-ae0-66.lightower.net (72.22.160.159) 5.719 ms 10.005 ms 5.828 ms 8 bstpmallj42-xe-0-2-0.lightower.net (72.22.160.149) 12.092 ms 12.879 ms 12.094 ms 9 xe-10-1-0.bar2.Boston1.Level3.net (4.53.54.5) 15.814 ms 11.628 ms 11.824 ms 10 ae-8-8.ebr1.NewYork1.Level3.net (4.69.140.98) 16.216 ms 17.513 ms 20.181 ms 11 ae-91-91.csw4.NewYork1.Level3.net (4.69.134.78) 24.959 ms 18.505 ms 16.826 ms 12 ae-74-74.ebr4.NewYork1.Level3.net (4.69.134.117) 22.706 ms ae-84-84.ebr4.NewYork1.Level3.net (4.69.134.121) 23.061 ms ae-94-94.ebr4.NewYork1.Level3.net (4.69.134.125) 35.255 ms 13 ae-3.ebr2.Dallas1.Level3.net (4.69.137.121) 57.115 ms 68.668 ms 58.914 ms 14 ge-7-0-70.ipcolo1.Dallas1.Level3.net (4.69.145.110) 53.817 ms ge-6-0-60.ipcolo1.Dallas1.Level3.net (4.69.145.46) 54.425 ms 53.667 ms 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * Tracing route to 174-143-170-178.static.cloud-ips.com [174.143.170.178] *from 204.15.4.2* over a maximum of 30 hops: 1 1 ms 1 ms 1 ms 204-15-4-1-static.webjogger.net [204.15.4.1] 2 1 ms 1 ms 1 ms 172.20.5.1 3 31 ms 3 ms 3 ms 204-15-4-241-static.webjogger.net [204.15.4.241] 4 5 ms 5 ms 4 ms webjog-hvdn.customer.hvdata.net [64.72.65.249] 5 4 ms 6 ms 4 ms nwbrnynwj91-ge-7-1-7.lightower.net [72.22.160.199] 6 5 ms 5 ms 5 ms hpjtnyxbj21-xe-2-3-0.lightower.net [72.22.160.103] 7 7 ms 6 ms 6 ms pghknyshj91-xe-7-3-0.lightower.net [72.22.160.104] 8 7 ms 6 ms 7 ms pghknyshj41-ae0-66.lightower.net [72.22.160.159] 9 12 ms 13 ms 39 ms bstpmallj42-xe-0-2-0.lightower.net [72.22.160.149] 10 19 ms 14 ms 12 ms xe-10-1-0.bar2.Boston1.Level3.net [4.53.54.5] 11 16 ms 18 ms 17 ms ae-8-8.ebr1.NewYork1.Level3.net [4.69.140.98] 12 17 ms 17 ms 18 ms ae-71-71.csw2.NewYork1.Level3.net [4.69.134.70] 13 * * * Request timed out. 14 64 ms 66 ms 63 ms ae-3.ebr2.Dallas1.Level3.net [4.69.137.121] 15 * 54 ms 54 ms ae-4-90.edge2.Dallas3.Level3.net [4.69.145.204] 16 * * * Request timed out. 17 * * * Request timed out. 18 * * * Request timed out. Adam -- Adam Greene Webjogger Internet Services http://www.webjogger.net (845) 757-4000

2 2

Off Topic Requezt
by Larry Sheldon 29 Oct '09

29 Oct '09

Would all of the people on this list using the "from address" of drinking.coffee(a)gmail.com please respond off-list. Thanks. -- Larry

1 0

Re: [outages] Level3 issues in Dallas?
by Larry Sheldon 29 Oct '09

29 Oct '09

In a recent screed (which I have since deleted here) I made sarcastic remarks that Matthew Walker thought were directed at him. They were not and I apologize. Quoting now from my response to him: > I'm sorry--I didn't mean to imply the beef was with you. (Or anybody > in particular, really). > > My beef (two beeves?) is with people that turned off ICMP and broke > some of the best tools we had for no gain. And with people who won't > acknowledge that the tools don't (and can't be relied upon to) work > any more. > > Kind of a if the shoe fits, wear it, otherwise I ain't talking to or > about you. > > I guess if I had to identify the most important beef, it would be > with the people whom turned the tools off. > > Again, I do apologize (and will publicly in a moment) for my lack of > clarity and the resultant misunderstanding. -- Requiescas in pace o email Two identifying characteristics of System Administrators: Ex turpi causa non oritur actio Infallibility, and the ability to learn from their mistakes. Eppure si rinfresca ICBM Targeting Information: http://tinyurl.com/4sqczs http://tinyurl.com/7tp8ml

1 0

Rack Space DFW Connectivity
by Ed Ramon 24 Oct '09

24 Oct '09

Hello I was wondering if I could pose a general question to all out there who watch and or interact with RackSapce specifically their DFW data center. It "feels" as if since the beginning of October our testing results which we hit Rack Space with at a interval of one minute have been coming back all over the place. Specifically we test out from Huston, DC and Chicago When I have called into them and chatted with their back bone teams I hear the same answer of "everything looks good." Interesting enough I now also hear some how Equinix being stated into their responses. So questions for anyone out there is: 1. Has anyone else seen the same oddities for Rack Space DFW? 2. Does anyone know if Rack Space DFW recently peered their traffic through Equinix DFW 3. The most recent alert/blip we received was 10-23-2009 at 2:49AM Eastern Thanks for your time and assistance, Ed

1 0

Bell Canada outage in Ontario?
by Denis Bazinet 23 Oct '09

23 Oct '09

Anyone else getting whacked? Multiple circuits only getting one-way traffic. (seeing mac addresses but not getting connectivity) Denis Bazinet Senior Manager, Network Operations Blink Communications

3 2

Fiber cut in WhiteRock TX - near dallas....
by Steven Fischer 23 Oct '09

23 Oct '09

affecting our site in Little Rock, AR on Verizon. Stand by for an update -- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy

1 0

Level 3 down in Atlanta
by Scott Howard 22 Oct '09

22 Oct '09

Went down about 35 minutes ago. Anyone know what's going on? Scott.

17 23

Level3 Problem in London
by Jason Dearborn 22 Oct '09

22 Oct '09

Level 3 is experiencing heavy packet loss (>60%) out of 6 Braham St, London UK. They claim one of their devices is having routing problems. Techs have been dispatched to the facility. Parent ticket is 3724603

1 1

Primary Radianz Circuit is down...
by Forestal, Andre Jr. 21 Oct '09

21 Oct '09

The Primary Radianz DS3 is currenly down. Traffic is being routed over secondary circuit. Ticket has been opened with Radianz Andre Forestal Jr. Network Engineer Wolverine Trading, LLC 111 Broadway New York, NY 10006 Office (646)485-6686 Cell (646)696-0298 Fax (646)485-7101 This message is intended only for the personal and confidential use of the recipients named above. If the reader of this email is not the intended recipient, you have received this email in error and any review, dissemination, distribution or copying is strictly prohibited. If you have received this email in error, please notify the sender immediately by return email and permanently delete the copy you received. This message is provided for informational purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments. Wolverine is not responsible for any recommendation, solicitation, offer or agreement or any information about any transaction, customer account or account activity that may be attached to or contained in this communication. Wolverine accepts no liability for any content contained in the email, or any errors or omissions arising as a result of email transmission. Any opinions contained in this email constitute the sender's best judgment at this time and are subject to change without notice.

1 1

Re: [outages] Level 3 down in Atlanta
by Bill Wichers 20 Oct '09

20 Oct '09

It's possible the faulty device was an aggregation router and only those customers connected to it saw the outage. Redundancy within Level(3)'s network would not have helped customers with only a single connection off the affected device. At some point it becomes the customer's responsibility to redundantly connect to their upstreams (as in buy two connections on different aggregation routers :-). I didn't see any L3 issues up here in Detroit, so whatever outage L3 had was apparently contained within that one POP or region. No cascade failures I can see... -Bill > Level 3 has a single router or switch handling packets at a major POP? > I doubt this, but the outage is confirmation something bad happened. > > That said: where's the redundancy, and why didn't it kick in? > > -- > | Jeremy Chadwick jdc(a)parodius.com | > | Parodius Networking http://www.parodius.com/ | > | UNIX Systems Administrator Mountain View, CA, USA | > | Making life hard for others since 1977. PGP: 4BD6C0CB |

2 1