January 2013 - Outages - Outages Lists

Re: [outages] Power Outage/Fiber Splice In Hawaii?
by Scott Weeks 15 Jan '13

15 Jan '13

--- brentr(a)progressive-solutions.com wrote: From: "Roberts, Brent" <brentr(a)progressive-solutions.com> Looks like there is either a rather Large Power outage or Fiber Cut In Hawaii at the moment. I personally have multiple circuits down and Random packet loss to Different Carriers on the Islands. Anyone else seeing anything similar? ----------------------------------------------- I can confirm that the ILEC here (Hawaiian Telcom) is/was having troubles that're even affecting their POTS. I no longer have any insight as to what the trouble is exactly, though, as I haven't worked there for a couple of years. scott _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

1 0

Twitter connectivity issues?
by Ross Halliday 12 Jan '13

12 Jan '13

Hi list, I'm seeing what appears to be a broken Layer 3 or 4 port channel (load balancing) somewhere in Twitter land. I have no idea if this is affecting anyone other than us as searching hasn't yielded anything, and Twitter's status page shows no issues. At the end of my message I've put some example of what I'm seeing. Behaviour changes based on source/dest IP, based on many customer and infrastructure addresses in different subnets, and a variety of the A records returned for twitter.com. We are multihomed so that might obscure the problem, I've included a sample out our 2nd provider. The problem appears to be somewhere past ae60.pao1-cr1.twttr.com (199.16.159.85) If anyone from Twitter is out there please drop me a line... we are AS 23252 If anyone else is experiencing this I'd love to know also Thanks --- Ross Halliday Network Operations WTC Communications Office: 613-547-6939 x203 Helpdesk: 866-547-6939 option 2 http://www.wtccommunications.ca edge-wprt-c6509#traceroute 199.59.150.39 source 66.102.70.237 Type escape sequence to abort. Tracing the route to r-199-59-150-39.twttr.com (199.59.150.39) 1 ge-6-1-117.car1.Toronto2.Level3.net (4.59.180.17) [AS 3356] 4 msec 4 msec 4 msec 2 4.69.153.77 [AS 3356] [MPLS: Label 493184 Exp 0] 4 msec 4 msec 4 msec 3 ae-0-11.bar2.Toronto1.Level3.net (4.69.151.242) [AS 3356] [MPLS: Label 492736 Exp 0] 4 msec 8 msec 4 msec 4 ae-7-1.ebr2.NewYork1.Level3.net (4.69.151.114) [AS 3356] [MPLS: Label 1494 Exp 0] 24 msec 24 msec 20 msec 5 ae-6-6.ebr2.NewYork2.Level3.net (4.69.141.22) [AS 3356] [MPLS: Label 1343 Exp 0] 16 msec 20 msec 16 msec 6 ae-1-100.ebr1.NewYork2.Level3.net (4.69.135.253) [AS 3356] [MPLS: Label 1278 Exp 0] 20 msec 24 msec 24 msec 7 ae-3-3.ebr2.Washington1.Level3.net (4.69.132.89) [AS 3356] [MPLS: Label 1355 Exp 0] 24 msec 24 msec 20 msec 8 ae-82-82.csw3.Washington1.Level3.net (4.69.134.154) [AS 3356] [MPLS: Label 1452 Exp 0] 24 msec ae-72-72.csw2.Washington1.Level3.net (4.69.134.150) [AS 3356] [MPLS: Label 1390 Exp 0] 24 msec ae-62-62.csw1.Washington1.Level3.net (4.69.134.146) [AS 3356] [MPLS: Label 1388 Exp 0] 24 msec 9 ae-2-70.edge2.Washington4.Level3.net (4.69.149.80) [AS 3356] 24 msec 24 msec ae-1-60.edge2.Washington4.Level3.net (4.69.149.16) [AS 3356] 36 msec 10 TWITTER-INC.edge2.Washington4.Level3.net (4.53.114.102) [AS 3356] 20 msec 20 msec 20 msec 11 ae60.pao1-cr1.twttr.com (199.16.159.85) [AS 13414] 96 msec 92 msec 92 msec 12 * * * 13 r-199-59-150-39.twttr.com (199.59.150.39) [AS 13414] 92 msec 100 msec 100 msec edge-wprt-c6509#traceroute 199.59.150.39 source 66.102.70.233 Type escape sequence to abort. Tracing the route to r-199-59-150-39.twttr.com (199.59.150.39) 1 ge-6-1-117.car1.Toronto2.Level3.net (4.59.180.17) [AS 3356] 4 msec 4 msec 4 msec 2 4.69.153.77 [AS 3356] [MPLS: Label 493184 Exp 0] 28 msec 32 msec 4 msec 3 ae-0-11.bar2.Toronto1.Level3.net (4.69.151.242) [AS 3356] [MPLS: Label 492736 Exp 0] 16 msec 36 msec 4 msec 4 ae-7-1.ebr2.NewYork1.Level3.net (4.69.151.114) [AS 3356] [MPLS: Label 1494 Exp 0] 24 msec 24 msec 24 msec 5 ae-6-6.ebr2.NewYork2.Level3.net (4.69.141.22) [AS 3356] [MPLS: Label 1343 Exp 0] 20 msec 16 msec 20 msec 6 ae-1-100.ebr1.NewYork2.Level3.net (4.69.135.253) [AS 3356] [MPLS: Label 1278 Exp 0] 16 msec 20 msec 16 msec 7 ae-3-3.ebr2.Washington1.Level3.net (4.69.132.89) [AS 3356] [MPLS: Label 1355 Exp 0] 20 msec 24 msec 24 msec 8 ae-92-92.csw4.Washington1.Level3.net (4.69.134.158) [AS 3356] [MPLS: Label 1396 Exp 0] 28 msec ae-72-72.csw2.Washington1.Level3.net (4.69.134.150) [AS 3356] [MPLS: Label 1390 Exp 0] 28 msec 24 msec 9 ae-4-90.edge2.Washington4.Level3.net (4.69.149.208) [AS 3356] 24 msec ae-2-70.edge2.Washington4.Level3.net (4.69.149.80) [AS 3356] 92 msec ae-3-80.edge2.Washington4.Level3.net (4.69.149.144) [AS 3356] 24 msec 10 TWITTER-INC.edge2.Washington4.Level3.net (4.53.114.102) [AS 3356] 20 msec 20 msec 24 msec 11 ae60.pao1-cr1.twttr.com (199.16.159.85) [AS 13414] 92 msec 92 msec 92 msec 12 * * * 13 * * * 14 * * * 15 * <snip> edge-wprt-c6509#traceroute 199.59.150.39 Type escape sequence to abort. Tracing the route to r-199-59-150-39.twttr.com (199.59.150.39) 1 ge5-1-5.core1.toronto1.nexicom.net (76.75.127.145) [AS 11666] 0 msec 8 msec 4 msec 2 xe2-2-0-1.core1.toronto1.nexicom.net (98.124.49.233) [AS 32881] 4 msec 4 msec 4 msec 3 206.126.236.97 [AS 8218] 20 msec 20 msec 20 msec 4 ae60.pao1-cr1.twttr.com (199.16.159.85) [AS 13414] 80 msec 80 msec 88 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * *

1 1

PLANNED OUTAGE: Wikipedia.org
by Jay Ashworth 11 Jan '13

11 Jan '13

Wikipedia is moving their Ashburn VA datacenter, 17-01UTC three days in a row, 22/3/4 January. Print this out and hang it on the wall you keep such things on, so everyone will know. Apologies for the crappy line wrap; I've had 36 RFC-compliance bugs open on Zimbra for about 4 years now. Cheers, -- jra ----- Forwarded Message ----- > From: "Ct Woo" <ctwoo(a)wikimedia.org> > To: "Wikimedia developers" <wikitech-l(a)lists.wikimedia.org>, "Development and Operations Engineers" > <engineering(a)lists.wikimedia.org> > Sent: Friday, January 11, 2013 3:07:15 PM > Subject: [Wikitech-l] Update on Ashburn data center switchover / migration – target date is week of 1/22/13 > All, > > The Migration team is in the last lap on completing the remaining > tasks to > ready our software stack and Ashburn infrastructure for the big > switchover > day. > > Per my last > update,<http://lists.wikimedia.org/pipermail/wikitech-l/2012-October/063668.html> > with the Fundraising activity behind us now, the team has scheduled > the *week > of 22nd January*, 2013 to perform the switchover. We are going to > block a > 8-hour migration window on the *22nd, 23rd and 24**th*. During those > periods, *17:00 UTC to 01:00 UTC hours (9am to 5pm PST*), there will > be > intermittent blackouts and they will be treated as 'planned' outages. > You > can follow the migration on irc.freenode.org in the > #wikimedia-operations > channel. > > The team is putting the finishing touches to the last few tasks and we > will > make the final Go/No decision on 18th Jan, 2013. An update will send > out > then. For those interested in tracking the progress, the meeting notes > are > captured on this wikitech > page<http://wikitech.wikimedia.org/view/Eqiad_Migration_Planning#Improving_Switc…> > . > > *Please note that we will be restricting code deployment during that > week, > allowing only emergency and critical ones only.* > > Thanks. > > CT Woo > _______________________________________________ > Wikitech-l mailing list > Wikitech-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274

1 0

Twitter issues
by Ross Halliday 11 Jan '13

11 Jan '13

(Hopefully the third try's the charm) Anybody else seeing connectivity issues reaching Twitter? (AS 13414). I'm seeing a what looks like a Layer 3 portchannel with a misbehaving or downed member link past ae60.pao1-cr1.twttr.com (199.16.159.85). Customer reports started rolling in about 11:30 AM EST yesterday for the main website and t.co. http://downrightnow.com/twitter also shows problems. I've been trying to get ahold of someone at Twitter's NOC but haven't had any luck so far. --- Ross Halliday Network Operations WTC Communications Office: 613-547-6939 x203 Helpdesk: 866-547-6939 option 2 http://www.wtccommunications.ca

1 0

CenturyLink DS-3 outage in midwest
by Frank Bulk 11 Jan '13

11 Jan '13

We learned from a partner that CenturyLink/Qwest has a DS-3 outage affecting some our partner's sites in Iowa. The full scope is unknown to us -- it could be just a single DS-3. Frank

1 1

Twitter connectivity issues?
by salad＠iri5.net 11 Jan '13

11 Jan '13

Hi list, I tried to send this yesterday, but after 15 hours I don't see my post, so appologies for any duplicates. Customer complaints started rolling in around 11:30 AM EST (GMT-0:500) yesterday, or approx 23 hours ago. I'm seeing what appears to be a broken Layer 3 or 4 port channel (load balancing) somewhere in Twitter land. I have no idea if this is affecting anyone other than us as searching hasn't yielded anything, and Twitter's status page shows no issues. At the end of my message I've put some example of what I'm seeing. Behaviour changes based on source/dest IP, based on many customer and infrastructure addresses in different subnets, and a variety of the A records returned for twitter.com. We are multihomed so that might obscure the problem, I've included a sample out our 2nd provider. The problem appears to be somewhere past ae60.pao1-cr1.twttr.com (199.16.159.85) If anyone from Twitter is out there please drop me a line... I've dropped a line to the ARIN NOC contact but haven't gotten a response. We are AS 23252 If anyone else is experiencing this I'd love to know also Thanks --- Ross Halliday Network Operations WTC Communications Office: 613-547-6939 x203 Helpdesk: 866-547-6939 option 2 http://www.wtccommunications.ca edge-wprt-c6509#traceroute 199.59.150.39 source 66.102.70.237 Type escape sequence to abort. Tracing the route to r-199-59-150-39.twttr.com (199.59.150.39) 1 ge-6-1-117.car1.Toronto2.Level3.net (4.59.180.17) [AS 3356] 4 msec 4 msec 4 msec 2 4.69.153.77 [AS 3356] [MPLS: Label 493184 Exp 0] 4 msec 4 msec 4 msec 3 ae-0-11.bar2.Toronto1.Level3.net (4.69.151.242) [AS 3356] [MPLS: Label 492736 Exp 0] 4 msec 8 msec 4 msec 4 ae-7-1.ebr2.NewYork1.Level3.net (4.69.151.114) [AS 3356] [MPLS: Label 1494 Exp 0] 24 msec 24 msec 20 msec 5 ae-6-6.ebr2.NewYork2.Level3.net (4.69.141.22) [AS 3356] [MPLS: Label 1343 Exp 0] 16 msec 20 msec 16 msec 6 ae-1-100.ebr1.NewYork2.Level3.net (4.69.135.253) [AS 3356] [MPLS: Label 1278 Exp 0] 20 msec 24 msec 24 msec 7 ae-3-3.ebr2.Washington1.Level3.net (4.69.132.89) [AS 3356] [MPLS: Label 1355 Exp 0] 24 msec 24 msec 20 msec 8 ae-82-82.csw3.Washington1.Level3.net (4.69.134.154) [AS 3356] [MPLS: Label 1452 Exp 0] 24 msec ae-72-72.csw2.Washington1.Level3.net (4.69.134.150) [AS 3356] [MPLS: Label 1390 Exp 0] 24 msec ae-62-62.csw1.Washington1.Level3.net (4.69.134.146) [AS 3356] [MPLS: Label 1388 Exp 0] 24 msec 9 ae-2-70.edge2.Washington4.Level3.net (4.69.149.80) [AS 3356] 24 msec 24 msec ae-1-60.edge2.Washington4.Level3.net (4.69.149.16) [AS 3356] 36 msec 10 TWITTER-INC.edge2.Washington4.Level3.net (4.53.114.102) [AS 3356] 20 msec 20 msec 20 msec 11 ae60.pao1-cr1.twttr.com (199.16.159.85) [AS 13414] 96 msec 92 msec 92 msec 12 * * * 13 r-199-59-150-39.twttr.com (199.59.150.39) [AS 13414] 92 msec 100 msec 100 msec edge-wprt-c6509#traceroute 199.59.150.39 source 66.102.70.233 Type escape sequence to abort. Tracing the route to r-199-59-150-39.twttr.com (199.59.150.39) 1 ge-6-1-117.car1.Toronto2.Level3.net (4.59.180.17) [AS 3356] 4 msec 4 msec 4 msec 2 4.69.153.77 [AS 3356] [MPLS: Label 493184 Exp 0] 28 msec 32 msec 4 msec 3 ae-0-11.bar2.Toronto1.Level3.net (4.69.151.242) [AS 3356] [MPLS: Label 492736 Exp 0] 16 msec 36 msec 4 msec 4 ae-7-1.ebr2.NewYork1.Level3.net (4.69.151.114) [AS 3356] [MPLS: Label 1494 Exp 0] 24 msec 24 msec 24 msec 5 ae-6-6.ebr2.NewYork2.Level3.net (4.69.141.22) [AS 3356] [MPLS: Label 1343 Exp 0] 20 msec 16 msec 20 msec 6 ae-1-100.ebr1.NewYork2.Level3.net (4.69.135.253) [AS 3356] [MPLS: Label 1278 Exp 0] 16 msec 20 msec 16 msec 7 ae-3-3.ebr2.Washington1.Level3.net (4.69.132.89) [AS 3356] [MPLS: Label 1355 Exp 0] 20 msec 24 msec 24 msec 8 ae-92-92.csw4.Washington1.Level3.net (4.69.134.158) [AS 3356] [MPLS: Label 1396 Exp 0] 28 msec ae-72-72.csw2.Washington1.Level3.net (4.69.134.150) [AS 3356] [MPLS: Label 1390 Exp 0] 28 msec 24 msec 9 ae-4-90.edge2.Washington4.Level3.net (4.69.149.208) [AS 3356] 24 msec ae-2-70.edge2.Washington4.Level3.net (4.69.149.80) [AS 3356] 92 msec ae-3-80.edge2.Washington4.Level3.net (4.69.149.144) [AS 3356] 24 msec 10 TWITTER-INC.edge2.Washington4.Level3.net (4.53.114.102) [AS 3356] 20 msec 20 msec 24 msec 11 ae60.pao1-cr1.twttr.com (199.16.159.85) [AS 13414] 92 msec 92 msec 92 msec 12 * * * 13 * * * 14 * * * 15 * edge-wprt-c6509#traceroute 199.59.150.39 Type escape sequence to abort. Tracing the route to r-199-59-150-39.twttr.com (199.59.150.39) 1 ge5-1-5.core1.toronto1.nexicom.net (76.75.127.145) [AS 11666] 0 msec 8 msec 4 msec 2 xe2-2-0-1.core1.toronto1.nexicom.net (98.124.49.233) [AS 32881] 4 msec 4 msec 4 msec 3 206.126.236.97 [AS 8218] 20 msec 20 msec 20 msec 4 ae60.pao1-cr1.twttr.com (199.16.159.85) [AS 13414] 80 msec 80 msec 88 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * *

1 0

Hostway Tampa
by Justin Scott 10 Jan '13

10 Jan '13

Something seems to be up at Hostway's facility in Tampa. We have servers there that suddenly went dark, tech support's Tampa phone number is giving a fast busy signal. I reached someone in Hostway sales that said they're aware of an issue in Tampa but their office hadn't been able to reach anyone in the Tampa NOC due to a phone problem, so no details or ETA.

1 2

Re: [outages] Twtelecom Menlo Park
by Phillip McKracken 09 Jan '13

09 Jan '13

> Anybody having BGP issues with twtelecom in Menlo Park? > > Thanks > Phillip > >

2 1

Level3 Downtown LA
by Carlos Alcantar 09 Jan '13

09 Jan '13

Anyone else seeing Level3 BGP session flapping in the downtown LA area? Started around 12:47AM PST Link flapped a couple times and now hard down since 1:02AM PST. Carlos Alcantar Race Communications / Race Team Member 1325 Howard Ave. #604, Burlingame, CA. 94010 Phone: +1 415 376 3314 / carlos(a)race.com / http://www.race.com

4 5

Windstream/Paetec Fiber cut in Springfield IL area
by Mitch 09 Jan '13

09 Jan '13

Just got confirmation from them of the fiber cut in the area. Any one else seeing services affected? Here is the email(with our circuit and ticket # removed) We are sending this email in regard to Paetec Repair Ticket #4871xxx on Circuit 0444xxxx. Information Update: We have confirmed that this service interuption is the result of a Fiber cut in Illinois. Technicians are currently onsite shooting the fiber. There is no ETTR as of this time. We will continue to provide updates as they become available. Thank you Paetec Repair

3 3