April 2011 - Outages - Outages Lists

Amazon EC2 outage AfterAction Report
by Jay Ashworth 29 Apr '11

29 Apr '11

http://aws.amazon.com/message/65648/ Please redirect discissions to -discuss, would you? Embarassingly, I can't make my email client (Zimbra6) add that header, that I know of. Cheers, -- jra

1 0

Potential Level 3 Issues
by Preston Parcell 29 Apr '11

29 Apr '11

Is anyone else seeing packet loss through Level 3 in the Dallas or Denver region? Preston

2 2

XO problems in Metro DC?
by Eric Van Tol 28 Apr '11

28 Apr '11

Anyone seeing any problems with XO transport in DC? We had a bunch of XO EoC circuits become unreachable, both through direct with XO and another provider who utilizes XO transport between COs. -evt

3 2

Re: [outages] Fwd: Update - Hibernia Atlantic TT#31978 - Update 11
by Nick D'Attilo 28 Apr '11

28 Apr '11

We have a protected circuit on there, been getting the same updates Seems pretty bad. Sent from my Android phone using TouchDown (www.nitrodesk.com) -----Original Message----- From: Kevin Blackham [blackham(a)gmail.com] Received: Wednesday, 27 Apr 2011, 20:31 To: outages(a)outages.org [outages(a)outages.org] Subject: [outages] Fwd: Update - Hibernia Atlantic TT#31978 - Update 11 Haven't seen any chatter on this messy cut. Thought I'd share for once. This is only affecting an unprotected path of mine. No indication that protecteds have failed. Kevin Blackham VMware/Mozy Begin forwarded message: From: Hibernia Atlantic NOC <HiberniaAtlanticNOC(a)hiberniaatlantic.com<mailto:HiberniaAtlanticNOC@hiberniaatlantic.com>> Date: April 27, 2011 16:23:40 PDT Subject: Update - Hibernia Atlantic TT#31978 - Update 11 Hibernia Atlantic TT#31978 Update 1: A fibre cut has been confirmed on the Hibernia UK East span between Southport and Bracewell. A team of engineers is en route to Bracewell to conduct OTDR testing to isolate the exact location of the break. Fibre crews are in the process of being activated. Update 2: No update is available at this time. Update 3: Fibre break has been isolated to 48km outside of the Hibernia Southport cable landing station (in the vicinity of Preston). No ETA is available as yet for fibre crews to reach site. Update 4: Repair representative due on site to carry out visual inspection at 15:50. This engineer will be joined by a civils and fibre repair team who are currently en route. Update 5: Civil and repair teams are expected onsite at 17:00 GMT. Update 6: Repair teams on site have found no obvious sources of damage to the ducting. They are now entering splice chambers to test in conjunction with the technician in Bracewell in order to isolate the exact location of the damage. Update 7: No obvious damage found in the area of the suspected break location (11.2Km from Bracewell), Emergency Response team are now going to enter a splice pit and test back towards Bracewell to pinpoint the exact location of the damage. A spare 4Km cable drum is on it's way from Liverpool, no ETA as yet. Update 8: Fibre break has been located to the junction of the A59 and Sawley Lane, near Sawley Village. Damage is extensive and appears to have been caused by the installation of new crash barriers. We are now formulating a repair plan with our Engineering team – please stand by for an update. Update 9: Due to the extent of the damage Hibernia are unable to execute a temporary repair. A new 4Km cable will be pulled in between the two nearest joints, prepped and then spliced to the existing cable. The new cable, spare ducting and the remainder of the emergency response team are due on site imminently. There is no ETTR as yet. The repair will be permanent. Update 10: Engineering team are now at the break location along with spare cable and ducting, engineers have accessed the pits at either end of the 4Km stretch and are now locating intermediate pits to enable the new cable to be pulled through. Update 11: After further analysis of the fibre route by the ground crew, it has become evident that a permanent repair / cable installation will be impossible tonight. There are major health and safety concerns such as working in very close proximity to a dual carriage way and poor light. Civils are now digging down at the break location with a view to pulling in slack from both ends and creating a temporary joint to get services back up. No ETTR as yet. Please be advised that Hibernia Atlantic is currently observing alarms on our UK network. We are currently investigating this outage and will provide an update shortly.

2 2

Fwd: Update - Hibernia Atlantic TT#31978 - Update 11
by Kevin Blackham 27 Apr '11

27 Apr '11

Haven't seen any chatter on this messy cut. Thought I'd share for once. This is only affecting an unprotected path of mine. No indication that protecteds have failed. Kevin Blackham VMware/Mozy Begin forwarded message: > From: Hibernia Atlantic NOC <HiberniaAtlanticNOC(a)hiberniaatlantic.com> > Date: April 27, 2011 16:23:40 PDT > > Subject: Update - Hibernia Atlantic TT#31978 - Update 11 > > Hibernia Atlantic TT#31978 > > Update 1: A fibre cut has been confirmed on the Hibernia UK East span between Southport and Bracewell. A team of engineers is en route to Bracewell to conduct OTDR testing to isolate the exact location of the break. Fibre crews are in the process of being activated. > > Update 2: No update is available at this time. > > Update 3: Fibre break has been isolated to 48km outside of the Hibernia Southport cable landing station (in the vicinity of Preston). No ETA is available as yet for fibre crews to reach site. > > Update 4: Repair representative due on site to carry out visual inspection at 15:50. This engineer will be joined by a civils and fibre repair team who are currently en route. > > Update 5: Civil and repair teams are expected onsite at 17:00 GMT. > > Update 6: Repair teams on site have found no obvious sources of damage to the ducting. They are now entering splice chambers to test in conjunction with the technician in Bracewell in order to isolate the exact location of the damage. > > Update 7: No obvious damage found in the area of the suspected break location (11.2Km from Bracewell), Emergency Response team are now going to enter a splice pit and test back towards Bracewell to pinpoint the exact location of the damage. A spare 4Km cable drum is on it's way from Liverpool, no ETA as yet. > > Update 8: Fibre break has been located to the junction of the A59 and Sawley Lane, near Sawley Village. Damage is extensive and appears to have been caused by the installation of new crash barriers. > We are now formulating a repair plan with our Engineering team – please stand by for an update. > > Update 9: Due to the extent of the damage Hibernia are unable to execute a temporary repair. A new 4Km cable will be pulled in between the two nearest joints, prepped and then spliced to the existing cable. The new cable, spare ducting and the remainder of the emergency response team are due on site imminently. There is no ETTR as yet. The repair will be permanent. > > Update 10: Engineering team are now at the break location along with spare cable and ducting, engineers have accessed the pits at either end of the 4Km stretch and are now locating intermediate pits to enable the new cable to be pulled through. > > Update 11: After further analysis of the fibre route by the ground crew, it has become evident that a permanent repair / cable installation will be impossible tonight. There are major health and safety concerns such as working in very close proximity to a dual carriage way and poor light. Civils are now digging down at the break location with a view to pulling in slack from both ends and creating a temporary joint to get services back up. No ETTR as yet. > > > Please be advised that Hibernia Atlantic is currently observing alarms on our UK network. > > We are currently investigating this outage and will provide an update shortly.

1 0

Fwd: Update - Hibernia Atlantic TT#31978 - Update 11
by Kevin Blackham 27 Apr '11

27 Apr '11

Thought I might contribute since I haven't seen any chatter on this messy cut. Kevin Blackham Mozy/VMware Begin forwarded message: > From: Hibernia Atlantic NOC <HiberniaAtlanticNOC(a)hiberniaatlantic.com> > Date: April 27, 2011 16:23:40 PDT > > Update 1: A fibre cut has been confirmed on the Hibernia UK East span between Southport and Bracewell. A team of engineers is en route to Bracewell to conduct OTDR testing to isolate the exact location of the break. Fibre crews are in the process of being activated. > > Update 2: No update is available at this time. > > Update 3: Fibre break has been isolated to 48km outside of the Hibernia Southport cable landing station (in the vicinity of Preston). No ETA is available as yet for fibre crews to reach site. > > Update 4: Repair representative due on site to carry out visual inspection at 15:50. This engineer will be joined by a civils and fibre repair team who are currently en route. > > Update 5: Civil and repair teams are expected onsite at 17:00 GMT. > > Update 6: Repair teams on site have found no obvious sources of damage to the ducting. They are now entering splice chambers to test in conjunction with the technician in Bracewell in order to isolate the exact location of the damage. > > Update 7: No obvious damage found in the area of the suspected break location (11.2Km from Bracewell), Emergency Response team are now going to enter a splice pit and test back towards Bracewell to pinpoint the exact location of the damage. A spare 4Km cable drum is on it's way from Liverpool, no ETA as yet. > > Update 8: Fibre break has been located to the junction of the A59 and Sawley Lane, near Sawley Village. Damage is extensive and appears to have been caused by the installation of new crash barriers. > We are now formulating a repair plan with our Engineering team – please stand by for an update. > > Update 9: Due to the extent of the damage Hibernia are unable to execute a temporary repair. A new 4Km cable will be pulled in between the two nearest joints, prepped and then spliced to the existing cable. The new cable, spare ducting and the remainder of the emergency response team are due on site imminently. There is no ETTR as yet. The repair will be permanent. > > Update 10: Engineering team are now at the break location along with spare cable and ducting, engineers have accessed the pits at either end of the 4Km stretch and are now locating intermediate pits to enable the new cable to be pulled through. > > Update 11: After further analysis of the fibre route by the ground crew, it has become evident that a permanent repair / cable installation will be impossible tonight. There are major health and safety concerns such as working in very close proximity to a dual carriage way and poor light. Civils are now digging down at the break location with a view to pulling in slack from both ends and creating a temporary joint to get services back up. No ETTR as yet. > > > Please be advised that Hibernia Atlantic is currently observing alarms on our UK network. > > We are currently investigating this outage and will provide an update shortly.

1 0

Verizon Outage Ashburn Va??
by John Mason Jr 21 Apr '11

21 Apr '11

T1 circuits down hard 3 days now, with no ETR yet. Wondering if other in the area (Beaumeade Circle)might be experiencing any issues Thanks John

2 1

Amazon EC2 outages in Virginia
by George Herbert 21 Apr '11

21 Apr '11

There are widespread reports going back to this morning of intermittent slowdowns and outages with EC2 services in the eastern datacenter. All over the press, but if you happen not to be watching and seeing it, here's an advisory. -- -george william herbert george.herbert(a)gmail.com

3 2

Surprised No Press Here - Playstation Down
by Scott, Robert D. 21 Apr '11

21 Apr '11

http://www.pcmag.com/article2/0,2817,2383924,00.asp Sony PlayStation Network Down for a 'Day or Two' Robert D. Scott Robert(a)ufl.edu Senior Network Engineer 352-273-0113 Phone CNS - Network Services 352-392-2061 CNS Phone Tree University of Florida 352-273-0743 FAX Florida Lambda Rail 352-294-3571 FLR NOC Gainesville, FL 32611 321-663-0421 Cell 3216630421(a)messaging.sprintpcs.com

4 3

XO IP Flex (and more?) outage in Chicago
by Bill Weiss 20 Apr '11

20 Apr '11

XO just confirmed for us that there's an outage in Chicago that affects at least our PRI over IP Flex. I didn't get a ticket number, but they said it's a known problem and may be fixed within the hour. -- Bill Weiss

1 1