There is a large fiber cut in Los Angeles affecting many providers at this time that's been going on since earlier today. I'm also being affected by this between LA and Dallas and seeing latency on other networks in the LA area as a result of congestion due to decreased capacity. -Vinny -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Anthony Ledesma Sent: Thursday, April 19, 2012 3:52 PM To: Martin Hepworth; outages@outages.org Subject: Re: [outages] Rightnow reporting several lines doen in bay area We have received a report from Zuora stating the following:

We have confirmed with our datacenter that there was a fiber line cut between Dallas and LA which is highly affecting Internet traffic.

We will continue to work with our datacenter and provide updates.  We will receive an estimate when it will be repaired in about 2 hours.  

We appreciate your patience with the Internet.

-- Anthony Ledesma | FireHost From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Martin Hepworth Sent: Thursday, April 19, 2012 12:43 PM To: outages@outages.org Subject: [outages] Rightnow reporting several lines doen in bay area Anyone got any further info, they are pointing the finger at their providers  --  Martin -- -- Martin Hepworth Oxford, UK _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

When did LA<->Dallas become part of the Bay Area? Owen On Apr 19, 2012, at 1:06 PM, <Vinny_Abello@dell.com> wrote:

There is a large fiber cut in Los Angeles affecting many providers at this time that's been going on since earlier today. I'm also being affected by this between LA and Dallas and seeing latency on other networks in the LA area as a result of congestion due to decreased capacity.

-Vinny

-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Anthony Ledesma Sent: Thursday, April 19, 2012 3:52 PM To: Martin Hepworth; outages@outages.org Subject: Re: [outages] Rightnow reporting several lines doen in bay area

We have received a report from Zuora stating the following:

...

We have confirmed with our datacenter that there was a fiber line cut between Dallas and LA which is highly affecting Internet traffic.

We will continue to work with our datacenter and provide updates. We will receive an estimate when it will be repaired in about 2 hours.

We appreciate your patience with the Internet.

-- Anthony Ledesma | FireHost

From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Martin Hepworth Sent: Thursday, April 19, 2012 12:43 PM To: outages@outages.org Subject: [outages] Rightnow reporting several lines doen in bay area

Anyone got any further info, they are pointing the finger at their providers

-- Martin

-- -- Martin Hepworth Oxford, UK

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Sounds like based on time, this could have been a cause of the major cBeyond issues this morning too. Blake Pfankuch From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Martin Hepworth Sent: Thursday, April 19, 2012 2:27 PM To: Owen DeLong Cc: outages@outages.org Subject: Re: [outages] Rightnow reporting several lines doen in bay area Looking at another report here looks like a fair amount if the western usa is affected by this -- Martin On Thursday, 19 April 2012, Owen DeLong wrote: When did LA<->Dallas become part of the Bay Area? Owen On Apr 19, 2012, at 1:06 PM, <Vinny_Abello@dell.com<javascript:;>> wrote:

There is a large fiber cut in Los Angeles affecting many providers at this time that's been going on since earlier today. I'm also being affected by this between LA and Dallas and seeing latency on other networks in the LA area as a result of congestion due to decreased capacity.

-Vinny

-----Original Message----- From: outages-bounces@outages.org<javascript:;> [mailto:outages-bounces@outages.org<javascript:;>] On Behalf Of Anthony Ledesma Sent: Thursday, April 19, 2012 3:52 PM To: Martin Hepworth; outages@outages.org<javascript:;> Subject: Re: [outages] Rightnow reporting several lines doen in bay area

We have received a report from Zuora stating the following:

...

We have confirmed with our datacenter that there was a fiber line cut between Dallas and LA which is highly affecting Internet traffic.

We will continue to work with our datacenter and provide updates. We will receive an estimate when it will be repaired in about 2 hours.

We appreciate your patience with the Internet.

-- Anthony Ledesma | FireHost

From: outages-bounces@outages.org<javascript:;> [mailto:outages-bounces@outages.org<javascript:;>] On Behalf Of Martin Hepworth Sent: Thursday, April 19, 2012 12:43 PM To: outages@outages.org<javascript:;> Subject: [outages] Rightnow reporting several lines doen in bay area

Anyone got any further info, they are pointing the finger at their providers

-- Martin

-- -- Martin Hepworth Oxford, UK

_______________________________________________ Outages mailing list Outages@outages.org<javascript:;> https://puck.nether.net/mailman/listinfo/outages

_______________________________________________ Outages mailing list Outages@outages.org<javascript:;> https://puck.nether.net/mailman/listinfo/outages

-- -- Martin Hepworth Oxford, UK

Here is what I see on Level 3's ticketing system... Network Event Summary: A partial fiber cut in Tustin, CA impacted transport unprotected, IP and Voice traffic. 911 calls were not impacted. Event Ticket ID: 5539336 Market Area Affected: Tustin, CA Ticket Create Date: 4/19/12 3:22:48 PM GMT Impacted For: 12 hours 52 minutes Event Status: Restored Resolve Date: 4/20/12 3:55:14 PM GMT Time Since Last Update: 15 hours 5 minutes 4/20/12 6:00:36 AM GMT Technical service center confirm all customer services have been restored. 4/20/12 5:30:05 AM GMT Transport NOC has completed system roll and all work has been completed. Technical service center is confirming service restoral at this time. 4/20/12 5:06:37 AM GMT The fiber repair crew has advised they have closed all tubes as splicing is completed. Transport NOC is seeing some alarms on systems they report are operating outside of span. The Transport NOC is working with the field technician to roll those systems back to original fibers now that they have completed splicing. Technical service center is awaiting Transport NOC to complete roll before they can verify service restoral with the customers. 4/20/12 4:06:23 AM GMT The fiber repair team report all splicing has been completed. Technical service center is confirming service restoral with the customer at this time. 4/20/12 3:42:14 AM GMT The fiber repair team advised that splicing thru 132 is now complete and they only have two more tubes remaining to splice. 4/20/12 3:02:55 AM GMT The fiber repair team advised that splicing in the south pit is complete. The north pit is complete thru 72 and crews are working on 73 thru 96 at this time. 4/20/12 2:07:47 AM GMT The fiber repair team has advised a second splice crew has arrived to help expedite the fiber repair. 4/20/12 1:28:50 AM GMT The fiber repair team has advised that they have completed splicing on over 48 fibers. The fiber restoration process is ongoing at this time and continues to be on schedule. 4/20/12 12:38:26 AM GMT The Transport NOC has advised that the fiber repair team has begun splicing onsite. The Transport NOC has confirmed that the fiber repair team has spliced 24 fibers, which has restored services to some customers. The technicians successfully re-routed traffic and restored services to as many customers as possible. The splicing process is on schedule at this time. 4/19/12 11:33:04 PM GMT The fiber repair team has advised that the fiber restoration strategy is to repair the fiber buffer tube by buffer tube in order. The fiber repair team is prepping the fiber for repair at this time and has updated the ETTR from 03:00 GMT to 06:00 GMT. The Transport NOC is investigating the possibility of re-routing customer traffic to expedite service restoration. 4/19/12 10:31:07 PM GMT The Transport NOC has advised that some customer whose services were supposed to restore when traffic was rolled to an alternate path have been confirmed as still down. The Transport NOC has advised the fiber repair team to begin the fiber repair to expedite the restoration of services for all of the affected customers. The fiber repair team has provided and ETR of 4 Hours. 4/19/12 9:26:18 PM GMT The Transport NOC advised that all systems have been rolled to an alternate path with the exception of some Dark Fiber customers. The Transport NOC advised that the Dark Fiber customers that are up could experience an additional impact to services once the fiber repair commences. The Technical Service Center is compiling a list of Dark Fiber customers that could be impacted so that they can be notified of the situation. The Splicing crew has the damaged fibers excavated and is in the process of placing new cable in the fiber ducks that were not damaged. 4/19/12 8:10:45 PM GMT The Transport NOC advised that protected traffic was restored as of 19:18 GMT. The Global Field Service technicians continue to work to roll the remaining impacted traffic to an alternate path. The splicing crew is onsite still awaiting the arrival of the digging equipment; however, no ETA was available at this time. 4/19/12 7:05:50 PM GMT The Transport NOC advises that three systems have been rolled and alarms have begun to clear at this time. Global Field Services is in the process of rolling the fourth system and all six systems should be rolled and restored by 20:00 GMT; the Transport NOC continues to monitor alarms. Global Field Services has reported that underground crews are onsite at the cut site and digging equipment should arrive to the site at approximately 19:45 GMT. The Technical Service Center reported that protected customer services are being impacted due to this issue. The Transport NOC confirmed this and has upgraded the event to a transport protected outage. 4/19/12 6:09:51 PM GMT The Transport NOC advised that the Global Field Service technicians will commence to roll traffic to an alternate route in approximately 15-minutes. The fiber splicing crew is expected to be onsite at the damage location in approximately 30-minutes. No estimated time to restore is available at this time. 4/19/12 5:08:34 PM GMT The Transport NOC advises that splice crews are en-route to the site at this time with an ETA of 17:45 GMT. Global Field Service Technicians are in the process of rolling services to spare fibers as temporary repair and advise that services should be migrated and restored shortly. The Transport NOC is monitoring alarms and will advise when all associated alarms have cleared and services are restored. 4/19/12 4:39:07 PM GMT The Transport NOC advises that the Field Technician arrived to the site and shot the fiber, and is seeing a partial fiber cut at approximately 42 km from the Tustin, CA site. Global Field Services advises that construction crews have been engaged and are en-route to the site with an ETA of 17:00 GMT. Splice crews have been engaged to dispatch to the site for investigation; no ETA has is available at this time. A Field Technician has reported that the partial cut appears to have been caused by a billboard crew who was boring in the area. The Transport NOC states that roll options are also being investigated at this time as there are 3 fibers remaining in the 156-count fiber cable to use to roll high-priority services too. No estimated time to complete the service roll has been provided. The Transport NOC confirms that IP and Voice services are also being impacted due to this issue and has upgraded this event to critical level. No estimated time to restore is available. 4/19/12 3:25:02 PM GMT The Transport NOC responded to loss of signal alarms, indicating a fiber cut in the Tustin, CA area, which is impacting unprotected traffic. Field Services have been engaged to dispatch to the site to shoot the fibers for further isolation. Estimated time of arrival to the site is 60 minutes. Regards, Tim From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Booher, Mike Sent: Thursday, April 19, 2012 4:48 PM To: outages@outages.org Subject: Re: [outages] Rightnow reporting several lines doen in bay area partial fiber cut at approximately 42 km from the Tustin, CA site. It has been reported that the partial cut appears to have been caused by a billboard crew who was boring in the area. The splicing crew is onsite still awaiting the arrival of the digging equipment; however, no ETA was available at this time Thanks! Mike From: outages-bounces@outages.org<mailto:outages-bounces@outages.org> [mailto:outages-bounces@outages.org]<mailto:[mailto:outages-bounces@outages.org]> On Behalf Of Blake Pfankuch Sent: Thursday, April 19, 2012 16:32 To: Martin Hepworth; Owen DeLong Cc: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Rightnow reporting several lines doen in bay area Sounds like based on time, this could have been a cause of the major cBeyond issues this morning too. Blake Pfankuch From: outages-bounces@outages.org<mailto:outages-bounces@outages.org> [mailto:outages-bounces@outages.org]<mailto:[mailto:outages-bounces@outages.org]> On Behalf Of Martin Hepworth Sent: Thursday, April 19, 2012 2:27 PM To: Owen DeLong Cc: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Rightnow reporting several lines doen in bay area Looking at another report here looks like a fair amount if the western usa is affected by this -- Martin On Thursday, 19 April 2012, Owen DeLong wrote: When did LA<->Dallas become part of the Bay Area? Owen On Apr 19, 2012, at 1:06 PM, <Vinny_Abello@dell.com<javascript:;>> wrote:

There is a large fiber cut in Los Angeles affecting many providers at this time that's been going on since earlier today. I'm also being affected by this between LA and Dallas and seeing latency on other networks in the LA area as a result of congestion due to decreased capacity.

-Vinny

-----Original Message----- From: outages-bounces@outages.org<javascript:;> [mailto:outages-bounces@outages.org<javascript:;>] On Behalf Of Anthony Ledesma Sent: Thursday, April 19, 2012 3:52 PM To: Martin Hepworth; outages@outages.org<javascript:;> Subject: Re: [outages] Rightnow reporting several lines doen in bay area

We have received a report from Zuora stating the following:

...

We have confirmed with our datacenter that there was a fiber line cut between Dallas and LA which is highly affecting Internet traffic.

We will continue to work with our datacenter and provide updates. We will receive an estimate when it will be repaired in about 2 hours.

We appreciate your patience with the Internet.

-- Anthony Ledesma | FireHost

From: outages-bounces@outages.org<javascript:;> [mailto:outages-bounces@outages.org<javascript:;>] On Behalf Of Martin Hepworth Sent: Thursday, April 19, 2012 12:43 PM To: outages@outages.org<javascript:;> Subject: [outages] Rightnow reporting several lines doen in bay area

Anyone got any further info, they are pointing the finger at their providers

-- Martin

-- -- Martin Hepworth Oxford, UK

_______________________________________________ Outages mailing list Outages@outages.org<javascript:;> https://puck.nether.net/mailman/listinfo/outages

_______________________________________________ Outages mailing list Outages@outages.org<javascript:;> https://puck.nether.net/mailman/listinfo/outages

-- -- Martin Hepworth Oxford, UK

On Thu, Apr 19, 2012 at 01:37:55PM -0700, Seth Mattinen wrote:

On 4/19/12 1:16 PM, Owen DeLong wrote:

...

When did LA<->Dallas become part of the Bay Area?

Possibly when a protect path starts being used.

That wouldn't explain why things are *still* down. The whole idea behind a protected path is that it's a failover path; primary path goes down, protected path takes over. Depending on the medium there's a few milliseconds of downtime (talking about SONET rings here). So this would make me ask: why are things still down? If a protected path was already in use (re: fibre cut), then that would make me ask: why hasn't anyone failed back to the primary path? I'm inclined to think there's a single carrier involved here, and possibly who lacks redundancy (i.e. both paths are shoved through the same conduit/fibre bundle which has been cut). There's no point in bothering with redundancy if there's a SPoF anywhere within the topology. But like everything else, this is all speculation. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |