Oddly enough, the ckt came up a few hours ago - I'm wondering if they re-routed us. No notes in the ticket (we're getting the updates now... they had a bad email address). At 04:53 PM 30/07/2014, Luke Rockwell wrote:

7/30/14 8:27:19 PM GMT Repair work is ongoing at the site. Updates will be provided as they are received. 7/30/14 7:32:21 PM GMTField Services continue excavation efforts and have estimated splicing will begin in approximately one hour. 7/30/14 7:10:12 PM GMT Repair efforts are ongoing at the site. No estimated restoral time is currently available. 7/30/14 6:18:25 PM GMTField Services report excavation efforts are in progress to access the damaged cable. The splice crew is onsite to continue with repair following excavation. 7/30/14 5:58:09 PM GMT Repair crews continue en route to the site to proceed with restoration efforts. 7/30/14 5:18:26 PM GMTA 108-count fiber cut has been confirmed to have been caused by construction near the railway. All necessary repair crews are en route with no ETA available at this time. 7/30/14 4:22:00 PM GMT Field Services report that there is construction in the area that has likely caused the fiber cut. An excavation crew has been engaged to uncover the suspected damaged portion of the span. No ETA has been provided for the excavation crew. 7/30/14 4:03:40 PM GMT Field Services continue investigation in conjunction with railway workers to further isolate the suspected fiber cut. 7/30/14 3:41:00 PM GMTField Services report that the fault location is in close proximity to a railway. As such, Field Services is coordinating with railway workers to ensure safe access to continue investigation. Splice crews have been dispatched to the site and are expected to arrive shortly. 7/30/14 2:58:44 PM GMT Field Services arrived in Southfield, MI and detected a fault approximately 52 miles from the site. Field Services is proceeding en route to the fault location. 7/30/14 2:11:36 PM GMTThe Transport NOC detected a loss of signal in Dexter, MI is impacting unprotected services. A fiber cut is suspected and Field Services have been engaged for investigation. Updates will be provided as they are received.

..................................... Luke Rockwell Systems Administrator Information Technology

.....................................

Cal Alumni Association | UC Berkeley 1 Alumni House, Berkeley, CA 94720 TÂ 510.900.8196 FÂ 510.642.6252

.....................................

Commitment, Support, Passion

_____________________________________ <http://alumni.berkeley.edu/>alumni.berkeley.edu

_____________________________________ <https://www.facebook.com/CalAlumni>Facebook | <http://www.linkedin.com/groups?gid=70245>LinkedIn

On Wed, Jul 30, 2014 at 9:41 AM, Bill Wichers <<mailto:billw@waveform.net>billw@waveform.net> wrote:

This is affecting XO in Detroit too, I can see them on protect path due to higher than normal latency Detroit<->Chicago.

Â

 -Bill

Â

From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Luke Rockwell via Outages Sent: Wednesday, July 30, 2014 12:14 PM To: Clayton Zekelman Cc: <mailto:Outages@outages.org>Outages@outages.org Subject: Re: [outages] Level3 10G DWDM Ckt Detroit to Chicago

Â

Latest update from Level 3 trouble board

Â

7/30/14 4:03:40 PM GMT

Field Services continue investigation in conjunction with railway workers to further isolate the suspected fiber cut.

7/30/14 3:41:00 PM GMT

Field Services report that the fault location is in close proximity to a railway. As such, Field Services is coordinating with railway workers to ensure safe access to continue investigation. Splice crews have been dispatched to the site and are expected to arrive shortly.

7/30/14 2:58:44 PM GMT

Field Services arrived in Southfield, MI and detected a fault approximately 52 miles from the site. Field Services is proceeding en route to the fault location.

7/30/14 2:11:36 PM GMT

The Transport NOC detected a loss of signal in Dexter, MI is impacting unprotected services. A fiber cut is suspected and Field Services have been engaged for investigation. Updates will be provided as they are received.

.....................................

Luke Rockwell Systems Administrator Information Technology

.....................................

Cal Alumni Association | UC Berkeley 1 Alumni House, Berkeley, CA 94720 TÂ 510.900.8196 FÂ 510.642.6252

.....................................

Commitment, Support, Passion

_____________________________________

<http://alumni.berkeley.edu/>alumni.berkeley.edu

_____________________________________

<https://www.facebook.com/CalAlumni>Facebook | <http://www.linkedin.com/groups?gid=70245>LinkedIn

Â

On Wed, Jul 30, 2014 at 8:47 AM, Clayton Zekelman via Outages <<mailto:outages@outages.org>outages@outages.org> wrote:

Well, the same 10 Gig DWDM is down again - 10:53am. Â Level3 reports a fibre cut this time - near the same location as the card failure on the 19th...

At 11:38 AM 19/07/2014, Clayton Zekelman via Outages wrote:

Looks like our circuit came back up a few minutes ago.

We took a couple of brief hits on 2 1Gig DIAs out of Southfield, MI though - not sure if that was related.

At 06:48 AM 19/07/2014, Clayton Zekelman wrote:

Level3 10G DWDM Ckt from Detroit to Chicago currently out of service (down at 6:15am EDT).

Possibly related to some sort of transport failure in Dexter, MI that happened around the same time.

---

Â

---

Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4

tel. 519-985-8410 fax. 519-985-8409 Â Â Â Â _______________________________________________ Outages mailing list <mailto:Outages@outages.org>Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Â

--- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409