Here's the latest: *** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time. 11/20/2015 00:40:36 GMT - *** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time. 11/19/2015 23:47:36 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:43:49 GMT From CASE: 10003567 - Event Restoration efforts are still ongoing, an estimated restoral time has not been provided. 11/19/2015 23:05:21 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:00:53 GMT From CASE: 10003567 - Event Field Services have excavated three hand holes to reach the 108-count fiber cable for repair. Efforts are ongoing at the site with no estimated restoral time currently available. 11/19/2015 22:29:51 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 22:25:57 GMT From CASE: 10003567 - Event Field Services have accessed one hand hole and are working on accessing a second. Evaluation of the cable damage is underway. 11/19/2015 21:51:05 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 21:47:47 GMT From CASE: 10003567 - Event Field Services are now excavating to the hand hole with a backhoe. Once access is available, evaluation of the fault will continue. 11/19/2015 20:54:23 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:50:55 GMT From CASE: 10003567 - Event Field Services are manually digging to access the hand hole closest to the suspected damage point. Once accessed, Field Services will be able to better evaluate the issue. 11/19/2015 20:12:15 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:08:11 GMT From CASE: 10003567 - Event The plow work at the site has made isolation of the damage point difficult to pinpoint. Field Services are working to access hand holes in order to continue investigation and restoration efforts. 11/19/2015 19:08:37 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 19:04:13 GMT From CASE: 10003567 - Event Field Services are onsite in the construction area working to locate the exact damage point. Replacement cable is being dispatched to the site to have available for repair and should arrive by 20:00 GMT.
We are still seeing some issue from yesterday’s fiber cut on Level3. Some circuit’s never came back up and other went back down this morning. I am working with Level3 on this now. Wondering if anyone else is still having issues between Chicago and St Louis? From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Thom Cleary via Outages Sent: Thursday, November 19, 2015 7:01 PM To: frnkblk@iname.com; outages@outages.org Subject: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015 Here's the latest: *** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time. 11/20/2015 00:40:36 GMT - *** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time. 11/19/2015 23:47:36 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:43:49 GMT From CASE: 10003567 - Event Restoration efforts are still ongoing, an estimated restoral time has not been provided. 11/19/2015 23:05:21 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:00:53 GMT From CASE: 10003567 - Event Field Services have excavated three hand holes to reach the 108-count fiber cable for repair. Efforts are ongoing at the site with no estimated restoral time currently available. 11/19/2015 22:29:51 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 22:25:57 GMT From CASE: 10003567 - Event Field Services have accessed one hand hole and are working on accessing a second. Evaluation of the cable damage is underway. 11/19/2015 21:51:05 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 21:47:47 GMT From CASE: 10003567 - Event Field Services are now excavating to the hand hole with a backhoe. Once access is available, evaluation of the fault will continue. 11/19/2015 20:54:23 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:50:55 GMT From CASE: 10003567 - Event Field Services are manually digging to access the hand hole closest to the suspected damage point. Once accessed, Field Services will be able to better evaluate the issue. 11/19/2015 20:12:15 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:08:11 GMT From CASE: 10003567 - Event The plow work at the site has made isolation of the damage point difficult to pinpoint. Field Services are working to access hand holes in order to continue investigation and restoration efforts. 11/19/2015 19:08:37 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 19:04:13 GMT From CASE: 10003567 - Event Field Services are onsite in the construction area working to locate the exact damage point. Replacement cable is being dispatched to the site to have available for repair and should arrive by 20:00 GMT. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
As far as I can tell, the route we are using to Chicago is back up, but am having a surprising number of ping timeouts/latency within the level3 path. I know they de-prioritize ICMP, but this degree of it is just silly and makes trying to understand what's going on to cause the performance issues we are having much harder. traceroute 8.8.8.8: https://gist.github.com/terribleplan/902f15e4729511e9b4c1 On Fri, Nov 20, 2015 at 10:01 AM, Erik Sundberg via Outages < outages@outages.org> wrote:
We are still seeing some issue from yesterday’s fiber cut on Level3. Some circuit’s never came back up and other went back down this morning. I am working with Level3 on this now.
Wondering if anyone else is still having issues between Chicago and St Louis?
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Thom Cleary via Outages *Sent:* Thursday, November 19, 2015 7:01 PM *To:* frnkblk@iname.com; outages@outages.org *Subject:* [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015
Here's the latest:
*** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time.
11/20/2015 00:40:36 GMT - *** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time.
11/19/2015 23:47:36 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:43:49 GMT From CASE: 10003567 - Event Restoration efforts are still ongoing, an estimated restoral time has not been provided.
11/19/2015 23:05:21 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:00:53 GMT From CASE: 10003567 - Event Field Services have excavated three hand holes to reach the 108-count fiber cable for repair. Efforts are ongoing at the site with no estimated restoral time currently available.
11/19/2015 22:29:51 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 22:25:57 GMT From CASE: 10003567 - Event Field Services have accessed one hand hole and are working on accessing a second. Evaluation of the cable damage is underway.
11/19/2015 21:51:05 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 21:47:47 GMT From CASE: 10003567 - Event Field Services are now excavating to the hand hole with a backhoe. Once access is available, evaluation of the fault will continue.
11/19/2015 20:54:23 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:50:55 GMT From CASE: 10003567 - Event Field Services are manually digging to access the hand hole closest to the suspected damage point. Once accessed, Field Services will be able to better evaluate the issue.
11/19/2015 20:12:15 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:08:11 GMT From CASE: 10003567 - Event The plow work at the site has made isolation of the damage point difficult to pinpoint. Field Services are working to access hand holes in order to continue investigation and restoration efforts.
11/19/2015 19:08:37 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 19:04:13 GMT From CASE: 10003567 - Event Field Services are onsite in the construction area working to locate the exact damage point. Replacement cable is being dispatched to the site to have available for repair and should arrive by 20:00 GMT.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Level3 Just said there is a another fiber cut outside Dallas. From: Erik Sundberg Sent: Friday, November 20, 2015 10:01 AM To: 'Thom Cleary' <clearythom@gmail.com>; frnkblk@iname.com Cc: 'outages@outages.org' <outages@outages.org> Subject: RE: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015 We are still seeing some issue from yesterday’s fiber cut on Level3. Some circuit’s never came back up and other went back down this morning. I am working with Level3 on this now. Wondering if anyone else is still having issues between Chicago and St Louis? From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Thom Cleary via Outages Sent: Thursday, November 19, 2015 7:01 PM To: frnkblk@iname.com<mailto:frnkblk@iname.com>; outages@outages.org<mailto:outages@outages.org> Subject: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015 Here's the latest: *** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time. 11/20/2015 00:40:36 GMT - *** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time. 11/19/2015 23:47:36 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:43:49 GMT From CASE: 10003567 - Event Restoration efforts are still ongoing, an estimated restoral time has not been provided. 11/19/2015 23:05:21 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:00:53 GMT From CASE: 10003567 - Event Field Services have excavated three hand holes to reach the 108-count fiber cable for repair. Efforts are ongoing at the site with no estimated restoral time currently available. 11/19/2015 22:29:51 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 22:25:57 GMT From CASE: 10003567 - Event Field Services have accessed one hand hole and are working on accessing a second. Evaluation of the cable damage is underway. 11/19/2015 21:51:05 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 21:47:47 GMT From CASE: 10003567 - Event Field Services are now excavating to the hand hole with a backhoe. Once access is available, evaluation of the fault will continue. 11/19/2015 20:54:23 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:50:55 GMT From CASE: 10003567 - Event Field Services are manually digging to access the hand hole closest to the suspected damage point. Once accessed, Field Services will be able to better evaluate the issue. 11/19/2015 20:12:15 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:08:11 GMT From CASE: 10003567 - Event The plow work at the site has made isolation of the damage point difficult to pinpoint. Field Services are working to access hand holes in order to continue investigation and restoration efforts. 11/19/2015 19:08:37 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 19:04:13 GMT From CASE: 10003567 - Event Field Services are onsite in the construction area working to locate the exact damage point. Replacement cable is being dispatched to the site to have available for repair and should arrive by 20:00 GMT. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
I’m seeing loss at Dallas to Cleveland on level3. If Chicago is out of route because of a fiber cut guessing Cleveland is the redundant path but doesn’t have the capacity. Joseph From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Erik Sundberg via Outages Sent: Friday, November 20, 2015 10:18 AM To: Thom Cleary; frnkblk@iname.com Cc: outages@outages.org Subject: Re: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015 Level3 Just said there is a another fiber cut outside Dallas. From: Erik Sundberg Sent: Friday, November 20, 2015 10:01 AM To: 'Thom Cleary' <clearythom@gmail.com<mailto:clearythom@gmail.com>>; frnkblk@iname.com<mailto:frnkblk@iname.com> Cc: 'outages@outages.org' <outages@outages.org<mailto:outages@outages.org>> Subject: RE: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015 We are still seeing some issue from yesterday’s fiber cut on Level3. Some circuit’s never came back up and other went back down this morning. I am working with Level3 on this now. Wondering if anyone else is still having issues between Chicago and St Louis? From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Thom Cleary via Outages Sent: Thursday, November 19, 2015 7:01 PM To: frnkblk@iname.com<mailto:frnkblk@iname.com>; outages@outages.org<mailto:outages@outages.org> Subject: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015 Here's the latest: *** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time. 11/20/2015 00:40:36 GMT - *** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time. 11/19/2015 23:47:36 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:43:49 GMT From CASE: 10003567 - Event Restoration efforts are still ongoing, an estimated restoral time has not been provided. 11/19/2015 23:05:21 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:00:53 GMT From CASE: 10003567 - Event Field Services have excavated three hand holes to reach the 108-count fiber cable for repair. Efforts are ongoing at the site with no estimated restoral time currently available. 11/19/2015 22:29:51 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 22:25:57 GMT From CASE: 10003567 - Event Field Services have accessed one hand hole and are working on accessing a second. Evaluation of the cable damage is underway. 11/19/2015 21:51:05 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 21:47:47 GMT From CASE: 10003567 - Event Field Services are now excavating to the hand hole with a backhoe. Once access is available, evaluation of the fault will continue. 11/19/2015 20:54:23 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:50:55 GMT From CASE: 10003567 - Event Field Services are manually digging to access the hand hole closest to the suspected damage point. Once accessed, Field Services will be able to better evaluate the issue. 11/19/2015 20:12:15 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:08:11 GMT From CASE: 10003567 - Event The plow work at the site has made isolation of the damage point difficult to pinpoint. Field Services are working to access hand holes in order to continue investigation and restoration efforts. 11/19/2015 19:08:37 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 19:04:13 GMT From CASE: 10003567 - Event Field Services are onsite in the construction area working to locate the exact damage point. Replacement cable is being dispatched to the site to have available for repair and should arrive by 20:00 GMT. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
Level3 has another fiber cut today this time in Dallas, went down around 9:00Am. From: Joseph Jackson [mailto:jjackson@aninetworks.net] Sent: Friday, November 20, 2015 10:20 AM To: Erik Sundberg <ESundberg@nitelusa.com>; Thom Cleary <clearythom@gmail.com>; frnkblk@iname.com; outages@outages.org Subject: RE: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015 I’m seeing loss at Dallas to Cleveland on level3. If Chicago is out of route because of a fiber cut guessing Cleveland is the redundant path but doesn’t have the capacity. Joseph From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Erik Sundberg via Outages Sent: Friday, November 20, 2015 10:18 AM To: Thom Cleary; frnkblk@iname.com<mailto:frnkblk@iname.com> Cc: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015 Level3 Just said there is a another fiber cut outside Dallas. From: Erik Sundberg Sent: Friday, November 20, 2015 10:01 AM To: 'Thom Cleary' <clearythom@gmail.com<mailto:clearythom@gmail.com>>; frnkblk@iname.com<mailto:frnkblk@iname.com> Cc: 'outages@outages.org' <outages@outages.org<mailto:outages@outages.org>> Subject: RE: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015 We are still seeing some issue from yesterday’s fiber cut on Level3. Some circuit’s never came back up and other went back down this morning. I am working with Level3 on this now. Wondering if anyone else is still having issues between Chicago and St Louis? From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Thom Cleary via Outages Sent: Thursday, November 19, 2015 7:01 PM To: frnkblk@iname.com<mailto:frnkblk@iname.com>; outages@outages.org<mailto:outages@outages.org> Subject: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015 Here's the latest: *** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time. 11/20/2015 00:40:36 GMT - *** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time. 11/19/2015 23:47:36 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:43:49 GMT From CASE: 10003567 - Event Restoration efforts are still ongoing, an estimated restoral time has not been provided. 11/19/2015 23:05:21 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:00:53 GMT From CASE: 10003567 - Event Field Services have excavated three hand holes to reach the 108-count fiber cable for repair. Efforts are ongoing at the site with no estimated restoral time currently available. 11/19/2015 22:29:51 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 22:25:57 GMT From CASE: 10003567 - Event Field Services have accessed one hand hole and are working on accessing a second. Evaluation of the cable damage is underway. 11/19/2015 21:51:05 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 21:47:47 GMT From CASE: 10003567 - Event Field Services are now excavating to the hand hole with a backhoe. Once access is available, evaluation of the fault will continue. 11/19/2015 20:54:23 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:50:55 GMT From CASE: 10003567 - Event Field Services are manually digging to access the hand hole closest to the suspected damage point. Once accessed, Field Services will be able to better evaluate the issue. 11/19/2015 20:12:15 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:08:11 GMT From CASE: 10003567 - Event The plow work at the site has made isolation of the damage point difficult to pinpoint. Field Services are working to access hand holes in order to continue investigation and restoration efforts. 11/19/2015 19:08:37 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 19:04:13 GMT From CASE: 10003567 - Event Field Services are onsite in the construction area working to locate the exact damage point. Replacement cable is being dispatched to the site to have available for repair and should arrive by 20:00 GMT. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
Zayo reports in another ticket there's a fiber cut between Falls and Denton, TX. Level 3 fiber map shows fiber in that area. Guessing it's related On Nov 20, 2015 10:25 AM, "Erik Sundberg" <ESundberg@nitelusa.com> wrote:
Level3 has another fiber cut today this time in Dallas, went down around 9:00Am.
*From:* Joseph Jackson [mailto:jjackson@aninetworks.net] *Sent:* Friday, November 20, 2015 10:20 AM *To:* Erik Sundberg <ESundberg@nitelusa.com>; Thom Cleary < clearythom@gmail.com>; frnkblk@iname.com; outages@outages.org *Subject:* RE: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015
I’m seeing loss at Dallas to Cleveland on level3. If Chicago is out of route because of a fiber cut guessing Cleveland is the redundant path but doesn’t have the capacity.
Joseph
*From:* Outages [mailto:outages-bounces@outages.org <outages-bounces@outages.org>] *On Behalf Of *Erik Sundberg via Outages *Sent:* Friday, November 20, 2015 10:18 AM *To:* Thom Cleary; frnkblk@iname.com *Cc:* outages@outages.org *Subject:* Re: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015
Level3 Just said there is a another fiber cut outside Dallas.
*From:* Erik Sundberg *Sent:* Friday, November 20, 2015 10:01 AM *To:* 'Thom Cleary' <clearythom@gmail.com>; frnkblk@iname.com *Cc:* 'outages@outages.org' <outages@outages.org> *Subject:* RE: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015
We are still seeing some issue from yesterday’s fiber cut on Level3. Some circuit’s never came back up and other went back down this morning. I am working with Level3 on this now.
Wondering if anyone else is still having issues between Chicago and St Louis?
*From:* Outages [mailto:outages-bounces@outages.org <outages-bounces@outages.org>] *On Behalf Of *Thom Cleary via Outages *Sent:* Thursday, November 19, 2015 7:01 PM *To:* frnkblk@iname.com; outages@outages.org *Subject:* [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015
Here's the latest:
*** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time.
11/20/2015 00:40:36 GMT - *** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time.
11/19/2015 23:47:36 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:43:49 GMT From CASE: 10003567 - Event Restoration efforts are still ongoing, an estimated restoral time has not been provided.
11/19/2015 23:05:21 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:00:53 GMT From CASE: 10003567 - Event Field Services have excavated three hand holes to reach the 108-count fiber cable for repair. Efforts are ongoing at the site with no estimated restoral time currently available.
11/19/2015 22:29:51 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 22:25:57 GMT From CASE: 10003567 - Event Field Services have accessed one hand hole and are working on accessing a second. Evaluation of the cable damage is underway.
11/19/2015 21:51:05 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 21:47:47 GMT From CASE: 10003567 - Event Field Services are now excavating to the hand hole with a backhoe. Once access is available, evaluation of the fault will continue.
11/19/2015 20:54:23 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:50:55 GMT From CASE: 10003567 - Event Field Services are manually digging to access the hand hole closest to the suspected damage point. Once accessed, Field Services will be able to better evaluate the issue.
11/19/2015 20:12:15 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:08:11 GMT From CASE: 10003567 - Event The plow work at the site has made isolation of the damage point difficult to pinpoint. Field Services are working to access hand holes in order to continue investigation and restoration efforts.
11/19/2015 19:08:37 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 19:04:13 GMT From CASE: 10003567 - Event Field Services are onsite in the construction area working to locate the exact damage point. Replacement cable is being dispatched to the site to have available for repair and should arrive by 20:00 GMT.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
*** CASCADED EXTERNAL NOTES 20-Nov-2015 16:31:59 GMT From CASE: 10009886 - Event Field Services arrived onsite in Dallas, TX to proceed with investigations. Testing indicated an issue approximately 43 kilometers from the site. Field Services also reported they were contacted by a road construction crew in that area who advised they identified a damaged fiber cable. Field Services are en route to the fault location for further evaluation. An ETA of 17:00 GMT has been provided. On Nov 20, 2015 10:32 AM, "Thom Cleary" <clearythom@gmail.com> wrote:
Zayo reports in another ticket there's a fiber cut between Falls and Denton, TX. Level 3 fiber map shows fiber in that area. Guessing it's related On Nov 20, 2015 10:25 AM, "Erik Sundberg" <ESundberg@nitelusa.com> wrote:
Level3 has another fiber cut today this time in Dallas, went down around 9:00Am.
*From:* Joseph Jackson [mailto:jjackson@aninetworks.net] *Sent:* Friday, November 20, 2015 10:20 AM *To:* Erik Sundberg <ESundberg@nitelusa.com>; Thom Cleary < clearythom@gmail.com>; frnkblk@iname.com; outages@outages.org *Subject:* RE: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015
I’m seeing loss at Dallas to Cleveland on level3. If Chicago is out of route because of a fiber cut guessing Cleveland is the redundant path but doesn’t have the capacity.
Joseph
*From:* Outages [mailto:outages-bounces@outages.org <outages-bounces@outages.org>] *On Behalf Of *Erik Sundberg via Outages *Sent:* Friday, November 20, 2015 10:18 AM *To:* Thom Cleary; frnkblk@iname.com *Cc:* outages@outages.org *Subject:* Re: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015
Level3 Just said there is a another fiber cut outside Dallas.
*From:* Erik Sundberg *Sent:* Friday, November 20, 2015 10:01 AM *To:* 'Thom Cleary' <clearythom@gmail.com>; frnkblk@iname.com *Cc:* 'outages@outages.org' <outages@outages.org> *Subject:* RE: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015
We are still seeing some issue from yesterday’s fiber cut on Level3. Some circuit’s never came back up and other went back down this morning. I am working with Level3 on this now.
Wondering if anyone else is still having issues between Chicago and St Louis?
*From:* Outages [mailto:outages-bounces@outages.org <outages-bounces@outages.org>] *On Behalf Of *Thom Cleary via Outages *Sent:* Thursday, November 19, 2015 7:01 PM *To:* frnkblk@iname.com; outages@outages.org *Subject:* [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015
Here's the latest:
*** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time.
11/20/2015 00:40:36 GMT - *** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time.
11/19/2015 23:47:36 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:43:49 GMT From CASE: 10003567 - Event Restoration efforts are still ongoing, an estimated restoral time has not been provided.
11/19/2015 23:05:21 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:00:53 GMT From CASE: 10003567 - Event Field Services have excavated three hand holes to reach the 108-count fiber cable for repair. Efforts are ongoing at the site with no estimated restoral time currently available.
11/19/2015 22:29:51 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 22:25:57 GMT From CASE: 10003567 - Event Field Services have accessed one hand hole and are working on accessing a second. Evaluation of the cable damage is underway.
11/19/2015 21:51:05 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 21:47:47 GMT From CASE: 10003567 - Event Field Services are now excavating to the hand hole with a backhoe. Once access is available, evaluation of the fault will continue.
11/19/2015 20:54:23 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:50:55 GMT From CASE: 10003567 - Event Field Services are manually digging to access the hand hole closest to the suspected damage point. Once accessed, Field Services will be able to better evaluate the issue.
11/19/2015 20:12:15 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:08:11 GMT From CASE: 10003567 - Event The plow work at the site has made isolation of the damage point difficult to pinpoint. Field Services are working to access hand holes in order to continue investigation and restoration efforts.
11/19/2015 19:08:37 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 19:04:13 GMT From CASE: 10003567 - Event Field Services are onsite in the construction area working to locate the exact damage point. Replacement cable is being dispatched to the site to have available for repair and should arrive by 20:00 GMT.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
*** CASCADED EXTERNAL NOTES 20-Nov-2015 17:38:27 GMT From CASE: 10009886 - Event Field Services are at the fault site evaluating the fiber damage.
Latest *** CASCADED EXTERNAL NOTES 20-Nov-2015 18:06:17 GMT From CASE: 10009886 - Event The fiber cut has been identified, caused by drilling works during road construction. A lot of cable has become wound around an auger. Field Services are excavating the 96-count fiber cable for repair. Replacement cable has been delivered to the site. No estimated restoral time is currently available. From: Thom Cleary [mailto:clearythom@gmail.com] Sent: Friday, November 20, 2015 12:04 PM To: Erik Sundberg <ESundberg@nitelusa.com> Cc: Joseph Jackson <jjackson@aninetworks.net>; frnkblk@iname.com; outages@outages.org Subject: XO / Level 3 Outage | Chicago - St Louis | 11/19/2015 *** CASCADED EXTERNAL NOTES 20-Nov-2015 17:38:27 GMT From CASE: 10009886 - Event Field Services are at the fault site evaluating the fiber damage. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
*11/20/2015 8:54 PM GMT* Field Services continue with restoration efforts at the site. As more information is gathered about the extent of the damage, repair plans are being evaluated to determine the best method for restoral. Further details will be provided as they are received. *11/20/2015 7:57 PM GMT* Field Services have been able to reach the conduit, which was buried 12 feet deep. They have determined approximately 300 feet of fiber cable has been damaged by the auger. Field Services are currently accessing additional portions of the conduit and a manhole for the repair. *11/20/2015 7:03 PM GMT* Excavation efforts are continuing to access the fiber cable for repair. All crews and materials are onsite to proceed with restoration as expeditiously as possible. On Fri, Nov 20, 2015 at 12:42 PM, Erik Sundberg via Outages < outages@outages.org> wrote:
Latest
*** CASCADED EXTERNAL NOTES 20-Nov-2015 18:06:17 GMT From CASE: 10009886 - Event The fiber cut has been identified, caused by drilling works during road construction. A lot of cable has become wound around an auger. Field Services are excavating the 96-count fiber cable for repair. Replacement cable has been delivered to the site. No estimated restoral time is currently available.
*From:* Thom Cleary [mailto:clearythom@gmail.com] *Sent:* Friday, November 20, 2015 12:04 PM *To:* Erik Sundberg <ESundberg@nitelusa.com> *Cc:* Joseph Jackson <jjackson@aninetworks.net>; frnkblk@iname.com; outages@outages.org *Subject:* XO / Level 3 Outage | Chicago - St Louis | 11/19/2015
*** CASCADED EXTERNAL NOTES 20-Nov-2015 17:38:27 GMT From CASE: 10009886 - Event Field Services are at the fault site evaluating the fiber damage.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Master Ticket # is 10009886, current ETA is 1700 GMT per Level 3's portal Nick Colton Director of Network Operations ALLO Communications 610 Broadway - PO Box 1123 Imperial, NE 69033 308-633-7814 On Fri, Nov 20, 2015 at 9:25 AM, Erik Sundberg via Outages < outages@outages.org> wrote:
Level3 has another fiber cut today this time in Dallas, went down around 9:00Am.
*From:* Joseph Jackson [mailto:jjackson@aninetworks.net] *Sent:* Friday, November 20, 2015 10:20 AM *To:* Erik Sundberg <ESundberg@nitelusa.com>; Thom Cleary < clearythom@gmail.com>; frnkblk@iname.com; outages@outages.org
*Subject:* RE: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015
I’m seeing loss at Dallas to Cleveland on level3. If Chicago is out of route because of a fiber cut guessing Cleveland is the redundant path but doesn’t have the capacity.
Joseph
*From:* Outages [mailto:outages-bounces@outages.org <outages-bounces@outages.org>] *On Behalf Of *Erik Sundberg via Outages *Sent:* Friday, November 20, 2015 10:18 AM *To:* Thom Cleary; frnkblk@iname.com *Cc:* outages@outages.org *Subject:* Re: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015
Level3 Just said there is a another fiber cut outside Dallas.
*From:* Erik Sundberg *Sent:* Friday, November 20, 2015 10:01 AM *To:* 'Thom Cleary' <clearythom@gmail.com>; frnkblk@iname.com *Cc:* 'outages@outages.org' <outages@outages.org> *Subject:* RE: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015
We are still seeing some issue from yesterday’s fiber cut on Level3. Some circuit’s never came back up and other went back down this morning. I am working with Level3 on this now.
Wondering if anyone else is still having issues between Chicago and St Louis?
*From:* Outages [mailto:outages-bounces@outages.org <outages-bounces@outages.org>] *On Behalf Of *Thom Cleary via Outages *Sent:* Thursday, November 19, 2015 7:01 PM *To:* frnkblk@iname.com; outages@outages.org *Subject:* [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015
Here's the latest:
*** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time.
11/20/2015 00:40:36 GMT - *** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time.
11/19/2015 23:47:36 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:43:49 GMT From CASE: 10003567 - Event Restoration efforts are still ongoing, an estimated restoral time has not been provided.
11/19/2015 23:05:21 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:00:53 GMT From CASE: 10003567 - Event Field Services have excavated three hand holes to reach the 108-count fiber cable for repair. Efforts are ongoing at the site with no estimated restoral time currently available.
11/19/2015 22:29:51 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 22:25:57 GMT From CASE: 10003567 - Event Field Services have accessed one hand hole and are working on accessing a second. Evaluation of the cable damage is underway.
11/19/2015 21:51:05 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 21:47:47 GMT From CASE: 10003567 - Event Field Services are now excavating to the hand hole with a backhoe. Once access is available, evaluation of the fault will continue.
11/19/2015 20:54:23 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:50:55 GMT From CASE: 10003567 - Event Field Services are manually digging to access the hand hole closest to the suspected damage point. Once accessed, Field Services will be able to better evaluate the issue.
11/19/2015 20:12:15 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:08:11 GMT From CASE: 10003567 - Event The plow work at the site has made isolation of the damage point difficult to pinpoint. Field Services are working to access hand holes in order to continue investigation and restoration efforts.
11/19/2015 19:08:37 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 19:04:13 GMT From CASE: 10003567 - Event Field Services are onsite in the construction area working to locate the exact damage point. Replacement cable is being dispatched to the site to have available for repair and should arrive by 20:00 GMT.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (6)
-
Erik Sundberg -
Joseph Jackson -
Kegan Myers -
Nick Colton -
Ryan Gelobter -
Thom Cleary