Centurylink Fiber cut in Phoenix
We have P2P link between Dallas and San Diego that went down. It is also impacting our connectivity to Dallas over our other CL links. Centurylink sent out this notification. Sending it out in case others are having similar issues. 07/17/2018 18:31:54 GMT - *** CASCADED EXTERNAL NOTES 17-Jul-2018 18:30:14 GMT From CASE: 14896924 - Event Field Operations have identified a 96-count buried fiber cable that has been cut by boring activities occurring in the area. The cable is buried 14 feet deep and is within 25 feet of a railroad line. As such, the railroad line provider will not allow excavations activities to commence until all necessary shoring supplies are on site. The supplies are being obtained at this time, however, the ETA to the site is not yet known. Once the shoring is in place, the excavation team will begin digging a trench to access the damaged fiber and perform the repairs. The trench will need to be approximately 50 feet in length and at least 15 feet deep to access the fiber and raise it for splicing repairs to commence. The initial activities are expected to take approximately two hours. As such the next update will be relayed at 20:30 GMT or as additional information becomes available. This issue has been escalated internally to the Executive level proactively. 07/17/2018 17:38:50 GMT - *** CASCADED EXTERNAL NOTES 17-Jul-2018 17:36:55 GMT From CASE: 14896924 - Event Field Operations and the OSPE team have arrived at the site and have identified a location where they believe a buried fiber cable has been damaged by construction activities occurring in the area. Additional assessments are ongoing at this time. A fiber splicing repair team has been engaged to assist with the repairs once the damaged fiber has been identified, however the ETA to the site is not yet known 07/17/2018 16:42:52 GMT - *** CASCADED EXTERNAL NOTES 17-Jul-2018 16:40:40 GMT From CASE: 14896924 - Event Field Operations have performed light measurement readings from the CenturyLink Gateway in Phoenix and have identified the fault condition 28.43 miles from the site. The Outside Plant Engineer (OSPE) has been engaged to assist with further isolation. Thanks, Bobin Joseph
Thanks, Joe. Below is the most recent update we received a bit ago: *** CASCADED EXTERNAL NOTES 17-Jul-2018 19:50:18 GMT - Event Field Operations have reported that the Construction team is en route to the site with backhoes and the shoring supplies. An ETA of 20:00 GMT has been provided. In addition, local utility providers are currently on site marking existing utility lines in the area to prevent impacts during the repairs. Once on site, the Construction team will commence with installing the necessary shoring around the damage site. It is estimated that the shoring activities will be completed at 23:00 GMT and excavation activities will commence at that time. The Transport NOC has reported that there are currently no reroute opportunities. The next update will be relayed once the shoring has been completed or additional information becomes available. Regards, Cory On Tue, Jul 17, 2018, 1:39 PM Bobin Joseph via Outages <outages@outages.org> wrote:
We have P2P link between Dallas and San Diego that went down. It is also impacting our connectivity to Dallas over our other CL links. Centurylink sent out this notification. Sending it out in case others are having similar issues.
07/17/2018 18:31:54 GMT -
*** CASCADED EXTERNAL NOTES 17-Jul-2018 18:30:14 GMT From CASE: 14896924 - Event Field Operations have identified a 96-count buried fiber cable that has been cut by boring activities occurring in the area. The cable is buried 14 feet deep and is within 25 feet of a railroad line. As such, the railroad line provider will not allow excavations activities to commence until all necessary shoring supplies are on site. The supplies are being obtained at this time, however, the ETA to the site is not yet known. Once the shoring is in place, the excavation team will begin digging a trench to access the damaged fiber and perform the repairs. The trench will need to be approximately 50 feet in length and at least 15 feet deep to access the fiber and raise it for splicing repairs to commence. The initial activities are expected to take approximately two hours. As such the next update will be relayed at 20:30 GMT or as additional information becomes available. This issue has been escalated internally to the Executive level proactively.
07/17/2018 17:38:50 GMT -
*** CASCADED EXTERNAL NOTES 17-Jul-2018 17:36:55 GMT From CASE: 14896924 - Event
Field Operations and the OSPE team have arrived at the site and have identified a location where they believe a buried fiber cable has been damaged by construction activities occurring in the area. Additional assessments are ongoing at this time. A fiber splicing repair team has been engaged to assist with the repairs once the damaged fiber has been identified, however the ETA to the site is not yet known
07/17/2018 16:42:52 GMT -
*** CASCADED EXTERNAL NOTES 17-Jul-2018 16:40:40 GMT From CASE: 14896924 - Event
Field Operations have performed light measurement readings from the CenturyLink Gateway in Phoenix and have identified the fault condition 28.43 miles from the site. The Outside Plant Engineer (OSPE) has been engaged to assist with further isolation.
Thanks,
Bobin Joseph
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Curious if anyone has any further updates on this fiber-cut.. I appreciate it On Tue, Jul 17, 2018 at 3:57 PM, Cory Sell via Outages <outages@outages.org> wrote:
Thanks, Joe. Below is the most recent update we received a bit ago:
*** CASCADED EXTERNAL NOTES 17-Jul-2018 19:50:18 GMT - Event
Field Operations have reported that the Construction team is en route to the site with backhoes and the shoring supplies. An ETA of 20:00 GMT has been provided. In addition, local utility providers are currently on site marking existing utility lines in the area to prevent impacts during the repairs. Once on site, the Construction team will commence with installing the necessary shoring around the damage site. It is estimated that the shoring activities will be completed at 23:00 GMT and excavation activities will commence at that time. The Transport NOC has reported that there are currently no reroute opportunities. The next update will be relayed once the shoring has been completed or additional information becomes available.
Regards,
Cory
On Tue, Jul 17, 2018, 1:39 PM Bobin Joseph via Outages < outages@outages.org> wrote:
We have P2P link between Dallas and San Diego that went down. It is also impacting our connectivity to Dallas over our other CL links. Centurylink sent out this notification. Sending it out in case others are having similar issues.
07/17/2018 18:31:54 GMT -
*** CASCADED EXTERNAL NOTES 17-Jul-2018 18:30:14 GMT From CASE: 14896924 - Event Field Operations have identified a 96-count buried fiber cable that has been cut by boring activities occurring in the area. The cable is buried 14 feet deep and is within 25 feet of a railroad line. As such, the railroad line provider will not allow excavations activities to commence until all necessary shoring supplies are on site. The supplies are being obtained at this time, however, the ETA to the site is not yet known. Once the shoring is in place, the excavation team will begin digging a trench to access the damaged fiber and perform the repairs. The trench will need to be approximately 50 feet in length and at least 15 feet deep to access the fiber and raise it for splicing repairs to commence. The initial activities are expected to take approximately two hours. As such the next update will be relayed at 20:30 GMT or as additional information becomes available. This issue has been escalated internally to the Executive level proactively.
07/17/2018 17:38:50 GMT -
*** CASCADED EXTERNAL NOTES 17-Jul-2018 17:36:55 GMT From CASE: 14896924 - Event
Field Operations and the OSPE team have arrived at the site and have identified a location where they believe a buried fiber cable has been damaged by construction activities occurring in the area. Additional assessments are ongoing at this time. A fiber splicing repair team has been engaged to assist with the repairs once the damaged fiber has been identified, however the ETA to the site is not yet known
07/17/2018 16:42:52 GMT -
*** CASCADED EXTERNAL NOTES 17-Jul-2018 16:40:40 GMT From CASE: 14896924 - Event
Field Operations have performed light measurement readings from the CenturyLink Gateway in Phoenix and have identified the fault condition 28.43 miles from the site. The Outside Plant Engineer (OSPE) has been engaged to assist with further isolation.
Thanks,
Bobin Joseph
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On 07/17/2018 07:10 PM, Murali Medikonda via Outages wrote:
Curious if anyone has any further updates on this fiber-cut.. I appreciate it
Here's the latest I've got: 07/18/2018 00:51:39 GMT - *** CASCADED EXTERNAL NOTES 18-Jul-2018 00:49:51 GMT From CASE: 14896554 - Event The excavation crew continues working to expose the rest of the conduit. 07/18/2018 00:12:38 GMT - *** CASCADED EXTERNAL NOTES 18-Jul-2018 00:05:57 GMT From CASE: 14896554 - Event Field Operations on site reports that 25 feet of conduit has been exposed and the damaged area has been located. 10 conduits have been determined to be destroyed. Crews on site continue excavating the area to completely uncover the damaged conduit; this process is expected to take 2 hours. After this process has been completed, the fiber will be brought to the surface so preparation efforts may begin for splicing. 07/17/2018 23:15:57 GMT - *** CASCADED EXTERNAL NOTES 17-Jul-2018 23:08:50 GMT From CASE: 14896554 - Event The Outside Plant team has located the conduit around 12 feet under the ground, and are currently excavating the area to discern the total extent of the damage. There are 12 conduits in total, so it may take time in isolating which conduits are damaged. 07/17/2018 19:51:58 GMT - *** CASCADED EXTERNAL NOTES 17-Jul-2018 19:50:18 GMT From CASE: 14896554 - Event Field Operations have reported that the Construction team is en route to the site with backhoes and the shoring supplies. An ETA of 20:00 GMT has been provided. In addition, local utility providers are currently on site marking existing utility lines in the area to prevent impacts during the repairs. Once on site, the Construction team will commence with installing the necessary shoring around the damage site. It is estimated that the shoring activities will be completed at 23:00 GMT and excavation activities will commence at that time. The Transport NOC has reported that there are currently no reroute opportunities. The next update will be relayed once the shoring has been completed or additional information becomes available. 07/17/2018 18:30:15 GMT - *** CASCADED EXTERNAL NOTES 17-Jul-2018 18:28:16 GMT From CASE: 14896554 - Event Field Operations have identified a 96-count buried fiber cable that has been cut by boring activities occurring in the area. The cable is buried 14 feet deep and is within 25 feet of a railroad line. As such, the railroad line provider will not allow excavations activities to commence until all necessary shoring supplies are on site. The supplies are being obtained at this time, however, the ETA to the site is not yet known. Once the shoring is in place, the excavation team will begin digging a trench to access the damaged fiber and perform the repairs. The trench will need to be approximately 50 feet in length and at least 15 feet deep to access the fiber and raise it for splicing repairs to commence. The initial activities are expected to take approximately two hours. As such the next update will be relayed at 20:30 GMT or as additional information becomes available. This issue has been escalated internally to the Executive level proactively. 07/17/2018 17:35:02 GMT - *** CASCADED EXTERNAL NOTES 17-Jul-2018 17:33:12 GMT From CASE: 14896554 - Event Field Operations and the OSPE team have arrived at the site and have identified a location where they believe a buried fiber cable has been damaged by construction activities occurring in the area. Additional assessments are on ongoing at this time. A fiber splicing repair team has been engaged to assist with the repairs once the damaged fiber has been identified, however the ETA to the site is not yet known. 07/17/2018 16:40:17 GMT - *** CASCADED EXTERNAL NOTES 17-Jul-2018 16:39:24 GMT From CASE: 14896554 - Event Field Operations have performed light measurement readings from the CenturyLink Gateway in Phoenix and have identified the fault condition 28.43 miles from the site. The Outside Plant Engineer (OSPE) has been engaged to assist with further isolation. 07/17/2018 16:20:14 GMT - 07/17/2018 15:54:16 GMT - The Transport NOC has reported that a suspected fiber cut in Phoenix, AZ is impacting unprotected transport services. Field Operations are being engaged to investigate further; however, an estimated time of arrival (ETA) is not yet known. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.
Last two updates; “The excavation crew continues working to expose the rest of the conduit.” “Field Operations on site reports that 25 feet of conduit has been exposed and the damaged area has been located. 10 conduits have been determined to be destroyed. Crews on site continue excavating the area to completely uncover the damaged conduit; this process is expected to take 2 hours. After this process has been completed, the fiber will be brought to the surface so preparation efforts may begin for splicing.” Sent from my iPhone On Jul 17, 2018, at 9:10 PM, Murali Medikonda via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Curious if anyone has any further updates on this fiber-cut.. I appreciate it On Tue, Jul 17, 2018 at 3:57 PM, Cory Sell via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Thanks, Joe. Below is the most recent update we received a bit ago: *** CASCADED EXTERNAL NOTES 17-Jul-2018 19:50:18 GMT - Event Field Operations have reported that the Construction team is en route to the site with backhoes and the shoring supplies. An ETA of 20:00 GMT has been provided. In addition, local utility providers are currently on site marking existing utility lines in the area to prevent impacts during the repairs. Once on site, the Construction team will commence with installing the necessary shoring around the damage site. It is estimated that the shoring activities will be completed at 23:00 GMT and excavation activities will commence at that time. The Transport NOC has reported that there are currently no reroute opportunities. The next update will be relayed once the shoring has been completed or additional information becomes available. Regards, Cory On Tue, Jul 17, 2018, 1:39 PM Bobin Joseph via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: We have P2P link between Dallas and San Diego that went down. It is also impacting our connectivity to Dallas over our other CL links. Centurylink sent out this notification. Sending it out in case others are having similar issues. 07/17/2018 18:31:54 GMT - *** CASCADED EXTERNAL NOTES 17-Jul-2018 18:30:14 GMT From CASE: 14896924 - Event Field Operations have identified a 96-count buried fiber cable that has been cut by boring activities occurring in the area. The cable is buried 14 feet deep and is within 25 feet of a railroad line. As such, the railroad line provider will not allow excavations activities to commence until all necessary shoring supplies are on site. The supplies are being obtained at this time, however, the ETA to the site is not yet known. Once the shoring is in place, the excavation team will begin digging a trench to access the damaged fiber and perform the repairs. The trench will need to be approximately 50 feet in length and at least 15 feet deep to access the fiber and raise it for splicing repairs to commence. The initial activities are expected to take approximately two hours. As such the next update will be relayed at 20:30 GMT or as additional information becomes available. This issue has been escalated internally to the Executive level proactively. 07/17/2018 17:38:50 GMT - *** CASCADED EXTERNAL NOTES 17-Jul-2018 17:36:55 GMT From CASE: 14896924 - Event Field Operations and the OSPE team have arrived at the site and have identified a location where they believe a buried fiber cable has been damaged by construction activities occurring in the area. Additional assessments are ongoing at this time. A fiber splicing repair team has been engaged to assist with the repairs once the damaged fiber has been identified, however the ETA to the site is not yet known 07/17/2018 16:42:52 GMT - *** CASCADED EXTERNAL NOTES 17-Jul-2018 16:40:40 GMT From CASE: 14896924 - Event Field Operations have performed light measurement readings from the CenturyLink Gateway in Phoenix and have identified the fault condition 28.43 miles from the site. The Outside Plant Engineer (OSPE) has been engaged to assist with further isolation. Thanks, Bobin Joseph _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
ETA is 0600 GMT. *** CASCADED EXTERNAL NOTES 18-Jul-2018 04:53:27 GMT From CASE: 14896924 - Event Field Operations report splicing in the South pit is progressing with 60 fibers completed. Splicing is anticipated to be complete in the South pit in 20 minutes. The North pit is bare testing fibers and are anticipating splicing to begin in approximately 20 minutes. Field technicians have been dispatched to various facilities to assist with equipment restoration if any problems occur. Systems should begin to restore as splicing in the North pit begins. The estimated restore time remains 06:00 GMT. Thanks, Bobin Joseph From: Outages <outages-bounces@outages.org> On Behalf Of Nikolas Geyer via Outages Sent: Tuesday, July 17, 2018 6:22 PM To: Murali Medikonda <mmedikonda@bandwidth.com> Cc: outages@outages.org Subject: Re: [outages] Centurylink Fiber cut in Phoenix Last two updates; “The excavation crew continues working to expose the rest of the conduit.” “Field Operations on site reports that 25 feet of conduit has been exposed and the damaged area has been located. 10 conduits have been determined to be destroyed. Crews on site continue excavating the area to completely uncover the damaged conduit; this process is expected to take 2 hours. After this process has been completed, the fiber will be brought to the surface so preparation efforts may begin for splicing.” Sent from my iPhone On Jul 17, 2018, at 9:10 PM, Murali Medikonda via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Curious if anyone has any further updates on this fiber-cut.. I appreciate it On Tue, Jul 17, 2018 at 3:57 PM, Cory Sell via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Thanks, Joe. Below is the most recent update we received a bit ago: *** CASCADED EXTERNAL NOTES 17-Jul-2018 19:50:18 GMT - Event Field Operations have reported that the Construction team is en route to the site with backhoes and the shoring supplies. An ETA of 20:00 GMT has been provided. In addition, local utility providers are currently on site marking existing utility lines in the area to prevent impacts during the repairs. Once on site, the Construction team will commence with installing the necessary shoring around the damage site. It is estimated that the shoring activities will be completed at 23:00 GMT and excavation activities will commence at that time. The Transport NOC has reported that there are currently no reroute opportunities. The next update will be relayed once the shoring has been completed or additional information becomes available. Regards, Cory On Tue, Jul 17, 2018, 1:39 PM Bobin Joseph via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: We have P2P link between Dallas and San Diego that went down. It is also impacting our connectivity to Dallas over our other CL links. Centurylink sent out this notification. Sending it out in case others are having similar issues. 07/17/2018 18:31:54 GMT - *** CASCADED EXTERNAL NOTES 17-Jul-2018 18:30:14 GMT From CASE: 14896924 - Event Field Operations have identified a 96-count buried fiber cable that has been cut by boring activities occurring in the area. The cable is buried 14 feet deep and is within 25 feet of a railroad line. As such, the railroad line provider will not allow excavations activities to commence until all necessary shoring supplies are on site. The supplies are being obtained at this time, however, the ETA to the site is not yet known. Once the shoring is in place, the excavation team will begin digging a trench to access the damaged fiber and perform the repairs. The trench will need to be approximately 50 feet in length and at least 15 feet deep to access the fiber and raise it for splicing repairs to commence. The initial activities are expected to take approximately two hours. As such the next update will be relayed at 20:30 GMT or as additional information becomes available. This issue has been escalated internally to the Executive level proactively. 07/17/2018 17:38:50 GMT - *** CASCADED EXTERNAL NOTES 17-Jul-2018 17:36:55 GMT From CASE: 14896924 - Event Field Operations and the OSPE team have arrived at the site and have identified a location where they believe a buried fiber cable has been damaged by construction activities occurring in the area. Additional assessments are ongoing at this time. A fiber splicing repair team has been engaged to assist with the repairs once the damaged fiber has been identified, however the ETA to the site is not yet known 07/17/2018 16:42:52 GMT - *** CASCADED EXTERNAL NOTES 17-Jul-2018 16:40:40 GMT From CASE: 14896924 - Event Field Operations have performed light measurement readings from the CenturyLink Gateway in Phoenix and have identified the fault condition 28.43 miles from the site. The Outside Plant Engineer (OSPE) has been engaged to assist with further isolation. Thanks, Bobin Joseph _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
*** EXTERNAL NOTES 18-Jul-2018 08:35:17 GMT Event Conclusion Summary Outage Start: July 17 2018 15:29 GMT Outage Stop: July 18 2018 06:55 GMT Root Cause: A fiber cut on a buried fiber cable in Phoenix, AZ was impacting unprotected services. Fix action: The fiber was spliced to restore. Summary: The Transport NOC reported that a suspected fiber cut in Phoenix, AZ was impacting unprotected transport services. Field Operations were engaged to investigate further and identified the fault condition 28.43 miles from the site. Field Operations and the Outside Plant team dispatched to site and isolated a 96-count buried fiber cable that has been cut by boring activities occurring in the area. The cable was buried 14 feet deep and was within 25 feet of a railroad line. As such, the railroad line provider would not allow excavations activities to commence until all necessary shoring supplies were on site. Once the shoring was in place, the excavation team will began digging a trench to access the damaged fiber and perform the repairs. In addition, local utility providers were on site marking existing utility lines in the area to prevent impacts during the repairs The trench needed to be approximately 50 feet in length and at least 15 feet deep to access the fiber and raise it for splicing repairs to commence. Field Operations reported 10 of the 12 conduits had been destroyed. Excavation was completed, slack was pulled and the fiber prepped, tested, and then spliced. Services were confirmed restored by the Repair Center with impacted customers. This service impact has concluded; however, there may be additional analysis and discovery that occurs as the incident is reviewed by NOC management. Any available updates will be relayed upon event ticket closure within the next 72 hours. At that time, a customer satisfaction survey link may be available. We strive to provide thorough communications containing the available information during a service disruption. Please let us know if the updates you received during this event were satisfactory. - Mike Bolitho On Tue, Jul 17, 2018 at 6:10 PM, Murali Medikonda via Outages < outages@outages.org> wrote:
Curious if anyone has any further updates on this fiber-cut.. I appreciate it
On Tue, Jul 17, 2018 at 3:57 PM, Cory Sell via Outages < outages@outages.org> wrote:
Thanks, Joe. Below is the most recent update we received a bit ago:
*** CASCADED EXTERNAL NOTES 17-Jul-2018 19:50:18 GMT - Event
Field Operations have reported that the Construction team is en route to the site with backhoes and the shoring supplies. An ETA of 20:00 GMT has been provided. In addition, local utility providers are currently on site marking existing utility lines in the area to prevent impacts during the repairs. Once on site, the Construction team will commence with installing the necessary shoring around the damage site. It is estimated that the shoring activities will be completed at 23:00 GMT and excavation activities will commence at that time. The Transport NOC has reported that there are currently no reroute opportunities. The next update will be relayed once the shoring has been completed or additional information becomes available.
Regards,
Cory
On Tue, Jul 17, 2018, 1:39 PM Bobin Joseph via Outages < outages@outages.org> wrote:
We have P2P link between Dallas and San Diego that went down. It is also impacting our connectivity to Dallas over our other CL links. Centurylink sent out this notification. Sending it out in case others are having similar issues.
07/17/2018 18:31:54 GMT -
*** CASCADED EXTERNAL NOTES 17-Jul-2018 18:30:14 GMT From CASE: 14896924 - Event Field Operations have identified a 96-count buried fiber cable that has been cut by boring activities occurring in the area. The cable is buried 14 feet deep and is within 25 feet of a railroad line. As such, the railroad line provider will not allow excavations activities to commence until all necessary shoring supplies are on site. The supplies are being obtained at this time, however, the ETA to the site is not yet known. Once the shoring is in place, the excavation team will begin digging a trench to access the damaged fiber and perform the repairs. The trench will need to be approximately 50 feet in length and at least 15 feet deep to access the fiber and raise it for splicing repairs to commence. The initial activities are expected to take approximately two hours. As such the next update will be relayed at 20:30 GMT or as additional information becomes available. This issue has been escalated internally to the Executive level proactively.
07/17/2018 17:38:50 GMT -
*** CASCADED EXTERNAL NOTES 17-Jul-2018 17:36:55 GMT From CASE: 14896924 - Event
Field Operations and the OSPE team have arrived at the site and have identified a location where they believe a buried fiber cable has been damaged by construction activities occurring in the area. Additional assessments are ongoing at this time. A fiber splicing repair team has been engaged to assist with the repairs once the damaged fiber has been identified, however the ETA to the site is not yet known
07/17/2018 16:42:52 GMT -
*** CASCADED EXTERNAL NOTES 17-Jul-2018 16:40:40 GMT From CASE: 14896924 - Event
Field Operations have performed light measurement readings from the CenturyLink Gateway in Phoenix and have identified the fault condition 28.43 miles from the site. The Outside Plant Engineer (OSPE) has been engaged to assist with further isolation.
Thanks,
Bobin Joseph
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Is anyone still seeing issues? The cut was restored, but we are still seeing layer 3 issues. On Wed, Jul 18, 2018, 10:48 AM Mike Bolitho via Outages <outages@outages.org> wrote:
*** EXTERNAL NOTES 18-Jul-2018 08:35:17 GMT Event Conclusion Summary
Outage Start: July 17 2018 15:29 GMT
Outage Stop: July 18 2018 06:55 GMT
Root Cause: A fiber cut on a buried fiber cable in Phoenix, AZ was impacting unprotected services.
Fix action: The fiber was spliced to restore.
Summary: The Transport NOC reported that a suspected fiber cut in Phoenix, AZ was impacting unprotected transport services. Field Operations were engaged to investigate further and identified the fault condition 28.43 miles from the site. Field Operations and the Outside Plant team dispatched to site and isolated a 96-count buried fiber cable that has been cut by boring activities occurring in the area. The cable was buried 14 feet deep and was within 25 feet of a railroad line. As such, the railroad line provider would not allow excavations activities to commence until all necessary shoring supplies were on site. Once the shoring was in place, the excavation team will began digging a trench to access the damaged fiber and perform the repairs. In addition, local utility providers were on site marking existing utility lines in the area to prevent impacts during the repairs The trench needed to be approximately 50 feet in length and at least 15 feet deep to access the fiber and raise it for splicing repairs to commence. Field Operations reported 10 of the 12 conduits had been destroyed. Excavation was completed, slack was pulled and the fiber prepped, tested, and then spliced. Services were confirmed restored by the Repair Center with impacted customers.
This service impact has concluded; however, there may be additional analysis and discovery that occurs as the incident is reviewed by NOC management. Any available updates will be relayed upon event ticket closure within the next 72 hours. At that time, a customer satisfaction survey link may be available. We strive to provide thorough communications containing the available information during a service disruption. Please let us know if the updates you received during this event were satisfactory.
- Mike Bolitho
On Tue, Jul 17, 2018 at 6:10 PM, Murali Medikonda via Outages < outages@outages.org> wrote:
Curious if anyone has any further updates on this fiber-cut.. I appreciate it
On Tue, Jul 17, 2018 at 3:57 PM, Cory Sell via Outages < outages@outages.org> wrote:
Thanks, Joe. Below is the most recent update we received a bit ago:
*** CASCADED EXTERNAL NOTES 17-Jul-2018 19:50:18 GMT - Event
Field Operations have reported that the Construction team is en route to the site with backhoes and the shoring supplies. An ETA of 20:00 GMT has been provided. In addition, local utility providers are currently on site marking existing utility lines in the area to prevent impacts during the repairs. Once on site, the Construction team will commence with installing the necessary shoring around the damage site. It is estimated that the shoring activities will be completed at 23:00 GMT and excavation activities will commence at that time. The Transport NOC has reported that there are currently no reroute opportunities. The next update will be relayed once the shoring has been completed or additional information becomes available.
Regards,
Cory
On Tue, Jul 17, 2018, 1:39 PM Bobin Joseph via Outages < outages@outages.org> wrote:
We have P2P link between Dallas and San Diego that went down. It is also impacting our connectivity to Dallas over our other CL links. Centurylink sent out this notification. Sending it out in case others are having similar issues.
07/17/2018 18:31:54 GMT -
*** CASCADED EXTERNAL NOTES 17-Jul-2018 18:30:14 GMT From CASE: 14896924 - Event Field Operations have identified a 96-count buried fiber cable that has been cut by boring activities occurring in the area. The cable is buried 14 feet deep and is within 25 feet of a railroad line. As such, the railroad line provider will not allow excavations activities to commence until all necessary shoring supplies are on site. The supplies are being obtained at this time, however, the ETA to the site is not yet known. Once the shoring is in place, the excavation team will begin digging a trench to access the damaged fiber and perform the repairs. The trench will need to be approximately 50 feet in length and at least 15 feet deep to access the fiber and raise it for splicing repairs to commence. The initial activities are expected to take approximately two hours. As such the next update will be relayed at 20:30 GMT or as additional information becomes available. This issue has been escalated internally to the Executive level proactively.
07/17/2018 17:38:50 GMT -
*** CASCADED EXTERNAL NOTES 17-Jul-2018 17:36:55 GMT From CASE: 14896924 - Event
Field Operations and the OSPE team have arrived at the site and have identified a location where they believe a buried fiber cable has been damaged by construction activities occurring in the area. Additional assessments are ongoing at this time. A fiber splicing repair team has been engaged to assist with the repairs once the damaged fiber has been identified, however the ETA to the site is not yet known
07/17/2018 16:42:52 GMT -
*** CASCADED EXTERNAL NOTES 17-Jul-2018 16:40:40 GMT From CASE: 14896924 - Event
Field Operations have performed light measurement readings from the CenturyLink Gateway in Phoenix and have identified the fault condition 28.43 miles from the site. The Outside Plant Engineer (OSPE) has been engaged to assist with further isolation.
Thanks,
Bobin Joseph
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Our VPLS and P2P’s are back up and running fine. This is between TX and SoCal. Thanks, Bobin Joseph From: Outages <outages-bounces@outages.org> On Behalf Of Cory Sell via Outages Sent: Wednesday, July 18, 2018 8:52 AM To: Mike Bolitho <mikebolitho@gmail.com> Cc: outages@outages.org Subject: Re: [outages] Centurylink Fiber cut in Phoenix Is anyone still seeing issues? The cut was restored, but we are still seeing layer 3 issues. On Wed, Jul 18, 2018, 10:48 AM Mike Bolitho via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: *** EXTERNAL NOTES 18-Jul-2018 08:35:17 GMT Event Conclusion Summary Outage Start: July 17 2018 15:29 GMT Outage Stop: July 18 2018 06:55 GMT Root Cause: A fiber cut on a buried fiber cable in Phoenix, AZ was impacting unprotected services. Fix action: The fiber was spliced to restore. Summary: The Transport NOC reported that a suspected fiber cut in Phoenix, AZ was impacting unprotected transport services. Field Operations were engaged to investigate further and identified the fault condition 28.43 miles from the site. Field Operations and the Outside Plant team dispatched to site and isolated a 96-count buried fiber cable that has been cut by boring activities occurring in the area. The cable was buried 14 feet deep and was within 25 feet of a railroad line. As such, the railroad line provider would not allow excavations activities to commence until all necessary shoring supplies were on site. Once the shoring was in place, the excavation team will began digging a trench to access the damaged fiber and perform the repairs. In addition, local utility providers were on site marking existing utility lines in the area to prevent impacts during the repairs The trench needed to be approximately 50 feet in length and at least 15 feet deep to access the fiber and raise it for splicing repairs to commence. Field Operations reported 10 of the 12 conduits had been destroyed. Excavation was completed, slack was pulled and the fiber prepped, tested, and then spliced. Services were confirmed restored by the Repair Center with impacted customers. This service impact has concluded; however, there may be additional analysis and discovery that occurs as the incident is reviewed by NOC management. Any available updates will be relayed upon event ticket closure within the next 72 hours. At that time, a customer satisfaction survey link may be available. We strive to provide thorough communications containing the available information during a service disruption. Please let us know if the updates you received during this event were satisfactory. - Mike Bolitho On Tue, Jul 17, 2018 at 6:10 PM, Murali Medikonda via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Curious if anyone has any further updates on this fiber-cut.. I appreciate it On Tue, Jul 17, 2018 at 3:57 PM, Cory Sell via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Thanks, Joe. Below is the most recent update we received a bit ago: *** CASCADED EXTERNAL NOTES 17-Jul-2018 19:50:18 GMT - Event Field Operations have reported that the Construction team is en route to the site with backhoes and the shoring supplies. An ETA of 20:00 GMT has been provided. In addition, local utility providers are currently on site marking existing utility lines in the area to prevent impacts during the repairs. Once on site, the Construction team will commence with installing the necessary shoring around the damage site. It is estimated that the shoring activities will be completed at 23:00 GMT and excavation activities will commence at that time. The Transport NOC has reported that there are currently no reroute opportunities. The next update will be relayed once the shoring has been completed or additional information becomes available. Regards, Cory On Tue, Jul 17, 2018, 1:39 PM Bobin Joseph via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: We have P2P link between Dallas and San Diego that went down. It is also impacting our connectivity to Dallas over our other CL links. Centurylink sent out this notification. Sending it out in case others are having similar issues. 07/17/2018 18:31:54 GMT - *** CASCADED EXTERNAL NOTES 17-Jul-2018 18:30:14 GMT From CASE: 14896924 - Event Field Operations have identified a 96-count buried fiber cable that has been cut by boring activities occurring in the area. The cable is buried 14 feet deep and is within 25 feet of a railroad line. As such, the railroad line provider will not allow excavations activities to commence until all necessary shoring supplies are on site. The supplies are being obtained at this time, however, the ETA to the site is not yet known. Once the shoring is in place, the excavation team will begin digging a trench to access the damaged fiber and perform the repairs. The trench will need to be approximately 50 feet in length and at least 15 feet deep to access the fiber and raise it for splicing repairs to commence. The initial activities are expected to take approximately two hours. As such the next update will be relayed at 20:30 GMT or as additional information becomes available. This issue has been escalated internally to the Executive level proactively. 07/17/2018 17:38:50 GMT - *** CASCADED EXTERNAL NOTES 17-Jul-2018 17:36:55 GMT From CASE: 14896924 - Event Field Operations and the OSPE team have arrived at the site and have identified a location where they believe a buried fiber cable has been damaged by construction activities occurring in the area. Additional assessments are ongoing at this time. A fiber splicing repair team has been engaged to assist with the repairs once the damaged fiber has been identified, however the ETA to the site is not yet known 07/17/2018 16:42:52 GMT - *** CASCADED EXTERNAL NOTES 17-Jul-2018 16:40:40 GMT From CASE: 14896924 - Event Field Operations have performed light measurement readings from the CenturyLink Gateway in Phoenix and have identified the fault condition 28.43 miles from the site. The Outside Plant Engineer (OSPE) has been engaged to assist with further isolation. Thanks, Bobin Joseph _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
participants (6)
-
Bobin Joseph -
Cory Sell -
Mike Bolitho -
Murali Medikonda -
Nikolas Geyer -
Tim Densmore