CenturyLink Internet Outage in Iowa, possibly Nationally?
Our CenturyLink services here in Iowa are down. Ethernet carriers are up, BGP sessions are up but we're not getting any BGP routes except 150 odd internal CenturyLink routes. Been waiting on hold with CL support for a while. Is anyone else seeing the same issue? -- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden.
Not national. 3356 and 209 are up in Denver. On Fri, Aug 23, 2019 at 9:27 AM Jeffrey Ollie via Outages < outages@outages.org> wrote:
Our CenturyLink services here in Iowa are down. Ethernet carriers are up, BGP sessions are up but we're not getting any BGP routes except 150 odd internal CenturyLink routes. Been waiting on hold with CL support for a while. Is anyone else seeing the same issue?
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I finally made it through the queue and was told that there's a possible fiber cut in the Altoona, Iowa area. 08/23/2019 15:30:46 GMT - The IP NOC is investigating an issue that is causing multiple routers to be isolated in Des Moines, IA that is impacting IP services. Field Operations are on site and have advised that there is power to the site and no immediate issues observed with the equipment. The Transport NOC is being engaged to investigate the transport portion of the network. 08/23/2019 15:18:36 GMT - On August 23, 2019 at 15:04 GMT, CenturyLink identified a service impact in Des Moines, IA. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting efforts and details may evolve as the discovery process evolves through to service resolution, ticket closure, or post incident review. On Fri, Aug 23, 2019 at 10:31 AM Joelle Maslak <jmaslak@antelope.net> wrote:
Not national. 3356 and 209 are up in Denver.
On Fri, Aug 23, 2019 at 9:27 AM Jeffrey Ollie via Outages < outages@outages.org> wrote:
Our CenturyLink services here in Iowa are down. Ethernet carriers are up, BGP sessions are up but we're not getting any BGP routes except 150 odd internal CenturyLink routes. Been waiting on hold with CL support for a while. Is anyone else seeing the same issue?
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden.
*** CASCADED EXTERNAL NOTES 23-Aug-2019 15:50:26 GMT From CASE: 16976512 - Event The Transport NOC have dispatched Field Operations to two sites in Des Moines, IA and have taken light level readings indicating a suspected fiber issue is present. Construction efforts in the area have been identified in the area, and Field Operations are dispatching to that area to further isolate the suspected damage. On Fri, Aug 23, 2019 at 10:39 AM Jeffrey Ollie <jeff@ocjtech.us> wrote:
I finally made it through the queue and was told that there's a possible fiber cut in the Altoona, Iowa area.
08/23/2019 15:30:46 GMT - The IP NOC is investigating an issue that is causing multiple routers to be isolated in Des Moines, IA that is impacting IP services. Field Operations are on site and have advised that there is power to the site and no immediate issues observed with the equipment. The Transport NOC is being engaged to investigate the transport portion of the network.
08/23/2019 15:18:36 GMT - On August 23, 2019 at 15:04 GMT, CenturyLink identified a service impact in Des Moines, IA. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting efforts and details may evolve as the discovery process evolves through to service resolution, ticket closure, or post incident review.
On Fri, Aug 23, 2019 at 10:31 AM Joelle Maslak <jmaslak@antelope.net> wrote:
Not national. 3356 and 209 are up in Denver.
On Fri, Aug 23, 2019 at 9:27 AM Jeffrey Ollie via Outages < outages@outages.org> wrote:
Our CenturyLink services here in Iowa are down. Ethernet carriers are up, BGP sessions are up but we're not getting any BGP routes except 150 odd internal CenturyLink routes. Been waiting on hold with CL support for a while. Is anyone else seeing the same issue?
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden.
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden.
*** CASCADED EXTERNAL NOTES 23-Aug-2019 17:23:30 GMT From CASE: 16976512 - Event The Transport NOC has advised that Field Operations continue to work to isolate the fiber damage. Additional light level testing was performed along the span, and Field Operations have advised the damage is suspected to be located 125 feet from the latest testing location. 08/23/2019 16:37:04 GMT - *** CASCADED EXTERNAL NOTES 23-Aug-2019 16:34:24 GMT From CASE: 16976512 - Event Field Operations is working to traverse the fiber span in an effort to identify the fiber damage. It has been advised that there are multiple areas of construction being performed along the span and Field Operations are actively investigating those areas. On Fri, Aug 23, 2019 at 10:56 AM Jeffrey Ollie <jeff@ocjtech.us> wrote:
*** CASCADED EXTERNAL NOTES 23-Aug-2019 15:50:26 GMT From CASE: 16976512 - Event The Transport NOC have dispatched Field Operations to two sites in Des Moines, IA and have taken light level readings indicating a suspected fiber issue is present. Construction efforts in the area have been identified in the area, and Field Operations are dispatching to that area to further isolate the suspected damage.
On Fri, Aug 23, 2019 at 10:39 AM Jeffrey Ollie <jeff@ocjtech.us> wrote:
I finally made it through the queue and was told that there's a possible fiber cut in the Altoona, Iowa area.
08/23/2019 15:30:46 GMT - The IP NOC is investigating an issue that is causing multiple routers to be isolated in Des Moines, IA that is impacting IP services. Field Operations are on site and have advised that there is power to the site and no immediate issues observed with the equipment. The Transport NOC is being engaged to investigate the transport portion of the network.
08/23/2019 15:18:36 GMT - On August 23, 2019 at 15:04 GMT, CenturyLink identified a service impact in Des Moines, IA. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting efforts and details may evolve as the discovery process evolves through to service resolution, ticket closure, or post incident review.
On Fri, Aug 23, 2019 at 10:31 AM Joelle Maslak <jmaslak@antelope.net> wrote:
Not national. 3356 and 209 are up in Denver.
On Fri, Aug 23, 2019 at 9:27 AM Jeffrey Ollie via Outages < outages@outages.org> wrote:
Our CenturyLink services here in Iowa are down. Ethernet carriers are up, BGP sessions are up but we're not getting any BGP routes except 150 odd internal CenturyLink routes. Been waiting on hold with CL support for a while. Is anyone else seeing the same issue?
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden.
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden.
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden.
*** CASCADED EXTERNAL NOTES 23-Aug-2019 18:09:40 GMT From CASE: 16976512 - Event Field Operations on site have identified that equipment laying rock in the area had damaged multiple cables. Efforts to identify the extent of the damage and formulate a repair plan are ongoing. While efforts to identify the fiber damage were ongoing, services were migrated to a diverse fiber path to restore all services. At this time all services have been restored but are currently in a Loss of Redundancy state while the fiber repairs remain ongoing. The Transport and IP NOC will continue to monitor services while the fiber repairs remain ongoing to ensure stability. A final notification will be provided momentarily. On Fri, Aug 23, 2019 at 12:52 PM Jeffrey Ollie <jeff@ocjtech.us> wrote:
*** CASCADED EXTERNAL NOTES 23-Aug-2019 17:23:30 GMT From CASE: 16976512 - Event The Transport NOC has advised that Field Operations continue to work to isolate the fiber damage. Additional light level testing was performed along the span, and Field Operations have advised the damage is suspected to be located 125 feet from the latest testing location.
08/23/2019 16:37:04 GMT - *** CASCADED EXTERNAL NOTES 23-Aug-2019 16:34:24 GMT From CASE: 16976512 - Event Field Operations is working to traverse the fiber span in an effort to identify the fiber damage. It has been advised that there are multiple areas of construction being performed along the span and Field Operations are actively investigating those areas.
On Fri, Aug 23, 2019 at 10:56 AM Jeffrey Ollie <jeff@ocjtech.us> wrote:
*** CASCADED EXTERNAL NOTES 23-Aug-2019 15:50:26 GMT From CASE: 16976512 - Event The Transport NOC have dispatched Field Operations to two sites in Des Moines, IA and have taken light level readings indicating a suspected fiber issue is present. Construction efforts in the area have been identified in the area, and Field Operations are dispatching to that area to further isolate the suspected damage.
On Fri, Aug 23, 2019 at 10:39 AM Jeffrey Ollie <jeff@ocjtech.us> wrote:
I finally made it through the queue and was told that there's a possible fiber cut in the Altoona, Iowa area.
08/23/2019 15:30:46 GMT - The IP NOC is investigating an issue that is causing multiple routers to be isolated in Des Moines, IA that is impacting IP services. Field Operations are on site and have advised that there is power to the site and no immediate issues observed with the equipment. The Transport NOC is being engaged to investigate the transport portion of the network.
08/23/2019 15:18:36 GMT - On August 23, 2019 at 15:04 GMT, CenturyLink identified a service impact in Des Moines, IA. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting efforts and details may evolve as the discovery process evolves through to service resolution, ticket closure, or post incident review.
On Fri, Aug 23, 2019 at 10:31 AM Joelle Maslak <jmaslak@antelope.net> wrote:
Not national. 3356 and 209 are up in Denver.
On Fri, Aug 23, 2019 at 9:27 AM Jeffrey Ollie via Outages < outages@outages.org> wrote:
Our CenturyLink services here in Iowa are down. Ethernet carriers are up, BGP sessions are up but we're not getting any BGP routes except 150 odd internal CenturyLink routes. Been waiting on hold with CL support for a while. Is anyone else seeing the same issue?
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden.
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden.
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden.
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden.
*** CASCADED EXTERNAL NOTES 23-Aug-2019 18:45:17 GMT From CASE: 16976512 - Event *Event Conclusion Summary* Outage Start: August 23, 2019 14:44 GMT Outage Stop: August 23, 2019 17:19 GMT Root Cause: A fiber cut caused multiple isolated routers in Des Moines, IA that was impacting IP services. Fix Action: Impacted services were migrated to a diverse fiber path. Summary: On August 23, 2019 at 14:44 GMT, CenturyLink identified a service impact in Des Moines, IA and the IP NOC began investigating. It was advised that multiple routers had become isolating causing impact to IP services. Field Operations were dispatched to the site of the isolated routers but confirmed power and were unable to identify any issues. The Transport NOC was then engaged and identified a suspected fiber issue. Field Operations were dispatched for initial light level testing. Initial light level testing confirmed the existence of a fiber cut in the area, and Field Operations advised there were multiple instances of constructions works ongoing in the area. Field Operations began traversing the span and taking additional light level readings from manholes along the span. Field Operations later identified that equipment laying rock in the area had damaged multiple cables. Efforts to identify the extent of the damage and formulate a repair plan remained ongo ing. While efforts to identify the fiber damage were ongoing, services were migrated to a diverse fiber path to restore all services. At this time all services have been restored but are currently in a Loss of Redundancy state while the fiber repairs remain ongoing. The Transport and IP NOC will continue to monitor services while the fiber repairs remain ongoing to ensure stability. A final notification will be provided momentarily. This service impact has concluded. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance. There may be additional analysis and discovery that occurs as the incident is reviewed by NOC management. Any available updates will be relayed upon event ticket closure within the next 72 hours. At that time, a customer satisfaction survey link may be available. We strive to provide thorough communications containing the available information during a service disruption. Please let us know if the updates you received during this event were satisfactory. On Fri, Aug 23, 2019 at 1:36 PM Jeffrey Ollie <jeff@ocjtech.us> wrote:
*** CASCADED EXTERNAL NOTES 23-Aug-2019 18:09:40 GMT From CASE: 16976512 - Event Field Operations on site have identified that equipment laying rock in the area had damaged multiple cables. Efforts to identify the extent of the damage and formulate a repair plan are ongoing.
While efforts to identify the fiber damage were ongoing, services were migrated to a diverse fiber path to restore all services. At this time all services have been restored but are currently in a Loss of Redundancy state while the fiber repairs remain ongoing. The Transport and IP NOC will continue to monitor services while the fiber repairs remain ongoing to ensure stability. A final notification will be provided momentarily.
On Fri, Aug 23, 2019 at 12:52 PM Jeffrey Ollie <jeff@ocjtech.us> wrote:
*** CASCADED EXTERNAL NOTES 23-Aug-2019 17:23:30 GMT From CASE: 16976512 - Event The Transport NOC has advised that Field Operations continue to work to isolate the fiber damage. Additional light level testing was performed along the span, and Field Operations have advised the damage is suspected to be located 125 feet from the latest testing location.
08/23/2019 16:37:04 GMT - *** CASCADED EXTERNAL NOTES 23-Aug-2019 16:34:24 GMT From CASE: 16976512 - Event Field Operations is working to traverse the fiber span in an effort to identify the fiber damage. It has been advised that there are multiple areas of construction being performed along the span and Field Operations are actively investigating those areas.
On Fri, Aug 23, 2019 at 10:56 AM Jeffrey Ollie <jeff@ocjtech.us> wrote:
*** CASCADED EXTERNAL NOTES 23-Aug-2019 15:50:26 GMT From CASE: 16976512 - Event The Transport NOC have dispatched Field Operations to two sites in Des Moines, IA and have taken light level readings indicating a suspected fiber issue is present. Construction efforts in the area have been identified in the area, and Field Operations are dispatching to that area to further isolate the suspected damage.
On Fri, Aug 23, 2019 at 10:39 AM Jeffrey Ollie <jeff@ocjtech.us> wrote:
I finally made it through the queue and was told that there's a possible fiber cut in the Altoona, Iowa area.
08/23/2019 15:30:46 GMT - The IP NOC is investigating an issue that is causing multiple routers to be isolated in Des Moines, IA that is impacting IP services. Field Operations are on site and have advised that there is power to the site and no immediate issues observed with the equipment. The Transport NOC is being engaged to investigate the transport portion of the network.
08/23/2019 15:18:36 GMT - On August 23, 2019 at 15:04 GMT, CenturyLink identified a service impact in Des Moines, IA. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting efforts and details may evolve as the discovery process evolves through to service resolution, ticket closure, or post incident review.
On Fri, Aug 23, 2019 at 10:31 AM Joelle Maslak <jmaslak@antelope.net> wrote:
Not national. 3356 and 209 are up in Denver.
On Fri, Aug 23, 2019 at 9:27 AM Jeffrey Ollie via Outages < outages@outages.org> wrote:
Our CenturyLink services here in Iowa are down. Ethernet carriers are up, BGP sessions are up but we're not getting any BGP routes except 150 odd internal CenturyLink routes. Been waiting on hold with CL support for a while. Is anyone else seeing the same issue?
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden.
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden.
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden.
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden.
-- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden.
----- Original Message -----
From: "Jeffrey Ollie via Outages" <outages@outages.org>
*** CASCADED EXTERNAL NOTES 23-Aug-2019 18:09:40 GMT From CASE: 16976512 - Event Field Operations on site have identified that equipment laying rock in the area had damaged multiple cables. Efforts to identify the extent of the damage and formulate a repair plan are ongoing.
While efforts to identify the fiber damage were ongoing, services were migrated to a diverse fiber path to restore all services. At this time all services have been restored but are currently in a Loss of Redundancy state while the fiber
Anyone who wants to discuss whether they were *already* in a Loss of Redundancy state if it took several hours to fail over is invited to do so on outages-discussion :-) Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
Jeff, Doesn't appear to be national now. We had problems with our CLink MPLS across WA, OR, CA, TX, and WI yesterday. It probably started (for us) around 1400-1430 UTC 08/22/19. We noticed by 1800-1900 UTC yesterday the circuits we tested were working. We didn't reactivate any of them fully until this morning because we had had no responses to our ticket. Also yesterday on the outage map most major cities across the country were a lighter orange for CLink. Today its primarily the Central US vertical line (OK, KS, NB, IO, IL, MN). This only affected our MPLS circuits, no IP or voice services. We also have not received any reason on our tickets. Hope that helps. Brian ________________________________ From: Outages <outages-bounces@outages.org> on behalf of Jeffrey Ollie via Outages <outages@outages.org> Sent: Friday, August 23, 2019 8:56 AM Cc: Outages@outages.org (outages@outages.org) <Outages@outages.org> Subject: Re: [outages] CenturyLink Internet Outage in Iowa, possibly Nationally? *** CASCADED EXTERNAL NOTES 23-Aug-2019 15:50:26 GMT From CASE: 16976512 - Event The Transport NOC have dispatched Field Operations to two sites in Des Moines, IA and have taken light level readings indicating a suspected fiber issue is present. Construction efforts in the area have been identified in the area, and Field Operations are dispatching to that area to further isolate the suspected damage. On Fri, Aug 23, 2019 at 10:39 AM Jeffrey Ollie <jeff@ocjtech.us<mailto:jeff@ocjtech.us>> wrote: I finally made it through the queue and was told that there's a possible fiber cut in the Altoona, Iowa area. 08/23/2019 15:30:46 GMT - The IP NOC is investigating an issue that is causing multiple routers to be isolated in Des Moines, IA that is impacting IP services. Field Operations are on site and have advised that there is power to the site and no immediate issues observed with the equipment. The Transport NOC is being engaged to investigate the transport portion of the network. 08/23/2019 15:18:36 GMT - On August 23, 2019 at 15:04 GMT, CenturyLink identified a service impact in Des Moines, IA. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting efforts and details may evolve as the discovery process evolves through to service resolution, ticket closure, or post incident review. On Fri, Aug 23, 2019 at 10:31 AM Joelle Maslak <jmaslak@antelope.net<mailto:jmaslak@antelope.net>> wrote: Not national. 3356 and 209 are up in Denver. On Fri, Aug 23, 2019 at 9:27 AM Jeffrey Ollie via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Our CenturyLink services here in Iowa are down. Ethernet carriers are up, BGP sessions are up but we're not getting any BGP routes except 150 odd internal CenturyLink routes. Been waiting on hold with CL support for a while. Is anyone else seeing the same issue? -- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages -- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden. -- Jeff Ollie The majestik møøse is one of the mäni interesting furry animals in Sweden.
participants (4)
-
Brian R -
Jay R. Ashworth -
Jeffrey Ollie -
Joelle Maslak