CenturyLink 10G Waves Down Between PA and FL
Anyone else having issues with connectivity between PA and FL today? We've got two, diverse 10G waves from CenturyLink and both are down at the moment. I swear... these have been the WORST circuits. We haven't had a single day where at least one of them is stable for 24+ hrs... This message is intended solely for the designated recipient(s). It may contain confidential or proprietary information and may be subject to attorney-client privilege or other confidentiality protections. If you are not a designated recipient you may not review, copy or distribute this message. If you receive this in error, please notify the sender by reply e-mail and delete this message. Thank you.
We have a circuit between Louisiana and Atlanta, GA that is currently flapping. It started flapping around 8:49am CST. /ns From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Alex Lembesis via Outages Sent: Friday, May 31, 2019 9:47 AM To: outages@outages.org Subject: [outages] CenturyLink 10G Waves Down Between PA and FL Anyone else having issues with connectivity between PA and FL today? We’ve got two, diverse 10G waves from CenturyLink and both are down at the moment. I swear… these have been the WORST circuits. We haven’t had a single day where at least one of them is stable for 24+ hrs… This message is intended solely for the designated recipient(s). It may contain confidential or proprietary information and may be subject to attorney-client privilege or other confidentiality protections. If you are not a designated recipient you may not review, copy or distribute this message. If you receive this in error, please notify the sender by reply e-mail and delete this message. Thank you.
Those are legacy L3 waves though, but the ATL link does follow the coast from LA to Tallahassee before turning north to Atlanta. So it could be related. Luke Ns From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Kody Vicknair via Outages Sent: Friday, May 31, 2019 10:25 AM To: Alex Lembesis Cc: outages@outages.org Subject: Re: [outages] CenturyLink 10G Waves Down Between PA and FL We have a circuit between Louisiana and Atlanta, GA that is currently flapping. It started flapping around 8:49am CST. /ns From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Alex Lembesis via Outages Sent: Friday, May 31, 2019 9:47 AM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] CenturyLink 10G Waves Down Between PA and FL Anyone else having issues with connectivity between PA and FL today? We’ve got two, diverse 10G waves from CenturyLink and both are down at the moment. I swear… these have been the WORST circuits. We haven’t had a single day where at least one of them is stable for 24+ hrs… This message is intended solely for the designated recipient(s). It may contain confidential or proprietary information and may be subject to attorney-client privilege or other confidentiality protections. If you are not a designated recipient you may not review, copy or distribute this message. If you receive this in error, please notify the sender by reply e-mail and delete this message. Thank you.
Please share more info if anyone knows outage details so i can try to include in visuals in infrapedia as accurate possible On Fri, May 31, 2019 at 08:47 Luke Guillory via Outages <outages@outages.org> wrote:
Those are legacy L3 waves though, but the ATL link does follow the coast from LA to Tallahassee before turning north to Atlanta. So it could be related.
Luke
Ns
*From:* Outages [mailto:outages-bounces@outages.org] * On Behalf Of *Kody Vicknair via Outages *Sent:* Friday, May 31, 2019 10:25 AM *To:* Alex Lembesis *Cc:* outages@outages.org *Subject:* Re: [outages] CenturyLink 10G Waves Down Between PA and FL
We have a circuit between Louisiana and Atlanta, GA that is currently flapping. It started flapping around 8:49am CST.
/ns
*From:* Outages [mailto:outages-bounces@outages.org <outages-bounces@outages.org>] *On Behalf Of *Alex Lembesis via Outages *Sent:* Friday, May 31, 2019 9:47 AM *To:* outages@outages.org *Subject:* [outages] CenturyLink 10G Waves Down Between PA and FL
Anyone else having issues with connectivity between PA and FL today? We’ve got two, diverse 10G waves from CenturyLink and both are down at the moment. I swear… these have been the WORST circuits. We haven’t had a single day where at least one of them is stable for 24+ hrs…
This message is intended solely for the designated recipient(s). It may contain confidential or proprietary information and may be subject to attorney-client privilege or other confidentiality protections. If you are not a designated recipient you may not review, copy or distribute this message. If you receive this in error, please notify the sender by reply e-mail and delete this message. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Mehmet +1-424-298-1903
All updates I’ve received from provider below: The local provider has located the correct fiber assignments and is actively working on dispatching repair teams. The local provider has not yet responded with an update and the issue has been escalated to the VP level. The Transport NOC has escalated to the Third Level with the local provider because the local provider is unable to confirm if their field technician is on site or if they have tested the fibers. The Outside Plant team has confirmed the fiber cut is on the local providers 432-count fiber cable. The Transport NOC has escalated with the local provider for status. Light level testing identified fiber damage 4.74 miles from the test site. Field Operations has sent this data to the Outside Plant team who will provide to the local provider and assist with their isolation efforts. Please find the update from Vendor; 05/31/2019 16:05:11 GMT - The Transport NOC have isolated part of the issue to be on a local provider's network, and the local provider has been engaged. Field Operations is in route to the test site to further isolate the fiber portion on the CenturyLink Network with an estimated time of arrival of 16:30 GMT. 05/31/2019 15:33:38 GMT - The Transport NOC has identified a degrading fiber span between two nodes and has engaged Field Operations to further investigate. 05/31/2019 15:25:25 GMT - On May 31, 2019 at 15:20 GMT CenturyLink identified a service impact between Atlanta, GA and College Park, GA. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve. From: Mehmet Akcin [mailto:mehmet@akcin.net] Sent: Friday, May 31, 2019 11:16 AM To: Luke Guillory <lguillory@reservetele.com> Cc: Alex Lembesis <Alex.Lembesis@tevapharm.com>; Kody Vicknair <kvicknair@reservetele.com>; outages@outages.org Subject: Re: [outages] CenturyLink 10G Waves Down Between PA and FL Please share more info if anyone knows outage details so i can try to include in visuals in infrapedia as accurate possible On Fri, May 31, 2019 at 08:47 Luke Guillory via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Those are legacy L3 waves though, but the ATL link does follow the coast from LA to Tallahassee before turning north to Atlanta. So it could be related. Luke Ns From: Outages [mailto:outages-bounces@outages.org<mailto:outages-bounces@outages.org>] On Behalf Of Kody Vicknair via Outages Sent: Friday, May 31, 2019 10:25 AM To: Alex Lembesis Cc: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] CenturyLink 10G Waves Down Between PA and FL We have a circuit between Louisiana and Atlanta, GA that is currently flapping. It started flapping around 8:49am CST. /ns From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Alex Lembesis via Outages Sent: Friday, May 31, 2019 9:47 AM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] CenturyLink 10G Waves Down Between PA and FL Anyone else having issues with connectivity between PA and FL today? We’ve got two, diverse 10G waves from CenturyLink and both are down at the moment. I swear… these have been the WORST circuits. We haven’t had a single day where at least one of them is stable for 24+ hrs… This message is intended solely for the designated recipient(s). It may contain confidential or proprietary information and may be subject to attorney-client privilege or other confidentiality protections. If you are not a designated recipient you may not review, copy or distribute this message. If you receive this in error, please notify the sender by reply e-mail and delete this message. Thank you. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages -- Mehmet +1-424-298-1903
Please see below the RFO for this outage: ################################################################################################################################## Event Conclusion Summary Outage Start: May 31, 2019 15:00 GMT Outage Stop: May 31, 2019 21:45 GMT Root Cause: A loose connector on a local provider's network between Atlanta, GA and College Park, GA was impacting unprotected services. Fix Action: The connection was secured, and the equipment rebooted, thus restoring services. Reason for Outage (RFO) Summary: On May 26, 2019, at 16:38 GMT CenturyLink identified a service impact between Atlanta, GA and College Park, GA. The Transport NOC was engaged and investigated in order to isolate the cause. The Transport NOC escalated with the local provider for status when no updated information was received and again to the Third Level with the local provider because the local provider was unable to confirm if their field technician were on site or if they had tested the fibers. The Outside Plant team confirmed the fiber issue was on the local providers 432-count fiber cable. The local provider had not yet responded with an update and the issue was escalated to the VP level and the local provider reported multiple teams were reviewing the issue. The local provider located the correct fiber assignments and was actively working on dispatching repair teams. Services appeared to have restored; however, alarms remained. The Transport NOC reset multiple cards and services were restored and continue to be monitored for stability. This service impact has concluded; if additional issues are experienced, please contact the CenturyLink Repair Center. This ticket will be closed in 24 hours. At that time, a customer satisfaction survey link may be available. We strive to provide thorough communications containing the available information during a service disruption. Please let us know if the updates you received during this event were satisfactory. From: Mehmet Akcin [mailto:mehmet@akcin.net] Sent: Friday, May 31, 2019 11:16 AM To: Luke Guillory <lguillory@reservetele.com> Cc: Alex Lembesis <Alex.Lembesis@tevapharm.com>; Kody Vicknair <kvicknair@reservetele.com>; outages@outages.org Subject: Re: [outages] CenturyLink 10G Waves Down Between PA and FL Please share more info if anyone knows outage details so i can try to include in visuals in infrapedia as accurate possible On Fri, May 31, 2019 at 08:47 Luke Guillory via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Those are legacy L3 waves though, but the ATL link does follow the coast from LA to Tallahassee before turning north to Atlanta. So it could be related. Luke Ns From: Outages [mailto:outages-bounces@outages.org<mailto:outages-bounces@outages.org>] On Behalf Of Kody Vicknair via Outages Sent: Friday, May 31, 2019 10:25 AM To: Alex Lembesis Cc: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] CenturyLink 10G Waves Down Between PA and FL We have a circuit between Louisiana and Atlanta, GA that is currently flapping. It started flapping around 8:49am CST. /ns From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Alex Lembesis via Outages Sent: Friday, May 31, 2019 9:47 AM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] CenturyLink 10G Waves Down Between PA and FL Anyone else having issues with connectivity between PA and FL today? We’ve got two, diverse 10G waves from CenturyLink and both are down at the moment. I swear… these have been the WORST circuits. We haven’t had a single day where at least one of them is stable for 24+ hrs… This message is intended solely for the designated recipient(s). It may contain confidential or proprietary information and may be subject to attorney-client privilege or other confidentiality protections. If you are not a designated recipient you may not review, copy or distribute this message. If you receive this in error, please notify the sender by reply e-mail and delete this message. Thank you. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages -- Mehmet +1-424-298-1903
Confirmed fiber cut in Wilmington, NC causing the issue on one of our circuits. CenturyLink is investigating the issue on our other circuit, but the problem has been isolated between Atlanta, GA and College Park, GA. Field techs are enroute. From: Luke Guillory (External) [mailto:lguillory@reservetele.com] Sent: Friday, May 31, 2019 11:48 AM To: Kody Vicknair; Alex Lembesis Cc: outages@outages.org Subject: RE: CenturyLink 10G Waves Down Between PA and FL Those are legacy L3 waves though, but the ATL link does follow the coast from LA to Tallahassee before turning north to Atlanta. So it could be related. Luke Ns From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Kody Vicknair via Outages Sent: Friday, May 31, 2019 10:25 AM To: Alex Lembesis Cc: outages@outages.org Subject: Re: [outages] CenturyLink 10G Waves Down Between PA and FL We have a circuit between Louisiana and Atlanta, GA that is currently flapping. It started flapping around 8:49am CST. /ns From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Alex Lembesis via Outages Sent: Friday, May 31, 2019 9:47 AM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] CenturyLink 10G Waves Down Between PA and FL Anyone else having issues with connectivity between PA and FL today? We’ve got two, diverse 10G waves from CenturyLink and both are down at the moment. I swear… these have been the WORST circuits. We haven’t had a single day where at least one of them is stable for 24+ hrs… This message is intended solely for the designated recipient(s). It may contain confidential or proprietary information and may be subject to attorney-client privilege or other confidentiality protections. If you are not a designated recipient you may not review, copy or distribute this message. If you receive this in error, please notify the sender by reply e-mail and delete this message. Thank you. This message is intended solely for the designated recipient(s). It may contain confidential or proprietary information and may be subject to attorney-client privilege or other confidentiality protections. If you are not a designated recipient you may not review, copy or distribute this message. If you receive this in error, please notify the sender by reply e-mail and delete this message. Thank you.
participants (4)
-
Alex Lembesis -
Kody Vicknair -
Luke Guillory -
Mehmet Akcin