Centurylink Packet Loss - Socal
Anyone experiencing CL internet packet loss in the San Diego area? We started seeing a spike around 7am PDT. Pings to multiple remote sites show around 40% loss. Downdetector also shows some activity. Thanks, Bobin
We have been seeing CL packet loss since around 9:30am CDT. Mostly on their Level3 network, seems to be between Atlanta and Minneapolis for us. Perhaps a broader CL/L3 issue? On 4/21/20 10:36 AM, Samuel Jackson via Outages wrote:
Anyone experiencing CL internet packet loss in the San Diego area? We started seeing a spike around 7am PDT. Pings to multiple remote sites show around 40% loss.
Downdetector also shows some activity.
Thanks, Bobin /*ATTENTION: This email is from an external source. Always use caution when opening attachments or clicking links from unknown senders or when receiving unexpected emails.*/
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We are also seeing ~10% loss and ~100ms extra latency on CTL; between Spokane, WA and DC. It does appear to be occurring on the old L3 network. --Eric -----Original Message----- From: Outages <outages-bounces@outages.org> On Behalf Of Douglas K. Rand via Outages Sent: Tuesday, April 21, 2020 8:43 AM To: outages@outages.org Subject: ADV: Re: [outages] Centurylink Packet Loss - Socal We have been seeing CL packet loss since around 9:30am CDT. Mostly on their Level3 network, seems to be between Atlanta and Minneapolis for us. Perhaps a broader CL/L3 issue? On 4/21/20 10:36 AM, Samuel Jackson via Outages wrote:
Anyone experiencing CL internet packet loss in the San Diego area? We started seeing a spike around 7am PDT. Pings to multiple remote sites show around 40% loss.
Downdetector also shows some activity.
Thanks, Bobin /*ATTENTION: This email is from an external source. Always use caution when opening attachments or clicking links from unknown senders or when receiving unexpected emails.*/
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Got this update from CL. 04/21/2020 16:07:37 GMT - *** CASCADED EXTERNAL NOTES 21-Apr-2020 16:07:34 GMT From CASE: 18573666 - Event IP NOC reports that services are impacted by multiple fiber cuts that are impacting the CenturyLink network. Field Operations and all necessary repair personnel have arrived to the first fiber cut location and are currently working to pull in the slack on the fiber span. Approximately 3,000 feet of fiber will be required to pulled in. 04/21/2020 16:05:08 GMT - 04/21/2020 15:34:02 GMT - On April 21 at 13:55 GMT, CenturyLink identified a service impact in multiple markets impacting IP services. As this network fault is impacting multiple customers, the event has increased visibility with CenturyLink leadership. As such, client trouble ticket associated to this fault have been automatically escalated to higher priority. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting efforts and details may evolve as the discovery process evolves through to service resolution, ticket closure, or post incident review. ________________________________ From: Outages <outages-bounces@outages.org> on behalf of Eric Flanery via Outages <outages@outages.org> Sent: Tuesday, April 21, 2020 8:57 AM To: outages@outages.org <outages@outages.org> Subject: Re: [outages] Centurylink Packet Loss - Socal We are also seeing ~10% loss and ~100ms extra latency on CTL; between Spokane, WA and DC. It does appear to be occurring on the old L3 network. --Eric -----Original Message----- From: Outages <outages-bounces@outages.org> On Behalf Of Douglas K. Rand via Outages Sent: Tuesday, April 21, 2020 8:43 AM To: outages@outages.org Subject: ADV: Re: [outages] Centurylink Packet Loss - Socal We have been seeing CL packet loss since around 9:30am CDT. Mostly on their Level3 network, seems to be between Atlanta and Minneapolis for us. Perhaps a broader CL/L3 issue? On 4/21/20 10:36 AM, Samuel Jackson via Outages wrote:
Anyone experiencing CL internet packet loss in the San Diego area? We started seeing a spike around 7am PDT. Pings to multiple remote sites show around 40% loss.
Downdetector also shows some activity.
Thanks, Bobin /*ATTENTION: This email is from an external source. Always use caution when opening attachments or clicking links from unknown senders or when receiving unexpected emails.*/
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages Bobin Joseph Senior Network Engineer Direct +1 858-380-3052 lytx.com =============== CONFIDENTIALITY NOTICE: This e-mail and any attachments contain information from Lytx, Inc. and/or its affiliates, and are intended solely for the use of the named recipient(s). This e-mail may contain confidential information of Lytx and its customers. Any dissemination of this e-mail by anyone other than an intended recipient is strictly prohibited. If you are not a named recipient, you are prohibited from any further viewing of the e-mail or any attachments or from making any use of the e-mail or attachments. If you believe you have received this e-mail in error, notify the sender immediately and permanently delete the e-mail, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the e-mail or attachments.
Latest update... 04/21/2020 16:58:13 GMT - IP NOC reports teams at the first cut site continue working to pull slack which is being slowed by having to pump out a manhole. Prep work is in progress on one side of the cable. Teams at the second cut site have identified the damage caused by construction excavation. A repair team is on site assessing the situation to begin repairs. ________________________________ From: Outages <outages-bounces@outages.org> on behalf of Bobin Joseph via Outages <outages@outages.org> Sent: Tuesday, April 21, 2020 10:22 AM To: outages@outages.org <outages@outages.org>; Eric Flanery <eflanery@fsr.com> Subject: Re: [outages] Centurylink Packet Loss - Socal Got this update from CL. 04/21/2020 16:07:37 GMT - *** CASCADED EXTERNAL NOTES 21-Apr-2020 16:07:34 GMT From CASE: 18573666 - Event IP NOC reports that services are impacted by multiple fiber cuts that are impacting the CenturyLink network. Field Operations and all necessary repair personnel have arrived to the first fiber cut location and are currently working to pull in the slack on the fiber span. Approximately 3,000 feet of fiber will be required to pulled in. 04/21/2020 16:05:08 GMT - 04/21/2020 15:34:02 GMT - On April 21 at 13:55 GMT, CenturyLink identified a service impact in multiple markets impacting IP services. As this network fault is impacting multiple customers, the event has increased visibility with CenturyLink leadership. As such, client trouble ticket associated to this fault have been automatically escalated to higher priority. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting efforts and details may evolve as the discovery process evolves through to service resolution, ticket closure, or post incident review. Bobin Joseph Senior Network Engineer Direct +1 858-380-3052 www.lytx.com<https://www.lytx.com/en-us> [Lytx Logo]<https://www.lytx.com/en-us> [LinkedIn]<https://www.linkedin.com/company/lytxinc> [imageFacebook]<https://www.facebook.com/LytxInc> [imageTwitter]<https://twitter.com/lytx> [imageYouTube]<https://www.youtube.com/user/LytxInc> ----- CONFIDENTIALITY NOTICE: This e-mail and any attachments contain information from Lytx, Inc. and/or its affiliates, and are intended solely for the use of the named recipient(s). This e-mail may contain confidential information of Lytx and its customers. Any dissemination of this e-mail by anyone other than an intended recipient is strictly prohibited. If you are not a named recipient, you are prohibited from any further viewing of the e-mail or any attachments or from making any use of the e-mail or attachments. If you believe you have received this e-mail in error, notify the sender immediately and permanently delete the e-mail, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the e-mail or attachments. ________________________________ From: Outages <outages-bounces@outages.org> on behalf of Eric Flanery via Outages <outages@outages.org> Sent: Tuesday, April 21, 2020 8:57 AM To: outages@outages.org <outages@outages.org> Subject: Re: [outages] Centurylink Packet Loss - Socal We are also seeing ~10% loss and ~100ms extra latency on CTL; between Spokane, WA and DC. It does appear to be occurring on the old L3 network. --Eric -----Original Message----- From: Outages <outages-bounces@outages.org> On Behalf Of Douglas K. Rand via Outages Sent: Tuesday, April 21, 2020 8:43 AM To: outages@outages.org Subject: ADV: Re: [outages] Centurylink Packet Loss - Socal We have been seeing CL packet loss since around 9:30am CDT. Mostly on their Level3 network, seems to be between Atlanta and Minneapolis for us. Perhaps a broader CL/L3 issue? On 4/21/20 10:36 AM, Samuel Jackson via Outages wrote:
Anyone experiencing CL internet packet loss in the San Diego area? We started seeing a spike around 7am PDT. Pings to multiple remote sites show around 40% loss.
Downdetector also shows some activity.
Thanks, Bobin /*ATTENTION: This email is from an external source. Always use caution when opening attachments or clicking links from unknown senders or when receiving unexpected emails.*/
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We're seeing issues in both Tampa and Las Vegas with them. On 4/21/20, 11:52 AM, "Outages on behalf of Douglas K. Rand via Outages" <outages-bounces@outages.org on behalf of outages@outages.org> wrote: We have been seeing CL packet loss since around 9:30am CDT. Mostly on their Level3 network, seems to be between Atlanta and Minneapolis for us. Perhaps a broader CL/L3 issue? On 4/21/20 10:36 AM, Samuel Jackson via Outages wrote: > Anyone experiencing CL internet packet loss in the San Diego area? We > started seeing a spike around 7am PDT. Pings to multiple remote sites > show around 40% loss. > > Downdetector also shows some activity. > > Thanks, > Bobin > /*ATTENTION: This email is from an external source. Always use caution > when opening attachments or clicking links from unknown senders or when > receiving unexpected emails.*/ > > _______________________________________________ > Outages mailing list > Outages@outages.org > https://puck.nether.net/mailman/listinfo/outages > _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
The occasional batch of dropped packets in Arkansas, old L3 network. On Tue, Apr 21, 2020 at 11:22 AM David Hubbard via Outages < outages@outages.org> wrote:
We're seeing issues in both Tampa and Las Vegas with them.
On 4/21/20, 11:52 AM, "Outages on behalf of Douglas K. Rand via Outages" < outages-bounces@outages.org on behalf of outages@outages.org> wrote:
We have been seeing CL packet loss since around 9:30am CDT. Mostly on their Level3 network, seems to be between Atlanta and Minneapolis for us. Perhaps a broader CL/L3 issue?
On 4/21/20 10:36 AM, Samuel Jackson via Outages wrote: > Anyone experiencing CL internet packet loss in the San Diego area? We > started seeing a spike around 7am PDT. Pings to multiple remote sites > show around 40% loss. > > Downdetector also shows some activity. > > Thanks, > Bobin > /*ATTENTION: This email is from an external source. Always use caution > when opening attachments or clicking links from unknown senders or when > receiving unexpected emails.*/ > > _______________________________________________ > Outages mailing list > Outages@outages.org > https://puck.nether.net/mailman/listinfo/outages >
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Do you see packet loss on the point to point links or transit peers? On Tue, Apr 21, 2020 at 9:46 AM Erik Lentz via Outages <outages@outages.org> wrote:
The occasional batch of dropped packets in Arkansas, old L3 network.
On Tue, Apr 21, 2020 at 11:22 AM David Hubbard via Outages < outages@outages.org> wrote:
We're seeing issues in both Tampa and Las Vegas with them.
On 4/21/20, 11:52 AM, "Outages on behalf of Douglas K. Rand via Outages" <outages-bounces@outages.org on behalf of outages@outages.org> wrote:
We have been seeing CL packet loss since around 9:30am CDT. Mostly on their Level3 network, seems to be between Atlanta and Minneapolis for us. Perhaps a broader CL/L3 issue?
On 4/21/20 10:36 AM, Samuel Jackson via Outages wrote: > Anyone experiencing CL internet packet loss in the San Diego area? We > started seeing a spike around 7am PDT. Pings to multiple remote sites > show around 40% loss. > > Downdetector also shows some activity. > > Thanks, > Bobin > /*ATTENTION: This email is from an external source. Always use caution > when opening attachments or clicking links from unknown senders or when > receiving unexpected emails.*/ > > _______________________________________________ > Outages mailing list > Outages@outages.org > https://puck.nether.net/mailman/listinfo/outages >
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Onur Karaagaoglu
participants (8)
-
Bobin Joseph -
David Hubbard -
Douglas K. Rand -
Eric Flanery -
Erik Lentz -
Onur Karaagaoglu -
Phillip Alger -
Samuel Jackson