Cogent cut on West Coast - seeing high latency to Arizona and East coast

From Cogent support at 10:22 pm Eastern on 5/13/2015 At this time, vendors are working to identify a splice point for repairs. There is currently no ETR. Please reference status.cogentco.com<http://status.cogentco.com/> for updates on this issue. To find answers to common technical questions you may have about your Cogent service, please review Cogent’s User Guide located athttp://www.cogentco.com/Guide/Global20Guide.pdf<http://www.cogentco.com/Guide/Global20Guide.pdf> To manage your account online go to https://ecogent.cogentco.com<https://ecogent.cogentco.com> Thank You, -----Original Message----- Dear Cogent Customer, We currently have a fiber cut on the west coast that is impacting many customers. You may see packet loss or latency until this is resolved. Our master ticket for this outage is 6157894<tel:6157894>. You can view updates at status.cogentco.com<http://status.cogentco.com/>. The information contained in this E-mail message is privileged, confidential, and may be protected from disclosure; please be aware that any other use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. If you think that you have received this E-mail message in error, please reply to the sender. This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. However, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by Kelley Drye & Warren LLP for any loss or damage arising in any way from its use.

This applies to Zayo too, I guess they share a southwest route. Been going on 24 hours now... ~Randy On 05/13/2015 7:55 pm, Silvey, David via Outages wrote:
From Cogent support at 10:22 pm Eastern on 5/13/2015
At this time, vendors are working to identify a splice point for repairs. There is currently no ETR. Please reference status.cogentco.com [2] for updates on this issue.
To find answers to common technical questions you may have about your Cogent service, please review Cogent's User Guide located athttp://www.cogentco.com/Guide/Global20Guide.pdf [3] To manage your account online go to https://ecogent.cogentco.com [4]
Thank You,
-----Original Message-----
Dear Cogent Customer,
We currently have a fiber cut on the west coast that is impacting many customers. You may see packet loss or latency until this is resolved. Our master ticket for this outage is 6157894 [5]. You can view updates at status.cogentco.com [2].
The information contained in this E-mail message is privileged, confidential, and may be protected from disclosure; please be aware that any other use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. If you think that you have received this E-mail message in error, please reply to the sender.
This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. However, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by Kelley Drye & Warren LLP for any loss or damage arising in any way from its use.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages [1]
Links: ------ [1] https://puck.nether.net/mailman/listinfo/outages [2] http://status.cogentco.com/ [3] http://www.cogentco.com/Guide/Global20Guide.pdf [4] https://ecogent.cogentco.com [5] tel:6157894

This is probably the same cut in Roll, AZ that Zayo suffered. There is a splicing crew on-site, but unfortunately it seems like the more they dig up and attempt to fix, the more they find wrong. Here’s the last update we received: "Our leased fiber provider are continuing to make repairs. We’ve been informed that the delay in restoration is due to a lack of fiber slack they thought was available to perform a butt-splice repair. They are currently installing a new splice enclosure in order to complete the cable repair. We’ve asked that Zayo fibers are prioritized once splicing begins.” There might also be some lingering issues with Telia due to the same fiber cut. We’re seeing 25-30% packet loss between Seattle and Phoenix via LAX. From: Randy McAnally via Outages Reply-To: <rsm@fast-serv.com> Date: Thursday, May 14, 2015 at 8:48 AM To: "Silvey, David" Cc: <outages@outages.org> Subject: Re: [outages] Cogent cut on West Coast - seeing high latency to Arizona and East coast

Can anyone confirm if this is now resolved or still ongoing? Cogent's status page shows everything is now normal. Thanks! On Thu, May 14, 2015 at 11:39 AM, Sean via Outages <outages@outages.org> wrote:
This is probably the same cut in Roll, AZ that Zayo suffered. There is a splicing crew on-site, but unfortunately it seems like the more they dig up and attempt to fix, the more they find wrong. Here’s the last update we received:
"Our leased fiber provider are continuing to make repairs. We’ve been informed that the delay in restoration is due to a lack of fiber slack they thought was available to perform a butt-splice repair. They are currently installing a new splice enclosure in order to complete the cable repair. We’ve asked that Zayo fibers are prioritized once splicing begins.”
There might also be some lingering issues with Telia due to the same fiber cut. We’re seeing 25-30% packet loss between Seattle and Phoenix via LAX.
From: Randy McAnally via Outages Reply-To: <rsm@fast-serv.com> Date: Thursday, May 14, 2015 at 8:48 AM To: "Silvey, David" Cc: <outages@outages.org> Subject: Re: [outages] Cogent cut on West Coast - seeing high latency to Arizona and East coast
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

It does indeed seem to be resolved. Zayo sent an "all clear" at about 9:30 MST yesterday evening, and all of my ping monitors (outbound over Cogent) normalized to expected levels shortly before then. --Jake

It seems to be resolved. Check http://status.cogentco.com/ On Fri, May 15, 2015 at 10:12 AM, Ryan Gelobter via Outages < outages@outages.org> wrote:
Can anyone confirm if this is now resolved or still ongoing? Cogent's status page shows everything is now normal.
Thanks!
On Thu, May 14, 2015 at 11:39 AM, Sean via Outages <outages@outages.org> wrote:
This is probably the same cut in Roll, AZ that Zayo suffered. There is a splicing crew on-site, but unfortunately it seems like the more they dig up and attempt to fix, the more they find wrong. Here’s the last update we received:
"Our leased fiber provider are continuing to make repairs. We’ve been informed that the delay in restoration is due to a lack of fiber slack they thought was available to perform a butt-splice repair. They are currently installing a new splice enclosure in order to complete the cable repair. We’ve asked that Zayo fibers are prioritized once splicing begins.”
There might also be some lingering issues with Telia due to the same fiber cut. We’re seeing 25-30% packet loss between Seattle and Phoenix via LAX.
From: Randy McAnally via Outages Reply-To: <rsm@fast-serv.com> Date: Thursday, May 14, 2015 at 8:48 AM To: "Silvey, David" Cc: <outages@outages.org> Subject: Re: [outages] Cogent cut on West Coast - seeing high latency to Arizona and East coast
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

OSP crews were off-hands as of 9:30PM MST last night. From: Rafael Possamai Date: Friday, May 15, 2015 at 8:17 AM To: Ryan Gelobter Cc: Sean, <outages@outages.org> Subject: Re: [outages] Cogent cut on West Coast - seeing high latency to Arizona and East coast It seems to be resolved. Check http://status.cogentco.com/ On Fri, May 15, 2015 at 10:12 AM, Ryan Gelobter via Outages <outages@outages.org> wrote: Can anyone confirm if this is now resolved or still ongoing? Cogent's status page shows everything is now normal. Thanks! On Thu, May 14, 2015 at 11:39 AM, Sean via Outages <outages@outages.org> wrote: This is probably the same cut in Roll, AZ that Zayo suffered. There is a splicing crew on-site, but unfortunately it seems like the more they dig up and attempt to fix, the more they find wrong. Here’s the last update we received: "Our leased fiber provider are continuing to make repairs. We’ve been informed that the delay in restoration is due to a lack of fiber slack they thought was available to perform a butt-splice repair. They are currently installing a new splice enclosure in order to complete the cable repair. We’ve asked that Zayo fibers are prioritized once splicing begins.” There might also be some lingering issues with Telia due to the same fiber cut. We’re seeing 25-30% packet loss between Seattle and Phoenix via LAX. From: Randy McAnally via Outages Reply-To: <rsm@fast-serv.com> Date: Thursday, May 14, 2015 at 8:48 AM To: "Silvey, David" Cc: <outages@outages.org> Subject: Re: [outages] Cogent cut on West Coast - seeing high latency to Arizona and East coast _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (6)
-
Jake Mertel
-
Rafael Possamai
-
Randy McAnally
-
Ryan Gelobter
-
Sean
-
Silvey, David