Anyone seen ETR yet? --Karl Karl Putland Senior Engineer *SimpleSignal* Anywhere: 303-242-8608 <http://www.simplesignal.com/explainer_video.php> On Thu, Jan 23, 2014 at 9:16 PM, Hugh Smallwood <hugh@ongoingoperations.com>wrote:
We have MPLS in CA MD VA all down
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I just had one of 2 of my circuits pop back up. On Fri, Jan 24, 2014 at 12:38 AM, SML <sml@lordsargon.com> wrote:
On 23 Jan 2014, at 23:34, Karl Putland wrote:
Anyone seen ETR yet?
My circuits are coming up.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Ditto, recovery in one location at 9:25 PM PT. Still down hard in a few others. From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Mitch Patterson Sent: Thursday, January 23, 2014 9:40 PM To: SML Cc: Outages@outages.org Subject: Re: [outages] CenturyLink outage? I just had one of 2 of my circuits pop back up. On Fri, Jan 24, 2014 at 12:38 AM, SML <sml@lordsargon.com<mailto:sml@lordsargon.com>> wrote: On 23 Jan 2014, at 23:34, Karl Putland wrote:
Anyone seen ETR yet?
My circuits are coming up. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
We were seeing DIA part of Enhanced ports come back up, but we weren't getting routes from the CUGs/VRFs. Looks like BGP session had gone stale. Kicked them all and we're back. --Karl Karl Putland Senior Engineer *SimpleSignal* Anywhere: 303-242-8608 <http://www.simplesignal.com/explainer_video.php> On Thu, Jan 23, 2014 at 10:46 PM, Steve Dispensa < Steve.Dispensa@microsoft.com> wrote:
Ditto, recovery in one location at 9:25 PM PT. Still down hard in a few others.
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Mitch Patterson *Sent:* Thursday, January 23, 2014 9:40 PM *To:* SML *Cc:* Outages@outages.org
*Subject:* Re: [outages] CenturyLink outage?
I just had one of 2 of my circuits pop back up.
On Fri, Jan 24, 2014 at 12:38 AM, SML <sml@lordsargon.com> wrote:
On 23 Jan 2014, at 23:34, Karl Putland wrote:
Anyone seen ETR yet?
My circuits are coming up.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Anyone see an RFO for this yet? [OCTI-Logo.gif] Bryan Riley Operations Manager 757.802.4991 (Anywhere) 757.351.4115 (fax) 757.802.4955 (main) www.oncalltelecom.com<http://www.oncalltelecom.com> From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Karl Putland Sent: Friday, January 24, 2014 12:58 AM To: Outages@outages.org Subject: Re: [outages] CenturyLink outage? We were seeing DIA part of Enhanced ports come back up, but we weren't getting routes from the CUGs/VRFs. Looks like BGP session had gone stale. Kicked them all and we're back. --Karl Karl Putland Senior Engineer SimpleSignal Anywhere: 303-242-8608<tel:303-242-8608> [Image removed by sender.]<http://www.simplesignal.com/explainer_video.php> [Image removed by sender.] On Thu, Jan 23, 2014 at 10:46 PM, Steve Dispensa <Steve.Dispensa@microsoft.com<mailto:Steve.Dispensa@microsoft.com>> wrote: Ditto, recovery in one location at 9:25 PM PT. Still down hard in a few others. From: Outages [mailto:outages-bounces@outages.org<mailto:outages-bounces@outages.org>] On Behalf Of Mitch Patterson Sent: Thursday, January 23, 2014 9:40 PM To: SML Cc: Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] CenturyLink outage? I just had one of 2 of my circuits pop back up. On Fri, Jan 24, 2014 at 12:38 AM, SML <sml@lordsargon.com<mailto:sml@lordsargon.com>> wrote: On 23 Jan 2014, at 23:34, Karl Putland wrote:
Anyone seen ETR yet?
My circuits are coming up. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Below is what we got. Last night from around 11:30pm until 2:30am<x-apple-data-detectors://0> there was a widespread outage on the MPLS backbone that impacted customers coast to coast. Tier 3 rebooted 125+ routers throughout the network to restore service, though a handful of customers may have taken slightly longer. The issue has been escalated and engineering is continuing to work to find out what caused the problem. As of now the network is stable, unfortunately I don't have much more info at this time to share. From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Bryan Riley Sent: Friday, January 24, 2014 10:37 AM To: Outages@outages.org Subject: Re: [outages] CenturyLink outage? Anyone see an RFO for this yet? [OCTI-Logo.gif] Bryan Riley Operations Manager 757.802.4991 (Anywhere) 757.351.4115 (fax) 757.802.4955 (main) www.oncalltelecom.com<http://www.oncalltelecom.com> From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Karl Putland Sent: Friday, January 24, 2014 12:58 AM To: Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] CenturyLink outage? We were seeing DIA part of Enhanced ports come back up, but we weren't getting routes from the CUGs/VRFs. Looks like BGP session had gone stale. Kicked them all and we're back. --Karl Karl Putland Senior Engineer SimpleSignal Anywhere: 303-242-8608<tel:303-242-8608> [Image removed by sender.]<http://www.simplesignal.com/explainer_video.php> [Image removed by sender.] On Thu, Jan 23, 2014 at 10:46 PM, Steve Dispensa <Steve.Dispensa@microsoft.com<mailto:Steve.Dispensa@microsoft.com>> wrote: Ditto, recovery in one location at 9:25 PM PT. Still down hard in a few others. From: Outages [mailto:outages-bounces@outages.org<mailto:outages-bounces@outages.org>] On Behalf Of Mitch Patterson Sent: Thursday, January 23, 2014 9:40 PM To: SML Cc: Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] CenturyLink outage? I just had one of 2 of my circuits pop back up. On Fri, Jan 24, 2014 at 12:38 AM, SML <sml@lordsargon.com<mailto:sml@lordsargon.com>> wrote: On 23 Jan 2014, at 23:34, Karl Putland wrote:
Anyone seen ETR yet?
My circuits are coming up. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
participants (6)
-
Bryan Riley -
Hugh Smallwood -
Karl Putland -
Mitch Patterson -
SML -
Steve Dispensa