Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance. -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral. -------- Original message -------- From: Jay Ashworth via Outages <outages@outages.org> Date: 8/30/20 9:36 AM (GMT-06:00) To: outages@outages.org Subject: [outages] 3556 problems Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance. -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Not on that ticket, so thanks a ton for the update! Execs are asking questions about what's going on and it's nice to give them feedback! On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages < outages@outages.org> wrote:
For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403
08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.
-------- Original message -------- From: Jay Ashworth via Outages <outages@outages.org> Date: 8/30/20 9:36 AM (GMT-06:00) To: outages@outages.org Subject: [outages] 3556 problems
Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Here is their latest update: The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance. Next update by: 2020-08-30 15:50 GMT This notification is sent from an unmonitored email account. Please click the link below to reply: mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A Notes History: 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance. 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral. 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time. 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time. 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve. From: Michael McDonald <mmcdonald@liquidweb.com> Sent: Sunday, August 30, 2020 9:56 AM To: Dylan Ebner <dylan.ebner@crlmed.com> Cc: Jay Ashworth <jra@baylink.com>; outages@outages.org Subject: Re: [outages] 3556 problems Not on that ticket, so thanks a ton for the update! Execs are asking questions about what's going on and it's nice to give them feedback! On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral. -------- Original message -------- From: Jay Ashworth via Outages <outages@outages.org<mailto:outages@outages.org>> Date: 8/30/20 9:36 AM (GMT-06:00) To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] 3556 problems Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Not sure if CenturyLink is facing Bogon routes issue, we are impacted intermittently on AS174 however we are accepting default route from this peer https://bgp.he.net/AS3356#_graph4 https://bgp.he.net/report/bogons#_bogonsv4asn Sent from my iPhone #Covid-19 (Be Safe | Maintain Social Distancing) On 30-Aug-2020, at 8:33 PM, Dylan Ebner via Outages <outages@outages.org> wrote: Here is their latest update: The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance. Next update by: 2020-08-30 15:50 GMT This notification is sent from an unmonitored email account. Please click the link below to reply: mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A Notes History: 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance. 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral. 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time. 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time. 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve. From: Michael McDonald <mmcdonald@liquidweb.com> Sent: Sunday, August 30, 2020 9:56 AM To: Dylan Ebner <dylan.ebner@crlmed.com> Cc: Jay Ashworth <jra@baylink.com>; outages@outages.org Subject: Re: [outages] 3556 problems Not on that ticket, so thanks a ton for the update! Execs are asking questions about what's going on and it's nice to give them feedback! On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral. -------- Original message -------- From: Jay Ashworth via Outages <outages@outages.org<mailto:outages@outages.org>> Date: 8/30/20 9:36 AM (GMT-06:00) To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] 3556 problems Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages<https://puck.nether.net/mailman/listinfo/outages> _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages This message is subject to Fortitude Law Associates's email communication policy. FLA-Disclaimer: https://protect-us.mimecast.com/s/hbb3COYERvH2nNq7hE2Z6w
It's recovering right now (for me , Munich). On 30 Aug 2020, at 16:59, Dylan Ebner via Outages wrote:
Here is their latest update:
The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.
Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.
Sent from my iPhone On Aug 30, 2020, at 10:21 AM, Jörg Kost via Outages <outages@outages.org> wrote: It's recovering right now (for me , Munich).
On 30 Aug 2020, at 16:59, Dylan Ebner via Outages wrote:
Here is their latest update:
The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.
Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Latest update from CenturyLink Notes History: 08/30/2020 15:31:25 GMT - The IP NOC continues to observe service affecting alarms clearing as the configuration that was deployed continues to propagate through the affected devices. Once affected services have been confirmed restored and stable, a final notification will be provided. 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance. 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral. 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time. 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time. 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve. Regards, Stephen Flynn Atlantic.Net Direct: (321) 206-1390 sflynn@staff.atlantic.net<mailto:sflynn@staff.atlantic.net> www.atlantic.net<http://www.atlantic.net/> From: Outages <outages-bounces@outages.org> On Behalf Of Dylan Ebner via Outages Sent: Sunday, August 30, 2020 11:00 AM To: Michael McDonald <mmcdonald@liquidweb.com> Cc: outages@outages.org Subject: Re: [outages] 3556 problems CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Here is their latest update: The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance. Next update by: 2020-08-30 15:50 GMT This notification is sent from an unmonitored email account. Please click the link below to reply: mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A Notes History: 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance. 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral. 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time. 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time. 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve. From: Michael McDonald <mmcdonald@liquidweb.com<mailto:mmcdonald@liquidweb.com>> Sent: Sunday, August 30, 2020 9:56 AM To: Dylan Ebner <dylan.ebner@crlmed.com<mailto:dylan.ebner@crlmed.com>> Cc: Jay Ashworth <jra@baylink.com<mailto:jra@baylink.com>>; outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] 3556 problems Not on that ticket, so thanks a ton for the update! Execs are asking questions about what's going on and it's nice to give them feedback! On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral. -------- Original message -------- From: Jay Ashworth via Outages <outages@outages.org<mailto:outages@outages.org>> Date: 8/30/20 9:36 AM (GMT-06:00) To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] 3556 problems Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
According to Centurylink tweet, their service has been restored. Can someone help me confirm that this is true before I start shifting the traffic back? From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Stephen Flynn via Outages Sent: Sunday, August 30, 2020 8:41 AM To: outages@outages.org Subject: Re: [outages] 3556 problems Latest update from CenturyLink Notes History: 08/30/2020 15:31:25 GMT - The IP NOC continues to observe service affecting alarms clearing as the configuration that was deployed continues to propagate through the affected devices. Once affected services have been confirmed restored and stable, a final notification will be provided. 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance. 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral. 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time. 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time. 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve. Regards, Stephen Flynn Atlantic.Net Direct: (321) 206-1390 sflynn@staff.atlantic.net <mailto:sflynn@staff.atlantic.net> www.atlantic.net <http://www.atlantic.net/> From: Outages <outages-bounces@outages.org <mailto:outages-bounces@outages.org> > On Behalf Of Dylan Ebner via Outages Sent: Sunday, August 30, 2020 11:00 AM To: Michael McDonald <mmcdonald@liquidweb.com <mailto:mmcdonald@liquidweb.com> > Cc: outages@outages.org <mailto:outages@outages.org> Subject: Re: [outages] 3556 problems CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Here is their latest update: The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance. Next update by: 2020-08-30 15:50 GMT This notification is sent from an unmonitored email account. Please click the link below to reply: mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403, <mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A> &body=Customer%20Reply%3A%0D%0A Notes History: 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance. 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral. 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time. 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time. 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve. From: Michael McDonald <mmcdonald@liquidweb.com <mailto:mmcdonald@liquidweb.com> > Sent: Sunday, August 30, 2020 9:56 AM To: Dylan Ebner <dylan.ebner@crlmed.com <mailto:dylan.ebner@crlmed.com> > Cc: Jay Ashworth <jra@baylink.com <mailto:jra@baylink.com> >; outages@outages.org <mailto:outages@outages.org> Subject: Re: [outages] 3556 problems Not on that ticket, so thanks a ton for the update! Execs are asking questions about what's going on and it's nice to give them feedback! On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral. -------- Original message -------- From: Jay Ashworth via Outages <outages@outages.org <mailto:outages@outages.org> > Date: 8/30/20 9:36 AM (GMT-06:00) To: outages@outages.org <mailto:outages@outages.org> Subject: [outages] 3556 problems Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
We were effected by this in reaching the Google SAS. It's been working without problems for 100 minutes. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Sun, Aug 30, 2020, 12:38 PM Tim Choe via Outages <outages@outages.org> wrote:
According to Centurylink tweet, their service has been restored. Can someone help me confirm that this is true before I start shifting the traffic back?
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Stephen Flynn via Outages *Sent:* Sunday, August 30, 2020 8:41 AM *To:* outages@outages.org *Subject:* Re: [outages] 3556 problems
Latest update from CenturyLink
Notes History:
08/30/2020 15:31:25 GMT - The IP NOC continues to observe service affecting alarms clearing as the configuration that was deployed continues to propagate through the affected devices. Once affected services have been confirmed restored and stable, a final notification will be provided.
08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.
Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.
08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.
08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time.
08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time.
08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.
The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.
Regards,
Stephen Flynn Atlantic.Net Direct: (321) 206-1390 sflynn@staff.atlantic.net www.atlantic.net
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Dylan Ebner via Outages *Sent:* Sunday, August 30, 2020 11:00 AM *To:* Michael McDonald <mmcdonald@liquidweb.com> *Cc:* outages@outages.org *Subject:* Re: [outages] 3556 problems
*CAUTION:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Here is their latest update:
The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.
Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.
Next update by: 2020-08-30 15:50 GMT
This notification is sent from an unmonitored email account. Please click the link below to reply:
mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A <SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A>
Notes History:
08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.
Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.
08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.
08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time.
08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time.
08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.
The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.
*From:* Michael McDonald <mmcdonald@liquidweb.com> *Sent:* Sunday, August 30, 2020 9:56 AM *To:* Dylan Ebner <dylan.ebner@crlmed.com> *Cc:* Jay Ashworth <jra@baylink.com>; outages@outages.org *Subject:* Re: [outages] 3556 problems
Not on that ticket, so thanks a ton for the update! Execs are asking questions about what's going on and it's nice to give them feedback!
On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages < outages@outages.org> wrote:
For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403
08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.
-------- Original message --------
From: Jay Ashworth via Outages <outages@outages.org>
Date: 8/30/20 9:36 AM (GMT-06:00)
To: outages@outages.org
Subject: [outages] 3556 problems
Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance. -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I just shifted back and everything looks normal. I'm in Waltham MA. On Sun, Aug 30, 2020 at 12:44 PM Josh Luthman via Outages < outages@outages.org> wrote:
We were effected by this in reaching the Google SAS. It's been working without problems for 100 minutes.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Sun, Aug 30, 2020, 12:38 PM Tim Choe via Outages <outages@outages.org> wrote:
According to Centurylink tweet, their service has been restored. Can someone help me confirm that this is true before I start shifting the traffic back?
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Stephen Flynn via Outages *Sent:* Sunday, August 30, 2020 8:41 AM *To:* outages@outages.org *Subject:* Re: [outages] 3556 problems
Latest update from CenturyLink
Notes History:
08/30/2020 15:31:25 GMT - The IP NOC continues to observe service affecting alarms clearing as the configuration that was deployed continues to propagate through the affected devices. Once affected services have been confirmed restored and stable, a final notification will be provided.
08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.
Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.
08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.
08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time.
08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time.
08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.
The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.
Regards,
Stephen Flynn Atlantic.Net Direct: (321) 206-1390 sflynn@staff.atlantic.net www.atlantic.net
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Dylan Ebner via Outages *Sent:* Sunday, August 30, 2020 11:00 AM *To:* Michael McDonald <mmcdonald@liquidweb.com> *Cc:* outages@outages.org *Subject:* Re: [outages] 3556 problems
*CAUTION:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Here is their latest update:
The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.
Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.
Next update by: 2020-08-30 15:50 GMT
This notification is sent from an unmonitored email account. Please click the link below to reply:
mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A <SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A>
Notes History:
08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.
Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.
08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.
08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time.
08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time.
08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.
The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.
*From:* Michael McDonald <mmcdonald@liquidweb.com> *Sent:* Sunday, August 30, 2020 9:56 AM *To:* Dylan Ebner <dylan.ebner@crlmed.com> *Cc:* Jay Ashworth <jra@baylink.com>; outages@outages.org *Subject:* Re: [outages] 3556 problems
Not on that ticket, so thanks a ton for the update! Execs are asking questions about what's going on and it's nice to give them feedback!
On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages < outages@outages.org> wrote:
For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403
08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.
-------- Original message --------
From: Jay Ashworth via Outages <outages@outages.org>
Date: 8/30/20 9:36 AM (GMT-06:00)
To: outages@outages.org
Subject: [outages] 3556 problems
Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance. -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com
“Can someone help me confirm that this is true before I start shifting the traffic back?” I won’t be activating our links until this evenings maintenance window. Regards, Stephen Flynn Atlantic.Net Direct: (321) 206-1390 sflynn@staff.atlantic.net<mailto:sflynn@staff.atlantic.net> www.atlantic.net<http://www.atlantic.net/> From: Outages <outages-bounces@outages.org> On Behalf Of Jason Kuehl via Outages Sent: Sunday, August 30, 2020 12:51 PM To: Josh Luthman <josh@imaginenetworksllc.com> Cc: OUTAGES Mailing List <outages@outages.org> Subject: Re: [outages] 3556 problems CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. I just shifted back and everything looks normal. I'm in Waltham MA. On Sun, Aug 30, 2020 at 12:44 PM Josh Luthman via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: We were effected by this in reaching the Google SAS. It's been working without problems for 100 minutes. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Sun, Aug 30, 2020, 12:38 PM Tim Choe via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: According to Centurylink tweet, their service has been restored. Can someone help me confirm that this is true before I start shifting the traffic back? From: Outages [mailto:outages-bounces@outages.org<mailto:outages-bounces@outages.org>] On Behalf Of Stephen Flynn via Outages Sent: Sunday, August 30, 2020 8:41 AM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] 3556 problems Latest update from CenturyLink Notes History: 08/30/2020 15:31:25 GMT - The IP NOC continues to observe service affecting alarms clearing as the configuration that was deployed continues to propagate through the affected devices. Once affected services have been confirmed restored and stable, a final notification will be provided. 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance. 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral. 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time. 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time. 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve. Regards, Stephen Flynn Atlantic.Net Direct: (321) 206-1390 sflynn@staff.atlantic.net<mailto:sflynn@staff.atlantic.net> www.atlantic.net<http://www.atlantic.net/> From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Dylan Ebner via Outages Sent: Sunday, August 30, 2020 11:00 AM To: Michael McDonald <mmcdonald@liquidweb.com<mailto:mmcdonald@liquidweb.com>> Cc: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] 3556 problems CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Here is their latest update: The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance. Next update by: 2020-08-30 15:50 GMT This notification is sent from an unmonitored email account. Please click the link below to reply: mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A Notes History: 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance. 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral. 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time. 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time. 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve. From: Michael McDonald <mmcdonald@liquidweb.com<mailto:mmcdonald@liquidweb.com>> Sent: Sunday, August 30, 2020 9:56 AM To: Dylan Ebner <dylan.ebner@crlmed.com<mailto:dylan.ebner@crlmed.com>> Cc: Jay Ashworth <jra@baylink.com<mailto:jra@baylink.com>>; outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] 3556 problems Not on that ticket, so thanks a ton for the update! Execs are asking questions about what's going on and it's nice to give them feedback! On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral. -------- Original message -------- From: Jay Ashworth via Outages <outages@outages.org<mailto:outages@outages.org>> Date: 8/30/20 9:36 AM (GMT-06:00) To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] 3556 problems Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages -- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com<mailto:jason.w.kuehl@gmail.com>
DqE Communications unpeered with Level3 this morning and advised they will not be repeering until Tues am as Level3 will not confirm SLAs at this time. ________________________________ From: Outages <outages-bounces@outages.org> on behalf of Josh Luthman via Outages <outages@outages.org> Sent: Sunday, August 30, 2020 12:41:16 PM To: Tim Choe <timch@inmotionhosting.com> Cc: OUTAGES Mailing List <outages@outages.org> Subject: [EXT] Re: [outages] 3556 problems We were effected by this in reaching the Google SAS. It's been working without problems for 100 minutes. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Sun, Aug 30, 2020, 12:38 PM Tim Choe via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: According to Centurylink tweet, their service has been restored. Can someone help me confirm that this is true before I start shifting the traffic back? From: Outages [mailto:outages-bounces@outages.org<mailto:outages-bounces@outages.org>] On Behalf Of Stephen Flynn via Outages Sent: Sunday, August 30, 2020 8:41 AM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] 3556 problems Latest update from CenturyLink Notes History: 08/30/2020 15:31:25 GMT - The IP NOC continues to observe service affecting alarms clearing as the configuration that was deployed continues to propagate through the affected devices. Once affected services have been confirmed restored and stable, a final notification will be provided. 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance. 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral. 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time. 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time. 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve. Regards, Stephen Flynn Atlantic.Net Direct: (321) 206-1390 sflynn@staff.atlantic.net<mailto:sflynn@staff.atlantic.net> www.atlantic.net<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fwww.atlantic.net%2f&c=E,1,5R2TRmkq32a1wr5n02HmSwEGqVxtmZJFs7gD-WK9HP6tIBknHShtrmznfoWajUGQ2l--WqPqpv8ZIJa8hUuzlf4tKq5kh44CiWUdfGtptHjq&typo=1> From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Dylan Ebner via Outages Sent: Sunday, August 30, 2020 11:00 AM To: Michael McDonald <mmcdonald@liquidweb.com<mailto:mmcdonald@liquidweb.com>> Cc: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] 3556 problems CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Here is their latest update: The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance. Next update by: 2020-08-30 15:50 GMT This notification is sent from an unmonitored email account. Please click the link below to reply: mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A Notes History: 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance. 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral. 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time. 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time. 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve. From: Michael McDonald <mmcdonald@liquidweb.com<mailto:mmcdonald@liquidweb.com>> Sent: Sunday, August 30, 2020 9:56 AM To: Dylan Ebner <dylan.ebner@crlmed.com<mailto:dylan.ebner@crlmed.com>> Cc: Jay Ashworth <jra@baylink.com<mailto:jra@baylink.com>>; outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] 3556 problems Not on that ticket, so thanks a ton for the update! Execs are asking questions about what's going on and it's nice to give them feedback! On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral. -------- Original message -------- From: Jay Ashworth via Outages <outages@outages.org<mailto:outages@outages.org>> Date: 8/30/20 9:36 AM (GMT-06:00) To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] 3556 problems Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fpuck.nether.net%2fmailman%2flistinfo%2foutages&c=E,1,mp1z4KyjL_2pXB9I4VZh6x0LBz1K44mlCSwdo8oeoxoApP9Bdirc5sr2-Yg6D3cSTzVPedMGS4XlTxR3lx-PAWIffTDnK86YnK6DlEHFAwLvRnE5z8Udfg,,&typo=1> _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fpuck.nether.net%2fmailman%2flistinfo%2foutages&c=E,1,hhxjdRHPd9c9gAxV65YFz__gKtgGs3qNTJr5G3ODLWVwy8g3uaOd38brYOKm2KdK5RKHP-aJCskrg53f-pP4opnRX3CJbUOfQ0oNHRxbV5Ws1sf4QXtxxcxksLys&typo=1>
Stable, latest statement (Flowspec): Summary: On August 30, 2020 10:04 GMT, CenturyLink identified an issue to be affecting users across multiple markets. The IP Network Operations Center (NOC) was engaged, and initial research identified that an offending flowspec announcement prevented Border Gateway Protocol (BGP) from establishing across multiple elements throughout the CenturyLink Network. The IP NOC deployed a global configuration change to block the offending flowspec announcement, which allowed BGP to begin to correctly establish. As the change propagated through the network, the IP NOC observed all associated service affecting alarms clearing and services returning to a stable state. On 30 Aug 2020, at 18:33, Tim Choe via Outages wrote:
According to Centurylink tweet, their service has been restored. Can someone help me confirm that this is true before I start shifting the traffic back?
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Stephen Flynn via Outages Sent: Sunday, August 30, 2020 8:41 AM To: outages@outages.org Subject: Re: [outages] 3556 problems
Latest update from CenturyLink
Notes History:
08/30/2020 15:31:25 GMT - The IP NOC continues to observe service affecting alarms clearing as the configuration that was deployed continues to propagate through the affected devices. Once affected services have been confirmed restored and stable, a final notification will be provided.
08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.
Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.
08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.
08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time.
08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time.
08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.
The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.
Regards,
Stephen Flynn Atlantic.Net Direct: (321) 206-1390 sflynn@staff.atlantic.net <mailto:sflynn@staff.atlantic.net> www.atlantic.net <http://www.atlantic.net/>
From: Outages <outages-bounces@outages.org <mailto:outages-bounces@outages.org> > On Behalf Of Dylan Ebner via Outages Sent: Sunday, August 30, 2020 11:00 AM To: Michael McDonald <mmcdonald@liquidweb.com <mailto:mmcdonald@liquidweb.com> > Cc: outages@outages.org <mailto:outages@outages.org> Subject: Re: [outages] 3556 problems
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Here is their latest update:
The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.
Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.
Next update by: 2020-08-30 15:50 GMT
This notification is sent from an unmonitored email account. Please click the link below to reply:
mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403, <mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A> &body=Customer%20Reply%3A%0D%0A
Notes History:
08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.
Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.
08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.
08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time.
08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time.
08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.
The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.
From: Michael McDonald <mmcdonald@liquidweb.com <mailto:mmcdonald@liquidweb.com> > Sent: Sunday, August 30, 2020 9:56 AM To: Dylan Ebner <dylan.ebner@crlmed.com <mailto:dylan.ebner@crlmed.com> > Cc: Jay Ashworth <jra@baylink.com <mailto:jra@baylink.com> >; outages@outages.org <mailto:outages@outages.org> Subject: Re: [outages] 3556 problems
Not on that ticket, so thanks a ton for the update! Execs are asking questions about what's going on and it's nice to give them feedback!
On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <outages@outages.org <mailto:outages@outages.org> > wrote:
For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403
08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.
-------- Original message --------
From: Jay Ashworth via Outages <outages@outages.org <mailto:outages@outages.org> >
Date: 8/30/20 9:36 AM (GMT-06:00)
To: outages@outages.org <mailto:outages@outages.org>
Subject: [outages] 3556 problems
Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance. -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Randy, I'm almost wondering if the FlowSpec didn't inadvertently target control plane or MP-BGP traffic inadvertently causing the undulating and unrecoverable flapping of their network. Alfredo Rodriguez PS: Good to see a PSC staffer here (from a former CMU student and 'Burgh resident). -----Original Message----- From: Outages <outages-bounces@outages.org> On Behalf Of Randy Bush via Outages Sent: Sunday, August 30, 2020 1:14 PM To: outages@outages.org Subject: Re: [outages] 3556 problems Sabre Email Notification: This email is from an EXTERNAL source. Please use caution when clicking on links or opening attachments from an unknown or suspicious sender.To report a suspected phishing email, browse to: help.sabre.com > Risk & Security > Report a Security Issue ______________________________________________________________________
initial research identified that an offending flowspec announcement prevented Border Gateway Protocol (BGP) from establishing
opsec paranoid here wonders about the content of the flowspec announcement and whence it originated. randy _______________________________________________ Outages mailing list Outages@outages.org https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman...
Apologies if this has been posted already - I found this interesting: https://blog.cloudflare.com/analysis-of-todays-centurylink-level-3-outage/ Hagen Miller Senior Systems Engineer Computer Services, Inc. -----Original Message----- From: Outages <outages-bounces@outages.org> On Behalf Of Rodriguez, Alfredo via Outages Sent: Monday, August 31, 2020 12:43 PM To: outages@outages.org Subject: Re: [outages] 3556 problems WARNING: This email is from an external source. Do not click links or attachments unless you recognize the sender and know the content is safe. Randy, I'm almost wondering if the FlowSpec didn't inadvertently target control plane or MP-BGP traffic inadvertently causing the undulating and unrecoverable flapping of their network. Alfredo Rodriguez PS: Good to see a PSC staffer here (from a former CMU student and 'Burgh resident). -----Original Message----- From: Outages <outages-bounces@outages.org> On Behalf Of Randy Bush via Outages Sent: Sunday, August 30, 2020 1:14 PM To: outages@outages.org Subject: Re: [outages] 3556 problems Sabre Email Notification: This email is from an EXTERNAL source. Please use caution when clicking on links or opening attachments from an unknown or suspicious sender.To report a suspected phishing email, browse to: help.sabre.com > Risk & Security > Report a Security Issue ______________________________________________________________________
initial research identified that an offending flowspec announcement prevented Border Gateway Protocol (BGP) from establishing
opsec paranoid here wonders about the content of the flowspec announcement and whence it originated. randy _______________________________________________ Outages mailing list Outages@outages.org https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman... _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages ________________________________ This e-mail and any files transmitted with it are the property of Computer Services, Inc. and/or its divisions, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. The sender does not accept any responsibility for any loss, disruption or damage to your data or computer systems that may occur from content of any sort contained in, transmitted with, or added to this e-mail. If you are not a named recipient or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited.
Reminder: let's move all the after-action traffic to outages-discuss. Thanks, -- jra ----- Original Message -----
From: "Randy Bush via Outages" <outages@outages.org> To: outages@outages.org Sent: Sunday, August 30, 2020 2:14:21 PM Subject: Re: [outages] 3556 problems
initial research identified that an offending flowspec announcement prevented Border Gateway Protocol (BGP) from establishing
opsec paranoid here wonders about the content of the flowspec announcement and whence it originated.
randy _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
On Sun 30 Aug 2020 09:33:33 GMT, Tim Choe via Outages wrote:
According to Centurylink tweet, their service has been restored. Can someone help me confirm that this is true before I start shifting the traffic back?
I’ve enabled back sessions at Ashburn and Paris, both are stable. -- Alarig
participants (16)
-
Alarig Le Lay -
Andres Herrera -
Bruce Freshwater -
Dylan Ebner -
Hagen Miller -
Jason Kuehl -
Jay Ashworth -
Jay R. Ashworth
-
Josh Luthman -
Jörg Kost -
Michael McDonald -
Rajwanshi, Bharat -
Randy Bush -
Rodriguez, Alfredo -
Stephen Flynn -
Tim Choe