CenturyLink Outages this morning
We are seeing a lot of centurylink circuits and services down this morning. Centurylink Control Center portal comes up but auth failing unable to login Waves down or bouncing Denver - Seattle (Bouncing) Denver - Chicago (Bouncing) New York - Los Angeles (Down) Chicago - Atlanta (Down) ENNI Down in Denver (Down) Still tying to open tickets with them there portal is not working and stuck on hold right now. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
Seeing the same in NC. Circuits are up but the routing between POPs has changed and we are seeing higher than normal latency. VoIP portal is down also. Allows you to login but you cannot make any changes or view any details. From: Outages <outages-bounces@outages.org> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 9:45 AM To: outages@outages.org Subject: [outages] CenturyLink Outages this morning CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. We are seeing a lot of centurylink circuits and services down this morning. Centurylink Control Center portal comes up but auth failing unable to login Waves down or bouncing Denver - Seattle (Bouncing) Denver - Chicago (Bouncing) New York - Los Angeles (Down) Chicago - Atlanta (Down) ENNI Down in Denver (Down) Still tying to open tickets with them there portal is not working and stuck on hold right now. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Holmes Security Systems may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system.
So it's a bad day for centurylink. The lady who answered the phone for there NOC said that we are having a nation wide outage and we are unable to open tickets at this time. Please call be in 60-90 mins. She had no info on what was going on ________________________________ From: Michael Bolton <mbolton@holmeselectricsecurity.com> Sent: Thursday, December 27, 2018 8:55:03 AM To: Erik Sundberg; outages@outages.org Subject: RE: CenturyLink Outages this morning Seeing the same in NC. Circuits are up but the routing between POPs has changed and we are seeing higher than normal latency. VoIP portal is down also. Allows you to login but you cannot make any changes or view any details. From: Outages <outages-bounces@outages.org> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 9:45 AM To: outages@outages.org Subject: [outages] CenturyLink Outages this morning CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. We are seeing a lot of centurylink circuits and services down this morning. Centurylink Control Center portal comes up but auth failing unable to login Waves down or bouncing Denver - Seattle (Bouncing) Denver - Chicago (Bouncing) New York - Los Angeles (Down) Chicago - Atlanta (Down) ENNI Down in Denver (Down) Still tying to open tickets with them there portal is not working and stuck on hold right now. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Holmes Security Systems may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
Anyone seeing indications of this affecting the legacy Qwest LD network? We're seeing an issue with an SS7 trunk group used for LD, haven't been able to reach anyone at CenturyLink yet. -Jason From: Outages <outages-bounces@outages.org> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 9:02 AM To: Michael Bolton <mbolton@holmeselectricsecurity.com>; outages@outages.org Subject: Re: [outages] CenturyLink Outages this morning So it's a bad day for centurylink. The lady who answered the phone for there NOC said that we are having a nation wide outage and we are unable to open tickets at this time. Please call be in 60-90 mins. She had no info on what was going on ________________________________ From: Michael Bolton <mbolton@holmeselectricsecurity.com<mailto:mbolton@holmeselectricsecurity.com>> Sent: Thursday, December 27, 2018 8:55:03 AM To: Erik Sundberg; outages@outages.org<mailto:outages@outages.org> Subject: RE: CenturyLink Outages this morning Seeing the same in NC. Circuits are up but the routing between POPs has changed and we are seeing higher than normal latency. VoIP portal is down also. Allows you to login but you cannot make any changes or view any details. From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 9:45 AM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] CenturyLink Outages this morning CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. We are seeing a lot of centurylink circuits and services down this morning. Centurylink Control Center portal comes up but auth failing unable to login Waves down or bouncing Denver - Seattle (Bouncing) Denver - Chicago (Bouncing) New York - Los Angeles (Down) Chicago - Atlanta (Down) ENNI Down in Denver (Down) Still tying to open tickets with them there portal is not working and stuck on hold right now. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Holmes Security Systems may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. Jason Baugher, Network Operations Manager 405 Emminga Road | PO Box 217 | Golden, IL 62339-0217 P:(217) 696-4411 | F:(217) 696-4811 | www.adams.net<http://www.adams.net/> [Adams-Logo]<http://adams.net/> ________________________________ The information contained in this email message is PRIVILEGED AND CONFIDENTIAL, and is intended for the use of the addressee and no one else. If you are not the intended recipient, please do not read, distribute, reproduce or use this email message (or the attachments) and notify the sender of the mistaken transmission. Thank you.
CenturyLink Control Center is back up. When your ticket will get looked at is another issues... We have had multiples waves restore, so they are making progress.... Sounds like a pretty bad DWDM NMS System outage with all the transports that we had down.(Just Guessing don't know for sure.) From: Outages <outages-bounces@outages.org> On Behalf Of Jason Baugher via Outages Sent: Thursday, December 27, 2018 11:52 AM To: outages@outages.org Subject: Re: [outages] CenturyLink Outages this morning Anyone seeing indications of this affecting the legacy Qwest LD network? We're seeing an issue with an SS7 trunk group used for LD, haven't been able to reach anyone at CenturyLink yet. -Jason From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 9:02 AM To: Michael Bolton <mbolton@holmeselectricsecurity.com<mailto:mbolton@holmeselectricsecurity.com>>; outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] CenturyLink Outages this morning So it's a bad day for centurylink. The lady who answered the phone for there NOC said that we are having a nation wide outage and we are unable to open tickets at this time. Please call be in 60-90 mins. She had no info on what was going on ________________________________ From: Michael Bolton <mbolton@holmeselectricsecurity.com<mailto:mbolton@holmeselectricsecurity.com>> Sent: Thursday, December 27, 2018 8:55:03 AM To: Erik Sundberg; outages@outages.org<mailto:outages@outages.org> Subject: RE: CenturyLink Outages this morning Seeing the same in NC. Circuits are up but the routing between POPs has changed and we are seeing higher than normal latency. VoIP portal is down also. Allows you to login but you cannot make any changes or view any details. From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 9:45 AM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] CenturyLink Outages this morning CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. We are seeing a lot of centurylink circuits and services down this morning. Centurylink Control Center portal comes up but auth failing unable to login Waves down or bouncing Denver - Seattle (Bouncing) Denver - Chicago (Bouncing) New York - Los Angeles (Down) Chicago - Atlanta (Down) ENNI Down in Denver (Down) Still tying to open tickets with them there portal is not working and stuck on hold right now. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Holmes Security Systems may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. Jason Baugher, Network Operations Manager 405 Emminga Road | PO Box 217 | Golden, IL 62339-0217 P:(217) 696-4411 | F:(217) 696-4811 | www.adams.net<http://www.adams.net/> [Adams-Logo]<http://adams.net/> ________________________________ The information contained in this email message is PRIVILEGED AND CONFIDENTIAL, and is intended for the use of the addressee and no one else. If you are not the intended recipient, please do not read, distribute, reproduce or use this email message (or the attachments) and notify the sender of the mistaken transmission. Thank you. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
CenturyLink finally acknowledged: Great Lakes Region CenturyLink Partners/Agents: FYI. Our network is experiencing a disruption affecting customer services. Customers are currently unable to generate tickets through the Control Center portal. We know how important these services are to our customers, and we are working to restore services as quickly as possible. Our technical teams are working to resolve the issue, and we will provide updates as the situation progresses. -bruce On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages" <outages@outages.org<mailto:outages@outages.org>> wrote: So it's a bad day for centurylink. The lady who answered the phone for there NOC said that we are having a nation wide outage and we are unable to open tickets at this time. Please call be in 60-90 mins. She had no info on what was going on ________________________________ From: Michael Bolton <mbolton@holmeselectricsecurity.com> Sent: Thursday, December 27, 2018 8:55:03 AM To: Erik Sundberg; outages@outages.org Subject: RE: CenturyLink Outages this morning Seeing the same in NC. Circuits are up but the routing between POPs has changed and we are seeing higher than normal latency. VoIP portal is down also. Allows you to login but you cannot make any changes or view any details. From: Outages <outages-bounces@outages.org> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 9:45 AM To: outages@outages.org Subject: [outages] CenturyLink Outages this morning CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. We are seeing a lot of centurylink circuits and services down this morning. Centurylink Control Center portal comes up but auth failing unable to login Waves down or bouncing Denver - Seattle (Bouncing) Denver - Chicago (Bouncing) New York - Los Angeles (Down) Chicago - Atlanta (Down) ENNI Down in Denver (Down) Still tying to open tickets with them there portal is not working and stuck on hold right now. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Holmes Security Systems may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
Finally received something useful from centurylink... I have more questions now like how broadcast traffic effect 4x Transport waves. Update: On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership. From: Outages <outages-bounces@outages.org> On Behalf Of Bruce Freshwater via Outages Sent: Thursday, December 27, 2018 1:24 PM To: outages@outages.org Subject: Re: [outages] CenturyLink Outages this morning CenturyLink finally acknowledged: Great Lakes Region CenturyLink Partners/Agents: FYI. Our network is experiencing a disruption affecting customer services. Customers are currently unable to generate tickets through the Control Center portal. We know how important these services are to our customers, and we are working to restore services as quickly as possible. Our technical teams are working to resolve the issue, and we will provide updates as the situation progresses. -bruce On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages" <outages@outages.org<mailto:outages@outages.org>> wrote: So it's a bad day for centurylink. The lady who answered the phone for there NOC said that we are having a nation wide outage and we are unable to open tickets at this time. Please call be in 60-90 mins. She had no info on what was going on ________________________________ From: Michael Bolton <mbolton@holmeselectricsecurity.com<mailto:mbolton@holmeselectricsecurity.com>> Sent: Thursday, December 27, 2018 8:55:03 AM To: Erik Sundberg; outages@outages.org<mailto:outages@outages.org> Subject: RE: CenturyLink Outages this morning Seeing the same in NC. Circuits are up but the routing between POPs has changed and we are seeing higher than normal latency. VoIP portal is down also. Allows you to login but you cannot make any changes or view any details. From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 9:45 AM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] CenturyLink Outages this morning CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. We are seeing a lot of centurylink circuits and services down this morning. Centurylink Control Center portal comes up but auth failing unable to login Waves down or bouncing Denver - Seattle (Bouncing) Denver - Chicago (Bouncing) New York - Los Angeles (Down) Chicago - Atlanta (Down) ENNI Down in Denver (Down) Still tying to open tickets with them there portal is not working and stuck on hold right now. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Holmes Security Systems may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
Finally received something useful from centurylink... I have more questions now like how broadcast traffic effect 4x Transport waves.
Ineptitude. Christopher Conley Systems Administrator | Fors Marsh Group Arlington, VA From: Outages <outages-bounces@outages.org> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 2:46 PM To: outages@outages.org Subject: Re: [outages] CenturyLink Outages this morning Finally received something useful from centurylink... I have more questions now like how broadcast traffic effect 4x Transport waves. Update: On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership. From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Bruce Freshwater via Outages Sent: Thursday, December 27, 2018 1:24 PM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] CenturyLink Outages this morning CenturyLink finally acknowledged: Great Lakes Region CenturyLink Partners/Agents: FYI. Our network is experiencing a disruption affecting customer services. Customers are currently unable to generate tickets through the Control Center portal. We know how important these services are to our customers, and we are working to restore services as quickly as possible. Our technical teams are working to resolve the issue, and we will provide updates as the situation progresses. -bruce On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages" <outages@outages.org<mailto:outages@outages.org>> wrote: So it's a bad day for centurylink. The lady who answered the phone for there NOC said that we are having a nation wide outage and we are unable to open tickets at this time. Please call be in 60-90 mins. She had no info on what was going on ________________________________ From: Michael Bolton <mbolton@holmeselectricsecurity.com<mailto:mbolton@holmeselectricsecurity.com>> Sent: Thursday, December 27, 2018 8:55:03 AM To: Erik Sundberg; outages@outages.org<mailto:outages@outages.org> Subject: RE: CenturyLink Outages this morning Seeing the same in NC. Circuits are up but the routing between POPs has changed and we are seeing higher than normal latency. VoIP portal is down also. Allows you to login but you cannot make any changes or view any details. From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 9:45 AM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] CenturyLink Outages this morning CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. We are seeing a lot of centurylink circuits and services down this morning. Centurylink Control Center portal comes up but auth failing unable to login Waves down or bouncing Denver - Seattle (Bouncing) Denver - Chicago (Bouncing) New York - Los Angeles (Down) Chicago - Atlanta (Down) ENNI Down in Denver (Down) Still tying to open tickets with them there portal is not working and stuck on hold right now. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Holmes Security Systems may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
I understand that the transport waves went out first, and then they try to reroute to something in Arizona that begin introducing all of the broadcast traffic. How legit that is I’m not sure Sent from my iPhone
On Dec 27, 2018, at 12:58 PM, Christopher Conley via Outages <outages@outages.org> wrote:
Finally received something useful from centurylink… I have more questions now like how broadcast traffic effect 4x Transport waves.
Ineptitude.
Christopher Conley Systems Administrator | Fors Marsh Group Arlington, VA
From: Outages <outages-bounces@outages.org> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 2:46 PM To: outages@outages.org Subject: Re: [outages] CenturyLink Outages this morning
Finally received something useful from centurylink… I have more questions now like how broadcast traffic effect 4x Transport waves.
Update: On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership.
From: Outages <outages-bounces@outages.org> On Behalf Of Bruce Freshwater via Outages Sent: Thursday, December 27, 2018 1:24 PM To: outages@outages.org Subject: Re: [outages] CenturyLink Outages this morning
CenturyLink finally acknowledged:
Great Lakes Region CenturyLink Partners/Agents:
FYI. Our network is experiencing a disruption affecting customer services. Customers are currently unable to generate tickets through the Control Center portal.
We know how important these services are to our customers, and we are working to restore services as quickly as possible.
Our technical teams are working to resolve the issue, and we will provide updates as the situation progresses.
-bruce
On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages" <outages@outages.org> wrote:
So it's a bad day for centurylink.
The lady who answered the phone for there NOC said that we are having a nation wide outage and we are unable to open tickets at this time. Please call be in 60-90 mins.
She had no info on what was going on
From: Michael Bolton <mbolton@holmeselectricsecurity.com> Sent: Thursday, December 27, 2018 8:55:03 AM To: Erik Sundberg; outages@outages.org Subject: RE: CenturyLink Outages this morning
Seeing the same in NC. Circuits are up but the routing between POPs has changed and we are seeing higher than normal latency.
VoIP portal is down also. Allows you to login but you cannot make any changes or view any details.
From: Outages <outages-bounces@outages.org> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 9:45 AM To: outages@outages.org Subject: [outages] CenturyLink Outages this morning
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. We are seeing a lot of centurylink circuits and services down this morning.
Centurylink Control Center portal comes up but auth failing unable to login
Waves down or bouncing
Denver - Seattle (Bouncing)
Denver - Chicago (Bouncing)
New York - Los Angeles (Down)
Chicago - Atlanta (Down)
ENNI Down in Denver (Down)
Still tying to open tickets with them there portal is not working and stuck on hold right now.
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Holmes Security Systems may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system.
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Maybe light at the end of the tunnel... latest we got from Centurylink was at 10:45pm CST. Our engineers and technicians have identified the network element that has affected our customer services. Services are restoring, and the current estimated time for full recovery is four hours. CenturyLink will be conducting an extensive post-incident investigation and root cause analysis to provide follow-up information to our customers impacted by this event. We will provide one more Sales Alert from this address once services are fully restored. From: Rich Breton <richdbreton@gmail.com> Sent: Thursday, December 27, 2018 2:59 PM To: Christopher Conley <CConley@forsmarshgroup.com> Cc: Erik Sundberg <ESundberg@nitelusa.com>; outages@outages.org Subject: Re: [outages] CenturyLink Outages this morning I understand that the transport waves went out first, and then they try to reroute to something in Arizona that begin introducing all of the broadcast traffic. How legit that is I’m not sure Sent from my iPhone On Dec 27, 2018, at 12:58 PM, Christopher Conley via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
Finally received something useful from centurylink… I have more questions now like how broadcast traffic effect 4x Transport waves.
Ineptitude. Christopher Conley Systems Administrator | Fors Marsh Group Arlington, VA From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 2:46 PM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] CenturyLink Outages this morning Finally received something useful from centurylink… I have more questions now like how broadcast traffic effect 4x Transport waves. Update: On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership. From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Bruce Freshwater via Outages Sent: Thursday, December 27, 2018 1:24 PM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] CenturyLink Outages this morning CenturyLink finally acknowledged: Great Lakes Region CenturyLink Partners/Agents: FYI. Our network is experiencing a disruption affecting customer services. Customers are currently unable to generate tickets through the Control Center portal. We know how important these services are to our customers, and we are working to restore services as quickly as possible. Our technical teams are working to resolve the issue, and we will provide updates as the situation progresses. -bruce On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages" <outages@outages.org<mailto:outages@outages.org>> wrote: So it's a bad day for centurylink. The lady who answered the phone for there NOC said that we are having a nation wide outage and we are unable to open tickets at this time. Please call be in 60-90 mins. She had no info on what was going on ________________________________ From: Michael Bolton <mbolton@holmeselectricsecurity.com<mailto:mbolton@holmeselectricsecurity.com>> Sent: Thursday, December 27, 2018 8:55:03 AM To: Erik Sundberg; outages@outages.org<mailto:outages@outages.org> Subject: RE: CenturyLink Outages this morning Seeing the same in NC. Circuits are up but the routing between POPs has changed and we are seeing higher than normal latency. VoIP portal is down also. Allows you to login but you cannot make any changes or view any details. From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 9:45 AM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] CenturyLink Outages this morning CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. We are seeing a lot of centurylink circuits and services down this morning. Centurylink Control Center portal comes up but auth failing unable to login Waves down or bouncing Denver - Seattle (Bouncing) Denver - Chicago (Bouncing) New York - Los Angeles (Down) Chicago - Atlanta (Down) ENNI Down in Denver (Down) Still tying to open tickets with them there portal is not working and stuck on hold right now. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Holmes Security Systems may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
Facebook is blowing up with 911 outages and claiming it's CenturyLink to blame. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Fri, Dec 28, 2018, 2:09 AM Erik Sundberg via Outages <outages@outages.org wrote:
Maybe light at the end of the tunnel...
latest we got from Centurylink was at 10:45pm CST.
Our engineers and technicians have identified the network element that has affected our customer services. Services are restoring, and the current estimated time for full recovery is four hours.
CenturyLink will be conducting an extensive post-incident investigation and root cause analysis to provide follow-up information to our customers impacted by this event. We will provide one more Sales Alert from this address once services are fully restored.
*From:* Rich Breton <richdbreton@gmail.com> *Sent:* Thursday, December 27, 2018 2:59 PM *To:* Christopher Conley <CConley@forsmarshgroup.com> *Cc:* Erik Sundberg <ESundberg@nitelusa.com>; outages@outages.org *Subject:* Re: [outages] CenturyLink Outages this morning
I understand that the transport waves went out first, and then they try to reroute to something in Arizona that begin introducing all of the broadcast traffic. How legit that is I’m not sure
Sent from my iPhone
On Dec 27, 2018, at 12:58 PM, Christopher Conley via Outages < outages@outages.org> wrote:
Finally received something useful from centurylink… I have more questions now like how broadcast traffic effect 4x Transport waves.
Ineptitude.
Christopher Conley
Systems Administrator | Fors Marsh Group
Arlington, VA
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Erik Sundberg via Outages *Sent:* Thursday, December 27, 2018 2:46 PM *To:* outages@outages.org *Subject:* Re: [outages] CenturyLink Outages this morning
Finally received something useful from centurylink… I have more questions now like how broadcast traffic effect 4x Transport waves.
Update:
On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership.
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Bruce Freshwater via Outages *Sent:* Thursday, December 27, 2018 1:24 PM *To:* outages@outages.org *Subject:* Re: [outages] CenturyLink Outages this morning
CenturyLink finally acknowledged:
Great Lakes Region CenturyLink Partners/Agents:
FYI. Our network is experiencing a disruption affecting customer services. Customers are currently unable to generate tickets through the Control Center portal.
We know how important these services are to our customers, and we are working to restore services as quickly as possible.
Our technical teams are working to resolve the issue, and we will provide updates as the situation progresses.
-bruce
On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages" < outages@outages.org> wrote:
So it's a bad day for centurylink.
The lady who answered the phone for there NOC said that we are having a nation wide outage and we are unable to open tickets at this time. Please call be in 60-90 mins.
She had no info on what was going on
------------------------------
*From:* Michael Bolton <mbolton@holmeselectricsecurity.com> *Sent:* Thursday, December 27, 2018 8:55:03 AM *To:* Erik Sundberg; outages@outages.org *Subject:* RE: CenturyLink Outages this morning
Seeing the same in NC. Circuits are up but the routing between POPs has changed and we are seeing higher than normal latency.
VoIP portal is down also. Allows you to login but you cannot make any changes or view any details.
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Erik Sundberg via Outages *Sent:* Thursday, December 27, 2018 9:45 AM *To:* outages@outages.org *Subject:* [outages] CenturyLink Outages this morning
*CAUTION:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We are seeing a lot of centurylink circuits and services down this morning.
Centurylink Control Center portal comes up but auth failing unable to login
Waves down or bouncing
Denver - Seattle (Bouncing)
Denver - Chicago (Bouncing)
New York - Los Angeles (Down)
Chicago - Atlanta (Down)
ENNI Down in Denver (Down)
Still tying to open tickets with them there portal is not working and stuck on hold right now.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Holmes Security Systems may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Everything on Facebook is true. 😏 It’s in the news as well: https://www.usatoday.com/story/news/nation/2018/12/28/centurylink-outage-kno... Many CenturyLink customers across the U.S. were without internet Thursday amid a lengthy outage that even affected some Verizon customers. The outage began early in the day but by late Thursday night, the company tweeted that its engineers had identified a “network element” that was affecting services and expected to fully restore services within hours. Monroe, Louisiana-based CenturyLink didn’t provide details of the problem and it didn’t indicate how many customers were affected. Customers from New York to California reported outages. Jessica Rosenworcel, a member of the Federal Communications Commission, said via Twitter it was a nationwide outage and her agency needed to investigate. The outage knocked out 911 emergency call services in parts of western Washington state. KOMO reports<https://komonews.com/news/local/centurylink-outage-knocks-out-911-calls-in-parts-of-w-wash> that some CenturyLink customers reported receiving busy signals when dialing 911. Other areas of the country also experiencing 911 outages included parts of Missouri, Idaho and Arizona. - andy From: Outages <outages-bounces@outages.org> on behalf of Josh Luthman via Outages <outages@outages.org> Reply-To: Josh Luthman <josh@imaginenetworksllc.com> Date: Friday, December 28, 2018 at 8:28 AM To: Erik Sundberg <ESundberg@nitelusa.com> Cc: OUTAGES Mailing List <outages@outages.org> Subject: Re: [outages] CenturyLink Outages this morning Facebook is blowing up with 911 outages and claiming it's CenturyLink to blame. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Fri, Dec 28, 2018, 2:09 AM Erik Sundberg via Outages <outages@outages.org<mailto:outages@outages.org> wrote: Maybe light at the end of the tunnel... latest we got from Centurylink was at 10:45pm CST. Our engineers and technicians have identified the network element that has affected our customer services. Services are restoring, and the current estimated time for full recovery is four hours. CenturyLink will be conducting an extensive post-incident investigation and root cause analysis to provide follow-up information to our customers impacted by this event. We will provide one more Sales Alert from this address once services are fully restored. From: Rich Breton <richdbreton@gmail.com<mailto:richdbreton@gmail.com>> Sent: Thursday, December 27, 2018 2:59 PM To: Christopher Conley <CConley@forsmarshgroup.com<mailto:CConley@forsmarshgroup.com>> Cc: Erik Sundberg <ESundberg@nitelusa.com<mailto:ESundberg@nitelusa.com>>; outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] CenturyLink Outages this morning I understand that the transport waves went out first, and then they try to reroute to something in Arizona that begin introducing all of the broadcast traffic. How legit that is I’m not sure Sent from my iPhone On Dec 27, 2018, at 12:58 PM, Christopher Conley via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
Finally received something useful from centurylink… I have more questions now like how broadcast traffic effect 4x Transport waves.
Ineptitude. Christopher Conley Systems Administrator | Fors Marsh Group Arlington, VA From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 2:46 PM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] CenturyLink Outages this morning Finally received something useful from centurylink… I have more questions now like how broadcast traffic effect 4x Transport waves. Update: On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership. From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Bruce Freshwater via Outages Sent: Thursday, December 27, 2018 1:24 PM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] CenturyLink Outages this morning CenturyLink finally acknowledged: Great Lakes Region CenturyLink Partners/Agents: FYI. Our network is experiencing a disruption affecting customer services. Customers are currently unable to generate tickets through the Control Center portal. We know how important these services are to our customers, and we are working to restore services as quickly as possible. Our technical teams are working to resolve the issue, and we will provide updates as the situation progresses. -bruce On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages" <outages@outages.org<mailto:outages@outages.org>> wrote: So it's a bad day for centurylink. The lady who answered the phone for there NOC said that we are having a nation wide outage and we are unable to open tickets at this time. Please call be in 60-90 mins. She had no info on what was going on ________________________________ From: Michael Bolton <mbolton@holmeselectricsecurity.com<mailto:mbolton@holmeselectricsecurity.com>> Sent: Thursday, December 27, 2018 8:55:03 AM To: Erik Sundberg; outages@outages.org<mailto:outages@outages.org> Subject: RE: CenturyLink Outages this morning Seeing the same in NC. Circuits are up but the routing between POPs has changed and we are seeing higher than normal latency. VoIP portal is down also. Allows you to login but you cannot make any changes or view any details. From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 9:45 AM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] CenturyLink Outages this morning CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. We are seeing a lot of centurylink circuits and services down this morning. Centurylink Control Center portal comes up but auth failing unable to login Waves down or bouncing Denver - Seattle (Bouncing) Denver - Chicago (Bouncing) New York - Los Angeles (Down) Chicago - Atlanta (Down) ENNI Down in Denver (Down) Still tying to open tickets with them there portal is not working and stuck on hold right now. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Holmes Security Systems may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Don't be a fear monger. I searched Facebook and found *three* posts about 911 outages, all of them simply reposts of local news stories. Facebook is not "blowing up". Sheesh! -mel via cell -mel via cell
On Dec 28, 2018, at 5:23 AM, Josh Luthman via Outages <outages@outages.org> wrote:
Facebook is blowing up with 911 outages and claiming it's CenturyLink to blame.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Fri, Dec 28, 2018, 2:09 AM Erik Sundberg via Outages <outages@outages.org wrote: Maybe light at the end of the tunnel...
latest we got from Centurylink was at 10:45pm CST.
Our engineers and technicians have identified the network element that has affected our customer services. Services are restoring, and the current estimated time for full recovery is four hours.
CenturyLink will be conducting an extensive post-incident investigation and root cause analysis to provide follow-up information to our customers impacted by this event. We will provide one more Sales Alert from this address once services are fully restored.
From: Rich Breton <richdbreton@gmail.com> Sent: Thursday, December 27, 2018 2:59 PM To: Christopher Conley <CConley@forsmarshgroup.com> Cc: Erik Sundberg <ESundberg@nitelusa.com>; outages@outages.org Subject: Re: [outages] CenturyLink Outages this morning
I understand that the transport waves went out first, and then they try to reroute to something in Arizona that begin introducing all of the broadcast traffic. How legit that is I’m not sure
Sent from my iPhone
On Dec 27, 2018, at 12:58 PM, Christopher Conley via Outages <outages@outages.org> wrote:
Finally received something useful from centurylink… I have more questions now like how broadcast traffic effect 4x Transport waves.
Ineptitude.
Christopher Conley
Systems Administrator | Fors Marsh Group
Arlington, VA
From: Outages <outages-bounces@outages.org> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 2:46 PM To: outages@outages.org Subject: Re: [outages] CenturyLink Outages this morning
Finally received something useful from centurylink… I have more questions now like how broadcast traffic effect 4x Transport waves.
Update:
On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership.
From: Outages <outages-bounces@outages.org> On Behalf Of Bruce Freshwater via Outages Sent: Thursday, December 27, 2018 1:24 PM To: outages@outages.org Subject: Re: [outages] CenturyLink Outages this morning
CenturyLink finally acknowledged:
Great Lakes Region CenturyLink Partners/Agents:
FYI. Our network is experiencing a disruption affecting customer services. Customers are currently unable to generate tickets through the Control Center portal.
We know how important these services are to our customers, and we are working to restore services as quickly as possible.
Our technical teams are working to resolve the issue, and we will provide updates as the situation progresses.
-bruce
On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages" <outages@outages.org> wrote:
So it's a bad day for centurylink.
The lady who answered the phone for there NOC said that we are having a nation wide outage and we are unable to open tickets at this time. Please call be in 60-90 mins.
She had no info on what was going on
From: Michael Bolton <mbolton@holmeselectricsecurity.com> Sent: Thursday, December 27, 2018 8:55:03 AM To: Erik Sundberg; outages@outages.org Subject: RE: CenturyLink Outages this morning
Seeing the same in NC. Circuits are up but the routing between POPs has changed and we are seeing higher than normal latency.
VoIP portal is down also. Allows you to login but you cannot make any changes or view any details.
From: Outages <outages-bounces@outages.org> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 9:45 AM To: outages@outages.org Subject: [outages] CenturyLink Outages this morning
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We are seeing a lot of centurylink circuits and services down this morning.
Centurylink Control Center portal comes up but auth failing unable to login
Waves down or bouncing
Denver - Seattle (Bouncing)
Denver - Chicago (Bouncing)
New York - Los Angeles (Down)
Chicago - Atlanta (Down)
ENNI Down in Denver (Down)
Still tying to open tickets with them there portal is not working and stuck on hold right now.
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Holmes Security Systems may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system.
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Most public safety agencies in Massachusetts posted on Facebook and Twitter between 6 AM and 8 AM EST about wireless 911 outages caused by CenturyLink. It was affecting wireless routing, so some calls were being misrouted to other towns, or even other states; some callers were getting reorder tones or other error messages. The problem was reported to be resolved prior to 8 AM EST. Seeing reports from PSAP's that the issue started again in the last 15 minutes. This is wireless only. Landline 911 calls were / are not affected. On Fri, Dec 28, 2018 at 9:32 AM Mel Beckman via Outages <outages@outages.org> wrote:
Don't be a fear monger. I searched Facebook and found *three* posts about 911 outages, all of them simply reposts of local news stories.
Facebook is not "blowing up".
Sheesh!
[image: image1.PNG]
-mel via cell
-mel via cell
On Dec 28, 2018, at 5:23 AM, Josh Luthman via Outages <outages@outages.org> wrote:
Facebook is blowing up with 911 outages and claiming it's CenturyLink to blame.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Fri, Dec 28, 2018, 2:09 AM Erik Sundberg via Outages < outages@outages.org wrote:
Maybe light at the end of the tunnel...
latest we got from Centurylink was at 10:45pm CST.
Our engineers and technicians have identified the network element that has affected our customer services. Services are restoring, and the current estimated time for full recovery is four hours.
CenturyLink will be conducting an extensive post-incident investigation and root cause analysis to provide follow-up information to our customers impacted by this event. We will provide one more Sales Alert from this address once services are fully restored.
*From:* Rich Breton <richdbreton@gmail.com> *Sent:* Thursday, December 27, 2018 2:59 PM *To:* Christopher Conley <CConley@forsmarshgroup.com> *Cc:* Erik Sundberg <ESundberg@nitelusa.com>; outages@outages.org *Subject:* Re: [outages] CenturyLink Outages this morning
I understand that the transport waves went out first, and then they try to reroute to something in Arizona that begin introducing all of the broadcast traffic. How legit that is I’m not sure
Sent from my iPhone
On Dec 27, 2018, at 12:58 PM, Christopher Conley via Outages < outages@outages.org> wrote:
Finally received something useful from centurylink… I have more questions now like how broadcast traffic effect 4x Transport waves.
Ineptitude.
Christopher Conley
Systems Administrator | Fors Marsh Group
Arlington, VA
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Erik Sundberg via Outages *Sent:* Thursday, December 27, 2018 2:46 PM *To:* outages@outages.org *Subject:* Re: [outages] CenturyLink Outages this morning
Finally received something useful from centurylink… I have more questions now like how broadcast traffic effect 4x Transport waves.
Update:
On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership.
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Bruce Freshwater via Outages *Sent:* Thursday, December 27, 2018 1:24 PM *To:* outages@outages.org *Subject:* Re: [outages] CenturyLink Outages this morning
CenturyLink finally acknowledged:
Great Lakes Region CenturyLink Partners/Agents:
FYI. Our network is experiencing a disruption affecting customer services. Customers are currently unable to generate tickets through the Control Center portal.
We know how important these services are to our customers, and we are working to restore services as quickly as possible.
Our technical teams are working to resolve the issue, and we will provide updates as the situation progresses.
-bruce
On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages" < outages@outages.org> wrote:
So it's a bad day for centurylink.
The lady who answered the phone for there NOC said that we are having a nation wide outage and we are unable to open tickets at this time. Please call be in 60-90 mins.
She had no info on what was going on
------------------------------
*From:* Michael Bolton <mbolton@holmeselectricsecurity.com> *Sent:* Thursday, December 27, 2018 8:55:03 AM *To:* Erik Sundberg; outages@outages.org *Subject:* RE: CenturyLink Outages this morning
Seeing the same in NC. Circuits are up but the routing between POPs has changed and we are seeing higher than normal latency.
VoIP portal is down also. Allows you to login but you cannot make any changes or view any details.
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Erik Sundberg via Outages *Sent:* Thursday, December 27, 2018 9:45 AM *To:* outages@outages.org *Subject:* [outages] CenturyLink Outages this morning
*CAUTION:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We are seeing a lot of centurylink circuits and services down this morning.
Centurylink Control Center portal comes up but auth failing unable to login
Waves down or bouncing
Denver - Seattle (Bouncing)
Denver - Chicago (Bouncing)
New York - Los Angeles (Down)
Chicago - Atlanta (Down)
ENNI Down in Denver (Down)
Still tying to open tickets with them there portal is not working and stuck on hold right now.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Holmes Security Systems may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Massachusetts emergency management agency (MEMA) sent this out this morning and followed up at 0736 that the issues had been resolved. *Massachusetts Emergency Management Agency* ------------------------------ *Massachusetts Emergency Management Agency* *Situational Awareness Statement* *December 28, 2018 7:00 AM* *Re: Cellular 911 Experiencing Technical Difficulties and Sporadic Outages* The Massachusetts Emergency Management Agency is sharing the following information from the Commonwealth Fusion Center (CFC) with our local partners and stakeholders. MEMA also is in touch with the State 911 Department. *From the Commonwealth Fusion Center* *Situation:* A nationwide outage affecting CenturyLink, an Internet communications company, is impacting wireless 911 call capability in various parts of the country. We believe Massachusetts has been impacted by the outage and urge residents using cellular phones, if necessary, to contact emergency first-responders by using the full 10-digit number of a public safety answering point emergency call center (PSAP) or their local police department. The Massachusetts State Police operate three regional PSAPs and a fourth is staffed by State 911 at MSP headquarters in Framingham. Residents using a cell phone to call for emergency services should first try to dial 911, and if it does not work, instead call one of these four 10-digit numbers: Shelburne Control PSAP 413-625-8200 New Braintree Control PSAP 508-867-1170 Northampton Control PSAP 413-586-1508 Framingham PSAP 508-820-2121. Residents using cell phones may also, if dialing 911 does not work, may call the 10-digit general number of their local police department. They may also call the 10-digit number of a State Police Barracks in their area (search for barracks location and phone numbers on the MSP web site at www.mass.gov and search for State Police locations). Other agencies operate additional PSAPs in Massachusetts beyond those listed above. The outage seems to only be affecting wireless 911 calls; landline calls to 911 appear to be working. CenturyLink has indicted it is working on repairing what the company called a “network element” that is causing the problem. *MEMA Operations* The State Emergency Operations Center (SEOC) is currently operating at Level 1 (Steady State Monitoring). MEMA will continue to monitor the situation and will disseminate additional Situational Awareness Statements as necessary. Cities and towns wishing to share information or request assistance should contact their MEMA Regional office or MEMA’s 24/7 communications center at 508-820-2000. *Stay Informed:* Utilize *Massachusetts Alerts* to receive emergency notifications and information from the Massachusetts Emergency Management Agency and the National Weather Service. *Massachusetts Alerts* is a free app that is available for Android and iPhones. To learn more about *Massachusetts Alerts*, and for information on how to download the free app onto your smartphone, visit: http://www.mass.gov/service-details/massachusetts-alerts-smartphone-app. Utilize MEMA’s real-time power outage viewer to stay informed about current power outages in your community and region, and across the state, including information from utility companies about restoration times: http://mema.mapsonline.net/public.html Utilize MEMA’s live weather radar and forecasting tools: http://www.mass.gov/map-resources *Online Resources:*For additional information and resources, visit: Massachusetts Emergency Management Agency at www.mass.gov/mema MEMA's Facebook page: http://www.facebook.com/MassachusettsEMA MEMA Twitter: @MassEMA <http://twitter.com/massema> Federal Emergency Management Agency at www.fema.gov National Weather Service Boston/Norton at www.weather.gov/boston National Weather Service Albany, NY at www.weather.gov/albany National Weather Service Weather Prediction Center: http://www.wpc.ncep.noaa.gov National Weather Service Storm Prediction Center: http://www.spc.noaa.gov/ Mass211 at www.mass211.org ------------------------------ This message has been sent from MEMA's ESF Team Listserv. To unsubscribe from this listserv, call MEMA Operations at (508) 820-2000. +1.781.300.0000 On Fri, Dec 28, 2018 at 9:52 AM Matt Brennan via Outages < outages@outages.org> wrote:
Most public safety agencies in Massachusetts posted on Facebook and Twitter between 6 AM and 8 AM EST about wireless 911 outages caused by CenturyLink. It was affecting wireless routing, so some calls were being misrouted to other towns, or even other states; some callers were getting reorder tones or other error messages. The problem was reported to be resolved prior to 8 AM EST.
Seeing reports from PSAP's that the issue started again in the last 15 minutes.
This is wireless only. Landline 911 calls were / are not affected.
On Fri, Dec 28, 2018 at 9:32 AM Mel Beckman via Outages < outages@outages.org> wrote:
Don't be a fear monger. I searched Facebook and found *three* posts about 911 outages, all of them simply reposts of local news stories.
Facebook is not "blowing up".
Sheesh!
[image: image1.PNG]
-mel via cell
-mel via cell
On Dec 28, 2018, at 5:23 AM, Josh Luthman via Outages < outages@outages.org> wrote:
Facebook is blowing up with 911 outages and claiming it's CenturyLink to blame.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Fri, Dec 28, 2018, 2:09 AM Erik Sundberg via Outages < outages@outages.org wrote:
Maybe light at the end of the tunnel...
latest we got from Centurylink was at 10:45pm CST.
Our engineers and technicians have identified the network element that has affected our customer services. Services are restoring, and the current estimated time for full recovery is four hours.
CenturyLink will be conducting an extensive post-incident investigation and root cause analysis to provide follow-up information to our customers impacted by this event. We will provide one more Sales Alert from this address once services are fully restored.
*From:* Rich Breton <richdbreton@gmail.com> *Sent:* Thursday, December 27, 2018 2:59 PM *To:* Christopher Conley <CConley@forsmarshgroup.com> *Cc:* Erik Sundberg <ESundberg@nitelusa.com>; outages@outages.org *Subject:* Re: [outages] CenturyLink Outages this morning
I understand that the transport waves went out first, and then they try to reroute to something in Arizona that begin introducing all of the broadcast traffic. How legit that is I’m not sure
Sent from my iPhone
On Dec 27, 2018, at 12:58 PM, Christopher Conley via Outages < outages@outages.org> wrote:
Finally received something useful from centurylink… I have more questions now like how broadcast traffic effect 4x Transport waves.
Ineptitude.
Christopher Conley
Systems Administrator | Fors Marsh Group
Arlington, VA
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Erik Sundberg via Outages *Sent:* Thursday, December 27, 2018 2:46 PM *To:* outages@outages.org *Subject:* Re: [outages] CenturyLink Outages this morning
Finally received something useful from centurylink… I have more questions now like how broadcast traffic effect 4x Transport waves.
Update:
On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership.
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Bruce Freshwater via Outages *Sent:* Thursday, December 27, 2018 1:24 PM *To:* outages@outages.org *Subject:* Re: [outages] CenturyLink Outages this morning
CenturyLink finally acknowledged:
Great Lakes Region CenturyLink Partners/Agents:
FYI. Our network is experiencing a disruption affecting customer services. Customers are currently unable to generate tickets through the Control Center portal.
We know how important these services are to our customers, and we are working to restore services as quickly as possible.
Our technical teams are working to resolve the issue, and we will provide updates as the situation progresses.
-bruce
On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages" < outages@outages.org> wrote:
So it's a bad day for centurylink.
The lady who answered the phone for there NOC said that we are having a nation wide outage and we are unable to open tickets at this time. Please call be in 60-90 mins.
She had no info on what was going on
------------------------------
*From:* Michael Bolton <mbolton@holmeselectricsecurity.com> *Sent:* Thursday, December 27, 2018 8:55:03 AM *To:* Erik Sundberg; outages@outages.org *Subject:* RE: CenturyLink Outages this morning
Seeing the same in NC. Circuits are up but the routing between POPs has changed and we are seeing higher than normal latency.
VoIP portal is down also. Allows you to login but you cannot make any changes or view any details.
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Erik Sundberg via Outages *Sent:* Thursday, December 27, 2018 9:45 AM *To:* outages@outages.org *Subject:* [outages] CenturyLink Outages this morning
*CAUTION:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We are seeing a lot of centurylink circuits and services down this morning.
Centurylink Control Center portal comes up but auth failing unable to login
Waves down or bouncing
Denver - Seattle (Bouncing)
Denver - Chicago (Bouncing)
New York - Los Angeles (Down)
Chicago - Atlanta (Down)
ENNI Down in Denver (Down)
Still tying to open tickets with them there portal is not working and stuck on hold right now.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Holmes Security Systems may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Received a WEA from MEMA in Boston, MA noting that 911 is down for cellular users. Dom Colangelo Senior Windows Systems Administrator Infrastructure and Online Services Simmons University 617.521.2669 On Fri, Dec 28, 2018 at 9:56 AM Bob Antia via Outages <outages@outages.org> wrote:
Massachusetts emergency management agency (MEMA) sent this out this morning and followed up at 0736 that the issues had been resolved.
*Massachusetts Emergency Management Agency* ------------------------------
*Massachusetts Emergency Management Agency*
*Situational Awareness Statement*
*December 28, 2018 7:00 AM*
*Re: Cellular 911 Experiencing Technical Difficulties and Sporadic Outages*
The Massachusetts Emergency Management Agency is sharing the following information from the Commonwealth Fusion Center (CFC) with our local partners and stakeholders. MEMA also is in touch with the State 911 Department.
*From the Commonwealth Fusion Center*
*Situation:*
A nationwide outage affecting CenturyLink, an Internet communications company, is impacting wireless 911 call capability in various parts of the country. We believe Massachusetts has been impacted by the outage and urge residents using cellular phones, if necessary, to contact emergency first-responders by using the full 10-digit number of a public safety answering point emergency call center (PSAP) or their local police department.
The Massachusetts State Police operate three regional PSAPs and a fourth is staffed by State 911 at MSP headquarters in Framingham. Residents using a cell phone to call for emergency services should first try to dial 911, and if it does not work, instead call one of these four 10-digit numbers: Shelburne Control PSAP 413-625-8200 New Braintree Control PSAP 508-867-1170 Northampton Control PSAP 413-586-1508 Framingham PSAP 508-820-2121.
Residents using cell phones may also, if dialing 911 does not work, may call the 10-digit general number of their local police department. They may also call the 10-digit number of a State Police Barracks in their area (search for barracks location and phone numbers on the MSP web site at www.mass.gov and search for State Police locations). Other agencies operate additional PSAPs in Massachusetts beyond those listed above.
The outage seems to only be affecting wireless 911 calls; landline calls to 911 appear to be working.
CenturyLink has indicted it is working on repairing what the company called a “network element” that is causing the problem.
*MEMA Operations*
The State Emergency Operations Center (SEOC) is currently operating at Level 1 (Steady State Monitoring). MEMA will continue to monitor the situation and will disseminate additional Situational Awareness Statements as necessary. Cities and towns wishing to share information or request assistance should contact their MEMA Regional office or MEMA’s 24/7 communications center at 508-820-2000.
*Stay Informed:*
Utilize *Massachusetts Alerts* to receive emergency notifications and information from the Massachusetts Emergency Management Agency and the National Weather Service. *Massachusetts Alerts* is a free app that is available for Android and iPhones. To learn more about *Massachusetts Alerts*, and for information on how to download the free app onto your smartphone, visit: http://www.mass.gov/service-details/massachusetts-alerts-smartphone-app.
Utilize MEMA’s real-time power outage viewer to stay informed about current power outages in your community and region, and across the state, including information from utility companies about restoration times: http://mema.mapsonline.net/public.html
Utilize MEMA’s live weather radar and forecasting tools: http://www.mass.gov/map-resources
*Online Resources:*For additional information and resources, visit: Massachusetts Emergency Management Agency at www.mass.gov/mema MEMA's Facebook page: http://www.facebook.com/MassachusettsEMA MEMA Twitter: @MassEMA <http://twitter.com/massema> Federal Emergency Management Agency at www.fema.gov National Weather Service Boston/Norton at www.weather.gov/boston National Weather Service Albany, NY at www.weather.gov/albany National Weather Service Weather Prediction Center: http://www.wpc.ncep.noaa.gov National Weather Service Storm Prediction Center: http://www.spc.noaa.gov/ Mass211 at www.mass211.org
------------------------------
This message has been sent from MEMA's ESF Team Listserv. To unsubscribe from this listserv, call MEMA Operations at (508) 820-2000.
+1.781.300.0000
On Fri, Dec 28, 2018 at 9:52 AM Matt Brennan via Outages < outages@outages.org> wrote:
Most public safety agencies in Massachusetts posted on Facebook and Twitter between 6 AM and 8 AM EST about wireless 911 outages caused by CenturyLink. It was affecting wireless routing, so some calls were being misrouted to other towns, or even other states; some callers were getting reorder tones or other error messages. The problem was reported to be resolved prior to 8 AM EST.
Seeing reports from PSAP's that the issue started again in the last 15 minutes.
This is wireless only. Landline 911 calls were / are not affected.
On Fri, Dec 28, 2018 at 9:32 AM Mel Beckman via Outages < outages@outages.org> wrote:
Don't be a fear monger. I searched Facebook and found *three* posts about 911 outages, all of them simply reposts of local news stories.
Facebook is not "blowing up".
Sheesh!
[image: image1.PNG]
-mel via cell
-mel via cell
On Dec 28, 2018, at 5:23 AM, Josh Luthman via Outages < outages@outages.org> wrote:
Facebook is blowing up with 911 outages and claiming it's CenturyLink to blame.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Fri, Dec 28, 2018, 2:09 AM Erik Sundberg via Outages < outages@outages.org wrote:
Maybe light at the end of the tunnel...
latest we got from Centurylink was at 10:45pm CST.
Our engineers and technicians have identified the network element that has affected our customer services. Services are restoring, and the current estimated time for full recovery is four hours.
CenturyLink will be conducting an extensive post-incident investigation and root cause analysis to provide follow-up information to our customers impacted by this event. We will provide one more Sales Alert from this address once services are fully restored.
*From:* Rich Breton <richdbreton@gmail.com> *Sent:* Thursday, December 27, 2018 2:59 PM *To:* Christopher Conley <CConley@forsmarshgroup.com> *Cc:* Erik Sundberg <ESundberg@nitelusa.com>; outages@outages.org *Subject:* Re: [outages] CenturyLink Outages this morning
I understand that the transport waves went out first, and then they try to reroute to something in Arizona that begin introducing all of the broadcast traffic. How legit that is I’m not sure
Sent from my iPhone
On Dec 27, 2018, at 12:58 PM, Christopher Conley via Outages < outages@outages.org> wrote:
Finally received something useful from centurylink… I have more questions now like how broadcast traffic effect 4x Transport waves.
Ineptitude.
Christopher Conley
Systems Administrator | Fors Marsh Group
Arlington, VA
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Erik Sundberg via Outages *Sent:* Thursday, December 27, 2018 2:46 PM *To:* outages@outages.org *Subject:* Re: [outages] CenturyLink Outages this morning
Finally received something useful from centurylink… I have more questions now like how broadcast traffic effect 4x Transport waves.
Update:
On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership.
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Bruce Freshwater via Outages *Sent:* Thursday, December 27, 2018 1:24 PM *To:* outages@outages.org *Subject:* Re: [outages] CenturyLink Outages this morning
CenturyLink finally acknowledged:
Great Lakes Region CenturyLink Partners/Agents:
FYI. Our network is experiencing a disruption affecting customer services. Customers are currently unable to generate tickets through the Control Center portal.
We know how important these services are to our customers, and we are working to restore services as quickly as possible.
Our technical teams are working to resolve the issue, and we will provide updates as the situation progresses.
-bruce
On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages" < outages@outages.org> wrote:
So it's a bad day for centurylink.
The lady who answered the phone for there NOC said that we are having a nation wide outage and we are unable to open tickets at this time. Please call be in 60-90 mins.
She had no info on what was going on
------------------------------
*From:* Michael Bolton <mbolton@holmeselectricsecurity.com> *Sent:* Thursday, December 27, 2018 8:55:03 AM *To:* Erik Sundberg; outages@outages.org *Subject:* RE: CenturyLink Outages this morning
Seeing the same in NC. Circuits are up but the routing between POPs has changed and we are seeing higher than normal latency.
VoIP portal is down also. Allows you to login but you cannot make any changes or view any details.
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Erik Sundberg via Outages *Sent:* Thursday, December 27, 2018 9:45 AM *To:* outages@outages.org *Subject:* [outages] CenturyLink Outages this morning
*CAUTION:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We are seeing a lot of centurylink circuits and services down this morning.
Centurylink Control Center portal comes up but auth failing unable to login
Waves down or bouncing
Denver - Seattle (Bouncing)
Denver - Chicago (Bouncing)
New York - Los Angeles (Down)
Chicago - Atlanta (Down)
ENNI Down in Denver (Down)
Still tying to open tickets with them there portal is not working and stuck on hold right now.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Holmes Security Systems may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Several times late last night we had our weather alert radio activate and broadcast an unintelligible message about 911 being down. It wasn't reception problems. The cadence of the dumb computer voice made it nearly impossible to understand. https://photos.app.goo.gl/M4spwKAkKPnoSrJw9 The alert came through ~5 hours after 911 went offline everywhere. -A On Fri, Dec 28, 2018, 08:44 Dom Colangelo via Outages <outages@outages.org wrote:
Received a WEA from MEMA in Boston, MA noting that 911 is down for cellular users.
Dom Colangelo
Senior Windows Systems Administrator
Infrastructure and Online Services
Simmons University
617.521.2669
On Fri, Dec 28, 2018 at 9:56 AM Bob Antia via Outages <outages@outages.org> wrote:
Massachusetts emergency management agency (MEMA) sent this out this morning and followed up at 0736 that the issues had been resolved.
*Massachusetts Emergency Management Agency* ------------------------------
*Massachusetts Emergency Management Agency*
*Situational Awareness Statement*
*December 28, 2018 7:00 AM*
*Re: Cellular 911 Experiencing Technical Difficulties and Sporadic Outages*
The Massachusetts Emergency Management Agency is sharing the following information from the Commonwealth Fusion Center (CFC) with our local partners and stakeholders. MEMA also is in touch with the State 911 Department.
*From the Commonwealth Fusion Center*
*Situation:*
A nationwide outage affecting CenturyLink, an Internet communications company, is impacting wireless 911 call capability in various parts of the country. We believe Massachusetts has been impacted by the outage and urge residents using cellular phones, if necessary, to contact emergency first-responders by using the full 10-digit number of a public safety answering point emergency call center (PSAP) or their local police department.
The Massachusetts State Police operate three regional PSAPs and a fourth is staffed by State 911 at MSP headquarters in Framingham. Residents using a cell phone to call for emergency services should first try to dial 911, and if it does not work, instead call one of these four 10-digit numbers: Shelburne Control PSAP 413-625-8200 New Braintree Control PSAP 508-867-1170 Northampton Control PSAP 413-586-1508 Framingham PSAP 508-820-2121.
Residents using cell phones may also, if dialing 911 does not work, may call the 10-digit general number of their local police department. They may also call the 10-digit number of a State Police Barracks in their area (search for barracks location and phone numbers on the MSP web site at www.mass.gov and search for State Police locations). Other agencies operate additional PSAPs in Massachusetts beyond those listed above.
The outage seems to only be affecting wireless 911 calls; landline calls to 911 appear to be working.
CenturyLink has indicted it is working on repairing what the company called a “network element” that is causing the problem.
*MEMA Operations*
The State Emergency Operations Center (SEOC) is currently operating at Level 1 (Steady State Monitoring). MEMA will continue to monitor the situation and will disseminate additional Situational Awareness Statements as necessary. Cities and towns wishing to share information or request assistance should contact their MEMA Regional office or MEMA’s 24/7 communications center at 508-820-2000.
*Stay Informed:*
Utilize *Massachusetts Alerts* to receive emergency notifications and information from the Massachusetts Emergency Management Agency and the National Weather Service. *Massachusetts Alerts* is a free app that is available for Android and iPhones. To learn more about *Massachusetts Alerts*, and for information on how to download the free app onto your smartphone, visit: http://www.mass.gov/service-details/massachusetts-alerts-smartphone-app.
Utilize MEMA’s real-time power outage viewer to stay informed about current power outages in your community and region, and across the state, including information from utility companies about restoration times: http://mema.mapsonline.net/public.html
Utilize MEMA’s live weather radar and forecasting tools: http://www.mass.gov/map-resources
*Online Resources:*For additional information and resources, visit: Massachusetts Emergency Management Agency at www.mass.gov/mema MEMA's Facebook page: http://www.facebook.com/MassachusettsEMA MEMA Twitter: @MassEMA <http://twitter.com/massema> Federal Emergency Management Agency at www.fema.gov National Weather Service Boston/Norton at www.weather.gov/boston National Weather Service Albany, NY at www.weather.gov/albany National Weather Service Weather Prediction Center: http://www.wpc.ncep.noaa.gov National Weather Service Storm Prediction Center: http://www.spc.noaa.gov/ Mass211 at www.mass211.org
------------------------------
This message has been sent from MEMA's ESF Team Listserv. To unsubscribe from this listserv, call MEMA Operations at (508) 820-2000.
+1.781.300.0000
On Fri, Dec 28, 2018 at 9:52 AM Matt Brennan via Outages < outages@outages.org> wrote:
Most public safety agencies in Massachusetts posted on Facebook and Twitter between 6 AM and 8 AM EST about wireless 911 outages caused by CenturyLink. It was affecting wireless routing, so some calls were being misrouted to other towns, or even other states; some callers were getting reorder tones or other error messages. The problem was reported to be resolved prior to 8 AM EST.
Seeing reports from PSAP's that the issue started again in the last 15 minutes.
This is wireless only. Landline 911 calls were / are not affected.
On Fri, Dec 28, 2018 at 9:32 AM Mel Beckman via Outages < outages@outages.org> wrote:
Don't be a fear monger. I searched Facebook and found *three* posts about 911 outages, all of them simply reposts of local news stories.
Facebook is not "blowing up".
Sheesh!
[image: image1.PNG]
-mel via cell
-mel via cell
On Dec 28, 2018, at 5:23 AM, Josh Luthman via Outages < outages@outages.org> wrote:
Facebook is blowing up with 911 outages and claiming it's CenturyLink to blame.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Fri, Dec 28, 2018, 2:09 AM Erik Sundberg via Outages < outages@outages.org wrote:
Maybe light at the end of the tunnel...
latest we got from Centurylink was at 10:45pm CST.
Our engineers and technicians have identified the network element that has affected our customer services. Services are restoring, and the current estimated time for full recovery is four hours.
CenturyLink will be conducting an extensive post-incident investigation and root cause analysis to provide follow-up information to our customers impacted by this event. We will provide one more Sales Alert from this address once services are fully restored.
*From:* Rich Breton <richdbreton@gmail.com> *Sent:* Thursday, December 27, 2018 2:59 PM *To:* Christopher Conley <CConley@forsmarshgroup.com> *Cc:* Erik Sundberg <ESundberg@nitelusa.com>; outages@outages.org *Subject:* Re: [outages] CenturyLink Outages this morning
I understand that the transport waves went out first, and then they try to reroute to something in Arizona that begin introducing all of the broadcast traffic. How legit that is I’m not sure
Sent from my iPhone
On Dec 27, 2018, at 12:58 PM, Christopher Conley via Outages < outages@outages.org> wrote:
Finally received something useful from centurylink… I have more questions now like how broadcast traffic effect 4x Transport waves.
Ineptitude.
Christopher Conley
Systems Administrator | Fors Marsh Group
Arlington, VA
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Erik Sundberg via Outages *Sent:* Thursday, December 27, 2018 2:46 PM *To:* outages@outages.org *Subject:* Re: [outages] CenturyLink Outages this morning
Finally received something useful from centurylink… I have more questions now like how broadcast traffic effect 4x Transport waves.
Update:
On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership.
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Bruce Freshwater via Outages *Sent:* Thursday, December 27, 2018 1:24 PM *To:* outages@outages.org *Subject:* Re: [outages] CenturyLink Outages this morning
CenturyLink finally acknowledged:
Great Lakes Region CenturyLink Partners/Agents:
FYI. Our network is experiencing a disruption affecting customer services. Customers are currently unable to generate tickets through the Control Center portal.
We know how important these services are to our customers, and we are working to restore services as quickly as possible.
Our technical teams are working to resolve the issue, and we will provide updates as the situation progresses.
-bruce
On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages" < outages@outages.org> wrote:
So it's a bad day for centurylink.
The lady who answered the phone for there NOC said that we are having a nation wide outage and we are unable to open tickets at this time. Please call be in 60-90 mins.
She had no info on what was going on
------------------------------
*From:* Michael Bolton <mbolton@holmeselectricsecurity.com> *Sent:* Thursday, December 27, 2018 8:55:03 AM *To:* Erik Sundberg; outages@outages.org *Subject:* RE: CenturyLink Outages this morning
Seeing the same in NC. Circuits are up but the routing between POPs has changed and we are seeing higher than normal latency.
VoIP portal is down also. Allows you to login but you cannot make any changes or view any details.
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Erik Sundberg via Outages *Sent:* Thursday, December 27, 2018 9:45 AM *To:* outages@outages.org *Subject:* [outages] CenturyLink Outages this morning
*CAUTION:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We are seeing a lot of centurylink circuits and services down this morning.
Centurylink Control Center portal comes up but auth failing unable to login
Waves down or bouncing
Denver - Seattle (Bouncing)
Denver - Chicago (Bouncing)
New York - Los Angeles (Down)
Chicago - Atlanta (Down)
ENNI Down in Denver (Down)
Still tying to open tickets with them there portal is not working and stuck on hold right now.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Holmes Security Systems may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
------------------------------
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Facebook is blowing up with 911 outages and claiming it's CenturyLink to blame
For what it's worth, here is a list of PSAPs which were reported as having problems, presumably due to the CenturyLink outages. Can't be sure if it's complete or not: Malden Police Department, MA Seekonk Police Department, MA Sharon Police Department, MA Amesbury Police Department, MA Georgetown Police Department, MA Freetown Police Department, MA Nantucket Police Department, MA North Attleboro Police Department, MA Cambridge Communications, MA McKinney Police Department, TX – 2106 MT Cleburne Police Department, TX – 2113 MT Blaine County Sheriff, MT Colstrip Police Department, MT Liberty County 9-1-1, MT Phillips County Sheriff’s Office, MT Powder River County Sheriff, MT Troy Area Dispatch, MT Valley County Sheriff’s Department, MT Montgomery County Sheriff’s Office, IA Des Moines Police Department, IA Clearlake Police Department, IA
We are seeing the same thing in Arizona and Texas From: Outages <outages-bounces@outages.org> On Behalf Of Erik Sundberg via Outages Sent: Thursday, December 27, 2018 7:45 AM To: outages@outages.org Subject: [outages] CenturyLink Outages this morning We are seeing a lot of centurylink circuits and services down this morning. Centurylink Control Center portal comes up but auth failing unable to login Waves down or bouncing Denver - Seattle (Bouncing) Denver - Chicago (Bouncing) New York - Los Angeles (Down) Chicago - Atlanta (Down) ENNI Down in Denver (Down) Still tying to open tickets with them there portal is not working and stuck on hold right now. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
participants (16)
-
Aaron C. de Bruyn -
Andy Jabbour -
Bob Antia -
Bruce Freshwater -
Christopher Conley -
Dom Colangelo -
Erik Sundberg -
Jason Baugher -
Jordan Morris -
Josh Luthman -
Matt Brennan -
Mel Beckman -
Michael Bolton -
Phil Lavin -
Rich Breton -
Rob_Knowles＠navyfederal.org