Saw our Centurylink Ethernet circuit lose L3 connectivity in SWFL around 0400 EDT. ANS/ABS states it's a nationwide routing issue. No ETR.
From what I've heard its about 10 states affected, mostly legacy Centurylink/embarq
Seems to be affecting DSL, T1 on our end, and from what you've said ethernet as well. On Tue, May 7, 2013 at 8:33 AM, Marco Prechel <marcoprechel@gmail.com>wrote:
Saw our Centurylink Ethernet circuit lose L3 connectivity in SWFL around 0400 EDT. ANS/ABS states it's a nationwide routing issue. No ETR. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
DSL seems fine in Ohio. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On May 7, 2013 8:49 AM, "Mitch" <mitpatterson@gmail.com> wrote:
From what I've heard its about 10 states affected, mostly legacy Centurylink/embarq
Seems to be affecting DSL, T1 on our end, and from what you've said ethernet as well.
On Tue, May 7, 2013 at 8:33 AM, Marco Prechel <marcoprechel@gmail.com>wrote:
Saw our Centurylink Ethernet circuit lose L3 connectivity in SWFL around 0400 EDT. ANS/ABS states it's a nationwide routing issue. No ETR. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Same issue in Tallahasee and Orlando FL. CenturyLink's outage has virtually crippled the Florida State Capital. http://cdn.l2net.com/content/video/CTLFL-Outage-BGP-05072013.mp4 Kraig Beahn, CEO L2Networks Corp. Office 850.701.4999 Operations 888.695.5905 Fax 877.734.3843 On May 7, 2013 8:43 AM, "Marco Prechel" <marcoprechel@gmail.com> wrote:
Saw our Centurylink Ethernet circuit lose L3 connectivity in SWFL around 0400 EDT. ANS/ABS states it's a nationwide routing issue. No ETR. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Centurylink Ethernet up and running in there hometown of Monroe LA as of 8:00 AM. On Tuesday, May 7, 2013, Kraig Beahn wrote:
Same issue in Tallahasee and Orlando FL. CenturyLink's outage has virtually crippled the Florida State Capital.
http://cdn.l2net.com/content/video/CTLFL-Outage-BGP-05072013.mp4
Kraig Beahn, CEO L2Networks Corp.
Office 850.701.4999 Operations 888.695.5905 Fax 877.734.3843 On May 7, 2013 8:43 AM, "Marco Prechel" <marcoprechel@gmail.com<javascript:_e({}, 'cvml', 'marcoprechel@gmail.com');>> wrote:
Saw our Centurylink Ethernet circuit lose L3 connectivity in SWFL around 0400 EDT. ANS/ABS states it's a nationwide routing issue. No ETR. _______________________________________________ Outages mailing list Outages@outages.org <javascript:_e({}, 'cvml', 'Outages@outages.org');> https://puck.nether.net/mailman/listinfo/outages
-- Jeremie Chism Triton Communications
Tons of latency in VA on IQ Enhanced Bryan Riley OnCall Telecom ________________________________ From: Outages [outages-bounces@outages.org] On Behalf Of Jeremie Chism [jchism2@gmail.com] Sent: Tuesday, May 07, 2013 9:03 AM To: Kraig Beahn Cc: outages@outages.org Subject: Re: [outages] Centurylink nationwide outage Centurylink Ethernet up and running in there hometown of Monroe LA as of 8:00 AM. On Tuesday, May 7, 2013, Kraig Beahn wrote: Same issue in Tallahasee and Orlando FL. CenturyLink's outage has virtually crippled the Florida State Capital. http://cdn.l2net.com/content/video/CTLFL-Outage-BGP-05072013.mp4 Kraig Beahn, CEO L2Networks Corp. Office 850.701.4999 Operations 888.695.5905 Fax 877.734.3843 On May 7, 2013 8:43 AM, "Marco Prechel" <marcoprechel@gmail.com<https://webmail.geeksoncall.com/owa/UrlBlockedError.aspx>> wrote: Saw our Centurylink Ethernet circuit lose L3 connectivity in SWFL around 0400 EDT. ANS/ABS states it's a nationwide routing issue. No ETR. _______________________________________________ Outages mailing list Outages@outages.org<https://webmail.geeksoncall.com/owa/UrlBlockedError.aspx> https://puck.nether.net/mailman/listinfo/outages -- Jeremie Chism Triton Communications
Been down and out for the count here in Kansas City on AS22186 (Legacy Embarq/Sprint) / 76.7.0.0/19 since around 3AM central time. The routing within CenturyLinks' network still appears to be functional (DSL customers), but anything outside of a local region with CenturyLink appears to not function. Unfortunately, I've not much more to say than that. =/ Regards, Brett On 5/7/2013 8:03 AM, Jeremie Chism wrote:
Centurylink Ethernet up and running in there hometown of Monroe LA as of 8:00 AM.
On Tuesday, May 7, 2013, Kraig Beahn wrote:
Same issue in Tallahasee and Orlando FL. CenturyLink's outage has virtually crippled the Florida State Capital.
http://cdn.l2net.com/content/video/CTLFL-Outage-BGP-05072013.mp4
Kraig Beahn, CEO L2Networks Corp.
Office 850.701.4999 Operations 888.695.5905 Fax 877.734.3843
On May 7, 2013 8:43 AM, "Marco Prechel" <marcoprechel@gmail.com <javascript:_e({}, 'cvml', 'marcoprechel@gmail.com');>> wrote:
Saw our Centurylink Ethernet circuit lose L3 connectivity in SWFL around 0400 EDT. ANS/ABS states it's a nationwide routing issue. No ETR. _______________________________________________ Outages mailing list Outages@outages.org <javascript:_e({}, 'cvml', 'Outages@outages.org');> https://puck.nether.net/mailman/listinfo/outages
-- Jeremie Chism Triton Communications
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Have a number of customer DSL sites down as well, southern US. Help Desk is telling me they can't get a ticket number from CL for this -- anyone have a master ticket from CL? --------------------------- Tom Mackintosh Sent from my iPhone On May 7, 2013, at 8:53 AM, Kraig Beahn <kraig@l2net.com> wrote:
Same issue in Tallahasee and Orlando FL. CenturyLink's outage has virtually crippled the Florida State Capital.
http://cdn.l2net.com/content/video/CTLFL-Outage-BGP-05072013.mp4
Kraig Beahn, CEO L2Networks Corp.
Office 850.701.4999 Operations 888.695.5905 Fax 877.734.3843
On May 7, 2013 8:43 AM, "Marco Prechel" <marcoprechel@gmail.com> wrote:
Saw our Centurylink Ethernet circuit lose L3 connectivity in SWFL around 0400 EDT. ANS/ABS states it's a nationwide routing issue. No ETR. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I first saw timeouts with www.qwest.com over IPv6 at 3:08 am Central -- I was wondering why, now I know. =) Frank -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Marco Prechel Sent: Tuesday, May 07, 2013 7:33 AM To: outages@outages.org Subject: [outages] Centurylink nationwide outage Saw our Centurylink Ethernet circuit lose L3 connectivity in SWFL around 0400 EDT. ANS/ABS states it's a nationwide routing issue. No ETR. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Sounds like it was scheduled maintenance gone bad: "Perrine said he spoke with CenturyLink at 6 a.m. where they advised him the company was doing scheduled maintenance. At 7:35 a.m. they told him something had gone wrong during the maintenance and it was affecting customers in 13 states." http://www.tallahassee.com/article/20130507/NEWS/130507010/Centurylink-outag e-affecting-local-internet-provider?nclick_check=1 I bite my tongue in regards to Network Tallahasee's singlehomedness and Perrine's comment on his questioning of the timing of the update. I encourage all my competitors to stay single homed and to do their maintenance Sunday morning, while their vendors are at their lowest staffing levels. Frank -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk Sent: Tuesday, May 07, 2013 8:29 AM To: 'Marco Prechel'; outages@outages.org Subject: Re: [outages] Centurylink nationwide outage I first saw timeouts with www.qwest.com over IPv6 at 3:08 am Central -- I was wondering why, now I know. =) Frank -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Marco Prechel Sent: Tuesday, May 07, 2013 7:33 AM To: outages@outages.org Subject: [outages] Centurylink nationwide outage Saw our Centurylink Ethernet circuit lose L3 connectivity in SWFL around 0400 EDT. ANS/ABS states it's a nationwide routing issue. No ETR. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Same thing happened in Monroe LA that took down all of north Louisiana. It was an update that went bad and the switch had to be manually checked line by line. No backup was done before the maintenance. 16 hours of downtime about 45 days ago. On Tuesday, May 7, 2013, Frank Bulk wrote:
Sounds like it was scheduled maintenance gone bad: "Perrine said he spoke with CenturyLink at 6 a.m. where they advised him the company was doing scheduled maintenance. At 7:35 a.m. they told him something had gone wrong during the maintenance and it was affecting customers in 13 states."
http://www.tallahassee.com/article/20130507/NEWS/130507010/Centurylink-outag e-affecting-local-internet-provider?nclick_check=1
I bite my tongue in regards to Network Tallahasee's singlehomedness and Perrine's comment on his questioning of the timing of the update.
I encourage all my competitors to stay single homed and to do their maintenance Sunday morning, while their vendors are at their lowest staffing levels.
Frank
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org <javascript:;>] On Behalf Of Frank Bulk Sent: Tuesday, May 07, 2013 8:29 AM To: 'Marco Prechel'; outages@outages.org <javascript:;> Subject: Re: [outages] Centurylink nationwide outage
I first saw timeouts with www.qwest.com over IPv6 at 3:08 am Central -- I was wondering why, now I know. =)
Frank
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org <javascript:;>] On Behalf Of Marco Prechel Sent: Tuesday, May 07, 2013 7:33 AM To: outages@outages.org <javascript:;> Subject: [outages] Centurylink nationwide outage
Saw our Centurylink Ethernet circuit lose L3 connectivity in SWFL around 0400 EDT. ANS/ABS states it's a nationwide routing issue. No ETR. _______________________________________________ Outages mailing list Outages@outages.org <javascript:;> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <javascript:;> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <javascript:;> https://puck.nether.net/mailman/listinfo/outages
-- Jeremie Chism Triton Communications
While going over some of our BGP data I noticed significant instability for AS5668 CenturyTel Internet Holdings, Inc. See for example: http://portal.bgpmon.net/data/AS5668-outages-may7-2013.png Clearly visible are the routes flapping (few minute outages) between 08:06 UTC and 15:34. Below an example for 98.125.0.0/16 (CenturyTel Service Group, LLC) down for 3 minutes 2013-05-07 08:06 - 2013-05-07 08:09 down for 3 minutes 2013-05-07 08:25 - 2013-05-07 08:28 down for 3 minutes 2013-05-07 08:53 - 2013-05-07 08:56 down for 4 minutes 2013-05-07 09:04 - 2013-05-07 09:08 down for 4 minutes 2013-05-07 09:20 - 2013-05-07 09:24 down for 3 minutes 2013-05-07 10:05 - 2013-05-07 10:08 down for 5 minutes 2013-05-07 10:11 - 2013-05-07 10:16 down for 3 minutes 2013-05-07 10:56 - 2013-05-07 10:59 down for 3 minutes 2013-05-07 12:14 - 2013-05-07 12:17 down for 3 minutes 2013-05-07 12:34 - 2013-05-07 12:37 down for 4 minutes 2013-05-07 13:02 - 2013-05-07 13:06 down for 3 minutes 2013-05-07 13:12 - 2013-05-07 13:15 down for 3 minutes 2013-05-07 14:11 - 2013-05-07 14:14 down for 9 minutes 2013-05-07 14:24 - 2013-05-07 14:33 down for 3 minutes 2013-05-07 14:45 - 2013-05-07 14:48 down for 3 minutes 2013-05-07 14:55 - 2013-05-07 14:58 down for 3 minutes 2013-05-07 15:24 - 2013-05-07 15:27 down for 3 minutes 2013-05-07 15:31 - 2013-05-07 15:34 Cheers, Andree .-- My secret spy satellite informs me that at 2013-05-07 9:33 AM Frank Bulk wrote:
Sounds like it was scheduled maintenance gone bad: "Perrine said he spoke with CenturyLink at 6 a.m. where they advised him the company was doing scheduled maintenance. At 7:35 a.m. they told him something had gone wrong during the maintenance and it was affecting customers in 13 states." http://www.tallahassee.com/article/20130507/NEWS/130507010/Centurylink-outag e-affecting-local-internet-provider?nclick_check=1
I bite my tongue in regards to Network Tallahasee's singlehomedness and Perrine's comment on his questioning of the timing of the update.
I encourage all my competitors to stay single homed and to do their maintenance Sunday morning, while their vendors are at their lowest staffing levels.
Frank
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk Sent: Tuesday, May 07, 2013 8:29 AM To: 'Marco Prechel'; outages@outages.org Subject: Re: [outages] Centurylink nationwide outage
I first saw timeouts with www.qwest.com over IPv6 at 3:08 am Central -- I was wondering why, now I know. =)
Frank
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Marco Prechel Sent: Tuesday, May 07, 2013 7:33 AM To: outages@outages.org Subject: [outages] Centurylink nationwide outage
Saw our Centurylink Ethernet circuit lose L3 connectivity in SWFL around 0400 EDT. ANS/ABS states it's a nationwide routing issue. No ETR. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Thanks for sharing that info. That lines up with us seeing HTTPv6 alarms clearing at 10:31 am Central, too. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Andree Toonk Sent: Tuesday, May 07, 2013 12:19 PM To: outages@outages.org Subject: Re: [outages] Centurylink nationwide outage While going over some of our BGP data I noticed significant instability for AS5668 CenturyTel Internet Holdings, Inc. See for example: http://portal.bgpmon.net/data/AS5668-outages-may7-2013.png Clearly visible are the routes flapping (few minute outages) between 08:06 UTC and 15:34. Below an example for 98.125.0.0/16 (CenturyTel Service Group, LLC) down for 3 minutes 2013-05-07 08:06 - 2013-05-07 08:09 down for 3 minutes 2013-05-07 08:25 - 2013-05-07 08:28 down for 3 minutes 2013-05-07 08:53 - 2013-05-07 08:56 down for 4 minutes 2013-05-07 09:04 - 2013-05-07 09:08 down for 4 minutes 2013-05-07 09:20 - 2013-05-07 09:24 down for 3 minutes 2013-05-07 10:05 - 2013-05-07 10:08 down for 5 minutes 2013-05-07 10:11 - 2013-05-07 10:16 down for 3 minutes 2013-05-07 10:56 - 2013-05-07 10:59 down for 3 minutes 2013-05-07 12:14 - 2013-05-07 12:17 down for 3 minutes 2013-05-07 12:34 - 2013-05-07 12:37 down for 4 minutes 2013-05-07 13:02 - 2013-05-07 13:06 down for 3 minutes 2013-05-07 13:12 - 2013-05-07 13:15 down for 3 minutes 2013-05-07 14:11 - 2013-05-07 14:14 down for 9 minutes 2013-05-07 14:24 - 2013-05-07 14:33 down for 3 minutes 2013-05-07 14:45 - 2013-05-07 14:48 down for 3 minutes 2013-05-07 14:55 - 2013-05-07 14:58 down for 3 minutes 2013-05-07 15:24 - 2013-05-07 15:27 down for 3 minutes 2013-05-07 15:31 - 2013-05-07 15:34 Cheers, Andree .-- My secret spy satellite informs me that at 2013-05-07 9:33 AM Frank Bulk wrote:
Sounds like it was scheduled maintenance gone bad: "Perrine said he spoke with CenturyLink at 6 a.m. where they advised him the company was doing scheduled maintenance. At 7:35 a.m. they told him something had gone wrong during the maintenance and it was affecting customers in 13 states."
http://www.tallahassee.com/article/20130507/NEWS/130507010/Centurylink-outag
e-affecting-local-internet-provider?nclick_check=1
I bite my tongue in regards to Network Tallahasee's singlehomedness and Perrine's comment on his questioning of the timing of the update.
I encourage all my competitors to stay single homed and to do their maintenance Sunday morning, while their vendors are at their lowest staffing levels.
Frank
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk Sent: Tuesday, May 07, 2013 8:29 AM To: 'Marco Prechel'; outages@outages.org Subject: Re: [outages] Centurylink nationwide outage
I first saw timeouts with www.qwest.com over IPv6 at 3:08 am Central -- I was wondering why, now I know. =)
Frank
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Marco Prechel Sent: Tuesday, May 07, 2013 7:33 AM To: outages@outages.org Subject: [outages] Centurylink nationwide outage
Saw our Centurylink Ethernet circuit lose L3 connectivity in SWFL around 0400 EDT. ANS/ABS states it's a nationwide routing issue. No ETR. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (10)
-
Andree Toonk -
Brett Cooper -
Bryan Riley -
Frank Bulk -
Jeremie Chism -
Josh Luthman -
Kraig Beahn -
Marco Prechel -
Mitch -
Thomas W Mackintosh Jr