Metro & North Atlanta (AT&T Metro-Ethernet Outage)

Any other operator seeing a massive AT&T metro-ethernet outage in the Atlanta territory? Appears to have started at roughly 11:50AM with partial recovery at 12:15PM and back out of service at 12:16:15PM. We are also being informed that this has affected some national broadcast video traffic (IP) transport as well.

We are seeing this as well. On Thu, Sep 20, 2012 at 1:36 PM, Kraig Beahn <Kraig@l2net.com> wrote:
Any other operator seeing a massive AT&T metro-ethernet outage in the Atlanta territory?
Appears to have started at roughly 11:50AM with partial recovery at 12:15PM and back out of service at 12:16:15PM.
We are also being informed that this has affected some national broadcast video traffic (IP) transport as well.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Just heard that a core Cisco 7600 router failed causing the outage. On Thu, Sep 20, 2012 at 1:39 PM, Aubrey Wells <aubrey@vocalcloud.com> wrote:
We are seeing this as well.
On Thu, Sep 20, 2012 at 1:36 PM, Kraig Beahn <Kraig@l2net.com> wrote:
Any other operator seeing a massive AT&T metro-ethernet outage in the Atlanta territory?
Appears to have started at roughly 11:50AM with partial recovery at 12:15PM and back out of service at 12:16:15PM.
We are also being informed that this has affected some national broadcast video traffic (IP) transport as well.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

I'm starting to sound like a broken record on outages@ and outages-discussion@: What happened to redundancy, *especially* if this was a "core router"? -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Thu, Sep 20, 2012 at 01:46:37PM -0400, Aubrey Wells wrote:
Just heard that a core Cisco 7600 router failed causing the outage.
On Thu, Sep 20, 2012 at 1:39 PM, Aubrey Wells <aubrey@vocalcloud.com> wrote:
We are seeing this as well.
On Thu, Sep 20, 2012 at 1:36 PM, Kraig Beahn <Kraig@l2net.com> wrote:
Any other operator seeing a massive AT&T metro-ethernet outage in the Atlanta territory?
Appears to have started at roughly 11:50AM with partial recovery at 12:15PM and back out of service at 12:16:15PM.
We are also being informed that this has affected some national broadcast video traffic (IP) transport as well.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Money. It all comes down to it. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Thu, Sep 20, 2012 at 2:01 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
I'm starting to sound like a broken record on outages@ and outages-discussion@:
What happened to redundancy, *especially* if this was a "core router"?
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Thu, Sep 20, 2012 at 01:46:37PM -0400, Aubrey Wells wrote:
Just heard that a core Cisco 7600 router failed causing the outage.
On Thu, Sep 20, 2012 at 1:39 PM, Aubrey Wells <aubrey@vocalcloud.com> wrote:
We are seeing this as well.
On Thu, Sep 20, 2012 at 1:36 PM, Kraig Beahn <Kraig@l2net.com> wrote:
Any other operator seeing a massive AT&T metro-ethernet outage in the Atlanta territory?
Appears to have started at roughly 11:50AM with partial recovery at 12:15PM and back out of service at 12:16:15PM.
We are also being informed that this has affected some national broadcast video traffic (IP) transport as well.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

On 9/20/12 11:01 AM, Jeremy Chadwick wrote:
I'm starting to sound like a broken record on outages@ and outages-discussion@:
What happened to redundancy, *especially* if this was a "core router"?
That's just the way AT&T does opt-e-man stuff: two strands of fiber from every switch in the field directly back to some 7600 in the CO. It's "core" to all the customers hanging off it, but probably no further than that. ~Seth

I think it goes further than that. I have customers as far south as Newnan and as far north as Alpharetta down, all coming out of different CO/wirecenters. On Thu, Sep 20, 2012 at 3:00 PM, Seth Mattinen <sethm@rollernet.us> wrote:
On 9/20/12 11:01 AM, Jeremy Chadwick wrote:
I'm starting to sound like a broken record on outages@ and outages-discussion@:
What happened to redundancy, *especially* if this was a "core router"?
That's just the way AT&T does opt-e-man stuff: two strands of fiber from every switch in the field directly back to some 7600 in the CO. It's "core" to all the customers hanging off it, but probably no further than that.
~Seth _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

We are seeing 3 circuits with Earthlink down as well (likley reselling someone else. On Thu, Sep 20, 2012 at 3:03 PM, Aubrey Wells <aubrey@vocalcloud.com> wrote:
I think it goes further than that. I have customers as far south as Newnan and as far north as Alpharetta down, all coming out of different CO/wirecenters.
On Thu, Sep 20, 2012 at 3:00 PM, Seth Mattinen <sethm@rollernet.us> wrote:
On 9/20/12 11:01 AM, Jeremy Chadwick wrote:
I'm starting to sound like a broken record on outages@ and outages-discussion@:
What happened to redundancy, *especially* if this was a "core router"?
That's just the way AT&T does opt-e-man stuff: two strands of fiber from every switch in the field directly back to some 7600 in the CO. It's "core" to all the customers hanging off it, but probably no further than that.
~Seth _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

On 9/20/12 12:03 PM, Aubrey Wells wrote:
I think it goes further than that. I have customers as far south as Newnan and as far north as Alpharetta down, all coming out of different CO/wirecenters.
Then it could be the top router that takes the all uplinks from the aggregation ones that ties them all together that failed. The last time I inquired how opt-e-man was laid out it's pretty basic. It's the cheap no frills no redundancy Ethernet offering. Otherwise you have to pay for SONET. ~Seth

Update from ATT with more details: The issue has been isolated to a LAG bundle between two 7609 routers in Atlanta and ATS is isolating the offending VLAN so issue can be mitigated. Network teams have identified the traffic is impacting ATLNGACS4CW. The source of the traffic has been traced back to ATLNGAPP4AW. Network teams are attempting to back trace the source of the traffic on ATLNGAPP4AW to either another router or an access circuit on this device. Troubleshooting through the network continues to locate the root source of the VLAN loop. This is a process of removing LAG bundles and core trunks from service until the CPU and traffic normalize. The process is underway with ATS and the Cisco is being engaged for additional support. There is no ETTR at this time. On Thu, Sep 20, 2012 at 3:11 PM, Seth Mattinen <sethm@rollernet.us> wrote:
On 9/20/12 12:03 PM, Aubrey Wells wrote:
I think it goes further than that. I have customers as far south as Newnan and as far north as Alpharetta down, all coming out of different CO/wirecenters.
Then it could be the top router that takes the all uplinks from the aggregation ones that ties them all together that failed. The last time I inquired how opt-e-man was laid out it's pretty basic. It's the cheap no frills no redundancy Ethernet offering. Otherwise you have to pay for SONET.
~Seth _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

All our circuits are back up now. Anybody else getting good news right about now? Reportedly this took out 1000+ MetroE circuits. Bad day to be a net eng at AT&T... On Thu, Sep 20, 2012 at 3:32 PM, Aubrey Wells <aubrey@vocalcloud.com> wrote:
Update from ATT with more details:
The issue has been isolated to a LAG bundle between two 7609 routers in Atlanta and ATS is isolating the offending VLAN so issue can be mitigated. Network teams have identified the traffic is impacting ATLNGACS4CW. The source of the traffic has been traced back to ATLNGAPP4AW. Network teams are attempting to back trace the source of the traffic on ATLNGAPP4AW to either another router or an access circuit on this device. Troubleshooting through the network continues to locate the root source of the VLAN loop. This is a process of removing LAG bundles and core trunks from service until the CPU and traffic normalize. The process is underway with ATS and the Cisco is being engaged for additional support. There is no ETTR at this time.
On Thu, Sep 20, 2012 at 3:11 PM, Seth Mattinen <sethm@rollernet.us> wrote:
On 9/20/12 12:03 PM, Aubrey Wells wrote:
I think it goes further than that. I have customers as far south as Newnan and as far north as Alpharetta down, all coming out of different CO/wirecenters.
Then it could be the top router that takes the all uplinks from the aggregation ones that ties them all together that failed. The last time I inquired how opt-e-man was laid out it's pretty basic. It's the cheap no frills no redundancy Ethernet offering. Otherwise you have to pay for SONET.
~Seth _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Our NOC is reporting that all of our metro-ethernet circuits were restored at 3:40:34PM (Eastern). Started with slight packet-loss on return to service and a little bit of latency as I would probably expect based upon the methodology in which AT&T is utilizing to troubleshoot/resolve the issue. AT&T has not confirmed continuity as of yet so i'm assuming they are still isolating the issue even though traffic is passing.

Our 3 earthlink fiber circuits have restored as of 7 minutes ago On Thu, Sep 20, 2012 at 3:32 PM, Aubrey Wells <aubrey@vocalcloud.com> wrote:
Update from ATT with more details:
The issue has been isolated to a LAG bundle between two 7609 routers in Atlanta and ATS is isolating the offending VLAN so issue can be mitigated. Network teams have identified the traffic is impacting ATLNGACS4CW. The source of the traffic has been traced back to ATLNGAPP4AW. Network teams are attempting to back trace the source of the traffic on ATLNGAPP4AW to either another router or an access circuit on this device. Troubleshooting through the network continues to locate the root source of the VLAN loop. This is a process of removing LAG bundles and core trunks from service until the CPU and traffic normalize. The process is underway with ATS and the Cisco is being engaged for additional support. There is no ETTR at this time.
On Thu, Sep 20, 2012 at 3:11 PM, Seth Mattinen <sethm@rollernet.us> wrote:
On 9/20/12 12:03 PM, Aubrey Wells wrote:
I think it goes further than that. I have customers as far south as Newnan and as far north as Alpharetta down, all coming out of different CO/wirecenters.
Then it could be the top router that takes the all uplinks from the aggregation ones that ties them all together that failed. The last time I inquired how opt-e-man was laid out it's pretty basic. It's the cheap no frills no redundancy Ethernet offering. Otherwise you have to pay for SONET.
~Seth _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Every AT&T rep I've ever dealt with on their Metro E offering says that AT&T does not guarantee any kind of redundancy in their Metro E network. That's probably there to protect their more expensive offerings like SONET, etc. RIchey -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jeremy Chadwick Sent: Thursday, September 20, 2012 2:01 PM To: outages@outages.org Subject: Re: [outages] Metro & North Atlanta (AT&T Metro-Ethernet Outage) I'm starting to sound like a broken record on outages@ and outages-discussion@: What happened to redundancy, *especially* if this was a "core router"? -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Thu, Sep 20, 2012 at 01:46:37PM -0400, Aubrey Wells wrote:
Just heard that a core Cisco 7600 router failed causing the outage.
On Thu, Sep 20, 2012 at 1:39 PM, Aubrey Wells <aubrey@vocalcloud.com> wrote:
We are seeing this as well.
On Thu, Sep 20, 2012 at 1:36 PM, Kraig Beahn <Kraig@l2net.com> wrote:
Any other operator seeing a massive AT&T metro-ethernet outage in the Atlanta territory?
Appears to have started at roughly 11:50AM with partial recovery at 12:15PM and back out of service at 12:16:15PM.
We are also being informed that this has affected some national broadcast video traffic (IP) transport as well.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Qwest has a recording on their 800 number saying there is an outage in Atlanta. Maybe something larger going on in ATL. Thanks, ML On Thu, Sep 20, 2012 at 11:36 AM, Kraig Beahn <Kraig@l2net.com> wrote:
Any other operator seeing a massive AT&T metro-ethernet outage in the Atlanta territory?
Appears to have started at roughly 11:50AM with partial recovery at 12:15PM and back out of service at 12:16:15PM.
We are also being informed that this has affected some national broadcast video traffic (IP) transport as well.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Mauricio Lizano 8821-6575 http://cr.linkedin.com/pub/mauricio-lizano/14/162/5b5

There is. Parts of AT&T's Metro Ethernet is down in Atlanta. Likely Qwest uses them for something. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Mauricio Lizano Sent: Thursday, September 20, 2012 1:46 PM To: Kraig Beahn Cc: outages@outages.org Subject: Re: [outages] Metro & North Atlanta (AT&T Metro-Ethernet Outage) Qwest has a recording on their 800 number saying there is an outage in Atlanta. Maybe something larger going on in ATL. Thanks, ML On Thu, Sep 20, 2012 at 11:36 AM, Kraig Beahn <Kraig@l2net.com> wrote: Any other operator seeing a massive AT&T metro-ethernet outage in the Atlanta territory? Appears to have started at roughly 11:50AM with partial recovery at 12:15PM and back out of service at 12:16:15PM. We are also being informed that this has affected some national broadcast video traffic (IP) transport as well. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages -- Mauricio Lizano 8821-6575 http://cr.linkedin.com/pub/mauricio-lizano/14/162/5b5

We are seeing this in Gainesville, GA. It is impacting Windstream (formerly Paetec) feed as well, they're probably subbing from ATTs line. Rob++ ----- Original Message -----
From: "Kraig Beahn" <Kraig@l2net.com> To: outages@outages.org Sent: Thursday, September 20, 2012 1:36:23 PM Subject: [outages] Metro & North Atlanta (AT&T Metro-Ethernet Outage)
Any other operator seeing a massive AT&T metro-ethernet outage in the Atlanta territory? Appears to have started at roughly 11:50AM with partial recovery at 12:15PM and back out of service at 12:16:15PM . We are also being informed that this has affected some national broadcast video traffic (IP) transport as well. -- pubkey: http://www.talgas.com/misc/rob_windsor-pubkey.txt Internet: windsor@warthog.com Life: Rob@Allen.Texas.USA.Earth
"They couldn't hit an elephant at this distance." -- Major General John Sedgwick

We currently do not have any customers reporting problems with their Metro-E circuits leaving our Data Center. Both from Windstream and AT&T direct. Regards, Bradley D. Bopp Director Of Engineering http://www.nationalnet.com http://as22384.peeringdb.net NationalNet is committed to the highest level of Customer Service available in the Web Hosting business. If you have any questions, comments or concerns feel free to contact us at 678-247-7000 ext 1 (or toll-free, 888-4-NATNET). On Sep 20, 2012, at Sep 20, 2012, 2:13 PM, Rob Windsor <windsor@warthog.com> wrote:
We are seeing this in Gainesville, GA.
It is impacting Windstream (formerly Paetec) feed as well, they're probably subbing from ATTs line.
Rob++
----- Original Message -----
From: "Kraig Beahn" <Kraig@l2net.com> To: outages@outages.org Sent: Thursday, September 20, 2012 1:36:23 PM Subject: [outages] Metro & North Atlanta (AT&T Metro-Ethernet Outage)
Any other operator seeing a massive AT&T metro-ethernet outage in the Atlanta territory?
Appears to have started at roughly 11:50AM with partial recovery at 12:15PM and back out of service at 12:16:15PM .
We are also being informed that this has affected some national broadcast video traffic (IP) transport as well. -- pubkey: http://www.talgas.com/misc/rob_windsor-pubkey.txt Internet: windsor@warthog.com Life: Rob@Allen.Texas.USA.Earth
"They couldn't hit an elephant at this distance." -- Major General John Sedgwick
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Kraig quoted here on this issue: http://www.fiercetelecom.com/story/atts-metro-ethernet-services-atlanta-affe cted-outage/2012-09-20 Frank From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Kraig Beahn Sent: Thursday, September 20, 2012 12:36 PM To: outages@outages.org Subject: [outages] Metro & North Atlanta (AT&T Metro-Ethernet Outage) Any other operator seeing a massive AT&T metro-ethernet outage in the Atlanta territory? Appears to have started at roughly 11:50AM with partial recovery at 12:15PM and back out of service at 12:16:15PM. We are also being informed that this has affected some national broadcast video traffic (IP) transport as well.
participants (12)
-
Aubrey Wells
-
Bradley Bopp
-
Eric Wieling
-
Frank Bulk
-
Jeremy Chadwick
-
Josh Luthman
-
Kraig Beahn
-
Mauricio Lizano
-
Mitch
-
My List Account
-
Rob Windsor
-
Seth Mattinen