Level 3 outage in Columbia, SC

All our clients on Level 3 appear to be down at the moment. Casey Crowe IT Systems Engineer Palmetto Computer Consulting Office: (803) 832-2653 Cell: (803) 386-9302 www.palmettocc.net<http://www.palmettocc.net/> [PCC_logo]

We've got a reasonably large internet connection from them, and a circuit to Sumter from Columbia, and they're both up. Berry On 8/24/2016 1:24 PM, Casey Crowe via Outages wrote:
All our clients on Level 3 appear to be down at the moment.
*Casey Crowe*
/IT Systems Engineer/
Palmetto Computer Consulting
Office: (803) 832-2653
Cell: (803) 386-9302
www.palmettocc.net <http://www.palmettocc.net/>
PCC_logo
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

We are seeing issues in Charlotte, NC and Atlanta, GA and are routing away from Level3 at this time. -- Devon
Casey Crowe via Outages <mailto:outages@outages.org> August 24, 2016 at 1:24 PM
All our clients on Level 3 appear to be down at the moment.
*Casey Crowe*
/IT Systems Engineer/
Palmetto Computer Consulting
Office: (803) 832-2653
Cell: (803) 386-9302
www.palmettocc.net <http://www.palmettocc.net/>
PCC_logo
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Devon

Been getting a lot of alert issues to boxes at aws. I wonder if it is related... Regards, Dovid -----Original Message----- From: Devon True via Outages <outages@outages.org> Sender: "Outages" <outages-bounces@outages.org>Date: Wed, 24 Aug 2016 13:49:02 To: Casey Crowe<casey@palmettocc.net> Reply-To: Devon True <devon@noved.org> Cc: outages@outages.org<outages@outages.org> Subject: Re: [outages] Level 3 outage in Columbia, SC _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

From about 1p EDT to 1:45p EDT.
QTS Datacenter. Have RFO/RCA ticket opened with Level3. Rob++ -- pubkey: http://www.talgas.com/misc/rob_windsor-pubkey.txt Internet: windsor@warthog.com Life: Rob@Allen.Texas.USA.Earth "They couldn't hit an elephant at this distance." -- Major General John Sedgwick

I just lost routes from Louisiana and Ohio to Florida. They did the second they hand off to L3: From Cleveland: 4. te0-0-1-3.agr12.cle04.atlas.cogentco.com 0.0% 283 4.5 4.6 4.1 9.2 0.7 5. te0-7-0-12.ccr21.cle04.atlas.cogentco.com 0.0% 283 4.4 4.5 4.1 8.7 0.7 6. be2993.ccr21.yyz02.atlas.cogentco.com 0.0% 283 11.6 11.7 11.3 18.4 0.8 7. level3.yyz02.atlas.cogentco.com 0.0% 283 19.5 19.6 19.2 38.8 1.5 8. ??? From Louisiana (right outside New Orleans on Comcast): 2. 50-255-74-110-static.hfc.comcastbusiness.net 0.0% 260 12.5 23.7 10.5 169.5 24.0 96.120.15.53 3. te-8-1-ur01.laplace.la.malt.comcast.net 0.0% 260 11.3 23.8 11.3 174.0 19.4 4. te-6-4-ur01.houma.la.malt.comcast.net 0.0% 259 24.9 25.2 12.7 196.8 19.0 te-6-1-ur01.houma.la.malt.comcast.net 5. te-0-6-0-4-ar02.dannythomas.tn.malt.comcast.net 0.4% 259 40.0 51.3 33.5 343.8 35.3 6. be-22258-cr02.dallas.tx.ibone.comcast.net 0.0% 259 61.6 62.2 44.4 333.5 44.0 7. 50.242.148.30 70.2% 259 1553. 2697. 1138. 10436 2103. 8. ??? --John On 08/24/2016 01:09 PM, Rob Windsor via Outages wrote:
From about 1p EDT to 1:45p EDT.
QTS Datacenter.
Have RFO/RCA ticket opened with Level3.
Rob++

As an aside, we're peering with 3356 and not seeing issues. I don't think that's legacy TWT. Berry On 8/24/2016 1:24 PM, Casey Crowe via Outages wrote:
All our clients on Level 3 appear to be down at the moment.
*Casey Crowe*
/IT Systems Engineer/
Palmetto Computer Consulting
Office: (803) 832-2653
Cell: (803) 386-9302
www.palmettocc.net <http://www.palmettocc.net/>
PCC_logo
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Update from Level3: Event Conclusion Summary Outage Start: August 24, 2016 16:58 GMT Outage Stop: August 24, 2016 17:38 GMT Root Cause: Customers experienced internet access issues in multiple markets of the southeastern United States. Fix action: An interconnect was shut down between two backbone routers to restore services. Summary: IP services were impacted in multiple markets of the southeastern United States. Reports were received that customers were experiencing internet access issues. Multiple teams were engaged for investigation and the highest level of attention was put toward restoring services. During troubleshooting, a problematic interconnect was shut down between two backbone routers. This action was successful in restoring services. During a maintenance window at a later date, Level 3 will repair the interconnect which will return greater resiliency. This service impact has concluded; however, there may be additional analysis and discovery that occurs as the incident is reviewed by NOC management. Any available updates will be relayed upon event ticket closure. If additional issues are experienced, please contact the Level 3 Technical Service Center at 1-877-4LEVEL3. Rob++ -- pubkey: http://www.talgas.com/misc/rob_windsor-pubkey.txt Internet: windsor@warthog.com Life: Rob@Allen.Texas.USA.Earth "They couldn't hit an elephant at this distance." -- Major General John Sedgwick
participants (6)
-
Berry Mobley
-
Casey Crowe
-
Devon True
-
Dovid Bender
-
John Barbieri
-
Rob Windsor