There was a maintenance scheduled for a cut over that we had been notified of but I'm having trouble confirming if it's related. It seems like a maintenance or maintenance prep work gone bad. I got that email at 1:00am central. GCR # 11424826<tel:11424826> Summary Level 3 will implement scheduled maintenance as part of a large scale global project to migrate traffic from our Voice platform to our IP platform. This work is taking place to migrate traffic off equipment that has reached end of life support and move it to a more stable environment. This GCR is one of several maintenances taking place across various locations. We will be performing this migration to increase network capacity, reliability, and stability. During this time you may experience up to three minutes of fast busies. Updates 04:11 GMT<x-apple-data-detectors://1> - This maintenance is now in progress. Harris Kleyman Data Center Engineer ---- Atomic Data(r) // Safe. Simple. Smart.(tm) 250 Marquette Ave, Suite 225 Minneapolis, MN 55401 Main // 612.466.2000 Direct // 612.466.2031 It's official. We're SOC 3 Certified, again!<http://www.atomicdata.com/articles/atomic-data%C2%AE-keeps-client%E2%80%99s-data-protected-soc-3%C2%AE-certified-security-and-controls> On Oct 4, 2016, at 4:55 PM, Frank Bulk via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: This is what we received from our partner, "The Level 3 VOICE department has made a few configuration changes on core network equipment, to provide a work around, and restore customer services. Currently, Level 3 has a ticket open with one of their vendors and the work around will stay in place until a root cause analysis is complete." Frank -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Rob Hall via Outages Sent: Tuesday, October 04, 2016 3:35 PM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Level3 VoIP Issues? Call flow appears to be normal as of 4:00PM Eastern across Charter, Comcast and Cox. Thanks! -- Rob Hall, CCNP R&S, CCP-V I.T. Manager @Work Personnel Service 3815 Hwy 66 S, Suite 4 Rogersville, TN 37857 Office: (423)272-7897, x10113 Mobile: (423)312-6003 -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Rob Hall via Outages Sent: Tuesday, October 4, 2016 2:17 PM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Level3 VoIP Issues? Also confirmed with Comcast and Cox voice service in VA. Thanks! -- Rob Hall, CCNP R&S, CCP-V I.T. Manager @Work Personnel Service 3815 Hwy 66 S, Suite 4 Rogersville, TN 37857 Office: (423)272-7897, x10113 Mobile: (423)312-6003 -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Rob Hall via Outages Sent: Tuesday, October 4, 2016 2:08 PM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Level3 VoIP Issues? FWIW, we're still seeing an outage via Charter's voice service. Local calling seems to be unaffected, but all inbound/outbound long distance is out. We're seeing this behavior in every location in which we have Charter's service, ranging across East TN and into Northern GA. Thanks! -- Rob Hall, CCNP R&S, CCP-V I.T. Manager @Work Personnel Service 3815 Hwy 66 S, Suite 4 Rogersville, TN 37857 Office: (423)272-7897, x10113 Mobile: (423)312-6003 -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Mike Castoro via Outages Sent: Tuesday, October 04, 2016 1:36 PM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Level3 VoIP Issues? Unofficially, I'm hearing that the Level 3 voice outage was caused by a "fire at a large switching center caused by a major fiber cut" -Mike -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Michael Cargile via Outages Sent: Tuesday, October 04, 2016 12:09 PM To: Peter Beckman <beckman@angryox.com<mailto:beckman@angryox.com>> Cc: Aubrey Wells via Outages <outages@outages.org<mailto:outages@outages.org>> Subject: Re: [outages] Level3 VoIP Issues? Both Bandwidth and Vitelity are just VoIP resellers. They use larger carriers like Level3 for their termination and origination. As such Level3's outage is effecting their services as well. On 2016-10-04 11:57, Peter Beckman via Outages wrote: Hey Aubrey -- Did you see origination or termination issues with Bandwidth and Vitelity? Or both? Beckman On Tue, 4 Oct 2016, Aubrey Wells via Outages wrote: Atlanta Ga, our trunks terminate in McLean VA and NYC. Same problem. Seeing issues with other providers like Bandwidth.com<http://Bandwidth.com> and Vitelity as well. --------------------- Aubrey Wells Manager, Network Engineering Digium Cloud Services Main: 888.305.3850 Support: 877.344.4861 or http://www.digium.com/en/support <http://www.digium.com/en/support?elq=65516445a5964d3597e25eaf566bc2c f&elqCampaignId=> --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com<mailto:beckman@angryox.com> http://www.angryox.com/ ---------------------------------------------------------------------- ----- _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages -- Michael Cargile Director of Consulting Vicidial Group www.vicidial.com<http://www.vicidial.com> _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages -- BEGIN-ANTISPAM-VOTING-LINKS ------------------------------------------------------ Teach CanIt if this mail (ID 01RPs9KqN) is spam: Spam: https://antispam.roaringpenguin.com/canit/b.php?i=01RPs9KqN&m=fab7ef2658be&t =20161004&c=s Fraud/Phish: https://antispam.roaringpenguin.com/canit/b.php?i=01RPs9KqN&m=fab7ef2658be&t =20161004&c=p Not spam: https://antispam.roaringpenguin.com/canit/b.php?i=01RPs9KqN&m=fab7ef2658be&t =20161004&c=n Forget vote: https://antispam.roaringpenguin.com/canit/b.php?i=01RPs9KqN&m=fab7ef2658be&t =20161004&c=f ------------------------------------------------------ END-ANTISPAM-VOTING-LINKS _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages