Comcast outage in the midwest?
We just got an alert for hundreds of devices that are on comcast in the mid west. It seems the routes have been pulled.
Northern VA, multiple customers on Comcast reporting outages for EDI, SIP trunks, etc. KP On Fri, Jun 29, 2018 at 12:37 PM Dovid Bender via Outages < outages@outages.org> wrote:
We just got an alert for hundreds of devices that are on comcast in the mid west. It seems the routes have been pulled.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
What do you call Midwest? Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Fri, Jun 29, 2018, 12:41 PM Kenneth Padgett via Outages < outages@outages.org> wrote:
Northern VA, multiple customers on Comcast reporting outages for EDI, SIP trunks, etc.
KP
On Fri, Jun 29, 2018 at 12:37 PM Dovid Bender via Outages < outages@outages.org> wrote:
We just got an alert for hundreds of devices that are on comcast in the mid west. It seems the routes have been pulled.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Midwest is as far west as Colorado Springs, CO. We are seeing problems getting to various services hosted in AWS US-East-1 N.Virginia. On Fri, Jun 29, 2018 at 10:43 AM Josh Luthman via Outages < outages@outages.org> wrote:
What do you call Midwest?
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Fri, Jun 29, 2018, 12:41 PM Kenneth Padgett via Outages < outages@outages.org> wrote:
Northern VA, multiple customers on Comcast reporting outages for EDI, SIP trunks, etc.
KP
On Fri, Jun 29, 2018 at 12:37 PM Dovid Bender via Outages < outages@outages.org> wrote:
We just got an alert for hundreds of devices that are on comcast in the mid west. It seems the routes have been pulled.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Josiah Ritchie Church Community Builder (719) 266-2888
We're seeing odd off-net behavior, primarily with Chicago market connections jumping from comcast to Tata, haven't come across other peers that are with issue, yet. Brad Riemann Senior Systems and Network Engineer Brad Riemann Senior Systems and Network Engineer briemann@nmi.com | www.nmi.com On Fri, Jun 29, 2018 at 11:52 AM Josiah Ritchie via Outages <outages@outages.org> wrote:
Midwest is as far west as Colorado Springs, CO. We are seeing problems getting to various services hosted in AWS US-East-1 N.Virginia.
On Fri, Jun 29, 2018 at 10:43 AM Josh Luthman via Outages <outages@outages.org> wrote:
What do you call Midwest?
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Fri, Jun 29, 2018, 12:41 PM Kenneth Padgett via Outages <outages@outages.org> wrote:
Northern VA, multiple customers on Comcast reporting outages for EDI, SIP trunks, etc.
KP
On Fri, Jun 29, 2018 at 12:37 PM Dovid Bender via Outages <outages@outages.org> wrote:
We just got an alert for hundreds of devices that are on comcast in the mid west. It seems the routes have been pulled.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Josiah Ritchie Church Community Builder (719) 266-2888 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We are also seeing issues in Atlanta. Having trouble getting to 8.8.8.8 and 8.8.4.4 but it seems to be coming back up. Bill -----Original Message----- From: Outages <outages-bounces@outages.org> On Behalf Of Brad Riemann via Outages Sent: Friday, June 29, 2018 12:56 PM To: jritchie@churchcommunitybuilder.com Cc: outages@outages.org Subject: Re: [outages] Comcast outage in the midwest? We're seeing odd off-net behavior, primarily with Chicago market connections jumping from comcast to Tata, haven't come across other peers that are with issue, yet. Brad Riemann Senior Systems and Network Engineer Brad Riemann Senior Systems and Network Engineer briemann@nmi.com | www.nmi.com On Fri, Jun 29, 2018 at 11:52 AM Josiah Ritchie via Outages <outages@outages.org> wrote:
Midwest is as far west as Colorado Springs, CO. We are seeing problems getting to various services hosted in AWS US-East-1 N.Virginia.
On Fri, Jun 29, 2018 at 10:43 AM Josh Luthman via Outages <outages@outages.org> wrote:
What do you call Midwest?
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Fri, Jun 29, 2018, 12:41 PM Kenneth Padgett via Outages <outages@outages.org> wrote:
Northern VA, multiple customers on Comcast reporting outages for EDI, SIP trunks, etc.
KP
On Fri, Jun 29, 2018 at 12:37 PM Dovid Bender via Outages <outages@outages.org> wrote:
We just got an alert for hundreds of devices that are on comcast in the mid west. It seems the routes have been pulled.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Josiah Ritchie Church Community Builder (719) 266-2888 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
If you trust downdetector.com as a source, there are an awful lot of charts that have the same profile, this could be bigger than just Comcast. But yeah, we are also seeing it for customers with Comcast. On Fri, Jun 29, 2018 at 1:01 PM, Warwick, Bill via Outages < outages@outages.org> wrote:
We are also seeing issues in Atlanta. Having trouble getting to 8.8.8.8 and 8.8.4.4 but it seems to be coming back up.
Bill
-----Original Message----- From: Outages <outages-bounces@outages.org> On Behalf Of Brad Riemann via Outages Sent: Friday, June 29, 2018 12:56 PM To: jritchie@churchcommunitybuilder.com Cc: outages@outages.org Subject: Re: [outages] Comcast outage in the midwest?
We're seeing odd off-net behavior, primarily with Chicago market connections jumping from comcast to Tata, haven't come across other peers that are with issue, yet.
Brad Riemann Senior Systems and Network Engineer
Brad Riemann
Senior Systems and Network Engineer
briemann@nmi.com | www.nmi.com
On Fri, Jun 29, 2018 at 11:52 AM Josiah Ritchie via Outages < outages@outages.org> wrote:
Midwest is as far west as Colorado Springs, CO. We are seeing problems
getting to various services hosted in AWS US-East-1 N.Virginia.
On Fri, Jun 29, 2018 at 10:43 AM Josh Luthman via Outages <
outages@outages.org> wrote:
What do you call Midwest?
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Fri, Jun 29, 2018, 12:41 PM Kenneth Padgett via Outages <
outages@outages.org> wrote:
Northern VA, multiple customers on Comcast reporting outages for EDI,
SIP trunks, etc.
KP
On Fri, Jun 29, 2018 at 12:37 PM Dovid Bender via Outages <
outages@outages.org> wrote:
We just got an alert for hundreds of devices that are on comcast in
the mid west. It seems the routes have been pulled.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Josiah Ritchie Church Community Builder (719) 266-2888 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Our technician got off the line with them and said that their "internal systems" are down for the next hour at minimum. This is a nationwide outage. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Warwick, Bill via Outages Sent: Friday, June 29, 2018 1:01 PM To: Brad Riemann <briemann@nmi.com>; jritchie@churchcommunitybuilder.com Cc: outages@outages.org Subject: Re: [outages] Comcast outage in the midwest? We are also seeing issues in Atlanta. Having trouble getting to 8.8.8.8 and 8.8.4.4 but it seems to be coming back up. Bill -----Original Message----- From: Outages <outages-bounces@outages.org> On Behalf Of Brad Riemann via Outages Sent: Friday, June 29, 2018 12:56 PM To: jritchie@churchcommunitybuilder.com Cc: outages@outages.org Subject: Re: [outages] Comcast outage in the midwest? We're seeing odd off-net behavior, primarily with Chicago market connections jumping from comcast to Tata, haven't come across other peers that are with issue, yet. Brad Riemann Senior Systems and Network Engineer Brad Riemann Senior Systems and Network Engineer briemann@nmi.com | www.nmi.com On Fri, Jun 29, 2018 at 11:52 AM Josiah Ritchie via Outages <outages@outages.org> wrote:
Midwest is as far west as Colorado Springs, CO. We are seeing problems getting to various services hosted in AWS US-East-1 N.Virginia.
On Fri, Jun 29, 2018 at 10:43 AM Josh Luthman via Outages <outages@outages.org> wrote:
What do you call Midwest?
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Fri, Jun 29, 2018, 12:41 PM Kenneth Padgett via Outages <outages@outages.org> wrote:
Northern VA, multiple customers on Comcast reporting outages for EDI, SIP trunks, etc.
KP
On Fri, Jun 29, 2018 at 12:37 PM Dovid Bender via Outages <outages@outages.org> wrote:
We just got an alert for hundreds of devices that are on comcast in the mid west. It seems the routes have been pulled.
_______________________________________________ Outages mailing list Outages@outages.org https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman...
_______________________________________________ Outages mailing list Outages@outages.org https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman...
_______________________________________________ Outages mailing list Outages@outages.org https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman...
-- Josiah Ritchie Church Community Builder (719) 266-2888 _______________________________________________ Outages mailing list Outages@outages.org https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman...
_______________________________________________ Outages mailing list Outages@outages.org https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman... _______________________________________________ Outages mailing list Outages@outages.org https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman...
Our Roseville MN location is down hard, well from here in MD. Also my fiber EDI to Comcast here in MD is running horrible, with spotty connectivity at best. So we are feeling the effects in various locations. Like others I tried calling Comcast, they call center is also down.. --- Howard Leadmon - howard@leadmon.net PBW Communications, LLC http://www.pbwcomm.com
Areas throughout central Pennsylvania are reporting some sites loading, and others not. Seems to be a routing issue as others have already suggested. [image: photo] Zach Bare Network Engineer at KINBER <https://www.facebook.com/Keystone-Initiative-for-Network-Based-Education-and-Research-188743104566075/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> <http://www.twitter.com/kinber?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> <https://www.linkedin.com/company/kinber-keystone-initiative-for-network-based-education-and-research-?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Address 5775 Allentown Blvd., Suite 101, Harrisburg, PA 17112 Phone 717-963-7490 | PennREN NOC 833-736-6736 (833-PENNREN) <717-963-7490+%7C+PennREN+NOC+833-736-6736+(833-PENNREN)?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Mobile 717-469-5461 <717-469-5461?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Email zbare@kinber.org On Fri, Jun 29, 2018 at 1:45 PM, Howard Leadmon via Outages < outages@outages.org> wrote:
Our Roseville MN location is down hard, well from here in MD. Also my fiber EDI to Comcast here in MD is running horrible, with spotty connectivity at best. So we are feeling the effects in various locations. Like others I tried calling Comcast, they call center is also down..
--- Howard Leadmon - howard@leadmon.net PBW Communications, LLC http://www.pbwcomm.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We're seeing outages in the Atlanta market on both EDI and coax service. It's not hard-down but appears routing related from what we can tell. I can't get through to Enterprise Support at all either, understandably. I just get the message "We are experiencing technical difficulties" and the line goes silent. Joe Jenkins *Director of Information Technology* Andretti Indoor Karting & Games Email jjenkins@funparksgroup.com | jjenkins@andrettikarting.com On Fri, Jun 29, 2018 at 1:57 PM Zach Bare via Outages <outages@outages.org> wrote:
Areas throughout central Pennsylvania are reporting some sites loading, and others not. Seems to be a routing issue as others have already suggested.
[image: photo] Zach Bare Network Engineer at KINBER
<https://www.facebook.com/Keystone-Initiative-for-Network-Based-Education-and-Research-188743104566075/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> <http://www.twitter.com/kinber?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> <https://www.linkedin.com/company/kinber-keystone-initiative-for-network-based-education-and-research-?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Address 5775 Allentown Blvd., Suite 101, Harrisburg, PA 17112 Phone 717-963-7490 | PennREN NOC 833-736-6736 (833-PENNREN) <717-963-7490+%7C+PennREN+NOC+833-736-6736+(833-PENNREN)?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Mobile 717-469-5461 <717-469-5461?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Email zbare@kinber.org
On Fri, Jun 29, 2018 at 1:45 PM, Howard Leadmon via Outages < outages@outages.org> wrote:
Our Roseville MN location is down hard, well from here in MD. Also my fiber EDI to Comcast here in MD is running horrible, with spotty connectivity at best. So we are feeling the effects in various locations. Like others I tried calling Comcast, they call center is also down..
--- Howard Leadmon - howard@leadmon.net PBW Communications, LLC http://www.pbwcomm.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We're back up in CA as of about 90 seconds ago. On Fri, Jun 29, 2018 at 10:55 AM, Zach Bare via Outages <outages@outages.org
wrote:
Areas throughout central Pennsylvania are reporting some sites loading, and others not. Seems to be a routing issue as others have already suggested.
[image: photo] Zach Bare Network Engineer at KINBER
<https://www.facebook.com/Keystone-Initiative-for-Network-Based-Education-and-Research-188743104566075/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> <http://www.twitter.com/kinber?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> <https://www.linkedin.com/company/kinber-keystone-initiative-for-network-based-education-and-research-?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Address 5775 Allentown Blvd., Suite 101, Harrisburg, PA 17112 Phone 717-963-7490 | PennREN NOC 833-736-6736 (833-PENNREN) <717-963-7490+%7C+PennREN+NOC+833-736-6736+(833-PENNREN)?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Mobile 717-469-5461 <717-469-5461?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Email zbare@kinber.org
On Fri, Jun 29, 2018 at 1:45 PM, Howard Leadmon via Outages < outages@outages.org> wrote:
Our Roseville MN location is down hard, well from here in MD. Also my fiber EDI to Comcast here in MD is running horrible, with spotty connectivity at best. So we are feeling the effects in various locations. Like others I tried calling Comcast, they call center is also down..
--- Howard Leadmon - howard@leadmon.net PBW Communications, LLC http://www.pbwcomm.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We are getting indications of recovery in the Chicago area. R. Sent from my iPhone, sorry for any typos, grammar errors, cryptic abbreviations, etc.
On Jun 29, 2018, at 1:04 PM, Terry Hardie via Outages <outages@outages.org> wrote:
We're back up in CA as of about 90 seconds ago.
On Fri, Jun 29, 2018 at 10:55 AM, Zach Bare via Outages <outages@outages.org> wrote: Areas throughout central Pennsylvania are reporting some sites loading, and others not. Seems to be a routing issue as others have already suggested.
Zach Bare Network Engineer at KINBER
Address 5775 Allentown Blvd., Suite 101, Harrisburg, PA 17112 Phone 717-963-7490 | PennREN NOC 833-736-6736 (833-PENNREN)
Mobile 717-469-5461
Email zbare@kinber.org
On Fri, Jun 29, 2018 at 1:45 PM, Howard Leadmon via Outages <outages@outages.org> wrote: Our Roseville MN location is down hard, well from here in MD. Also my fiber EDI to Comcast here in MD is running horrible, with spotty connectivity at best. So we are feeling the effects in various locations. Like others I tried calling Comcast, they call center is also down..
--- Howard Leadmon - howard@leadmon.net PBW Communications, LLC http://www.pbwcomm.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We are seeing a routing change. We are in NY and both NTT and HE were sending the traffic to their NY routers. The traffic is now going: HE is sending the traffic to ATL and peering with Comcast there. From there Comcast is sending it to Chicago. NTT is sending it on their own network to Chicago and handing it off to Comcast there. On Fri, Jun 29, 2018 at 2:10 PM, Rob Repp via Outages <outages@outages.org> wrote:
We are getting indications of recovery in the Chicago area. R.
Sent from my iPhone, sorry for any typos, grammar errors, cryptic abbreviations, etc.
On Jun 29, 2018, at 1:04 PM, Terry Hardie via Outages <outages@outages.org> wrote:
We're back up in CA as of about 90 seconds ago.
On Fri, Jun 29, 2018 at 10:55 AM, Zach Bare via Outages < outages@outages.org> wrote:
Areas throughout central Pennsylvania are reporting some sites loading, and others not. Seems to be a routing issue as others have already suggested.
[image: photo] Zach Bare Network Engineer at KINBER
<https://www.facebook.com/Keystone-Initiative-for-Network-Based-Education-and-Research-188743104566075/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> <http://www.twitter.com/kinber?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> <https://www.linkedin.com/company/kinber-keystone-initiative-for-network-based-education-and-research-?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Address 5775 Allentown Blvd., Suite 101, Harrisburg, PA 17112 Phone 717-963-7490 | PennREN NOC 833-736-6736 (833-PENNREN) <717-963-7490+%7C+PennREN+NOC+833-736-6736+(833-PENNREN)?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Mobile 717-469-5461 <717-469-5461?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Email zbare@kinber.org
On Fri, Jun 29, 2018 at 1:45 PM, Howard Leadmon via Outages < outages@outages.org> wrote:
Our Roseville MN location is down hard, well from here in MD. Also my fiber EDI to Comcast here in MD is running horrible, with spotty connectivity at best. So we are feeling the effects in various locations. Like others I tried calling Comcast, they call center is also down..
--- Howard Leadmon - howard@leadmon.net PBW Communications, LLC http://www.pbwcomm.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
BGP session to Comcast has dropped for us on the East Coast. We were unaffected until that happened. I suspect other East Coast providers have seen similar, which is the cause of the routing change. From: Outages <outages-bounces@outages.org> On Behalf Of Dovid Bender via Outages Sent: 29 June 2018 19:16 To: Rob Repp <robrepp@winnetka36.org> Cc: outages@outages.org Subject: Re: [outages] Comcast outage in the midwest? We are seeing a routing change. We are in NY and both NTT and HE were sending the traffic to their NY routers. The traffic is now going: HE is sending the traffic to ATL and peering with Comcast there. From there Comcast is sending it to Chicago. NTT is sending it on their own network to Chicago and handing it off to Comcast there. On Fri, Jun 29, 2018 at 2:10 PM, Rob Repp via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: We are getting indications of recovery in the Chicago area. R. Sent from my iPhone, sorry for any typos, grammar errors, cryptic abbreviations, etc. On Jun 29, 2018, at 1:04 PM, Terry Hardie via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: We're back up in CA as of about 90 seconds ago. On Fri, Jun 29, 2018 at 10:55 AM, Zach Bare via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Areas throughout central Pennsylvania are reporting some sites loading, and others not. Seems to be a routing issue as others have already suggested. [photo] Zach Bare Network Engineer at KINBER [https://s3.amazonaws.com/images.wisestamp.com/social_icons/square/facebook.png]<https://www.facebook.com/Keystone-Initiative-for-Network-Based-Education-and-Research-188743104566075/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> [https://s3.amazonaws.com/images.wisestamp.com/social_icons/square/twitter.png]<http://www.twitter.com/kinber?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> [https://s3.amazonaws.com/images.wisestamp.com/social_icons/square/linkedin.png]<https://www.linkedin.com/company/kinber-keystone-initiative-for-network-based-education-and-research-?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Address 5775 Allentown Blvd., Suite 101, Harrisburg, PA 17112 Phone 717-963-7490 | PennREN NOC 833-736-6736 (833-PENNREN) <tel:717-963-7490+%7C+PennREN+NOC+833-736-6736+(833-PENNREN)?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Mobile 717-469-5461 <tel:717-469-5461?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Email zbare@kinber.org <mailto:zbare@kinber.org> On Fri, Jun 29, 2018 at 1:45 PM, Howard Leadmon via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Our Roseville MN location is down hard, well from here in MD. Also my fiber EDI to Comcast here in MD is running horrible, with spotty connectivity at best. So we are feeling the effects in various locations. Like others I tried calling Comcast, they call center is also down.. --- Howard Leadmon - howard@leadmon.net<mailto:howard@leadmon.net> PBW Communications, LLC http://www.pbwcomm.com _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Comcast is now saying this is due to a cut at another provider they believe is affecting other isps as well. Any indication that this is true? I haven't seen any other outages today. On Fri, Jun 29, 2018, 2:20 PM Phil Lavin via Outages <outages@outages.org> wrote:
BGP session to Comcast has dropped for us on the East Coast. We were unaffected until that happened. I suspect other East Coast providers have seen similar, which is the cause of the routing change.
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Dovid Bender via Outages *Sent:* 29 June 2018 19:16 *To:* Rob Repp <robrepp@winnetka36.org> *Cc:* outages@outages.org *Subject:* Re: [outages] Comcast outage in the midwest?
We are seeing a routing change. We are in NY and both NTT and HE were sending the traffic to their NY routers. The traffic is now going:
HE is sending the traffic to ATL and peering with Comcast there. From there Comcast is sending it to Chicago.
NTT is sending it on their own network to Chicago and handing it off to Comcast there.
On Fri, Jun 29, 2018 at 2:10 PM, Rob Repp via Outages <outages@outages.org> wrote:
We are getting indications of recovery in the Chicago area.
R.
Sent from my iPhone, sorry for any typos, grammar errors, cryptic abbreviations, etc.
On Jun 29, 2018, at 1:04 PM, Terry Hardie via Outages <outages@outages.org> wrote:
We're back up in CA as of about 90 seconds ago.
On Fri, Jun 29, 2018 at 10:55 AM, Zach Bare via Outages < outages@outages.org> wrote:
Areas throughout central Pennsylvania are reporting some sites loading, and others not. Seems to be a routing issue as others have already suggested.
[image: photo]
*Zach Bare* Network Engineer at KINBER
*Address *5775 Allentown Blvd., Suite 101, Harrisburg, PA 17112 *Phone * 717-963-7490 | PennREN NOC 833-736-6736 (833-PENNREN) <717-963-7490+%7C+PennREN+NOC+833-736-6736+(833-PENNREN)?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> *Mobile * 717-469-5461 <717-469-5461?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> *Email * zbare@kinber.org
On Fri, Jun 29, 2018 at 1:45 PM, Howard Leadmon via Outages < outages@outages.org> wrote:
Our Roseville MN location is down hard, well from here in MD. Also my fiber EDI to Comcast here in MD is running horrible, with spotty connectivity at best. So we are feeling the effects in various locations. Like others I tried calling Comcast, they call center is also down..
--- Howard Leadmon - howard@leadmon.net PBW Communications, LLC http://www.pbwcomm.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
There is a Reddit /r/sysadmin contributor indicating that he is participating in inter-ISP discussion and that there are indeed major fiber issues across multiple providers: https://old.reddit.com/r/sysadmin/comments/8uukck/comcast_level_3_issues_eas... That may be a better forum for further minor updates rather than ringing the Outages list On Fri, Jun 29, 2018 at 3:10 PM, Sean Hayes via Outages <outages@outages.org
wrote:
Comcast is now saying this is due to a cut at another provider they believe is affecting other isps as well. Any indication that this is true? I haven't seen any other outages today.
On Fri, Jun 29, 2018, 2:20 PM Phil Lavin via Outages <outages@outages.org> wrote:
BGP session to Comcast has dropped for us on the East Coast. We were unaffected until that happened. I suspect other East Coast providers have seen similar, which is the cause of the routing change.
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Dovid Bender via Outages *Sent:* 29 June 2018 19:16 *To:* Rob Repp <robrepp@winnetka36.org> *Cc:* outages@outages.org *Subject:* Re: [outages] Comcast outage in the midwest?
We are seeing a routing change. We are in NY and both NTT and HE were sending the traffic to their NY routers. The traffic is now going:
HE is sending the traffic to ATL and peering with Comcast there. From there Comcast is sending it to Chicago.
NTT is sending it on their own network to Chicago and handing it off to Comcast there.
On Fri, Jun 29, 2018 at 2:10 PM, Rob Repp via Outages < outages@outages.org> wrote:
We are getting indications of recovery in the Chicago area.
R.
Sent from my iPhone, sorry for any typos, grammar errors, cryptic abbreviations, etc.
On Jun 29, 2018, at 1:04 PM, Terry Hardie via Outages < outages@outages.org> wrote:
We're back up in CA as of about 90 seconds ago.
On Fri, Jun 29, 2018 at 10:55 AM, Zach Bare via Outages < outages@outages.org> wrote:
Areas throughout central Pennsylvania are reporting some sites loading, and others not. Seems to be a routing issue as others have already suggested.
[image: photo]
*Zach Bare* Network Engineer at KINBER
*Address *5775 Allentown Blvd., Suite 101, Harrisburg, PA 17112 *Phone * 717-963-7490 | PennREN NOC 833-736-6736 (833-PENNREN) <717-963-7490+%7C+PennREN+NOC+833-736-6736+(833-PENNREN)?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> *Mobile * 717-469-5461 <717-469-5461?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> *Email * zbare@kinber.org
On Fri, Jun 29, 2018 at 1:45 PM, Howard Leadmon via Outages < outages@outages.org> wrote:
Our Roseville MN location is down hard, well from here in MD. Also my fiber EDI to Comcast here in MD is running horrible, with spotty connectivity at best. So we are feeling the effects in various locations. Like others I tried calling Comcast, they call center is also down..
--- Howard Leadmon - howard@leadmon.net PBW Communications, LLC http://www.pbwcomm.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Sounds like a load they are feeding to customers. We are an ISP peering with several other upstreams and I see no issues with any of them.
On Jun 29, 2018, at 15:10, Sean Hayes via Outages <outages@outages.org> wrote:
Comcast is now saying this is due to a cut at another provider they believe is affecting other isps as well. Any indication that this is true? I haven't seen any other outages today.
On Fri, Jun 29, 2018, 2:20 PM Phil Lavin via Outages <outages@outages.org> wrote: BGP session to Comcast has dropped for us on the East Coast. We were unaffected until that happened. I suspect other East Coast providers have seen similar, which is the cause of the routing change.
From: Outages <outages-bounces@outages.org> On Behalf Of Dovid Bender via Outages Sent: 29 June 2018 19:16 To: Rob Repp <robrepp@winnetka36.org> Cc: outages@outages.org Subject: Re: [outages] Comcast outage in the midwest?
We are seeing a routing change. We are in NY and both NTT and HE were sending the traffic to their NY routers. The traffic is now going:
HE is sending the traffic to ATL and peering with Comcast there. From there Comcast is sending it to Chicago.
NTT is sending it on their own network to Chicago and handing it off to Comcast there.
On Fri, Jun 29, 2018 at 2:10 PM, Rob Repp via Outages <outages@outages.org> wrote:
We are getting indications of recovery in the Chicago area.
R.
Sent from my iPhone, sorry for any typos, grammar errors, cryptic abbreviations, etc.
On Jun 29, 2018, at 1:04 PM, Terry Hardie via Outages <outages@outages.org> wrote:
We're back up in CA as of about 90 seconds ago.
On Fri, Jun 29, 2018 at 10:55 AM, Zach Bare via Outages <outages@outages.org> wrote:
Areas throughout central Pennsylvania are reporting some sites loading, and others not. Seems to be a routing issue as others have already suggested.
Zach Bare Network Engineer at KINBER
Address 5775 Allentown Blvd., Suite 101, Harrisburg, PA 17112 Phone 717-963-7490 | PennREN NOC 833-736-6736 (833-PENNREN) Mobile 717-469-5461 Email zbare@kinber.org
On Fri, Jun 29, 2018 at 1:45 PM, Howard Leadmon via Outages <outages@outages.org> wrote:
Our Roseville MN location is down hard, well from here in MD. Also my fiber EDI to Comcast here in MD is running horrible, with spotty connectivity at best. So we are feeling the effects in various locations. Like others I tried calling Comcast, they call center is also down..
--- Howard Leadmon - howard@leadmon.net PBW Communications, LLC http://www.pbwcomm.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Jun 29, 2018, at 12:10 PM, Sean Hayes via Outages <outages@outages.org> wrote:
Comcast is now saying this is due to a cut at another provider they believe is affecting other isps as well. Any indication that this is true? I haven't seen any other outages today.
There's a Level3 fiber cut in North Carolina that affected one of our waves: "*** CASCADED EXTERNAL NOTES 29-Jun-2018 15:13:37 GMT From CASE: 14815828 - Event Field Operations dispatched to the estimated failure location and upon arrival was advised by local law enforcement there was a truck that struck a utility pole, which then fell across the street and struck another pole, which then tore down the aerial fiber lines. In addition to the fiber lines being down, live power lines are down as well. OSP will be dispatching a fiber repair and construction crew to the site; however, until the local utility company has completed their repairs and deemed the area safe, fiber repairs cannot begin. *** CASCADED EXTERNAL NOTES 29-Jun-2018 19:21:05 GMT From CASE: 14815828 - Event The OSP has contacted the personnel transporting the replacement fiber, and they advised they remain en route, and on schedule for their ETA of 20:15 GMT. The next update will be provided upon their arrival, or as new information becomes available. Steve
We're seeing Comcast come back up in Nashville, and now seeing Chicago drop. On Fri, Jun 29, 2018 at 2:05 PM Terry Hardie via Outages < outages@outages.org> wrote:
We're back up in CA as of about 90 seconds ago.
On Fri, Jun 29, 2018 at 10:55 AM, Zach Bare via Outages < outages@outages.org> wrote:
Areas throughout central Pennsylvania are reporting some sites loading, and others not. Seems to be a routing issue as others have already suggested.
[image: photo] Zach Bare Network Engineer at KINBER
<https://www.facebook.com/Keystone-Initiative-for-Network-Based-Education-and-Research-188743104566075/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> <http://www.twitter.com/kinber?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> <https://www.linkedin.com/company/kinber-keystone-initiative-for-network-based-education-and-research-?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Address 5775 Allentown Blvd., Suite 101, Harrisburg, PA 17112 Phone 717-963-7490 | PennREN NOC 833-736-6736 (833-PENNREN) <717-963-7490+%7C+PennREN+NOC+833-736-6736+(833-PENNREN)?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Mobile 717-469-5461 <717-469-5461?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> Email zbare@kinber.org
On Fri, Jun 29, 2018 at 1:45 PM, Howard Leadmon via Outages < outages@outages.org> wrote:
Our Roseville MN location is down hard, well from here in MD. Also my fiber EDI to Comcast here in MD is running horrible, with spotty connectivity at best. So we are feeling the effects in various locations. Like others I tried calling Comcast, they call center is also down..
--- Howard Leadmon - howard@leadmon.net PBW Communications, LLC http://www.pbwcomm.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Our Comcast service is actually still up, which is odd because it goes down all the time, (actually went down yesterday) but we’re having issues with a cloud based app, calling into support for it, they are telling us it is down for everyone across the country, I am wondering if this Comcast outage has anything to do with it. Chris Cherry *The Abstract & Title Guaranty Co., Inc.* *71 West Marion Street <x-apple-data-detectors://0/1>* *Danville IN, 46122 <x-apple-data-detectors://0/1>* *P: (317) 745-4300 <(317)%20745-4300>* *F: (317) 745-3038 <(317)%20745-3038>* ccherry@abstract-title.com Sent from my iPad*, **Please excuse any abbreviations and auto correct errors.* On Jun 29, 2018, at 1:01 PM, Warwick, Bill via Outages <outages@outages.org> wrote: We are also seeing issues in Atlanta. Having trouble getting to 8.8.8.8 and 8.8.4.4 but it seems to be coming back up. Bill -----Original Message----- From: Outages <outages-bounces@outages.org> On Behalf Of Brad Riemann via Outages Sent: Friday, June 29, 2018 12:56 PM To: jritchie@churchcommunitybuilder.com Cc: outages@outages.org Subject: Re: [outages] Comcast outage in the midwest? We're seeing odd off-net behavior, primarily with Chicago market connections jumping from comcast to Tata, haven't come across other peers that are with issue, yet. Brad Riemann Senior Systems and Network Engineer Brad Riemann Senior Systems and Network Engineer briemann@nmi.com | www.nmi.com On Fri, Jun 29, 2018 at 11:52 AM Josiah Ritchie via Outages < outages@outages.org> wrote: Midwest is as far west as Colorado Springs, CO. We are seeing problems getting to various services hosted in AWS US-East-1 N.Virginia. On Fri, Jun 29, 2018 at 10:43 AM Josh Luthman via Outages < outages@outages.org> wrote: What do you call Midwest? Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Fri, Jun 29, 2018, 12:41 PM Kenneth Padgett via Outages < outages@outages.org> wrote: Northern VA, multiple customers on Comcast reporting outages for EDI, SIP trunks, etc. KP On Fri, Jun 29, 2018 at 12:37 PM Dovid Bender via Outages < outages@outages.org> wrote: We just got an alert for hundreds of devices that are on comcast in the mid west. It seems the routes have been pulled. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages -- Josiah Ritchie Church Community Builder (719) 266-2888 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages -- **WARNING – FRAUDULENT FUNDING INSTRUCTIONS** Email hacking and fraud are on the rise to fraudulently misdirect funds. Please call your escrow officer immediately using contact information found from an independent source, such as the sales contract or internet, to verify any funding instructions received. We are not responsible for any wires sent by you to an incorrect bank account. *CONFIDENTIALITY NOTICE:* This e-mail transmission and any materials attached hereto are private and confidential and are the property of the sender. The information contained herein is privileged and is intended only for the use of the individual(s) or entity(ies) named above. If you are not the intended recipient, be advised that any unauthorized disclosure, copying, distribution, use or the taking of any action in reliance on this transmission is strictly prohibited. If you have received this transmission in error, please notify the sender immediately to arrange for its return.
I am wrong about the routes being pulled but connectivity is still an issue. On Fri, Jun 29, 2018 at 12:36 PM, Dovid Bender <dovid@telecurve.com> wrote:
We just got an alert for hundreds of devices that are on comcast in the mid west. It seems the routes have been pulled.
Seeing same thing on the east coast From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Dovid Bender via Outages Sent: Friday, June 29, 2018 9:39 AM To: outages@outages.org Subject: Re: [outages] Comcast outage in the midwest? I am wrong about the routes being pulled but connectivity is still an issue. On Fri, Jun 29, 2018 at 12:36 PM, Dovid Bender <dovid@telecurve.com<mailto:dovid@telecurve.com>> wrote: We just got an alert for hundreds of devices that are on comcast in the mid west. It seems the routes have been pulled.
Seeing the same for sites hosted at Peak 10 in Raleigh NC On Fri, Jun 29, 2018 at 11:44 AM Jordan Morris via Outages < outages@outages.org> wrote:
Seeing same thing on the east coast
*From:* Outages [mailto:outages-bounces@outages.org] * On Behalf Of *Dovid Bender via Outages *Sent:* Friday, June 29, 2018 9:39 AM *To:* outages@outages.org
*Subject:* Re: [outages] Comcast outage in the midwest?
I am wrong about the routes being pulled but connectivity is still an issue.
On Fri, Jun 29, 2018 at 12:36 PM, Dovid Bender <dovid@telecurve.com> wrote:
We just got an alert for hundreds of devices that are on comcast in the mid west. It seems the routes have been pulled.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Jeremie Chism Triton Communications
we just lost a bunch of sites on east coast as well On Fri, Jun 29, 2018 at 12:37 PM Dovid Bender via Outages < outages@outages.org> wrote:
We just got an alert for hundreds of devices that are on comcast in the mid west. It seems the routes have been pulled.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Anyone from comcast on the list? We see national impact Sean Salvadalena From: Outages <outages-bounces@outages.org> on behalf of "Alex S. via Outages" <outages@outages.org> Reply-To: "Alex S." <4500811@gmail.com> Date: Friday, June 29, 2018 at 11:42 To: "dovid@telecurve.com" <dovid@telecurve.com> Cc: "outages@outages.org" <outages@outages.org> Subject: Re: [outages] Comcast outage in the midwest? we just lost a bunch of sites on east coast as well On Fri, Jun 29, 2018 at 12:37 PM Dovid Bender via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: We just got an alert for hundreds of devices that are on comcast in the mid west. It seems the routes have been pulled. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages<https://url.emailprotection.link/?aUewINOCRlYV9Def85FT01vqq1lmstKVDFmqwCsWNREgtbkB6lmTHqE8S7_EjWsdZHZ3KZ9ZMIxfrhSP_Gc2ZUQ~~>
It does appear to be route related -- my home internet feed in FL (through comcast) has lost the ability to route to GTT's network for example. Some other destinations connect with no service degradation. Jim Hudson InfoRailway LLC Engineering | Web | Development | Telecom jim@inforailway.com | +1 (407) 613-2299 PO Box 2058, Windermere, FL 34786
On Jun 29, 2018, at 12:36 PM, Dovid Bender via Outages <outages@outages.org> wrote:
We just got an alert for hundreds of devices that are on comcast in the mid west. It seems the routes have been pulled.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
yes it does seem to be route specific cant route to certain EDI sites in VA and IL also seeing routing issues with certain ENS sites on east coast On Fri, Jun 29, 2018 at 12:44 PM Jim Hudson via Outages <outages@outages.org> wrote:
It does appear to be route related -- my home internet feed in FL (through comcast) has lost the ability to route to GTT's network for example. Some other destinations connect with no service degradation.
*Jim HudsonInfoRailway LLC * Engineering | Web | Development | Telecom jim@inforailway.com | +1 (407) 613-2299 PO Box 2058, Windermere, FL 34786
On Jun 29, 2018, at 12:36 PM, Dovid Bender via Outages < outages@outages.org> wrote:
We just got an alert for hundreds of devices that are on comcast in the mid west. It seems the routes have been pulled.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (25)
-
Alex S. -
Biddle, Josh -
Brad Riemann -
Chris Cherry -
Dovid Bender -
Howard Leadmon -
Jeremie Chism -
Jim Hudson -
Joe Jenkins -
Jordan Morris -
Josh Luthman -
Josiah Ritchie -
Kenneth Padgett -
Matt Hoppes -
Mike -
Pete Eisengrein -
Phil Lavin -
Rob Repp -
Ryan Easton -
Sean Hayes -
Sean Salvadalena -
Steve Feldman -
Terry Hardie -
Warwick, Bill -
Zach Bare