Re: [outages] Cogeco - Toronto outage?
Is the NNI with them for LAN type services or is it TPIA? We have both and are not experiencing any issues so sounds like possibly the demarcation switch you are connected to possibly …. Paul From: Outages <outages-bounces@outages.org> on behalf of Nick Pron via Outages <outages@outages.org> Reply-To: Nick Pron <Nick.Pron@xbase.com> Date: Wednesday, May 16, 2018 at 8:54 AM To: "outages@outages.org List" <outages@outages.org> Subject: [outages] Cogeco - Toronto outage? Anyone have info about this? We’ve been having issues with our NNI since Monday with our customers (down, packet loss, etc) Cogeco’s master tickets 718506 and 719701 claim they’ve swapped line cards in a core switch, upgraded IOS and still working with Cisco. Our rep seems to hint at that there’s more at play here than the tickets are saying.. Last update from them was 7:35 AM EDT Unfortunately, we have seen further instability on the affected network device and are continuing to work with our Cisco partner to isolate the issue and implement a restoration plan. We are still unable to provide an estimated time to restore but we will provide further updates as soon as more information becomes available. Once again we apologize for any disruption to your services that this may be causing. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
LAN type. That’s what it seems like. We did notice that a customer we have on ICA we can’t reach them as one of the hops goes through Cogeco From: Paul Stewart [mailto:paul@paulstewart.org] Sent: Wednesday, May 16, 2018 8:57 AM To: Nick Pron <Nick.Pron@xbase.com>; outages@outages.org List <outages@outages.org> Subject: Re: [outages] Cogeco - Toronto outage? Is the NNI with them for LAN type services or is it TPIA? We have both and are not experiencing any issues so sounds like possibly the demarcation switch you are connected to possibly …. Paul From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> on behalf of Nick Pron via Outages <outages@outages.org<mailto:outages@outages.org>> Reply-To: Nick Pron <Nick.Pron@xbase.com<mailto:Nick.Pron@xbase.com>> Date: Wednesday, May 16, 2018 at 8:54 AM To: "outages@outages.org List<mailto:outages@outages.org%20List>" <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] Cogeco - Toronto outage? Anyone have info about this? We’ve been having issues with our NNI since Monday with our customers (down, packet loss, etc) Cogeco’s master tickets 718506 and 719701 claim they’ve swapped line cards in a core switch, upgraded IOS and still working with Cisco. Our rep seems to hint at that there’s more at play here than the tickets are saying.. Last update from them was 7:35 AM EDT Unfortunately, we have seen further instability on the affected network device and are continuing to work with our Cisco partner to isolate the issue and implement a restoration plan. We are still unable to provide an estimated time to restore but we will provide further updates as soon as more information becomes available. Once again we apologize for any disruption to your services that this may be causing. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
We are also seeing some outages around the Toronto area. One of our NNI is also flapping from time to time. The weird part is that I'm seeing two way traffic, but way under the usual levels on the circuits. Anyone has any concrete info from Cogeco? On Wed, May 16, 2018 at 9:20 AM, Nick Pron via Outages <outages@outages.org> wrote:
LAN type. That’s what it seems like.
We did notice that a customer we have on ICA we can’t reach them as one of the hops goes through Cogeco
*From:* Paul Stewart [mailto:paul@paulstewart.org] *Sent:* Wednesday, May 16, 2018 8:57 AM *To:* Nick Pron <Nick.Pron@xbase.com>; outages@outages.org List < outages@outages.org> *Subject:* Re: [outages] Cogeco - Toronto outage?
Is the NNI with them for LAN type services or is it TPIA? We have both and are not experiencing any issues so sounds like possibly the demarcation switch you are connected to possibly ….
Paul
*From: *Outages <outages-bounces@outages.org> on behalf of Nick Pron via Outages <outages@outages.org> *Reply-To: *Nick Pron <Nick.Pron@xbase.com> *Date: *Wednesday, May 16, 2018 at 8:54 AM *To: *"outages@outages.org List" <outages@outages.org> *Subject: *[outages] Cogeco - Toronto outage?
Anyone have info about this? We’ve been having issues with our NNI since Monday with our customers (down, packet loss, etc)
Cogeco’s master tickets 718506 and 719701 claim they’ve swapped line cards in a core switch, upgraded IOS and still working with Cisco.
Our rep seems to hint at that there’s more at play here than the tickets are saying..
Last update from them was 7:35 AM EDT
Unfortunately, we have seen further instability on the affected network device and are continuing to work with our Cisco partner to isolate the issue and implement a restoration plan.
We are still unable to provide an estimated time to restore but we will provide further updates as soon as more information becomes available.
Once again we apologize for any disruption to your services that this may be causing.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- *Laurent Dumont* *Spécialiste réseau / Network specialist* *Fibrenoire* - www.fibrenoire.ca A: 550 , avenue Beaumont, bureau 320, Montréal (Québec) H3N 1V1 T. 514 907-3002 x132 C.438-825-4642 ldumont@fibrenoire.ca <farchambault@fibrenoire.ca> Twitter: Twitter Fibrenoire <https://twitter.com/fibrenoire> *--Note de confidentialité--* *Ce message électronique et toutes pièces jointes contiennent des informations privilégiées et confidentielles. Si vous n'êtes pas le destinataire prévu, il est strictement interdit d'utiliser, de copier ou divulguer les informations ainsi transmises; merci d'en aviser l'expéditeur et de les supprimer de votre ordinateur.* *--Confidentiality Note--* *This email message and any attachments contain privileged and confidential information. If you are not the intended recipient, your use, copying or disclosure of this information is prohibited; please contact the sender and delete the material from your computer.*
Appears they're having some bigger issues, I couldn't reach www.esasafe.com from a Cogeco business fiber line, but could from a connection that is based off cogeco's cable network(start.ca). MTR got completely lost on the fiber link(stops two hops past burl-ts1.cgocable.net, with heavy(8-10%) packet loss from that point on), but made it to Rogers (the hoster of esasafe.com) from the home connection just fine. Just retested after typing the above up and it appears to work again, so they're having some issues. On Wed, May 16, 2018 at 9:20 AM, Nick Pron via Outages <outages@outages.org> wrote:
LAN type. That’s what it seems like.
We did notice that a customer we have on ICA we can’t reach them as one of the hops goes through Cogeco
*From:* Paul Stewart [mailto:paul@paulstewart.org] *Sent:* Wednesday, May 16, 2018 8:57 AM *To:* Nick Pron <Nick.Pron@xbase.com>; outages@outages.org List < outages@outages.org> *Subject:* Re: [outages] Cogeco - Toronto outage?
Is the NNI with them for LAN type services or is it TPIA? We have both and are not experiencing any issues so sounds like possibly the demarcation switch you are connected to possibly ….
Paul
*From: *Outages <outages-bounces@outages.org> on behalf of Nick Pron via Outages <outages@outages.org> *Reply-To: *Nick Pron <Nick.Pron@xbase.com> *Date: *Wednesday, May 16, 2018 at 8:54 AM *To: *"outages@outages.org List" <outages@outages.org> *Subject: *[outages] Cogeco - Toronto outage?
Anyone have info about this? We’ve been having issues with our NNI since Monday with our customers (down, packet loss, etc)
Cogeco’s master tickets 718506 and 719701 claim they’ve swapped line cards in a core switch, upgraded IOS and still working with Cisco.
Our rep seems to hint at that there’s more at play here than the tickets are saying..
Last update from them was 7:35 AM EDT
Unfortunately, we have seen further instability on the affected network device and are continuing to work with our Cisco partner to isolate the issue and implement a restoration plan.
We are still unable to provide an estimated time to restore but we will provide further updates as soon as more information becomes available.
Once again we apologize for any disruption to your services that this may be causing.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Might be an attack of some sort. We pressured our rep and he seemed to point it at a security issue, but wasn’t able to give any more details. Their latest update seems to confirm it: Update 12:43 EDT Hrs - 16 MAY 2018 The Cogeco Peer 1 Networking team and our equipment partner, Cisco, have been enabling additional security features on the affected switch. We continue to monitor the success of that feature and will send an update once we have more information available. From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Mike Dresser via Outages Sent: Wednesday, May 16, 2018 12:39 PM To: outages@outages.org List <outages@outages.org> Subject: Re: [outages] Cogeco - Toronto outage? Appears they're having some bigger issues, I couldn't reach www.esasafe.com<http://www.esasafe.com> from a Cogeco business fiber line, but could from a connection that is based off cogeco's cable network(start.ca<http://start.ca>). MTR got completely lost on the fiber link(stops two hops past burl-ts1.cgocable.net<http://burl-ts1.cgocable.net>, with heavy(8-10%) packet loss from that point on), but made it to Rogers (the hoster of esasafe.com<http://esasafe.com>) from the home connection just fine. Just retested after typing the above up and it appears to work again, so they're having some issues. On Wed, May 16, 2018 at 9:20 AM, Nick Pron via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: LAN type. That’s what it seems like. We did notice that a customer we have on ICA we can’t reach them as one of the hops goes through Cogeco From: Paul Stewart [mailto:paul@paulstewart.org<mailto:paul@paulstewart.org>] Sent: Wednesday, May 16, 2018 8:57 AM To: Nick Pron <Nick.Pron@xbase.com<mailto:Nick.Pron@xbase.com>>; outages@outages.org<mailto:outages@outages.org> List <outages@outages.org<mailto:outages@outages.org>> Subject: Re: [outages] Cogeco - Toronto outage? Is the NNI with them for LAN type services or is it TPIA? We have both and are not experiencing any issues so sounds like possibly the demarcation switch you are connected to possibly …. Paul From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> on behalf of Nick Pron via Outages <outages@outages.org<mailto:outages@outages.org>> Reply-To: Nick Pron <Nick.Pron@xbase.com<mailto:Nick.Pron@xbase.com>> Date: Wednesday, May 16, 2018 at 8:54 AM To: "outages@outages.org List<mailto:outages@outages.org%20List>" <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] Cogeco - Toronto outage? Anyone have info about this? We’ve been having issues with our NNI since Monday with our customers (down, packet loss, etc) Cogeco’s master tickets 718506 and 719701 claim they’ve swapped line cards in a core switch, upgraded IOS and still working with Cisco. Our rep seems to hint at that there’s more at play here than the tickets are saying.. Last update from them was 7:35 AM EDT Unfortunately, we have seen further instability on the affected network device and are continuing to work with our Cisco partner to isolate the issue and implement a restoration plan. We are still unable to provide an estimated time to restore but we will provide further updates as soon as more information becomes available. Once again we apologize for any disruption to your services that this may be causing. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Looks like it was resolved around noon today EST for us. *Update 2:43 EDT Hrs - 16 MAY 2018* We have diagnosed a MAC address flooding issue which was affecting connectivity for a number of our clients. We have implemented a temporary control to resolve the impact. This temporary control was implemented just over an hour ago and since then we have not seen any reoccurrence of the issue and all ports remain enabled. We will continue to closely monitor the situation as we implement a permanent resolution to the issue. On Wed, May 16, 2018 at 12:54 PM, Nick Pron via Outages <outages@outages.org
wrote:
Might be an attack of some sort. We pressured our rep and he seemed to point it at a security issue, but wasn’t able to give any more details.
Their latest update seems to confirm it:
Update 12:43 EDT Hrs - 16 MAY 2018
The Cogeco Peer 1 Networking team and our equipment partner, Cisco, have been enabling additional *security features* on the affected switch. We continue to monitor the success of that feature and will send an update once we have more information available.
*From:* Outages [mailto:outages-bounces@outages.org] * On Behalf Of *Mike Dresser via Outages *Sent:* Wednesday, May 16, 2018 12:39 PM *To:* outages@outages.org List <outages@outages.org>
*Subject:* Re: [outages] Cogeco - Toronto outage?
Appears they're having some bigger issues, I couldn't reach www.esasafe.com from a Cogeco business fiber line, but could from a connection that is based off cogeco's cable network(start.ca). MTR got completely lost on the fiber link(stops two hops past burl-ts1.cgocable.net, with heavy(8-10%) packet loss from that point on), but made it to Rogers (the hoster of esasafe.com) from the home connection just fine.
Just retested after typing the above up and it appears to work again, so they're having some issues.
On Wed, May 16, 2018 at 9:20 AM, Nick Pron via Outages < outages@outages.org> wrote:
LAN type. That’s what it seems like.
We did notice that a customer we have on ICA we can’t reach them as one of the hops goes through Cogeco
*From:* Paul Stewart [mailto:paul@paulstewart.org] *Sent:* Wednesday, May 16, 2018 8:57 AM *To:* Nick Pron <Nick.Pron@xbase.com>; outages@outages.org List < outages@outages.org> *Subject:* Re: [outages] Cogeco - Toronto outage?
Is the NNI with them for LAN type services or is it TPIA? We have both and are not experiencing any issues so sounds like possibly the demarcation switch you are connected to possibly ….
Paul
*From: *Outages <outages-bounces@outages.org> on behalf of Nick Pron via Outages <outages@outages.org> *Reply-To: *Nick Pron <Nick.Pron@xbase.com> *Date: *Wednesday, May 16, 2018 at 8:54 AM *To: *"outages@outages.org List" <outages@outages.org> *Subject: *[outages] Cogeco - Toronto outage?
Anyone have info about this? We’ve been having issues with our NNI since Monday with our customers (down, packet loss, etc)
Cogeco’s master tickets 718506 and 719701 claim they’ve swapped line cards in a core switch, upgraded IOS and still working with Cisco.
Our rep seems to hint at that there’s more at play here than the tickets are saying..
Last update from them was 7:35 AM EDT
Unfortunately, we have seen further instability on the affected network device and are continuing to work with our Cisco partner to isolate the issue and implement a restoration plan.
We are still unable to provide an estimated time to restore but we will provide further updates as soon as more information becomes available.
Once again we apologize for any disruption to your services that this may be causing.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- *Laurent Dumont* *Spécialiste réseau / Network specialist* *Fibrenoire* - www.fibrenoire.ca A: 550 , avenue Beaumont, bureau 320, Montréal (Québec) H3N 1V1 T. 514 907-3002 x132 C.438-825-4642 ldumont@fibrenoire.ca <farchambault@fibrenoire.ca> Twitter: Twitter Fibrenoire <https://twitter.com/fibrenoire> *--Note de confidentialité--* *Ce message électronique et toutes pièces jointes contiennent des informations privilégiées et confidentielles. Si vous n'êtes pas le destinataire prévu, il est strictement interdit d'utiliser, de copier ou divulguer les informations ainsi transmises; merci d'en aviser l'expéditeur et de les supprimer de votre ordinateur.* *--Confidentiality Note--* *This email message and any attachments contain privileged and confidential information. If you are not the intended recipient, your use, copying or disclosure of this information is prohibited; please contact the sender and delete the material from your computer.*
Some more flapping today, our ENNI port has also been going up and down. Not much from Cogeco so far. Anyone else? On Wed, May 16, 2018 at 3:31 PM, Laurent Dumont <ldumont@fibrenoire.ca> wrote:
Looks like it was resolved around noon today EST for us.
*Update 2:43 EDT Hrs - 16 MAY 2018* We have diagnosed a MAC address flooding issue which was affecting connectivity for a number of our clients. We have implemented a temporary control to resolve the impact. This temporary control was implemented just over an hour ago and since then we have not seen any reoccurrence of the issue and all ports remain enabled. We will continue to closely monitor the situation as we implement a permanent resolution to the issue.
On Wed, May 16, 2018 at 12:54 PM, Nick Pron via Outages < outages@outages.org> wrote:
Might be an attack of some sort. We pressured our rep and he seemed to point it at a security issue, but wasn’t able to give any more details.
Their latest update seems to confirm it:
Update 12:43 EDT Hrs - 16 MAY 2018
The Cogeco Peer 1 Networking team and our equipment partner, Cisco, have been enabling additional *security features* on the affected switch. We continue to monitor the success of that feature and will send an update once we have more information available.
*From:* Outages [mailto:outages-bounces@outages.org] * On Behalf Of *Mike Dresser via Outages *Sent:* Wednesday, May 16, 2018 12:39 PM *To:* outages@outages.org List <outages@outages.org>
*Subject:* Re: [outages] Cogeco - Toronto outage?
Appears they're having some bigger issues, I couldn't reach www.esasafe.com from a Cogeco business fiber line, but could from a connection that is based off cogeco's cable network(start.ca). MTR got completely lost on the fiber link(stops two hops past burl-ts1.cgocable.net, with heavy(8-10%) packet loss from that point on), but made it to Rogers (the hoster of esasafe.com) from the home connection just fine.
Just retested after typing the above up and it appears to work again, so they're having some issues.
On Wed, May 16, 2018 at 9:20 AM, Nick Pron via Outages < outages@outages.org> wrote:
LAN type. That’s what it seems like.
We did notice that a customer we have on ICA we can’t reach them as one of the hops goes through Cogeco
*From:* Paul Stewart [mailto:paul@paulstewart.org] *Sent:* Wednesday, May 16, 2018 8:57 AM *To:* Nick Pron <Nick.Pron@xbase.com>; outages@outages.org List < outages@outages.org> *Subject:* Re: [outages] Cogeco - Toronto outage?
Is the NNI with them for LAN type services or is it TPIA? We have both and are not experiencing any issues so sounds like possibly the demarcation switch you are connected to possibly ….
Paul
*From: *Outages <outages-bounces@outages.org> on behalf of Nick Pron via Outages <outages@outages.org> *Reply-To: *Nick Pron <Nick.Pron@xbase.com> *Date: *Wednesday, May 16, 2018 at 8:54 AM *To: *"outages@outages.org List" <outages@outages.org> *Subject: *[outages] Cogeco - Toronto outage?
Anyone have info about this? We’ve been having issues with our NNI since Monday with our customers (down, packet loss, etc)
Cogeco’s master tickets 718506 and 719701 claim they’ve swapped line cards in a core switch, upgraded IOS and still working with Cisco.
Our rep seems to hint at that there’s more at play here than the tickets are saying..
Last update from them was 7:35 AM EDT
Unfortunately, we have seen further instability on the affected network device and are continuing to work with our Cisco partner to isolate the issue and implement a restoration plan.
We are still unable to provide an estimated time to restore but we will provide further updates as soon as more information becomes available.
Once again we apologize for any disruption to your services that this may be causing.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- *Laurent Dumont* *Spécialiste réseau / Network specialist* *Fibrenoire* - www.fibrenoire.ca A: 550 , avenue Beaumont, bureau 320, Montréal (Québec) H3N 1V1 T. 514 907-3002 x132 C.438-825-4642 ldumont@fibrenoire.ca <farchambault@fibrenoire.ca> Twitter: Twitter Fibrenoire <https://twitter.com/fibrenoire>
*--Note de confidentialité--*
*Ce message électronique et toutes pièces jointes contiennent des informations privilégiées et confidentielles. Si vous n'êtes pas le destinataire prévu, il est strictement interdit d'utiliser, de copier ou divulguer les informations ainsi transmises; merci d'en aviser l'expéditeur et de les supprimer de votre ordinateur.*
*--Confidentiality Note--*
*This email message and any attachments contain privileged and confidential information. If you are not the intended recipient, your use, copying or disclosure of this information is prohibited; please contact the sender and delete the material from your computer.*
-- *Laurent Dumont* *Spécialiste réseau / Network specialist* *Fibrenoire* - www.fibrenoire.ca A: 550 , avenue Beaumont, bureau 320, Montréal (Québec) H3N 1V1 T. 514 907-3002 x132 C.438-825-4642 ldumont@fibrenoire.ca <farchambault@fibrenoire.ca> Twitter: Twitter Fibrenoire <https://twitter.com/fibrenoire> *--Note de confidentialité--* *Ce message électronique et toutes pièces jointes contiennent des informations privilégiées et confidentielles. Si vous n'êtes pas le destinataire prévu, il est strictement interdit d'utiliser, de copier ou divulguer les informations ainsi transmises; merci d'en aviser l'expéditeur et de les supprimer de votre ordinateur.* *--Confidentiality Note--* *This email message and any attachments contain privileged and confidential information. If you are not the intended recipient, your use, copying or disclosure of this information is prohibited; please contact the sender and delete the material from your computer.*
participants (4)
-
Laurent Dumont -
Mike Dresser -
Nick Pron -
Paul Stewart