Seems Telia/AS1299 has a dodgy line card in New York causing packet loss across their network. Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype) http://www.peeringdb.com/view.php?asn=13335
Agreed, seeing weird things via Telia as well. Ticket is opened with Telia. Aaron Endly Operations Center Manager | Cologix, Inc. 9 Wing Drive, Cedar Knolls, NJ 07927 P: 973.590.5107 Email (aaron.endly@cologix.com) -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Marty Strong via Outages Sent: Thursday, February 11, 2016 5:57 PM To: outages@outages.org Subject: [outages] Telia in New York Seems Telia/AS1299 has a dodgy line card in New York causing packet loss across their network. Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype) http://www.peeringdb.com/view.php?asn=13335 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
They’ve been slow to even open a ticket, seems from a couple of sources that it’s a big issue for them. Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype) http://www.peeringdb.com/view.php?asn=13335
On 11 Feb 2016, at 23:01, Aaron Endly <aaron.endly@cologix.com> wrote:
Agreed, seeing weird things via Telia as well. Ticket is opened with Telia.
Aaron Endly Operations Center Manager | Cologix, Inc. 9 Wing Drive, Cedar Knolls, NJ 07927 P: 973.590.5107 Email (aaron.endly@cologix.com)
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Marty Strong via Outages Sent: Thursday, February 11, 2016 5:57 PM To: outages@outages.org Subject: [outages] Telia in New York
Seems Telia/AS1299 has a dodgy line card in New York causing packet loss across their network.
Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype)
http://www.peeringdb.com/view.php?asn=13335
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I confirm a lot of reported incidents around this today as well. Have not heard anything direct from Telia though. On Thu, Feb 11, 2016 at 5:57 PM, Marty Strong via Outages < outages@outages.org> wrote:
Seems Telia/AS1299 has a dodgy line card in New York causing packet loss across their network.
Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype)
http://www.peeringdb.com/view.php?asn=13335
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- *James Cornman* *Chief Technology Officer*jcornman@atlanticmetro.net 212.792.9950 - ext 101 *Atlantic Metro Communications* *4 Century Drive, Parsippany NJ 07054* *Colocation • Cloud Hosting • Network Connectivity • Managed Services* Follow us on Twitter: @atlanticmetro <https://twitter.com/atlanticmetro> *• Like us on Facebook <https://www.facebook.com/atlanticmetro>* www.atlanticmetro.net
I did end up hearing directly from the Telia NOC. They're aware of the problem and working to repair it, but do not have an ETTR as of about 30 mins ago. --j On Thu, Feb 11, 2016 at 3:02 PM, James Cornman via Outages <outages@outages.org> wrote:
I confirm a lot of reported incidents around this today as well. Have not heard anything direct from Telia though.
On Thu, Feb 11, 2016 at 5:57 PM, Marty Strong via Outages <outages@outages.org> wrote:
Seems Telia/AS1299 has a dodgy line card in New York causing packet loss across their network.
Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype)
http://www.peeringdb.com/view.php?asn=13335
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
--
James Cornman
Chief Technology Officer jcornman@atlanticmetro.net 212.792.9950 - ext 101
Atlantic Metro Communications
4 Century Drive, Parsippany NJ 07054
Colocation • Cloud Hosting • Network Connectivity • Managed Services Follow us on Twitter: @atlanticmetro • Like us on Facebook www.atlanticmetro.net
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Does anyone know the Device name of the router having issues? From: Outages [mailto:outages-bounces@outages.org] On Behalf Of James Cornman via Outages Sent: Thursday, February 11, 2016 5:03 PM To: Marty Strong <marty@cloudflare.com> Cc: outages@outages.org Subject: Re: [outages] Telia in New York I confirm a lot of reported incidents around this today as well. Have not heard anything direct from Telia though. On Thu, Feb 11, 2016 at 5:57 PM, Marty Strong via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Seems Telia/AS1299 has a dodgy line card in New York causing packet loss across their network. Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty@cloudflare.com<mailto:marty@cloudflare.com> +44 7584 906 055<tel:%2B44%207584%20906%20055> smartflare (Skype) http://www.peeringdb.com/view.php?asn=13335 _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages -- James Cornman Chief Technology Officer jcornman@atlanticmetro.net<mailto:jcornman@atlanticmetro.net> 212.792.9950 - ext 101 Atlantic Metro Communications 4 Century Drive, Parsippany NJ 07054 Colocation • Cloud Hosting • Network Connectivity • Managed Services Follow us on Twitter: @atlanticmetro<https://twitter.com/atlanticmetro> • Like us on Facebook<https://www.facebook.com/atlanticmetro> www.atlanticmetro.net<https://www.atlanticmetro.net/> ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
I'm observing the 100% loss on a leg of the path that used to show up between "nyk-bb2-link.telia.net" and "ldn-b7-link.telia.net" --j On Thu, Feb 11, 2016 at 3:06 PM, Erik Sundberg via Outages <outages@outages.org> wrote:
Does anyone know the Device name of the router having issues?
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of James Cornman via Outages Sent: Thursday, February 11, 2016 5:03 PM To: Marty Strong <marty@cloudflare.com> Cc: outages@outages.org Subject: Re: [outages] Telia in New York
I confirm a lot of reported incidents around this today as well. Have not heard anything direct from Telia though.
On Thu, Feb 11, 2016 at 5:57 PM, Marty Strong via Outages <outages@outages.org> wrote:
Seems Telia/AS1299 has a dodgy line card in New York causing packet loss across their network.
Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype)
http://www.peeringdb.com/view.php?asn=13335
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
--
James Cornman
Chief Technology Officer jcornman@atlanticmetro.net 212.792.9950 - ext 101
Atlantic Metro Communications
4 Century Drive, Parsippany NJ 07054
Colocation • Cloud Hosting • Network Connectivity • Managed Services Follow us on Twitter: @atlanticmetro • Like us on Facebook www.atlanticmetro.net
________________________________
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Dear Customer,
Our engineer will attempt to reset several switch interface boards in the router which are alarming in an attempt to clear packet loss. Packet loss will increase during this time over a period of 30-60 minutes. This will occur in the next 30 minutes.
Kind Regards, Josh Little TeliaSonera International Carrier Customer Care Phone +46 771 191170 Fax +46 8 750 5427 carrier-csc@teliasonera.com
Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype) http://www.peeringdb.com/view.php?asn=13335
On 11 Feb 2016, at 23:14, Jonathan Lassoff via Outages <outages@outages.org> wrote:
I'm observing the 100% loss on a leg of the path that used to show up between "nyk-bb2-link.telia.net" and "ldn-b7-link.telia.net"
--j
On Thu, Feb 11, 2016 at 3:06 PM, Erik Sundberg via Outages <outages@outages.org> wrote:
Does anyone know the Device name of the router having issues?
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of James Cornman via Outages Sent: Thursday, February 11, 2016 5:03 PM To: Marty Strong <marty@cloudflare.com> Cc: outages@outages.org Subject: Re: [outages] Telia in New York
I confirm a lot of reported incidents around this today as well. Have not heard anything direct from Telia though.
On Thu, Feb 11, 2016 at 5:57 PM, Marty Strong via Outages <outages@outages.org> wrote:
Seems Telia/AS1299 has a dodgy line card in New York causing packet loss across their network.
Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype)
http://www.peeringdb.com/view.php?asn=13335
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
--
James Cornman
Chief Technology Officer jcornman@atlanticmetro.net 212.792.9950 - ext 101
Atlantic Metro Communications
4 Century Drive, Parsippany NJ 07054
Colocation • Cloud Hosting • Network Connectivity • Managed Services Follow us on Twitter: @atlanticmetro • Like us on Facebook www.atlanticmetro.net
________________________________
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Carrier-csc@teliasonera.com 6:57 PM (0 minutes ago) to me Dear support, We are currently experiencing an issue on our backbone router in New York. We have dispatched a technician to remove faulty hardware, no ETA at this time. Further updates will come via Service Disturbance notification case 00533792. On Thu, Feb 11, 2016 at 6:21 PM, Marty Strong via Outages < outages@outages.org> wrote:
Dear Customer,
Our engineer will attempt to reset several switch interface boards in the router which are alarming in an attempt to clear packet loss. Packet loss will increase during this time over a period of 30-60 minutes. This will occur in the next 30 minutes.
Kind Regards, Josh Little TeliaSonera International Carrier Customer Care Phone +46 771 191170 Fax +46 8 750 5427 carrier-csc@teliasonera.com
Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype)
http://www.peeringdb.com/view.php?asn=13335
On 11 Feb 2016, at 23:14, Jonathan Lassoff via Outages < outages@outages.org> wrote:
I'm observing the 100% loss on a leg of the path that used to show up between "nyk-bb2-link.telia.net" and "ldn-b7-link.telia.net"
--j
On Thu, Feb 11, 2016 at 3:06 PM, Erik Sundberg via Outages <outages@outages.org> wrote:
Does anyone know the Device name of the router having issues?
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of James Cornman via Outages Sent: Thursday, February 11, 2016 5:03 PM To: Marty Strong <marty@cloudflare.com> Cc: outages@outages.org Subject: Re: [outages] Telia in New York
I confirm a lot of reported incidents around this today as well. Have not heard anything direct from Telia though.
On Thu, Feb 11, 2016 at 5:57 PM, Marty Strong via Outages <outages@outages.org> wrote:
Seems Telia/AS1299 has a dodgy line card in New York causing packet loss across their network.
Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype)
http://www.peeringdb.com/view.php?asn=13335
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
--
James Cornman
Chief Technology Officer jcornman@atlanticmetro.net 212.792.9950 - ext 101
Atlantic Metro Communications
4 Century Drive, Parsippany NJ 07054
Colocation • Cloud Hosting • Network Connectivity • Managed Services Follow us on Twitter: @atlanticmetro • Like us on Facebook www.atlanticmetro.net
________________________________
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- *James Cornman* *Chief Technology Officer*jcornman@atlanticmetro.net 212.792.9950 - ext 101 *Atlantic Metro Communications* *4 Century Drive, Parsippany NJ 07054* *Colocation • Cloud Hosting • Network Connectivity • Managed Services* Follow us on Twitter: @atlanticmetro <https://twitter.com/atlanticmetro> *• Like us on Facebook <https://www.facebook.com/atlanticmetro>* www.atlanticmetro.net
FWIW -- we're back online through this nyk->ldn path as of 2016-02-12T00:18:00Z On Thu, Feb 11, 2016 at 3:58 PM, James Cornman via Outages < outages@outages.org> wrote:
Carrier-csc@teliasonera.com 6:57 PM (0 minutes ago) to me
Dear support,
We are currently experiencing an issue on our backbone router in New York. We have dispatched a technician to remove faulty hardware, no ETA at this time. Further updates will come via Service Disturbance notification case 00533792.
On Thu, Feb 11, 2016 at 6:21 PM, Marty Strong via Outages < outages@outages.org> wrote:
Dear Customer,
Our engineer will attempt to reset several switch interface boards in the router which are alarming in an attempt to clear packet loss. Packet loss will increase during this time over a period of 30-60 minutes. This will occur in the next 30 minutes.
Kind Regards, Josh Little TeliaSonera International Carrier Customer Care Phone +46 771 191170 Fax +46 8 750 5427 carrier-csc@teliasonera.com
Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype)
http://www.peeringdb.com/view.php?asn=13335
On 11 Feb 2016, at 23:14, Jonathan Lassoff via Outages < outages@outages.org> wrote:
I'm observing the 100% loss on a leg of the path that used to show up between "nyk-bb2-link.telia.net" and "ldn-b7-link.telia.net"
--j
On Thu, Feb 11, 2016 at 3:06 PM, Erik Sundberg via Outages <outages@outages.org> wrote:
Does anyone know the Device name of the router having issues?
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of James Cornman via Outages Sent: Thursday, February 11, 2016 5:03 PM To: Marty Strong <marty@cloudflare.com> Cc: outages@outages.org Subject: Re: [outages] Telia in New York
I confirm a lot of reported incidents around this today as well. Have not heard anything direct from Telia though.
On Thu, Feb 11, 2016 at 5:57 PM, Marty Strong via Outages <outages@outages.org> wrote:
Seems Telia/AS1299 has a dodgy line card in New York causing packet loss across their network.
Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype)
http://www.peeringdb.com/view.php?asn=13335
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
--
James Cornman
Chief Technology Officer jcornman@atlanticmetro.net 212.792.9950 - ext 101
Atlantic Metro Communications
4 Century Drive, Parsippany NJ 07054
Colocation • Cloud Hosting • Network Connectivity • Managed Services Follow us on Twitter: @atlanticmetro • Like us on Facebook www.atlanticmetro.net
________________________________
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
--
*James Cornman*
*Chief Technology Officer*jcornman@atlanticmetro.net 212.792.9950 - ext 101
*Atlantic Metro Communications*
*4 Century Drive, Parsippany NJ 07054*
*Colocation • Cloud Hosting • Network Connectivity • Managed Services* Follow us on Twitter: @atlanticmetro <https://twitter.com/atlanticmetro> *• Like us on Facebook <https://www.facebook.com/atlanticmetro>* www.atlanticmetro.net
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Can confirm. I've been tracking an issue there since about 2016-02-11T18:02:00Z (5 hours and counting now...) This smells like a bad link in a logical aggregate bundle that isn't getting signaled down. Adjacent hosts on either side of the broken path work and don't work. Only certain src/dest IP tuples seem to be getting dropped. I'm only observing this on the New York -> London Telia path. Cheers, jof On Thu, Feb 11, 2016 at 2:57 PM, Marty Strong via Outages <outages@outages.org> wrote:
Seems Telia/AS1299 has a dodgy line card in New York causing packet loss across their network.
Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty@cloudflare.com +44 7584 906 055 smartflare (Skype)
http://www.peeringdb.com/view.php?asn=13335
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (5)
-
Aaron Endly -
Erik Sundberg -
James Cornman -
Jonathan Lassoff -
Marty Strong