Anyone else seeing any issues in NYC with Lightower/Sidera? Starting around 6:05AM (EST) we saw heavy packet loss across their network (from Sidera legacy) and some of our point to points bounce. The P to P's have some what stabilized (though we've seen a couple more bounces since then) the IP side of things have been pretty much unusable (we shut our session down.) It also looks like their phone system is having a problem - calls into them seem to drop after trying to connect to the NOC - which I'm sure is pretty busy. -Jayson
Yes. Heavy packet loss on their circuits in NYC. - Gaurav. On Nov 7, 2013 5:28 PM, "Jayson Navitsky" <jnavitsky@gmail.com> wrote:
Anyone else seeing any issues in NYC with Lightower/Sidera? Starting around 6:05AM (EST) we saw heavy packet loss across their network (from Sidera legacy) and some of our point to points bounce. The P to P's have some what stabilized (though we've seen a couple more bounces since then) the IP side of things have been pretty much unusable (we shut our session down.) It also looks like their phone system is having a problem - calls into them seem to drop after trying to connect to the NOC - which I'm sure is pretty busy.
-Jayson
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We are seeing the same issues with Sidera in New York. --Vincent From: Gaurav Taparia <gaurav.taparia@gmail.com> To: Jayson Navitsky <jnavitsky@gmail.com> Cc: outages@outages.org Date: 11/07/2013 07:06 AM Subject: Re: [outages] Lightower/Sidera - NYC Sent by: "Outages" <outages-bounces@outages.org> Yes. Heavy packet loss on their circuits in NYC. - Gaurav. On Nov 7, 2013 5:28 PM, "Jayson Navitsky" <jnavitsky@gmail.com> wrote: Anyone else seeing any issues in NYC with Lightower/Sidera? Starting around 6:05AM (EST) we saw heavy packet loss across their network (from Sidera legacy) and some of our point to points bounce. The P to P's have some what stabilized (though we've seen a couple more bounces since then) the IP side of things have been pretty much unusable (we shut our session down.) It also looks like their phone system is having a problem - calls into them seem to drop after trying to connect to the NOC - which I'm sure is pretty busy. -Jayson _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We use them for L2 transport between Nycm and BSTP. It started taking errors at approx. 6:05am EST. Thankfully this is just a protect path for us. Calling their NOC results in busy signals or just dead air. ~ Karl On Thu, Nov 7, 2013 at 7:08 AM, Vincent Aniello <VAniello@portware.com>wrote:
We are seeing the same issues with Sidera in New York.
--Vincent
From: Gaurav Taparia <gaurav.taparia@gmail.com> To: Jayson Navitsky <jnavitsky@gmail.com> Cc: outages@outages.org Date: 11/07/2013 07:06 AM Subject: Re: [outages] Lightower/Sidera - NYC Sent by: "Outages" <outages-bounces@outages.org> ------------------------------
Yes. Heavy packet loss on their circuits in NYC.
- Gaurav.
On Nov 7, 2013 5:28 PM, "Jayson Navitsky" <*jnavitsky@gmail.com*<jnavitsky@gmail.com>> wrote: Anyone else seeing any issues in NYC with Lightower/Sidera? Starting around 6:05AM (EST) we saw heavy packet loss across their network (from Sidera legacy) and some of our point to points bounce. The P to P's have some what stabilized (though we've seen a couple more bounces since then) the IP side of things have been pretty much unusable (we shut our session down.) It also looks like their phone system is having a problem - calls into them seem to drop after trying to connect to the NOC - which I'm sure is pretty busy.
-Jayson
_______________________________________________ Outages mailing list *Outages@outages.org* <Outages@outages.org> *https://puck.nether.net/mailman/listinfo/outages*<https://puck.nether.net/mailman/listinfo/outages> _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Sales guy states it's a "large scale MPLS outage" at 111 8th, tech on-site @7 a.m. On Nov 7, 2013, at 7:52 AM, Karl Clapp wrote:
We use them for L2 transport between Nycm and BSTP. It started taking errors at approx. 6:05am EST. Thankfully this is just a protect path for us.
Calling their NOC results in busy signals or just dead air.
~ Karl
On Thu, Nov 7, 2013 at 7:08 AM, Vincent Aniello <VAniello@portware.com> wrote: We are seeing the same issues with Sidera in New York.
--Vincent
From: Gaurav Taparia <gaurav.taparia@gmail.com> To: Jayson Navitsky <jnavitsky@gmail.com> Cc: outages@outages.org Date: 11/07/2013 07:06 AM Subject: Re: [outages] Lightower/Sidera - NYC Sent by: "Outages" <outages-bounces@outages.org>
Yes. Heavy packet loss on their circuits in NYC. - Gaurav.
On Nov 7, 2013 5:28 PM, "Jayson Navitsky" <jnavitsky@gmail.com> wrote: Anyone else seeing any issues in NYC with Lightower/Sidera? Starting around 6:05AM (EST) we saw heavy packet loss across their network (from Sidera legacy) and some of our point to points bounce. The P to P's have some what stabilized (though we've seen a couple more bounces since then) the IP side of things have been pretty much unusable (we shut our session down.) It also looks like their phone system is having a problem - calls into them seem to drop after trying to connect to the NOC - which I'm sure is pretty busy.
-Jayson
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Similar experience for me. IP is down and I get a busy signal when I call their NOC. Sent from my iPhone On Nov 7, 2013, at 6:58 AM, "Jayson Navitsky" <jnavitsky@gmail.com> wrote: Anyone else seeing any issues in NYC with Lightower/Sidera? Starting around 6:05AM (EST) we saw heavy packet loss across their network (from Sidera legacy) and some of our point to points bounce. The P to P's have some what stabilized (though we've seen a couple more bounces since then) the IP side of things have been pretty much unusable (we shut our session down.) It also looks like their phone system is having a problem - calls into them seem to drop after trying to connect to the NOC - which I'm sure is pretty busy. -Jayson _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages "This communication is intended solely for the addressee and is confidential and not for third party unauthorized distribution"
Likely related to the fiber cut they experienced on their network earlier this week which they routed around. Shane Ronan, Vice President – Technology Architect State Street Global Exchange | 1230 Avenue of the Americas, 18th Fl | New York, NY 10020 P (212) 259-3023 | M (347) 413-4503 sronan@eexchange.com<mailto:sronan@eexchange.com> From: Jayson Navitsky <jnavitsky@gmail.com<mailto:jnavitsky@gmail.com>> Date: Thursday, November 7, 2013 at 6:55 AM To: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] Lightower/Sidera - NYC Anyone else seeing any issues in NYC with Lightower/Sidera? Starting around 6:05AM (EST) we saw heavy packet loss across their network (from Sidera legacy) and some of our point to points bounce. The P to P's have some what stabilized (though we've seen a couple more bounces since then) the IP side of things have been pretty much unusable (we shut our session down.) It also looks like their phone system is having a problem - calls into them seem to drop after trying to connect to the NOC - which I'm sure is pretty busy. -Jayson ________________________________ http://www.statestreet.com/emaildisclaimer/
LightTower/Sidera master ticker for this is LT1323725. I was able to get through their support line now. The ETR is 1-2 hours from now. --- Karl Clapp Network Engineering Manager Biddeford Internet Corp. d.b.a. GWI (AS5760) Office: 207.602.1144 Mobile: 207.641.3907 Fax: 207.286.2061 On Thu, Nov 7, 2013 at 9:39 AM, <SRonan@eexchange.com> wrote:
Likely related to the fiber cut they experienced on their network earlier this week which they routed around.
Shane Ronan, Vice President – Technology Architect
State Street Global Exchange | 1230 Avenue of the Americas, 18th Fl | New York, NY 10020
P (212) 259-3023 | M (347) 413-4503
sronan@eexchange.com
From: Jayson Navitsky <jnavitsky@gmail.com> Date: Thursday, November 7, 2013 at 6:55 AM To: "outages@outages.org" <outages@outages.org> Subject: [outages] Lightower/Sidera - NYC
Anyone else seeing any issues in NYC with Lightower/Sidera? Starting around 6:05AM (EST) we saw heavy packet loss across their network (from Sidera legacy) and some of our point to points bounce. The P to P's have some what stabilized (though we've seen a couple more bounces since then) the IP side of things have been pretty much unusable (we shut our session down.) It also looks like their phone system is having a problem - calls into them seem to drop after trying to connect to the NOC - which I'm sure is pretty busy.
-Jayson
------------------------------
http://www.statestreet.com/emaildisclaimer/
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Just as a data point, LT for me is routing into Equinix NY5 over HE, so at least there's connectivity... My traceroute [v0.82] icmp (0.0.0.0) Thu Nov 7 07:24:09 2013 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.1.1 0.0% 29 6.7 3.2 0.8 12.2 2.7 2. 10.0.1.49 0.0% 28 8.0 4.3 1.5 13.1 3.0 3. 10.0.24.73 0.0% 28 6.7 4.5 1.9 14.7 3.4 4. 25.66.208.web-pass.com 0.0% 28 1.9 11.2 1.9 48.0 12.3 5. 2-1.760.core.fre.web-pass.com 0.0% 28 9.2 6.4 3.1 19.7 4.0 6. gige-g6-2.core1.fmt1.he.net 0.0% 28 9.6 12.4 3.5 26.4 6.2 7. 10gigabitethernet1-2.core1.sjc2.he.net 0.0% 28 11.9 9.5 3.5 18.7 4.3 8. 10gigabitethernet9-1.core1.nyc4.he.net 0.0% 28 87.9 79.5 73.3 92.7 5.5 9. 10gigabitethernet4-1.core1.nyc5.he.net 0.0% 28 76.6 80.5 73.6 106.7 7.0 10. lightower-fiber-networks.10gigabitether 0.0% 28 78.0 82.3 74.4 108.0 9.4 11. xe-8-3-0.nycmnyzrj91.lightower.net 0.0% 28 89.1 83.6 74.2 144.7 15.8 12. ae2-jrcynj67j41.lightower.net 0.0% 28 85.4 88.6 80.1 131.5 12.1 13. 192.154.116.14.lightower.net 0.0% 28 85.6 83.9 80.2 95.6 3.5 14. ??? John On Nov 7, 2013, at 7:19 AM, Karl Clapp <kclapp@staff.gwi.net> wrote:
LightTower/Sidera master ticker for this is LT1323725.
I was able to get through their support line now.
The ETR is 1-2 hours from now.
--- Karl Clapp Network Engineering Manager Biddeford Internet Corp. d.b.a. GWI (AS5760) Office: 207.602.1144 Mobile: 207.641.3907 Fax: 207.286.2061
On Thu, Nov 7, 2013 at 9:39 AM, <SRonan@eexchange.com> wrote: Likely related to the fiber cut they experienced on their network earlier this week which they routed around.
Shane Ronan, Vice President – Technology Architect State Street Global Exchange | 1230 Avenue of the Americas, 18th Fl | New York, NY 10020 P (212) 259-3023 | M (347) 413-4503 sronan@eexchange.com
From: Jayson Navitsky <jnavitsky@gmail.com> Date: Thursday, November 7, 2013 at 6:55 AM To: "outages@outages.org" <outages@outages.org> Subject: [outages] Lightower/Sidera - NYC
Anyone else seeing any issues in NYC with Lightower/Sidera? Starting around 6:05AM (EST) we saw heavy packet loss across their network (from Sidera legacy) and some of our point to points bounce. The P to P's have some what stabilized (though we've seen a couple more bounces since then) the IP side of things have been pretty much unusable (we shut our session down.) It also looks like their phone system is having a problem - calls into them seem to drop after trying to connect to the NOC - which I'm sure is pretty busy.
-Jayson
http://www.statestreet.com/emaildisclaimer/
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I contacted their NOC about 45min ago. They have stated they are still working on resolution. They have reports about some services being functional, while other still experiencing packet-loss and latency. It does not sound like they have been able to identify the root cause and are still troubleshooting. Here is the number I used to get through to their NOC: 888-583-4237, option 1. From there, just follow the prompts. Fwiw, my customer circuit from NYC to NJ is still affected. On Thu, Nov 7, 2013 at 10:25 AM, John Kinsella <jlk@thrashyour.com> wrote:
Just as a data point, LT for me is routing into Equinix NY5 over HE, so at least there's connectivity...
My traceroute [v0.82] icmp (0.0.0.0) Thu Nov 7 07:24:09 2013 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.1.1 0.0% 29 6.7 3.2 0.8 12.2 2.7 2. 10.0.1.49 0.0% 28 8.0 4.3 1.5 13.1 3.0 3. 10.0.24.73 0.0% 28 6.7 4.5 1.9 14.7 3.4 4. 25.66.208.web-pass.com 0.0% 28 1.9 11.2 1.9 48.0 12.3 5. 2-1.760.core.fre.web-pass.com 0.0% 28 9.2 6.4 3.1 19.7 4.0 6. gige-g6-2.core1.fmt1.he.net 0.0% 28 9.6 12.4 3.5 26.4 6.2 7. 10gigabitethernet1-2.core1.sjc2.he.net 0.0% 28 11.9 9.5 3.5 18.7 4.3 8. 10gigabitethernet9-1.core1.nyc4.he.net 0.0% 28 87.9 79.5 73.3 92.7 5.5 9. 10gigabitethernet4-1.core1.nyc5.he.net 0.0% 28 76.6 80.5 73.6 106.7 7.0 10. lightower-fiber-networks.10gigabitether 0.0% 28 78.0 82.3 74.4 108.0 9.4 11. xe-8-3-0.nycmnyzrj91.lightower.net 0.0% 28 89.1 83.6 74.2 144.7 15.8 12. ae2-jrcynj67j41.lightower.net 0.0% 28 85.4 88.6 80.1 131.5 12.1 13. 192.154.116.14.lightower.net 0.0% 28 85.6 83.9 80.2 95.6 3.5 14. ???
John
On Nov 7, 2013, at 7:19 AM, Karl Clapp <kclapp@staff.gwi.net> wrote:
LightTower/Sidera master ticker for this is LT1323725.
I was able to get through their support line now.
The ETR is 1-2 hours from now.
--- Karl Clapp Network Engineering Manager Biddeford Internet Corp. d.b.a. GWI (AS5760) Office: 207.602.1144 Mobile: 207.641.3907 Fax: 207.286.2061
On Thu, Nov 7, 2013 at 9:39 AM, <SRonan@eexchange.com> wrote:
Likely related to the fiber cut they experienced on their network earlier this week which they routed around.
Shane Ronan, Vice President – Technology Architect State Street Global Exchange | 1230 Avenue of the Americas, 18th Fl | New York, NY 10020 P (212) 259-3023 | M (347) 413-4503 sronan@eexchange.com
From: Jayson Navitsky <jnavitsky@gmail.com> Date: Thursday, November 7, 2013 at 6:55 AM To: "outages@outages.org" <outages@outages.org> Subject: [outages] Lightower/Sidera - NYC
Anyone else seeing any issues in NYC with Lightower/Sidera? Starting around 6:05AM (EST) we saw heavy packet loss across their network (from Sidera legacy) and some of our point to points bounce. The P to P's have some what stabilized (though we've seen a couple more bounces since then) the IP side of things have been pretty much unusable (we shut our session down.) It also looks like their phone system is having a problem - calls into them seem to drop after trying to connect to the NOC - which I'm sure is pretty busy.
-Jayson
------------------------------
http://www.statestreet.com/emaildisclaimer/
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Our circuits stabilized mid-afternoon, but shortly after 4pm EST we started seeing issues again. --Vincent From: Karl Clapp <kclapp@staff.gwi.net> To: "outages@outages.org" <outages@outages.org> Date: 11/07/2013 03:23 PM Subject: Re: [outages] Lightower/Sidera - NYC Sent by: "Outages" <outages-bounces@outages.org> I contacted their NOC about 45min ago. They have stated they are still working on resolution. They have reports about some services being functional, while other still experiencing packet-loss and latency. It does not sound like they have been able to identify the root cause and are still troubleshooting. Here is the number I used to get through to their NOC: 888-583-4237, option 1. From there, just follow the prompts. Fwiw, my customer circuit from NYC to NJ is still affected. On Thu, Nov 7, 2013 at 10:25 AM, John Kinsella <jlk@thrashyour.com> wrote: Just as a data point, LT for me is routing into Equinix NY5 over HE, so at least there's connectivity... My traceroute [v0.82] icmp (0.0.0.0) Thu Nov 7 07:24:09 2013 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.1.1 0.0% 29 6.7 3.2 0.8 12.2 2.7 2. 10.0.1.49 0.0% 28 8.0 4.3 1.5 13.1 3.0 3. 10.0.24.73 0.0% 28 6.7 4.5 1.9 14.7 3.4 4. 25.66.208.web-pass.com 0.0% 28 1.9 11.2 1.9 48.0 12.3 5. 2-1.760.core.fre.web-pass.com 0.0% 28 9.2 6.4 3.1 19.7 4.0 6. gige-g6-2.core1.fmt1.he.net 0.0% 28 9.6 12.4 3.5 26.4 6.2 7. 10gigabitethernet1-2.core1.sjc2.he.net 0.0% 28 11.9 9.5 3.5 18.7 4.3 8. 10gigabitethernet9-1.core1.nyc4.he.net 0.0% 28 87.9 79.5 73.3 92.7 5.5 9. 10gigabitethernet4-1.core1.nyc5.he.net 0.0% 28 76.6 80.5 73.6 106.7 7.0 10. lightower-fiber-networks.10gigabitether 0.0% 28 78.0 82.3 74.4 108.0 9.4 11. xe-8-3-0.nycmnyzrj91.lightower.net 0.0% 28 89.1 83.6 74.2 144.7 15.8 12. ae2-jrcynj67j41.lightower.net 0.0% 28 85.4 88.6 80.1 131.5 12.1 13. 192.154.116.14.lightower.net 0.0% 28 85.6 83.9 80.2 95.6 3.5 14. ??? John On Nov 7, 2013, at 7:19 AM, Karl Clapp <kclapp@staff.gwi.net> wrote: LightTower/Sidera master ticker for this is LT1323725. I was able to get through their support line now. The ETR is 1-2 hours from now. --- Karl Clapp Network Engineering Manager Biddeford Internet Corp. d.b.a. GWI (AS5760) Office: 207.602.1144 Mobile: 207.641.3907 Fax: 207.286.2061 On Thu, Nov 7, 2013 at 9:39 AM, <SRonan@eexchange.com> wrote: Likely related to the fiber cut they experienced on their network earlier this week which they routed around. Shane Ronan, Vice President – Technology Architect State Street Global Exchange | 1230 Avenue of the Americas, 18th Fl | New York, NY 10020 P (212) 259-3023 | M (347) 413-4503 sronan@eexchange.com From: Jayson Navitsky <jnavitsky@gmail.com> Date: Thursday, November 7, 2013 at 6:55 AM To: "outages@outages.org" <outages@outages.org> Subject: [outages] Lightower/Sidera - NYC Anyone else seeing any issues in NYC with Lightower/Sidera? Starting around 6:05AM (EST) we saw heavy packet loss across their network (from Sidera legacy) and some of our point to points bounce. The P to P's have some what stabilized (though we've seen a couple more bounces since then) the IP side of things have been pretty much unusable (we shut our session down.) It also looks like their phone system is having a problem - calls into them seem to drop after trying to connect to the NOC - which I'm sure is pretty busy. -Jayson http://www.statestreet.com/emaildisclaimer/ _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Likewise. Must be addressing root cause now? Andrew Bunde Team Lead, Storage & Network (405) 217 3811 Direct (405) 543 8288 Mobile (877) 405 2446 Agio Support andrew.bunde@agio.com<mailto:andrew.bunde@agio.com> Agio | agio.com From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Vincent Aniello Sent: Thursday, November 07, 2013 3:29 PM To: kclapp@staff.gwi.net Cc: outages@outages.org; Outages Subject: Re: [outages] Lightower/Sidera - NYC Our circuits stabilized mid-afternoon, but shortly after 4pm EST we started seeing issues again. --Vincent From: Karl Clapp <kclapp@staff.gwi.net<mailto:kclapp@staff.gwi.net>> To: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Date: 11/07/2013 03:23 PM Subject: Re: [outages] Lightower/Sidera - NYC Sent by: "Outages" <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> ________________________________ I contacted their NOC about 45min ago. They have stated they are still working on resolution. They have reports about some services being functional, while other still experiencing packet-loss and latency. It does not sound like they have been able to identify the root cause and are still troubleshooting. Here is the number I used to get through to their NOC: 888-583-4237, option 1. From there, just follow the prompts. Fwiw, my customer circuit from NYC to NJ is still affected. On Thu, Nov 7, 2013 at 10:25 AM, John Kinsella <jlk@thrashyour.com<mailto:jlk@thrashyour.com>> wrote: Just as a data point, LT for me is routing into Equinix NY5 over HE, so at least there's connectivity... My traceroute [v0.82] icmp (0.0.0.0) Thu Nov 7 07:24:09 2013 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.1.1 0.0% 29 6.7 3.2 0.8 12.2 2.7 2. 10.0.1.49 0.0% 28 8.0 4.3 1.5 13.1 3.0 3. 10.0.24.73 0.0% 28 6.7 4.5 1.9 14.7 3.4 4. 25.66.208.web-pass.com<http://web-pass.com/> 0.0% 28 1.9 11.2 1.9 48.0 12.3 5. 2-1.760.core.fre.web-pass.com<http://core.fre.web-pass.com/> 0.0% 28 9.2 6.4 3.1 19.7 4.0 6. gige-g6-2.core1.fmt1.he.net<http://core1.fmt1.he.net/> 0.0% 28 9.6 12.4 3.5 26.4 6.2 7. 10gigabitethernet1-2.core1.sjc2.he.net<http://core1.sjc2.he.net/> 0.0% 28 11.9 9.5 3.5 18.7 4.3 8. 10gigabitethernet9-1.core1.nyc4.he.net<http://core1.nyc4.he.net/> 0.0% 28 87.9 79.5 73.3 92.7 5.5 9. 10gigabitethernet4-1.core1.nyc5.he.net<http://core1.nyc5.he.net/> 0.0% 28 76.6 80.5 73.6 106.7 7.0 10. lightower-fiber-networks.10gigabitether 0.0% 28 78.0 82.3 74.4 108.0 9.4 11. xe-8-3-0.nycmnyzrj91.lightower.net<http://xe-8-3-0.nycmnyzrj91.lightower.net/> 0.0% 28 89.1 83.6 74.2 144.7 15.8 12. ae2-jrcynj67j41.lightower.net<http://ae2-jrcynj67j41.lightower.net/> 0.0% 28 85.4 88.6 80.1 131.5 12.1 13. 192.154.116.14.lightower.net<http://lightower.net/> 0.0% 28 85.6 83.9 80.2 95.6 3.5 14. ??? John On Nov 7, 2013, at 7:19 AM, Karl Clapp <kclapp@staff.gwi.net<mailto:kclapp@staff.gwi.net>> wrote: LightTower/Sidera master ticker for this is LT1323725. I was able to get through their support line now. The ETR is 1-2 hours from now. --- Karl Clapp Network Engineering Manager Biddeford Internet Corp. d.b.a. GWI (AS5760) Office: 207.602.1144<tel:207.602.1144> Mobile: 207.641.3907<tel:207.641.3907> Fax: 207.286.2061<tel:207.286.2061> On Thu, Nov 7, 2013 at 9:39 AM, <SRonan@eexchange.com<mailto:SRonan@eexchange.com>> wrote: Likely related to the fiber cut they experienced on their network earlier this week which they routed around. Shane Ronan, Vice President – Technology Architect State Street Global Exchange | 1230 Avenue of the Americas, 18th Fl | New York, NY 10020 P (212) 259-3023<tel:%28212%29%20259-3023> | M (347) 413-4503<tel:%28347%29%20413-4503> sronan@eexchange.com<mailto:sronan@eexchange.com> From: Jayson Navitsky <jnavitsky@gmail.com<mailto:jnavitsky@gmail.com>> Date: Thursday, November 7, 2013 at 6:55 AM To: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] Lightower/Sidera - NYC Anyone else seeing any issues in NYC with Lightower/Sidera? Starting around 6:05AM (EST) we saw heavy packet loss across their network (from Sidera legacy) and some of our point to points bounce. The P to P's have some what stabilized (though we've seen a couple more bounces since then) the IP side of things have been pretty much unusable (we shut our session down.) It also looks like their phone system is having a problem - calls into them seem to drop after trying to connect to the NOC - which I'm sure is pretty busy. -Jayson ________________________________ http://www.statestreet.com/emaildisclaimer/ _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages The information in this transmittal may be legally privileged, confidential, and/or otherwise protected by law from disclosure, and is intended only for the recipient(s) listed above. If you are neither the intended recipient(s) nor a person responsible for the delivery of this transmittal to the intended recipient(s), you are hereby notified that any distribution or copying of this transmittal is prohibited. If you have received this transmittal in error, please notify Agio LLC immediately at (405) 217-3800 or by return e-mail and take the steps necessary to delete it completely from your computer system. Thank you.
On Nov 7, 2013, at 4:29 PM, Vincent Aniello wrote:
Our circuits stabilized mid-afternoon, but shortly after 4pm EST we started seeing issues again.
Confirmed here as well, 7 out of 7 L2 tail circuits not passing traffic after a good hour of instability. FWIW, I don't recall any recent maintenance announcements from them, so this is a bit troubling (I'd expect this if we saw a big rollout of new code the night previous) since it came out of the blue. I can also confirm we saw our first outage much earlier than they acknowledged, one circuit lost connectivity at 5:23 a.m. I can also confirm that we did not see the physical layer go down once on any of our circuits, this seems to be occurring in the core, not the rings that actually service the buildings. We're legacy ConEd/RCN/Sidera, have no idea if this is happening with the LT network or not. We got what appears to be a bogus "all restored" email at 2:50 p.m. Charles
--Vincent
From: Karl Clapp <kclapp@staff.gwi.net> To: "outages@outages.org" <outages@outages.org> Date: 11/07/2013 03:23 PM Subject: Re: [outages] Lightower/Sidera - NYC Sent by: "Outages" <outages-bounces@outages.org>
I contacted their NOC about 45min ago. They have stated they are still working on resolution. They have reports about some services being functional, while other still experiencing packet-loss and latency. It does not sound like they have been able to identify the root cause and are still troubleshooting.
Here is the number I used to get through to their NOC: 888-583-4237, option 1. From there, just follow the prompts.
Fwiw, my customer circuit from NYC to NJ is still affected.
On Thu, Nov 7, 2013 at 10:25 AM, John Kinsella <jlk@thrashyour.com> wrote: Just as a data point, LT for me is routing into Equinix NY5 over HE, so at least there's connectivity...
My traceroute [v0.82] icmp (0.0.0.0) Thu Nov 7 07:24:09 2013 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.1.1 0.0% 29 6.7 3.2 0.8 12.2 2.7 2. 10.0.1.49 0.0% 28 8.0 4.3 1.5 13.1 3.0 3. 10.0.24.73 0.0% 28 6.7 4.5 1.9 14.7 3.4 4. 25.66.208.web-pass.com 0.0% 28 1.9 11.2 1.9 48.0 12.3 5. 2-1.760.core.fre.web-pass.com 0.0% 28 9.2 6.4 3.1 19.7 4.0 6. gige-g6-2.core1.fmt1.he.net 0.0% 28 9.6 12.4 3.5 26.4 6.2 7. 10gigabitethernet1-2.core1.sjc2.he.net 0.0% 28 11.9 9.5 3.5 18.7 4.3 8. 10gigabitethernet9-1.core1.nyc4.he.net 0.0% 28 87.9 79.5 73.3 92.7 5.5 9. 10gigabitethernet4-1.core1.nyc5.he.net 0.0% 28 76.6 80.5 73.6 106.7 7.0 10. lightower-fiber-networks.10gigabitether 0.0% 28 78.0 82.3 74.4 108.0 9.4 11. xe-8-3-0.nycmnyzrj91.lightower.net 0.0% 28 89.1 83.6 74.2 144.7 15.8 12. ae2-jrcynj67j41.lightower.net 0.0% 28 85.4 88.6 80.1 131.5 12.1 13. 192.154.116.14.lightower.net 0.0% 28 85.6 83.9 80.2 95.6 3.5 14. ???
John
On Nov 7, 2013, at 7:19 AM, Karl Clapp <kclapp@staff.gwi.net> wrote:
LightTower/Sidera master ticker for this is LT1323725.
I was able to get through their support line now.
The ETR is 1-2 hours from now.
--- Karl Clapp Network Engineering Manager Biddeford Internet Corp. d.b.a. GWI (AS5760) Office: 207.602.1144 Mobile: 207.641.3907 Fax: 207.286.2061
On Thu, Nov 7, 2013 at 9:39 AM, <SRonan@eexchange.com> wrote: Likely related to the fiber cut they experienced on their network earlier this week which they routed around.
Shane Ronan, Vice President – Technology Architect State Street Global Exchange | 1230 Avenue of the Americas, 18th Fl | New York, NY 10020 P (212) 259-3023 | M (347) 413-4503 sronan@eexchange.com
From: Jayson Navitsky <jnavitsky@gmail.com> Date: Thursday, November 7, 2013 at 6:55 AM To: "outages@outages.org" <outages@outages.org> Subject: [outages] Lightower/Sidera - NYC
Anyone else seeing any issues in NYC with Lightower/Sidera? Starting around 6:05AM (EST) we saw heavy packet loss across their network (from Sidera legacy) and some of our point to points bounce. The P to P's have some what stabilized (though we've seen a couple more bounces since then) the IP side of things have been pretty much unusable (we shut our session down.) It also looks like their phone system is having a problem - calls into them seem to drop after trying to connect to the NOC - which I'm sure is pretty busy.
-Jayson
http://www.statestreet.com/emaildisclaimer/
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
As another datapoint, the traditional HV/Datanet/Lightower side of the network has been stable during this reported outage. On Thu, Nov 7, 2013 at 4:45 PM, Charles Sprickman <spork@bway.net> wrote:
On Nov 7, 2013, at 4:29 PM, Vincent Aniello wrote:
Our circuits stabilized mid-afternoon, but shortly after 4pm EST we started seeing issues again.
Confirmed here as well, 7 out of 7 L2 tail circuits not passing traffic after a good hour of instability.
FWIW, I don't recall any recent maintenance announcements from them, so this is a bit troubling (I'd expect this if we saw a big rollout of new code the night previous) since it came out of the blue. I can also confirm we saw our first outage much earlier than they acknowledged, one circuit lost connectivity at 5:23 a.m. I can also confirm that we did not see the physical layer go down once on any of our circuits, this seems to be occurring in the core, not the rings that actually service the buildings.
We're legacy ConEd/RCN/Sidera, have no idea if this is happening with the LT network or not.
We got what appears to be a bogus "all restored" email at 2:50 p.m.
Charles
--Vincent
From: Karl Clapp <kclapp@staff.gwi.net> To: "outages@outages.org" <outages@outages.org> Date: 11/07/2013 03:23 PM Subject: Re: [outages] Lightower/Sidera - NYC Sent by: "Outages" <outages-bounces@outages.org> ------------------------------
I contacted their NOC about 45min ago. They have stated they are still working on resolution. They have reports about some services being functional, while other still experiencing packet-loss and latency. It does not sound like they have been able to identify the root cause and are still troubleshooting.
Here is the number I used to get through to their NOC: 888-583-4237, option 1. From there, just follow the prompts.
Fwiw, my customer circuit from NYC to NJ is still affected.
On Thu, Nov 7, 2013 at 10:25 AM, John Kinsella <*jlk@thrashyour.com*<jlk@thrashyour.com>> wrote: Just as a data point, LT for me is routing into Equinix NY5 over HE, so at least there's connectivity...
My traceroute [v0.82] icmp (0.0.0.0) Thu Nov 7 07:24:09 2013 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.1.1 0.0% 29 6.7 3.2 0.8 12.2 2.7 2. 10.0.1.49 0.0% 28 8.0 4.3 1.5 13.1 3.0 3. 10.0.24.73 0.0% 28 6.7 4.5 1.9 14.7 3.4 4. 25.66.208.*web-pass.com* <http://web-pass.com/> 0.0% 28 1.9 11.2 1.9 48.0 12.3 5. 2-1.760.*core.fre.web-pass.com* <http://core.fre.web-pass.com/> 0.0% 28 9.2 6.4 3.1 19.7 4.0 6. gige-g6-2.*core1.fmt1.he.net* <http://core1.fmt1.he.net/> 0.0% 28 9.6 12.4 3.5 26.4 6.2 7. 10gigabitethernet1-2.*core1.sjc2.he.net* <http://core1.sjc2.he.net/> 0.0% 28 11.9 9.5 3.5 18.7 4.3 8. 10gigabitethernet9-1.*core1.nyc4.he.net* <http://core1.nyc4.he.net/> 0.0% 28 87.9 79.5 73.3 92.7 5.5 9. 10gigabitethernet4-1.*core1.nyc5.he.net* <http://core1.nyc5.he.net/> 0.0% 28 76.6 80.5 73.6 106.7 7.0 10. lightower-fiber-networks.10gigabitether 0.0% 28 78.0 82.3 74.4 108.0 9.4 11. *xe-8-3-0.nycmnyzrj91.lightower.net*<http://xe-8-3-0.nycmnyzrj91.lightower.net/> 0.0% 28 89.1 83.6 74.2 144.7 15.8 12. *ae2-jrcynj67j41.lightower.net*<http://ae2-jrcynj67j41.lightower.net/> 0.0% 28 85.4 88.6 80.1 131.5 12.1 13. 192.154.116.14.*lightower.net* <http://lightower.net/> 0.0% 28 85.6 83.9 80.2 95.6 3.5 14. ???
John
On Nov 7, 2013, at 7:19 AM, Karl Clapp <*kclapp@staff.gwi.net*<kclapp@staff.gwi.net>> wrote:
LightTower/Sidera master ticker for this is LT1323725.
I was able to get through their support line now.
The ETR is 1-2 hours from now.
--- Karl Clapp Network Engineering Manager Biddeford Internet Corp. d.b.a. GWI (AS5760) Office: *207.602.1144* <207.602.1144> Mobile: *207.641.3907* <207.641.3907> Fax: *207.286.2061* <207.286.2061>
On Thu, Nov 7, 2013 at 9:39 AM, <*SRonan@eexchange.com*<SRonan@eexchange.com>> wrote: Likely related to the fiber cut they experienced on their network earlier this week which they routed around.
Shane Ronan, Vice President – Technology Architect State Street Global Exchange | 1230 Avenue of the Americas, 18th Fl | New York, NY 10020 P *(212) 259-3023* <%28212%29%20259-3023> | M *(347) 413-4503*<%28347%29%20413-4503> *sronan@eexchange.com* <sronan@eexchange.com>
*From: *Jayson Navitsky <*jnavitsky@gmail.com* <jnavitsky@gmail.com>> * Date: *Thursday, November 7, 2013 at 6:55 AM * To: *"*outages@outages.org* <outages@outages.org>" < *outages@outages.org* <outages@outages.org>> * Subject: *[outages] Lightower/Sidera - NYC
Anyone else seeing any issues in NYC with Lightower/Sidera? Starting around 6:05AM (EST) we saw heavy packet loss across their network (from Sidera legacy) and some of our point to points bounce. The P to P's have some what stabilized (though we've seen a couple more bounces since then) the IP side of things have been pretty much unusable (we shut our session down.) It also looks like their phone system is having a problem - calls into them seem to drop after trying to connect to the NOC - which I'm sure is pretty busy.
-Jayson
------------------------------
*http://www.statestreet.com/emaildisclaimer/*<http://www.statestreet.com/emaildisclaimer/>
_______________________________________________ Outages mailing list *Outages@outages.org* <Outages@outages.org> *https://puck.nether.net/mailman/listinfo/outages*<https://puck.nether.net/mailman/listinfo/outages>
_______________________________________________ Outages mailing list *Outages@outages.org* <Outages@outages.org> *https://puck.nether.net/mailman/listinfo/outages*<https://puck.nether.net/mailman/listinfo/outages>
_______________________________________________ Outages mailing list *Outages@outages.org* <Outages@outages.org> *https://puck.nether.net/mailman/listinfo/outages*<https://puck.nether.net/mailman/listinfo/outages>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Tim:>
This outage is still on-going. I spoke with an executive at LT earlier today and the word then was that some downstream customer of the Sidera/ConEd/RCN legacy network is leaking PVST BPDUs and its wreaking havoc on their network. I'm not sure how legitimate that is anymore since its been several hours. Apparently though, since our circuits are fluctuating up and down every few minutes, it would seem they're not very close to resolution. We're approaching the 12 hour mark for certain circuits. They also don't seem to have any tools to notify/communicate with their customers about the on-going efforts..and they don't tweet anything but marketing crap. The management of this incident is asinine. -James On Thu, Nov 7, 2013 at 4:54 PM, Tim Durack <tdurack@gmail.com> wrote:
As another datapoint, the traditional HV/Datanet/Lightower side of the network has been stable during this reported outage.
On Thu, Nov 7, 2013 at 4:45 PM, Charles Sprickman <spork@bway.net> wrote:
On Nov 7, 2013, at 4:29 PM, Vincent Aniello wrote:
Our circuits stabilized mid-afternoon, but shortly after 4pm EST we started seeing issues again.
Confirmed here as well, 7 out of 7 L2 tail circuits not passing traffic after a good hour of instability.
FWIW, I don't recall any recent maintenance announcements from them, so this is a bit troubling (I'd expect this if we saw a big rollout of new code the night previous) since it came out of the blue. I can also confirm we saw our first outage much earlier than they acknowledged, one circuit lost connectivity at 5:23 a.m. I can also confirm that we did not see the physical layer go down once on any of our circuits, this seems to be occurring in the core, not the rings that actually service the buildings.
We're legacy ConEd/RCN/Sidera, have no idea if this is happening with the LT network or not.
We got what appears to be a bogus "all restored" email at 2:50 p.m.
Charles
--Vincent
From: Karl Clapp <kclapp@staff.gwi.net> To: "outages@outages.org" <outages@outages.org> Date: 11/07/2013 03:23 PM Subject: Re: [outages] Lightower/Sidera - NYC Sent by: "Outages" <outages-bounces@outages.org> ________________________________
I contacted their NOC about 45min ago. They have stated they are still working on resolution. They have reports about some services being functional, while other still experiencing packet-loss and latency. It does not sound like they have been able to identify the root cause and are still troubleshooting.
Here is the number I used to get through to their NOC: 888-583-4237, option 1. From there, just follow the prompts.
Fwiw, my customer circuit from NYC to NJ is still affected.
On Thu, Nov 7, 2013 at 10:25 AM, John Kinsella <jlk@thrashyour.com> wrote: Just as a data point, LT for me is routing into Equinix NY5 over HE, so at least there's connectivity...
My traceroute [v0.82] icmp (0.0.0.0) Thu Nov 7 07:24:09 2013 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.1.1 0.0% 29 6.7 3.2 0.8 12.2 2.7 2. 10.0.1.49 0.0% 28 8.0 4.3 1.5 13.1 3.0 3. 10.0.24.73 0.0% 28 6.7 4.5 1.9 14.7 3.4 4. 25.66.208.web-pass.com 0.0% 28 1.9 11.2 1.9 48.0 12.3 5. 2-1.760.core.fre.web-pass.com 0.0% 28 9.2 6.4 3.1 19.7 4.0 6. gige-g6-2.core1.fmt1.he.net 0.0% 28 9.6 12.4 3.5 26.4 6.2 7. 10gigabitethernet1-2.core1.sjc2.he.net 0.0% 28 11.9 9.5 3.5 18.7 4.3 8. 10gigabitethernet9-1.core1.nyc4.he.net 0.0% 28 87.9 79.5 73.3 92.7 5.5 9. 10gigabitethernet4-1.core1.nyc5.he.net 0.0% 28 76.6 80.5 73.6 106.7 7.0 10. lightower-fiber-networks.10gigabitether 0.0% 28 78.0 82.3 74.4 108.0 9.4 11. xe-8-3-0.nycmnyzrj91.lightower.net 0.0% 28 89.1 83.6 74.2 144.7 15.8 12. ae2-jrcynj67j41.lightower.net 0.0% 28 85.4 88.6 80.1 131.5 12.1 13. 192.154.116.14.lightower.net 0.0% 28 85.6 83.9 80.2 95.6 3.5 14. ???
John
On Nov 7, 2013, at 7:19 AM, Karl Clapp <kclapp@staff.gwi.net> wrote:
LightTower/Sidera master ticker for this is LT1323725.
I was able to get through their support line now.
The ETR is 1-2 hours from now.
--- Karl Clapp Network Engineering Manager Biddeford Internet Corp. d.b.a. GWI (AS5760) Office: 207.602.1144 Mobile: 207.641.3907 Fax: 207.286.2061
On Thu, Nov 7, 2013 at 9:39 AM, <SRonan@eexchange.com> wrote: Likely related to the fiber cut they experienced on their network earlier this week which they routed around.
Shane Ronan, Vice President – Technology Architect State Street Global Exchange | 1230 Avenue of the Americas, 18th Fl | New York, NY 10020 P (212) 259-3023 | M (347) 413-4503 sronan@eexchange.com
From: Jayson Navitsky <jnavitsky@gmail.com> Date: Thursday, November 7, 2013 at 6:55 AM To: "outages@outages.org" <outages@outages.org> Subject: [outages] Lightower/Sidera - NYC
Anyone else seeing any issues in NYC with Lightower/Sidera? Starting around 6:05AM (EST) we saw heavy packet loss across their network (from Sidera legacy) and some of our point to points bounce. The P to P's have some what stabilized (though we've seen a couple more bounces since then) the IP side of things have been pretty much unusable (we shut our session down.) It also looks like their phone system is having a problem - calls into them seem to drop after trying to connect to the NOC - which I'm sure is pretty busy.
-Jayson
________________________________
http://www.statestreet.com/emaildisclaimer/
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Tim:>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- James Cornman Chief Technology Officer jcornman@atlanticmetro.net 212.792.9950 - ext 101 Atlantic Metro Communications, Inc. 4 Century Drive, Parsippany NJ 07054 Data Centers • Managed Hosting • Internet Access • Cloud Computing Follow @atlanticmetro on Twitter to learn more about daily specials. www.atlanticmetro.net
Thanks James. I was told of some sort of packet storm causing the outage on their MPLS network. Maybe these BPDUs have spanning tree going haywire. Maybe they had to isolate some PEs from the network and all of us are on those PEs? -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of James Cornman Sent: Thursday, November 07, 2013 5:35 PM To: outages@outages.org Subject: Re: [outages] Lightower/Sidera - NYC This outage is still on-going. I spoke with an executive at LT earlier today and the word then was that some downstream customer of the Sidera/ConEd/RCN legacy network is leaking PVST BPDUs and its wreaking havoc on their network. I'm not sure how legitimate that is anymore since its been several hours. Apparently though, since our circuits are fluctuating up and down every few minutes, it would seem they're not very close to resolution. We're approaching the 12 hour mark for certain circuits. They also don't seem to have any tools to notify/communicate with their customers about the on-going efforts..and they don't tweet anything but marketing crap. The management of this incident is asinine. -James On Thu, Nov 7, 2013 at 4:54 PM, Tim Durack <tdurack@gmail.com> wrote:
As another datapoint, the traditional HV/Datanet/Lightower side of the network has been stable during this reported outage.
On Thu, Nov 7, 2013 at 4:45 PM, Charles Sprickman <spork@bway.net> wrote:
On Nov 7, 2013, at 4:29 PM, Vincent Aniello wrote:
Our circuits stabilized mid-afternoon, but shortly after 4pm EST we started seeing issues again.
Confirmed here as well, 7 out of 7 L2 tail circuits not passing traffic after a good hour of instability.
FWIW, I don't recall any recent maintenance announcements from them, so this is a bit troubling (I'd expect this if we saw a big rollout of new code the night previous) since it came out of the blue. I can also confirm we saw our first outage much earlier than they acknowledged, one circuit lost connectivity at 5:23 a.m. I can also confirm that we did not see the physical layer go down once on any of our circuits, this seems to be occurring in the core, not the rings that actually service the buildings.
We're legacy ConEd/RCN/Sidera, have no idea if this is happening with the LT network or not.
We got what appears to be a bogus "all restored" email at 2:50 p.m.
Charles
--Vincent
From: Karl Clapp <kclapp@staff.gwi.net> To: "outages@outages.org" <outages@outages.org> Date: 11/07/2013 03:23 PM Subject: Re: [outages] Lightower/Sidera - NYC Sent by: "Outages" <outages-bounces@outages.org> ________________________________
I contacted their NOC about 45min ago. They have stated they are still working on resolution. They have reports about some services being functional, while other still experiencing packet-loss and latency. It does not sound like they have been able to identify the root cause and are still troubleshooting.
Here is the number I used to get through to their NOC: 888-583-4237, option 1. From there, just follow the prompts.
Fwiw, my customer circuit from NYC to NJ is still affected.
On Thu, Nov 7, 2013 at 10:25 AM, John Kinsella <jlk@thrashyour.com> wrote: Just as a data point, LT for me is routing into Equinix NY5 over HE, so at least there's connectivity...
My traceroute [v0.82] icmp (0.0.0.0) Thu Nov 7 07:24:09 2013 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.1.1 0.0% 29 6.7 3.2 0.8 12.2 2.7 2. 10.0.1.49 0.0% 28 8.0 4.3 1.5 13.1 3.0 3. 10.0.24.73 0.0% 28 6.7 4.5 1.9 14.7 3.4 4. 25.66.208.web-pass.com 0.0% 28 1.9 11.2 1.9 48.0 12.3 5. 2-1.760.core.fre.web-pass.com 0.0% 28 9.2 6.4 3.1 19.7 4.0 6. gige-g6-2.core1.fmt1.he.net 0.0% 28 9.6 12.4 3.5 26.4 6.2 7. 10gigabitethernet1-2.core1.sjc2.he.net 0.0% 28 11.9 9.5 3.5 18.7 4.3 8. 10gigabitethernet9-1.core1.nyc4.he.net 0.0% 28 87.9 79.5 73.3 92.7 5.5 9. 10gigabitethernet4-1.core1.nyc5.he.net 0.0% 28 76.6 80.5 73.6 106.7 7.0 10. lightower-fiber-networks.10gigabitether 0.0% 28 78.0 82.3 74.4 108.0 9.4 11. xe-8-3-0.nycmnyzrj91.lightower.net 0.0% 28 89.1 83.6 74.2 144.7 15.8 12. ae2-jrcynj67j41.lightower.net 0.0% 28 85.4 88.6 80.1 131.5 12.1 13. 192.154.116.14.lightower.net 0.0% 28 85.6 83.9 80.2 95.6 3.5 14. ???
John
On Nov 7, 2013, at 7:19 AM, Karl Clapp <kclapp@staff.gwi.net> wrote:
LightTower/Sidera master ticker for this is LT1323725.
I was able to get through their support line now.
The ETR is 1-2 hours from now.
--- Karl Clapp Network Engineering Manager Biddeford Internet Corp. d.b.a. GWI (AS5760) Office: 207.602.1144 Mobile: 207.641.3907 Fax: 207.286.2061
On Thu, Nov 7, 2013 at 9:39 AM, <SRonan@eexchange.com> wrote: Likely related to the fiber cut they experienced on their network earlier this week which they routed around.
Shane Ronan, Vice President - Technology Architect State Street Global Exchange | 1230 Avenue of the Americas, 18th Fl | New York, NY 10020 P (212) 259-3023 | M (347) 413-4503 sronan@eexchange.com
From: Jayson Navitsky <jnavitsky@gmail.com> Date: Thursday, November 7, 2013 at 6:55 AM To: "outages@outages.org" <outages@outages.org> Subject: [outages] Lightower/Sidera - NYC
Anyone else seeing any issues in NYC with Lightower/Sidera? Starting around 6:05AM (EST) we saw heavy packet loss across their network (from Sidera legacy) and some of our point to points bounce. The P to P's have some what stabilized (though we've seen a couple more bounces since then) the IP side of things have been pretty much unusable (we shut our session down.) It also looks like their phone system is having a problem - calls into them seem to drop after trying to connect to the NOC - which I'm sure is pretty busy.
-Jayson
________________________________
http://www.statestreet.com/emaildisclaimer/
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Tim:>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- James Cornman Chief Technology Officer jcornman@atlanticmetro.net 212.792.9950 - ext 101 Atlantic Metro Communications, Inc. 4 Century Drive, Parsippany NJ 07054 Data Centers * Managed Hosting * Internet Access * Cloud Computing Follow @atlanticmetro on Twitter to learn more about daily specials. www.atlanticmetro.net _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Looks like it stopped OTC market trading for a bit. http://www.cnbc.com/id/101181346 On Thu, Nov 7, 2013 at 2:54 PM, <michael.mccoy@bt.com> wrote:
Thanks James. I was told of some sort of packet storm causing the outage on their MPLS network. Maybe these BPDUs have spanning tree going haywire. Maybe they had to isolate some PEs from the network and all of us are on those PEs?
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of James Cornman Sent: Thursday, November 07, 2013 5:35 PM To: outages@outages.org Subject: Re: [outages] Lightower/Sidera - NYC
This outage is still on-going. I spoke with an executive at LT earlier today and the word then was that some downstream customer of the Sidera/ConEd/RCN legacy network is leaking PVST BPDUs and its wreaking havoc on their network. I'm not sure how legitimate that is anymore since its been several hours. Apparently though, since our circuits are fluctuating up and down every few minutes, it would seem they're not very close to resolution. We're approaching the 12 hour mark for certain circuits.
They also don't seem to have any tools to notify/communicate with their customers about the on-going efforts..and they don't tweet anything but marketing crap. The management of this incident is asinine.
-James
On Thu, Nov 7, 2013 at 4:54 PM, Tim Durack <tdurack@gmail.com> wrote:
As another datapoint, the traditional HV/Datanet/Lightower side of the network has been stable during this reported outage.
On Thu, Nov 7, 2013 at 4:45 PM, Charles Sprickman <spork@bway.net> wrote:
On Nov 7, 2013, at 4:29 PM, Vincent Aniello wrote:
Our circuits stabilized mid-afternoon, but shortly after 4pm EST we started seeing issues again.
Confirmed here as well, 7 out of 7 L2 tail circuits not passing traffic after a good hour of instability.
FWIW, I don't recall any recent maintenance announcements from them, so this is a bit troubling (I'd expect this if we saw a big rollout of new code the night previous) since it came out of the blue. I can also confirm we saw our first outage much earlier than they acknowledged, one circuit lost connectivity at 5:23 a.m. I can also confirm that we did not see the physical layer go down once on any of our circuits, this seems to be occurring in the core, not the rings
that actually service the buildings.
We're legacy ConEd/RCN/Sidera, have no idea if this is happening with the LT network or not.
We got what appears to be a bogus "all restored" email at 2:50 p.m.
Charles
--Vincent
From: Karl Clapp <kclapp@staff.gwi.net> To: "outages@outages.org" <outages@outages.org> Date: 11/07/2013 03:23 PM Subject: Re: [outages] Lightower/Sidera - NYC Sent by: "Outages" <outages-bounces@outages.org> ________________________________
I contacted their NOC about 45min ago. They have stated they are still working on resolution. They have reports about some services being functional, while other still experiencing packet-loss and latency. It does not sound like they have been able to identify the root cause and are still troubleshooting.
Here is the number I used to get through to their NOC: 888-583-4237, option 1. From there, just follow the prompts.
Fwiw, my customer circuit from NYC to NJ is still affected.
On Thu, Nov 7, 2013 at 10:25 AM, John Kinsella <jlk@thrashyour.com>
wrote:
Just as a data point, LT for me is routing into Equinix NY5 over HE, so at least there's connectivity...
My traceroute [v0.82] icmp (0.0.0.0) Thu Nov 7 07:24:09 2013 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.1.1 0.0% 29 6.7 3.2 0.8 12.2 2.7 2. 10.0.1.49 0.0% 28 8.0 4.3 1.5 13.1 3.0 3. 10.0.24.73 0.0% 28 6.7 4.5 1.9 14.7 3.4 4. 25.66.208.web-pass.com 0.0% 28 1.9 11.2 1.9 48.0 12.3 5. 2-1.760.core.fre.web-pass.com 0.0% 28 9.2 6.4 3.1 19.7 4.0 6. gige-g6-2.core1.fmt1.he.net 0.0% 28 9.6 12.4 3.5 26.4 6.2 7. 10gigabitethernet1-2.core1.sjc2.he.net 0.0% 28 11.9 9.5 3.5 18.7 4.3 8. 10gigabitethernet9-1.core1.nyc4.he.net 0.0% 28 87.9 79.5 73.3 92.7 5.5 9. 10gigabitethernet4-1.core1.nyc5.he.net 0.0% 28 76.6 80.5 73.6 106.7 7.0 10. lightower-fiber-networks.10gigabitether 0.0% 28 78.0 82.3 74.4 108.0 9.4 11. xe-8-3-0.nycmnyzrj91.lightower.net 0.0% 28 89.1 83.6 74.2 144.7 15.8 12. ae2-jrcynj67j41.lightower.net 0.0% 28 85.4 88.6 80.1 131.5 12.1 13. 192.154.116.14.lightower.net 0.0% 28 85.6 83.9 80.2 95.6 3.5 14. ???
John
On Nov 7, 2013, at 7:19 AM, Karl Clapp <kclapp@staff.gwi.net> wrote:
LightTower/Sidera master ticker for this is LT1323725.
I was able to get through their support line now.
The ETR is 1-2 hours from now.
--- Karl Clapp Network Engineering Manager Biddeford Internet Corp. d.b.a. GWI (AS5760) Office: 207.602.1144 Mobile: 207.641.3907 Fax: 207.286.2061
On Thu, Nov 7, 2013 at 9:39 AM, <SRonan@eexchange.com> wrote: Likely related to the fiber cut they experienced on their network earlier this week which they routed around.
Shane Ronan, Vice President - Technology Architect State Street Global Exchange | 1230 Avenue of the Americas, 18th Fl | New York, NY 10020 P (212) 259-3023 | M (347) 413-4503 sronan@eexchange.com
From: Jayson Navitsky <jnavitsky@gmail.com> Date: Thursday, November 7, 2013 at 6:55 AM To: "outages@outages.org" <outages@outages.org> Subject: [outages] Lightower/Sidera - NYC
Anyone else seeing any issues in NYC with Lightower/Sidera? Starting around 6:05AM (EST) we saw heavy packet loss across their network (from Sidera legacy) and some of our point to points bounce. The P to P's have some what stabilized (though we've seen a couple more bounces since then) the IP side of things have been pretty much unusable (we shut our session down.) It also looks like their phone system is having a problem - calls into them seem to drop after trying to connect to the NOC - which I'm sure is pretty busy.
-Jayson
________________________________
http://www.statestreet.com/emaildisclaimer/
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Tim:>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- James Cornman
Chief Technology Officer jcornman@atlanticmetro.net 212.792.9950 - ext 101
Atlantic Metro Communications, Inc.
4 Century Drive, Parsippany NJ 07054
Data Centers * Managed Hosting * Internet Access * Cloud Computing Follow @atlanticmetro on Twitter to learn more about daily specials. www.atlanticmetro.net
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (13)
-
Bunde, Andrew -
Charles Sprickman -
Gaurav Taparia -
James Cornman -
Jared Geiger -
Jayson Navitsky -
John Kinsella -
Karl Clapp -
michael.mccoy＠bt.com -
SRonan＠eexchange.com -
Tillinger, Steve -
Tim Durack -
Vincent Aniello