We have some Fiberlight circuits impacted as of about 9:45 AM Eastern. Still down. Here's what they said: FiberLight NOC detected multiple alarms between devices in the greater WDC area, including links down between Vienna VA and our WDC datacenters.*Current Status:* FiberLight technician was dispatched to Vienna VA to isolate the issue. Initial readings put the fiber break less than 1km from that location. Troubleshooting is ongoing. *Preliminary RFO:* Fiber cut near 7990 Science Applications Ct, Vienna VA
We also have some Wave (RCN) circuits impacted by this cut. A little more info: As of ~7pm Eastern: Damage traced to construction at Gallows Rd and Saic Ct. Damage is due to construction boring and is 10 feet deep. Crews on site. No ETR at this time. Then at ~9pm Eastern (this one we got over the phone from Wave who got it from Fiberlight): they are "routing a spare conduit to check for damages" and continuing to do escavation [sic], no ETR, 432-count fiber Nothing since. Anyone have a picture of the damage? Or live close enough to go take one? On Tue, Oct 20, 2020 at 6:36 PM Ross Tajvar <ross@tajvar.io> wrote:
We have some Fiberlight circuits impacted as of about 9:45 AM Eastern. Still down. Here's what they said:
FiberLight NOC detected multiple alarms between devices in the greater WDC area, including links down between Vienna VA and our WDC datacenters.*Current Status:* FiberLight technician was dispatched to Vienna VA to isolate the issue. Initial readings put the fiber break less than 1km from that location. Troubleshooting is ongoing. *Preliminary RFO:* Fiber cut near 7990 Science Applications Ct, Vienna VA
Sorry, we got an update that I missed around 10pm Eastern: Safety Shoring is in, and excavation is underway. That's referring to this previous update: Trench shoring taking place due to the depth of the damage (10 foot deep trench). Damaged fiber is 432ct. So, looks like it's still going to be a while. On Tue, Oct 20, 2020 at 10:47 PM Ross Tajvar <ross@tajvar.io> wrote:
We also have some Wave (RCN) circuits impacted by this cut. A little more info:
As of ~7pm Eastern:
Damage traced to construction at Gallows Rd and Saic Ct. Damage is due to construction boring and is 10 feet deep. Crews on site. No ETR at this time.
Then at ~9pm Eastern (this one we got over the phone from Wave who got it from Fiberlight): they are "routing a spare conduit to check for damages" and continuing to do escavation [sic], no ETR, 432-count fiber
Nothing since. Anyone have a picture of the damage? Or live close enough to go take one?
On Tue, Oct 20, 2020 at 6:36 PM Ross Tajvar <ross@tajvar.io> wrote:
We have some Fiberlight circuits impacted as of about 9:45 AM Eastern. Still down. Here's what they said:
FiberLight NOC detected multiple alarms between devices in the greater WDC area, including links down between Vienna VA and our WDC datacenters.*Current Status:* FiberLight technician was dispatched to Vienna VA to isolate the issue. Initial readings put the fiber break less than 1km from that location. Troubleshooting is ongoing. *Preliminary RFO:* Fiber cut near 7990 Science Applications Ct, Vienna VA
The North American Fiber-Seeking Backhoe strikes again. We need to start an annual culling of that herd. -- Christopher Conley Systems Administrator | Fors Marsh Group 1010 N. Glebe Rd, Suite 510 Arlington, VA 22201 ________________________________ From: Ross Tajvar via Outages <outages@outages.org> Sent: Tuesday, October 20, 2020 10:54 PM To: OUTAGES Mailing List Subject: Re: [outages] Fiber cut in NoVA We also have some Wave (RCN) circuits impacted by this cut. A little more info: As of ~7pm Eastern: Damage traced to construction at Gallows Rd and Saic Ct. Damage is due to construction boring and is 10 feet deep. Crews on site. No ETR at this time. Then at ~9pm Eastern (this one we got over the phone from Wave who got it from Fiberlight): they are "routing a spare conduit to check for damages" and continuing to do escavation [sic], no ETR, 432-count fiber Nothing since. Anyone have a picture of the damage? Or live close enough to go take one? On Tue, Oct 20, 2020 at 6:36 PM Ross Tajvar <ross@tajvar.io<mailto:ross@tajvar.io>> wrote: We have some Fiberlight circuits impacted as of about 9:45 AM Eastern. Still down. Here's what they said: FiberLight NOC detected multiple alarms between devices in the greater WDC area, including links down between Vienna VA and our WDC datacenters.Current Status: FiberLight technician was dispatched to Vienna VA to isolate the issue. Initial readings put the fiber break less than 1km from that location. Troubleshooting is ongoing. Preliminary RFO: Fiber cut near 7990 Science Applications Ct, Vienna VA
Its was a duct bank seeking directional drill actually... :) On Tue, Oct 20, 2020 at 11:00 PM Christopher Conley via Outages < outages@outages.org> wrote:
The North American Fiber-Seeking Backhoe strikes again.
We need to start an annual culling of that herd.
-- Christopher Conley Systems Administrator | Fors Marsh Group 1010 N. Glebe Rd, Suite 510 Arlington, VA 22201 ------------------------------ *From:* Ross Tajvar via Outages <outages@outages.org> *Sent:* Tuesday, October 20, 2020 10:54 PM *To:* OUTAGES Mailing List *Subject:* Re: [outages] Fiber cut in NoVA
We also have some Wave (RCN) circuits impacted by this cut. A little more info:
As of ~7pm Eastern:
Damage traced to construction at Gallows Rd and Saic Ct. Damage is due to construction boring and is 10 feet deep. Crews on site. No ETR at this time.
Then at ~9pm Eastern (this one we got over the phone from Wave who got it from Fiberlight): they are "routing a spare conduit to check for damages" and continuing to do escavation [sic], no ETR, 432-count fiber
Nothing since. Anyone have a picture of the damage? Or live close enough to go take one?
On Tue, Oct 20, 2020 at 6:36 PM Ross Tajvar <ross@tajvar.io> wrote:
We have some Fiberlight circuits impacted as of about 9:45 AM Eastern. Still down. Here's what they said:
FiberLight NOC detected multiple alarms between devices in the greater WDC area, including links down between Vienna VA and our WDC datacenters.*Current Status:* FiberLight technician was dispatched to Vienna VA to isolate the issue. Initial readings put the fiber break less than 1km from that location. Troubleshooting is ongoing. *Preliminary RFO:* Fiber cut near 7990 Science Applications Ct, Vienna VA
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- [image: photo] *Stephen Klenert, CSMP* Chief Strategy Officer, Atlantic Metro 212.792.9950 x 113 | 609.510.3665 | steve@atlanticmetro.net | www.atlanticmetro.net | 300 Blvd East Weehawken, NJ 07086 <https://www.facebook.com/atlanticmetro/> <http://linkedin.com/in/getklenert> <https://twitter.com/atlanticmetro> *COVID-19 Response -* https://www.atlanticmetro.net/covid-19-response/
12am Eastern: Crews are having to dig 10 feet in both directions along fiber path due to more damage being identified. No ETR available. 1am Eastern: Crews have repaired half the duct, and have 1 of 2 cables placed to the first splice point. Splicers are preparing cable while construction crews continue to repair the other half of the duct. No ETR available. 2am Eastern: Crews have the second section of duct while continuing to prepare for splicing. No ETR available. On Tue, Oct 20, 2020, 11:03 PM Stephen Klenert <sklenert@atlanticmetro.net> wrote:
Its was a duct bank seeking directional drill actually... :)
On Tue, Oct 20, 2020 at 11:00 PM Christopher Conley via Outages < outages@outages.org> wrote:
The North American Fiber-Seeking Backhoe strikes again.
We need to start an annual culling of that herd.
-- Christopher Conley Systems Administrator | Fors Marsh Group 1010 N. Glebe Rd, Suite 510 Arlington, VA 22201 ------------------------------ *From:* Ross Tajvar via Outages <outages@outages.org> *Sent:* Tuesday, October 20, 2020 10:54 PM *To:* OUTAGES Mailing List *Subject:* Re: [outages] Fiber cut in NoVA
We also have some Wave (RCN) circuits impacted by this cut. A little more info:
As of ~7pm Eastern:
Damage traced to construction at Gallows Rd and Saic Ct. Damage is due to construction boring and is 10 feet deep. Crews on site. No ETR at this time.
Then at ~9pm Eastern (this one we got over the phone from Wave who got it from Fiberlight): they are "routing a spare conduit to check for damages" and continuing to do escavation [sic], no ETR, 432-count fiber
Nothing since. Anyone have a picture of the damage? Or live close enough to go take one?
On Tue, Oct 20, 2020 at 6:36 PM Ross Tajvar <ross@tajvar.io> wrote:
We have some Fiberlight circuits impacted as of about 9:45 AM Eastern. Still down. Here's what they said:
FiberLight NOC detected multiple alarms between devices in the greater WDC area, including links down between Vienna VA and our WDC datacenters.*Current Status:* FiberLight technician was dispatched to Vienna VA to isolate the issue. Initial readings put the fiber break less than 1km from that location. Troubleshooting is ongoing. *Preliminary RFO:* Fiber cut near 7990 Science Applications Ct, Vienna VA
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
--
[image: photo] *Stephen Klenert, CSMP* Chief Strategy Officer, Atlantic Metro 212.792.9950 x 113 | 609.510.3665 | steve@atlanticmetro.net | www.atlanticmetro.net | 300 Blvd East Weehawken, NJ 07086 <https://www.facebook.com/atlanticmetro/> <http://linkedin.com/in/getklenert> <https://twitter.com/atlanticmetro> *COVID-19 Response -* https://www.atlanticmetro.net/covid-19-response/
9am Eastern: Crews have spliced the first 216 of the first 432. Working to pull 2nd 432. Some alarms have started to clear. No ETR available On Wed, Oct 21, 2020 at 2:32 AM Ross Tajvar <ross@tajvar.io> wrote:
12am Eastern: Crews are having to dig 10 feet in both directions along fiber path due to more damage being identified. No ETR available.
1am Eastern: Crews have repaired half the duct, and have 1 of 2 cables placed to the first splice point. Splicers are preparing cable while construction crews continue to repair the other half of the duct. No ETR available.
2am Eastern: Crews have the second section of duct while continuing to prepare for splicing. No ETR available.
On Tue, Oct 20, 2020, 11:03 PM Stephen Klenert <sklenert@atlanticmetro.net> wrote:
Its was a duct bank seeking directional drill actually... :)
On Tue, Oct 20, 2020 at 11:00 PM Christopher Conley via Outages < outages@outages.org> wrote:
The North American Fiber-Seeking Backhoe strikes again.
We need to start an annual culling of that herd.
-- Christopher Conley Systems Administrator | Fors Marsh Group 1010 N. Glebe Rd, Suite 510 Arlington, VA 22201 ------------------------------ *From:* Ross Tajvar via Outages <outages@outages.org> *Sent:* Tuesday, October 20, 2020 10:54 PM *To:* OUTAGES Mailing List *Subject:* Re: [outages] Fiber cut in NoVA
We also have some Wave (RCN) circuits impacted by this cut. A little more info:
As of ~7pm Eastern:
Damage traced to construction at Gallows Rd and Saic Ct. Damage is due to construction boring and is 10 feet deep. Crews on site. No ETR at this time.
Then at ~9pm Eastern (this one we got over the phone from Wave who got it from Fiberlight): they are "routing a spare conduit to check for damages" and continuing to do escavation [sic], no ETR, 432-count fiber
Nothing since. Anyone have a picture of the damage? Or live close enough to go take one?
On Tue, Oct 20, 2020 at 6:36 PM Ross Tajvar <ross@tajvar.io> wrote:
We have some Fiberlight circuits impacted as of about 9:45 AM Eastern. Still down. Here's what they said:
FiberLight NOC detected multiple alarms between devices in the greater WDC area, including links down between Vienna VA and our WDC datacenters.*Current Status:* FiberLight technician was dispatched to Vienna VA to isolate the issue. Initial readings put the fiber break less than 1km from that location. Troubleshooting is ongoing. *Preliminary RFO:* Fiber cut near 7990 Science Applications Ct, Vienna VA
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
--
[image: photo] *Stephen Klenert, CSMP* Chief Strategy Officer, Atlantic Metro 212.792.9950 x 113 | 609.510.3665 | steve@atlanticmetro.net | www.atlanticmetro.net | 300 Blvd East Weehawken, NJ 07086 <https://www.facebook.com/atlanticmetro/> <http://linkedin.com/in/getklenert> <https://twitter.com/atlanticmetro> *COVID-19 Response -* https://www.atlanticmetro.net/covid-19-response/
If you have a better address, I'm about a 10 minute drive from there. Saic Drive does not cross Gallows. Gallows Road ends at Route 7, turns into International Drive. I could not find an Saic Ct. On Tue, 20 Oct 2020, Ross Tajvar via Outages wrote:
We also have some Wave (RCN) circuits impacted by this cut. A little more info:
As of ~7pm Eastern:
Damage traced to construction at Gallows Rd and Saic Ct. Damage is due to construction boring and is 10 feet deep. Crews on site. No ETR at this time.
Then at ~9pm Eastern (this one we got over the phone from Wave who got it from Fiberlight): they are "routing a spare conduit to check for damages" and continuing to do escavation [sic], no ETR, 432-count fiber
Nothing since. Anyone have a picture of the damage? Or live close enough to go take one?
On Tue, Oct 20, 2020 at 6:36 PM Ross Tajvar <ross@tajvar.io> wrote:
We have some Fiberlight circuits impacted as of about 9:45 AM Eastern. Still down. Here's what they said:
FiberLight NOC detected multiple alarms between devices in the greater WDC area, including links down between Vienna VA and our WDC datacenters.*Current Status:* FiberLight technician was dispatched to Vienna VA to isolate the issue. Initial readings put the fiber break less than 1km from that location. Troubleshooting is ongoing. *Preliminary RFO:* Fiber cut near 7990 Science Applications Ct, Vienna VA
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Hmm, that's all I have for an address. They did say it was "near 7990 Science Applications Ct, Vienna VA". I bet if you start there and drive north on Gallows you'll see it. Might be after it turns into International Dr even - I wouldn't be surprised if they got it wrong. On Wed, Oct 21, 2020, 2:09 AM Peter Beckman via Outages <outages@outages.org> wrote:
If you have a better address, I'm about a 10 minute drive from there.
Saic Drive does not cross Gallows.
Gallows Road ends at Route 7, turns into International Drive.
I could not find an Saic Ct.
On Tue, 20 Oct 2020, Ross Tajvar via Outages wrote:
We also have some Wave (RCN) circuits impacted by this cut. A little more info:
As of ~7pm Eastern:
Damage traced to construction at Gallows Rd and Saic Ct. Damage is due to construction boring and is 10 feet deep. Crews on site. No ETR at this time.
Then at ~9pm Eastern (this one we got over the phone from Wave who got it from Fiberlight): they are "routing a spare conduit to check for damages" and continuing to do escavation [sic], no ETR, 432-count fiber
Nothing since. Anyone have a picture of the damage? Or live close enough to go take one?
On Tue, Oct 20, 2020 at 6:36 PM Ross Tajvar <ross@tajvar.io> wrote:
We have some Fiberlight circuits impacted as of about 9:45 AM Eastern. Still down. Here's what they said:
FiberLight NOC detected multiple alarms between devices in the greater WDC area, including links down between Vienna VA and our WDC datacenters.*Current Status:* FiberLight technician was dispatched to Vienna VA to isolate the issue. Initial readings put the fiber break less than 1km from that location. Troubleshooting is ongoing. *Preliminary RFO:* Fiber cut near 7990 Science Applications Ct, Vienna VA
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/
---------------------------------------------------------------------------_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (4)
-
Christopher Conley -
Peter Beckman -
Ross Tajvar -
Stephen Klenert