Hi, I've got confirmed reports (from Cogent and Megapath) that there is a fiber cut affecting service through the South Bay. We have Cogent fiber circuits down in Campbell and San Martin. In San Martin, landlines have no dialtone and AT&T cell phones have no service whatsoever. All calls to Gilroy landlines are failing to All Circuits Busy. Anyone have any further details? The issue seems to have started around 1:00am PDT or so. -Robert
Similar confirmed reports for fiber services between San Jose and San Martin with Zayo and Level(3). I have an ETR of 11AM PDT, splicing in progress. -Rob -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Robert Glover Sent: Tuesday, April 16, 2013 10:37 AM To: outages@outages.org Subject: [outages] Fiber Cut in SF Bay Area Hi, I've got confirmed reports (from Cogent and Megapath) that there is a fiber cut affecting service through the South Bay. We have Cogent fiber circuits down in Campbell and San Martin. In San Martin, landlines have no dialtone and AT&T cell phones have no service whatsoever. All calls to Gilroy landlines are failing to All Circuits Busy. Anyone have any further details? The issue seems to have started around 1:00am PDT or so. -Robert _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Yes, Level(3) did find the Fiber cut for the Sunnyvale, CA area but the cut was in the San Jose area. The fiber cut was spliced and fused earlier, but re-dispatched for a customer's dark fiber strains, but other than that everyone should back in service if affected by this cut. *If you are experiencing a service disruption, please report the issue to Level 3 by submitting a ticket via the MyLevel3 Portal or calling 877-4LEVEL3.* Joe On Tue, Apr 16, 2013 at 12:13 PM, Rob Laird (rolaird) <rolaird@cisco.com>wrote:
Similar confirmed reports for fiber services between San Jose and San Martin with Zayo and Level(3). I have an ETR of 11AM PDT, splicing in progress.
-Rob
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Robert Glover Sent: Tuesday, April 16, 2013 10:37 AM To: outages@outages.org Subject: [outages] Fiber Cut in SF Bay Area
Hi,
I've got confirmed reports (from Cogent and Megapath) that there is a fiber cut affecting service through the South Bay. We have Cogent fiber circuits down in Campbell and San Martin. In San Martin, landlines have no dialtone and AT&T cell phones have no service whatsoever. All calls to Gilroy landlines are failing to All Circuits Busy.
Anyone have any further details? The issue seems to have started around 1:00am PDT or so.
-Robert _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
This reminds me of my favorite Level3 article ever, about fiber cuts. http://blog.level3.com/level-3-network/the-10-most-bizarre-and-annoying-caus... Frankly, I think every fiber cut should be coupled with pictures posted about the fiber cut. Can't help but be curious sometimes! From: Joe Sanchez <marco207p@gmail.com<mailto:marco207p@gmail.com>> Date: Tuesday, April 16, 2013 5:12 PM To: "Rob Laird (rolaird)" <rolaird@cisco.com<mailto:rolaird@cisco.com>> Cc: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: Re: [outages] Fiber Cut in SF Bay Area Yes, Level(3) did find the Fiber cut for the Sunnyvale, CA area but the cut was in the San Jose area. The fiber cut was spliced and fused earlier, but re-dispatched for a customer's dark fiber strains, but other than that everyone should back in service if affected by this cut. If you are experiencing a service disruption, please report the issue to Level 3 by submitting a ticket via the MyLevel3 Portal or calling 877-4LEVEL3. Joe On Tue, Apr 16, 2013 at 12:13 PM, Rob Laird (rolaird) <rolaird@cisco.com<mailto:rolaird@cisco.com>> wrote: Similar confirmed reports for fiber services between San Jose and San Martin with Zayo and Level(3). I have an ETR of 11AM PDT, splicing in progress. -Rob -----Original Message----- From: Outages [mailto:outages-bounces@outages.org<mailto:outages-bounces@outages.org>] On Behalf Of Robert Glover Sent: Tuesday, April 16, 2013 10:37 AM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] Fiber Cut in SF Bay Area Hi, I've got confirmed reports (from Cogent and Megapath) that there is a fiber cut affecting service through the South Bay. We have Cogent fiber circuits down in Campbell and San Martin. In San Martin, landlines have no dialtone and AT&T cell phones have no service whatsoever. All calls to Gilroy landlines are failing to All Circuits Busy. Anyone have any further details? The issue seems to have started around 1:00am PDT or so. -Robert _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Number 4 on that list happens here in my city almost weekly, whether the lines be for telephone, cable, or power. The residential line techs I've talked to indicate the common cause are big rigs driving around in areas where the cables are low-hanging. Of course nobody reports this until there's an actual outage... :/ Much worse, not to mention dangerous, occasionally someone with a bucket truck forgets to lower the arm all the way and proceeds to drive around for a while before wondering what all the noise is about. When this happens, it's like a trail of bread crumbs for the fire department. Owning a scanner (e.g. fire/police/city) comes in handy for this exact reason. Number 5 is also a common problem here too, at least for residential providers. Rain doesn't tend to do much damage (barring in-line power amps that aren't properly closed by techs), but wind is an entirely different problem. Goes hand in hand with Number 8. And finally, we get number 2 too, though my city is blessed with a "special" kind of grey squirrel: psychotic and extremely aggressive (search Google for "Mountain View squirrels" if you think I'm kidding. I've had "personal encounters" with them, the most notable being one that dropped from a tree onto my handlebars while biking through a park; it chattered at me rudely for a full 5-6 seconds *while I was still moving*, before scurrying off somewhere. Maybe this is what the Second Amendment is really for... ;-) ) P.S. -- To all you field techs out there, thank you for all the hard work you do. You deserve a lot more credit than you get. :-) -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Wed, Apr 17, 2013 at 12:29:11AM +0000, Darren Schreiber wrote:
This reminds me of my favorite Level3 article ever, about fiber cuts.
http://blog.level3.com/level-3-network/the-10-most-bizarre-and-annoying-caus...
Frankly, I think every fiber cut should be coupled with pictures posted about the fiber cut. Can't help but be curious sometimes!
From: Joe Sanchez <marco207p@gmail.com<mailto:marco207p@gmail.com>> Date: Tuesday, April 16, 2013 5:12 PM To: "Rob Laird (rolaird)" <rolaird@cisco.com<mailto:rolaird@cisco.com>> Cc: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: Re: [outages] Fiber Cut in SF Bay Area
Yes, Level(3) did find the Fiber cut for the Sunnyvale, CA area but the cut was in the San Jose area. The fiber cut was spliced and fused earlier, but re-dispatched for a customer's dark fiber strains, but other than that everyone should back in service if affected by this cut.
If you are experiencing a service disruption, please report the issue to Level 3 by submitting a ticket via the MyLevel3 Portal or calling 877-4LEVEL3.
Joe
On Tue, Apr 16, 2013 at 12:13 PM, Rob Laird (rolaird) <rolaird@cisco.com<mailto:rolaird@cisco.com>> wrote: Similar confirmed reports for fiber services between San Jose and San Martin with Zayo and Level(3). I have an ETR of 11AM PDT, splicing in progress.
-Rob
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org<mailto:outages-bounces@outages.org>] On Behalf Of Robert Glover Sent: Tuesday, April 16, 2013 10:37 AM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] Fiber Cut in SF Bay Area
Hi,
I've got confirmed reports (from Cogent and Megapath) that there is a fiber cut affecting service through the South Bay. We have Cogent fiber circuits down in Campbell and San Martin. In San Martin, landlines have no dialtone and AT&T cell phones have no service whatsoever. All calls to Gilroy landlines are failing to All Circuits Busy.
Anyone have any further details? The issue seems to have started around 1:00am PDT or so.
-Robert _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Tue, Apr 16, 2013 at 6:17 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Number 4 on that list happens here in my city almost weekly, whether the lines be for telephone, cable, or power.
The residential line techs I've talked to indicate the common cause are big rigs driving around in areas where the cables are low-hanging. Of course nobody reports this until there's an actual outage... :/
This happened to my neighbors about 3 years ago, in the middle of the night. Fortunately only took their phone and cable line out, but we heard the "Bang" as it was pulled off the house and then the truck screeching to a halt... -- -george william herbert george.herbert@gmail.com
Latest from mega path: *UPDATE: *ILEC/Carrier working on Fiber Splicing. ETR is 2:00 AM PST, Next notification will be sent at 00:00 AM PST. On Tue, Apr 16, 2013 at 11:00 PM, George Herbert <george.herbert@gmail.com>wrote:
On Tue, Apr 16, 2013 at 6:17 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Number 4 on that list happens here in my city almost weekly, whether the lines be for telephone, cable, or power.
The residential line techs I've talked to indicate the common cause are big rigs driving around in areas where the cables are low-hanging. Of course nobody reports this until there's an actual outage... :/
This happened to my neighbors about 3 years ago, in the middle of the night. Fortunately only took their phone and cable line out, but we heard the "Bang" as it was pulled off the house and then the truck screeching to a halt...
-- -george william herbert george.herbert@gmail.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
All but 2 of the 25 Covad^H^H^H^H^H Megapath circuits that were down in the GLRYCAXF CO are now back up. Also, landlines and Verizon DSL is back up in parts of Gilroy. Looks like AT&T is making progress on the fiber splicing. On 4/16/2013 8:45 PM, Mitch wrote:
Latest from mega path: *UPDATE: *ILEC/Carrier working on Fiber Splicing. ETR is 2:00 AM PST, Next notification will be sent at 00:00 AM PST.
On Tue, Apr 16, 2013 at 11:00 PM, George Herbert <george.herbert@gmail.com <mailto:george.herbert@gmail.com>> wrote:
On Tue, Apr 16, 2013 at 6:17 PM, Jeremy Chadwick <jdc@koitsu.org <mailto:jdc@koitsu.org>> wrote:
Number 4 on that list happens here in my city almost weekly, whether the lines be for telephone, cable, or power.
The residential line techs I've talked to indicate the common cause are big rigs driving around in areas where the cables are low-hanging. Of course nobody reports this until there's an actual outage... :/
This happened to my neighbors about 3 years ago, in the middle of the night. Fortunately only took their phone and cable line out, but we heard the "Bang" as it was pulled off the house and then the truck screeching to a halt...
-- -george william herbert george.herbert@gmail.com <mailto:george.herbert@gmail.com>
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
The single DSL line we had down from this outage is restored(from covad/megapath) On Wed, Apr 17, 2013 at 1:36 AM, Robert Glover <robertg@garlic.com> wrote:
All but 2 of the 25 Covad^H^H^H^H^H Megapath circuits that were down in the GLRYCAXF CO are now back up.
Also, landlines and Verizon DSL is back up in parts of Gilroy. Looks like AT&T is making progress on the fiber splicing.
On 4/16/2013 8:45 PM, Mitch wrote:
Latest from mega path: *UPDATE: *ILEC/Carrier working on Fiber Splicing. ETR is 2:00 AM PST, Next notification will be sent at 00:00 AM PST.
On Tue, Apr 16, 2013 at 11:00 PM, George Herbert <george.herbert@gmail.com
wrote:
On Tue, Apr 16, 2013 at 6:17 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Number 4 on that list happens here in my city almost weekly, whether the lines be for telephone, cable, or power.
The residential line techs I've talked to indicate the common cause are big rigs driving around in areas where the cables are low-hanging. Of course nobody reports this until there's an actual outage... :/
This happened to my neighbors about 3 years ago, in the middle of the night. Fortunately only took their phone and cable line out, but we heard the "Bang" as it was pulled off the house and then the truck screeching to a halt...
-- -george william herbert george.herbert@gmail.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages
Looks like Megapath has given the all clear on their side *Description:* Service Interruption at the GLRYCAXF central office in San Jose, CA has been resolved. *Restoration Information: *ILEC/Carrier Equipment - Repaired On Wed, Apr 17, 2013 at 1:57 AM, Mitch <mitpatterson@gmail.com> wrote:
The single DSL line we had down from this outage is restored(from covad/megapath)
On Wed, Apr 17, 2013 at 1:36 AM, Robert Glover <robertg@garlic.com> wrote:
All but 2 of the 25 Covad^H^H^H^H^H Megapath circuits that were down in the GLRYCAXF CO are now back up.
Also, landlines and Verizon DSL is back up in parts of Gilroy. Looks like AT&T is making progress on the fiber splicing.
On 4/16/2013 8:45 PM, Mitch wrote:
Latest from mega path: *UPDATE: *ILEC/Carrier working on Fiber Splicing. ETR is 2:00 AM PST, Next notification will be sent at 00:00 AM PST.
On Tue, Apr 16, 2013 at 11:00 PM, George Herbert < george.herbert@gmail.com> wrote:
On Tue, Apr 16, 2013 at 6:17 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Number 4 on that list happens here in my city almost weekly, whether the lines be for telephone, cable, or power.
The residential line techs I've talked to indicate the common cause are big rigs driving around in areas where the cables are low-hanging. Of course nobody reports this until there's an actual outage... :/
This happened to my neighbors about 3 years ago, in the middle of the night. Fortunately only took their phone and cable line out, but we heard the "Bang" as it was pulled off the house and then the truck screeching to a halt...
-- -george william herbert george.herbert@gmail.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages
Received notification from TelePacific that AT&T experienced a fiber cut due to vandalism. Crews are actively working on resolving the issue, but there is no available completion time estimate at this time. On Tue, Apr 16, 2013 at 7:37 AM, Robert Glover <robertg@garlic.com> wrote:
Hi,
I've got confirmed reports (from Cogent and Megapath) that there is a fiber cut affecting service through the South Bay. We have Cogent fiber circuits down in Campbell and San Martin. In San Martin, landlines have no dialtone and AT&T cell phones have no service whatsoever. All calls to Gilroy landlines are failing to All Circuits Busy.
Anyone have any further details? The issue seems to have started around 1:00am PDT or so.
-Robert ______________________________**_________________ Outages mailing list Outages@outages.org https://puck.nether.net/**mailman/listinfo/outages<https://puck.nether.net/mailman/listinfo/outages>
-- ------------------------------------------------------------------- On the Internet, you can be anything you want. It's strange so many people choose to be stupid.
This is all I have from Megapath: *Description:* We are currently experiencing a service interruption due to equipment failure at the GLRYCAXF central office in San Jose, CA. *UPDATE: *Megapath NOC is still following up with Carrier/ ILEC to resolve the fibercut issue in their network. No ETR available. *Master TT number: *5303870 *Severity Level: *P1 *Total End-Users Affected: *1 *Start Time: *04/16/2013 01:15:00 AM PDT On Tue, Apr 16, 2013 at 10:37 AM, Robert Glover <robertg@garlic.com> wrote:
Hi,
I've got confirmed reports (from Cogent and Megapath) that there is a fiber cut affecting service through the South Bay. We have Cogent fiber circuits down in Campbell and San Martin. In San Martin, landlines have no dialtone and AT&T cell phones have no service whatsoever. All calls to Gilroy landlines are failing to All Circuits Busy.
Anyone have any further details? The issue seems to have started around 1:00am PDT or so.
-Robert ______________________________**_________________ Outages mailing list Outages@outages.org https://puck.nether.net/**mailman/listinfo/outages<https://puck.nether.net/mailman/listinfo/outages>
Mitch, Yep that is exactly what I see in Megapath CONNECT as well for our myriad of Gilroy circuits..
From what I have gathered from various news outlets and other mailing lists, a power substation in San Jose was vandalized, along with fiber that runs nearby it. HAZMAT had to get the sub-station vandalism cleaned up before AT&T could get in to start splicing.
-Robert On 04/16/2013 11:01 AM, Mitch wrote:
This is all I have from Megapath: *Description:* We are currently experiencing a service interruption due to equipment failure at the GLRYCAXF central office in San Jose, CA.
*UPDATE: *Megapath NOC is still following up with Carrier/ ILEC to resolve the fibercut issue in their network. No ETR available.
*Master TT number: *5303870 *Severity Level: *P1 *Total End-Users Affected: *1 *Start Time: *04/16/2013 01:15:00 AM PDT
On Tue, Apr 16, 2013 at 10:37 AM, Robert Glover <robertg@garlic.com <mailto:robertg@garlic.com>> wrote:
Hi,
I've got confirmed reports (from Cogent and Megapath) that there is a fiber cut affecting service through the South Bay. We have Cogent fiber circuits down in Campbell and San Martin. In San Martin, landlines have no dialtone and AT&T cell phones have no service whatsoever. All calls to Gilroy landlines are failing to All Circuits Busy.
Anyone have any further details? The issue seems to have started around 1:00am PDT or so.
-Robert _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
*Latest update* * **Description:* We are currently experiencing a service interruption due to equipment failure at the GLRYCAXF central office in San Jose, CA. *UPDATE: *Megapath NOC is following up with the ILEC/ Carrier, as per the current updates their fiber crew is working to restore the fibercut . No ETR available. *Master TT number: *5303870 *Severity Level: *P1 *Total End-Users Affected: *1 *Start Time: *04/16/2013 01:15:00 AM PDT *Estimated Time to Resolution: *Unknown Historical time to repair this type of outage is not available. On Tue, Apr 16, 2013 at 2:01 PM, Mitch <mitpatterson@gmail.com> wrote:
This is all I have from Megapath: *Description:* We are currently experiencing a service interruption due to equipment failure at the GLRYCAXF central office in San Jose, CA.
*UPDATE: *Megapath NOC is still following up with Carrier/ ILEC to resolve the fibercut issue in their network. No ETR available.
*Master TT number: *5303870 *Severity Level: *P1 *Total End-Users Affected: *1 *Start Time: *04/16/2013 01:15:00 AM PDT
On Tue, Apr 16, 2013 at 10:37 AM, Robert Glover <robertg@garlic.com>wrote:
Hi,
I've got confirmed reports (from Cogent and Megapath) that there is a fiber cut affecting service through the South Bay. We have Cogent fiber circuits down in Campbell and San Martin. In San Martin, landlines have no dialtone and AT&T cell phones have no service whatsoever. All calls to Gilroy landlines are failing to All Circuits Busy.
Anyone have any further details? The issue seems to have started around 1:00am PDT or so.
-Robert ______________________________**_________________ Outages mailing list Outages@outages.org https://puck.nether.net/**mailman/listinfo/outages<https://puck.nether.net/mailman/listinfo/outages>
All, Just got off the horn with Cogent. They called AT&T while they had me on hold, AT&T told them they have extracted the fiber, which was, according to them, "very heavily damaged". They have brought a new cable in, and are in the process of splicing. -Robert On 4/16/2013 11:40 AM, Mitch wrote:
*Latest update* * * * **Description:* We are currently experiencing a service interruption due to equipment failure at the GLRYCAXF central office in San Jose, CA.
*UPDATE: *Megapath NOC is following up with the ILEC/ Carrier, as per the current updates their fiber crew is working to restore the fibercut . No ETR available.
*Master TT number: *5303870 *Severity Level: *P1 *Total End-Users Affected: *1 *Start Time: *04/16/2013 01:15:00 AM PDT *Estimated Time to Resolution: *Unknown Historical time to repair this type of outage is not available.
On Tue, Apr 16, 2013 at 2:01 PM, Mitch <mitpatterson@gmail.com <mailto:mitpatterson@gmail.com>> wrote:
This is all I have from Megapath: *Description:* We are currently experiencing a service interruption due to equipment failure at the GLRYCAXF central office in San Jose, CA.
*UPDATE: *Megapath NOC is still following up with Carrier/ ILEC to resolve the fibercut issue in their network. No ETR available.
*Master TT number: *5303870 *Severity Level: *P1 *Total End-Users Affected: *1 *Start Time: *04/16/2013 01:15:00 AM PDT
On Tue, Apr 16, 2013 at 10:37 AM, Robert Glover <robertg@garlic.com <mailto:robertg@garlic.com>> wrote:
Hi,
I've got confirmed reports (from Cogent and Megapath) that there is a fiber cut affecting service through the South Bay. We have Cogent fiber circuits down in Campbell and San Martin. In San Martin, landlines have no dialtone and AT&T cell phones have no service whatsoever. All calls to Gilroy landlines are failing to All Circuits Busy.
Anyone have any further details? The issue seems to have started around 1:00am PDT or so.
-Robert _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
participants (8)
-
Darren Schreiber -
George Herbert -
Jeremy Chadwick -
Joe Sanchez -
Mitch -
Rob Laird (rolaird) -
Robert Glover -
walkabouttigger