fibre cut near 200 Paul, San Francisco
Hey, I am hearing from multiple people about connectivity problems in the bay area, and they all seem to have 200 Paul in common. ISC is reporting a fibre cut between 529 Bryant, Palo Alto and 200 Paul, SF. At least one Unitedlayer customer in 200 Paul seems to be off the air. Anybody know any specifics? Joe
On Thu, Apr 09, 2009 at 08:43:47AM -0400, Joe Abley wrote:
I am hearing from multiple people about connectivity problems in the bay area, and they all seem to have 200 Paul in common. ISC is reporting a fibre cut between 529 Bryant, Palo Alto and 200 Paul, SF. At least one Unitedlayer customer in 200 Paul seems to be off the air.
Anybody know any specifics?
Only thing I've heard is how Abovenet has a "supposed" fibre cut somewhere between San Francisco and Los Angeles, which is affecting a large number of customers in the Bay Area. One of our gigE circuits utilising that path is hard down (even now). Could be related, then again might not be. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
My co-worker tells me there's been an announcement of the below fibre cut on 810AM radio this morning in the Bay, stating it's affecting AT&T, Verizon, Verizon Wireless, and other carriers. The physical location of the cut has not yet been disclosed. So, the news/media is aware of the problem. Good to know they all have redundant circuits running through alternate locations and conduits. *shakes head* -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | On Thu, Apr 09, 2009 at 08:43:47AM -0400, Joe Abley wrote:
Hey,
I am hearing from multiple people about connectivity problems in the bay area, and they all seem to have 200 Paul in common. ISC is reporting a fibre cut between 529 Bryant, Palo Alto and 200 Paul, SF. At least one Unitedlayer customer in 200 Paul seems to be off the air.
Anybody know any specifics?
Joe _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
http://sandbox.bitgravity.com/blog/2009/04/09/destroy-the-internet-with-a-ha...
"rather than going down into the manhole....." Great... give the kids some new ideas. geeze. -Jim P.
Yeah, I read that and my first thought was "just what we need, people *explaining* how to cause problems". I've never been a big fan of publicizing detailed fiber route maps though, I guess this just gives me more reasons to make that argument... -Bill
http://sandbox.bitgravity.com/blog/2009/04/09/destroy-the-internet-with- a-hacksaw/
"rather than going down into the manhole....."
Great... give the kids some new ideas. geeze.
-Jim P. _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Thu, Apr 9, 2009 at 3:51 PM, Bill Wichers <billw@waveform.net> wrote:
Yeah, I read that and my first thought was "just what we need, people *explaining* how to cause problems".
Terrorists who want to damage the US and world economies by attacking the Internet's soft underbelly no doubt prolly already figured it out.
Most likely yes, but I was thinking mostly of the home-grown troublemakers that have always been around. Somehow telecom seems to be mostly immune from them. Since there really is no practical way to secure the outside plant (and to the one that mentioned lockable manhole covers -- have a look at the "key" to those sometime, there isn't very much security there), the only thing we can do to keep it safe is to not make a big deal about where it is or what can harm it. -Bill
On Thu, Apr 9, 2009 at 3:51 PM, Bill Wichers <billw@waveform.net> wrote:
Yeah, I read that and my first thought was "just what we need, people *explaining* how to cause problems".
Terrorists who want to damage the US and world economies by attacking the Internet's soft underbelly no doubt prolly already figured it out. _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
the only thing we can do to keep it safe is to not make a big deal about where it is or what can harm it.
Agreed. Then we'll always know it's only the CWA members who are ever to blame, like when they did this in my area in '04: http://www.cwalocal4250.org/operatorservices/binarydata/Meet%20With%20Mediat or.pdf Of course, after that one, the police were investigating it as a "Terrorist Activity" under the "Patriot Act", but no arrests were made in our area, despite the eyewitness who practically saw it happen... Any way we can petition the git with the blog entry to pull that map of his page? Geez! Kirk Coviello Digital West Networks, Inc. V: 805.781.WEST (9378) Email: kirk@digitalwest.net
On Thu, Apr 9, 2009 at 2:42 PM, Kirk Coviello <kcoviello@digitalwest.net> wrote:
Any way we can petition the git with the blog entry to pull that map of his page? Geez!
for someone looking to damage fiber paths, it's painfully easy (but never impresses dates) to walk/drive around and physically trace them yourself. hiding maps is security theater
Bill Wichers wrote:
Most likely yes, but I was thinking mostly of the home-grown troublemakers that have always been around. Somehow telecom seems to be mostly immune from them.
Since there really is no practical way to secure the outside plant (and to the one that mentioned lockable manhole covers -- have a look at the "key" to those sometime, there isn't very much security there), the only thing we can do to keep it safe is to not make a big deal about where it is or what can harm it.
Some news reports suggest this may be and "inside" (so to speak) job. I have no facts, beyond what I have seen here.
On Thu, Apr 09, 2009 at 04:42:53PM -0500, Larry Sheldon wrote:
Bill Wichers wrote:
Most likely yes, but I was thinking mostly of the home-grown troublemakers that have always been around. Somehow telecom seems to be mostly immune from them.
Since there really is no practical way to secure the outside plant (and to the one that mentioned lockable manhole covers -- have a look at the "key" to those sometime, there isn't very much security there), the only thing we can do to keep it safe is to not make a big deal about where it is or what can harm it.
Some news reports suggest this may be and "inside" (so to speak) job.
I have no facts, beyond what I have seen here.
CWA claims no involvement: http://www.msnbc.msn.com/id/30139776/ A CWA member states "only a nut would do this": http://www.dslreports.com/shownews/ATT-Fiber-Cut-Impacts-Silicon-Valley-1018... Video of the cable cut itself indicates thieves may have been out "cutting for copper". It's quite a clean cut (01:48 to 2:02): http://cbs5.com/video/?id=48654@kpix.dayport.com The video may also help someone determine the exact location and which of the two cuts it was. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
As fashionable as it is to be paranoid, this does seem to fit the 'never attribute to malice that which can be more easily explained by stupidity' realm. IIRC, there have been a number of cases recently where various fiber trunks (some of them aerial runs) have been cut by miscreants looking for copper - they must have been so disappointed..... Jeremy Chadwick wrote:
Video of the cable cut itself indicates thieves may have been out "cutting for copper". It's quite a clean cut (01:48 to 2:02): http://cbs5.com/video/?id=48654@kpix.dayport.com
-- Scanned by MailScanner
Jeremy Chadwick writes:
Video of the cable cut itself indicates thieves may have been out "cutting for copper". It's quite a clean cut (01:48 to 2:02): http://cbs5.com/video/?id=48654@kpix.dayport.com
The video may also help someone determine the exact location and which of the two cuts it was.
Clearly not the San Carlos cut - seems to be very close to W121.80542 N37.25718 Beginning of the ramp up from southbound Monterey Highway to Westbound Blossom Hill Road. There's a tree there in Google Earth - but you can see pieces of cut down tree in the video, and the lightpole which is in the nearby background up the ramp. -george william herbert gherbert@retro.com
On Thu, Apr 9, 2009 at 17:26, Bill Wichers <billw@waveform.net> wrote:
Most likely yes, but I was thinking mostly of the home-grown troublemakers that have always been around.
As was i, that and the slacker crowd looking for something to do on a slow day. The issue wasn't whether security by obscurity is effective, it was about giving ideas to those who don't need any encouragement. -Jim P.
On Thu, Apr 9, 2009 at 1:51 PM, Bill Wichers <billw@waveform.net> wrote:
Yeah, I read that and my first thought was "just what we need, people *explaining* how to cause problems".
I've never been a big fan of publicizing detailed fiber route maps though, I guess this just gives me more reasons to make that argument...
who needs detailed fiber maps when you can have the 1-800-DIG people come out and spray paint the important routes for you? people who want to destroy things will figure out how to destroy them. hack saw, gasoline, or otherwise. maybe we should lay so much fiber it wont matter what they firebomb (-:
On Thu, Apr 9, 2009 at 4:20 PM, Aaron Glenn <aaron.glenn@gmail.com> wrote:
maybe we should lay so much fiber it wont matter what they firebomb (-:
We could always clawback the USF funds and require they be used sensibly, as in your "lay so much fiber it wont matter."
Aaron Glenn wrote:
On Thu, Apr 9, 2009 at 1:51 PM, Bill Wichers <billw@waveform.net> wrote:
Yeah, I read that and my first thought was "just what we need, people *explaining* how to cause problems".
I've never been a big fan of publicizing detailed fiber route maps though, I guess this just gives me more reasons to make that argument...
who needs detailed fiber maps when you can have the 1-800-DIG people come out and spray paint the important routes for you? people who want to destroy things will figure out how to destroy them. hack saw, gasoline, or otherwise.
Even easier: walk down the street and look for anything visibly marked with bright orange. Or look in manholes marked as telcom for the same. It's likely to be fiber. ~Seth
Jim Popovitch wrote:
http://sandbox.bitgravity.com/blog/2009/04/09/destroy-the-internet-with-a-ha...
"rather than going down into the manhole....."
Great... give the kids some new ideas. geeze.
Should have waited a bit--an Executive Order would have covered the situation.
On Thu, 9 Apr 2009, Joe Abley wrote:
I am hearing from multiple people about connectivity problems in the bay area, and they all seem to have 200 Paul in common. ISC is reporting a fibre cut between 529 Bryant, Palo Alto and 200 Paul, SF. At least one Unitedlayer customer in 200 Paul seems to be off the air.
We're showing spikes in latency out in LAX on Cogent, as well as a few other interesting anomalies that are consistent with shifts in traffic in a number of different locations. Has anyone noticed any network routing issues outside of the immediately-effected area, as a result of this? 9 gi0-0-0.core01.lax05.atlas.cogentco.com (154.54.6.185) 65.447 ms 65.449 ms 65.434 ms 10 xo.lax05.atlas.cogentco.com (154.54.11.242) 127.708 ms 127.683 ms 127.680 ms -- William R. Lorenz
has anyone been able to pin point the cut? There have been mentions of Redwood City, San Carlos, San Francisco. Where exactly is the cut? William R. Lorenz wrote:
On Thu, 9 Apr 2009, Joe Abley wrote:
I am hearing from multiple people about connectivity problems in the bay area, and they all seem to have 200 Paul in common. ISC is reporting a fibre cut between 529 Bryant, Palo Alto and 200 Paul, SF. At least one Unitedlayer customer in 200 Paul seems to be off the air.
We're showing spikes in latency out in LAX on Cogent, as well as a few other interesting anomalies that are consistent with shifts in traffic in a number of different locations. Has anyone noticed any network routing issues outside of the immediately-effected area, as a result of this?
9 gi0-0-0.core01.lax05.atlas.cogentco.com (154.54.6.185) 65.447 ms 65.449 ms 65.434 ms 10 xo.lax05.atlas.cogentco.com (154.54.11.242) 127.708 ms 127.683 ms 127.680 ms
-- William R. Lorenz
The reports seem to be settling on 2 distinct sites -- South San Jose and San Carlos. The former had 4 cuts and latter just 1. -r On Thu, Apr 09, 2009 at 06:27:53PM -0700, John Martinez wrote:
has anyone been able to pin point the cut? There have been mentions of Redwood City, San Carlos, San Francisco. Where exactly is the cut?
William R. Lorenz wrote:
On Thu, 9 Apr 2009, Joe Abley wrote:
I am hearing from multiple people about connectivity problems in the bay area, and they all seem to have 200 Paul in common. ISC is reporting a fibre cut between 529 Bryant, Palo Alto and 200 Paul, SF. At least one Unitedlayer customer in 200 Paul seems to be off the air.
We're showing spikes in latency out in LAX on Cogent, as well as a few other interesting anomalies that are consistent with shifts in traffic in a number of different locations. Has anyone noticed any network routing issues outside of the immediately-effected area, as a result of this?
9 gi0-0-0.core01.lax05.atlas.cogentco.com (154.54.6.185) 65.447 ms 65.449 ms 65.434 ms 10 xo.lax05.atlas.cogentco.com (154.54.11.242) 127.708 ms 127.683 ms 127.680 ms
-- William R. Lorenz
participants (16)
-
Aaron Glenn -
Bill Wichers -
Gadi Evron -
George William Herbert -
Jason Lewis -
Jeremy Chadwick -
Jim Popovitch -
Joe Abley -
John Martinez -
Kirk Coviello -
Larry Sheldon -
Larry Vaden -
Ravi Pina -
Rich Parker -
Seth Mattinen -
William R. Lorenz