We currently have at least 5 sites down on Verizon Business MPLS in the Midwest. (Chicago, Milwaukee, Indy) We've been on hold with TAC for 30+ minutes. Any info is appreciated.
We got this message from our account rep: "We are have a catastrophic event (fiber cut) going in Riverdale, IL" Not sure if it's related On Thu, Aug 26, 2010 at 12:54 PM, notifications <notifications@mattfilip.com
wrote:
We currently have at least 5 sites down on Verizon Business MPLS in the Midwest. (Chicago, Milwaukee, Indy) We’ve been on hold with TAC for 30+ minutes. Any info is appreciated.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Yeah, this is it. Took us awhile to get info. Their IVR didn't report a fiber cut, just that they were working on it. From: Kenny Sallee [mailto:kenny.sallee@gmail.com] Sent: Thursday, August 26, 2010 3:48 PM To: notifications Cc: outages@outages.org Subject: Re: [outages] Verizon Business We got this message from our account rep: "We are have a catastrophic event (fiber cut) going in Riverdale, IL" Not sure if it's related On Thu, Aug 26, 2010 at 12:54 PM, notifications <notifications@mattfilip.com> wrote: We currently have at least 5 sites down on Verizon Business MPLS in the Midwest. (Chicago, Milwaukee, Indy) We've been on hold with TAC for 30+ minutes. Any info is appreciated. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages No virus found in this incoming message. Checked by AVG - www.avg.com Version: 9.0.851 / Virus Database: 271.1.1/3095 - Release Date: 08/26/10 01:34:00
Catastrophic event. Interesting language for a fibre cut? ~ jason
We got this message from our account rep:
"We are have a catastrophic event (fiber cut) going in Riverdale, IL"
Not sure if it's related
On Thu, Aug 26, 2010 at 12:54 PM, notifications < notifications@mattfilip.com
wrote:
We currently have at least 5 sites down on Verizon Business MPLS in the Midwest. (Chicago, Milwaukee, Indy) We’ve been on hold with TAC for 30+ minutes. Any info is appreciated.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Thu, Aug 26, 2010 at 2:11 PM, Kenny Sallee <kenny.sallee@gmail.com>wrote:
On Thu, Aug 26, 2010 at 2:10 PM, Jason L. Sparks <jlsparks@gmail.com>wrote:
Catastrophic event. Interesting language for a fibre cut?
Guess it depends on your perspective.
... and how many backup paths you have... Scott.
On Thu, Aug 26, 2010 at 02:11:26PM -0700, Kenny Sallee wrote:
On Thu, Aug 26, 2010 at 2:10 PM, Jason L. Sparks <[1]jlsparks@gmail.com> wrote:
Catastrophic event. Interesting language for a fibre cut?
~ jason
Guess it depends on your perspective.
Sounds catastrophic to me, given that they lack physical redundancy in their MPLS mesh. Customers should be asking them, post-mortem, why failover paths weren't available. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Jeremy, That's the catastrophic part. ~jason On Thu, Aug 26, 2010 at 5:30 PM, Jeremy Chadwick <outages@jdc.parodius.com>wrote:
On Thu, Aug 26, 2010 at 02:11:26PM -0700, Kenny Sallee wrote:
On Thu, Aug 26, 2010 at 2:10 PM, Jason L. Sparks <[1]jlsparks@gmail.com> wrote:
Catastrophic event. Interesting language for a fibre cut?
~ jason
Guess it depends on your perspective.
Sounds catastrophic to me, given that they lack physical redundancy in their MPLS mesh. Customers should be asking them, post-mortem, why failover paths weren't available.
-- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
On Aug 26, 2010, at 2:11 PM, Kenny Sallee wrote:
On Thu, Aug 26, 2010 at 2:10 PM, Jason L. Sparks <jlsparks@gmail.com> wrote: Catastrophic event. Interesting language for a fibre cut?
~ jason
Guess it depends on your perspective.
And whether you have a protect side, or, a second-side of the ring with even more circuits on it (famous Worldcom stunt). Owen
No kidding. Someone should ask that rep, if they have ever heard of MPLS Fast Re-Route. Not that hard... On Thu, Aug 26, 2010 at 2:10 PM, Jason L. Sparks <jlsparks@gmail.com> wrote:
Catastrophic event. Interesting language for a fibre cut?
~ jason
We got this message from our account rep:
"We are have a catastrophic event (fiber cut) going in Riverdale, IL"
Not sure if it's related
On Thu, Aug 26, 2010 at 12:54 PM, notifications < notifications@mattfilip.com
wrote:
We currently have at least 5 sites down on Verizon Business MPLS in the Midwest. (Chicago, Milwaukee, Indy) We’ve been on hold with TAC for 30+ minutes. Any info is appreciated.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Wait you mean have the network do something WITHOUT going through RCMAC?! BLASPHEMY! </sarcasm> :) A little more seriously though, often times theres one of several issues going on, either a 'collapsed loop' where the "redundant" paths are in the same trench, or, there just is no redundant path in their network. Happens far more often than any telco would want to admit to any customer. If I were affected by this I'd push for a root cause and a documented fix, along with credit heh. --On Thursday, August 26, 2010 2:17 PM -0700 Robin Pimentel <robin.pimentel@gmail.com> wrote:
No kidding. Someone should ask that rep, if they have ever heard of MPLS Fast Re-Route. Not that hard...
On Thu, Aug 26, 2010 at 2:10 PM, Jason L. Sparks <jlsparks@gmail.com> wrote:
Catastrophic event. Interesting language for a fibre cut?
~ jason
On Aug 26, 2010, at 4:10 PM, Jason L. Sparks wrote:
Catastrophic event. Interesting language for a fibre cut?
~ jason
We got this message from our account rep:
"We are have a catastrophic event (fiber cut) going in Riverdale, IL"
Not sure if it's related
On Thu, Aug 26, 2010 at 12:54 PM, notifications <notifications@mattfilip.com
wrote:
We currently have at least 5 sites down on Verizon Business MPLS in the Midwest. (Chicago, Milwaukee, Indy) We’ve been on hold with TAC for 30+ minutes. Any info is appreciated.
Well, Riverdale does have about four very large railroad yards. So if the fiber cut is, for instance, hypothetically speaking, underneath a big burning pile of locomotives and tank cars carrying HazMat, I suppose that could be considered a "catastrophic event." Just sayin'.... ---- Andy Ringsmuth andy@newslink.com News Link – Manager Technology 1845 S. 11th St., Lincoln, NE 68502 (402) 475-6397 (402) 304-0083 cellular
Catastrophic event sounds better, and makes customers less liked to be annoyed.. "We had a catastrophic event and that's why you network is down" sounds somewhat reasonable and you get the feeling that they are heroically struggling against terrorist and train wrecks and similar. "Some clod with a spade sliced our cable in half and now we are screwed..." makes them sound incompetent... Warren Kumari ------ Please excuse typing, etc -- This was sent from a device with a tiny keyboard. On Aug 26, 2010, at 5:10 PM, "Jason L. Sparks" <jlsparks@gmail.com> wrote:
Catastrophic event. Interesting language for a fibre cut?
~ jason
We got this message from our account rep:
"We are have a catastrophic event (fiber cut) going in Riverdale, IL"
Not sure if it's related
On Thu, Aug 26, 2010 at 12:54 PM, notifications <notifications@mattfilip.com
wrote:
We currently have at least 5 sites down on Verizon Business MPLS in the Midwest. (Chicago, Milwaukee, Indy) We’ve been on hold with TAC for 30+ minutes. Any info is appreciated.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Fair point. I acknowledge I jumped the gun. Carry on. ~jason
Catastrophic event sounds better, and makes customers less liked to be annoyed..
"We had a catastrophic event and that's why you network is down" sounds somewhat reasonable and you get the feeling that they are heroically struggling against terrorist and train wrecks and similar.
"Some clod with a spade sliced our cable in half and now we are screwed..." makes them sound incompetent...
Warren Kumari ------ Please excuse typing, etc -- This was sent from a device with a tiny keyboard.
On Aug 26, 2010, at 5:10 PM, "Jason L. Sparks" <jlsparks@gmail.com> wrote:
Catastrophic event. Interesting language for a fibre cut?
~ jason
We got this message from our account rep:
"We are have a catastrophic event (fiber cut) going in Riverdale, IL"
Not sure if it's related
On Thu, Aug 26, 2010 at 12:54 PM, notifications < notifications@mattfilip.com
wrote:
We currently have at least 5 sites down on Verizon Business MPLS in the Midwest. (Chicago, Milwaukee, Indy) We’ve been on hold with TAC for 30+ minutes. Any info is appreciated.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (10)
-
Andy Ringsmuth -
Jason L. Sparks -
Jeremy Chadwick -
Kenny Sallee -
Michael Loftis -
notifications -
Owen DeLong -
Robin Pimentel -
Scott Howard -
Warren Kumari