Verizon IP VPN (MPLS service) problems in the U.S., anyone?

I've got 100% lost packets involving multiple locations throughout the US and see odd traceroute behavior. Thanks! Note: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer.

We don't use Verizon MPLS at my workplace (instead just direct IP peering, e.g. BGP and physical connectivity), but we aren't seeing any anomalies. We also use Verizon IPTF, which I believe goes across their MPLS-based backbone, and we're not seeing any anomalies from any of Verizon's SBCs across the US (Georgia, New York, and California). Do you have an MPLS-capable traceroute available on your routers which you can use to narrow down which portions in their network are responsible (usually will show MPLS tags)? -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Tue, Jan 17, 2012 at 05:27:52PM -0500, Brad Beck wrote:
I've got 100% lost packets involving multiple locations throughout the US and see odd traceroute behavior.
Thanks!
Note: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Brad et al, A little birdie within Verizon tells me that there was, quote, "a major, *MAJOR* outage that has occurred". I didn't get any more details (because we saw no impact relating to my workplace), but our Account Rep was surprised to know we knew of "an MPLS issue". :-) -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Tue, Jan 17, 2012 at 02:33:35PM -0800, Jeremy Chadwick wrote:
We don't use Verizon MPLS at my workplace (instead just direct IP peering, e.g. BGP and physical connectivity), but we aren't seeing any anomalies.
We also use Verizon IPTF, which I believe goes across their MPLS-based backbone, and we're not seeing any anomalies from any of Verizon's SBCs across the US (Georgia, New York, and California).
Do you have an MPLS-capable traceroute available on your routers which you can use to narrow down which portions in their network are responsible (usually will show MPLS tags)?
-- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Tue, Jan 17, 2012 at 05:27:52PM -0500, Brad Beck wrote:
I've got 100% lost packets involving multiple locations throughout the US and see odd traceroute behavior.
Thanks!
Note: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Thanks Jeremy! @jdc.parodius.com] Sent: Tuesday, January 17, 2012 6:12 PM To: Brad Beck Cc: outages@outages.org Subject: Re: [outages] Verizon IP VPN (MPLS service) problems in the U.S., anyone? Brad et al, A little birdie within Verizon tells me that there was, quote, "a major, *MAJOR* outage that has occurred". I didn't get any more details (because we saw no impact relating to my workplace), but our Account Rep was surprised to know we knew of "an MPLS issue". :-) -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Tue, Jan 17, 2012 at 02:33:35PM -0800, Jeremy Chadwick wrote:
We don't use Verizon MPLS at my workplace (instead just direct IP peering, e.g. BGP and physical connectivity), but we aren't seeing any anomalies.
We also use Verizon IPTF, which I believe goes across their MPLS-based backbone, and we're not seeing any anomalies from any of Verizon's SBCs across the US (Georgia, New York, and California).
Do you have an MPLS-capable traceroute available on your routers which you can use to narrow down which portions in their network are responsible (usually will show MPLS tags)?
-- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Tue, Jan 17, 2012 at 05:27:52PM -0500, Brad Beck wrote:
I've got 100% lost packets involving multiple locations throughout the US and see odd traceroute behavior.
Thanks!
Note: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Note: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer.

* Brad Beck <Brad.Beck@meritain.com> wrote:
I've got 100% lost packets involving multiple locations throughout the US and see odd traceroute behavior.
"...world wide ticket ..." #2012011723964, according to https://twitter.com/#!/tonhe/status/159411814126456832 -- Jeremy L. Gaddis

I've got 100% lost packets involving multiple locations throughout the US and see odd traceroute behavior.
We saw the outage as well, however we didn't have 100% packet loss. We only had loss to internet destinations (seems as though 0/0 was blackholed). The prefixes in PIP that were destined to non-internet stuff stayed up. Anyone else see this kind of behavior?? -- max On Tue, Jan 17, 2012 at 4:27 PM, Brad Beck <Brad.Beck@meritain.com> wrote:
I've got 100% lost packets involving multiple locations throughout the US and see odd traceroute behavior.
Thanks!
Note: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

* N. Max Pierson <nmaxpierson@gmail.com> wrote:
We saw the outage as well, however we didn't have 100% packet loss. We only had loss to internet destinations (seems as though 0/0 was blackholed). The prefixes in PIP that were destined to non-internet stuff stayed up. Anyone else see this kind of behavior??
Rumor is that they somehow injected a 0/0 into an assload of VRFs, somewhere on the east coast. Sounds like a resume generating event... -- Jeremy L. Gaddis http://evilrouters.net
participants (4)
-
Brad Beck
-
Jeremy Chadwick
-
Jeremy L. Gaddis
-
N. Max Pierson