seeing similar on the east coast; somewhere between alter.net (vzb) and level3 in nyc just before the chi hop. at least that's what my traceroute is telling me. On Oct 2, 2009, at 3:54 PM, Jake VanDewater wrote:

We just opened a ticket with Verizon and were informed of outages for Verizon on East Coast.

-----Original Message----- From: outages-bounces@outages.org [mailto:outages- bounces@outages.org] On Behalf Of Joseph Jackson Sent: Friday, October 02, 2009 3:46 PM To: outages@outages.org Subject: [outages] Verizon west coast outage?

Anyone seeing packet loss going through .so-6-0-0.xt1.lax7.alter.net for hosts on Verizon but on the East Coast dying?

Have a prefix that isn't available if data comes into Verizon on the west coast only.

Joseph

_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages

-- Wes http://claimid.com/wesyoung

I'll be very interested to know what the root cause of this was. All the discussion indicates there is a single point of failure somewhere within Verizon's network out east; route propagation and reconvergence should have mitigated things, so the question is why it didn't. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | On Fri, Oct 02, 2009 at 09:12:47PM +0100, Lee Hetherington wrote:

This also seemed to have effected Europe connecting to Verizon routes in Denver and Sydney... and a Circuit we have on VZB in New York reaching much of the internet...

Looks to be stable again now...

Lee

-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jake VanDewater Sent: 02 October 2009 20:55 To: Joseph Jackson; outages@outages.org Subject: Re: [outages] Verizon west coast outage?

We just opened a ticket with Verizon and were informed of outages for Verizon on East Coast.

-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Joseph Jackson Sent: Friday, October 02, 2009 3:46 PM To: outages@outages.org Subject: [outages] Verizon west coast outage?

Anyone seeing packet loss going through .so-6-0-0.xt1.lax7.alter.net for hosts on Verizon but on the East Coast dying?

Have a prefix that isn't available if data comes into Verizon on the west coast only.

Joseph

_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages

______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________

______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________ _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages

This was posted on VZ's website as an explanation for the outage ! Original Link: http://forums.verizon.com/t5/Verizon-at-Home/Northeast-Router-Failure/ba -p/87001 ! "As many of you may know, a router failed in the Verizon network this afternoon causing problems for some Verizon FiOS and High Speed Internet users. Here's what happened. The router in question is in New York City and handles traffic from the Verizon network to the larger Internet. It is one of a number of such routers in our network. When routers have problems, they are designed to report that they are sick. Internet traffic is rerouted to adjacent routers automatically and sent around the trouble spot. In this case, that didn't happen. The router went into a hung state and did not appear to the rest of the network as though it was having problems. That meant that some user traffic from the northeast continued to flow to the stalled router, but couldn't be processed. Presto - an outage for those users. Verizon FiOS, High Speed Internet and small business customers were affected. The problem occurred at approximately 3:15 EDT and was detected and resolved :40 minutes later. Traffic began to flow normally and customers returned to service at that point. Currently, we're monitoring the router and the network watching for any other problems." ! Andre -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Seamus Hartmann Sent: Friday, October 02, 2009 3:55 PM To: outages@outages.org Subject: Re: [outages] Verizon west coast outage? Lost east coast as well. Issue started 3:13:58 PM, ended 3:50:11 PM EST. Seamus -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Joseph Jackson Sent: Friday, October 02, 2009 3:46 PM To: outages@outages.org Subject: [outages] Verizon west coast outage? Anyone seeing packet loss going through .so-6-0-0.xt1.lax7.alter.net for hosts on Verizon but on the East Coast dying? Have a prefix that isn't available if data comes into Verizon on the west coast only. Joseph _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages This message is intended only for the personal and confidential use of the recipients named above. If the reader of this email is not the intended recipient, you have received this email in error and any review, dissemination, distribution or copying is strictly prohibited. If you have received this email in error, please notify the sender immediately by return email and permanently delete the copy you received. This message is provided for informational purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments. Wolverine is not responsible for any recommendation, solicitation, offer or agreement or any information about any transaction, customer account or account activity that may be attached to or contained in this communication. Wolverine accepts no liability for any content contained in the email, or any errors or omissions arising as a result of email transmission. Any opinions contained in this email constitute the sender's best judgment at this time and are subject to change without notice.

And so the end of the story is... Epik has a more reliable network than Verizon. Ha! Matt Kollander Atlanta | Toronto | Los Angeles | Vancouver |Denver| -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jeremy Chadwick Sent: Friday, October 02, 2009 3:53 PM To: outages@outages.org Subject: Re: [outages] Verizon west coast outage? On Fri, Oct 02, 2009 at 04:55:30PM -0500, Forestal, Andre Jr. wrote:

When routers have problems, they are designed to report that they are sick. Internet traffic is rerouted to adjacent routers automatically and sent around the trouble spot. In this case, that didn't happen. The router went into a hung state and did not appear to the rest of the network as though it was having problems.

It appears BFD is a protocol foreign to Verizon. And why didn't hold timers expire on associated iBGP peers? Questions, questions... -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages

http://twitter.com/verizon On 10/2/09 4:12 PM, "Patrick W. Gilmore" <patrick@ianai.net> wrote:

Over 1000 calls in the VZ queue. They know they're broken. No idea why or ETR yet.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 There appears to be a major outage on VZB Internet facing side of the switch (east coast). VZB west coast seems to be ok, at least for AS701 (uunet). regards, /virendra Jeremy Chadwick wrote:

I'll add here that NANOG also has a couple threads going on about this which appear quite relevant, but it's hard to tell what's broken and why at this point.

http://www.merit.edu/mail.archives/nanog/msg00678.html

http://www.merit.edu/mail.archives/nanog/msg00676.html http://www.merit.edu/mail.archives/nanog/msg00679.html

http://www.merit.edu/mail.archives/nanog/msg00677.html http://www.merit.edu/mail.archives/nanog/msg00680.html http://www.merit.edu/mail.archives/nanog/msg00681.html

And possibly related, but unsure given that this report is significantly earlier than the others (AS7132 = AT&T).

http://www.merit.edu/mail.archives/nanog/msg00671.html

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKxmZHpbZvCIJx1bcRAhRcAKCYlU1KBLJJ2X33k9yge1FKqGWLuQCgyqVx Uh5QZp/5VNeVnkbFtcU8KoQ= =pJuM -----END PGP SIGNATURE-----