Zayo/Abovenet Issues in SF Bay Area

We're seeing routing loops and blackholing of SF Bay Area Zayo IPv4 space. In our case, our outbound traffic is blackholing at cr2.lax112.us.above.net. I filed a ticket, and Zayo didn't know of the outage yet. Other netops friends in the area are mentioning similar failures. Cheers, jof

Confirmed, Zayo is aware, we also have a ticket open and have depeered them until they are cleaned up. Peter Kranz <http://www.unwiredltd.com/> www.UnwiredLtd.com Desk: 510-868-1614 x100 Mobile: 510-207-0000 <mailto:pkranz@unwiredltd.com> pkranz@unwiredltd.com From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Jonathan Lassoff Sent: Monday, February 10, 2014 2:41 PM To: Outages Mailing List Subject: [outages] Zayo/Abovenet Issues in SF Bay Area We're seeing routing loops and blackholing of SF Bay Area Zayo IPv4 space. In our case, our outbound traffic is blackholing at cr2.lax112.us.above.net <http://cr2.lax112.us.above.net> . I filed a ticket, and Zayo didn't know of the outage yet. Other netops friends in the area are mentioning similar failures. Cheers, jof

On Mon, Feb 10, 2014 at 02:40:43PM -0800, Jonathan Lassoff wrote:
I filed a ticket, and Zayo didn't know of the outage yet. Other netops friends in the area are mentioning similar failures.
Yes. We've also opened a ticket but haven't yet received confirmation of any event. Our issue began at approximately 21:38 UTC. -Snow

I've also been seeing issues with Zayo, since early afternoon. Only in the bay area. Hey guys! -Yuri B On Mon, Feb 10, 2014 at 2:48 PM, James Snow <snow@teardrop.org> wrote:
On Mon, Feb 10, 2014 at 02:40:43PM -0800, Jonathan Lassoff wrote:
I filed a ticket, and Zayo didn't know of the outage yet. Other netops friends in the area are mentioning similar failures.
Yes. We've also opened a ticket but haven't yet received confirmation of any event. Our issue began at approximately 21:38 UTC.
-Snow
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Has anyone heard of a cause or estimated time to repair? -Snow

So far, Zayo has given no update to us and claims that there is no ETA/ETR on the parent ticket. The rumor mill is suggesting that a customer of theirs is suffering a large NTP amplification DDOS. Issues for us started at around 1:44pm (-0800). We still observe traffic blackholing or routing loops occasionally, it lasts for ~10min and then it comes right back for 20 - 45min. I've just shutdown Abovenet/Zayo BGP sessions for now. Cheers, jof On Mon, Feb 10, 2014 at 6:36 PM, James Snow <snow@teardrop.org> wrote:
Has anyone heard of a cause or estimated time to repair?
-Snow
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

This happened 1/21/14 from 12:40 PST - 14:15 PST as well.. I wonder if it's the same customer under attack which seems to cripple parts of their bay area network. Under Above.net we received excellent information about network events, under Zayo is has not been the same experience. -Peter

Last time we reached out to Zayo it took over a month. This is just sales and not any incident regarding service. Stop enabling bad service. On Feb 10, 2014 9:16 PM, "Peter Kranz" <pkranz@unwiredltd.com> wrote:
This happened 1/21/14 from 12:40 PST - 14:15 PST as well.. I wonder if it's the same customer under attack which seems to cripple parts of their bay area network. Under Above.net we received excellent information about network events, under Zayo is has not been the same experience.
-Peter
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (5)
-
James Snow
-
Jonathan Lassoff
-
Peter Kranz
-
Rogan Schlassa
-
Yuri Bank