Anyone noticing problems with Zayo transit within the last 15 minutes? Peers are about 125k routes lower in Chicago and Amsterdam. Mike
Yup, stuff was routing all over the universe for us all the sudden (literally). It has resolved now. From: Outages <outages-bounces@outages.org> on behalf of Mike Davis via Outages <Outages@outages.org> Reply-To: Mike Davis <mdavis@singlehop.com> Date: Friday, September 29, 2017 at 11:11 AM To: "Outages@outages.org" <Outages@outages.org> Subject: [outages] Zayo Transit Anyone noticing problems with Zayo transit within the last 15 minutes? Peers are about 125k routes lower in Chicago and Amsterdam. Mike
Yes, We are seeing tracert’s fail to our destination in New York from France, Sweden, Denmark, Toronto, CA, and Philadelphia, PA. Regards, Ryan
On Sep 29, 2017, at 14:11, Mike Davis via Outages <outages@outages.org> wrote:
Anyone noticing problems with Zayo transit within the last 15 minutes?
Peers are about 125k routes lower in Chicago and Amsterdam.
Mike _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Charter/Spectrum has lost routes to Zayo, GTT looks to be affected as well. Ive checked a few others, Comcast and Level3 look unaffected. Sites are not totally offline or unreachable but are seeing limited reachability. We only have 2 with Zayo, Memphis and Charlotte, but both are affected. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Ryan Collins via Outages Sent: Friday, September 29, 2017 1:14 PM To: Mike Davis Cc: Outages@outages.org Subject: Re: [outages] Zayo Transit Yes, We are seeing tracert’s fail to our destination in New York from France, Sweden, Denmark, Toronto, CA, and Philadelphia, PA. Regards, Ryan
On Sep 29, 2017, at 14:11, Mike Davis via Outages <outages@outages.org> wrote:
Anyone noticing problems with Zayo transit within the last 15 minutes?
Peers are about 125k routes lower in Chicago and Amsterdam.
Mike _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We are still seeing problems with Zayo (indirectly) as of now.
:| sigh... created by Eric Zalesak on Sep 29, 2017 12:02:03 PM Turn up customer was Singtel. We are still seeing sessions dropping after mitigating Singtel. created by Eric Zalesak on Sep 29, 2017 11:55:59 AM Suspected turn up with open policy. On Fri, Sep 29, 2017 at 11:22 AM, Steven Raymond via Outages < outages@outages.org> wrote:
We are still seeing problems with Zayo (indirectly) as of now.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Looks like zayo went above their 60k prefix limit causing peers to drop if configured to do so. Looks like they updated to 65k per peeringdb. On Sep 29, 2017, at 3:12 PM, Mike Davis via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: :| sigh... created by Eric Zalesak on Sep 29, 2017 12:02:03 PM Turn up customer was Singtel. We are still seeing sessions dropping after mitigating Singtel. created by Eric Zalesak on Sep 29, 2017 11:55:59 AM Suspected turn up with open policy. On Fri, Sep 29, 2017 at 11:22 AM, Steven Raymond via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: We are still seeing problems with Zayo (indirectly) as of now. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Zayo Master Ticket TTN-0001950534 On Fri, Sep 29, 2017 at 12:59 PM, Adam Gregory via Outages < outages@outages.org> wrote:
Looks like zayo went above their 60k prefix limit causing peers to drop if configured to do so. Looks like they updated to 65k per peeringdb.
On Sep 29, 2017, at 3:12 PM, Mike Davis via Outages <outages@outages.org> wrote:
:| sigh...
created by Eric Zalesak on Sep 29, 2017 12:02:03 PM Turn up customer was Singtel. We are still seeing sessions dropping after mitigating Singtel.
created by Eric Zalesak on Sep 29, 2017 11:55:59 AM Suspected turn up with open policy.
On Fri, Sep 29, 2017 at 11:22 AM, Steven Raymond via Outages < outages@outages.org> wrote:
We are still seeing problems with Zayo (indirectly) as of now.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We use Zayo for IP access and are still experiencing issues. Here's the response we've recived from Zayo Support: We are experiencing a major network event across multiple peering points from the global internet table that is impacting traffic throughout the internet. We are working to mitigate the incoming traffic across these peers at this time to lessen the impact until the root-cause of the influx can be identified and halted. We expect to have a resolution within the hour, however this appears to be incoming to the Zayo global internet routing table from our major peering partners. We appreciate your patience as our technical teams are fully engaged to cease impact to our customers at this time and we will update you as the status changes. We have opened the following customer-facing bridge to assist with real-time updates as they are known to our NCC team: <number removed> -andy On Fri, Sep 29, 2017 at 12:59 PM, Adam Gregory via Outages < outages@outages.org> wrote:
Looks like zayo went above their 60k prefix limit causing peers to drop if configured to do so. Looks like they updated to 65k per peeringdb.
On Sep 29, 2017, at 3:12 PM, Mike Davis via Outages <outages@outages.org> wrote:
:| sigh...
created by Eric Zalesak on Sep 29, 2017 12:02:03 PM Turn up customer was Singtel. We are still seeing sessions dropping after mitigating Singtel.
created by Eric Zalesak on Sep 29, 2017 11:55:59 AM Suspected turn up with open policy.
On Fri, Sep 29, 2017 at 11:22 AM, Steven Raymond via Outages < outages@outages.org> wrote:
We are still seeing problems with Zayo (indirectly) as of now.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Last update we received: Dear Zayo Customer, The majority of our peering links with our affected peering partners have restored with ancillary points still left. We are working to clear damping and reset those sessions left and some customers may still experience latency until this is completed. Thank you for your patience and if you are still experiencing a hard down issue, let us know so we can work to resolve it in a timely manner. From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Andy Litzinger via Outages Sent: Friday, September 29, 2017 1:07 PM To: Adam Gregory <adam.gregory@heidmar.com> Cc: outages@outages.org Subject: Re: [outages] Zayo Transit We use Zayo for IP access and are still experiencing issues. Here's the response we've recived from Zayo Support: We are experiencing a major network event across multiple peering points from the global internet table that is impacting traffic throughout the internet. We are working to mitigate the incoming traffic across these peers at this time to lessen the impact until the root-cause of the influx can be identified and halted. We expect to have a resolution within the hour, however this appears to be incoming to the Zayo global internet routing table from our major peering partners. We appreciate your patience as our technical teams are fully engaged to cease impact to our customers at this time and we will update you as the status changes. We have opened the following customer-facing bridge to assist with real-time updates as they are known to our NCC team: <number removed> -andy On Fri, Sep 29, 2017 at 12:59 PM, Adam Gregory via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: Looks like zayo went above their 60k prefix limit causing peers to drop if configured to do so. Looks like they updated to 65k per peeringdb. On Sep 29, 2017, at 3:12 PM, Mike Davis via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: :| sigh... created by Eric Zalesak on Sep 29, 2017 12:02:03 PM Turn up customer was Singtel. We are still seeing sessions dropping after mitigating Singtel. created by Eric Zalesak on Sep 29, 2017 11:55:59 AM Suspected turn up with open policy. On Fri, Sep 29, 2017 at 11:22 AM, Steven Raymond via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: We are still seeing problems with Zayo (indirectly) as of now. _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Looks like im.aol.com and webmail.aol.com (which both resolve to Akamai IPs) die in the Zayo network still...
On Sep 29, 2017, at 13:07, Andy Litzinger via Outages <outages@outages.org> wrote:
We use Zayo for IP access and are still experiencing issues. Here's the response we've recived from Zayo Support:
We are experiencing a major network event across multiple peering points from the global internet table that is impacting traffic throughout the internet. We are working to mitigate the incoming traffic across these peers at this time to lessen the impact until the root-cause of the influx can be identified and halted. We expect to have a resolution within the hour, however this appears to be incoming to the Zayo global internet routing table from our major peering partners. We appreciate your patience as our technical teams are fully engaged to cease impact to our customers at this time and we will update you as the status changes. We have opened the following customer-facing bridge to assist with real-time updates as they are known to our NCC team: <number removed>
-andy
On Fri, Sep 29, 2017 at 12:59 PM, Adam Gregory via Outages <outages@outages.org> wrote: Looks like zayo went above their 60k prefix limit causing peers to drop if configured to do so. Looks like they updated to 65k per peeringdb.
On Sep 29, 2017, at 3:12 PM, Mike Davis via Outages <outages@outages.org> wrote:
:| sigh...
created by Eric Zalesak on Sep 29, 2017 12:02:03 PM Turn up customer was Singtel. We are still seeing sessions dropping after mitigating Singtel.
created by Eric Zalesak on Sep 29, 2017 11:55:59 AM Suspected turn up with open policy.
On Fri, Sep 29, 2017 at 11:22 AM, Steven Raymond via Outages <outages@outages.org> wrote: We are still seeing problems with Zayo (indirectly) as of now.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Yeah, seeing a huge spike in updates, especially in Chicago. On Fri, Sep 29, 2017 at 2:11 PM, Mike Davis via Outages <outages@outages.org> wrote:
Anyone noticing problems with Zayo transit within the last 15 minutes?
Peers are about 125k routes lower in Chicago and Amsterdam.
Mike
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (11)
-
Adam Gregory -
Andy Litzinger -
Darren Schreiber -
Luke Rockwell -
Mike Davis -
mike.lyon＠gmail.com -
Neader, Brent -
Nicholas Suan -
Ryan Collins -
Sean Pedersen -
Steven Raymond