Level3 DECIX/Frankfurt Packet Loss
Hey All, we are currently experiencing very high packet loss over DECIX/Frankfurt with two different routings. The hops which are dropping the packets seem to be:
4.69.154.135 (ae-3-80.edge3.Frankfurt1.Level3.net) 62.4.65.106 (62.4.65.106.not.updated.fra.de.eunx.net)
Someone else has recognized similar problems?
Hi, indeed, we were also having serious issues and confirmed the packetloss/connectivity-loss (worldwide) via RIPE Atlas queries. We had to switch it off. Ticket opened @level3. Best Regards Max M. On 12.06.2015 11:29, Harald Haas (hahaas) via Outages wrote:
Hey All,
we are currently experiencing very high packet loss over DECIX/Frankfurt with two different routings.
The hops which are dropping the packets seem to be:
4.69.154.135 (ae-3-80.edge3.Frankfurt1.Level3.net) 62.4.65.106 (62.4.65.106.not.updated.fra.de.eunx.net)
Someone else has recognized similar problems?
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We are seeing packetloss across all of our EU sites backed by L3. Possibly related to the "AS4788 Telecom Malaysia major route leak" thread on NANOG? On Fri, Jun 12, 2015 at 4:41 AM, Max Mühlbronner <outages@outages.org> wrote:
Hi,
indeed, we were also having serious issues and confirmed the packetloss/connectivity-loss (worldwide) via RIPE Atlas queries. We had to switch it off. Ticket opened @level3.
Best Regards
Max M.
On 12.06.2015 11:29, Harald Haas (hahaas) via Outages wrote:
Hey All,
we are currently experiencing very high packet loss over DECIX/Frankfurt with two different routings.
The hops which are dropping the packets seem to be:
4.69.154.135 (ae-3-80.edge3.Frankfurt1.Level3.net) 62.4.65.106 (62.4.65.106.not.updated.fra.de.eunx.net)
Someone else has recognized similar problems?
_______________________________________________ Outages mailing listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Max, do you know if there is a master L3 ticket yet? On Fri, Jun 12, 2015 at 4:48 AM, James Baldwin <jbaldwin@antinode.net> wrote:
We are seeing packetloss across all of our EU sites backed by L3.
Possibly related to the "AS4788 Telecom Malaysia major route leak" thread on NANOG?
On Fri, Jun 12, 2015 at 4:41 AM, Max Mühlbronner <outages@outages.org> wrote:
Hi,
indeed, we were also having serious issues and confirmed the packetloss/connectivity-loss (worldwide) via RIPE Atlas queries. We had to switch it off. Ticket opened @level3.
Best Regards
Max M.
On 12.06.2015 11:29, Harald Haas (hahaas) via Outages wrote:
Hey All,
we are currently experiencing very high packet loss over DECIX/Frankfurt with two different routings.
The hops which are dropping the packets seem to be:
4.69.154.135 (ae-3-80.edge3.Frankfurt1.Level3.net) 62.4.65.106 (62.4.65.106.not.updated.fra.de.eunx.net)
Someone else has recognized similar problems?
_______________________________________________ Outages mailing listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Hi, i don't think so, nothing shown at my level3. On 12.06.2015 11:48, James Baldwin wrote:
Max, do you know if there is a master L3 ticket yet?
On Fri, Jun 12, 2015 at 4:48 AM, James Baldwin <jbaldwin@antinode.net <mailto:jbaldwin@antinode.net>> wrote:
We are seeing packetloss across all of our EU sites backed by L3.
Possibly related to the "AS4788 Telecom Malaysia major route leak" thread on NANOG?
On Fri, Jun 12, 2015 at 4:41 AM, Max Mühlbronner <outages@outages.org <mailto:outages@outages.org>> wrote:
Hi,
indeed, we were also having serious issues and confirmed the packetloss/connectivity-loss (worldwide) via RIPE Atlas queries. We had to switch it off. Ticket opened @level3.
Best Regards
Max M.
On 12.06.2015 11:29, Harald Haas (hahaas) via Outages wrote:
Hey All,
we are currently experiencing very high packet loss over DECIX/Frankfurt with two different routings.
The hops which are dropping the packets seem to be:
> 4.69.154.135 (ae-3-80.edge3.Frankfurt1.Level3.net <http://ae-3-80.edge3.Frankfurt1.Level3.net>) > 62.4.65.106 (62.4.65.106.not.updated.fra.de.eunx.net <http://62.4.65.106.not.updated.fra.de.eunx.net>)
Someone else has recognized similar problems?
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Looks like it stabilized. Regards, Dovid -----Original Message----- From: Max Mühlbronner via Outages <outages@outages.org> Sender: "Outages" <outages-bounces@outages.org>Date: Fri, 12 Jun 2015 11:52:59 To: James Baldwin<jbaldwin@antinode.net> Reply-To: Max Mühlbronner <mm@42com.com> Cc: <outages@outages.org> Subject: Re: [outages] Level3 DECIX/Frankfurt Packet Loss _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Hi, Level3 raised an Event Ticket ID 9340550 with following information: 06/12/2015 10:06 AM GMT The IP NOC is investigating routing issues in Multiple Markets across the network globally that are impacting IP services. Troubleshooting efforts are currently ongoing with no estimated time to restore services available at this point. Please be advised that updates for this event will be relayed every half an hour unless otherwise noted. Best Regards, Plamen On Fri, Jun 12, 2015 at 11:52 AM, Max Mühlbronner <outages@outages.org> wrote:
Hi,
i don't think so, nothing shown at my level3.
On 12.06.2015 11:48, James Baldwin wrote:
Max, do you know if there is a master L3 ticket yet?
On Fri, Jun 12, 2015 at 4:48 AM, James Baldwin <jbaldwin@antinode.net> wrote:
We are seeing packetloss across all of our EU sites backed by L3.
Possibly related to the "AS4788 Telecom Malaysia major route leak" thread on NANOG?
On Fri, Jun 12, 2015 at 4:41 AM, Max Mühlbronner <outages@outages.org> wrote:
Hi,
indeed, we were also having serious issues and confirmed the packetloss/connectivity-loss (worldwide) via RIPE Atlas queries. We had to switch it off. Ticket opened @level3.
Best Regards
Max M.
On 12.06.2015 11:29, Harald Haas (hahaas) via Outages wrote:
Hey All,
we are currently experiencing very high packet loss over DECIX/Frankfurt with two different routings.
The hops which are dropping the packets seem to be:
4.69.154.135 (ae-3-80.edge3.Frankfurt1.Level3.net) 62.4.65.106 (62.4.65.106.not.updated.fra.de.eunx.net)
Someone else has recognized similar problems?
_______________________________________________ Outages mailing listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Possibly related: http://seclists.org/nanog/2015/Jun/586 and http://seclists.org/nanog/2015/Jun/607 From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Plamen Stoev via Outages Sent: Friday, June 12, 2015 6:40 To: Max Mühlbronner Cc: outages@outages.org Subject: Re: [outages] Level3 DECIX/Frankfurt Packet Loss Hi, Level3 raised an Event Ticket ID 9340550 with following information: 06/12/2015 10:06 AM GMT The IP NOC is investigating routing issues in Multiple Markets across the network globally that are impacting IP services. Troubleshooting efforts are currently ongoing with no estimated time to restore services available at this point. Please be advised that updates for this event will be relayed every half an hour unless otherwise noted. Best Regards, Plamen On Fri, Jun 12, 2015 at 11:52 AM, Max Mühlbronner <outages@outages.org<mailto:outages@outages.org>> wrote: Hi, i don't think so, nothing shown at my level3. On 12.06.2015 11:48, James Baldwin wrote: Max, do you know if there is a master L3 ticket yet? On Fri, Jun 12, 2015 at 4:48 AM, James Baldwin <jbaldwin@antinode.net<mailto:jbaldwin@antinode.net>> wrote: We are seeing packetloss across all of our EU sites backed by L3. Possibly related to the "AS4788 Telecom Malaysia major route leak" thread on NANOG? On Fri, Jun 12, 2015 at 4:41 AM, Max Mühlbronner <outages@outages.org<mailto:outages@outages.org>> wrote: Hi, indeed, we were also having serious issues and confirmed the packetloss/connectivity-loss (worldwide) via RIPE Atlas queries. We had to switch it off. Ticket opened @level3. Best Regards Max M. On 12.06.2015 11:29, Harald Haas (hahaas) via Outages wrote: Hey All, we are currently experiencing very high packet loss over DECIX/Frankfurt with two different routings. The hops which are dropping the packets seem to be:
4.69.154.135 (ae-3-80.edge3.Frankfurt1.Level3.net<http://ae-3-80.edge3.Frankfurt1.Level3.net>) 62.4.65.106 (62.4.65.106.not.updated.fra.de.eunx.net<http://62.4.65.106.not.updated.fra.de.eunx.net>)
Someone else has recognized similar problems? _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages itevomcid
Also see here for more details: http://www.bgpmon.net/massive-route-leak-cause-internet-slowdown/ .-- My secret spy satellite informs me that at 2015-06-12 6:17 AM Hoyle Anderson (AM) via Outages wrote:
Possibly related: http://seclists.org/nanog/2015/Jun/586 and http://seclists.org/nanog/2015/Jun/607
*From:*Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Plamen Stoev via Outages *Sent:* Friday, June 12, 2015 6:40 *To:* Max Mühlbronner *Cc:* outages@outages.org *Subject:* Re: [outages] Level3 DECIX/Frankfurt Packet Loss
Hi,
Level3 raised an Event Ticket ID 9340550 with following information:
06/12/2015 10:06 AM GMT
The IP NOC is investigating routing issues in Multiple Markets across the network globally that are impacting IP services. Troubleshooting efforts are currently ongoing with no estimated time to restore services available at this point. Please be advised that updates for this event will be relayed every half an hour unless otherwise noted.
Best Regards,
Plamen
On Fri, Jun 12, 2015 at 11:52 AM, Max Mühlbronner <outages@outages.org <mailto:outages@outages.org>> wrote:
Hi,
i don't think so, nothing shown at my level3.
On 12.06.2015 11:48, James Baldwin wrote:
Max, do you know if there is a master L3 ticket yet?
On Fri, Jun 12, 2015 at 4:48 AM, James Baldwin <jbaldwin@antinode.net <mailto:jbaldwin@antinode.net>> wrote:
We are seeing packetloss across all of our EU sites backed by L3.
Possibly related to the "AS4788 Telecom Malaysia major route leak" thread on NANOG?
On Fri, Jun 12, 2015 at 4:41 AM, Max Mühlbronner <outages@outages.org <mailto:outages@outages.org>> wrote:
Hi,
indeed, we were also having serious issues and confirmed the packetloss/connectivity-loss (worldwide) via RIPE Atlas queries. We had to switch it off. Ticket opened @level3.
Best Regards
Max M.
On 12.06.2015 11:29, Harald Haas (hahaas) via Outages wrote:
Hey All,
we are currently experiencing very high packet loss over DECIX/Frankfurt with two different routings.
The hops which are dropping the packets seem to be:
> 4.69.154.135 (ae-3-80.edge3.Frankfurt1.Level3.net <http://ae-3-80.edge3.Frankfurt1.Level3.net>)
> 62.4.65.106 (62.4.65.106.not.updated.fra.de.eunx.net <http://62.4.65.106.not.updated.fra.de.eunx.net>)
Someone else has recognized similar problems?
_______________________________________________
Outages mailing list
Outages@outages.org <mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
itevomcid
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Similar issues with Level 3 chicago, atlanta and tampa getting in and out of their network to several others. From Brighthouse residential I can't get to their own website. Judging by the fact that I've been on hold for about 30 minutes now waiting for them to pick up, I'd say they're having a massive issue. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Harald Haas (hahaas) via Outages Sent: Friday, June 12, 2015 5:30 AM To: outages@outages.org Subject: [outages] Level3 DECIX/Frankfurt Packet Loss Hey All, we are currently experiencing very high packet loss over DECIX/Frankfurt with two different routings. The hops which are dropping the packets seem to be:
4.69.154.135 (ae-3-80.edge3.Frankfurt1.Level3.net) 62.4.65.106 (62.4.65.106.not.updated.fra.de.eunx.net)
Someone else has recognized similar problems?
participants (8)
-
Andree Toonk -
David Hubbard -
Dovid Bender -
Harald Haas (hahaas) -
Hoyle Anderson (AM) -
James Baldwin -
Max Mühlbronner -
Plamen Stoev