We've been observing a small about of intermittent packet loss to/from AT&T and Level3 here in the San Francisco Bay Area this morning. In examining the situation, it seems like a evenly-spaced dropping of packets to certain IPs. This appears to end users like some IPs on destination networks are reachable, while some adjacent ones are not. I suspect this is due to a failed LAG leg or ECMP leg on Level3's side. We were able to route around the problem for now. Cheers, jof
I see some problems too with our level3 connection. Franck Martin http://www.avonsys.com/ http://www.facebook.com/Avonsys http://www.linkedin.com/company/avonsys twitter: FranckMartin Avonsys Check your domain reputation: http://gurl.im/b69d4o Application Monitoring: http://gurl.im/4d39Gu ----- Original Message ----- From: "Jonathan Lassoff" <jof@thejof.com> To: "Outages Mailing List" <outages@outages.org> Sent: Wednesday, 9 March, 2011 11:21:05 AM Subject: [outages] AT&T <-> Level3 Packet Loss in SF Bay Area We've been observing a small about of intermittent packet loss to/from AT&T and Level3 here in the San Francisco Bay Area this morning. In examining the situation, it seems like a evenly-spaced dropping of packets to certain IPs. This appears to end users like some IPs on destination networks are reachable, while some adjacent ones are not. I suspect this is due to a failed LAG leg or ECMP leg on Level3's side. We were able to route around the problem for now. Cheers, jof _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Hi - On 03/09/2011 11:21 AM, Jonathan Lassoff wrote:
We've been observing a small about of intermittent packet loss to/from AT&T and Level3 here in the San Francisco Bay Area this morning. In examining the situation, it seems like a evenly-spaced dropping of packets to certain IPs.
We experienced similar issue last week off our level3 connection for a specific /21 customer block. After further troubleshooting we decided to re-route our /21 to the other provider where packet loss issue went away. We are still scratching our heads over this issue. regards, /virendra
This appears to end users like some IPs on destination networks are reachable, while some adjacent ones are not.
I suspect this is due to a failed LAG leg or ECMP leg on Level3's side.
We were able to route around the problem for now.
Cheers, jof _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Have you seen this issue with any other carriers? We have seen a *very* similar issue with several Comcast-announced routes in the Midwest and we've never been able to track it down to any "for sure" type of problem. -Bill
Hi -
On 03/09/2011 11:21 AM, Jonathan Lassoff wrote:
We've been observing a small about of intermittent packet loss to/from AT&T and Level3 here in the San Francisco Bay Area this morning. In examining the situation, it seems like a evenly-spaced dropping of packets to certain IPs.
We experienced similar issue last week off our level3 connection for a specific /21 customer block. After further troubleshooting we decided to re-route our /21 to the other provider where packet loss issue went away. We are still scratching our heads over this issue.
regards, /virendra
Level3 is reporting that it may be an issue with a core switch experiencing high CPU utilization. Regards Neal Lewis "Bill Wichers" <billw@waveform.n et> To Sent by: <virendra.rode@outages.org>, outages-bounces@o <outages@outages.org> utages.org cc Subject 03/09/2011 02:02 Re: [outages] AT&T <-> Level3 PM Packet Loss in SF Bay Area Have you seen this issue with any other carriers? We have seen a *very* similar issue with several Comcast-announced routes in the Midwest and we've never been able to track it down to any "for sure" type of problem. -Bill
Hi -
On 03/09/2011 11:21 AM, Jonathan Lassoff wrote:
We've been observing a small about of intermittent packet loss to/from AT&T and Level3 here in the San Francisco Bay Area this morning. In examining the situation, it seems like a evenly-spaced dropping of packets to certain IPs.
We experienced similar issue last week off our level3 connection for a specific /21 customer block. After further troubleshooting we decided to re-route our /21 to the other provider where packet loss issue went away. We are still scratching our heads over this issue.
regards, /virendra
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Wed, Mar 9, 2011 at 1:46 PM, virendra rode <virendra.rode@outages.org> wrote:
Hi -
On 03/09/2011 11:21 AM, Jonathan Lassoff wrote:
We've been observing a small about of intermittent packet loss to/from AT&T and Level3 here in the San Francisco Bay Area this morning. In examining the situation, it seems like a evenly-spaced dropping of packets to certain IPs.
------------------- We experienced similar issue last week off our level3 connection for a specific /21 customer block. After further troubleshooting we decided to re-route our /21 to the other provider where packet loss issue went away. We are still scratching our heads over this issue.
Hrm. That's been my solution for this type of problem as well: just temporarily switch providers in both directions. This can be a little tricker if course if you're not a directly-downstream customer of the affected transit AS. I've been able to identify this problem by running mtr or traceroute simultaneously to several (at least 16+) IPs that are adjacent in a remote network and seeing different source IPs coming back for the ICMP type 11/code 0 (TTL Exceeded in transit) responses, and correlating loss to some subset of those IPs. Of course control plane policing or loaded CPUs on remote routers can confound these numbers. I feel like MTR does this especially well. Hope that helps. Good luck. Enjoy your routers.
According to Level 3 They have routed traffic from the problem module in the switch, and high CPU utilization has dropped. They are currently checking with their customers to confirm improvement. Regards Neal Lewis Jonathan Lassoff <jof@thejof.com> Sent by: To outages-bounces@o virendra.rode@outages.org utages.org cc outages@outages.org Subject 03/09/2011 03:39 Re: [outages] AT&T <-> Level3 PM Packet Loss in SF Bay Area On Wed, Mar 9, 2011 at 1:46 PM, virendra rode <virendra.rode@outages.org> wrote:
Hi -
On 03/09/2011 11:21 AM, Jonathan Lassoff wrote:
We've been observing a small about of intermittent packet loss to/from AT&T and Level3 here in the San Francisco Bay Area this morning. In examining the situation, it seems like a evenly-spaced dropping of packets to certain IPs.
------------------- We experienced similar issue last week off our level3 connection for a specific /21 customer block. After further troubleshooting we decided to re-route our /21 to the other provider where packet loss issue went away. We are still scratching our heads over this issue.
Hrm. That's been my solution for this type of problem as well: just temporarily switch providers in both directions. This can be a little tricker if course if you're not a directly-downstream customer of the affected transit AS. I've been able to identify this problem by running mtr or traceroute simultaneously to several (at least 16+) IPs that are adjacent in a remote network and seeing different source IPs coming back for the ICMP type 11/code 0 (TTL Exceeded in transit) responses, and correlating loss to some subset of those IPs. Of course control plane policing or loaded CPUs on remote routers can confound these numbers. I feel like MTR does this especially well. Hope that helps. Good luck. Enjoy your routers. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On 03/09/2011 03:46 PM, Cornelius_Lewis@ahm.honda.com wrote:
According to Level 3
They have routed traffic from the problem module in the switch, and high CPU utilization has dropped. They are currently checking with their customers to confirm improvement.
Regards
Neal Lewis
Thanks Neal! regards, /virendra
Jonathan Lassoff <jof@thejof.com> Sent by: To outages-bounces@o virendra.rode@outages.org utages.org cc outages@outages.org Subject 03/09/2011 03:39 Re: [outages] AT&T<-> Level3 PM Packet Loss in SF Bay Area
On Wed, Mar 9, 2011 at 1:46 PM, virendra rode<virendra.rode@outages.org> wrote:
Hi -
On 03/09/2011 11:21 AM, Jonathan Lassoff wrote:
We've been observing a small about of intermittent packet loss to/from AT&T and Level3 here in the San Francisco Bay Area this morning. In examining the situation, it seems like a evenly-spaced dropping of packets to certain IPs.
------------------- We experienced similar issue last week off our level3 connection for a specific /21 customer block. After further troubleshooting we decided to re-route our /21 to the other provider where packet loss issue went away. We are still scratching our heads over this issue.
Hrm. That's been my solution for this type of problem as well: just temporarily switch providers in both directions. This can be a little tricker if course if you're not a directly-downstream customer of the affected transit AS.
I've been able to identify this problem by running mtr or traceroute simultaneously to several (at least 16+) IPs that are adjacent in a remote network and seeing different source IPs coming back for the ICMP type 11/code 0 (TTL Exceeded in transit) responses, and correlating loss to some subset of those IPs. Of course control plane policing or loaded CPUs on remote routers can confound these numbers.
I feel like MTR does this especially well.
Hope that helps. Good luck. Enjoy your routers. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (5)
-
Bill Wichers -
Cornelius_Lewis＠ahm.honda.com -
Franck Martin -
Jonathan Lassoff -
virendra rode