On 03/09/2011 03:46 PM, Cornelius_Lewis@ahm.honda.com wrote:
According to Level 3
They have routed traffic from the problem module in the switch, and high CPU utilization has dropped. They are currently checking with their customers to confirm improvement.
Regards
Neal Lewis
Thanks Neal! regards, /virendra
Jonathan Lassoff <jof@thejof.com> Sent by: To outages-bounces@o virendra.rode@outages.org utages.org cc outages@outages.org Subject 03/09/2011 03:39 Re: [outages] AT&T<-> Level3 PM Packet Loss in SF Bay Area
On Wed, Mar 9, 2011 at 1:46 PM, virendra rode<virendra.rode@outages.org> wrote:
Hi -
On 03/09/2011 11:21 AM, Jonathan Lassoff wrote:
We've been observing a small about of intermittent packet loss to/from AT&T and Level3 here in the San Francisco Bay Area this morning. In examining the situation, it seems like a evenly-spaced dropping of packets to certain IPs.
------------------- We experienced similar issue last week off our level3 connection for a specific /21 customer block. After further troubleshooting we decided to re-route our /21 to the other provider where packet loss issue went away. We are still scratching our heads over this issue.
Hrm. That's been my solution for this type of problem as well: just temporarily switch providers in both directions. This can be a little tricker if course if you're not a directly-downstream customer of the affected transit AS.
I've been able to identify this problem by running mtr or traceroute simultaneously to several (at least 16+) IPs that are adjacent in a remote network and seeing different source IPs coming back for the ICMP type 11/code 0 (TTL Exceeded in transit) responses, and correlating loss to some subset of those IPs. Of course control plane policing or loaded CPUs on remote routers can confound these numbers.
I feel like MTR does this especially well.
Hope that helps. Good luck. Enjoy your routers. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages