We have a colo in the El Segundo Savvis facility and as of about 30 minutes ago we started experiencing much reduced inbound traffic (email) and outbound traffic (websites) and large packet loss. Doing a traceroute to yahoo, I see: 2: gw.iv.ivenue.com (208.89.136.1) 0.617ms 3: no reply 4: hr2-v3008.la1.savvis.net (64.70.11.61) 0.810ms 5: 204.70.203.90 (204.70.203.90) 15.891ms 6: no reply 7: 208.178.58.221 (208.178.58.221) 13.790ms 8: no reply 9: no reply 10: YAHOOPEER.Te2-3.1101.ar5.CHI2.gblx.net (206.57.3.210) 67.495ms asymm 13 11: ae-7.pat1.nez.yahoo.com (216.115.104.124) 80.223ms asymm 15 12: xe-5-0-0.msr1.ne1.yahoo.com (216.115.100.1) 80.218ms asymm 16 13: xe-4-0-0.clr1-a-gdc.ne1.yahoo.com (98.138.144.21) 74.594ms asymm 17 Hop 5 is Savvis, hop 7 is Level 3. I do not know the geographical location of those routers. The master case number is 3877881, but I can't view any details of that case from their portal (and they "closed" mine that I created when they attached it to the master case). I know it's impacting the El Segundo and Orange County facilities, not sure about Burbank or any other locations. Is there something going on with either of these two (Savvis and/or Level 3) at the moment that anybody is aware of in the SoCal area? Details, we're looking for details. ...Todd -- The total budget at all receivers for solving senders' problems is $0. If you want them to accept your mail and manage it the way you want, send it the way the spec says to. --John Levine
On Wed, Apr 24, 2013 at 1:57 PM, Todd Lyons <tlyons@ivenue.com> wrote:
We have a colo in the El Segundo Savvis facility and as of about 30 minutes ago we started experiencing much reduced inbound traffic (email) and outbound traffic (websites) and large packet loss. Doing
Everything seems better as of about 5 minutes ago. I have no idea what was the problem. ...Todd -- The total budget at all receivers for solving senders' problems is $0. If you want them to accept your mail and manage it the way you want, send it the way the spec says to. --John Levine
Todd, http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N4... 1. Can you provide source and destination IPs? 2. Do you have a return path traceroute? This is incredibly important. 3. Can you use a utility like mtr instead, from both endspoints? 4. Is there some reason your below traceroute is in some format that looks almost hand-written and lacks any indication of packet loss (e.g. no "*" indicators)? It's not that I don't believe you, it's that the evidence needs to be presented. If you need a destination to reach (that hopefully utilises the network path you're seeing problems with), you can use my box: 206.125.172.42. I do not filter ICMP. If you provide an answer to #1 I can provide return-path results. But in general, testing against a site where you don't have return-path access/visibility makes troubleshooting very difficult. See the PDF I linked. :-) -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Wed, Apr 24, 2013 at 01:57:22PM -0700, Todd Lyons wrote:
We have a colo in the El Segundo Savvis facility and as of about 30 minutes ago we started experiencing much reduced inbound traffic (email) and outbound traffic (websites) and large packet loss. Doing a traceroute to yahoo, I see:
2: gw.iv.ivenue.com (208.89.136.1) 0.617ms 3: no reply 4: hr2-v3008.la1.savvis.net (64.70.11.61) 0.810ms 5: 204.70.203.90 (204.70.203.90) 15.891ms 6: no reply 7: 208.178.58.221 (208.178.58.221) 13.790ms 8: no reply 9: no reply 10: YAHOOPEER.Te2-3.1101.ar5.CHI2.gblx.net (206.57.3.210) 67.495ms asymm 13 11: ae-7.pat1.nez.yahoo.com (216.115.104.124) 80.223ms asymm 15 12: xe-5-0-0.msr1.ne1.yahoo.com (216.115.100.1) 80.218ms asymm 16 13: xe-4-0-0.clr1-a-gdc.ne1.yahoo.com (98.138.144.21) 74.594ms asymm 17
Hop 5 is Savvis, hop 7 is Level 3. I do not know the geographical location of those routers. The master case number is 3877881, but I can't view any details of that case from their portal (and they "closed" mine that I created when they attached it to the master case). I know it's impacting the El Segundo and Orange County facilities, not sure about Burbank or any other locations.
Is there something going on with either of these two (Savvis and/or Level 3) at the moment that anybody is aware of in the SoCal area? Details, we're looking for details.
...Todd -- The total budget at all receivers for solving senders' problems is $0. If you want them to accept your mail and manage it the way you want, send it the way the spec says to. --John Levine _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Traffic levels are back to normal. On my way to pick up my kid from school, Savvis called to verify the issue had been resolved. On Wed, Apr 24, 2013 at 2:27 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Todd,
http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N4...
I'll be more cognizant of these things, even when hair mode = fire.
1. Can you provide source and destination IPs?
nat'd src 208.89.136.20 to dst 98.139.183.24
2. Do you have a return path traceroute? This is incredibly important.
I was in too much of a hurry. You should be able to traceroute back to 64.14.201.31 or 208.89.138.21.
3. Can you use a utility like mtr instead, from both endspoints?
Next time I'll go out and back from our rackspace machine in HK. Obviously I would have to use something different as the remote since my trace was to yahoo.com and I am not a yahoo employee. I just picked something different from my co-worker (he used google) to make sure that in saying "the Internet is down", I wasn't just some idiot if it was just "google is down".
4. Is there some reason your below traceroute is in some format that looks almost hand-written and lacks any indication of packet loss (e.g. no "*" indicators)? It's not that I don't believe you, it's that the evidence needs to be presented.
Agreed. I used tracepath (CentOS 6.x) instead of mtr or traceroute. Again, too much of a hurry.
If you need a destination to reach (that hopefully utilises the network path you're seeing problems with), you can use my box: 206.125.172.42. I do not filter ICMP. If you provide an answer to #1 I can provide return-path results.
Thanks for the offer, but no need. The master ticket has been resolved, I'll know more technical details in 5 days or less.
But in general, testing against a site where you don't have return-path access/visibility makes troubleshooting very difficult. See the PDF I linked. :-)
Agreed. Too much going on at once and didn't get all the info out I should have. Thanks for the reply Jeremy, it's much appreciated. ...Todd -- The total budget at all receivers for solving senders' problems is $0. If you want them to accept your mail and manage it the way you want, send it the way the spec says to. --John Levine
participants (2)
-
Jeremy Chadwick -
Todd Lyons