SoftLayer Latency Jump 4x last 30 minutes
Anyone else seeing increased latency to SoftLayer destinations? Beginning at 2:33:25pm EST and continuing for the last 26 minutes, Round Trip Time jumped from a fairly constant 36-44ms to 142-225ms. This may not seem like much, but it impacts Voice Quality for VoIP calls. HOST: www.nova.org Loss% Snt Last Avg Best Wrst StDev 1. xl0.tic.nova.org 0.0% 15 0.2 0.2 0.1 0.2 0.0 2. wsip-70-168-111-17.dc.dc.cox 0.0% 15 1.4 16.4 0.9 204.2 52.4 3. ip68-100-3-109.dc.dc.cox.net 0.0% 15 0.9 1.1 0.8 2.3 0.5 4. ashbbprj01-ae2.rd.as.cox.net 0.0% 15 1.6 2.3 1.5 8.7 1.8 5. xe-1-0-2.bbr02.eq01.wdc02.ne 0.0% 15 3.9 2.1 1.7 3.9 0.6 6. ae1.bbr02.eq01.chi01.network 0.0% 15 225.8 225.4 223.3 226.1 0.8 7. ae7.bbr01.eq01.chi01.network 6.7% 15 226.4 226.1 225.3 226.7 0.4 8. ae20.bbr01.eq01.dal03.networ 0.0% 15 142.2 143.0 141.8 151.9 2.5 9. ae0.dar02.sr01.dal01.network 0.0% 15 142.1 142.0 140.6 142.5 0.5 10. po2.fcr03.sr04.dal01.network 6.7% 15 142.3 142.8 140.8 151.4 2.5 11. 67.228.7.104-static.reverse. 13.3% 15 142.5 142.3 141.5 142.5 0.3 --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------
On Feb 26, 2015, at 3:00 PM, Peter Beckman via Outages <outages@outages.org> wrote:
Anyone else seeing increased latency to SoftLayer destinations?
Beginning at 2:33:25pm EST and continuing for the last 26 minutes, Round Trip Time jumped from a fairly constant 36-44ms to 142-225ms. This may not seem like much, but it impacts Voice Quality for VoIP calls.
Funny you mention that, I’m ssh’d into a server in Dallas on SL and I was just about to check my cable modem to see why it’s feeling so laggy. Here’s more traceroutes. CV/OOL: 1 10.240.176.233 (10.240.176.233) 10.276 ms 7.123 ms 8.078 ms 2 67.59.225.244 (67.59.225.244) 9.073 ms 9.097 ms 7.532 ms 3 rtr1-ge1-10.mhe.prnynj.cv.net (67.83.242.1) 10.638 ms 10.773 ms ool-4353f209.dyn.optonline.net (67.83.242.9) 9.704 ms 4 64.15.7.41 (64.15.7.41) 11.120 ms 65.19.119.193 (65.19.119.193) 9.245 ms 64.15.7.57 (64.15.7.57) 10.095 ms 5 451be0de.cst.lightpath.net (65.19.120.222) 13.601 ms 11.940 ms 64.15.0.86 (64.15.0.86) 11.531 ms 6 rtr7-ge2-1.in.nycmnyzr.cv.net (64.15.0.106) 20.782 ms 19.739 ms 18.669 ms 7 te3-5.bbr01.eq01.wdc01.networklayer.com (206.126.236.185) 16.764 ms 17.737 ms 18.897 ms 8 ae7.bbr02.eq01.wdc02.networklayer.com (173.192.18.195) 21.071 ms 21.332 ms 16.649 ms 9 ae1.bbr02.eq01.chi01.networklayer.com (173.192.18.155) 139.147 ms 140.299 ms 138.833 ms 10 ae7.bbr01.eq01.chi01.networklayer.com (173.192.18.170) 139.805 ms 138.216 ms 137.771 ms 11 ae20.bbr01.eq01.dal03.networklayer.com (173.192.18.136) 169.957 ms 181.421 ms 166.108 ms 12 ae6.dar02.sr01.dal05.networklayer.com (50.97.18.193) 162.461 ms ae5.dar01.sr01.dal05.networklayer.com (173.192.18.215) 164.874 ms ae6.dar02.sr01.dal05.networklayer.com (50.97.18.193) 163.429 ms 13 po1.fcr02.sr02.dal05.networklayer.com (173.192.118.139) 168.017 ms po2.fcr02.sr02.dal05.networklayer.com (173.192.118.141) 165.814 ms 168.669 ms NAC (looks good): 2 vlan808.tbr2.oct.nac.net (209.123.10.41) 0.470 ms 1.003 ms 0.227 ms 3 0.e1-3.tbr1.oct.nac.net (209.123.10.45) 0.357 ms 0.e1-1.tbr2.mmu.nac.net (209.123.10.22) 0.421 ms 0.e1-3.tbr1.oct.nac.net (209.123.10.45) 0.467 ms 4 0.e1-4.tbr1.ewr.nac.net (209.123.10.121) 0.961 ms 6.215 ms 0.860 ms 5 0.e1-2.tbr1.tl9.nac.net (209.123.10.130) 5.856 ms 1.466 ms 2.235 ms 6 0.e2-1.pr2.tl9.nac.net (209.123.11.142) 1.234 ms 6.714 ms 1.360 ms 7 te1-7.bbr01.tl01.nyc01.networklayer.com (198.32.160.27) 1.358 ms 1.342 ms 1.361 ms 8 ae7.bbr02.tl01.nyc01.networklayer.com (173.192.18.177) 1.608 ms 1.592 ms 1.861 ms 9 ae1.bbr01.eq01.chi01.networklayer.com (173.192.18.132) 24.098 ms 23.452 ms 23.354 ms 10 ae20.bbr01.eq01.dal03.networklayer.com (173.192.18.136) 47.329 ms 47.066 ms 49.333 ms 11 ae6.dar02.sr01.dal05.networklayer.com (50.97.18.193) 44.711 ms ae5.dar01.sr01.dal05.networklayer.com (173.192.18.215) 50.813 ms ae6.dar02.sr01.dal05.networklayer.com (50.97.18.193) 47.320 ms 12 po2.fcr02.sr02.dal05.networklayer.com (173.192.118.141) 45.437 ms po1.fcr02.sr02.dal05.networklayer.com (173.192.118.139) 46.267 ms 45.218 ms 13 jag.devel.ettops.com (75.126.200.52) 47.188 ms 48.017 ms 48.072 ms Level3: 2 ge-9-16.car4.NewYork1.Level3.net (4.53.92.73) 0.750 ms 0.368 ms 0.319 ms 3 ae-3-80.edge1.Washington4.Level3.net (4.69.149.143) 6.213 ms ae-4-90.edge1.Washington4.Level3.net (4.69.149.207) 6.553 ms ae-3-80.edge1.Washington4.Level3.net (4.69.149.143) 6.497 ms 4 SOFTLAYER-T.edge1.Washington4.Level3.net (4.53.112.26) 5.762 ms 5.960 ms 6.034 ms 5 ae1.bbr02.eq01.chi01.networklayer.com (173.192.18.155) 123.806 ms 124.001 ms 130.060 ms 6 ae7.bbr01.eq01.chi01.networklayer.com (173.192.18.170) 123.679 ms 124.940 ms 124.669 ms 7 ae20.bbr01.eq01.dal03.networklayer.com (173.192.18.136) 144.650 ms 149.134 ms 149.576 ms 8 ae5.dar01.sr01.dal05.networklayer.com (173.192.18.215) 145.526 ms 145.449 ms 149.142 ms 9 po1.fcr02.sr02.dal05.networklayer.com (173.192.118.139) 149.053 ms 150.462 ms 146.063 ms 10 jag.devel.ettops.com (75.126.200.52) 145.958 ms 145.493 ms 145.753 ms Digital Ocean, London (looks OK): 2 5.101.111.237 (5.101.111.237) 1.183 ms 5.101.111.241 (5.101.111.241) 0.435 ms 5.101.111.233 (5.101.111.233) 0.399 ms 3 bbr01.lon01.networklayer.com (195.66.225.183) 1.581 ms 1.539 ms 1.382 ms 4 ae1.bbr02.tl01.nyc01.networklayer.com (50.97.18.204) 69.226 ms 69.132 ms 69.193 ms 5 ae1.bbr01.eq01.chi01.networklayer.com (173.192.18.132) 91.666 ms 91.268 ms 92.855 ms 6 ae20.bbr01.eq01.dal03.networklayer.com (173.192.18.136) 110.398 ms 110.168 ms 108.914 ms 7 ae5.dar01.sr01.dal05.networklayer.com (173.192.18.215) 110.246 ms ae6.dar02.sr01.dal05.networklayer.com (50.97.18.193) 114.383 ms ae5.dar01.sr01.dal05.networklayer.com (173.192.18.215) 114.327 ms 8 po1.fcr02.sr02.dal05.networklayer.com (173.192.118.139) 113.562 ms 112.323 ms 116.488 ms 9 jag.devel.ettops.com (75.126.200.52) 111.026 ms 112.016 ms 109.644 ms
HOST: www.nova.org Loss% Snt Last Avg Best Wrst StDev 1. xl0.tic.nova.org 0.0% 15 0.2 0.2 0.1 0.2 0.0 2. wsip-70-168-111-17.dc.dc.cox 0.0% 15 1.4 16.4 0.9 204.2 52.4 3. ip68-100-3-109.dc.dc.cox.net 0.0% 15 0.9 1.1 0.8 2.3 0.5 4. ashbbprj01-ae2.rd.as.cox.net 0.0% 15 1.6 2.3 1.5 8.7 1.8 5. xe-1-0-2.bbr02.eq01.wdc02.ne 0.0% 15 3.9 2.1 1.7 3.9 0.6 6. ae1.bbr02.eq01.chi01.network 0.0% 15 225.8 225.4 223.3 226.1 0.8 7. ae7.bbr01.eq01.chi01.network 6.7% 15 226.4 226.1 225.3 226.7 0.4 8. ae20.bbr01.eq01.dal03.networ 0.0% 15 142.2 143.0 141.8 151.9 2.5 9. ae0.dar02.sr01.dal01.network 0.0% 15 142.1 142.0 140.6 142.5 0.5 10. po2.fcr03.sr04.dal01.network 6.7% 15 142.3 142.8 140.8 151.4 2.5 11. 67.228.7.104-static.reverse. 13.3% 15 142.5 142.3 141.5 142.5 0.3
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We saw an increase in latency + packet loss from Dallas 1,5, and 9 into Washington 1 for the last 30 minutes. Appears to be improving now though. Andrew McLean ________________________________________ From: Outages [outages-bounces@outages.org] on behalf of Peter Beckman via Outages [outages@outages.org] Sent: Thursday, February 26, 2015 1:00 PM To: outages@outages.org Subject: [outages] SoftLayer Latency Jump 4x last 30 minutes Anyone else seeing increased latency to SoftLayer destinations? Beginning at 2:33:25pm EST and continuing for the last 26 minutes, Round Trip Time jumped from a fairly constant 36-44ms to 142-225ms. This may not seem like much, but it impacts Voice Quality for VoIP calls. HOST: www.nova.org Loss% Snt Last Avg Best Wrst StDev 1. xl0.tic.nova.org 0.0% 15 0.2 0.2 0.1 0.2 0.0 2. wsip-70-168-111-17.dc.dc.cox 0.0% 15 1.4 16.4 0.9 204.2 52.4 3. ip68-100-3-109.dc.dc.cox.net 0.0% 15 0.9 1.1 0.8 2.3 0.5 4. ashbbprj01-ae2.rd.as.cox.net 0.0% 15 1.6 2.3 1.5 8.7 1.8 5. xe-1-0-2.bbr02.eq01.wdc02.ne 0.0% 15 3.9 2.1 1.7 3.9 0.6 6. ae1.bbr02.eq01.chi01.network 0.0% 15 225.8 225.4 223.3 226.1 0.8 7. ae7.bbr01.eq01.chi01.network 6.7% 15 226.4 226.1 225.3 226.7 0.4 8. ae20.bbr01.eq01.dal03.networ 0.0% 15 142.2 143.0 141.8 151.9 2.5 9. ae0.dar02.sr01.dal01.network 0.0% 15 142.1 142.0 140.6 142.5 0.5 10. po2.fcr03.sr04.dal01.network 6.7% 15 142.3 142.8 140.8 151.4 2.5 11. 67.228.7.104-static.reverse. 13.3% 15 142.5 142.3 141.5 142.5 0.3 --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Thanks for the replies. It looks like it recovered at 3:28pm EST yesterday according to our logs. Average latency has increased by about 8% (36ms to 39ms RTT) since the issue. Beckman On Thu, 26 Feb 2015, Andrew McLean wrote:
We saw an increase in latency + packet loss from Dallas 1,5, and 9 into Washington 1 for the last 30 minutes. Appears to be improving now though.
________________________________________ From: Outages [outages-bounces@outages.org] on behalf of Peter Beckman via Outages [outages@outages.org] To: outages@outages.org Subject: [outages] SoftLayer Latency Jump 4x last 30 minutes
Anyone else seeing increased latency to SoftLayer destinations?
Beginning at 2:33:25pm EST and continuing for the last 26 minutes, Round Trip Time jumped from a fairly constant 36-44ms to 142-225ms. This may not seem like much, but it impacts Voice Quality for VoIP calls.
HOST: www.nova.org Loss% Snt Last Avg Best Wrst StDev 1. xl0.tic.nova.org 0.0% 15 0.2 0.2 0.1 0.2 0.0 2. wsip-70-168-111-17.dc.dc.cox 0.0% 15 1.4 16.4 0.9 204.2 52.4 3. ip68-100-3-109.dc.dc.cox.net 0.0% 15 0.9 1.1 0.8 2.3 0.5 4. ashbbprj01-ae2.rd.as.cox.net 0.0% 15 1.6 2.3 1.5 8.7 1.8 5. xe-1-0-2.bbr02.eq01.wdc02.ne 0.0% 15 3.9 2.1 1.7 3.9 0.6 6. ae1.bbr02.eq01.chi01.network 0.0% 15 225.8 225.4 223.3 226.1 0.8 7. ae7.bbr01.eq01.chi01.network 6.7% 15 226.4 226.1 225.3 226.7 0.4 8. ae20.bbr01.eq01.dal03.networ 0.0% 15 142.2 143.0 141.8 151.9 2.5 9. ae0.dar02.sr01.dal01.network 0.0% 15 142.1 142.0 140.6 142.5 0.5 10. po2.fcr03.sr04.dal01.network 6.7% 15 142.3 142.8 140.8 151.4 2.5 11. 67.228.7.104-static.reverse. 13.3% 15 142.5 142.3 141.5 142.5 0.3
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------
participants (3)
-
Andrew McLean -
Charles Sprickman -
Peter Beckman