peering in LA between XO and Savvis
We are seeing large (20-30%) packet loss in the forward and reverse path from XO to Savvis in the Los Angeles peering point. This was happening yesterday for about 2 hours, and has picked back up this morning. This has made some sites hosted in Savvis unusable for us (like Netsuite). Appropriate traces (some hops stripped to protect the innocent): From Draper UT to Savvis SC4 datacenter HOST: amds-mon Loss% Snt Last Avg Best Wrst StDev 1. stipped 0.0% 50 0.5 1.0 0.4 3.5 0.9 2. stripped 0.0% 50 29.1 10.7 0.2 38.6 13.9 3. stripped 0.0% 50 35.1 12.7 1.1 40.8 14.5 4. stripped 0.0% 50 43.1 21.8 2.9 48.9 15.8 5. p6-0-0d0.mar2.saltlake-ut.us 0.0% 50 40.0 25.2 3.1 155.9 25.8 6. p3-2-0d0.rar2.la-ca.us.xo.ne 0.0% 50 21.1 41.2 17.1 128.8 24.1 7. 207.88.12.157.ptr.us.xo.net 4.0% 50 26.6 34.9 17.8 60.9 15.7 8. 207.88.12.158.ptr.us.xo.net 0.0% 50 25.4 34.1 17.0 65.2 16.4 9. bpr2-so-6-1-0.losangeles.sav 30.0% 50 72.3 81.3 54.1 161.9 25.5 10. cr2-gig-0-7-5-2.losangeles.s 32.0% 50 86.1 80.3 57.0 107.8 17.0 11. cr2-pos0-0-0-0.sanfrancisco. 26.0% 50 76.7 90.9 65.5 116.8 16.3 12. er1-te-2-0-1.sanjose3equinix 44.0% 50 70.7 86.6 66.2 112.8 16.6 13. hr1-te-2-0-0.santaclarasc5.s 38.0% 50 70.8 86.2 66.6 116.5 16.8 14. 216.34.2.226 34.0% 50 92.1 174.0 66.8 823.9 203.2 15. stripped 36.0% 50 109.1 90.3 66.1 118.5 17.4 16. end-point 44.0% 50 107.1 89.7 66.8 115.6 18.9
From SC4 to same Draper UT network (reverse path, hop 16 is firewalled):
HOST: prod-monitor Loss% Snt Last Avg Best Wrst StDev 1. stripped 0.0% 50 0.4 0.4 0.3 0.5 0.1 2. stripped 0.0% 50 318.4 52.4 0.6 683.9 150.8 3. hr1-ge-7-47.santaclarasc5.sa 0.0% 50 0.9 3.3 0.6 128.2 18.0 4. hr1-te-1-0-1.santaclarasc4.s 0.0% 50 1.0 0.9 0.6 9.4 1.2 5. hr1-te-2-0-0.santaclarasc9.s 0.0% 50 1.1 1.8 0.6 28.6 4.1 6. pr1-ge-4-0-0.SanJoseEquinix. 0.0% 50 1.3 7.1 1.0 128.2 22.7 7. pr2-so-0-0-0.PaloAltoPaix.sa 0.0% 50 10.5 5.7 2.2 81.6 12.5 8. p2-2.IR1.PaloAlto-CA.us.xo.n 0.0% 50 2.7 2.3 2.0 3.1 0.2 9. 65.106.1.66.ptr.us.xo.net 0.0% 50 3.7 3.8 3.1 13.0 1.4 10. 65.106.1.65.ptr.us.xo.net 0.0% 50 15.7 6.1 3.0 35.9 7.8 11. p6-0-0.RAR1.LA-CA.us.xo.net 30.0% 50 56.4 57.8 49.9 91.1 8.7 12. p0-0-0d0.rar2.la-ca.us.xo.ne 34.0% 50 55.1 63.6 43.9 157.8 25.7 13. p1-0-0d0.mar2.saltlake-ut.us 42.0% 50 68.8 74.0 59.1 151.4 16.8 14. p15-0.chr1.saltlake-ut.us.xo 36.0% 50 72.8 68.2 60.6 76.3 3.6 15. stripped 36.0% 50 73.7 72.7 62.8 81.7 3.7 16. ??? 100.0 50 0.0 0.0 0.0 0.0 0.0 --Justin
After working with both vendors, it seems that a DDoS attack, aimed else where, which transited XO's peering router in LA was the root cause of this issue. Seems to be cleared up now. Justin Sharp wrote:
We are seeing large (20-30%) packet loss in the forward and reverse path from XO to Savvis in the Los Angeles peering point. This was happening yesterday for about 2 hours, and has picked back up this morning. This has made some sites hosted in Savvis unusable for us (like Netsuite).
Appropriate traces (some hops stripped to protect the innocent):
From Draper UT to Savvis SC4 datacenter
HOST: amds-mon Loss% Snt Last Avg Best Wrst StDev 1. stipped 0.0% 50 0.5 1.0 0.4 3.5 0.9 2. stripped 0.0% 50 29.1 10.7 0.2 38.6 13.9 3. stripped 0.0% 50 35.1 12.7 1.1 40.8 14.5 4. stripped 0.0% 50 43.1 21.8 2.9 48.9 15.8 5. p6-0-0d0.mar2.saltlake-ut.us 0.0% 50 40.0 25.2 3.1 155.9 25.8 6. p3-2-0d0.rar2.la-ca.us.xo.ne 0.0% 50 21.1 41.2 17.1 128.8 24.1 7. 207.88.12.157.ptr.us.xo.net 4.0% 50 26.6 34.9 17.8 60.9 15.7 8. 207.88.12.158.ptr.us.xo.net 0.0% 50 25.4 34.1 17.0 65.2 16.4 9. bpr2-so-6-1-0.losangeles.sav 30.0% 50 72.3 81.3 54.1 161.9 25.5 10. cr2-gig-0-7-5-2.losangeles.s 32.0% 50 86.1 80.3 57.0 107.8 17.0 11. cr2-pos0-0-0-0.sanfrancisco. 26.0% 50 76.7 90.9 65.5 116.8 16.3 12. er1-te-2-0-1.sanjose3equinix 44.0% 50 70.7 86.6 66.2 112.8 16.6 13. hr1-te-2-0-0.santaclarasc5.s 38.0% 50 70.8 86.2 66.6 116.5 16.8 14. 216.34.2.226 34.0% 50 92.1 174.0 66.8 823.9 203.2 15. stripped 36.0% 50 109.1 90.3 66.1 118.5 17.4 16. end-point 44.0% 50 107.1 89.7 66.8 115.6 18.9
From SC4 to same Draper UT network (reverse path, hop 16 is firewalled):
HOST: prod-monitor Loss% Snt Last Avg Best Wrst StDev
1. stripped 0.0% 50 0.4 0.4 0.3 0.5 0.1 2. stripped 0.0% 50 318.4 52.4 0.6 683.9 150.8 3. hr1-ge-7-47.santaclarasc5.sa 0.0% 50 0.9 3.3 0.6 128.2 18.0 4. hr1-te-1-0-1.santaclarasc4.s 0.0% 50 1.0 0.9 0.6 9.4 1.2 5. hr1-te-2-0-0.santaclarasc9.s 0.0% 50 1.1 1.8 0.6 28.6 4.1 6. pr1-ge-4-0-0.SanJoseEquinix. 0.0% 50 1.3 7.1 1.0 128.2 22.7 7. pr2-so-0-0-0.PaloAltoPaix.sa 0.0% 50 10.5 5.7 2.2 81.6 12.5 8. p2-2.IR1.PaloAlto-CA.us.xo.n 0.0% 50 2.7 2.3 2.0 3.1 0.2 9. 65.106.1.66.ptr.us.xo.net 0.0% 50 3.7 3.8 3.1 13.0 1.4 10. 65.106.1.65.ptr.us.xo.net 0.0% 50 15.7 6.1 3.0 35.9 7.8 11. p6-0-0.RAR1.LA-CA.us.xo.net 30.0% 50 56.4 57.8 49.9 91.1 8.7 12. p0-0-0d0.rar2.la-ca.us.xo.ne 34.0% 50 55.1 63.6 43.9 157.8 25.7 13. p1-0-0d0.mar2.saltlake-ut.us 42.0% 50 68.8 74.0 59.1 151.4 16.8 14. p15-0.chr1.saltlake-ut.us.xo 36.0% 50 72.8 68.2 60.6 76.3 3.6 15. stripped 36.0% 50 73.7 72.7 62.8 81.7 3.7 16. ??? 100.0 50 0.0 0.0 0.0 0.0 0.0
--Justin
_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (1)
-
Justin Sharp