We are seeing some issues w/ XO/Savvis peering.. Trace from XO to Savvis IP space (64.75.10.151) Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. scrubbed 0.0% 6 0.6 0.5 0.4 0.6 0.1 2. ip65-44-114-97.z114-44-65.customer.algx.net 0.0% 6 1.3 1.3 1.2 1.4 0.1 3. ??? Trace from Savvis to XO IP space (65.44.114.97) 1. scrubbed 0.0% 38 0.4 0.4 0.3 0.5 0.1 2. 64.41.199.129 0.0% 37 1.0 24.0 0.6 330.2 80.4 3. hr1-ge-7-47.santaclarasc5.savvis.net 0.0% 37 0.7 1.4 0.6 27.3 4.4 4. er1-te-1-0-0.sanjose3equinix.savvis.net 0.0% 37 0.7 5.2 0.6 140.3 23.2 5. cr1-tenge-0-7-5-0.sanfrancisco.savvis.net 2.7% 37 2.9 4.0 2.6 16.6 2.5 6. cr2-pos-0-0-3-3.dallas.savvis.net 0.0% 37 42.6 43.1 42.3 51.4 1.4 7. dpr1-ge-4-0-0.dallasequinix.savvis.net 0.0% 37 43.1 44.8 42.9 76.9 6.7 8. er1-te-2-1.dallasequinix.savvis.net 0.0% 37 43.3 49.2 42.8 233.6 31.6 9. 208.175.175.90 0.0% 37 43.0 42.8 42.6 43.6 0.2 10. 65.106.1.102 75.0% 37 43.5 46.5 43.4 62.9 6.3 11. 65.106.1.101 0.0% 37 43.4 47.8 43.2 112.3 12.5 12. 65.106.0.41 0.0% 37 57.5 65.1 57.1 177.3 21.0 13. 65.106.1.73 0.0% 37 57.4 66.5 57.1 162.1 24.2 14. ??? Trying to call into XO and they aren't even taking calls, they mention something about network issues in Spokane. Any ideas as to what is going on/ETA to fix? --Justin
Also seeing some issues with XO out here: Tracing route to yahoo.com [68.180.206.184] over a maximum of 30 hops: 1 2 ms 1 ms 1 ms 10.100.20.1 2 2 ms 1 ms 2 ms sjcisr01-int.wyse.com [10.100.1.15] 3 3 ms 2 ms 2 ms 132.237.245.1 4 3 ms 3 ms 2 ms 207.88.3.37.ptr.us.xo.net [207.88.3.37] 5 7 ms 3 ms 3 ms p3-0-0.mar2.fremont-ca.us.xo.net [207.88.80.181] 6 16 ms 5 ms 3 ms p4-0-0.rar2.sanjose-ca.us.xo.net [65.106.5.137] 7 12 ms 14 ms 11 ms p6-0-0.rar1.la-ca.us.xo.net [65.106.0.17] 8 17 ms 11 ms 11 ms p0-0-0d0.rar2.la-ca.us.xo.net [65.106.1.50] 9 44 ms 50 ms 52 ms p6-0-0.rar1.dallas-tx.us.xo.net [65.106.0.13] 10 * * * Request timed out. 11 44 ms 44 ms 44 ms 207.88.12.150.ptr.us.xo.net [207.88.12.150] 12 53 ms 85 ms 52 ms 206.111.5.42.ptr.us.xo.net [206.111.5.42] ^C On Mon, Sep 22, 2008 at 2:35 PM, Justin Sharp <sharp@sharpone.net> wrote:
We are seeing some issues w/ XO/Savvis peering..
Trace from XO to Savvis IP space (64.75.10.151)
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. scrubbed 0.0% 6 0.6 0.5 0.4 0.6 0.1 2. ip65-44-114-97.z114-44-65.customer.algx.net 0.0% 6 1.3 1.3 1.2 1.4 0.1 3. ???
Trace from Savvis to XO IP space (65.44.114.97)
1. scrubbed 0.0% 38 0.4 0.4 0.3 0.5 0.1 2. 64.41.199.129 0.0% 37 1.0 24.0 0.6 330.2 80.4 3. hr1-ge-7-47.santaclarasc5.savvis.net 0.0% 37 0.7 1.4 0.6 27.3 4.4 4. er1-te-1-0-0.sanjose3equinix.savvis.net 0.0% 37 0.7 5.2 0.6 140.3 23.2 5. cr1-tenge-0-7-5-0.sanfrancisco.savvis.net 2.7% 37 2.9 4.0 2.6 16.6 2.5 6. cr2-pos-0-0-3-3.dallas.savvis.net 0.0% 37 42.6 43.1 42.3 51.4 1.4 7. dpr1-ge-4-0-0.dallasequinix.savvis.net 0.0% 37 43.1 44.8 42.9 76.9 6.7 8. er1-te-2-1.dallasequinix.savvis.net 0.0% 37 43.3 49.2 42.8 233.6 31.6 9. 208.175.175.90 0.0% 37 43.0 42.8 42.6 43.6 0.2 10. 65.106.1.102 75.0% 37 43.5 46.5 43.4 62.9 6.3 11. 65.106.1.101 0.0% 37 43.4 47.8 43.2 112.3 12.5 12. 65.106.0.41 0.0% 37 57.5 65.1 57.1 177.3 21.0 13. 65.106.1.73 0.0% 37 57.4 66.5 57.1 162.1 24.2 14. ???
Trying to call into XO and they aren't even taking calls, they mention something about network issues in Spokane. Any ideas as to what is going on/ETA to fix?
--Justin
Seems like Savvis/XO routes have been restored, albeit through an unusual Dallas route (usually takes an XO Los Angeles route).. 1. scrubbed 0.0% 204 0.4 0.7 0.3 7.0 0.9 2. ip65-44-114-97.z114-44-65.customer.algx.net 0.0% 204 1.8 1.8 1.1 8.4 1.2 3. ip65-46-48-29.z48-46-65.customer.algx.net 0.0% 204 3.3 6.3 2.8 96.4 9.0 4. p6-0-0-0.mar1.saltlake-ut.us.xo.net 0.5% 204 8.1 14.4 3.1 161.0 26.3 5. p4-1-0-0.rar1.denver-co.us.xo.net 0.0% 204 15.5 25.6 15.3 191.4 26.7 6. p0-0-0d0.rar2.denver-co.us.xo.net 0.0% 204 16.2 22.4 16.1 129.2 14.4 7. p1-0-0.rar2.dallas-tx.us.xo.net 0.0% 204 45.3 38.3 29.9 201.3 21.3 8. te-3-4-0.rar3.dallas-tx.us.xo.net 66.0% 204 30.3 34.2 29.6 116.2 11.6 9. 207.88.12.146.ptr.us.xo.net 0.0% 204 33.1 32.7 29.4 106.7 8.5 10. 208.175.175.89 0.0% 204 29.8 35.7 29.5 195.4 20.0 11. dpr1-ge-2-0-0.dallasequinix.savvis.net 0.0% 204 34.3 33.8 29.6 115.2 10.5 12. cr2-tengige0-7-5-0.dallas.savvis.net 1.5% 204 30.2 33.3 29.7 68.6 5.4 13. cr2-loopback.sfo.savvis.net 0.0% 203 75.1 209.8 74.0 3270. 493.7 14. er1-te-2-0-1.sanjose3equinix.savvis.net 0.0% 203 71.8 75.6 71.7 219.5 11.7 15. hr1-te-2-0-0.santaclarasc5.savvis.net 0.0% 203 72.1 75.5 72.0 194.0 9.4 16. 216.34.2.226 0.0% 203 72.3 105.4 72.1 723.7 102.5 17. 64.41.199.140 28.6% 203 77.5 74.9 72.0 183.1 9.7 18. scrubbed 0.0% 203 73.8 76.0 72.4 207.9 11.3 Hop 8 looks busy.. --Justin Mike Lyon wrote:
Also seeing some issues with XO out here:
Tracing route to yahoo.com [68.180.206.184] over a maximum of 30 hops:
1 2 ms 1 ms 1 ms 10.100.20.1 2 2 ms 1 ms 2 ms sjcisr01-int.wyse.com [10.100.1.15] 3 3 ms 2 ms 2 ms 132.237.245.1 4 3 ms 3 ms 2 ms 207.88.3.37.ptr.us.xo.net [207.88.3.37] 5 7 ms 3 ms 3 ms p3-0-0.mar2.fremont-ca.us.xo.net [207.88.80.181]
6 16 ms 5 ms 3 ms p4-0-0.rar2.sanjose-ca.us.xo.net [65.106.5.137]
7 12 ms 14 ms 11 ms p6-0-0.rar1.la-ca.us.xo.net [65.106.0.17] 8 17 ms 11 ms 11 ms p0-0-0d0.rar2.la-ca.us.xo.net [65.106.1.50] 9 44 ms 50 ms 52 ms p6-0-0.rar1.dallas-tx.us.xo.net [65.106.0.13] 10 * * * Request timed out. 11 44 ms 44 ms 44 ms 207.88.12.150.ptr.us.xo.net [207.88.12.150] 12 53 ms 85 ms 52 ms 206.111.5.42.ptr.us.xo.net [206.111.5.42] ^C
On Mon, Sep 22, 2008 at 2:35 PM, Justin Sharp <sharp@sharpone.net> wrote:
We are seeing some issues w/ XO/Savvis peering..
Trace from XO to Savvis IP space (64.75.10.151)
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. scrubbed 0.0% 6 0.6 0.5 0.4 0.6 0.1 2. ip65-44-114-97.z114-44-65.customer.algx.net 0.0% 6 1.3 1.3 1.2 1.4 0.1 3. ???
Trace from Savvis to XO IP space (65.44.114.97)
1. scrubbed 0.0% 38 0.4 0.4 0.3 0.5 0.1 2. 64.41.199.129 0.0% 37 1.0 24.0 0.6 330.2 80.4 3. hr1-ge-7-47.santaclarasc5.savvis.net 0.0% 37 0.7 1.4 0.6 27.3 4.4 4. er1-te-1-0-0.sanjose3equinix.savvis.net 0.0% 37 0.7 5.2 0.6 140.3 23.2 5. cr1-tenge-0-7-5-0.sanfrancisco.savvis.net 2.7% 37 2.9 4.0 2.6 16.6 2.5 6. cr2-pos-0-0-3-3.dallas.savvis.net 0.0% 37 42.6 43.1 42.3 51.4 1.4 7. dpr1-ge-4-0-0.dallasequinix.savvis.net 0.0% 37 43.1 44.8 42.9 76.9 6.7 8. er1-te-2-1.dallasequinix.savvis.net 0.0% 37 43.3 49.2 42.8 233.6 31.6 9. 208.175.175.90 0.0% 37 43.0 42.8 42.6 43.6 0.2 10. 65.106.1.102 75.0% 37 43.5 46.5 43.4 62.9 6.3 11. 65.106.1.101 0.0% 37 43.4 47.8 43.2 112.3 12.5 12. 65.106.0.41 0.0% 37 57.5 65.1 57.1 177.3 21.0 13. 65.106.1.73 0.0% 37 57.4 66.5 57.1 162.1 24.2 14. ???
Trying to call into XO and they aren't even taking calls, they mention something about network issues in Spokane. Any ideas as to what is going on/ETA to fix?
--Justin
We're a major XO shop, IP and voip. They're whole IP net seems to have been down from about 17:20 EDT to about 17:30 - 17:35 EDT. I'm sure the echo effects will last a bit as this was pretty major. I don't know the cause. It affected data and VoIP as well all over the US. -----Original Message----- From: Mike Lyon [mailto:mike.lyon@gmail.com] Sent: Monday, September 22, 2008 5:44 PM To: Justin Sharp Cc: outages@outages.org; Nanog Mailing list Subject: Re: [outages] XO Outage Also seeing some issues with XO out here: Tracing route to yahoo.com [68.180.206.184] over a maximum of 30 hops: 1 2 ms 1 ms 1 ms 10.100.20.1 2 2 ms 1 ms 2 ms sjcisr01-int.wyse.com [10.100.1.15] 3 3 ms 2 ms 2 ms 132.237.245.1 4 3 ms 3 ms 2 ms 207.88.3.37.ptr.us.xo.net [207.88.3.37] 5 7 ms 3 ms 3 ms p3-0-0.mar2.fremont-ca.us.xo.net [207.88.80.181] 6 16 ms 5 ms 3 ms p4-0-0.rar2.sanjose-ca.us.xo.net [65.106.5.137] 7 12 ms 14 ms 11 ms p6-0-0.rar1.la-ca.us.xo.net [65.106.0.17] 8 17 ms 11 ms 11 ms p0-0-0d0.rar2.la-ca.us.xo.net [65.106.1.50] 9 44 ms 50 ms 52 ms p6-0-0.rar1.dallas-tx.us.xo.net [65.106.0.13] 10 * * * Request timed out. 11 44 ms 44 ms 44 ms 207.88.12.150.ptr.us.xo.net [207.88.12.150] 12 53 ms 85 ms 52 ms 206.111.5.42.ptr.us.xo.net [206.111.5.42] ^C On Mon, Sep 22, 2008 at 2:35 PM, Justin Sharp <sharp@sharpone.net> wrote:
We are seeing some issues w/ XO/Savvis peering..
Trace from XO to Savvis IP space (64.75.10.151)
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. scrubbed 0.0% 6 0.6 0.5 0.4 0.6 0.1 2. ip65-44-114-97.z114-44-65.customer.algx.net 0.0% 6 1.3 1.3 1.2 1.4 0.1 3. ???
Trace from Savvis to XO IP space (65.44.114.97)
1. scrubbed 0.0% 38 0.4 0.4 0.3 0.5 0.1 2. 64.41.199.129 0.0% 37 1.0 24.0 0.6 330.2 80.4 3. hr1-ge-7-47.santaclarasc5.savvis.net 0.0% 37 0.7 1.4 0.6 27.3 4.4 4. er1-te-1-0-0.sanjose3equinix.savvis.net 0.0% 37 0.7 5.2 0.6 140.3 23.2 5. cr1-tenge-0-7-5-0.sanfrancisco.savvis.net 2.7% 37 2.9 4.0 2.6 16.6 2.5 6. cr2-pos-0-0-3-3.dallas.savvis.net 0.0% 37 42.6 43.1 42.3 51.4 1.4 7. dpr1-ge-4-0-0.dallasequinix.savvis.net 0.0% 37 43.1 44.8 42.9 76.9 6.7 8. er1-te-2-1.dallasequinix.savvis.net 0.0% 37 43.3 49.2 42.8 233.6 31.6 9. 208.175.175.90 0.0% 37 43.0 42.8 42.6 43.6 0.2 10. 65.106.1.102 75.0% 37 43.5 46.5 43.4 62.9 6.3 11. 65.106.1.101 0.0% 37 43.4 47.8 43.2 112.3 12.5 12. 65.106.0.41 0.0% 37 57.5 65.1 57.1 177.3 21.0 13. 65.106.1.73 0.0% 37 57.4 66.5 57.1 162.1 24.2 14. ???
Trying to call into XO and they aren't even taking calls, they mention something about network issues in Spokane. Any ideas as to what is going on/ETA to fix?
--Justin
_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I am seeing it on my end also: traceroute: Warning: www.cnn.com has multiple addresses; using 157.166.224.25 traceroute to www.cnn.com (157.166.224.25), 64 hops max, 40 byte packets 1 hq-rtr1.genius.local (64.244.66.1) 0.891 ms 0.429 ms 0.449 ms 2 ip65-46-253-157.z253-46-65.customer.algx.net (65.46.253.157) 1.856 ms 2.860 ms 1.881 ms 3 p3-0-0.mar2.fremont-ca.us.xo.net (207.88.80.181) 16.922 ms 2.041 ms 2.013 ms 4 p4-3-0.rar2.sanjose-ca.us.xo.net (65.106.5.161) 2.637 ms 2.192 ms 2.823 ms 5 p6-0-0.rar1.la-ca.us.xo.net (65.106.0.17) 10.308 ms 10.258 ms 10.386 ms 6 207.88.13.22.ptr.us.xo.net (207.88.13.22) 10.931 ms 10.535 ms 10.037 ms 7 *^C Justin Sharp wrote:
We are seeing some issues w/ XO/Savvis peering..
Trace from XO to Savvis IP space (64.75.10.151)
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. scrubbed
0.0% 6 0.6 0.5 0.4 0.6 0.1 2. ip65-44-114-97.z114-44-65.customer.algx.net 0.0% 6 1.3 1.3 1.2 1.4 0.1 3. ???
Trace from Savvis to XO IP space (65.44.114.97)
1. scrubbed
0.0% 38 0.4 0.4 0.3 0.5 0.1 2. 64.41.199.129 0.0% 37 1.0 24.0 0.6 330.2 80.4 3. hr1-ge-7-47.santaclarasc5.savvis.net 0.0% 37 0.7 1.4 0.6 27.3 4.4 4. er1-te-1-0-0.sanjose3equinix.savvis.net 0.0% 37 0.7 5.2 0.6 140.3 23.2 5. cr1-tenge-0-7-5-0.sanfrancisco.savvis.net 2.7% 37 2.9 4.0 2.6 16.6 2.5 6. cr2-pos-0-0-3-3.dallas.savvis.net 0.0% 37 42.6 43.1 42.3 51.4 1.4 7. dpr1-ge-4-0-0.dallasequinix.savvis.net 0.0% 37 43.1 44.8 42.9 76.9 6.7 8. er1-te-2-1.dallasequinix.savvis.net 0.0% 37 43.3 49.2 42.8 233.6 31.6 9. 208.175.175.90 0.0% 37 43.0 42.8 42.6 43.6 0.2 10. 65.106.1.102 75.0% 37 43.5 46.5 43.4 62.9 6.3 11. 65.106.1.101 0.0% 37 43.4 47.8 43.2 112.3 12.5 12. 65.106.0.41 0.0% 37 57.5 65.1 57.1 177.3 21.0 13. 65.106.1.73 0.0% 37 57.4 66.5 57.1 162.1 24.2 14. ???
Trying to call into XO and they aren't even taking calls, they mention something about network issues in Spokane. Any ideas as to what is going on/ETA to fix?
--Justin
Seeing outage here in San Diego. On Sep 22, 2008, at 2:35 PM, Justin Sharp wrote:
We are seeing some issues w/ XO/Savvis peering..
Trace from XO to Savvis IP space (64.75.10.151)
Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss % Snt Last Avg Best Wrst StDev 1. scrubbed 0.0 % 6 0.6 0.5 0.4 0.6 0.1 2. ip65 -44 -114 -97 .z114 -44 -65 .customer .algx.net 0.0% 6 1.3 1.3 1.2 1.4 0.1 3. ???
Trace from Savvis to XO IP space (65.44.114.97)
1. scrubbed 0.0 % 38 0.4 0.4 0.3 0.5 0.1 2. 64.41.199.129 0.0 % 37 1.0 24.0 0.6 330.2 80.4 3. hr1- ge -7 -47 .santaclarasc5 .savvis .net 0.0% 37 0.7 1.4 0.6 27.3 4.4 4. er1- te -1 -0 -0 .sanjose3equinix .savvis .net 0.0% 37 0.7 5.2 0.6 140.3 23.2 5. cr1- tenge -0 -7 -5 -0 .sanfrancisco .savvis .net 2.7% 37 2.9 4.0 2.6 16.6 2.5 6. cr2- pos -0 -0 -3 -3 .dallas .savvis .net
0.0% 37 42.6 43.1 42.3 51.4 1.4 7. dpr1- ge -4 -0 -0 .dallasequinix .savvis .net 0.0% 37 43.1 44.8 42.9 76.9 6.7 8. er1- te -2 -1 .dallasequinix .savvis .net 0.0% 37 43.3 49.2 42.8 233.6 31.6 9. 208.175.175.90 0.0 % 37 43.0 42.8 42.6 43.6 0.2 10. 65.106.1.102 75.0 % 37 43.5 46.5 43.4 62.9 6.3 11. 65.106.1.101 0.0 % 37 43.4 47.8 43.2 112.3 12.5 12. 65.106.0.41 0.0 % 37 57.5 65.1 57.1 177.3 21.0 13. 65.106.1.73 0.0 % 37 57.4 66.5 57.1 162.1 24.2 14. ???
Trying to call into XO and they aren't even taking calls, they mention something about network issues in Spokane. Any ideas as to what is going on/ETA to fix?
--Justin
-- Paul Norton Systems Administrator Neoverve - www.neoverve.com Neoverve Blog - http://blog.neoverve.com/
Little late on my part, but yes... XO partitioned for 20-30 mins today. Said it was a peering issue. I'll share the RFO when I get it. Sent from my BlackBerry® smartphone with SprintSpeed -----Original Message----- From: Paul Norton <paul@neoverve.com> Date: Mon, 22 Sep 2008 14:53:43 To: Justin Sharp<sharp@sharpone.net> Cc: <outages@outages.org> Subject: Re: [outages] XO Outage Seeing outage here in San Diego. On Sep 22, 2008, at 2:35 PM, Justin Sharp wrote:
We are seeing some issues w/ XO/Savvis peering..
Trace from XO to Savvis IP space (64.75.10.151)
Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss % Snt Last Avg Best Wrst StDev 1. scrubbed 0.0 % 6 0.6 0.5 0.4 0.6 0.1 2. ip65 -44 -114 -97 .z114 -44 -65 .customer .algx.net 0.0% 6 1.3 1.3 1.2 1.4 0.1 3. ???
Trace from Savvis to XO IP space (65.44.114.97)
1. scrubbed 0.0 % 38 0.4 0.4 0.3 0.5 0.1 2. 64.41.199.129 0.0 % 37 1.0 24.0 0.6 330.2 80.4 3. hr1- ge -7 -47 .santaclarasc5 .savvis .net 0.0% 37 0.7 1.4 0.6 27.3 4.4 4. er1- te -1 -0 -0 .sanjose3equinix .savvis .net 0.0% 37 0.7 5.2 0.6 140.3 23.2 5. cr1- tenge -0 -7 -5 -0 .sanfrancisco .savvis .net 2.7% 37 2.9 4.0 2.6 16.6 2.5 6. cr2- pos -0 -0 -3 -3 .dallas .savvis .net
0.0% 37 42.6 43.1 42.3 51.4 1.4 7. dpr1- ge -4 -0 -0 .dallasequinix .savvis .net 0.0% 37 43.1 44.8 42.9 76.9 6.7 8. er1- te -2 -1 .dallasequinix .savvis .net 0.0% 37 43.3 49.2 42.8 233.6 31.6 9. 208.175.175.90 0.0 % 37 43.0 42.8 42.6 43.6 0.2 10. 65.106.1.102 75.0 % 37 43.5 46.5 43.4 62.9 6.3 11. 65.106.1.101 0.0 % 37 43.4 47.8 43.2 112.3 12.5 12. 65.106.0.41 0.0 % 37 57.5 65.1 57.1 177.3 21.0 13. 65.106.1.73 0.0 % 37 57.4 66.5 57.1 162.1 24.2 14. ???
Trying to call into XO and they aren't even taking calls, they mention something about network issues in Spokane. Any ideas as to what is going on/ETA to fix?
--Justin
-- Paul Norton Systems Administrator Neoverve - www.neoverve.com Neoverve Blog - http://blog.neoverve.com/ _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (6)
-
blackham＠gmail.com -
Justin Sharp -
Michael R. Self -
Mike Lyon -
Paul Norton -
Zaid Ali