Finally got the RFO: Summary: On Monday September 22, 2008 at 4:12 PM all XO Peering Routers were isolated due to a mis-configuration caused by human error. An XO technician inadvertently ran an incomplete script while attempting to complete non-impacting changes on an SNMP filter. XO data customers were unable to reach external websites following the scripting push. XO identified the issue immediately and began the process of logging into the Peering routers and corrected the configuration to restore the Peering router connectivity by 5:20 PM. As of this writing, XO operations is implementing a verification screen prior to script launching to ensure that only the intended scripts are being deployed. In addition, all scripting will require a Scheduled Maintenance Request or Repair event. Cause: Inadvertent mis-configuration caused by human error Resolution: XO technicians corrected configuration to restore peering router connectivity. Sent from my BlackBerry® smartphone with SprintSpeed -----Original Message----- From: blackham@gmail.com Date: Tue, 23 Sep 2008 00:41:42 To: Paul Norton<paul@neoverve.com>; <outages-bounces@outages.org>; Justin Sharp<sharp@sharpone.net> Cc: <outages@outages.org> Subject: Re: [outages] XO Outage Little late on my part, but yes... XO partitioned for 20-30 mins today. Said it was a peering issue. I'll share the RFO when I get it. Sent from my BlackBerry® smartphone with SprintSpeed -----Original Message----- From: Paul Norton <paul@neoverve.com> Date: Mon, 22 Sep 2008 14:53:43 To: Justin Sharp<sharp@sharpone.net> Cc: <outages@outages.org> Subject: Re: [outages] XO Outage Seeing outage here in San Diego. On Sep 22, 2008, at 2:35 PM, Justin Sharp wrote:

We are seeing some issues w/ XO/Savvis peering..

Trace from XO to Savvis IP space (64.75.10.151)

Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss % Snt Last Avg Best Wrst StDev 1. scrubbed 0.0 % 6 0.6 0.5 0.4 0.6 0.1 2. ip65 -44 -114 -97 .z114 -44 -65 .customer .algx.net 0.0% 6 1.3 1.3 1.2 1.4 0.1 3. ???

Trace from Savvis to XO IP space (65.44.114.97)

1. scrubbed 0.0 % 38 0.4 0.4 0.3 0.5 0.1 2. 64.41.199.129 0.0 % 37 1.0 24.0 0.6 330.2 80.4 3. hr1- ge -7 -47 .santaclarasc5 .savvis .net 0.0% 37 0.7 1.4 0.6 27.3 4.4 4. er1- te -1 -0 -0 .sanjose3equinix .savvis .net 0.0% 37 0.7 5.2 0.6 140.3 23.2 5. cr1- tenge -0 -7 -5 -0 .sanfrancisco .savvis .net 2.7% 37 2.9 4.0 2.6 16.6 2.5 6. cr2- pos -0 -0 -3 -3 .dallas .savvis .net

0.0% 37 42.6 43.1 42.3 51.4 1.4 7. dpr1- ge -4 -0 -0 .dallasequinix .savvis .net 0.0% 37 43.1 44.8 42.9 76.9 6.7 8. er1- te -2 -1 .dallasequinix .savvis .net 0.0% 37 43.3 49.2 42.8 233.6 31.6 9. 208.175.175.90 0.0 % 37 43.0 42.8 42.6 43.6 0.2 10. 65.106.1.102 75.0 % 37 43.5 46.5 43.4 62.9 6.3 11. 65.106.1.101 0.0 % 37 43.4 47.8 43.2 112.3 12.5 12. 65.106.0.41 0.0 % 37 57.5 65.1 57.1 177.3 21.0 13. 65.106.1.73 0.0 % 37 57.4 66.5 57.1 162.1 24.2 14. ???

Trying to call into XO and they aren't even taking calls, they mention something about network issues in Spokane. Any ideas as to what is going on/ETA to fix?

--Justin

-- Paul Norton Systems Administrator Neoverve - www.neoverve.com Neoverve Blog - http://blog.neoverve.com/ _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages