September 2008 - Outages

Perhaps this explains why Qwest was having problems...
by netgeek 30 Sep '08

30 Sep '08

Firefighters from 6 departments battle blaze at Qwest building in Tukwila Nearly 75 firefighters from across King County worked together Monday afternoon to put out a three-alarm fire in the roof of a Qwest Communications building in Tukwila. By Noelene Clark Seattle Times staff reporter Scores of firefighters from across King County worked together Monday afternoon to put out a three-alarm fire in the roof of a Qwest Communications building in Tukwila. Nobody was hurt in the fire, started by roofers working with hot tar at about 3:45 p.m., said Tukwila Fire Battalion Commander Marty Grisham. The fire got into the insulation, spreading beneath catwalks and air-conditioning systems, he said. "It was a concealed fire, and those are just really tough," Grisham said. "It wasn't a big fire, but it was a very manpower-intensive fire." Almost 75 firefighters from Tukwila, Seattle, Renton, King County Fire District 20, Kent and SeaTac departments were called to help contain the fire, he said. Everyone in the building — fewer than 10 people — evacuated, a Tukwila fire spokesman said. Firefighters had to cut part of the roof away to get to the blaze, Grisham said. They also ran a salvage operation to rescue computers and other expensive equipment from water damage, he said. "It was very physically demanding," Grisham said. Several medic units stood by, and teenagers participating in Tukwila's Fire Explorers program ran a "rehab station" with food, drinks and emergency first-aid equipment to support the firefighters as they battled the blaze for more than two hours, Grisham said. Fire marshals are expected to make a damage estimate Tuesday, he said.

1 0

qwest issues from seattle?
by Christopher Rogers 29 Sep '08

29 Sep '08

Anyone else having any problems with qwest(AS209) today? Their BGP session has bounced twice in the past half hour, and has been down now for about 15 minutes. We're peered with them at 1914 3rd ave, (securities and exchange building- next to the westin) in seattle. Anyone in the westin having similar issues? Thanks chris Christopher Rogers Network Engineer - Marchex, Inc w:+1.206.331.3368 m:+1.206.234.1423

1 1

www.house.gov not reachable.
by virendra rode 29 Sep '08

29 Sep '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Appears to be slashdot effect. Is anyone able to reach it? $ traceroute-nanog www.house.gov traceroute to www.house.gov (143.228.144.184), 30 hops max, 40 byte packets 1 * * 192.168.1.1 (192.168.1.1) 0.694 ms 2 rrcs-24-43-96-129.west.biz.rr.com (24.43.96.129) 12.547 ms 14.746 ms 11.975 ms 3 cpe-24-30-162-209.socal.rr.com (24.30.162.209) 13.230 ms 10.382 ms 10.428 ms 4 tge4-0-2.lsanca4-rtr1.socal.rr.com (24.30.162.233) 15.257 ms 16.641 ms 19.161 ms 5 * * * 6 te-4-1.car1.Tustin1.Level3.net (4.71.104.157) 21.439 ms te-1-3.car1.Tustin1.Level3.net (4.79.140.37) 18.273 ms te-1-4.car1.Tustin1.Level3.net (4.79.140.1) 18.181 ms 7 ae-2-3.bar1.Tustin1.Level3.net (4.69.132.218) 20.702 ms 20.403 ms 17.741 ms 8 ae-6-6.ebr1.LosAngeles1.Level3.net (4.69.136.202) 21.904 ms 19.457 ms 17.880 ms 9 ae-81-81.csw3.LosAngeles1.Level3.net (4.69.137.10) 29.696 ms ae-71-71.csw2.LosAngeles1.Level3.net (4.69.137.6) 22.903 ms ae-61-61.csw1.LosAngeles1.Level3.net (4.69.137.2) 39.060 ms 10 ae-4-99.edge1.LosAngeles1.Level3.net (4.68.20.199) 19.504 ms ae-2-79.edge1.LosAngeles1.Level3.net (4.68.20.71) 24.409 ms ae-1-69.edge1.LosAngeles1.Level3.net (4.68.20.7) 20.251 ms 11 192.205.33.225 (192.205.33.225) 98.923 ms level3-gw.la2ca.ip.att.net (192.205.33.229) 18.833 ms 192.205.33.225 (192.205.33.225) 17.658 ms 12 tbr2.la2ca.ip.att.net (12.127.3.214) 84.049 ms 83.400 ms 82.004 ms 13 cr2.la2ca.ip.att.net (12.122.19.213) 87.134 ms 83.234 ms 83.367 ms 14 12.122.30.29 (12.122.30.29) 85.060 ms 83.920 ms 82.151 ms 15 cr2.dvmco.ip.att.net (12.122.30.26) 85.061 ms 82.887 ms 82.259 ms 16 * cr1.cgcil.ip.att.net (12.122.31.86) 83.637 ms 82.557 ms 17 cr1.cl2oh.ip.att.net (12.122.2.206) 82.341 ms 83.166 ms 84.391 ms 18 cr2.cl2oh.ip.att.net (12.122.2.126) 84.642 ms 84.020 ms 85.446 ms 19 cr2.phlpa.ip.att.net (12.122.2.210) 199.738 ms 81.896 ms 82.662 ms 20 cr1.wswdc.ip.att.net (12.122.4.54) 84.546 ms 82.374 ms 82.072 ms 21 12.123.10.1 (12.123.10.1) 85.794 ms 83.924 ms 82.481 ms 22 * * * 23 * * * 24 * * * 25 * * * regards, /virendra -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFI4AbKpbZvCIJx1bcRAiCWAKDGEnwwlKkjtPp20/oBj+v4+XOw1wCg9VBr BnYN/jFjC4TEbzUdbytrWFY= =gAsi -----END PGP SIGNATURE-----

12 19

[Fwd: Re: T+Mobile data outage from detroit to new england]
by virendra rode 28 Sep '08

28 Sep '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - -------- Original Message -------- Subject: Re: [outages] T+Mobile data outage from detroit to new england Date: Sun, 28 Sep 2008 19:43:17 -0400 From: Jonathan Brockmeier <brockj(a)hope.edu> To: virendra.rode(a)gmail.com References: <48E00221.7050908(a)gmail.com> <ba72f1210809281609l37f6252cn8f392df0b27f2947(a)mail.gmail.com> <48E00FF5.3080909(a)gmail.com> I get connected at gsm not grps or edge, so I would say it is down and their customer service has a message upfront that my area has issues and they are working on having it back soon... Jonathan Brockmeier On 9/28/08, virendra rode <virendra.rode(a)gmail.com> wrote: > Jonathan Brockmeier wrote: >> Yes I can confirm on my t-mobile phone in michgan > --------------------- > Is it up or down? > > > regards, > /virendra > > >> Jonathan Brockmeier > >> On 9/28/08, virendra rode <virendra.rode(a)gmail.com> wrote: >> I am informed that there has been data outage from detroit to new england. > >> Can someone confirm this? So far no eta has been given. > > >> regards, >> /virendra > > _______________________________________________ > outages mailing list > outages(a)outages.org > https://puck.nether.net/mailman/listinfo/outages >>> > - -- Jonathan Brockmeier, CIT Hope College 616-395-7670 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFI4DOMpbZvCIJx1bcRAunWAKDnu4ZxDN4OGjk4N4+pqptrZ+xi9gCcD8uT n+DHxYqVHT1MhieQ/XEQKLQ= =yXyc -----END PGP SIGNATURE-----

1 0

[Fwd: Re: T+Mobile data outage from detroit to new england]
by virendra rode 28 Sep '08

28 Sep '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - -------- Original Message -------- Subject: Re: [outages] T+Mobile data outage from detroit to new england Date: Sun, 28 Sep 2008 19:30:00 -0400 From: Rusty Dekema <rdekema(a)gmail.com> To: virendra.rode(a)gmail.com References: <48E00221.7050908(a)gmail.com> I can confirm that t-mo's gprs/edge service was unavailable approx 25 miles west of detroit from approximately midnight last night until sometime after 3 pm today. This is based on my observation of my own t-mo phone; I wasn't keeping a close eye on it or doing any kind of logging of availability. - -Rusty On 9/28/08, virendra rode <virendra.rode(a)gmail.com> wrote: > I am informed that there has been data outage from detroit to new england. > > Can someone confirm this? So far no eta has been given. > > > regards, > /virendra > _______________________________________________ outages mailing list outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages - -- Sent from Gmail for mobile | mobile.google.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFI4DNSpbZvCIJx1bcRAmOgAKC0csqoUFnB3flTOi4d4IloOQPC9gCfapzB 1fMFtZ63Sbx+QJ6jNQzirXA= =W4gz -----END PGP SIGNATURE-----

1 0

Re: [outages] www.house.gov not reachable.
by John Macleod 28 Sep '08

28 Sep '08

I'm sure its a typical enterprise/government announcement. No prior communication to IT staff, although they will be required to troubleshoot and fix! As now it appears "broken", although completely avoidable! Communication and planning - its another foreign policy! -- John Macleod CTO Alentus Corporation 28202 Cabot Road Suite 205 Laguna Niguel, CA 92677 Tel: +1.877.922.9903 DID - US: +1.949.243.0490 UK: +44.(0)208.819.0350 www.alentus.com www.serversfirst.com - Sent via the Alentus Corporate Mobile Messaging System - ----- Original Message ----- From: outages-bounces(a)outages.org <outages-bounces(a)outages.org> To: Christian Koch <christian(a)broknrobot.com> Cc: Laurence F. Sheldon, Jr. <LarrySheldon(a)cox.net>; outages(a)outages.org <outages(a)outages.org> Sent: Sun Sep 28 16:55:25 2008 Subject: Re: [outages] www.house.gov not reachable. -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Just found out, congress released their draft rescue plan on their website. Just wondering if the bill includes any $$$$ for load-balancers :-) regards, /virendra Christian Koch wrote: > Trying 143.228.144.184... > Connected to www.house.gov. > Escape character is '^]'. > GET / HTTP/1.0 > > . > > > HTTP/1.1 200 OK > Server: "USHR Webserver Ver 5.4.1" > Date: Sun, 28 Sep 2008 22:43:33 GMT > Content-type: text/html > Connection: close > > <?xml version="1.0" encoding="iso-8859-1"?> > <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" > "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> > <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"> > > <head> > <title>United States House of Representatives, 110th Congress, 2nd > Session</title> > > > > having issues puling up via a browser though... > > On Sun, Sep 28, 2008 at 6:43 PM, Laurence F. Sheldon, Jr. > <LarrySheldon(a)cox.net> wrote: >> virendra rode wrote: >> >>> Appears to be slashdot effect. Is anyone able to reach it? >> I don't have full diagnostic skills--it seems to server up a blank page. >> >> They apparently block ICMP as current best practice seems to require. >> >> _______________________________________________ >> outages mailing list >> outages(a)outages.org >> https://puck.nether.net/mailman/listinfo/outages >> > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFI4AtdpbZvCIJx1bcRAkT+AJ4nY7zUprLeUTHHdGKKJiQ3uYOuMwCgqbFa hJlT0knbWF6XtGd+0a6byVA= =z/1S -----END PGP SIGNATURE----- _______________________________________________ outages mailing list outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

1 0

T+Mobile data outage from detroit to new england
by virendra rode 28 Sep '08

28 Sep '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I am informed that there has been data outage from detroit to new england. Can someone confirm this? So far no eta has been given. regards, /virendra -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFI4AIhpbZvCIJx1bcRAsAzAJ9JbhJ6yP1KwVLFm90aHlogEI4J/QCgwjXS kVabR28mPYWWCsg2MoqM2NY= =jj/j -----END PGP SIGNATURE-----

1 0

ColoServe Outage
by Matt Baldwin 26 Sep '08

26 Sep '08

Looks like ColoServe is experiencing a DoS attack which has taken many of its services offline. -matt

1 0

Re: [outages] XO Outage
by blackham＠gmail.com 25 Sep '08

25 Sep '08

Finally got the RFO: Summary: On Monday September 22, 2008 at 4:12 PM all XO Peering Routers were isolated due to a mis-configuration caused by human error. An XO technician inadvertently ran an incomplete script while attempting to complete non-impacting changes on an SNMP filter. XO data customers were unable to reach external websites following the scripting push. XO identified the issue immediately and began the process of logging into the Peering routers and corrected the configuration to restore the Peering router connectivity by 5:20 PM. As of this writing, XO operations is implementing a verification screen prior to script launching to ensure that only the intended scripts are being deployed. In addition, all scripting will require a Scheduled Maintenance Request or Repair event. Cause: Inadvertent mis-configuration caused by human error Resolution: XO technicians corrected configuration to restore peering router connectivity. Sent from my BlackBerry® smartphone with SprintSpeed -----Original Message----- From: blackham(a)gmail.com Date: Tue, 23 Sep 2008 00:41:42 To: Paul Norton<paul(a)neoverve.com>; <outages-bounces(a)outages.org>; Justin Sharp<sharp(a)sharpone.net> Cc: <outages(a)outages.org> Subject: Re: [outages] XO Outage Little late on my part, but yes... XO partitioned for 20-30 mins today. Said it was a peering issue. I'll share the RFO when I get it. Sent from my BlackBerry® smartphone with SprintSpeed -----Original Message----- From: Paul Norton <paul(a)neoverve.com> Date: Mon, 22 Sep 2008 14:53:43 To: Justin Sharp<sharp(a)sharpone.net> Cc: <outages(a)outages.org> Subject: Re: [outages] XO Outage Seeing outage here in San Diego. On Sep 22, 2008, at 2:35 PM, Justin Sharp wrote: > We are seeing some issues w/ XO/Savvis peering.. > > Trace from XO to Savvis IP space (64.75.10.151) > > Keys: Help Display mode Restart statistics Order of fields > quit > Packets > Pings > Host > Loss > % Snt Last Avg Best Wrst StDev > 1. > scrubbed > 0.0 > % 6 0.6 0.5 0.4 0.6 0.1 > 2. > ip65 > -44 > -114 > -97 > .z114 > -44 > -65 > .customer > .algx.net > 0.0% 6 1.3 1.3 1.2 1.4 0.1 > 3. ??? > > > Trace from Savvis to XO IP space (65.44.114.97) > > 1. > scrubbed > 0.0 > % 38 0.4 0.4 0.3 0.5 0.1 > 2. > 64.41.199.129 > 0.0 > % 37 1.0 24.0 0.6 330.2 80.4 > 3. hr1- > ge > -7 > -47 > .santaclarasc5 > .savvis > .net > 0.0% 37 0.7 1.4 0.6 27.3 4.4 > 4. er1- > te > -1 > -0 > -0 > .sanjose3equinix > .savvis > .net > 0.0% 37 0.7 5.2 0.6 140.3 23.2 > 5. cr1- > tenge > -0 > -7 > -5 > -0 > .sanfrancisco > .savvis > .net > 2.7% 37 2.9 4.0 2.6 16.6 2.5 > 6. cr2- > pos > -0 > -0 > -3 > -3 > .dallas > .savvis > .net > > 0.0% 37 42.6 43.1 42.3 51.4 1.4 > 7. dpr1- > ge > -4 > -0 > -0 > .dallasequinix > .savvis > .net > 0.0% 37 43.1 44.8 42.9 76.9 6.7 > 8. er1- > te > -2 > -1 > .dallasequinix > .savvis > .net > 0.0% 37 43.3 49.2 42.8 233.6 31.6 > 9. > 208.175.175.90 > 0.0 > % 37 43.0 42.8 42.6 43.6 0.2 > 10. > 65.106.1.102 > 75.0 > % 37 43.5 46.5 43.4 62.9 6.3 > 11. > 65.106.1.101 > 0.0 > % 37 43.4 47.8 43.2 112.3 12.5 > 12. > 65.106.0.41 > 0.0 > % 37 57.5 65.1 57.1 177.3 21.0 > 13. > 65.106.1.73 > 0.0 > % 37 57.4 66.5 57.1 162.1 24.2 > 14. ??? > > Trying to call into XO and they aren't even taking calls, they > mention something about network issues in Spokane. Any ideas as to > what is going on/ETA to fix? > > --Justin > > -- Paul Norton Systems Administrator Neoverve - www.neoverve.com Neoverve Blog - http://blog.neoverve.com/ _______________________________________________ outages mailing list outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

1 0

puck outage this am.
by virendra rode 24 Sep '08

24 Sep '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, You may or may not have noticed puck.nether.net (hosting outages.org mailing list) suffered an outage this am. This was caused due to a file system issue (possible inconsistence state) on a fairly large (1tb) filesystem which caused services on puck to be unresponsive. Many thanks to Jared Mauch in addressing this issue. Like always, appreciate your feedback and your constructive criticism. regards, /virendra -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFI2wsgpbZvCIJx1bcRArpUAJsECd19nRQs06wkgvnfS3Xhk2TvOQCghhTC OyRIHk8YHPR72l+WRmh15qU= =fy4U -----END PGP SIGNATURE-----

1 0