Sorry I was interested in outages not Jays mistakes. I really just need a site that reports outages as it effects business and not the details on how Jay fixed his issues. From: Jay Ashworth <jra@baylink.com> To: outages@outages.org Sent: Friday, September 23, 2011 7:17 PM Subject: [outages] Lessons Learned: RRTB outage So I had to renumber some servers this afternoon, cause I was expanding to a larger netblock (a 28 instead of a 29). I renumbered my servers and my DNS (which I'd set the TTL on to 300 like a good boy on Wednesday), and then pulled the trigger with Road Runner. He "rescripted" his SMC router (the likely cause of some standard deviation noted by a couple of reporters -- the router, not the rescripting), and I pinged it and it was ok, and I mtr'd it and it was ok, so I hit the webserver, and that came up fine, too. So then my boss calls me 15 minutes later: it's not working. "I wonder what that could be", sez I; I'd even traced and hit the webserver from my Android phone (Sprint; Opera Mobile 11), and it had worked fine. That was Red Herring #1. So my boss uses a Mac. So does my best friend, and while he was on the way out the door to a second-anniversary-wake for a guy we went to school with, he took a moment to try to hit it as well. No luck. That was Red Herring #2 (both of them use Macs). Those of you who've been playing close, careful attention here may have noticed by now the thing I did *not* say I'd done: Changing the default gateway on the server. My office lan could hit it *because its uplink was in the same network*; *it* had a route for that network. Everyone else... couldn't. Apparently, Sprint operates a caching server, even if you're using the version of Opera (Mobile, not Mini) that does *not*, which explains Red Herring #1. As for Red Herring #2, well... Macs don't, apparently, hard-cache IPs the way WinXP does (I'm looking at *you*, "ipconfig/ flushdns"), but I already knew that, because boss had the right address. Lesson Learned: Make sure you know what your diagnostic tests are telling you, before you use them to rule out possible problems. Better yet: don't rule those potential problems out at all: work your whole diagnostic tree every time Oh: I forgot Red Herring #3: the traces that broke *didn't hit that carrier edge router* for some reason. No clue why. Thanks to the dozen or so people who responded; a couple of whom have way too {much time,many servers} on their hands. :-) Followups to -discuss Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages