Went down about 35 minutes ago. Anyone know what's going on? Scott.
I saw our BGP peer to them bounce at 11:00:21 EST as well. It came back at 11:00:55 and the link stayed up. -Scott From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Scott Howard Sent: Tuesday, October 20, 2009 11:20 AM To: outages@outages.org Subject: [outages] Level 3 down in Atlanta Went down about 35 minutes ago. Anyone know what's going on? Scott.
We saw the level 3 peering bounce, but it looked like after it came back up, there was some traffic being blackholed between level3 and regional providers. Things have recovered in the last 10-15 minutes. - Gaurav On Tue, Oct 20, 2009 at 9:36 AM, Scott Berkman <scott@sberkman.net> wrote:
I saw our BGP peer to them bounce at 11:00:21 EST as well. It came back at 11:00:55 and the link stayed up.
-Scott
*From:* outages-bounces@outages.org [mailto:outages-bounces@outages.org] *On Behalf Of *Scott Howard *Sent:* Tuesday, October 20, 2009 11:20 AM *To:* outages@outages.org *Subject:* [outages] Level 3 down in Atlanta
Went down about 35 minutes ago.
Anyone know what's going on?
Scott.
_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We seem to be back to normal now as well. David Hiers CCIE (R/S, V), CISSP ADP Dealer Services 2525 SW 1st Ave. Suite 300W Portland, OR 97201 o: 503-205-4467 f: 503-402-3277 From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Gaurav Taparia Sent: Tuesday, October 20, 2009 8:59 AM To: Scott Berkman Cc: outages@outages.org Subject: Re: [outages] Level 3 down in Atlanta We saw the level 3 peering bounce, but it looked like after it came back up, there was some traffic being blackholed between level3 and regional providers. Things have recovered in the last 10-15 minutes. - Gaurav On Tue, Oct 20, 2009 at 9:36 AM, Scott Berkman <scott@sberkman.net<mailto:scott@sberkman.net>> wrote: I saw our BGP peer to them bounce at 11:00:21 EST as well. It came back at 11:00:55 and the link stayed up. -Scott From: outages-bounces@outages.org<mailto:outages-bounces@outages.org> [mailto:outages-bounces@outages.org<mailto:outages-bounces@outages.org>] On Behalf Of Scott Howard Sent: Tuesday, October 20, 2009 11:20 AM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] Level 3 down in Atlanta Went down about 35 minutes ago. Anyone know what's going on? Scott. _______________________________________________ outages mailing list outages@outages.org<mailto:outages@outages.org> https://puck.nether.net/mailman/listinfo/outages This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
If you are a Level3 customer, check the Portal. The only Atlanta issue I see was earlier, but someone could have flipped a wrong switch working on this: 3720051 Atlanta, GA ~ Voice / IP Outage ~ Dial Interfaces Down Atlanta, GA 10/20/09 2:38:00 PM GMT Open 10/20/09 3:43:03 PM GMT Robert D. Scott Robert@ufl.edu Senior Network Engineer 352-273-0113 Phone CNS - Network Services 352-392-2061 CNS Phone Tree University of Florida 352-392-9440 FAX Florida Lambda Rail 352-294-3571 FLR NOC Gainesville, FL 32611 321-663-0421 Cell -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Scott Berkman Sent: Tuesday, October 20, 2009 11:37 AM To: 'Scott Howard'; outages@outages.org Subject: Re: [outages] Level 3 down in Atlanta I saw our BGP peer to them bounce at 11:00:21 EST as well. It came back at 11:00:55 and the link stayed up. -Scott From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Scott Howard Sent: Tuesday, October 20, 2009 11:20 AM To: outages@outages.org Subject: [outages] Level 3 down in Atlanta Went down about 35 minutes ago. Anyone know what's going on? Scott.
Looks like it's all back up as of about 30 mins ago. Apparently either a core switch or router failed, which took down much of their network in Atlanta, as well as Memphis and Nashville. Thanks, Scott. On Tue, Oct 20, 2009 at 8:19 AM, Scott Howard <scott@doc.net.au> wrote:
Went down about 35 minutes ago.
Anyone know what's going on?
Scott.
Just joined the list for this exact case! We are in 56 Marietta and are operating currently (second time since 11AM). Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 "When you have eliminated the impossible, that which remains, however improbable, must be the truth." --- Sir Arthur Conan Doyle On Tue, Oct 20, 2009 at 12:28 PM, Scott Howard <scott@doc.net.au> wrote:
Looks like it's all back up as of about 30 mins ago.
Apparently either a core switch or router failed, which took down much of their network in Atlanta, as well as Memphis and Nashville.
Thanks, Scott.
On Tue, Oct 20, 2009 at 8:19 AM, Scott Howard <scott@doc.net.au> wrote:
Went down about 35 minutes ago.
Anyone know what's going on?
Scott.
_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Tue, Oct 20, 2009 at 09:28:21AM -0700, Scott Howard wrote:
Looks like it's all back up as of about 30 mins ago.
Apparently either a core switch or router failed, which took down much of their network in Atlanta, as well as Memphis and Nashville.
Level 3 has a single router or switch handling packets at a major POP? I doubt this, but the outage is confirmation something bad happened. That said: where's the redundancy, and why didn't it kick in? -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
There are four routes into Atlanta, I believe. Getting to Atlanta wasn't a problem. Between the Atlanta hub and the customers was the problem. Can't give every customer two connections for the price of one =/ Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 "When you have eliminated the impossible, that which remains, however improbable, must be the truth." --- Sir Arthur Conan Doyle On Tue, Oct 20, 2009 at 1:00 PM, Jeremy Chadwick <outages@jdc.parodius.com>wrote:
On Tue, Oct 20, 2009 at 09:28:21AM -0700, Scott Howard wrote:
Looks like it's all back up as of about 30 mins ago.
Apparently either a core switch or router failed, which took down much of their network in Atlanta, as well as Memphis and Nashville.
Level 3 has a single router or switch handling packets at a major POP? I doubt this, but the outage is confirmation something bad happened.
That said: where's the redundancy, and why didn't it kick in?
-- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Just got an RFO from Level 3... A Core router in Atlanta crashed and failed over. But the redundant router didn't pass any traffic so they had to reload the router that crashed. They have opened a ticket with their hardware vendor. My tech that called wasn't as nosey as I wanted him to be so he didn't get any device and node information. But this sounds like Cisco Cisco and more Cisco issues. The GSR's are EOL so I bet it was one of them. Sean From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Josh Luthman Sent: Tuesday, October 20, 2009 1:13 PM To: Jeremy Chadwick Cc: outages@outages.org Subject: Re: [outages] Level 3 down in Atlanta There are four routes into Atlanta, I believe. Getting to Atlanta wasn't a problem. Between the Atlanta hub and the customers was the problem. Can't give every customer two connections for the price of one =/ Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 "When you have eliminated the impossible, that which remains, however improbable, must be the truth." --- Sir Arthur Conan Doyle On Tue, Oct 20, 2009 at 1:00 PM, Jeremy Chadwick <outages@jdc.parodius.com<mailto:outages@jdc.parodius.com>> wrote: On Tue, Oct 20, 2009 at 09:28:21AM -0700, Scott Howard wrote:
Looks like it's all back up as of about 30 mins ago.
Apparently either a core switch or router failed, which took down much of their network in Atlanta, as well as Memphis and Nashville. Level 3 has a single router or switch handling packets at a major POP? I doubt this, but the outage is confirmation something bad happened.
That said: where's the redundancy, and why didn't it kick in? -- | Jeremy Chadwick jdc@parodius.com<mailto:jdc@parodius.com> | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | _______________________________________________ outages mailing list outages@outages.org<mailto:outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Sean Clark wrote:
The GSR’s are EOL so I bet it was one of them.
No they aren't. The original three chassis are EOL (12008, 12012, and the five-rail 12016), but the other nine chassis (three in the 12000 series, four in the 12400 series, and two in the 12800 series) do appear to be alive and kicking. pt
There are multiple devices there handling intra and inter-pop traffic. NOC still investigating with vendor. Redundancy won't kick in if soft or silent failures occur and the control plane keeps protocols up but forwarding is affected. Unfortunately these types of failures still happen from time to time especially with multi-member LAG groups. No details on this specific event however. * Jeremy Chadwick was thought to have said:
Level 3 has a single router or switch handling packets at a major POP? I doubt this, but the outage is confirmation something bad happened.
That said: where's the redundancy, and why didn't it kick in?
I have a peering with them in Atlanta which bounced during the outage. Per their noc the switch just froze and began eating (my term) packets without actually causing peerings to go down. They rebooted it which is when our peerings were reset. They are monitoring and have been stable since the issue. From: Craig Pierantozzi <tozz@tozz.net> To: outages@outages.org Date: 10/20/2009 03:29 PM Subject: Re: [outages] Level 3 down in Atlanta Sent by: <outages-bounces@outages.org> There are multiple devices there handling intra and inter-pop traffic. NOC still investigating with vendor. Redundancy won't kick in if soft or silent failures occur and the control plane keeps protocols up but forwarding is affected. Unfortunately these types of failures still happen from time to time especially with multi-member LAG groups. No details on this specific event however. * Jeremy Chadwick was thought to have said:
Level 3 has a single router or switch handling packets at a major POP? I doubt this, but the outage is confirmation something bad happened.
That said: where's the redundancy, and why didn't it kick in?
outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
----- "Jeremy Chadwick" <outages@jdc.parodius.com> wrote:
On Tue, Oct 20, 2009 at 09:28:21AM -0700, Scott Howard wrote:
Looks like it's all back up as of about 30 mins ago.
Apparently either a core switch or router failed, which took down much of their network in Atlanta, as well as Memphis and Nashville.
Level 3 has a single router or switch handling packets at a major POP? I doubt this, but the outage is confirmation something bad happened. That said: where's the redundancy, and why didn't it kick in?
Oh; you're *always* asking that. :-) The Internet Backbone<tm> has been a commercial, rather than an engineering, construct for over 15 years now. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com '87 e24 St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274 Start a man a fire, and he'll be warm all night. Set a man on fire, and he'll be warm for the rest of his life.
On Thu, Oct 22, 2009 at 7:03 PM, Jay R. Ashworth <jra@baylink.com> wrote:
----- "Jeremy Chadwick" <outages@jdc.parodius.com> wrote:
On Tue, Oct 20, 2009 at 09:28:21AM -0700, Scott Howard wrote:
Looks like it's all back up as of about 30 mins ago.
Apparently either a core switch or router failed, which took down much of their network in Atlanta, as well as Memphis and Nashville.
Level 3 has a single router or switch handling packets at a major POP? I doubt this, but the outage is confirmation something bad happened. That said: where's the redundancy, and why didn't it kick in?
Oh; you're *always* asking that.
:-)
The Internet Backbone<tm> has been a commercial, rather than an engineering, construct for over 15 years now.
The RFO that went out somewhat after he asked that was more useful... N=2 redundancy was in place. However, when primary had hardware failure, secondary had (unknown / unstated) software, config, or hardware failure that hadn't been detected or checked, and it didn't work either. It's hard to test clusters of things well when they have near-100% uptime requirements. The dependability of the untested failover unit is low, as you're not testing it well. Sometimes you can test failovers in stream. But sometimes those supposedly harmless failover tests fail for baroque reasons, taking down a service when the primary was in fact just fine. This isn't (just) an economics problem. Reliability of complex problems is an mathematically exponentially hard problem to crack from the engineering and theoretical levels. Some people don't try - and get what they deserve - and some people give it a good or best commercial reasonable effort, and still fail. Doing better than that is really hard. -- -george william herbert george.herbert@gmail.com
This is the purpose of learning from your mistakes in the past. Create a maintenance plan so it doesn't happen again! Fool me once... Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 "When you have eliminated the impossible, that which remains, however improbable, must be the truth." --- Sir Arthur Conan Doyle On Thu, Oct 22, 2009 at 10:43 PM, George Herbert <george.herbert@gmail.com>wrote:
On Thu, Oct 22, 2009 at 7:03 PM, Jay R. Ashworth <jra@baylink.com> wrote:
----- "Jeremy Chadwick" <outages@jdc.parodius.com> wrote:
On Tue, Oct 20, 2009 at 09:28:21AM -0700, Scott Howard wrote:
Looks like it's all back up as of about 30 mins ago.
Apparently either a core switch or router failed, which took down much of their network in Atlanta, as well as Memphis and Nashville.
Level 3 has a single router or switch handling packets at a major POP? I doubt this, but the outage is confirmation something bad happened. That said: where's the redundancy, and why didn't it kick in?
Oh; you're *always* asking that.
:-)
The Internet Backbone<tm> has been a commercial, rather than an engineering, construct for over 15 years now.
The RFO that went out somewhat after he asked that was more useful... N=2 redundancy was in place. However, when primary had hardware failure, secondary had (unknown / unstated) software, config, or hardware failure that hadn't been detected or checked, and it didn't work either.
It's hard to test clusters of things well when they have near-100% uptime requirements. The dependability of the untested failover unit is low, as you're not testing it well.
Sometimes you can test failovers in stream. But sometimes those supposedly harmless failover tests fail for baroque reasons, taking down a service when the primary was in fact just fine.
This isn't (just) an economics problem. Reliability of complex problems is an mathematically exponentially hard problem to crack from the engineering and theoretical levels.
Some people don't try - and get what they deserve - and some people give it a good or best commercial reasonable effort, and still fail. Doing better than that is really hard.
-- -george william herbert george.herbert@gmail.com _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Thu, 22 Oct 2009, George Herbert wrote:
On Thu, Oct 22, 2009 at 7:03 PM, Jay R. Ashworth <jra@baylink.com> wrote:
Level 3 has a single router or switch handling packets at a major POP? I doubt this, but the outage is confirmation something bad happened. That said: where's the redundancy, and why didn't it kick in?
Oh; you're *always* asking that.
The RFO that went out somewhat after he asked that was more useful... N=2 redundancy was in place. However, when primary had hardware failure, secondary had (unknown / unstated) software, config, or hardware failure that hadn't been detected or checked, and it didn't
I'm not in Atlanta but from what was mentioned on the list, it was a soft failure which is why the other routers didn't failover w/ HSRP or whatnot: https://puck.nether.net/pipermail/outages/2009-October/001607.html https://puck.nether.net/pipermail/outages/2009-October/001608.html The real question should be why nobody powered down that device the first or second time, considering it didn't failover properly the first time. https://puck.nether.net/pipermail/outages/2009-October/001600.html https://puck.nether.net/pipermail/outages/2009-October/001611.html These things happen from time-to-time -- that's the Internet. -- William R. Lorenz
FWIW, here in Nashville on (3), we never saw any real problems, except with reachability to Atlanta... -- Tim On Tue, Oct 20, 2009 at 11:28 AM, Scott Howard <scott@doc.net.au> wrote:
Looks like it's all back up as of about 30 mins ago.
Apparently either a core switch or router failed, which took down much of their network in Atlanta, as well as Memphis and Nashville.
Thanks, Scott.
On Tue, Oct 20, 2009 at 8:19 AM, Scott Howard <scott@doc.net.au> wrote:
Went down about 35 minutes ago.
Anyone know what's going on?
Scott.
_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Same here. David Hiers CCIE (R/S, V), CISSP ADP Dealer Services 2525 SW 1st Ave. Suite 300W Portland, OR 97201 o: 503-205-4467 f: 503-402-3277 From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Scott Howard Sent: Tuesday, October 20, 2009 2:01 PM To: outages@outages.org Subject: Re: [outages] Level 3 down in Atlanta Looks like they are having problem again, as of about 3-4 minutes ago. Scott. On Tue, Oct 20, 2009 at 8:19 AM, Scott Howard <scott@doc.net.au<mailto:scott@doc.net.au>> wrote: Went down about 35 minutes ago. Anyone know what's going on? Scott. This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
Yep. From: "Hiers, David" <David_Hiers@adp.com> To: Scott Howard <scott@doc.net.au>, "outages@outages.org" <outages@outages.org> Date: 10/20/2009 05:21 PM Subject: Re: [outages] Level 3 down in Atlanta Sent by: <outages-bounces@outages.org> Same here. David Hiers CCIE (R/S, V), CISSP ADP Dealer Services 2525 SW 1st Ave. Suite 300W Portland, OR 97201 o: 503-205-4467 f: 503-402-3277 From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Scott Howard Sent: Tuesday, October 20, 2009 2:01 PM To: outages@outages.org Subject: Re: [outages] Level 3 down in Atlanta Looks like they are having problem again, as of about 3-4 minutes ago. Scott. On Tue, Oct 20, 2009 at 8:19 AM, Scott Howard <scott@doc.net.au> wrote: Went down about 35 minutes ago. Anyone know what's going on? Scott. This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Same here. Changed my forward path away from Atlanta and I can now get to alot of sites I could not before. On Tue, Oct 20, 2009 at 4:33 PM, <Keegan.Holley@sungard.com> wrote:
Yep.
From: "Hiers, David" <David_Hiers@adp.com> To: Scott Howard < scott@doc.net.au>, "outages@outages.org" <outages@outages.org> Date: 10/20/2009 05:21 PM Subject: Re: [outages] Level 3 down in Atlanta Sent by: <outages-bounces@outages.org>
------------------------------
Same here.
David Hiers
CCIE (R/S, V), CISSP ADP Dealer Services 2525 SW 1st Ave. Suite 300W Portland, OR 97201 o: 503-205-4467 f: 503-402-3277
*From:* outages-bounces@outages.org [mailto:outages-bounces@outages.org<outages-bounces@outages.org>] *On Behalf Of *Scott Howard* Sent:* Tuesday, October 20, 2009 2:01 PM* To:* outages@outages.org* Subject:* Re: [outages] Level 3 down in Atlanta
Looks like they are having problem again, as of about 3-4 minutes ago.
Scott.
On Tue, Oct 20, 2009 at 8:19 AM, Scott Howard <*scott@doc.net.au*<scott@doc.net.au>> wrote: Went down about 35 minutes ago.
Anyone know what's going on?
Scott.
------------------------------ This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. _______________________________________________
outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Looks like it's starting to come back for me. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 "When you have eliminated the impossible, that which remains, however improbable, must be the truth." --- Sir Arthur Conan Doyle On Tue, Oct 20, 2009 at 6:03 PM, Clue Store <cluestore@gmail.com> wrote:
Same here. Changed my forward path away from Atlanta and I can now get to alot of sites I could not before.
On Tue, Oct 20, 2009 at 4:33 PM, <Keegan.Holley@sungard.com> wrote:
Yep.
From: "Hiers, David" <David_Hiers@adp.com> To: Scott Howard < scott@doc.net.au>, "outages@outages.org" <outages@outages.org> Date: 10/20/2009 05:21 PM Subject: Re: [outages] Level 3 down in Atlanta Sent by: <outages-bounces@outages.org>
------------------------------
Same here.
David Hiers
CCIE (R/S, V), CISSP ADP Dealer Services 2525 SW 1st Ave. Suite 300W Portland, OR 97201 o: 503-205-4467 f: 503-402-3277
*From:* outages-bounces@outages.org [mailto:outages-bounces@outages.org<outages-bounces@outages.org>] *On Behalf Of *Scott Howard* Sent:* Tuesday, October 20, 2009 2:01 PM* To:* outages@outages.org* Subject:* Re: [outages] Level 3 down in Atlanta
Looks like they are having problem again, as of about 3-4 minutes ago.
Scott.
On Tue, Oct 20, 2009 at 8:19 AM, Scott Howard <*scott@doc.net.au*<scott@doc.net.au>> wrote: Went down about 35 minutes ago.
Anyone know what's going on?
Scott.
------------------------------ This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. _______________________________________________
outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
They claim it should be back up now. Same router failed that the majority of the traffic was flowing through. L3 tech told me they've powered it down now. I haven't verified this for myself yet. -Will On Tue, Oct 20, 2009 at 5:00 PM, Scott Howard <scott@doc.net.au> wrote:
Looks like they are having problem again, as of about 3-4 minutes ago.
Scott.
On Tue, Oct 20, 2009 at 8:19 AM, Scott Howard <scott@doc.net.au> wrote:
Went down about 35 minutes ago.
Anyone know what's going on?
Scott.
participants (17)
-
Byrd, William -
Clue Store -
Craig Pierantozzi -
Gaurav Taparia -
George Herbert -
Hiers, David -
Jay R. Ashworth -
Jeremy Chadwick -
Josh Luthman -
Keegan.Holley＠sungard.com -
Pete Templin -
Robert D. Scott -
Scott Berkman -
Scott Howard -
Sean Clark -
Tim Jackson -
William R. Lorenz