This should probably go to outages-discuss, but I have to say something. These kinds of outages were regular occurrences at HE Fremont (FMT2 had not been built yet, but I doubt it matters). Here's a few choice examples -- and these are just a few -- of recurring things which never got addressed. I will never forget how packets with a Comcast source, HE Fremont destination, went through an AT&T "device" somewhere in the mix which would regularly burp/flake out and HE couldn't do anything about it. No alternate routes were propagated, so literally Comcast->HE was hard down. Single point of failure. I will never forget how packets with a SBC DSL (now AT&T) source, HE Fremont destination, went through Telia (a Swedish ISP with no North American NOC) for a single hop, and that Telia would drop their BGP session or severe their link for whatever reason. Like the above, no alternate routes were propagated, so literally SBC->HE was hard down. Likewise, individuals in Sweden who used Telia also saw the same thing going from Telia Sweden->HE Fremont. Single point of failure. I will never forget the Cisco router which would reboot every 4-5 months for no reason... a problem which went on for literally years, and all I was ever told was "well it's back up now" and "we have a case open with Cisco". Single point of failure. I will never forget how HE refused to use VLANs to segregate customers on a layer 2 level, instead preferring some strange layer 3 implementation. When we witnessed an unexpected massive (7-8mbit/sec) increase in inbound traffic, only to find that the destination IPs of these packets were for another customer in a completely different netblock/area of the Fremont facility, we were told by support "that's impossible". Full tcpdump captures were given, and we were told "this makes no sense, this can't happen". 4-5 hours later, we were told the root cause was "a customer who had misconfigured their load balancer". Right. I will never forget the two separate times there were full-scale power outages both caused by "UPS maintenance". Gas generators? They have them, but when I asked why they didn't kick in, I was told "we don't know". When I asked if there would be a follow-up investigation as to why that didn't happen, so the issue wouldn't recur, I was told "probably". I will never forget the "maintenances" that we were never told of, because scheduled maintenances are not announced to customers. I was left with the impression they're done on a whim vs. scheduled. Safe to say when our contract ended, we left. I will not recommend Hurricane Electric to anyone who wants co-location with reliable, redundant connectivity. I *will* recommend them for cheap "I just need a 1U box stuck somewhere and don't really care about quality" co-location, although compared to some of their competitors, they're actually more expensive. Also, if HE or HE customers read this and want to flame/argue/do burn-outs in a Pacer over this -- don't bother. I won't be responding to any mails. Why? Because all outages/incidents witnessed, including the above, were sent to our account rep. when we were asked "why aren't you renewing?" I received no response past that point. I'll leave you with this: if FMT2 had redundancy, then how were things hard down for over an hour? Think about it. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | On Sat, Sep 26, 2009 at 01:17:22AM -0700, Scott Howard wrote:
It's back up as of a few seconds ago.
Scott.
On Sat, Sep 26, 2009 at 1:13 AM, Scott Howard <scott@doc.net.au> wrote:
Outage started at 11:37pm Pacific, give or take a minute.
I spoke to them at about 11:43pm and they were aware there was an issue, but didn't know the cause. Since then it's been impossible to get through to anything but voicemail :(
Scott
On Sat, Sep 26, 2009 at 1:09 AM, George Herbert <george.herbert@gmail.com>wrote:
Hurricane Electric has a datacenter outage at FMT2, cause as yet unspecified, but there are SF bay area and possibly wider network issues eminating from it at low to moderate intensity.
Out around 12:15am PST Sat morning.
_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages