[Freelance Traveller] Site Outage 2010.Feb.05

[FreeTrav]

Our hosting provider seems to have met disaster, and their entire network is down - and has been so for over 21 hours as I write this. We may be looking for a new hosting provider very soon. If anyone has any insights into what may be happening in Canton, GA that might have caused this - other than suggesting that our provider's disaster might be fundamentally fiscal rather than technical (believe me, we're wondering that ourselves) - we'd be interested in hearing it.

Because of this outage, oure email addresses at freelancetraveller.com are also out of service.

As a temporary measure, email to us can be sent to freetrav@gmail.com

You can also visit our website mirror at http://freelancetraveller.downport.com

We will try to keep our fans updated as we find out more.

 

[rust]

If anyone has any insights into what may be happening in Canton, GA that might have caused this - other than suggesting that our provider's disaster might be fundamentally fiscal rather than technical (believe me, we're wondering that ourselves) - we'd be interested in hearing it.

There are a couple of reports of customers of at least one provider (Wind-
stream) in Canton, GA, who obviously had serious problems since mid Ja-
nuary, with extreme fluctuations of download speed and thelike.

I have seen exactly the same problems with two providers over here, and
in both cases their servers crashed soon afterwards and remained down
for more than a day.

Edit.:
According to my network diagnostics software, your DNS server is down.
The entire CyberWeb Net is currently down completely.

 

[FreeTrav]

As I write this, it's been down for a minimum of 45 hours. I just checked again, and they have updated as of 18:00 Eastern time 2010.Feb.06 - they are saying that two separate and (ostensibly) unrelated incidents took out both their primary and backup systems, and OnTrac has been called in 'at great expense' to do data recovery. They do not have an estimate for restoration of service, other than 'as soon as we can', and they are claiming that they will have new and better redundancy/backup solutions after restoration.

This, obviously, does not help us right now.

We're still watching and waiting and fuming. And I'm going to be talking to some friends about this sort of situation, from a theoretical standpoint. Which may not stay theoretical, depending on what I learn.

 

[GamerDude]

Good luck and hope the site is back up soon..

 

[BP]

... they are saying that two separate and (ostensibly) unrelated incidents took out both their primary and backup systems, and OnTrac has been called in 'at great expense' to do data recovery.

Ah.. amateurs :roll:

But don't be alarmed, I worked with billion dollar companies who spent 10s of millions every year on backups that they could not restore, backups that transported to an unsecure and connected location every week, backup facilities located within the same geographic 'danger' zones, awesome robotic tape backup archive rooms that never got tapes changed because the 24/7 operators weren't given keys (for real!), redundant communications methods all dependent on a single piece of equipment or running accoss the same land line, etc.

A personal favorite - that happens all the time - RAID arrays with drives all bought at the same time with the data improperly spanned (so 2 out of 5 drives fail there is no recovery) - these drives will typically fail at the same time! Simple common sense.

Anyway, hope you find a new host - these guys have been down way to long regardless of any technical difficulties - the difficulties are in planning and competency (even the poor planning that led to this should have been overcome by now - at least to some extent - excepting physical damage like a high temp fire to the only copies of harddrives).

Good luck!

 

[FreeTrav]

BP, you're not describing anything I'm unfamiliar with; I've done IT for many companies, in different industries, for close to 25 years, and there hasn't been a one that didn't take a shortcut somewhere in their disaster planning. Most of the time, they wouldn't get bitten where it would hurt. This particular time, well...

 

[BP]

BP, you're not describing anything I'm unfamiliar with; I've done IT for many companies, in different industries, for close to 25 years, and there hasn't been a one that didn't take a shortcut somewhere in their disaster planning. Most of the time, they wouldn't get bitten where it would hurt. This particular time, well...

And you selected these clowns for hosting your excellent site?

 

[FreeTrav]

And you selected these clowns for hosting your excellent site?

It's not like most hosting companies reveal their system setup or disaster recovery plans; they simply advertise "X% UPTIME GUARANTEED!". It's why a GOOD website will have a provision for a mirror - as I do; the site is still accessible at http://freelancetraveller.downport.com, though because of the way Downport handles things, you can't just tack on path data to that URL; you have to "walk" the links to get where you want to be.

I'm looking at some other possibilities, and I learned a little something that might just make it possible/easy to arrange automatic fallback/fallforward in the future.

 

[The Chef]

I work in DR, we deal with a number of servers a month, because back ups are not done...

I know Ontrac... they are expensive, but they should do a good job

Chef

 

[FreeTrav]

WE'RE BACK!

Some time between 18:00 EST yesterday and 05:00 this morning, our hosting provider got their services running. At the present time, the main Freelance Traveller website and all email addresses @freelancetraveller.com are back in service. We apologise for the outage, and are continuing to look at options for redundancy and fallover. Thank you for bearing with us during this trying week.

 

[GamerDude]

Jeff,

It wasn't you fault, you dealt with it quickly and were up front on everything. Can't ask for more than that.