Downspout.ca was being run in an OpenVZ/Virtuozzo based VPS system. On Wednesday Oct 8, 2008 it disappeared from the internet along with an unknown number of others, probably hundreds, of sites. The last I saw of it was 11:45pm. It was gone by 11:50pm.

At 4:40 am I received a message that they were suffering technical difficulties and my patience was required. I’d given up waiting and gone to bed by then, so there was all the patience in the world available.

At some point on the following day the user forums the company runs disappeared. I couldn’t tell if it was as a result of failure or deliberate removal. Before then there was clearly a very wide spread problem with at least four hardware nodes that were dead not including the one I was on. Lots of yelling, whining and crying was detected in the forums but only a single small message from the company suggesting equipment failure and/or power failure. According to rumour the ticket system and support email was also dead. DNS was dead.

The forum link was replaced with an “announcement” link saying very little other than new customer sign up was suspended and the data looked like it was gone. I think it also said that some customers were going to get the boot.

On Friday I received a notice via email saying “… due to some unforeseen circumstances an entire segment of our network has been deleted.” and all backup data had been lost and couldn’t be retrieved.

I don’t care I had my own backup.

They also said they lacked the space to give me a replacement server so they were cancelling my account and providing a refund.

So much for their 100% uptime guarantee.

Sigh… I moved.

On Monday Oct 13 1:54 am I receive a message telling me all my data had been lost (so what) and that my account had been re-enabled and I’d get a 45 day credit.

I’d already moved on the Friday after they told me to go away. I’m not moving back now.

On Monday Oct 13 2:13pm they sent a message coming clean on what happened. Briefly, new employee makes critically stupid security error by disabling two firewalls to allow a different employee access while “abroad”. Some script kiddy gets in using a default password on some unspecified new appliance and starts deleting everything.

They seem to be back together now with the exception of the data loss. The forums are back and there are lots of unhappy people. Most of them have no backups at all and lost years of data. Umm… just who’s fault is that again?

I don’t blame the company for the data loss, however their lack of feedback was a bad idea. Telling me to go away was probably dumb given they re-enabled the service later on though.

I’m using a Xen based VPS now. OpenVZ/Xen lessons learned, more later.