Welcome back to TheAngelForever.com If you tried to stop by here over the last 31 hours, you probably noticed there was a problem going on. When TechyDad and I woke up yesterday, I went to check something on my blog. Much to my horror, my website was down. I went to TechyDad’s blog and nada there either. We were getting 500 Error messages about the server.
As time went by, we contacted Arvixe, the company that is our webhost. No real answers, lack of time schedule on the fix and on and on throughout the day. By noon, they finally admitted it was a hardware fail and were trying to fix things on the server. Then we noticed that data was not there. Thank goodness, TechyDad backs up our information nightly. The worst case scenario would have been me losing my Betty Crocker post (could redo if needed).
I had people that were questioning what was going on, so TechyDad set up something on the URLs of our blogs to let people know we were aware of the problem. We updated and put up a clock. This is what was here if you looked recently (click to enlarge):
Thankfully, we are up and running again and looking for a new host. We understand that things happen, but there is no excuse with not telling paying customers what is going on and updating things regularly – especially when we are talking over 24 hours of down time.
With a day lost, I will be trying to play even more catch up, so please bear with me as I settle back in. If you have any suggestions on web hosting companies please let us know. TechyDad is in charge of that and knows what he needs for our websites and his work.
Thank you again to everyone for understanding the technical difficulties that were beyond our control.
https://support.arvixe.com/index.php?_m=news&_a=viewnews&newsid=24 was posted on our website 1 hour after the incident. I understand your concerns. However it is almost impossible for us to gauge the severity of a downtime in a shorter amount than that. Also, we posted updates AS we found out more about the situation on the server.
From an internal perspective, to bring the server up in the amount of time that we did (6 hours) and to bring MySQL up without any data loss is something I don’t believe any other company would be able to do, big or small. (Considering the failure)
You are noting a 24+ hour downtime. Please note that all services were functioning properly within 6 hours. It took an entire day for 2 of our staff members to fix the corruption caused within MySQL after that 8 hour period.
If you did have a backup of your MySQL database, the site could have been restored shortly after the 8 hour downtime.
Arvand,
First of all, we knew about that news page, but it wasn’t updated often enough. It first gave an estimate of “a couple of hours.” Around 1pm – the 8 hour mark – it mentioned needing possibly 4 – 6 hours more. Then this morning – past the 24 hour mark – there was still no update. The last update was posted only *after* I complained about no updates being posted. Having a “news” page is useless if no new news is posted in a timely fashion.
Secondly, you didn’t get the servers back up and running completely in 6 hours time. True, the servers themselves were up and running, but the mySQL databases weren’t working. A WordPress blog can’t be considered “back up and running” without mySQL. mySQL (and thus our websites) were finally restored at the 29 hour mark.
While I did have a backup, I didn’t restore it for 3 reasons. First of all, it wasn’t as recent as the backup that Arvixe had. Secondly, I kept being told that my database was being restored. I figured that restoring from my backup might throw the process off. At the very least, we might get comments on our blog only to have them overwritten by the Arvixe backup being restored. Finally, the admin panel kept reporting “InnoDB” errors on the mySQL database. I didn’t think restoring my database would fix this. In fact, I was afraid it might worsen matters (by changing things in the middle of your debugging.) I mentioned this error to the support staff and was told it was being worked on. So I waited for it to be fixed.
We’re not so much mad about the hardware failure. Those things happen. And we’re happy you were able to restore with no data loss. What we aren’t happy about was the lack of communication during the process. That news page should have been updated more often with a timestamp on the update so people can see when the last update was. And if the final problem really was simply a corrupted mySQL data file, you should have given users the option of having a restore done from the last backup (quicker but might lose some data) or waiting for the fix (slower but might recover more data).
.-= TechyDad´s last blog ..Aloha Friday: Teaching Kids About Computers =-.
Arvand,
Thank you for the reply. I would like to note that I was not writing this post to bash Arvixe, it was to let my readers know why my blog was down for 31 hours. As you have probably seen in the comments, my husband/TechyDad has replied to you (http://www.theangelforever.com/?p=2795). No one will argue that we are beyond thrilled that all of the technicians worked long and hard to get everything back up and running without any data loss. That is fabulous news. What we are upset about is the lack of communication. You have the means to update your customers not only with the news page (which was not time stamped with status updates regularly), but you can use social media like your Twitter account and Facebook Fan Page. Be honest, let us know when something goes wrong. If you don’t know an ETA fine, but by all means please update with that every few hours with a sentence or two. Two updates over 24+ hours is not acceptable to customers that are left in the dark and not given answers when they make contact with your customer support.
TechyDad and I have recommended Arvixe to countless bloggers and other businesses for web hosting. We may look at other options, but that does not mean we will leave Arvixe. IF something like this happens again we hope that your company will keep us and others informed about things in a timely fashion.
[…] and hope to go to more in the future. As I mentioned Wednesday (before our websites went belly up again for 24+ hours), I went to a Cook’s Ham event held by Price Chopper Supermarkets on Tuesday. […]