Upgrading and other such games

Well, that was, err, interesting.

For reasons that seemed quite sensible at the time, I decided that today would be a good day to upgrade my Linode virtual server to a more recent version of its Linux distribution. There’s a quite nicely documented process for this, so away I went, pausing only to not take a backup more recent than this morning’s automagic one[1].

Well, it ran through all the expected stuff, but then cam up with some sulky messages which suggested that it wasn’t at all happy with the installed firewall, and as a result of that it wasn’t going to let MySQL start, so there. Now as this site is ever so slightly dependent on MySQL, that was a bit of a problem.

I tried some of the basic troubleshooting steps, but this didn’t help at all. After a bit more fiddling, I decided it was time to back out and revert to a backup. This is a reasonably simple process with Linode:

  1. Shut down the running server
  2. Resize its disk partition to make room for a restore
  3. Restore from last backup
  4. Wait
  5. Reboot into restored system

There were a few more jobs I needed to do, though. As I’d typed a couple of posts after the backup was taken, I copied the actual database files from the slightly duff server to my computer first, then when I had the restored one running, I stopped the MySQL demon, replaced the database files from my copy then restarted MySQL. All appeared to be well, but I had missed a step.

When I copied the files from the duff server, I did it using the root account, and did the same when I copied them back to the good server. This meant that they were owned by root, and while the MySQL server could open the database, it couldn’t actually save any changes, which caused me a few moments of confusion. It was then just a matter of stoping MySQL again – quite likely not necessary, but I thought I’d play safe – then running a quick

chown -R mysql:mysql foldername

to restore ownership. I then restarted MySQL and all was as it should be.

Finally, as I was happy that I was back where I should be, I deleted the disk partitions used by the duff server, shut down the good one and resized its disk to use all the space, then restarted it.

And if you can read this, it all worked. If you can’t read this, it didn’t.

I’ll try the upgrade another time, most likely by a less exciting method involving starting a new Linode, updating that, then moving all my stuff across to it and shutting down and removing the old one. As Linode charge only for the time you actually have a Linode in use, it’s not an expensive option to bring up a temporary server. I probably should have done that the first time…

[1] Yes, yes, silly me, etc