Forum backups and occasional slow response - how to help

[overscan (PaulMM)]

I've implemented the final version of the forum backup system. You may have suffered from slow response the last coupla hours while I was running and rerunning the scripts fotr debugging.

The forum database is automatically backed up every day to a local FTP server provided by my hosting company. 8 backups are kept (7 daily backups and one monthly). The same backups are replicated to Amazon S3 storage. Forum attachments are saved daily to Amazon S3.

It take about 2 minutes to do a full database dump. During this time, the site won't respond. That happens once a day. For about 5 mins after that, its compressing the backup, but this doesn't seem to cause any performance issues.

Without major reengineering work, this is the best I can do for the size of the database (500MB+ and growing).

How can we make the database smaller and quicker to backup? Well, actually you can help here!

Firstly, you can prune old PMs. When viewing PMs go to Actions menu - Prune Messages.

Secondly, you can mark all messages as read occasionally. Keeping track of how far every user has read in each one of 14082 topics takes quite a bit of space. There's a button to do this between the forum boards and the recent posts panel on the right.

Hopefully the forum performance is good overall.

 

[overscan (PaulMM)]

Re: Forum backups and occasional slow response - mostly resolved

I have fixed an issue causing slowness during backups to Amazon S3.

This was caused by s3cmd doing an MD5 hash of all files to check if they had changed - this included all 20GB of image attachments. This is very cpu and disk intensive and resulted in slow performance at times during backups. Backups were also taking much longer than expected.

This option has been turned off, due to nature of the forum software it isn't needed. Backups now take a fraction of the time and don't slow the server at all.

There will still be a 2-3 minute period per day of no response while the database is backed up - this is trickier to fix and probably requires a switch to a new database format & software.

 

[Stargazer2006]

Thanks a lot, Paul.