(If there are better conventions for this kind of thing, please let me know.)
P&E Dashboard, running at outreachdashboard.wmflabs.org, requires server interventions to keep it running smoothly. I'm going to start documenting what I've done to try to keep it up and going.
T188864 - server ran out of disk space.
- mounted extra disk space on /srv and moving database backup files there.
An update process has been failing because of lack of space to write tmp tables to disk during particular database queries. The main filesystem has about 1.2 GB free.
- edited /etc/environment to set /srv/tmp as the TMPDIR environment variable
- restarted mysql
Free storage was down to ~600MB on the root filesystem, largely because of the continued growth of the MySQL database.
- Disabled the dashboard via a2dissite
- Moved the mysql datadir from /var/lib/mysql to /srv/mysql/mysql, following this guide: https://www.digitalocean.com/community/tutorials/how-to-move-a-mysql-data-directory-to-a-new-location-on-ubuntu-16-04
- Turned off mysql service
- rsync'd the database files
- moved the originals to mysql.bak
- added an AppArmor alias
- restarted AppArmor
- restarted mysql service
- Reenabled dashboard via a2ensite
Downtime was about 15 minutes.
During Febrary, Wes Reid and Sage Ross set up a new Debian VPS and transferred outreachdashboard.wmflabs.org to it, since the Ubuntu server was no longer supported on labs.
We attempted, with partial success, to set it up so that it can be deployed by multiple people (instead of only by Sage, as previously). In the process, we encountered a number of conflicts that can happen when a second user deploys. For now, deploys from multiple users are not practical until we solve those issues, although they can be manually worked around.
All data was transferred without issue, downtime was about 30 minutes, and the system has been running smoothly on the new server for the last two weeks.