We just got a hard deadline for this since Bahnhof is removing the platform where the server lives May 7.

If we want the same performance as our current server (2 GB RAM, 2 CPUs, 100 GB disk) it will cost 173,80 SEK/month. This is without backups. We've said that we don't need that for now, only when the lexicon is actively edited. Even then we can probably get away with periodically making dumps of the lexicon and saving somewhere rather than proper server backups. I'm guessing there's away to write to Google Drive from server via API, if we want them there.

Also, now that we have Grafana we don't need the smaller server with InfluxDB.

We've decided to primarily use Cloud VPS.

I created an instance for both wiki and Speecoid, since this is what we currently have on the Bahnhof server. I chose 2 CPUs and 4 GB, the other server has 2 CPUs and 2 GB.

The instance only has 20 GB regardless of flavour so I created a volume (doc) to store utterance files. I gave it 40 GB now, which is about twice what's in use. The idea is to have only utterance files on it, though it looks like there are other things taking up quite a bit of space too. Let's see what happens when installing things.

I used the Puppet class to setup LAMP: https://wikitech.wikimedia.org/wiki/Help:LAMP_instances. There are instructions for using Vagrant too, but according to the general Vagrant documentation it shouldn't be used for production. Also if something goes wrong in Vagrant it's usually a bit of a mess to sort out.

For some reason MariaDB stopped working giving error:

Fatal error: Can't open and lock privilege tables: Table 'mysql.db' doesn't exist

Running sudo mariadb-install-db seems to have solved it, but now all the databases are gone😒.

I made a first attempt at transferring the utterance files with ssh -3. It went for about an hour and then got Broken pipe. That only managed to transfer 1.1 %, probably due to the amount of files (over 1 million). There may also be overhead from running the command on a third machine, not sure how that works.

I ended up setting up ssh on the new machine and running rsync on it to get the files from the old machine. This was significantly faster, taking about an hour for all the files.

I installed Speechoid following the documentation. The Sox proxy doesn't install properly, but otherwise it seems to work.

In T331429#8720539, @Sebastian_Berlin-WMSE wrote:

For some reason MariaDB stopped working giving error:

Fatal error: Can't open and lock privilege tables: Table 'mysql.db' doesn't exist

Running sudo mariadb-install-db seems to have solved it, but now all the databases are gone😒.

This may have been related to Cloud VPS database config changes (T329571).

There's currently an error with creating new utterance files which prevents listening. I think this is due to a miss match between the utterances in the database and files. Hopefully re-importing the database will fix this, but I'll do that after the issue above has been fixed.

In T331429#8778127, @Sebastian_Berlin-WMSE wrote:

Running sudo mariadb-install-db seems to have solved it, but now all the databases are gone😒.

This may have been related to Cloud VPS database config changes (T329571).

Hi, happy to help here. But we gotta start with some general concepts. If you use puppet to install the whole LAMP system for you, that's a good idea and will give you the same system as before, but if you do that you should not also manually install mariadb packages. That is likely to conflict and make things confusing.

Now that being said, the ticket you linked above indeed has to do with the path that is used for the database files.

In your project Hiera config, in Horizon, please remove the override that sets it to /srv/sqldata.

Then I recommend simply starting with a new instance, applying the puppet role simplelamp2, as you have in the past, and running puppet.

Then you should get a working mariadb server that uses the standard path /var/lib/mysql and there won't be issues anymore with disappearing databases.

This will be the cleanest way to install things, there is no manual install and next time you get the exact same thing. It will be easy to recreate as many times as you like.

On the current system, you can look for old databases and verify things as such:

• check what the current datadir is that mysql actually uses currently, get this from mysql console with "'SHOW VARIABLES WHERE Variable_Name = "datadir"'"

• check what datadir is configured in /etc/my.cnf (this can be different from above if the service has not been restarted since the config was edited last time)

• if there is a difference between the 2, adjust the config but do NOT edit /etc/my.cnf manually. instead change the setting in your project Hiera and run puppet, puppet will change the file for you.

• manually restart the service with 'systemctl restart mysql'. Now the running datadir and config shoud be the same thing

• it's possible your databases that you are missing are in /srv/sqldata and now you are running on /var/lib/mysql or vice versa. look into both places, if they both have files you can switch between them as above and look at the difference. That there are 2 different pathes by default was part of the linked bug but /srv/ is NOT the default anymore.

• TLDR: do NOT use /srv/sqldata, easiest is to start over, do not manually run things, let puppet handle it, bug has been fixed, shouldn't happen again. forget about vagrant

Let me know if you need help or have questions.

also willing to talk in some realtime tool about it if you want. I am in US timezone though.

Thanks, @Dzahn. I think I got it working.

I've closed down the two bahnhof servers and requested a deletion of them.

Separately I've also requested a refund for the part of the month after the discontinuation date.

I've added a redirect in Loopia for wikispeech.wikimedia.se -> https://wikispeech-producer.wmcloud.org/. That should be the last thing needed for this.