Note that in addition to these database VMs, the 'wikilabels' project itself has a Stretch VM that needs to be moved too: T306110: Cloud VPS "wikilabels" project Stretch deprecation

A few notes:

• wikilabels-02.wikilabels.eqiad1.wikimedia.cloud is currently shut down
• Examining wikilabels-staging-02 reveals that there is no local Postgres instance
• The tech wiki page on Wikilabels points this out:

wikilabels-02.eqiad1.wikimedia.cloud The main node and uses Postgresql (db1004.eqiad.wmnet) to work. It's accessible from labels.wmflabs.org
wikilabels-staging-01.eqiad1.wikimedia.cloud: The staging node, uses similar setup and accessible from labels-staging.wmflabs.org

• The onyl thing running on the staging machine is a uwsgi application:
  
  /usr/bin/uwsgi --die-on-term --ini /etc/uwsgi/apps-enabled/wikilabels-web.ini

My best guess is that the main VM is down for some reason (use is rare, as the tech wiki page points out. All the data should be living on machines managed by the Data Persistence team. Upgrading the machine to a new OS should be straightforward, assuming that the dependency set of the uwsgi application (and its dependency on Python features) are okay with rolling forward.

I will look into cloning the staging machine so I can do some potentially-destructive testing.

In T312564#8161734, @klausman wrote:

My best guess is that the main VM is down for some reason (use is rare, as the tech wiki page points out.

This is because of T306110: Cloud VPS "wikilabels" project Stretch deprecation, see the admin log https://sal.toolforge.org/wikilabels

All the data should be living on machines managed by the Data Persistence team. Upgrading the machine to a new OS should be straightforward, assuming that the dependency set of the uwsgi application (and its dependency on Python features) are okay with rolling forward.

Please see T307389: Upgrade wikilabels databases to buster/bullseye, which is why this task exists in the first place (not sure why we have these two separate tasks)

The wikilabels-02.wikilabels.eqiad1.wikimedia.cloud host has been replaced with wikilabels-03.wikilabels.eqiad1.wikimedia.cloud (on Buster), the wikilabels endpoint is back up and running. We have established that we need the endpoint, at least for the moment, to support ORES re-training of models, so we'll need to own and keep up the db part too.

As just added to T307389: DBs have been migrated and docs updated. Taavi has shut down the old clouddb instances and if we don't find we still need them for some reason, will delete them in a week.