Page MenuHomePhabricator

all.dblist in beta-labs is a live hack
Closed, ResolvedPublic

Description

On beta, the /home/wikipedia/common/all.dblist is a live hack. It holds the list of database configured on the cluster which, on beta, is different from the one on production.

The beta databases should be listed in a new file named beta.dblist or wmflabs.dblist and all our tools must be adapted to detect they are on beta and load the new file.

Currently, when someone does a git pull and erase the all.dblist, the first database is listed as aawikibooks which does not exist on beta. That causes at least the job runner to die since it can not find the database and throw an exception.


Version: unspecified
Severity: normal

Details

Reference
bz41133

Related Objects

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 12:56 AM
bzimport set Reference to bz41133.

Some scripts in extensions/WikimediaMaintenance still refers to all.dblist explicitly. Among others:

addWiki.php
dumpInterwiki.php
rebuildInterwiki.php
storage/make-all-blobs

I have a vague plan to fix this as a general case, by adjusting everything I can find that looks at *.dblist to first load the realm from /etc/wikimedia-realm (if that exists) and check if *-$realm.dblist exists as an override. Then we can actually commit all the beta-specific versions of these files into git where they should be.

It looks like I'll also wind up doing the same for wikiversions.dat, and hopefully cleaning up some of the rest of the untracked changes in the beta config.

Ok, I wound up having to touch operations/mediawiki-config, operations/mediawiki-multiversion, and mediawiki/extensions/WikimediaMaintenance to make this change. And it all needs thorough review.

Gerrit changeset 32167 has the changes to operations/mediawiki-config
Gerrit changeset 32168 has the changes to operations/mediawiki-multiversion
Gerrit changeset 32169 has the changes to mediawiki/extensions/WikimediaMaintenance

I think that issue has been fixed when we implemented the multiple datacenter features in mediawiki-config. There is an all-labs.dblist file now since https://gerrit.wikimedia.org/r/#/c/39059/ and IIRC it should be loaded properly when using getRealmSpecificFilename( '/path/to/all.dblist').

Will have to double check all our script properly use that list of databases.

We now vary based on the realm. The all.dblist is replaced by all-labs.dblist whenever we run on labs.