Page MenuHomePhabricator

More automation for the deployment of cxserver and Wikidata when creating new wikis
Open, MediumPublic

Description

Thanks to @Ladsgroup's work to T253439: Eliminate the toil in WMF wiki creation, wiki creation is now easier than it had been.

However, there are still several things that could possibly become even easier. The two most notable ones are the connection of the new wiki to Wikidata and to cxserver, the service that makes using the Content Translation extension possible (the latter is only relevant for Wikipedias at the moment, while the former is relevant for almost all wikis).

These two things usually get enabled within about a week after the wiki is created, and requires some manual deployment steps. Ideally, this time should be somewhere between zero minutes and maybe ten minutes from the wiki's creation.

I'm not a true expert in how wiki creation works, but some knowledgeable people told me that it's probably doable without making very huge changes, and suggested that I create a task, so here it is. I'm putting Wikidata, Parsoid and cxserver together because they are so closely related. In addition, Content Translation cannot fully function without Wikidata and Parsoid, so whenever this happens, Wikidata and Parsoid have to be done first, and cxserver deployment immediately after that.

  • Wikidata
  • Parsoid
  • cxserver

Event Timeline

We need to decouple dependency of cxserver/config/languages.yaml here and read list of languages from elsewhere (mediawiki-config?). This will remove dependency of cxserver deployment from Wiki creation process completely.

I can't think of any way to read mediawiki-config from cxserver. We could use some API to get list of Wikipedia periodically and cache it.

So on the Wikidata-side, it's documented here: https://wikitech.wikimedia.org/wiki/Add_a_wiki#Wikidata

Technically it's rather straightforward to automate (it's just rebuilding sites table for all wikis) but in reality, the script fails from time to time (and you need to run it on more than 1K wikis separately, one by one) and without human supervision it happens to break basically in every new wiki and the important part is that it needs to immediately re-ran it otherwise the whole wiki breaks.

That can be sorta mitigated too but the underlying problem is that we have 1k site store instead of a centralized one. So T113034: RFC: Overhaul Interwiki map, unify with Sites and WikiMap would automatically fix the issue but so far there hasn't been a push to actually implement the approved RFC.

Hello,

I can't think of any way to read mediawiki-config from cxserver. We could use some API to get list of Wikipedia periodically and cache it.

can you read https://noc.wikimedia.org/conf/highlight.php?file=langlist or https://noc.wikimedia.org/conf/highlight.php?file=dblists/wikipedia.dblist? That could be ae a solution IMO.

Noc is not acceptable. It has to be a proper api with stability and uptime agreements.

Pginer-WMF triaged this task as Medium priority.Jul 18 2024, 8:54 AM
Pginer-WMF moved this task from Backlog to Deployment preparation on the CX-cxserver board.