Page MenuHomePhabricator

Setup new wikis in Beta Cluster for Content Translation
Closed, DeclinedPublic

Description

Content Translation depends on user testing for new languages using Beta Cluster. To make sure user and developers can test well, we need to have all language wikis that Content Translation supports right now (and more as we grow).

We also need to effectively test the redirects to target language wikis, publish in target wikis, beta feature propagation etc Since, Beta is not exact configuration as production, issues like, https://www.mediawiki.org/w/index.php?title=Topic:Scfdmyjhftringtd tends to happen.

As per documentation, process is described at: https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/Add_a_wiki

Details

Related Gerrit Patches:
integration/config : masterBeta: Add wikis for ContentTranslation
operations/mediawiki-config : masterBeta: Add wikis for ContentTranslation

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Arrbee moved this task from Backlog to discussing on the LE-Sprint-84 board.Mar 11 2015, 9:07 AM
Arrbee raised the priority of this task from Medium to High.Mar 11 2015, 10:04 AM
Arrbee moved this task from discussing to Backlog on the LE-Sprint-84 board.
Arrbee moved this task from Backlog to In Progress on the LE-Sprint-84 board.Mar 24 2015, 8:05 AM
Arrbee edited a custom field.
Pginer-WMF edited a custom field.Apr 1 2015, 9:36 AM

Change 202689 had a related patch set uploaded (by KartikMistry):
Beta: Add wikis for ContentTranslation

https://gerrit.wikimedia.org/r/202689

KartikMistry moved this task from Backlog to In Progress on the LE-Sprint-85 board.Apr 8 2015, 7:31 AM

I dont think anything is blocked on Releng. The Add a wiki tutorial should covers all steps required and they are doable by anyone able to +2 on operations/mediawiki-config and in the wmf LDAP group to refresh the Jenkins job.

If there is something actually blocking, please be specific :)

Arrbee moved this task from In Progress to In Review on the LE-Sprint-85 board.Apr 14 2015, 8:57 AM
Arrbee moved this task from Long term to CX5 on the ContentTranslation board.Apr 20 2015, 6:06 AM
Arrbee edited projects, added LE-Sprint-86; removed LE-Sprint-85.Apr 21 2015, 7:26 AM
Arrbee moved this task from Backlog to In Progress on the LE-Sprint-86 board.Apr 21 2015, 7:30 AM
greg added a comment.Apr 21 2015, 3:28 PM

Just added a few more people to that review list (SPOFs are bad ;) ).

Change 202689 merged by jenkins-bot:
Beta: Add wikis for ContentTranslation

https://gerrit.wikimedia.org/r/202689

Change 206389 had a related patch set uploaded (by Hashar):
Beta: Add wikis for ContentTranslation

https://gerrit.wikimedia.org/r/206389

The wiki page https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/Add_a_wiki list several steps and the last one (configure Jenkins) hasn't been completed. I have proposed https://gerrit.wikimedia.org/r/#/c/206389/ to do so.

One has to verify that all steps have been completed ie:

  • databases created with addWiki.php)
  • search setup
  • Parsoid configuration
  • Jenkins update
KartikMistry moved this task from Blocked to In Review on the LE-Sprint-86 board.Apr 28 2015, 3:06 PM

@KartikMistry: I'm working on updating the docs on wikitech and hopefully automating the process a bit more.

@mmodell, that would be fantastic! Thank you!!

@KartikMistry it should be correct now, I'm going to try to further automate the process but this is less confusing now. Thanks to @demon, he did most of the documentation cleanup.

Legoktm added a subscriber: Legoktm.May 1 2015, 6:27 PM

So....right now CentralAuth is broken because these wikis haven't been set up yet.

Reverted the all-labs.dblist/wikiversions.json changes in https://gerrit.wikimedia.org/r/#/c/208170/ as it was breaking CentralAuth. The databases need to be created FIRST and then config changes done.

demon added a comment.May 1 2015, 7:22 PM

The databases need to be created FIRST and then config changes done.

This.

That add a wiki instruction is way outdated (and should be merged into the primary one). You should create the wiki prior to adding to wikiversions. In fact, addWiki does that for you! I think the idea was to use the new wiki's dbname as the --wiki param to addWiki.php but this is incorrect.

The invocation of addWiki.php is incredibly arcane.

mwscript addWiki.php --wiki=aawiki [lang] [site] [dbname] [domain]

The --wiki parameter actually turns out to be important. We need a master DB connection to an existing wiki on the same cluster as the target wiki. An existing wiki. Ideally this shouldn't happen but there's always a chance of a hook doing something dumb prior to getting to the core tables created and us ending up hitting a non-existent table. Also I'm not sure of the behavior on trying to get db config for a non-existing wiki (and even end up on the right cluster--this is untested territory).

Usually this will be s3. So most wikis (including the aawiki example) would be fine here. But if you really want to end up on a different cluster--which should be preconfigured in db-*.php as it is--then aawiki won't cut it.

The --wiki parameter actually turns out to be important. We need a master DB connection to an existing wiki on the same cluster as the target wiki. An existing wiki. Ideally this shouldn't happen but there's always a chance of a hook doing something dumb prior to getting to the core tables created and us ending up hitting a non-existent table. Also I'm not sure of the behavior on trying to get db config for a non-existing wiki (and even end up on the right cluster--this is untested territory).

does it really make sense to let hooks run in administration scripts?

demon added a comment.EditedMay 4 2015, 2:52 PM

The --wiki parameter actually turns out to be important. We need a master DB connection to an existing wiki on the same cluster as the target wiki. An existing wiki. Ideally this shouldn't happen but there's always a chance of a hook doing something dumb prior to getting to the core tables created and us ending up hitting a non-existent table. Also I'm not sure of the behavior on trying to get db config for a non-existing wiki (and even end up on the right cluster--this is untested territory).

does it really make sense to let hooks run in administration scripts?

In most cases yes, you would want hooks to run. For example: if you were running refreshLinks.php on a wiki you would want to make sure all parser-related hooks are there so you end up parsing the links as intended.

For addWiki we could possibly unset $wgHooks prior to execution. I'm still on the fence about it.

The Jenkins job that update the databases can probably read the .dblist T98342

Change 206389 abandoned by Hashar:
Beta: Add wikis for ContentTranslation

Reason:
The Jenkins job has been refactored and now reads the databases directly from all-labs.dblist

T98342 beta-update-databases-eqiad should get list of DB to update from mediawiki-config all-labs.dblist

Code https://gerrit.wikimedia.org/r/#/c/210619/3/jjb/beta.yaml,unified

That has been made simpler following your recent wiki additions :]

https://gerrit.wikimedia.org/r/206389

Arrbee edited projects, added LE-Sprint-87; removed LE-Sprint-86.May 25 2015, 8:08 AM
Arrbee moved this task from Backlog to In Progress on the LE-Sprint-87 board.May 25 2015, 8:15 AM

/poke

what's the status of this?

@mmodell, can you explain parameters of,

mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki \
    en wikimedia loginwiki login.wikimedia.beta.wmflabs.org

en = ?

demon added a comment.Jun 2 2015, 1:41 PM

--help explains them:

Arguments:
    <language>: Language code of new site, e.g. en
    <site>: Type of site, e.g. wikipedia
    <dbname>: Name of database to create, e.g. enwiki
    <domain>: Domain name of the wiki, e.g. en.wikipedia.org
Arrbee edited projects, added LE-Sprint-88; removed LE-Sprint-87.Jun 9 2015, 9:31 AM
dduvall added a subscriber: dduvall.

Unblocked per discussion in SoS. We'll revisit this once the Staging project is un-paused.

@KartikMistry is there anything left to do on this task? Seems all wiki have been created and are functional. If that is the case please close this bug, else poke me anytime so we can pair/sprint/finish it :-]

Arrbee lowered the priority of this task from High to Low.Jun 23 2015, 9:31 AM

Do we need any more testing environments? If not, and if I understand the title correctly, this can probably be closed.

Restricted Application added a subscriber: Luke081515. · View Herald TranscriptJul 2 2015, 7:12 AM
Amire80 moved this task from CX5 to CX6 on the ContentTranslation board.Jul 2 2015, 4:32 PM
Pginer-WMF moved this task from CX6 to CX7 on the ContentTranslation board.Jul 21 2015, 4:27 PM

Moved to externally blocked on the beta-cluster work-board.

New wiki databases are not on deployment-prep as of right now. If this is no longer a priority for ContentTranslation feel free to close.

hashar closed this task as Declined.Aug 24 2015, 10:30 AM

Apparently not needed anymore right now. Feel free to reopen when some more wikis are needed.

Restricted Application removed a subscriber: Liuxinyu970226. · View Herald TranscriptAug 24 2015, 10:30 AM