Exclude testwikis and private wikis from CX draft purge script and separate CX database on testwiki
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	KartikMistry
	Sep 21 2020, 9:05 AM

Description

From T261189:

The pre-existing issue is that the list of wikis include test.wikipedia.org. This is a special wiki where CX has been configured to "translate in the target", but still used the shared, global database tables.

This means that when the loop gets to testwiki, the purge script detects this condition and instead of only going over drafts *->testwiki (I guess en, but doesn't matter), it goes over all drafts and purges and notifies them. So for any wiki that appears after testwiki in the list, our script would not do anything, since testwiki already went over them. To make things worse, the purges for testwiki succeed, but any notification will fail (because it would need to send the notification to an user in another wiki).

Suggested fixes:

Immediately remove testwiki from the script runs

Setup CX on testwiki so that it does not used the shared database tables, but instead of a local copy of the tables. If this is done, testwiki can be restored on the script runs.
- Create CX tables on the testwiki.
- Remove testwiki from extension1 cluster.

Details

Subject	Repo	Branch	Lines +/-
Follow-up for 483beb2: wmgContentTranslationDatabase => wgContentTranslationDatabase	operations/mediawiki-config	master	+1 -1
ContentTranslation: Do not use wikishared DB for testwiki	operations/mediawiki-config	master	+6 -2
Revert "Set ContentTranslationDatabase to wikishared only if ContentTranslationCluster is set"	operations/mediawiki-config	master	+2 -1
Revert "ContentTranslation: Remove testwiki from extension1 cluster"	operations/mediawiki-config	master	+0 -1
Set ContentTranslationDatabase to wikishared only if ContentTranslationCluster is set	operations/mediawiki-config	master	+1 -2
ContentTranslation: Remove testwiki from extension1 cluster	operations/mediawiki-config	master	+1 -0
Exclude testwikis and private wikis from CX draft purge script run	operations/puppet	production	+1 -1

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		KartikMistry	T261189 Enable user notifications for old unpublished ContentTranslation draft purge script
		Resolved		KartikMistry	T263417 Exclude testwikis and private wikis from CX draft purge script and separate CX database on testwiki

Event Timeline

KartikMistry created this task.Sep 21 2020, 9:05 AM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptSep 21 2020, 9:05 AM

Change 628758 had a related patch set uploaded (by KartikMistry; owner: KartikMistry):
[operations/puppet@production] Exclude testwikis and private wikis from CX draft purge script run

https://gerrit.wikimedia.org/r/628758

gerritbot added a project: Patch-For-Review.Sep 21 2020, 9:05 AM

KartikMistry moved this task from Quarter Backlog to In Review on the Language-Team (Language-2020-July-September) board.Sep 21 2020, 11:06 AM

My plan is:

Create CX tables on testwiki (during B&C deployment) and,

Set wmgContentTranslationCluster to false for testwiki.

KartikMistry updated the task description. (Show Details)Sep 21 2020, 12:38 PM

Change 628790 had a related patch set uploaded (by KartikMistry; owner: KartikMistry):
[operations/mediawiki-config@master] ContentTranslation: Move testwiki off extension1 cluster

https://gerrit.wikimedia.org/r/628790

KartikMistry added a subscriber: • Marostegui.Sep 21 2020, 12:40 PM

These new tables contain or will store any private data that shouldn't be public?

These tables are equivalent to the CX tables in wikishared. These tables can follow the same policy as those in wikishared, which I think are all public but not in public replicas at the moment.

We are doing this because testwiki being special is causing issues with CX if it uses the shared tables.

CX tables aren't marked as private from what I can see, but wikishared isn't replicated to the replicas (as the whole x1 isn't included).
So I won't add any special filter for those new tables then and will let them replicate.

Thank you!

• Marostegui unsubscribed.Sep 21 2020, 3:45 PM

Change 628758 merged by Alexandros Kosiaris:
[operations/puppet@production] Exclude testwikis and private wikis from CX draft purge script run

https://gerrit.wikimedia.org/r/628758

KartikMistry updated the task description. (Show Details)Sep 23 2020, 6:14 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-23T11:08:55Z] <Urbanecm> Create ContentTranslation tables at testwiki using SQL files from /srv/mediawiki/php-1.36.0-wmf.10/extensions/ContentTranslation/sql (T263417

Change 628790 merged by jenkins-bot:
[operations/mediawiki-config@master] ContentTranslation: Remove testwiki from extension1 cluster

https://gerrit.wikimedia.org/r/628790

Change 629360 had a related patch set uploaded (by KartikMistry; owner: KartikMistry):
[operations/mediawiki-config@master] Set ContentTranslationDatabase to wikishared only if ContentTranslationCluster is set

https://gerrit.wikimedia.org/r/629360

Change 629360 merged by jenkins-bot:
[operations/mediawiki-config@master] Set ContentTranslationDatabase to wikishared only if ContentTranslationCluster is set

https://gerrit.wikimedia.org/r/629360

Change 629204 had a related patch set uploaded (by Urbanecm; owner: Urbanecm):
[operations/mediawiki-config@master] Revert "ContentTranslation: Remove testwiki from extension1 cluster"

https://gerrit.wikimedia.org/r/629204

Change 629205 had a related patch set uploaded (by Urbanecm; owner: Urbanecm):
[operations/mediawiki-config@master] Revert "Set ContentTranslationDatabase to wikishared only if ContentTranslationCluster is set"

https://gerrit.wikimedia.org/r/629205

KartikMistry updated the task description. (Show Details)Sep 23 2020, 11:42 AM

Change 629204 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "ContentTranslation: Remove testwiki from extension1 cluster"

https://gerrit.wikimedia.org/r/629204

Change 629205 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "Set ContentTranslationDatabase to wikishared only if ContentTranslationCluster is set"

https://gerrit.wikimedia.org/r/629205

Change 629371 had a related patch set uploaded (by KartikMistry; owner: KartikMistry):
[operations/mediawiki-config@master] WIP: ContentTranslation: Do not use wikishared for testwiki

https://gerrit.wikimedia.org/r/629371

KartikMistry triaged this task as Medium priority.Sep 28 2020, 5:13 AM

Change 629371 merged by jenkins-bot:
[operations/mediawiki-config@master] ContentTranslation: Do not use wikishared DB for testwiki

https://gerrit.wikimedia.org/r/629371

Change 630565 had a related patch set uploaded (by Urbanecm; owner: Urbanecm):
[operations/mediawiki-config@master] Follow-up for 483beb2: wmgContentTranslationDatabase => wgContentTranslationDatabase

https://gerrit.wikimedia.org/r/630565

Change 630565 merged by jenkins-bot:
[operations/mediawiki-config@master] Follow-up for 483beb2: wmgContentTranslationDatabase => wgContentTranslationDatabase

https://gerrit.wikimedia.org/r/630565

Mentioned in SAL (#wikimedia-operations) [2020-09-28T11:20:20Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 483beb2452caead8c44dfb8e608812778033fba0: ContentTranslation: Do not use wikishared DB for testwiki (T263417; follow-up af09303a4a155681b198ac70468494c2155868df also included in this sync) (duration: 00m 57s)

KartikMistry renamed this task from Exclude testwikis and private wikis from old unpublished CX draft purge script run to Exclude testwikis and private wikis from CX draft purge script and separate CX database on testwiki.Sep 28 2020, 11:27 AM

KartikMistry updated the task description. (Show Details)

KartikMistry moved this task from In Review to Needs QA on the Language-Team (Language-2020-July-September) board.

Notes for QA

ContentTranslation is working on testwiki.
Same user can't see Drafts from other Wikipedias on testwiki.
testwiki draft is not available in other Wikipedias.

Mentioned in SAL (#wikimedia-operations) [2020-09-28T11:43:36Z] <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: 483beb2452caead8c44dfb8e608812778033fba0: ContentTranslation: Do not use wikishared DB for testwiki (T263417; follow-up af09303a4a155681b198ac70468494c2155868df also included in this sync) (duration: 00m 56s)