Page MenuHomePhabricator

Deploy MediaModeration Extension to Wikimedia Production
Closed, ResolvedPublic

Description

This is the top-level task for deploying MediaModeration.

MediaModeration is an extension developed by the Platform Engineering together with Trust-and-Safety team for automatic detection of child abuse images using PhotoDNA provider.

  • Add the new extension submodule to the git mediawiki/extensions repo
  • Code Review.
  • Security Review for MediaModeration T248483
  • Review by the product owner @CCicalese_WMF
  • Move extension CI config to wikimedia-deployed section
  • Add extension to the make-wmf-branch release tool
  • Deploy on beta sites
    • Add extension to extension-list
    • Add secret keys to Beta Cluster config
    • Load extension in mediawiki-config
  • Add secret keys to production config
  • Enable on group0
  • Update https://www.mediawiki.org/wiki/Extension:MediaModeration accordingly ("Release status" etc)
  • Deploy on all wikis

Event Timeline

Pchelolo claimed this task.Mar 18 2020, 2:03 PM

Assigning to myself to fill in the details.

CCicalese_WMF renamed this task from Deploy MediaWiki Extension (start deployment pipeline) to Deploy MediaModeration Extension to Wikimedia (start deployment pipeline).Mar 18 2020, 2:23 PM
CCicalese_WMF triaged this task as Medium priority.
Pchelolo renamed this task from Deploy MediaModeration Extension to Wikimedia (start deployment pipeline) to Deploy MediaModeration Extension to Wikimedia Production.Mar 25 2020, 2:42 PM
Pchelolo updated the task description. (Show Details)
Pchelolo added a subscriber: CCicalese_WMF.
Pchelolo updated the task description. (Show Details)Mar 25 2020, 3:08 PM
Jdforrester-WMF added a subscriber: Jdforrester-WMF.

The "Add extension to the make-wmf-branch release tool" step has to happen two weeks before you start to deploy (i.e., be included in both production branches), or production's i18n build step will blow up.

Change 599363 had a related patch set uploaded (by Cicalese; owner: Cicalese):
[operations/mediawiki-config@master] added MediaModeration

https://gerrit.wikimedia.org/r/599363

Change 599362 had a related patch set uploaded (by Cicalese; owner: Cicalese):
[integration/config@master] move MediaModeration to production

https://gerrit.wikimedia.org/r/599362

Change 599366 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[mediawiki/tools/release@master] Start branching the MediaModeration extension for production

https://gerrit.wikimedia.org/r/599366

Change 599366 merged by jenkins-bot:
[mediawiki/tools/release@master] Start branching the MediaModeration extension for production

https://gerrit.wikimedia.org/r/599366

Change 599369 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[operations/mediawiki-config@master] Install MediaModeration extension - I: Add i18n

https://gerrit.wikimedia.org/r/599369

Change 599370 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[operations/mediawiki-config@master] Install MediaModeration extension - II: Add flag to IS

https://gerrit.wikimedia.org/r/599370

Change 599362 merged by jenkins-bot:
[integration/config@master] layout: Move MediaModeration to production section and enforce

https://gerrit.wikimedia.org/r/599362

Mentioned in SAL (#wikimedia-releng) [2020-05-28T16:10:10Z] <James_F> Zuul: Move MediaModeration to production section and enforce T247943

The "Add extension to the make-wmf-branch release tool" step has to happen two weeks before you start to deploy (i.e., be included in both production branches), or production's i18n build step will blow up.

Thank you for the additional direction, @Jdforrester-WMF.

Is there a way to shorten that period in any way, or would that introduce unacceptable instability? For future reference, where should we be looking for details like the timing involved?

Do you see any other potential issues in the set of steps documented in the task description? If so, please fee free to add more detail there, such as where we'll need to pause. Thank you.

The "Add extension to the make-wmf-branch release tool" step has to happen two weeks before you start to deploy (i.e., be included in both production branches), or production's i18n build step will blow up.

Thank you for the additional direction, @Jdforrester-WMF.

Is there a way to shorten that period in any way, or would that introduce unacceptable instability?

Theoretically in a emergency we could try to back-port it and do a full production deployment out of sequence, but I'd feel a lot better with Manager (Tyler) or even Director (Greg) sign-off there; it's a pretty serious difference from normal, and though the risk is probably OK, that's not generally my call. :-) (There are steps in the production deployment process which can't be rolled back quickly; spending 30 minutes with major i18n issues in all production wikis whilst waiting for the re-build would be Not Fun.)

For future reference, where should we be looking for details like the timing involved?

Some of this is documented (see e.g. Preparing for deployment), but because we keep changing things to try to make systems more robust and easier to use, the best thing is to meet with RelEng (me, or whomever Tyler has available) to talk about deployment. We're always happy to help.

Do you see any other potential issues in the set of steps documented in the task description? If so, please fee free to add more detail there, such as where we'll need to pause. Thank you.

No, other than the concern about PrivateSettings for Labs; I've tweaked the runlist, but it looks good to me.

OK, thanks for the context.

Aklapper updated the task description. (Show Details)May 29 2020, 6:58 AM

Change 599369 merged by jenkins-bot:
[operations/mediawiki-config@master] Install MediaModeration extension - I: Add i18n

https://gerrit.wikimedia.org/r/599369

Change 599370 merged by jenkins-bot:
[operations/mediawiki-config@master] Install MediaModeration extension - II: Add flag to IS

https://gerrit.wikimedia.org/r/599370

Mentioned in SAL (#wikimedia-operations) [2020-06-17T09:02:20Z] <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T247943 Install MediaModeration extension - II: Add flag to IS (duration: 01m 05s)

Mentioned in SAL (#wikimedia-releng) [2020-06-17T15:05:43Z] <James_F> Private config for MediaModeration extension added on Beta Cluster for T247943

Change 599363 merged by jenkins-bot:
[operations/mediawiki-config@master] Install MediaModeration extension - III: Install where enabled

https://gerrit.wikimedia.org/r/599363

Mentioned in SAL (#wikimedia-operations) [2020-06-17T15:11:34Z] <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: T247943 Install MediaModeration extension - III: Install where enabled (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2020-06-17T15:17:25Z] <jforrester@deploy1001> Synchronized private/PrivateSettings.php: T247943 Add API key and recipient config for MediaModeration (duration: 00m 55s)

Change 606209 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[operations/mediawiki-config@master] Install MediaModeration extension - IV: Enable on Beta Clusetr

https://gerrit.wikimedia.org/r/606209

Change 606209 merged by jenkins-bot:
[operations/mediawiki-config@master] Install MediaModeration extension - IV: Enable on Beta Clusetr

https://gerrit.wikimedia.org/r/606209

Change 606222 had a related patch set uploaded (by Cicalese; owner: Cicalese):
[operations/mediawiki-config@master] Add temporary logging for mediamoderation

https://gerrit.wikimedia.org/r/606222

Change 606222 merged by jenkins-bot:
[operations/mediawiki-config@master] Add temporary logging for mediamoderation

https://gerrit.wikimedia.org/r/606222

Mentioned in SAL (#wikimedia-operations) [2020-06-17T18:31:46Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 96153f9: Add temporary logging for mediamoderation (T247943) (duration: 00m 56s)

Change 606239 had a related patch set uploaded (by Cicalese; owner: Cicalese):
[operations/mediawiki-config@master] DO NOT MERGE Remove temporary logging for mediamoderation

https://gerrit.wikimedia.org/r/606239

Change 607327 had a related patch set uploaded (by Cicalese; owner: Cicalese):
[operations/mediawiki-config@master] Enable MediaModeration on group0

https://gerrit.wikimedia.org/r/607327

Change 607327 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable MediaModeration on group0

https://gerrit.wikimedia.org/r/607327

Change 608062 had a related patch set uploaded (by Cicalese; owner: Cicalese):
[operations/mediawiki-config@master] Add HTTP proxy to MediaModeration.

https://gerrit.wikimedia.org/r/608062

tomasz added a subscriber: tomasz.EditedJun 28 2020, 11:14 AM

Please can the release of this extension be advertised to all local oversighters at least a few days in advance, as this would be a very important addition to their tools. The Commons oversighters, for example, have not received any (semi-)official information as of yet.

drochford added a comment.EditedJun 29 2020, 2:57 PM

Please can the release of this extension be advertised to all local oversighters at least a few days in advance, as this would be a very important addition to their tools. The Commons oversighters, for example, have not received any (semi-)official information as of yet.

This tool should not be accessible to oversighters in any way. The output of the tool is internal facing only.

tomasz removed a subscriber: tomasz.Jun 29 2020, 4:06 PM

Change 608062 merged by jenkins-bot:
[operations/mediawiki-config@master] Add HTTP proxy to MediaModeration.

https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/ /608062

Mentioned in SAL (#wikimedia-operations) [2020-06-29T18:15:15Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: c86fcd4: Add HTTP proxy to MediaModeration (T247943) (duration: 00m 58s)

CCicalese_WMF updated the task description. (Show Details)Jul 1 2020, 1:24 AM

Change 608753 had a related patch set uploaded (by Cicalese; owner: Cicalese):
[operations/mediawiki-config@master] Deploy MediaModeration on all production wikis.

https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/ /608753

Change 608753 merged by jenkins-bot:
[operations/mediawiki-config@master] Deploy MediaModeration on all production wikis.

https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/ /608753

Iniquity added a comment.EditedJul 2 2020, 10:58 AM

It just seems to me that the Extension is quite important, and this can attract more volunteers for it. If it is necessary, of course.

Telling users that there is some special software they can't see, can't use, and can't help with, which won't change anything ahout their experience on the wikis, and which unless they're already aware through T&S will never hear of again doesn't seem a good use of Tech News, or of the tech ambassadors mailing list. What do you think is necessary?

Iniquity added a comment.EditedJul 2 2020, 11:14 AM

Yes, you're right, I read the description incorrectly, I decided that there is manual work. In any case, it seems to me that it will be useful to convey information about the existence of the extension to the community. At a minimum, it will become known that there is a second level of protection against illegal materials. Maybe I'm wrong, thanks for the answer anyway :)

CCicalese_WMF closed this task as Resolved.Jul 2 2020, 12:58 PM

Resolved?

Indeed!