Page MenuHomePhabricator

Split MediaSearch out into its own extension
Open, Needs TriagePublic

Description

As discussed in a recent engineering team meeting, there are several advantages to splitting MediaSearch out of WikibaseMediaInfo into its own extension:

  • We can set up Jest for front-end testing without having to deal with the existing node-qunit frameowrk in WBMI
  • We can use the opportunity to create some boilerplate code for future projects (test framework, potentially webpack config or Storybook implementation)
  • If an opportunity to use MediaSeach on another wiki comes up in the future, the code will already be modular and separate from WBMI
  • WBMI is becoming quite large and modular code will be easier to maintain going forward

Here's a rough outline of what needs to be done; I'll split some of this out into subtasks later:

  • Request a new Gerrit repo
  • Build new extension, get CR from team
  • Determine if we need security readiness review and, if so, get the ball rolling on that
  • Remove code from WBMI
  • Coordinate deployment (part of which will be T278121)
  • Documentation: extension page and help page on mediawiki.org, add to https://www.mediawiki.org/wiki/Developers/Maintainers

Deployment process

Deployment 0 (can be regular deployment train)

  • Prepare i18n messages:
    • Build MediaSearch i18n messages from 2 sources. Patch: 674609
    • Use consistent prefix for mediasearch i18n messages. Patch: 682095
    • Remove messages. Patch: 682096
  • Get Extension:MediaSearch repo in production; see documentation: Patch: 682101

Deployment 1 (backport & config deployment window)

  • testcommons
    • Prepare any new config vars that MediaSearch requires (if any) & include the extension on testcommons. Patch: 682102
    • Deploy config
    • Verify that nothing break
  • commons

Deployment 2 (backport & config deployment window)

  • Prepare, merge, backport & pull these patches onto deployment host:
    • Disable search widget takeover (in WikibaseMediaInfo) - Patch: 674015
    • Enable search widget takeover (in MediaSearch) - Patch: 674010
    • Rename Special:MediaSearch to Special:OldMediaSearch (in WikibaseMediaInfo) - Patch: 676613
    • Rename Special:NewMediaSearch to Special:MediaSearch (in MediaSearch) - Patch: 676614
  • Pull both of these WikibaseMediaInfo changes in staging server & check that nothing unexpected breaks
  • Pull both of these MediaSearch changes in staging server & test that everything works
  • Sync both of these WikibaseMediaInfo changes (Special:MediaSearch will not be accessible for a couple of minutes until next step; search widget requests will go to Special:Search)
  • Sync both of these MediaSearch patches (Special:MediaSearch is back, and so is the search widget takeover)
  • Verify all Special:MediaSearch functionality again

Post-deployment 2: work with translatewiki folks to move messages from WBMI to MediaSearch (see subtask for details)

Deployment 3 (can be regular deployment train)

  • Remove all relevant code from WikibaseMediaInfo
  • Verify all Special:MediaSearch functionality again

Event Timeline

Mentioned in SAL (#wikimedia-releng) [2020-10-20T22:15:08Z] <James_F> Zuul: [mediawiki/extensions/MediaSearch] Install CI for this new prod repo T265939

Jdforrester-WMF added a subscriber: Jdforrester-WMF.

I've installed CI for this new repo; shout if you need help with anything else in the process.

Change 636499 had a related patch set uploaded (by Anne Tomasevich; owner: Anne Tomasevich):
[mediawiki/extensions/MediaSearch@master] [WIP] To do: - Transfer i18n messages from WBMI to this extension. Thus far, only the api files and the en and qqq main files have been copied over - Determine strategy for the above change to avoid disrupting translatewiki.net - Determine strategy for deploying, given we need to disable and eventually remove this code from WBMI to avoid conflicts

https://gerrit.wikimedia.org/r/636499

Change 636506 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] Zuul: [mediawiki/extensions/MediaSearch] Add WikibaseCirrus dep.

https://gerrit.wikimedia.org/r/636506

Change 636506 merged by jenkins-bot:
[integration/config@master] Zuul: [mediawiki/extensions/MediaSearch] Add WikibaseCirrus dep.

https://gerrit.wikimedia.org/r/636506

Mentioned in SAL (#wikimedia-releng) [2020-10-26T22:21:16Z] <James_F> Zuul: [mediawiki/extensions/MediaSearch] Add WikibaseCirrus dependency T265939

Change 636499 abandoned by Anne Tomasevich:
[mediawiki/extensions/MediaSearch@master] [WIP] Move MediaSearch code into its own extension

Reason:
We've decided not to move forward with this

https://gerrit.wikimedia.org/r/636499

Have we decided that we're not moving forward with this *at all* (in which case this ticket can be declined), or simply that we won't move forward with the in-progress patch but will likely pick up again at some later point, when we have the time to wait it out (in which case we can move this ticket elsewhere)?

I think there's a case for the latter if we continue work on MediaSearch or if it would benefit others to have the MediaSearch front-end broken out and made more agnostic (if that's even feasible). Let's discuss at the first engineering office hours of the new year?

Yeah, makes sense. FWIW, I also still think it'd be worth separating the frontend into it's own thing at some point (to allow it to be installed on other wikis without the current stringent dependency requirements)
Moving to "blocked" for the time being, until we're in a position where we have the time to re-init & wait out security review etc.

Change 636499 restored by Anne Tomasevich:
[mediawiki/extensions/MediaSearch@master] [WIP] Move MediaSearch code into its own extension

https://gerrit.wikimedia.org/r/636499

Change 662792 had a related patch set uploaded (by Anne Tomasevich; owner: Anne Tomasevich):
[operations/mediawiki-config@master] Add external entity search URI for new MediaSearch extension

https://gerrit.wikimedia.org/r/662792

Patch has been approved, but won't be testable until it's enabled on-wiki, which is blocked on security readiness review (T266513)
Moving to blocked until that is complete.
After that, we can pick up the remaining work of enabling the extension.

Change 662792 merged by jenkins-bot:
[operations/mediawiki-config@master] Add external entity search URI for new MediaSearch extension

https://gerrit.wikimedia.org/r/662792

Change 636499 merged by jenkins-bot:
[mediawiki/extensions/MediaSearch@master] Move MediaSearch code into its own extension

https://gerrit.wikimedia.org/r/636499

Change 663790 had a related patch set uploaded (by Matthias Mullie; owner: Matthias Mullie):
[mediawiki/extensions/MediaSearch@master] Apply diff up to dc7b9a1b218eac25a18c30a5f9afc36ca7e040da

https://gerrit.wikimedia.org/r/663790

Change 663790 merged by jenkins-bot:
[mediawiki/extensions/MediaSearch@master] Apply diff up to dc7b9a1b218eac25a18c30a5f9afc36ca7e040da

https://gerrit.wikimedia.org/r/663790

Change 670182 had a related patch set uploaded (by Matthias Mullie; owner: Matthias Mullie):
[mediawiki/extensions/MediaSearch@master] Apply diff up to 25b8791bcadac46d4bc3da0189fe225cbc5d4fcd

https://gerrit.wikimedia.org/r/670182

Change 670182 merged by jenkins-bot:
[mediawiki/extensions/MediaSearch@master] Apply diff up to 25b8791bcadac46d4bc3da0189fe225cbc5d4fcd

https://gerrit.wikimedia.org/r/670182

Change 673608 had a related patch set uploaded (by Anne Tomasevich; owner: Anne Tomasevich):
[mediawiki/extensions/MediaSearch@master] Apply diff up to d10a4f6c8d592d7b29d0b035ee3d0b2363f8f9f2

https://gerrit.wikimedia.org/r/673608

Change 673608 merged by jenkins-bot:
[mediawiki/extensions/MediaSearch@master] Apply diff up to d10a4f6c8d592d7b29d0b035ee3d0b2363f8f9f2

https://gerrit.wikimedia.org/r/673608

@matthiasmullie would you mind outlining the process you think we should follow to transition from Special:NewMediaSearch to Special:MediaSearch in the new extension? Will we need to merge this change right before deployment with a corresponding one in WBMI?

I'm thinking something like this:

Deployment 0 (backport & config deployment window)

  • Prepare any new config vars that MediaSearch requires (if any)
  • Deploy config

Deployment 1 (can be regular deployment train)

Deployment 2 (backport & config deployment window)

  • Prepare, merge, backport & pull these patches onto deployment host:
    • Disable search widget takeover (in WikibaseMediaInfo) - Patch: 674015
    • Enable search widget takeover (in MediaSearch) - Patch: 674010
    • Rename Special:MediaSearch to Special:OldMediaSearch (in WikibaseMediaInfo) - Patch TODO
    • Rename Special:NewMediaSearch to Special:MediaSearch (in MediaSearch) - Patch TODO
  • Pull both of these WikibaseMediaInfo changes in staging server & check that nothing unexpected breaks
  • Pull both of these MediaSearch changes in staging server & test that everything works
  • Sync both of these WikibaseMediaInfo changes (Special:MediaSearch will not be accessible for a couple of minutes until next step; search widget requests will go to Special:Search)
  • Sync both of these MediaSearch patches (Special:MediaSearch is back, and so is the search widget takeover)
  • Verify all Special:MediaSearch functionality again

Deployment 3 (can be regular deployment train)

  • Remove all relevant code from WikibaseMediaInfo
  • Verify all Special:MediaSearch functionality again

Done!

Thanks @matthiasmullie! I think the only thing missing from that plan is the messages - seems like we should update the MediaSearch extension to depend on WBMI messages for now, then as part of deployment 3(?), we can go through the process outlined here to move WBMI messages to MediaSearch, rename them, and remove them from WBMI.

Change 676613 had a related patch set uploaded (by Anne Tomasevich; author: Anne Tomasevich):

[mediawiki/extensions/WikibaseMediaInfo@master] Rename Special:MediaSearch to Special:OldMediaSearch

https://gerrit.wikimedia.org/r/676613

Change 676614 had a related patch set uploaded (by Anne Tomasevich; author: Anne Tomasevich):

[mediawiki/extensions/MediaSearch@master] Rename Special:NewMediaSearch to Special:MediaSearch

https://gerrit.wikimedia.org/r/676614

Change 677024 had a related patch set uploaded (by Anne Tomasevich; author: Anne Tomasevich):

[mediawiki/extensions/MediaSearch@master] Apply diff up to 7aab874136853735c03ece84a91e89c73a0014dd

https://gerrit.wikimedia.org/r/677024

Change 677024 merged by jenkins-bot:

[mediawiki/extensions/MediaSearch@master] Apply diff up to 7aab874136853735c03ece84a91e89c73a0014dd

https://gerrit.wikimedia.org/r/677024

Change 680299 had a related patch set uploaded (by Matthias Mullie; author: Matthias Mullie):

[mediawiki/extensions/MediaSearch@master] Apply diff up to 9128a771d6ee76c3162df8f144ef33c515e712ee

https://gerrit.wikimedia.org/r/680299

Change 680299 merged by jenkins-bot:

[mediawiki/extensions/MediaSearch@master] Apply diff up to 9128a771d6ee76c3162df8f144ef33c515e712ee

https://gerrit.wikimedia.org/r/680299

Change 682101 had a related patch set uploaded (by Matthias Mullie; author: Matthias Mullie):

[mediawiki/tools/release@master] Add extensions/MediaSearch

https://gerrit.wikimedia.org/r/682101

Change 682102 had a related patch set uploaded (by Matthias Mullie; author: Matthias Mullie):

[operations/mediawiki-config@master] Enable Extension:MediaSeach on testcommons

https://gerrit.wikimedia.org/r/682102

Change 682105 had a related patch set uploaded (by Matthias Mullie; author: Matthias Mullie):

[operations/mediawiki-config@master] Enable Extension:MediaSearch on commons

https://gerrit.wikimedia.org/r/682105

Change 682101 merged by jenkins-bot:

[mediawiki/tools/release@master] Start branching MediaSearch extension for Wikimedia production

https://gerrit.wikimedia.org/r/682101

Change 685752 had a related patch set uploaded (by Matthias Mullie; author: Matthias Mullie):

[operations/mediawiki-config@master] Enable Extension:MediaSearch on betacommons

https://gerrit.wikimedia.org/r/685752

Change 685752 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable Extension:MediaSearch on betacommons

https://gerrit.wikimedia.org/r/685752

Mentioned in SAL (#wikimedia-operations) [2021-05-06T11:34:13Z] <mlitn@deploy1002> sync-file aborted: Config: [[gerrit:685752|Enable Extension:MediaSearch on betacommons (T265939)]] (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2021-05-06T11:35:22Z] <mlitn@deploy1002> Synchronized wmf-config: Config: [[gerrit:685752|Enable Extension:MediaSearch on betacommons (T265939)]] (duration: 01m 06s)