Page MenuHomePhabricator

Determine how to migrate messages from WBMI to new MediaSearch extension
Open, Needs TriagePublic

Description

The Structured Data team is planning to extract existing code for the Media Search feature out of WikibaseMediaInfo into a separate extension (see parent task). One aspect of this will be transferring messages from WBMI to the new MediaSearch extension. This will include removing Media Search-specific messages from files in the WBMI i18n directory and adding them to the MediaSearch extension, and removing the entire i18n/api directory and all its files from WBMI and moving them to MediaSearch.

We need to coordinate this work on 2 levels:

  1. Avoid either duplicating or missing messages in production during the switch from WBMI to MediaSearch
  2. Ensure we're not creating a negative experience on translatewiki.net

Event Timeline

Hi @Nikerabbit, as noted in the task description, we're planning on migrating a number of messages (and some entire message files) from the WikibaseMediaInfo extension into a new extension, MediaSearch. We want to avoid disrupting things on translatewiki.net and I was hoping to get your advice on making this transition as smooth as possible. My general plan is to merge a patch in WBMI removing the relevant messages and message files in tandem with the MediaSearch extension deployment, which will include the messages pulled from WBMI with different keys (swapping out the extension prefixes, mostly).

Is there anything we need to do differently to prevent duplicate work on translatewiki.net, or any other concerns? For example, if we merge a patch in MediaSearch with the messages from WBMI before MediaSearch is deployed, will those messages show up on translatewiki.net, or will that only happen post-deployment? I want to make sure we're getting the timing right.

My recommendation would be the following:

  1. (we) Synchronize with translatewiki.net to ensure everything is up to date
  2. Copy or move all message files to MediaSearch to preserve authorship and translations in languages under threshold (if copying, WBMI syncing should be disabled temporarily).
  3. Remove messages that you do not need in MediaSearch en.json(s)
  4. (we) Synchronize with translatewiki.net. This will drop unused keys from translation files.
  5. Rename message keys
  6. (we) Synchronize with translatewiki.net (we will perform mass rename on our own, which can take hours)
  7. Remove unused messages from WBMI (if not done earlier) and restore syncing
  8. (we) Synchronize with translatewiki.net
  9. You can deploy now

@Raymond Can you handle this or do you need help?

@Raymond Can you handle this or do you need help?

Should be doable by me.

@AnneT Please add me as reviewer for all these patches so that I can do the necessary actions on translatewiki.net

Thanks very much for the information, @Nikerabbit and @Raymond.

FYI, it seems unlikely that we will be pursuing this separate extension for the time being. If we ever do, or if we move messages like this in the future, we will certainly follow the processes outlined here.

Hey @Raymond, I wanted to let you know that the Structured Data team has decided to go ahead and move forward with this plan to remove the Special:MediaSearch code from the WikibaseMediaInfo extension and move it to a new extension, MediaSearch.

Right now my team is reviewing an initial patch to migrate the code. Once that patch is ready to merge, can I ping you to ask you to synch the WBMI messages with translatewiki, so I can include an up-to-date batch of messages in the MediaSearch patch? I'm hoping we'll be ready to do this within a week.

I'll add you as a reviewer on that patch—right now I've copied over and renamed the English messages just for the sake of being able to test the extension locally, but when we're ready to merge I'll copy the i18n files directly, remove unneeded messages, and wait before renaming the messages, as outlined in Nikerabbit's comment.

Actually @Raymond, scratch that: I'm realizing that since we can't remove the messages from WBMI immediately and don't want to turn off synching to WBMI for a long time, we're going to need to do all of this in tandem with deploying the MediaSearch extension, which won't happen for a little while.

Here's a question: could you turn off synching to MediaSearch instead? This would allow us to at least include the en.json and qqq.json files in the master branch so we could test locally without errors until we can actually deploy the new extension.

Reopening this since we are going to move forward with the new extension.

Hey @Raymond, I just wanted to check in on my previous comment—does it make sense to turn off synching for the new MediaSearch extension until we're closer to deploying it? I want to make sure all our ducks are in a row before we merge the initial patch that contains just the en.json and qqq.json files for testing purposes.

Hey @Raymond, I just wanted to check in on my previous comment—does it make sense to turn off synching for the new MediaSearch extension until we're closer to deploying it? I want to make sure all our ducks are in a row before we merge the initial patch that contains just the en.json and qqq.json files for testing purposes.

@AnneT Sorry, I missed your comment. But good news: The new MediaSearch extension was not and is not registered on translatewiki. Therefore nothing to do for now. Please give me a ping once I may register MediaSearch on translatewiki.net

Aha, perfect! Thanks, will do :)

Change 674609 had a related patch set uploaded (by Matthias Mullie; author: Matthias Mullie):

[mediawiki/extensions/MediaSearch@master] Build MediaSearch i18n messages from 2 sources

https://gerrit.wikimedia.org/r/674609

  1. Remove unused messages from WBMI (if not done earlier) and restore syncing
  2. ...
  3. You can deploy now

Above is a bit of a problem. We already have the exact same code live, in production (in Extension:WikibaseMediaInfo).
We can't remove the i18n files from Extension:WikibaseMediaInfo until after we've fully deployed Extension:MediaInfo (which will replace what is currently served from Extension:WikibaseMediaInfo)

AIUI, we'll end up in one of these scenarios:

  1. we first deploy Extension:MediaSearch, then sync the messages afterwards, but this means we'll have a brief moment with stale & very incomplete messages.
  2. we first sync the messages, then deploy Extension:MediaSearch, but this means Extension:WikibaseMediaInfo may no longer receive message updates in the meantime (this could be days, if we encounter issues with deployment)

I guess we could bite the bullet and live with stale messages for a few days if we have to, but I'd rather not.

I just submitted a patch that would leverage the old PHP-style message format to serve as a stopgap solution by reading message files from both extensions & merge them for a short time.
The idea is basically that:

  • Extension:MediaSearch registers ExtensionMessagesFiles instead of MessagesDirs
  • The messagesfile in Extension:MediaSearch reads both its own (if any) and Extension:WikibaseMediaInfo's json files (but changes wikibasemediainfo-special-mediasearch- prefixes to mediasearch-)
  • As long as Extension:WikibaseMediaInfo json continue to receive updates to the relevant messages, they'll continue to be used in Extension:MediaSearch
  • We can copy json files to Extension:MediaSearch whenever we want
  • Once json files have been copied, we remove them from Extension:MediaSearch - now the script starts showing the Extension:MediaSearch versions
  • And we can now remove the ExtensionMessagesFiles script and simply register MessagesDirs instead, which will continue to show thee same messages

I think we can still follow the process outlined by @Nikerabbit, but it decouples the i18n message copying from the rest of the deployment: we could deploy any time before, during or after the messages copy, since it'll fall over to the messages from the other extension.

However, does the above hacky workaround work with the translatewiki systems & processes? Other things I've overlooked?

@Raymond do you have thoughts on Matthias's comment above?

@Raymond do you have thoughts on Matthias's comment above?

Sounds good for me. I cannot test @matthiasmullie 's patch at the moment but this way it should works.

One suggestion: Commit a patch to the translatewiki.net repo whith enabling Extension:MediaSearch and disabling Extension:WBMI (by # comments, not complete removing at the moment) in the same patch. This way the switch can be done a bt faster

Change 682095 had a related patch set uploaded (by Matthias Mullie; author: Matthias Mullie):

[mediawiki/extensions/WikibaseMediaInfo@master] Use consistent prefix for mediasearch i18n messages

https://gerrit.wikimedia.org/r/682095

Change 682096 had a related patch set uploaded (by Matthias Mullie; author: Matthias Mullie):

[mediawiki/extensions/MediaSearch@master] Remove messages

https://gerrit.wikimedia.org/r/682096

We will soon start deploying Extension:MediaSearch without i18n messages, using the stopgap solution described above (T266345#6967371).
We expect it'll take ~3 weeks before everything is up & running and has overtaken the original extension (which will continue to be the place that the i18n messages live)
Once all of that has settled and the risk of having to revert anything has gone away, we'll continue the i18n messages migration as outlined by @Nikerabbit (T266345#6587909) & @Raymond (T266345#7001995)
Does that make sense?

Change 682095 merged by jenkins-bot:

[mediawiki/extensions/WikibaseMediaInfo@master] Use consistent prefix for mediasearch i18n messages

https://gerrit.wikimedia.org/r/682095

Change 674609 merged by jenkins-bot:

[mediawiki/extensions/MediaSearch@master] Build MediaSearch i18n messages from 2 sources

https://gerrit.wikimedia.org/r/674609

Change 682096 merged by jenkins-bot:

[mediawiki/extensions/MediaSearch@master] Remove messages

https://gerrit.wikimedia.org/r/682096