This can likely be split into multiple phases:

• Algorithm that finds pure renames (100% match and 1:1 mapping)
• Algorithm that finds fuzzy renames (less than 100% with 1:1 mapping)
• User interface to "break" automatically detect renames that is wrong
• User interface to connect renames that were not detected [optional]
• Backend code to perform the renames

Hi @Pginer-WMF

Need your inputs on this task. To summarize what we are doing here,

We use the Special:ManageMessageGroups page to merge incoming message changes from the source repositories projects into Translatewiki.net. Sometimes these message changes might include the message key itself being changed but not the translation. In such cases we have to rename the key on Translatewiki.net in order to avoid translators having to re-translate the same message again.

See an image below of how the UI looks currently,

Currently this process is manual, but this task deals with making some of it automatic based on if the translation content of a renamed message. If the content of a new message matches the content of a message that is no longer present on the source repo, but is present in Translatewiki.net that means we've to perform a rename.

The following document (https://docs.google.com/document/d/1xmjiuo53gZyuPjPCBuE-3jqb_yN751Mjd-I-Xa6Yr9U/edit?usp=sharing) summarizes the changes we're planning to make. The UI changes are listed under sections,

1. UI Update for displaying renames and allowing “breaks”
2. UI Update to connect missed renames

I'd love for you to review the changes suggested, and if possible provide icons for break and unbreak buttons.

We've reduced the scope of this task and broken out Adding support for fuzzy renames to T222566: Automatically detect message key renames with minor content changes in Special:ManageMessageGroups

In T196601#5158867, @abi_ wrote:

Hi @Pginer-WMF

Need your inputs on this task. To summarize what we are doing here,

I think it may help to expose less and more specific actions to the user. With the current proposal, it may not be obvious the effects of the broken link icon. Below I was illustrating some ideas that may help with this:

• Show an indicator spelling out that the message has been renamed (or at least the system interprets it this way).
• Next to the "remaned" message, provide a menu access allowing users to modify/correct such decision. There users can make a specific choice on whether the message is a new independent message or a rename of a different one. If we expect this list to grow very long, we can consider a different interaction element than a plain list of options.
• Make the "Ignore" action more explicit about the new message being discarded.

This is just an initial set of ideas put together to get some more feedback on the general direction. If this looks promising, I can iterate with more details.

I think this UI looks much more intuitive. Few points,

If we expect this list to grow very long, we can consider a different interaction element than a plain list of options.

This list can grow long, so we might have to use a different interaction element. I initially thought of using a dialog to list down the messages.

So then the items under the dropdown menu might then be updated to show,

Add as a new message
-----
Add as a rename for:
message-a1   (edit icon)   (selected)

The edit icon would pop out the dialog box.

Slightly off topic,

1. Add as new message
2. Add as rename for:

Would still cause a postback, and refresh the page.

@Nikerabbit - Thoughts? I haven't seen dialog boxes being used much on Translatewiki.net or the Translate extension.

Dialog boxes (especially modal ones) are often discouraged. For example English Wikipedia says:

A modal dialog interrupts the main workflow. This effect has either been sought by the developer because it focuses on the completion of the task at hand or rejected because it prevents the user from changing to a different task when needed.

In this case user is looking to complete a task (fix up rename). The options to me seem to be either provide a control in-place, or a dialog box. Either seems okay from workflow point.

But if step backward a bit and consider how the user learns about the fact that the automatically generated renames are not correct. I see two cases:

• The user is explicitly informed about renames (e.g. via email) from developers. The developer might have listed the exact renames or the user may check the diffs to see what has actually been renamed.
• The user looks at the processing page, scrolls up and down to see what has been changed and notices there is a more appropriate match.
• The user looks at the proposed rename, and just straight away deems it incorrect. The user may want to inspect other suggestions or just break it up. This can happen with fuzzy matching.

This brings up the following thoughts to me that I didn't think of earlier:

• Input or search by key should be possible.
• The menu does not provide an obvious ability to preview the values of different keys[1]. A dialog box would have space to show diffs before selection as well.
• The menu can list the best matches sorted by similarity in ascending order and maybe show similarity-% as well. This could be limited to top N.
• In the mock the message keys are very short. In practice they are usually much longer. Mostly this would mean that the menu is wider.

[1] Before that the only option would be to update the diff area on hover, but the menu itself hides that when expanded.

In summary, I believe that an interface that allows switching between even the TOP 3 keys based on most similar content, would handle 99% of cases if we have fuzzy matching support. Without fuzzy matching support, it's likely that additional renames need to be input by hand since we can't rank suggestions. . Top 3 combined with an option to select alternative item through search feels sufficient to me, without need for dialogs.

My intention with the dialog box was to have something similar to what Phab has when editing related tasks

The dialog box will have all the newly added keys in the group along with their content displayed with a feature to search through them. It is important for the user to see the content of the message to be able to determine renames. Even with Fuzzy rename support, selecting from the top 3 might require looking at the content which is not possible with the dropdown. If we can come up with another approach that does not need a dialog box, but still displays the content, that should be OK.

Ok. Based on the comments above, I explored a process where the user is provided two options (add as new or as a rename). The later allows to select by searching and surfaces the items with better matches (and the current selected message highlighted). I tried to explore if this can be supported in-place, which allows the user to navigate through the page to collect necessary information:

Alternatively, the rename process can be made into a detached dialog.

Does this make sense? Do you see any step where the user may lack the information needed to make the decision?

Does this make sense? Do you see any step where the user may lack the information needed to make the decision?

Yes, makes sense and all the necessary information is there.

We have,

1. the message key
2. the content
3. similarity percentage

@Pginer-WMF - I think we should also add a confirmation button as a step that needs to be carried out after selecting the matching message as renaming will cause a page refresh.

@Nikerabbit - Which approach out of the two do you prefer? Personally, the dialog box fits better, and provides more space.

Both alternatives are good.

What kind of code changes would be required to avoid page refresh?

We will have to,

1. Make an AJAX call to update the CDB file which is simple
2. Also update the DOM to reflect the updated rename or adding the message as completely new.
3. Determine the post limit, and if are we crossing it.

The second and third points are difficult.

We can send the HTML of the modified messages from the server to the client, but will still require a lot of JavaScript to update the DOM appropriately to reflect the contents of the CDB file.

Regarding the post limit, I'm not sure how we will determine whether we've exceeded it. We could keep the currently limit in the DOM and then send it to the server while making the AJAX request. There will be other corner-cases such as a broken message not appearing due to the limit being crossed.

One possible and relatively easy approach would be to regenerate and replace the DOM related to the entire group where the rename happened but will have to take additional care to ensure that the previously selected radio options are re-selected.

• Renames are determined based on content within a language in a group. If a message key has been renamed, the old key in the wiki will appear as deleted, and the new incoming key will appear as a new message. We determine renames by comparing the content of the messages deleted from the wiki, and new incoming messages. If the content is the same, we consider them a rename.
• In a group, if the source language key is renamed, then all the other language message keys will also be renamed.
• It is not possible to ignore renames, but the matched renamed messages can be broken and added as a new messages.
• If the content of the incoming new key, and the deleted wiki key is not similar, the system will not be able to recognize them as renames. In such cases, it is possible to manually mark a new message as a rename of an old message using the Rename dialog.
• A check has been added to ensure that two users cannot update the changes at the same time. For eg: User A and B load the ManageMessageGroup page. User B breaks a rename, but user A is still looking at an old screen that does not reflect this updated change. When A submits the ManageMessageGroup page, the system will warn them that they have outdated changes and they should reload the page to see the latest changes.

During testing, the following scenarios were identified and tested,

1. A message is renamed, with only source key changed.
2. A message is renamed, with source and all the translation keys changed as well.
3. A message is renamed, with source language message key changed, but a translation message having a content change with the old key.
4. A message is renamed, with source language message key changed, and a translation message having a content change with newly renamed message key.
5. A message is renamed, with source language message key changed and the source language also having a content change.

Following major parts of the code base have been changed,

• ExternalMessageSourceStateComparator - Checks for changes during the imports
• Special:ManageGroups - To display the renames and handles the submission of user input. Renames are displayed in a dialog box.
• MessageUpdateJob - Background job that processes and performs the necessary renames
• ApiManageMessageGroups & ApiQueryManageMessageGroups - Provides functionality for the purpose of performing manual renames and breaking renames on the Special:ManageGroups.
• MessageSourceChange - To track all the changes for a MessageGroup, and provide an interface for easy querying

Screenshot

Question: Should we specify somewhere that this is the similarity? Might be difficult for users to understand what it is.

Change 515617 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/Translate@master] Add rename and auto-match functionality for imports

https://gerrit.wikimedia.org/r/515617

Change 519634 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/Translate@master] Add rename and auto-match functionality for imports

https://gerrit.wikimedia.org/r/519634

Change 519649 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/Translate@master] Display identified renames, and allow manual modification

https://gerrit.wikimedia.org/r/519649

Change 515617 abandoned by Abijeet Patro:
Add rename and auto-match functionality for imports

Reason:
Broken into smaller patches - I0596559c82aad3a8ff317a55bad74cbf665ae865, Ied5565ae5c259d64367114ce6326fd9508c5fe76, I2a39c8eada6e734d122451419f7879af88aa023d

https://gerrit.wikimedia.org/r/515617

Change 519651 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/Translate@master] Process renames as per user submissions

https://gerrit.wikimedia.org/r/519651

After this task is done, we need to Inform project maintainers and community by adding a news item under Translatewiki.net. See {T222577: Setup and document process for large key renames}

Change 519634 merged by jenkins-bot:
[mediawiki/extensions/Translate@master] Process incoming changes to identify potential renames

https://gerrit.wikimedia.org/r/519634

Change 519651 merged by jenkins-bot:
[mediawiki/extensions/Translate@master] Process renames as per user submissions

https://gerrit.wikimedia.org/r/519651

Change 519649 merged by jenkins-bot:
[mediawiki/extensions/Translate@master] Display identified renames, and allow manual modification

https://gerrit.wikimedia.org/r/519649

Updated documentation on https://translatewiki.net/wiki/Repository_management#How_to_use_Special:ManageMessageGroups to describe this functionality.

Next thing to do,

After this task is done, we need to Inform project maintainers and community by adding a news item under Translatewiki.net. See {T222577: Setup and document process for large key renames}

This patch has been deployed on Translatewiki.net and we're (Raymond, Niklas and I) are currently using this new functionality. I will be resolving this task once we've used it for a while and are confident that it works as intended.

Raymond said it worked well for some simple renames. No complaints from me either.