This is a follow-up to T48833 where I implemented repository-state synchronization.
For fully automated exports, even stronger synchronization is required: we should make sure that source changes are processed in translatewiki before we use a particular revision.
See https://translatewiki.net/wiki/Repository_management#Repository_state_synchronization for detailed description of the issues we have observed.
Current implementation plan
We're creating a group synchronization cache that tracks:
- incoming changes to messages,
- failed and passed message updates
Based on this we will display warnings to administrators who can then manually fix failed messages updates.
Eventually based on failure/failure resolution tracking we will stop import/exports of messages from Translatewiki.
Things to do
- Track incoming changes to messages from various groups: addition, modification and deletion
- Identify incoming message changes that failed to properly update on the wiki
- Track failed/timed-out messages updates and display them to the administrator
- Allow administrators to mark failures as "fixed".
- Run MessageIndexRebuild job once there are no MessageUpdate Jobs in the synchronization cache.
- Warn translation administrators when they try to export translations while there are errors in groups.
- Halt imports for message groups that have unresolved failures.
- Do not allow administrators to process changes from Special:ManageMessageGroups incase of unresolved failures.
Other minor to do
- Timeout should be based on the number of messages to be processed for a group
- Update translatewiki configuration to pass the new flags to check group sync cache for import and exports
- Logging when a group / message is marked as resolved by the administrator
- What should happen when a group export does not happen due to sync issues; administrator has to retry again after sometime? They will have to ensure that they check the export logs.
- What should happen if a group import does not happen due to sync issues; these are automatically run. Should we increase frequency of how often the import runs?
- Exports and imports should not run simultaneously. This will be outside the scope of the group synchronization cache but something may want, to achieve "strong synchronization".
- Should exports be stopped if messages are waiting to be processed on Special:ManageMessageGroups? I would say that it should. We will have to check specifically for MediaiWiki / non-MediaiWiki exports.
- 20-01-2021: Changes for this patch caused a production error: T272428: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist
- 02-02-2021: https://phabricator.wikimedia.org/T182433#6797412
The list of Gerrit patches submitted for this task (including subtasks) can be found here: https://gerrit.wikimedia.org/r/q/topic:%22strong-sync%22+(status:open%20OR%20status:merged)