# Initial requirement
This is a follow-up to T48833 where I implemented repository-state synchronization.
For fully automated exports, even stronger synchronization is required: we should make sure that source changes are processed in translatewiki before we use a particular revision.
See https://translatewiki.net/wiki/Repository_management#Repository_state_synchronization for detailed description of the issues we have observed.
# Current implementation plan
We're creating a //group synchronization cache// that tracks:
* incoming changes to messages,
* failed and passed message updates
Based on this we will display warnings to administrators who can then manually fix failed messages updates.
Eventually based on failure/failure resolution tracking we will stop import/exports of messages from Translatewiki.
# Things to do
- [x] Track incoming changes to messages from various groups: addition, modification and deletion
- [x] Identify incoming message changes that failed to properly update on the wiki
- [x] Track failed/timed-out messages updates and display them to the administrator
- [x] Allow administrators to mark failed/timed-out messages updates as "fixed".
- [x] Run `MessageIndexRebuild` job once there are no `MessageUpdate` jobs in the synchronization cache.
- [x] Warn translation administrators when they try to export message groups while they have:
* synchronization errors
* changes in review on Special:ManageMessageGroup
* synchronization is in progress
- [x] Halt imports for message groups that have:.
* synchronization errors
* synchronization is in progress
- [x] Do not allow administrators to process changes from `Special:ManageMessageGroups` incase of unresolved failures.
## Other minor to do
- [x] Timeout should be based on the number of messages to be processed for a group
- [x] Update translatewiki configuration to remove the `--skip-group-sync-check`
- [x] Logging when a group / message is marked as resolved by the administrator
# Pending decisions
- [x] 1. What should happen when a group export does not happen due to sync issues; administrator has to retry again after sometime? They will have to ensure that they check the export logs.
* **Decision**: Nothing special to do. It will be automatically re-attempted next time.
- [x] 2. What should happen if a group import does not happen due to sync issues; these are automatically run. Should we increase frequency of how often the import runs?
* **Decision**: We've decided to block imports and the current frequency of every two hours is enough. People are not looking into it even that often necessarily.
- [x] 3. Exports and imports should not run simultaneously. This will be outside the scope of the group synchronization cache but something may want, to achieve "strong synchronization".
* **Decision**: Yes, this needs to be done.
- [x] 4. Should exports be stopped if messages are waiting to be processed on `Special:ManageMessageGroups`? I would say that it should. We will have to check specifically for MediaiWiki / non-MediaiWiki exports.
* **Decision**: Yes, this needs to be done.
## Update log
1. **20-01-2021**: Changes for this patch caused a production error: {T272428}
2. **02-02-2021**: https://phabricator.wikimedia.org/T182433#6797412
3. **26-04-2021**: https://phabricator.wikimedia.org/T182433#7033993
4. **13-05-2021**: https://phabricator.wikimedia.org/T182433#7084380
5. **21-06-2021**: https://phabricator.wikimedia.org/T182433#7165680
6. **12-07-2021**: https://phabricator.wikimedia.org/T182433#7204410
## Patches
The list of Gerrit patches submitted for this task (including subtasks) can be found here: https://gerrit.wikimedia.org/r/q/topic:%22strong-sync%22+(status:open%20OR%20status:merged)