Automatically export translations from translatewiki.net
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Paladox
	Jun 21 2015, 9:36 AM

Description

Hi this is a feature suggestion for the translation bot to upload translations automatically so that it makes it less work for the translation team.

Outcome

Non-MediaWiki translations are automatically pushed to software repositories at scheduled times. Additional safeguards and monitoring was implemented to ensure exports are safe and that translation admins will notice if they are not working. Automation makes export schedule consistent and predictable and frees up translation admin time for other activities.

Details

	Subject	Repo	Branch	Lines +/-
	Add systemd timer for autoexport	translatewiki	master	+41 -0
	Add execute-and-notify	translatewiki	master	+42 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Nikerabbit	T103258 Automatically export translations from translatewiki.net
Resolved	abi_	T182433 Implement a stronger synchronization in RepoNG and Translate
Resolved	abi_	T273671 Track failed/timed-out messages updates and display them to the administrator
Resolved	abi_	T276197 Allow administrators to resolve message/group failures
Resolved	Nikerabbit	T182436 Secure the use of translatewiki.net key
Resolved	Nikerabbit	T279934 Centralized location of repositories for translation exports

Event Timeline

Paladox created this task.Jun 21 2015, 9:36 AM

Paladox raised the priority of this task from to Needs Triage.

Paladox updated the task description. (Show Details)

Paladox added a project: translatewiki.net.

Paladox added subscribers: Paladox, Nikerabbit, siebrand, Raymond.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 21 2015, 9:36 AM

There is already a bug for this, but it's impossible to find it in Phabricator. I'll try to find it when the bugzilla search is restored.

In T103258#1400663, @Nemo_bis wrote:

There is already a bug for this

Stuff that might be related: T90223: Improve translatewiki.net deployment system, T91707: L10n-bot should not force-merge / override Jenkins (breaks the build), T93308: Spot check on-wiki messages against prior day's translatewiki updates.

Aklapper renamed this task from [FEATURE SUGGESTION] Automatically upload translation daily without needing the translation team to do it to Automatically upload translation daily without needing the translation team to do it.Jun 25 2015, 3:19 PM

Aklapper set Security to None.

Anyway, this is a long term goal which is currently prototyped at http://intense.wmflabs.org/.

Nemo_bis renamed this task from Automatically upload translation daily without needing the translation team to do it to Automatically export translations daily without needing the TWN staff to do it.Jul 1 2015, 5:33 AM

Nikerabbit moved this task from Backlog to System admin stuff on the translatewiki.net board.Nov 21 2017, 2:23 PM

Nikerabbit added a subtask: T182433: Implement a stronger synchronization in RepoNG and Translate.Dec 8 2017, 2:59 PM

Nikerabbit added a subtask: T182436: Secure the use of translatewiki.net key.Dec 8 2017, 3:13 PM

Nikerabbit closed subtask T182436: Secure the use of translatewiki.net key as Resolved.Oct 19 2018, 12:17 PM

Nikerabbit mentioned this in T206694: Determine desired architecture to update localization strings for Wikimedia.Nov 12 2018, 3:49 PM

Basically, what is left is to implement T182433: Implement a stronger synchronization in RepoNG and Translate.

We recently had an issue that L10n-bot updated qqq.json by removing new messages added by a developer in an earlier patch. Our simple chronology protector takes care of that most of the time, but our JobQueue was stuck this time, so the assumption that it is safe to commit a patch on top of the current commit we have in read-only clones was broken. The stronger synchronization is about not using that assumption, but rather more explicit import state tracking.

Nikerabbit moved this task from System admin stuff to Features on the translatewiki.net board.May 8 2019, 12:47 PM

Pginer-WMF added projects: Language-Team (Language-2019-July-September), Epic.Jun 19 2019, 11:50 AM

Pginer-WMF removed a project: Language-Team (Language-2019-July-September).Dec 30 2019, 1:25 PM

Nikerabbit added a subtask: T279934: Centralized location of repositories for translation exports.Apr 12 2021, 3:05 PM

Nikerabbit closed subtask T279934: Centralized location of repositories for translation exports as Resolved.Jun 21 2021, 8:01 AM

We now have strong synchronization in place. Niklas and I had some discussion regarding what we can be done next to get to fully automated imports / exports.

We would like to have a IRC ping when exports / imports are run, and an email with the output of the command. systemd will be used to trigger the commands periodically and will write output to a log file. A wrapper script (possibly named exec-and-notify) that will trigger the command, then notify on IRC and email. If the command run by the wrapper script has a non-zero exit code, the email sent should have a subject prefix - FAILED:

When exporting mediawiki today:

raymond@web2:~$ autoexport-mediawiki
chmod: changing permissions of '/tmp/sync.lock': Operation not permitted

That is fixed now.

Nikerabbit claimed this task.Jun 29 2021, 7:19 AM

Nikerabbit added a project: Language-Team (Language-2021-April-June).

Change 702074 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[translatewiki@master] Add execute-and-notify

https://gerrit.wikimedia.org/r/702074

gerritbot added a project: Patch-For-Review.Jun 29 2021, 7:57 AM

Change 702077 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[translatewiki@master] Add systemd timer for autoexport

https://gerrit.wikimedia.org/r/702077

Nikerabbit raised the priority of this task from Low to Medium.Jun 29 2021, 10:51 AM

Nikerabbit moved this task from Quarter Backlog to In Review on the Language-Team (Language-2021-April-June) board.

Change 702074 merged by jenkins-bot:

[translatewiki@master] Add execute-and-notify

https://gerrit.wikimedia.org/r/702074

Change 702077 merged by jenkins-bot:

[translatewiki@master] Add systemd timer for autoexport

https://gerrit.wikimedia.org/r/702077

jenkins-bot mentioned this in rGTWNe99acb32ece5: Add systemd timer for autoexport.Jun 30 2021, 12:37 PM

jenkins-bot mentioned this in rGTWN47a4adeb490a: Add execute-and-notify.

Nikerabbit removed a project: Patch-For-Review.Jun 30 2021, 12:58 PM

Nikerabbit moved this task from In Review to Recheck after deployment on the Language-Team (Language-2021-April-June) board.

Nikerabbit renamed this task from Automatically export translations daily without needing the TWN staff to do it to Automatically export translations from translatewiki.net.Jul 12 2021, 7:09 AM

Nikerabbit closed this task as Resolved.

Nikerabbit updated the task description. (Show Details)

Nikerabbit moved this task from Recheck after deployment to Done on the Language-Team (Language-2021-April-June) board.

abi_ closed subtask T182433: Implement a stronger synchronization in RepoNG and Translate as Resolved.Jul 20 2021, 4:44 PM

Automatically export translations from translatewiki.netClosed, ResolvedPublicActions