Maniphest T279934

Centralized location of repositories for translation exports
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Nikerabbit
	Apr 12 2021, 3:05 PM

Tags

Referenced Files

None

Subscribers

Description

Currently each translation admin has their own clone of repositories. This has multiple drawbacks:

Manual setup procedure
Risk of having outdated repoconfig.yaml
Increased disk space usage (one full set of repositories currently takes 16G)
Increased disk IOps usage
Increased file system cache churn

All of this could be solved by using a single, central location. There a few considerations to take into account:

File system write permissions [can use same logic as for read-only clones]
Locking to prevent concurrent access [can be postponed until fully automated exports are implemented]

Status

This change has been deployed and verified
Locking is being implemented separately
Cleanup of now unused export directories has started

Outcome

We now use shared set of repositories for doing translation exports, instead of each committer having to have separate clones. New committers no longer need to do any manual setup and potential errors with that are removed. Server resource use has improved.

Details

	Project	Branch	Lines +/-	Subject
	translatewiki	master	+135 -24	Central exports directory

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Nikerabbit	T103258 Automatically export translations from translatewiki.net
		Resolved		Nikerabbit	T279934 Centralized location of repositories for translation exports

Event Timeline

Nikerabbit created this task.Apr 12 2021, 3:05 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 12 2021, 3:05 PM

Nikerabbit added a parent task: T103258: Automatically export translations from translatewiki.net.Apr 12 2021, 3:05 PM

Change 681702 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[translatewiki@master] Central exports directory

https://gerrit.wikimedia.org/r/681702

gerritbot added a project: Patch-For-Review.Apr 26 2021, 1:11 PM

Nikerabbit claimed this task.Apr 26 2021, 1:11 PM

Nikerabbit added a project: Language-Team (Language-2021-April-June).

Nikerabbit moved this task from Backlog to Processes on the translatewiki.net board.

Nikerabbit moved this task from Quarter Backlog to In Review on the Language-Team (Language-2021-April-June) board.

Nikerabbit triaged this task as Medium priority.Apr 28 2021, 11:43 AM

Change 681702 merged by jenkins-bot:

[translatewiki@master] Central exports directory

https://gerrit.wikimedia.org/r/681702

jenkins-bot mentioned this in rGTWNbd1b60cae2ae: Central exports directory.Jun 14 2021, 7:46 AM

Maintenance_bot removed a project: Patch-For-Review.Jun 14 2021, 8:10 AM

Tested with non-MediaWiki exports. Waiting to see if any issues come up with MediaWiki exports.

Nikerabbit closed this task as Resolved.Jun 21 2021, 8:01 AM

Nikerabbit moved this task from Recheck after deployment to Done on the Language-Team (Language-2021-April-June) board.

Nikerabbit updated the task description. (Show Details)Jun 21 2021, 8:29 AM

Nikerabbit mentioned this in T257205: Track translatewiki.net translation synchronization incidents.Jun 24 2021, 7:43 AM

abi_ mentioned this in T228034: Use SVN_SSH to specify known_hosts during imports/exports.Jul 12 2021, 3:13 PM