Page MenuHomePhabricator

Set up git replication on integration slaves
Closed, DeclinedPublic

Description

Figure out how to create and maintain replication on integration slaves. After T96629, we can do it from a scheduled job. But that is the "when", not the "how".

See also:
T96629: Convert pool from a few large slaves (4X) to more smaller slaves (1X)

Event Timeline

Krinkle created this task.Apr 21 2015, 2:40 PM
Krinkle raised the priority of this task from to Normal.
Krinkle updated the task description. (Show Details)
Krinkle added subscribers: Aklapper, greg, hashar, Krinkle.

That is similar to T87294: Nodepool images need Gerrit mirror for git-clone performance. Though with Nodepool that would be done on image creations and for our current permanent slave we would probably need some kind of weekly cron to refresh them.

From that task @Krinkle pointed to OpenStack script https://github.com/openstack-infra/project-config/blob/59e04c19850a067e57b6304ee398b20438b295a1/nodepool/scripts/cache_git_repos.py

That is similar to T87294: Nodepool images need Gerrit mirror for git-clone performance. Though with Nodepool that would be done on image creations and for our current permanent slave we would probably need some kind of weekly cron to refresh them.

Cron would interfere with builds and create races that corrupt git-clone. Per advice from people in #openstack-infra, we should use a Jenkins job scheduled to run on each instance. That way it will not run concurrently with other jobs.

From that task @Krinkle pointed to OpenStack script https://github.com/openstack-infra/project-config/blob/59e04c19850a067e57b6304ee398b20438b295a1/nodepool/scripts/cache_git_repos.py

That script is for creating the replication, it does not have logic for updating replication. We will need a list of commands to run for each repo. E.g.: git fetch origin. And maybe disable auto-gc, run gc after update, and maybe repack?

Change 206074 had a related patch set uploaded (by Krinkle):
[WIP] Implement git-cache-update script

https://gerrit.wikimedia.org/r/206074

hashar closed this task as Declined.Oct 6 2015, 12:29 PM
hashar claimed this task.

This will be done as part of migrating to Nodepool instances. The specific task is T87294: Nodepool images need Gerrit mirror for git-clone performance

Change 206074 abandoned by Krinkle:
Implement git-cache-update script

https://gerrit.wikimedia.org/r/206074