Page MenuHomePhabricator

scap3 should repack / pack-refs git repos under /srv/deployment
Closed, ResolvedPublic

Description

The git repositories on tin.eqiad.wmnet under /srv/deployment are never repacked or garbage collected. We could use a Trebuchet entry point to do the maintenance work or add a weekly cron to handle it for us.

On the Zuul merger git repositories, there is a daily cron to pack the references (less files under .git/refs/:

find /srv/deployment -maxdepth 3 -type d -name '.git' -exec git --git-dir='{}' pack-refs --all \;

Could probably add git repack -A -d which rebuild objects in a single pack file and put unreachable objects out of the pack as loose objects. They can then be removed with git gc --prune=all.

Revisions and Commits

Event Timeline

hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added subscribers: Legoktm, Krenair, Dzahn and 4 others.
hashar renamed this task from Trebuchet should repack / pack-refs git repos under /srv/deployment to scap3 should repack / pack-refs git repos under /srv/deployment.Oct 27 2015, 2:30 PM

s/Trebuchet/scap3/

The git repositories on tin.eqiad.wmnet under /srv/deployment are never repacked or garbage collected.

Yes they are, git does gc on repositories from time to time on its own.

I think it's just a matter of whether we think we need a more tightly packed repository. Wouldn't hurt I suppose, considering it gets a fair bit of traffic on a weekly basis.

dduvall triaged this task as High priority.Feb 12 2016, 7:33 PM
dduvall lowered the priority of this task from High to Medium.
dduvall moved this task from Needs triage to Debt on the Scap board.
dduvall moved this task from Needs triage to Debt on the Scap board.
hashar added a revision: Restricted Differential Revision.Dec 9 2016, 11:10 AM

Tyler has a patch for it D503 :}

Unlicking this cookie {D503} provides needed functions for this, and does some tag cleanup, but not the actual gc-ing.

We've got support in git.py for gc operations now. We could run this $every_so_often. Remember we'll need to handle this for sumodules, as repack and gc do not operate on submodules.

We could also tweak the git config for repos so they'll be more aggressive in how tightly they repack, prune refs, and how often they run. This buys us some automatic benefits.

greg subscribed.

Adding our Release-Engineering-Team (Kanban) project as we would like to work on this in the coming quarter or two (no promises though, this is not a "goal" only "other hoped for work").

mmodell subscribed.

this is pretty easy I will try to get this in before the end of quarter.

@mmodell will take a look at this again :)

After the last round of changes to the way submodules are handled, I don't think this is relevant any longer. We now do a shallow clone of submodules so there would be no reason to pack anything on the submodules or targets. We already do git gc on the deploy repo. Marking resolved rather than invalid since we did implement the spirit of this task even if not exactly the letter of it.