Page MenuHomePhabricator

Run zuul-clear-refs.py daily on all our repositories to reclaim Zuul references
Open, MediumPublic

Description

Zuul-merger creates a bunch of references which ends up slowing down git fetch (T70481). We need to run the reference cleaner on all repository on a daily/weekly basis.

Event Timeline

hashar created this task.Jun 23 2015, 2:15 PM
hashar claimed this task.
hashar raised the priority of this task from to Medium.
hashar updated the task description. (Show Details)
hashar added subscribers: JanZerebecki, greg, Krinkle, hashar.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 23 2015, 2:15 PM
hashar removed hashar as the assignee of this task.Jul 22 2015, 1:02 PM

So we have the packages on apt.wikimedia.org with the zuul-clear-refs.py script.

What I have noticed though is that when it garbage collect old references, the zuul-merger fails to merge patch for the repo with a stacktrace related to iterating refs.

I think there is a race condition where zuul-merger iter the references, clear-refs delete one, and the iterator dies horribly because the reference is no more present.

So we can not quite auto run it or at a very non busy / odd hour to minimize races.

Example stacktrace:

2016-09-15 11:20:36,855 ERROR zuul.Merger: Unable to reset repo <zuul.merger.merger.Repo object at 0x7fe0315e2c90>
Traceback (most recent call last):
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 322, in _mergeItem
    repo.reset()
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 104, in reset
    for ref in origin.refs:
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/remote.py", line 513, in refs
    out_refs.extend(RemoteReference.list_items(self.repo, remote=self.name))
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/util.py", line 745, in list_items
    out_list.extend(cls.iter_items(repo, *args, **kwargs))
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/refs/symbolic.py", line 594, in _iter_items
    for sha, rela_path in cls._iter_packed_refs(repo):
ValueError: need more than 1 value to unpack
Krinkle removed a subscriber: Krinkle.Sep 15 2016, 7:47 PM

Gave the package a try but the shebang for zuul-clear-refs.py is not rewritten :(

The /usr/bin/zuul* are symlinks to /usr/share/python/zuul/bin/zuul* EXCEPT for zuul-clear-ref which is not even installed in the venv :(

The symlinking is done via debian/links.

We can now run /usr/bin/zuul-clear-ref. But since it is playing with the ref, it is safer to have the zuul-merger service stopped while running the script :( Else there are random merge failure occuring due to lacking refs :(