Page MenuHomePhabricator

Run daily on all our repositories to reclaim Zuul references
Closed, DeclinedPublic


Zuul-merger creates a bunch of references which ends up slowing down git fetch (T70481). We need to run the reference cleaner on all repository on a daily/weekly basis.

Event Timeline

hashar claimed this task.
hashar raised the priority of this task from to Medium.
hashar updated the task description. (Show Details)
hashar added subscribers: JanZerebecki, greg, Krinkle, hashar.

So we have the packages on with the script.

What I have noticed though is that when it garbage collect old references, the zuul-merger fails to merge patch for the repo with a stacktrace related to iterating refs.

I think there is a race condition where zuul-merger iter the references, clear-refs delete one, and the iterator dies horribly because the reference is no more present.

So we can not quite auto run it or at a very non busy / odd hour to minimize races.

Example stacktrace:

2016-09-15 11:20:36,855 ERROR zuul.Merger: Unable to reset repo <zuul.merger.merger.Repo object at 0x7fe0315e2c90>
Traceback (most recent call last):
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/merger/", line 322, in _mergeItem
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/merger/", line 104, in reset
    for ref in origin.refs:
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/", line 513, in refs
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/", line 745, in list_items
    out_list.extend(cls.iter_items(repo, *args, **kwargs))
  File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/refs/", line 594, in _iter_items
    for sha, rela_path in cls._iter_packed_refs(repo):
ValueError: need more than 1 value to unpack

Gave the package a try but the shebang for is not rewritten :(

The /usr/bin/zuul* are symlinks to /usr/share/python/zuul/bin/zuul* EXCEPT for zuul-clear-ref which is not even installed in the venv :(

The symlinking is done via debian/links.

We can now run /usr/bin/zuul-clear-ref. But since it is playing with the ref, it is safer to have the zuul-merger service stopped while running the script :( Else there are random merge failure occuring due to lacking refs :(