|Resolved||• hashar||T70480 Scale merge operations (tracking)|
|Resolved||• hashar||T70481 Zuul repositories have too many refs causing slow updates|
|Open||None||T103528 Run zuul-clear-refs.py daily on all our repositories to reclaim Zuul references|
|Resolved||• hashar||T103529 Package / puppetize zuul-clear-refs.py|
|Resolved||elukey||T145057 Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger)|
|Resolved||• hashar||T147073 Upgrade Zuul on scandium to 2.5.0-8-gcbc7f62-wmf3jessie1|
So we have the packages on apt.wikimedia.org with the zuul-clear-refs.py script.
What I have noticed though is that when it garbage collect old references, the zuul-merger fails to merge patch for the repo with a stacktrace related to iterating refs.
I think there is a race condition where zuul-merger iter the references, clear-refs delete one, and the iterator dies horribly because the reference is no more present.
So we can not quite auto run it or at a very non busy / odd hour to minimize races.
2016-09-15 11:20:36,855 ERROR zuul.Merger: Unable to reset repo <zuul.merger.merger.Repo object at 0x7fe0315e2c90> Traceback (most recent call last): File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 322, in _mergeItem repo.reset() File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 104, in reset for ref in origin.refs: File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/remote.py", line 513, in refs out_refs.extend(RemoteReference.list_items(self.repo, remote=self.name)) File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/util.py", line 745, in list_items out_list.extend(cls.iter_items(repo, *args, **kwargs)) File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/refs/symbolic.py", line 594, in _iter_items for sha, rela_path in cls._iter_packed_refs(repo): ValueError: need more than 1 value to unpack
Gave the package a try but the shebang for zuul-clear-refs.py is not rewritten :(
The /usr/bin/zuul* are symlinks to /usr/share/python/zuul/bin/zuul* EXCEPT for zuul-clear-ref which is not even installed in the venv :(
The symlinking is done via debian/links.
We can now run /usr/bin/zuul-clear-ref. But since it is playing with the ref, it is safer to have the zuul-merger service stopped while running the script :( Else there are random merge failure occuring due to lacking refs :(