Zuul-merger creates a bunch of references which ends up slowing down git fetch (T70481). We need to run the reference cleaner on all repository on a daily/weekly basis.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | hashar | T70480 Scale merge operations (tracking) | |||
| Resolved | hashar | T70481 Zuul repositories have too many refs causing slow updates | |||
| Declined | None | T103528 Run zuul-clear-refs.py daily on all our repositories to reclaim Zuul references | |||
| Resolved | hashar | T103529 Package / puppetize zuul-clear-refs.py | |||
| Resolved | elukey | T145057 Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger) | |||
| Resolved | hashar | T147073 Upgrade Zuul on scandium to 2.5.0-8-gcbc7f62-wmf3jessie1 |
Event Timeline
Comment Actions
So we have the packages on apt.wikimedia.org with the zuul-clear-refs.py script.
What I have noticed though is that when it garbage collect old references, the zuul-merger fails to merge patch for the repo with a stacktrace related to iterating refs.
I think there is a race condition where zuul-merger iter the references, clear-refs delete one, and the iterator dies horribly because the reference is no more present.
So we can not quite auto run it or at a very non busy / odd hour to minimize races.
Comment Actions
Example stacktrace:
2016-09-15 11:20:36,855 ERROR zuul.Merger: Unable to reset repo <zuul.merger.merger.Repo object at 0x7fe0315e2c90>
Traceback (most recent call last):
File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 322, in _mergeItem
repo.reset()
File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 104, in reset
for ref in origin.refs:
File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/remote.py", line 513, in refs
out_refs.extend(RemoteReference.list_items(self.repo, remote=self.name))
File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/util.py", line 745, in list_items
out_list.extend(cls.iter_items(repo, *args, **kwargs))
File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/refs/symbolic.py", line 594, in _iter_items
for sha, rela_path in cls._iter_packed_refs(repo):
ValueError: need more than 1 value to unpackComment Actions
Gave the package a try but the shebang for zuul-clear-refs.py is not rewritten :(
The /usr/bin/zuul* are symlinks to /usr/share/python/zuul/bin/zuul* EXCEPT for zuul-clear-ref which is not even installed in the venv :(
The symlinking is done via debian/links.
Comment Actions
We can now run /usr/bin/zuul-clear-ref. But since it is playing with the ref, it is safer to have the zuul-merger service stopped while running the script :( Else there are random merge failure occuring due to lacking refs :(