Page MenuHomePhabricator

Package / puppetize zuul-clear-refs.py
Closed, ResolvedPublic

Description

We need to puppetize or package the zuul-clear-refs.py script https://review.openstack.org/#/c/109276/

Event Timeline

hashar created this task.Jun 23 2015, 2:16 PM
hashar claimed this task.
hashar raised the priority of this task from to Medium.
hashar updated the task description. (Show Details)
hashar added subscribers: hashar, Krinkle, greg and 2 others.
hashar renamed this task from Package / puppets zuul-clear-refs.py to Package / puppetize zuul-clear-refs.py.Jun 25 2015, 10:56 AM
hashar removed hashar as the assignee of this task.
hashar set Security to None.
hashar claimed this task.Jun 30 2015, 1:51 PM
hashar removed hashar as the assignee of this task.Jul 22 2015, 1:02 PM
hashar claimed this task.Jul 27 2015, 2:40 PM
hashar lowered the priority of this task from Medium to Low.

Still being reviewed upstream at https://review.openstack.org/#/c/109276/

hashar removed hashar as the assignee of this task.Oct 8 2015, 8:25 PM
hashar moved this task from In-progress to Backlog on the Continuous-Integration-Infrastructure board.

The patch I have proposed upstream has been approved :-} https://review.openstack.org/#/c/109276/

http://docs.openstack.org/infra/zuul/merger.html#clearing-old-references

The references created under refs/zuul are not garbage collected. Since git fetch send them all to Gerrit to sync the repositories, the time spent on merge will slightly grow overtime and start being noticeable.

To clean them you can use the tools/zuul-clear-refs.py script on each repositories. It will delete Zuul references that point to commits for which the commit date is older than a given amount of days (default 360):

We would want to include the utility in the Zuul Debian package.

"Just" have to figure out a way to have the debian package to copy /tools/zuul-clear-refs.py to /usr/bin/zuul-clear-refs. Should be fairly good first task.

hashar claimed this task.Aug 29 2016, 10:08 AM

I have included the script in our Debian package for Precise which ship it as /usr/bin/zuul-clear-ref. That is meant to be used on zuul-merger instance, the one we have is scandium.eqiad.wmnet which is running Jessie, so this patch is now pending for a port of the Zuul package to jessie.

Krinkle removed a subscriber: Krinkle.Aug 30 2016, 1:10 AM
Krenair added a subscriber: Krenair.Sep 7 2016, 3:53 PM

I have build a new Jessie package and it is going to be upgraded on scandium (zuul merger) via T145057

2.5.0-8-gcbc7f62-wmf2jessie1 has been deployed on scandium.eqiad.wmnet (the zuul-merger) but I screwed it up. The python shebang needs to be adjusted to point to the venv:

$ zuul-clear-refs
Traceback (most recent call last):
  File "/usr/bin/zuul-clear-refs", line 30, in <module>
    import git
ImportError: No module named git
$ head -n 1 `which zuul-cloner`
#!/usr/share/python/zuul/bin/python

The custom patch debian/patches/0002-Replace-python-shebang-with-python2.7.patch from the branch patch-queue/debian/precise-wikimedia needs to be rebased.

Change 310277 had a related patch set uploaded (by Hashar):
New release 2.5.0-8-gcbc7f62-wmf3precise1

https://gerrit.wikimedia.org/r/310277

Change 310277 merged by Hashar:
New release 2.5.0-8-gcbc7f62-wmf3precise1

https://gerrit.wikimedia.org/r/310277

Change 310285 had a related patch set uploaded (by Hashar):
New release 2.5.0-8-gcbc7f62-wmf3jessie1

https://gerrit.wikimedia.org/r/310285

Change 310285 merged by Hashar:
New release 2.5.0-8-gcbc7f62-wmf3jessie1

https://gerrit.wikimedia.org/r/310285

I have rebased the shebang patch. Rebuild package for both Precise and Jessie:

https://people.wikimedia.org/~hashar/debs/zuul_2.5.0-8-gcbc7f62-wmf3precise1/
https://people.wikimedia.org/~hashar/debs/zuul_2.5.0-8-gcbc7f62-wmf3jessie1/

debdiff shows they are noop

comparing the package contents with previous version shows that three files get their shebang updated.

Will get them pushed to the zuul systems and apt.wikimedia.org

akosiaris closed this task as Resolved.Sep 15 2016, 4:51 PM

Packages have been uploaded to apt.wikimedia.org \O/

hashar reopened this task as Open.Oct 11 2016, 3:22 PM

Does not work. The script is not properly installed and the shebang is not adjusted to point to the venv.

Change 315281 had a related patch set uploaded (by Hashar):
WMF: install zuul-clear-refs

https://gerrit.wikimedia.org/r/315281

Change 315281 merged by Hashar:
WMF: install zuul-clear-refs

https://gerrit.wikimedia.org/r/315281

Change 315290 had a related patch set uploaded (by Hashar):
Fix zuul-clear-refs install and shebang

https://gerrit.wikimedia.org/r/315290

Change 315290 merged by Hashar:
Fix zuul-clear-refs install and shebang

https://gerrit.wikimedia.org/r/315290

Change 315352 had a related patch set uploaded (by Hashar):
2.5.0-8-gcbc7f62-wmf4precise1

https://gerrit.wikimedia.org/r/315352

Change 315352 merged by Hashar:
2.5.0-8-gcbc7f62-wmf4precise1

https://gerrit.wikimedia.org/r/315352

Change 315430 had a related patch set uploaded (by Hashar):
2.5.0-8-gcbc7f62-wmf4jessie1

https://gerrit.wikimedia.org/r/315430

Change 315430 merged by Hashar:
2.5.0-8-gcbc7f62-wmf4jessie1

https://gerrit.wikimedia.org/r/315430

zuul_2.5.0-8-gcbc7f62-wmf4* fix it:

$ which zuul-clear-refs
/usr/bin/zuul-clear-refs
$ zuul-clear-refs
usage: zuul-clear-refs [-h] [--until DAYS_AGO] [-n] [-v] gitrepo
zuul-clear-refs: error: too few arguments

hashar closed this task as Resolved.Oct 12 2016, 2:34 PM
hashar added a subscriber: elukey.

scandium:~$ zuul-clear-refs
usage: zuul-clear-refs [-h] [--until DAYS_AGO] [-n] [-v] gitrepo
zuul-clear-refs: error: too few arguments

FINALLY! Huge thanks to @elukey for all the package upgrades :}