Page MenuHomePhabricator

Setup a local repo for toollabs that supports separate trusty and precise packages
Closed, ResolvedPublic

Description

Currently they all go into one pot, and I do not know what happens if a trusty only package tries to be installed on precise. Nothing good, probably.

Event Timeline

yuvipanda raised the priority of this task from to Needs Triage.
yuvipanda updated the task description. (Show Details)
yuvipanda added a project: Toolforge.
yuvipanda changed Security from none to None.
yuvipanda added subscribers: yuvipanda, coren.

You're right, nothing good. I've just run into a wall where Ubuntu's python-matplotlib on Trusty requires libtcl8.6. On Precise it doesn't, so our home-baked tcl 8.6 doesn't provide it, but Trusty's tcl 8.6 does. If we were to use Ubuntu's tcl 8.5 on Precise, everything would be peachy, but there was adamant user demand for 8.6 on Precise. *argl*

So we need to have one repository for Precise and one for Trusty as a stopgap, or change our priority specification for our local repository in the way that it is no longer preferred always, but only if it provides the latest version of a package. Need to think about that.

Also, we should make an effort to reduce the size of our local repository (to zero :-)). I. e., every package there should probably be pushed upstream, for review, maintenance & Co. In the past we usually just used py2dsc or something similar to hammer something into "good enough". Debian packagers are pretty picky, so a) this would only increase the quality of the packages and their durability and b) it's rather easy to outsource to volunteers who don't want to plug holes under the water line for one or two users, but are interested in creating something lasting that is widely used and offers prestige.

scfc triaged this task as High priority.
scfc moved this task from Backlog to In Progress on the Toolforge board.

The WMF repositories (wikimedia-precise & Co.) have a priority of 1001. So if the priority of our local repository was set in a way that it would just supplement the Ubuntu repositories for missing/obsolete packages, we couldn't override packages in the WMF repositories locally. Ergo splitting the repository is the way to go.

In this case, I won't automatically move the packages from /data/project/.system/deb to /data/project/.system/deb-precise, but do it manually after merging :-). I would fill /data/project/.system/deb-trusty (and /data/project/.system/deb-jessie) only with jobutils and misctools for starters and copy additional packages there only if it is necessary for the OS release.

Change 195079 had a related patch set uploaded (by Tim Landscheidt):
Tools: Split local apt repository by OS release

https://gerrit.wikimedia.org/r/195079

Change 195079 merged by Yuvipanda:
Tools: Split local apt repository by OS release

https://gerrit.wikimedia.org/r/195079

Everything seems ok. Now to remove the unneeded packages from trusty...

You will end up with :-):

scfc@toolsbeta-puppetmaster3:~$ ls -l /data/project/.system/deb-trusty
total 1564
-r--r--r-- 1 root tools.admin   8922 Mar  8 05:23 jobutils_1.0.10_all.deb
-r--r--r-- 1 root tools.admin   8954 Mar  8 05:23 jobutils_1.0.11_all.deb
-rw-r--r-- 1 root tools.admin  65278 Mar  8 05:28 libmediawiki-bot-perl_5.005006-1_all.deb
-r--r--r-- 1 root tools.admin  11342 Mar  8 05:23 misctools_1.0.10_amd64.deb
-r--r--r-- 1 root tools.admin  11390 Mar  8 05:23 misctools_1.0.11_amd64.deb
-rw-r--r-- 1 root tools.admin  13612 Mar  8 05:23 misctools_1.0.12_amd64.deb
-rw-r--r-- 1 root tools.admin  13654 Mar  8 05:23 misctools_1.0.13_amd64.deb
-rw-r--r-- 1 root tools.admin  13808 Mar  8 05:23 misctools_1.0.14_amd64.deb
-rw-r--r-- 1 root tools.admin  11680 Mar  8 05:23 misctools_1.1_amd64.deb
-rw-r--r-- 1 root tools.admin   4773 Mar  8 05:38 Packages.gz
-rw-r--r-- 1 root tools.admin   6766 Mar  8 05:37 python-backports.ssl-match-hostname_3.4.0.2-1_all.deb
-rw-r--r-- 1 root tools.admin   2590 Mar  8 05:28 python-celery-with-redis_3.0-1_all.deb
-rw-r--r-- 1 root tools.admin   7512 Mar  8 05:23 python-flask-oauth_0.12-1_all.deb
-rw-r--r-- 1 root tools.admin  60458 Mar  8 05:28 python-mwparserfromhell_0.3.3-1_amd64.deb
-rw-r--r-- 1 root tools.admin 109128 Mar  8 05:28 python-oursql_0.9.3.1-1_amd64.deb
-rw-r--r-- 1 root tools.admin  18896 Mar  8 05:28 python-sh_1.09.1~precise-1_all.deb
-rw-r--r-- 1 root tools.admin  11568 Mar  8 05:28 python-socketio-client_0.5.3-1_all.deb
-rw-r--r-- 1 root tools.admin 183878 Mar  8 05:34 python-websocket-client_0.23.0-1_all.deb
-rw-r--r-- 1 root tools.admin  18150 Mar  8 05:28 python-wikitools_1.1.1-1_all.deb
-rw-r--r-- 1 root tools.admin 983392 Mar  8 05:23 sbt.deb
scfc@toolsbeta-puppetmaster3:~$

Did you remove some from deb-trusty for tools? misctools etc is missing there..

Yes, accidentally, I started with a clean plate, added what was missing, and as misctools was already installed everywhere … Added back again.

After removing the unneeded packages from deb-trusty and running Puppet and apt-get update on the Trusty instances a bunch of times, now all Puppet runs succeed.