Page MenuHomePhabricator

Move cloudera packages to a separate archive section
Closed, ResolvedPublic3 Estimated Story Points

Description

We're using a wide range of external packages from Cloudera. Right now these are all in the reprepro archive section "thirdparty". Right now "thirdparty" is present on all machines in the production cluster and chd packages contain packages which are only present in standard Debian and which we are using outside the analytics clusters (zookeeper at least, there might be more, needs a complete review), so we don't have a clean way to restrict the CDH packages to analytics.

I would prefer if we could migrate them to a separate archive section (e.g. cdh or maybe analytics-external), which is only added via /etc/apt/sources.list.d/cdh on analytics machines. This doesn't need to be done immediate, but could be e.g. done when importing the next set of Hadoop packages.

Event Timeline

@MoritzMuehlenhoff, do you a preference re 'cdh' vs 'analytics-external'? 'analytics-external' might be more future proof, but what if we need to install stuff from cdh on non analytics hosts...or we want to put things in analytics-external that could be needed on non analytics hosts? Either way has pros and cons.

I think I'm leaning towards cdh. Then it can be an exact (whitelisted) mirror of cdh apt.

I also have a (slight) preference towards cdh.

Ottomata edited projects, added: Analytics-Kanban; removed: Analytics.
Ottomata set the point value for this task to 3.

Change 337657 had a related patch set uploaded (by Ottomata):
Add cdh-trusty and cdh-jessie reprepro updates and mirror them to a new cdh component

https://gerrit.wikimedia.org/r/337657

TODO after CDH upgrade, remove old cloudera/thirdparty updates.

Change 337657 merged by Ottomata:
Add cloudera-trusty and cloudera-jessie reprepro updates and mirror them to a new cloudera component

https://gerrit.wikimedia.org/r/337657

Change 337877 had a related patch set uploaded (by Ottomata):
Configure analytics cluster nodes to use thirdparty/cloudera apt component

https://gerrit.wikimedia.org/r/337877

Change 337877 merged by Ottomata:
Configure analytics cluster nodes to use thirdparty/cloudera apt component

https://gerrit.wikimedia.org/r/337877

Ha! OOPS I knew I would make a mistake when cleaning up old packages! I accidentally removed almost all CDH packages from thirdparty/cloudera. Luckily, updates will pull them back in for us. Annnnnnd done! https://apt.wikimedia.org/wikimedia/pool/thirdparty/cloudera/