I'm going to build a new anaconda-wmf package soon. When we do, we can update or include any conda or pip package that is out there. These packages will installed on stat boxes as well as all Hadoop worker nodes in /usr/lib/anaconda-wmf.

If there's a package you'd like us to include that we don't already, please let me know in this ticket soon!

Aside from the already huge number of packages from the base Anaconda distribution, we also install these:

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	Add requested packages to base anaconda-wmf	Ottomata	operations/debs/anaconda-wmf	debian	+7 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Ottomata	T224658 Newpyter - SWAP Juypter Rewrite
Resolved	Ottomata	T262847 Decomission SWAP
Resolved	Ottomata	T271960 New anaconda-wmf release with updated packages

Event Timeline

Ottomata created this task.Jan 13 2021, 5:14 PM

If we could upgrade jupyterhub-systemdspawner to 0.15 it would be nice :)

Thanks @Ottomata! The three that I can think of quickly are:

mwapi for making easy requests to Mediawiki APIs. Obviously less relevant for worker nodes but I often use this in the same notebook as I'm running PySpark jobs for various data augmentation etc. I think only available via pip.
mwxml for easy access to the dumps. Again, less for worker nodes and more for being able to mix analyses that depend on the cluster with dump processing. I think only available via pip.
shapely for spatial analyses. If this one is a big package, it's okay to leave it off as it's probably more specific to me. Available via pip but also a conda package.

To clarify, the base Anaconda packages we'll have access to are only the ones marked "In Installer" in the link you shared?

To clarify, the base Anaconda packages we'll have access to are only the ones marked "In Installer" in the link you shared?

I actually don't know for sure, but I'd assume yes.

As I understand it, this just determines what's installed by default; the user can always manually install packages that aren't included (at least on the analytics clients). Is that right?

That's right! To get the deps on the worker nodes if they aren't already there is a bit of extra work for the user (that's you!) though.

• razzi moved this task from Incoming to Data Exploration Tools on the Analytics board.Jan 14 2021, 6:06 PM

• razzi added a project: Analytics-Kanban.

LGoto moved this task from Triage to Tracking on the Product-Analytics board.Jan 19 2021, 6:13 PM

Ottomata claimed this task.Jan 22 2021, 4:09 PM

Change 668453 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/debs/anaconda-wmf@debian] Add requested packages to base anaconda-wmf

https://gerrit.wikimedia.org/r/668453

gerritbot added a project: Patch-For-Review.Mar 4 2021, 2:56 PM

Change 668453 merged by Ottomata:
[operations/debs/anaconda-wmf@debian] Add requested packages to base anaconda-wmf

https://gerrit.wikimedia.org/r/668453

Maintenance_bot removed a project: Patch-For-Review.Mar 4 2021, 10:10 PM

Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.Mar 11 2021, 6:24 PM

Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.Mar 22 2021, 1:25 PM

Ottomata closed this task as Resolved.Apr 6 2021, 7:12 PM