Page MenuHomePhabricator

Install all pywikibot python optional dependencies on tool labs
Closed, ResolvedPublic

Description

Following on from T65539 ..

The following packages are optional dependencies of pywikibot ( http://git.wikimedia.org/blob/pywikibot%2Fcore.git/HEAD/requirements.txt ), but needed for some scripts and library components to function, and appear missing from modules/toollabs/manifests/*.pp , and should be installed for both python 2 and python 3 unless marked otherwise, roughly in order of importance:

  • unicodecsv (Debian package python-unicodecsv - py2 only ; this will likely soon become a mandatory dependency)
  • unittest2 (Debian package python-unittest2 - py2 only ; mandatory for the test suite on py2.6, but we may soon need to adopt it for py2.7 and py3.3 testing also, as unittest2 is the same as py3.4 unittest, and is .. 'the future')
  • ipaddr (Debian package python-ipaddr/ python3-ipaddr )
  • tkinter (Debian package python-tk / python3-tk )
  • stdnum (Debian package python-stdnum / python3-stdnum)
  • pycountry (Debian package python-pycountry - no py3 package)
  • daemonize
  • flickrapi
  • pydot (Debian package python-pydot is unsuitable)
  • crontab
  • pYsearch (py2 only iirc)
  • Pillow, GUI interface, used for flickrripper
  • lunatic-python is an unsolved problem; there are many forks of this module and the pywiki core team hasnt done any testing on which one is best. @DrTrigon is the main user of this as far as we know.
  • google is a click-through license nightmare, and probably not appropriate for installation on Wikimedia servers.

If there are any which cant be installed for whatever reason, the pywikibot team may be able to update the code to use an alternative library of approach.

Event Timeline

jayvdb raised the priority of this task from to Needs Triage.
jayvdb updated the task description. (Show Details)
jayvdb added a project: Toolforge.
jayvdb added a project: Pywikibot.
jayvdb set Security to None.
jayvdb added a subscriber: jayvdb.

Pywikibot compat has more dependencies, many of which will become core dependencies as part of the ongoing OPW Pywikibot-compat-to-core project porting scripts to core. T71980 (over four months old) is raised for one of them.

Hmm, I'm wondering if we should figure out a virtualenv based way of setting up pywikibot on toollabs. This would give pywikibot maintainers the ability to maintain the required packages themselves without relying on the labs admins.

However, if the pywikibot folks think that maintaining the virtualenv themselves is going to be too much work / not something they want to do, I'm happy to install all these packages.

@yuvipanda, it would be great if you could install these dependencies in the first instance. Some of them are distributed as part of the OS package collection. Also I believe Pillow needs a sysadmin action, as Ubuntu puts python's Tkinter into a separate debian package.

If there are any packages which you dont feel comfortable installing globally, or if the dev team adds dependencies too frequently, we need to consider setting up maintained virtualenv package set, but hopefully we can avoid that. Most of these libraries are generally useful to anyone doing wiki stuff in python.

Please also add "socketIO_client" to the list; it is used to connect to the newish RecentChanges stream , so that it available to all python users.

Change 184298 had a related patch set uploaded (by Yuvipanda):
tools: Install some pywikibot dependencies

https://gerrit.wikimedia.org/r/184298

Patch-For-Review

Change 184298 merged by Yuvipanda:
tools: Install some pywikibot dependencies

https://gerrit.wikimedia.org/r/184298

So I've installed:

  1. unittest2
  2. pycountry
  3. flickrapi
  4. pydot

pysearch seems to be just pointing to python-yahoo now in ubuntu. Is that ok to be installed?

unicodecsv and socketIO require new packages to be built, so I shall do that later today.

Why are daemonize and crontab required?

daemonize supports the library command line option -daemonize. I dont
know much more than that, but I can dig into it.

crontab is used to parse a crontab stored on a wiki page, used to
schedule tasks. It is only used by a script that is not yet
functional, so not a high priority atm.

will look into the yahoo package

I'm not sure if we should support daemonize. Jobs should just go on the grid instead.

Change 184302 had a related patch set uploaded (by Yuvipanda):
tools: Install python-unicodecsv

https://gerrit.wikimedia.org/r/184302

Patch-For-Review

Change 184302 merged by Yuvipanda:
tools: Install python-unicodecsv

https://gerrit.wikimedia.org/r/184302

Change 184887 had a related patch set uploaded (by Yuvipanda):
tools: Install python-socketio-client

https://gerrit.wikimedia.org/r/184887

Patch-For-Review

Change 184887 merged by Yuvipanda:
tools: Install python-socketio-client

https://gerrit.wikimedia.org/r/184887

yuvipanda changed the task status from Open to Stalled.Jan 14 2015, 2:56 PM

So, socketio client has been installed as well.

Open Questions:

  1. is python-search still needed if it is just an alias for python-yahoo?
  2. Should we allow deamonize? My gut feeling is 'no' (I realize that this won't prevent people from actually using it via virtualenv, but jobs should go on the grid)
  3. Is crontab still required? ToolLabs has a strangeish cron setup, so I'm hoping whatever pywikibot feature this is it sets up jobs to execute via jsub rather than directly.

(sorry it took so long for me to get back to this)

  1. python pysearch , I think I see what you are saying.

https://packages.debian.org/source/wheezy/python-pysearch is a transitional package for https://packages.debian.org/wheezy/python-yahoo

The filelist of https://packages.debian.org/wheezy/all/python-yahoo/filelist looks very similar to the source http://sourceforge.net/p/pysearch/code/HEAD/tree/trunk/yahoo/search/ listed at https://pypi.python.org/pypi/pYsearch , and it the appropriate .egg

So, I think only installing 'python-yahoo' is fine.


  1. daemonize . Sorry, this can be ignored, as this external dependency is due to my misunderstanding; see T89581 .

p.s. If you dont want -daemonize used on tool labs, create a task to add a warning or error if it is used on tool labs. Tool labs is pywikibot's primary customer, so ... IMO the software should try to help enforce/promote the way you want the software used on your free hardware.


  1. crontab

This module is only a crontab parser.

https://pypi.python.org/pypi/crontab
is
https://github.com/josiahcarlson/parse-crontab

It is used to parse the second item in each own of this page

https://de.wikipedia.org/wiki/Benutzer:DrTrigon/DrTrigonBot/script_wui-crontab.css

The first item is a revid, which is currently

https://de.wikipedia.org/wiki/?oldid=125193969

Those on-wiki 'files' are semi-protected due to the fact they are .css files.

The bot and operator was last active in August 2014 : https://de.wikipedia.org/wiki/Spezial:Beitr%C3%A4ge/DrTrigonBot / https://de.wikipedia.org/wiki/Spezial:Beitr%C3%A4ge/DrTrigon (https://tools.wmflabs.org/guc/?user=DrTrigon) https://gerrit.wikimedia.org/r/#/q/owner:%22DrTrigon+%253Cdr.trigon%2540surfeu.ch%253E%22,n,z They dont have a Phab account that I can see.

I would not mind if we deferred the 'what to do' with this script and its crontab dependency a little longer to see if DrTrigon becomes active. The dependency is optional, and nobody else is seriously working on improving this script at the moment (see T70797), so the dependency isn't needed for regression testing. The Lua dependency (mentioned in the task description) needs to be resolved before the crontab dependency is a serious impediment.


But wait, there is more!

Other dependencies added recently:

python3: six (probably already installed)
python2.6: future (hopefully you're not maintaining a py2.6 environment..;-)

Now that T76286 is merged, another dependency for py2.6 and 2.7 is https://pypi.python.org/pypi/ipaddress

scfc triaged this task as Low priority.Apr 6 2015, 10:33 AM
scfc moved this task from Triage to Backlog on the Toolforge board.
scfc added a subscriber: scfc.

Change 209978 had a related patch set uploaded (by Merlijn van Deen):
Add python-ipaddress package

https://gerrit.wikimedia.org/r/209978

jayvdb renamed this task from Install all pywikibot python dependencies on tool labs to Install all pywikibot python optional dependencies on tool labs.May 27 2015, 1:35 PM

Change 209978 abandoned by Merlijn van Deen:
Add python-ipaddress package

https://gerrit.wikimedia.org/r/209978

Now that T76286 is merged, another dependency for py2.6 and 2.7 is https://pypi.python.org/pypi/ipaddress

Now that T100603: Support Debian package python-ipaddr is resolved, that dependency for py2.7 has changed to Debian package python-ipaddr. Could it be installed please, and is it optional, but quite critical for performance.

jayvdb updated the task description. (Show Details)

Change 216904 had a related patch set uploaded (by Tim Landscheidt):
Tools: Add python-ipaddr to exec_environ

https://gerrit.wikimedia.org/r/216904

Change 216904 merged by Yuvipanda:
Tools: Add python-ipaddr to exec_environ

https://gerrit.wikimedia.org/r/216904

jayvdb updated the task description. (Show Details)
jayvdb updated the task description. (Show Details)

While Pillow is not yet installed, most of the core optional dependencies have been installed, and I think the desired approach is to create separate tasks and organise debian packages for them.

jayvdb claimed this task.