Page MenuHomePhabricator

Cleaning up and refactoring python-wikitools (for use with Python 3)
Open, Needs TriagePublic


Based on the following two discussions ...

  1. Py3 branch can not be installed with setuptools (dependency_links, install_requires), only through requirements.txt, due to version overload @ github
  2. Update python-wikitools on Tool Labs @ phabricator

... I'd be interested in cleaning up and possibly refactoring this module. I am building an application, which is already using this module and I am in the process of preparing it for publication. Because I am developing on Python 3, this is where my main interest is.

The problem with regards to python-wikitools can be summarized as follows: The currently publicly available version of python-wikitools (1.3 on PyPI) works with Python 2 (aka legacy Python) only. This is also what the master branch on GitHub contains. Next to it, there is a branch named "py3" on GitHub, which contains a working port of python-wikitools (labeled version 1.2) for contemporary Python 3. As opposed to the master branch, the branch py3 contains a quite broad and promising set of (unit) tests. Both branches have in common that they do not follow best practices in terms of Python packaging and repository layouts, which prohibits easy packaging and distribution, both on PyPI and various Linux distributions. Both branches also have in common that the code lacks a bit of a clear maintainable structure, but this is the lesser of the evils at this point. The packaging and distribution part is the real problem right now.

For reasons explained in the above mentioned discussion on GitHub, the Python 3 branch can basically not be used as a dependency in real-life Python packages and modules because it can not properly be defined as a dependency with setuptools. I requires (a) a hack in its source code and (b) dropping security measures when installing a package depending on it with pip (the controversial "--process-dependency-links" flag).

I'd be willing to invest some time into resolving the packaging situation if I receive some feedback on the following questions:

  1. How relevant is Python 2 support at this point? (I'd say it is relevant and there should be working packages for Python 2, but this comes at the expense of a lot of extra work).
  2. Related to the last question, what kind of environment is used by Tool Labs (if relevant)?
  3. If BOTH Python 2 and 3 must be supported, which form is preferred? I'd vote against a six-based approach with a shared code-base and rather maintain two similar-ish code-bases right next to each other, but this is really a question of taste and of who is interested in maintaining this in the long term.
  4. I'd start by setting up CI support (probably for Travis), which is fairly trivial on its own for Python packages, but I would be needing a working (and "self-cleaning") MediaWiki instance, vm image or docker image or similar for most of the tests. The latter is beyond my available time - could someone here help me out with this? It does not make sense to clean up python-wikitools if there was not a test infrastructure.
  5. Beyond PyPI, I'd be packaging for SuSE-based (i.e. rpm-based) distributions, but I'd prefer if someone else takes care of Debian-based ones.

Event Timeline

smernst created this task.Sep 18 2017, 7:35 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 18 2017, 7:35 AM
smernst updated the task description. (Show Details)Sep 18 2017, 7:42 AM

This looks like (support) questions, not like a valid bug report or enhancement request?
#1 to #3 should probably be asked on the mailing list at (or on IRC)
#4: / /

smernst added a comment.EditedSep 18 2017, 8:28 AM

@Aklapper You're right. I am tracking down the stakeholders of the python-wikitools project and I though here is a good place to start ... it's one of the best (and the few actually maintained) Python MediaWiki API wrappers out there. Thanks for the links.

Hi @smernst. Are you still interested in discussing this Python module?

Xqt added a subscriber: Xqt.Jul 24 2019, 8:37 AM
bd808 edited projects, added Tools; removed Toolforge.Jan 4 2020, 9:12 PM