Page MenuHomePhabricator

Pywikibot 8 requires mwparserfromhell, which is not present on the Toolforge grid
Closed, DeclinedPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

  • Run python3 /data/project/shared/pywikibot/core_stable/pwb.py replace.py on the bastion host

What happens?:
The 'mwparserfromhell>=0.6.3' distribution was not found and is required by the application

A package necessary for pywikibot is missing.
Please update required module with:

pip install "mwparserfromhell>=0.6.3"

What should have happened instead?:
Should work. If replace.py is invoked directly ($ python3 /data/project/shared/pywikibot/core_stable/scripts/replace.py), no error is given.

Software version (skip for WMF-hosted wikis like Wikipedia):

Other information (browser name/version, screenshots, etc.):
Probably introduced in T326498

Event Timeline

taavi renamed this task from pwb.py on Toolforge throws mwparserfromhell error on Pywikibot 8 release to Pywikibot 8 requires mwparserfromhell, which is not present on the Toolforge grid.Jan 22 2023, 2:09 PM
taavi edited projects, added Toolforge (Software install/update); removed Toolforge.
taavi moved this task from Unsorted to Job grid on the Toolforge (Software install/update) board.

For whatever reason we only install python-mwparserfromhell (which is Python 2), and not the python3 variant. Apparently this dates all the way back to 2014 (T65539) and originated as a Toolforge-specific package. Nowadays the package is available in the main Debian repos (maintained by @Legoktm), however 0.6.3 or newer is not available:

taavi@tools-sgebastion-11:~ $ apt-cache policy python3-mwparserfromhell
python3-mwparserfromhell:
  Installed: (none)
  Candidate: 0.5.2-2
  Version table:
     0.6-1~bpo10+1 100
        100 http://mirrors.wikimedia.org/debian buster-backports/main amd64 Packages
     0.5.2-2 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages

Buster has 0.5.2 and buster-backports has 0.6.0. Additionally buster-backports should have been retired already, so we can't rely on that either. We might be able to do a custom backport, but I'd really rather not spend time on this given the future plans for the job grid.

Change 882220 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::grid: install python3-mwparserfromhell

https://gerrit.wikimedia.org/r/882220

This looks strange for me:

tools.xqtools@tools-sgebastion-10:~$ python3
Python 3.7.3 (default, Oct 31 2022, 14:04:00)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import mwparserfromhell
>>> mwparserfromhell.__version__
'0.6.3'
>>>

Where does this version come from?

This looks strange for me:

You have this in the .profile of that tool:

export PYTHONPATH=/data/project/shared/pywikibot/stable:/data/project/shared/pywikibot/core

So it's being loaded from /data/project/shared/pywikibot/stable/mwparserfromhell which contains the library

taavi@tools-sgebastion-11:/data/project/shared/pywikibot/stable $ ls mwparserfromhell/
definitions.py  __init__.py  nodes  parser  smart_list  string_mixin.py  utils.py  wikicode.py

So this becomes more of a question to the maintainers of the pywikibot tool on how everything is intended to work in the first place.

Would it help to decrease mwparserfromhelp dependency to =>0.5.2?

To use the shared pywikibot install, it is necessary to modify PYTHONPATH so that dependencies are available. (docs)

Change 882220 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] P:toolforge::grid: install python3-mwparserfromhell

https://gerrit.wikimedia.org/r/882220

Change 905545 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [fix] Decrease needed mwpfh version to 0.5.2

https://gerrit.wikimedia.org/r/905545

Xqt triaged this task as High priority.Apr 4 2023, 8:19 AM

Change 905545 merged by Xqt:

[pywikibot/core@master] [fix] Decrease needed mwpfh version to 0.5.2

https://gerrit.wikimedia.org/r/905545

Change 908467 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@stable] [fix] Decrease needed mwpfh version to 0.5.2

https://gerrit.wikimedia.org/r/908467

Change 908467 merged by Xqt:

[pywikibot/core@stable] [fix] Decrease needed mwpfh version to 0.5.2

https://gerrit.wikimedia.org/r/908467

Xqt lowered the priority of this task from Medium to Low.Apr 15 2023, 5:16 PM

I'm closing all grid engine install/update tasks as the [[ grid engine is being removed | https://wikitech.wikimedia.org/wiki/News/Toolforge_Grid_Engine_deprecation ]]. Please see the instructions page on how to migrate your tool to Kubernetes. Thanks.