Not sure why the Toolforge project was removed by @JJMC89 . A package is missing on the generic infrastructure affecting multiple tools. Also no mention of this on https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot#Clone_pywikibot_git_repo

You can request Toolforge software installs by creating a task with Toolforge (Software install/update); however, there is no requirement for Toolforge to have Pywikibot dependencies installed. mwparserfromhell is provided centrally in the shared install, which is the recommended way to use Pywikibot on Toolforge.

The AttributeError will not be raised when the dependencies are installed.

We (Toolforge admins) encourage people to use virtualenvs to install dependencies for their tools as those will get you much more up to date versions than what Debian ships and don't require us to install the dependencies globally on all grid nodes. You need to have Pywikibot's dependencies installed somehow if you plan to use it. I don't believe we've ever guaranteed to include dependencies needed by clones of (popular) frameworks as it's outside of our control when those frameworks add new/updated requirements (although it looks like Pywikibot has been working with what's installed on the grid by default in the past).

That being said, we do install mwparserfromhell on all grid nodes, but for Python 2 only. Installing the Python 3 version is likely possible as well, but the version on Debian Stretch is 0.4 (compared to 0.5 what the error message claims to require). Upgrading the grid to a newer operating system release has been largely stalled as we've been working on creating Kubernetes-based replacements for the grid.

To summarize a bit:

• Toolforge policy is to not install Python packages system-wide because it causes confusion and conflicts over versions. People should use virtualenvs or pip install --user instead.
  • Debian Stretch (what the bastions/grid run) contains mwparserfromhell 0.4.2, which wouldn't meet Pywikibot's requirement of >=0.5.0 anyways.
• Pywikibot requires external dependencies and the shared Toolforge installation (maintained by Pywikibot developers, not Toolforge admins) installs them for you.
  • As of April / T106763: Mandatory dependency on mwparserfromhell, mwparserfromhell or wikitextparser is required.
  • Pywikibot also requires requests and setuptools, but those are installed by coincidence because other system packages need them.

If you want to maintain your own Pywikibot installation, IMO using a virtualenv is the easiest. You could also pip install --user mwparserfromhell too, I don't remember exactly what downsides that has over a virtualenv.

Otherwise, I'm not really sure what's actionable in this ticket besides asking for a revert of the mwparserfromhell/wikitextparser dependency.

• MediaWiki markup parser dependency was introduced because the Pywikibot's regex parser failed for nested templates with depth 3 and the old compat parser was preplaced years ago due to other bugs, see T106763.
• mwparserfromhell >= 5.0 is required due to T71384 and solved issues with iterables
• the exception on exit time occures because Pywikibot framework isn't fully loaded when the dependency is checked and the script is exited in this early state. This is indeed ugly but circular dependency of framework parts mainly of the logging module is still problematic in such cases (btw circular Import problems was the main reason to give up Python 3.5.0-3.5.2). Will be solved shortly I guess.

What can we do on Pywikibot's side?

• We can check the impacts for downgrade the mwpfh version to 4.2 but this is 6 years old and I am not sure whether it works with newer Pythons and if we get back other bugs. Unfortunately we have lost the Travis CI for testing.
• We can recover the old regex parser but a lot of bugs are unsolved then and we have a different behavior of the results
• we can have our own MW markup parser but I have no time to recover it from compat and fix the known issues

I am a bit confused about a preinstalled mwpfh noted above because textlib itself doesn’t care about its version [1] and should not raise that given exception. Is it really preinstalled? Or is it for Py2 only?

[1] https://gerrit.wikimedia.org/r/plugins/gitiles/pywikibot/core/+/13c2008f4eb6fe44450339372085f3faf22434a4/pywikibot/textlib.py

The "AttributeError: module 'pywikibot' has no attribute 'Site' error is fixed with T298384. Is anything left to do here?

In T291771#7591567, @Xqt wrote:

The "AttributeError: module 'pywikibot' has no attribute 'Site' error is fixed with T298384. Is anything left to do here?

Did you try a blank install on Toolforge without virtualenv? As long as that doesn't work, this problem isn't solved. A work-around exists so you can lower the priority. To really fix this two options exist:

• Toolforge should do the (bit overdue) update of Debian so that all dependencies are installed by default
• Pywikibot should strip out some of the dependencies so it runs on the current Toolforge

Hi @Multichill, thanks for your response.

Did you try a blank install on Toolforge without virtualenv?

I have von toolforge access, therefore I have no glue what packages are preinstalled there. It would be a great help if you could give me the response of pip freeze which shows the preinstalled packages with their package version.

Toolforge should do the (bit overdue) update of Debian so that all dependencies are installed by default

I suggest to create a separate task for it (if it's not already done)

Pywikibot should strip out some of the dependencies so it runs on the current Toolforge

A wikitext parser become mandatory due to T106763 with Pywikibot 6.3 (rPWBC21f9f7d), the old behaviour was deprecated with 6.1 (rPWBC525dbc7) and a lot of bugs could be solved with that change.
I don't know about the installation status of other packages which are mandatory. What about requests, which is mandatory since Pywikibot 3.0 I guess. What about setuptools which is shipped with Python but a minimum version is required for versioning.
Is using venv to complicated on toolforge? An idea would be to include the needed wikitext parser as exterals like we had previously with BeautifullSoup in compat branch. Would that be appropriate and help in this matter?