Page MenuHomePhabricator

phe-tools: Match&Split bot is not running because of python2 deprecation in pywikibot
Open, Needs TriagePublic

Description

Phe's Match&Split bot shows as "match_and_split robot is not running. Please try again later."

https://phetools.toolforge.org/match_and_split.php

It was slowly grinding to a halt yesterday (possibly coincident with the MW deploy train hitting the Wikisourcen), and has been completely down all day today.

Could someone with tool access give it a little kick?

Related Objects

StatusSubtypeAssignedTask
ResolvedDalba
ResolvedDalba
OpenNone
ResolvedXqt
DeclinedNone
ResolvedXqt
DeclinedNone
ResolvedXqt
ResolvedXqt
ResolvedGoalXqt
ResolvedDvorapa
ResolvedLegoktm
OpenBUG REPORTNone
ResolvedBUG REPORTDvorapa
OpenNone
ResolvedBUG REPORTDvorapa
OpenNone
ResolvedXqt
ResolvedBUG REPORTDalba
ResolvedBUG REPORTDalba
ResolvedBUG REPORTXqt
ResolvedBUG REPORTXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedBUG REPORTDalba
ResolvedXqt
ResolvedXqt
ResolvedBUG REPORTDalba
ResolvedDalba
OpenBUG REPORTNone
ResolvedBUG REPORTXqt
ResolvedBUG REPORTNone
ResolvedBUG REPORTDzahn
ResolvedBUG REPORTDalba
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
OpenNone
ResolvedXqt
OpenNone
DeclinedNone
InvalidNone
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedBUG REPORTXqt
ResolvedDvorapa
ResolvedDvorapa
OpenNone
OpenNone
ResolvedUrbanecm
OpenNone
ResolvedDvorapa
ResolvedBUG REPORTDvorapa
ResolvedDvorapa
OpenNone
OpenNone
ResolvedXqt
ResolvedXqt
OpenNone
ResolvedDvorapa
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
DeclinedNone
InvalidNone
ResolvedUrbanecm
ResolvedXqt
ResolvedXqt
DeclinedNone
Resolvedjayvdb
DeclinedNone
DuplicateNone
ResolvedDvorapa
ResolvedXqt
DeclinedNone
DeclinedNone
Resolvedhashar
ResolvedMpaa
ResolvedAndrew
DeclinedNone
ResolvedXqt
ResolvedDvorapa
ResolvedXqt
ResolvedDvorapa
DeclinedNone
ResolvedXqt
DeclinedNone
ResolvedDvorapa
ResolvedXqt
ResolvedXqt
DeclinedNone

Event Timeline

Removing Community-Tech as it's unclear why that team was added.

@Aklapper Indeed. Community-Tech was added as their Toolforge group account is one of the four accounts set as admin for the phetools Toolforge project.

Eh, sorry and thanks for the explanation!

MusikAnimal added a subscriber: MusikAnimal.

I think we're only concerned with phetools/credits.py because it is used by WS Export. I'm afraid we can't help with the issue here. Sorry!

@Candalua That leaves you as the only admin on phetools with any likelihood of having the spare cycles to look at this (Phe and Tpt are highly unlikely to be available for this any time soon). Any chance you could poke around here a bit?

Alternately, could you give me a pointer on the process you followed to be added to the project? While PHP and Python aren't quite my cuppa, I can certainly do basic *nix debugging and HUP any process that needs a kick (“This is how we fix problem on Russian space station!”).

@Xover I'm not familiar with the match&split tool, normally I just tend to the statistics tool. However I added you to the project, any help that you can contribute is more than welcome.

@Candalua Thanks!

I took a quick peek and it looks like the problem is that pywikibot (which match_and_split.py imports) was updated to a version that requires Python 3.x on October 1st, and the phetools codebase is executing under Python 2.x. The code uses syntax that Python 3.x chokes on, so we can't simply sub in python3 in phe/run_service.sh.

I'll try to dig a bit further tomorrow and see if there are any workarounds short of porting phetools to Python 3.x wholesale.

bd808 renamed this task from phe-tools: Match&Split bot is not running to phe-tools: Match&Split bot is not running because of python2 deprecation in pywikibot.Nov 4 2020, 8:33 PM

I took a quick peek and it looks like the problem is that pywikibot (which match_and_split.py imports) was updated to a version that requires Python 3.x on October 1st, and the phetools codebase is executing under Python 2.x. The code uses syntax that Python 3.x chokes on, so we can't simply sub in python3 in phe/run_service.sh.

I'll try to dig a bit further tomorrow and see if there are any workarounds short of porting phetools to Python 3.x wholesale.

The quickest fix would be to use the py2 version of pywikibot.
For Toolforge shared: Instead of using /shared/pywikibot/core_stable [most recent release] (or /shared/pywikibot/core [nightly updates from master]) use /shared/pywikibot/core_python2 [final python2 release].
For other sources: use a version < 4.0

@JJMC89 Thanks!

I switched the environment to point at /shared/pywikibot/core_python2 and that seems to have been sufficient for match_and_split, but the codebase for all the tools is pretty tightly integrated so it may yet turn out that this breaks other parts of phetools.

In particular, phetools was using a local copy of httplib2 whose provenance I am uncertain of (it now uses the system-provided version); and a local copy of the pywikibot scripts/ directory that may or may not have local modifications.

I'll try to do some testing on the other components to verify that they too work with this change.