Page MenuHomePhabricator

[BREAKING] Change: Switch from httplib2 to requests
Closed, InvalidPublic

Description

After T98671, the following occurs:

C:\pwb\core>pwb.py redirect double -moves -total:10 -always -lang:pa
ImportError: No module named requests
Python module requests is required.
Try running 'pip install requests'.

Not a good idea to break the bot without any hint!

Details

Related Gerrit Patches:
pywikibot/core : master[FEAT] Basic httplib2 support

Event Timeline

Xqt created this task.Jun 3 2015, 7:43 AM
Xqt raised the priority of this task from to High.
Xqt updated the task description. (Show Details)
Xqt added a project: Pywikibot.
Xqt added a subscriber: Xqt.
Restricted Application added subscribers: Aklapper, Unknown Object (MLST). · View Herald TranscriptJun 3 2015, 7:43 AM
jayvdb added a subscriber: jayvdb.Jun 3 2015, 8:21 AM

The hint is there: install requests.
See also the email to the mailing list.

Its still bad form to break a framework without having a fallback. What should have happened is introduce new system, throw warnings to user about missing module, give users reasonable time to fix, then remove the fallback for maintainability. However stability of the framework seems to have degraded significantly since moving from compat. Introducing such a breaking chance without a fallback is a Really Bad Idea™, and is one of the main reasons I know people continue to use compat since there are far fewer stability issues.

The fact master is going to be unstable was announced after the RC was released on May 28, with explicit recommendation to not update your master for bots running on production wikis.

https://lists.wikimedia.org/pipermail/pywikibot/2015-May/009268.html

The tarball releases are currently locked to the git revision before
the Lyon Hackathon.  This was done to prevent large code merges
affecting users of the tarballs and Wikimedia labs shared pywikibot.
There was one large change merged for RC 1, and there are a few more
large changes which will be merged before the final pywikibot 2.0.

I suggest bot operators avoid the unstable master by not updating
regularly using git, but using the labs shared version or the pip
package.  We'll announce each new release candidate for people to
upgrade to.
XZise added a subscriber: XZise.Jun 3 2015, 11:51 AM

The hint is there: install requests.
See also the email to the mailing list.

I guess @Xqt meant with “hint” a warning beforehand.

Xqt added a comment.Jun 3 2015, 11:57 AM

I don't understand such a recommendations. We always recommended "keep your bot up-to-date". And now "please dont update" ... and wait for bugfixes, translations, features until we have finished the bot in any spring; the current release may be broken. But please merge to core and don't use the ... I should give up.

Change 215603 had a related patch set uploaded (by XZise):
[FEAT] Basic httplib2 support

https://gerrit.wikimedia.org/r/215603

I don't understand such a recommendations. We always recommended "keep your bot up-to-date". And now "please dont update" ... and wait for bugfixes, translations, features until we have finished the bot in any spring; the current release may be broken. But please merge to core and don't use the ... I should give up.

The reason to move to a versioned system is to provide better long-term stability for bot users. Unfortunately, we're still in the process of figuring out how to do this exactly, so not everything will be perfect on the first attempt.

In the current situation, master would break every now and then, and require changes to bots. This was typically not planned, but it did mean that blindly 'git pull'ing always was a bad idea. The idea for versioning is the following. When 2.0 is released, only bug fixes will be backported, so it's completely safe to update to the latest 2.0.x. More major changes would end up in 2.1, and really bit changes (e.g. the removal of deprecated compat wrappers) would become 3.0.

In this specific situation, I think it might be advisable to actually have this change in 2.1, not in 2.0. Then current users can switch to the 2.0 branch safely, and we can work on 2.1 without worrying about changes such as these.

jayvdb added a comment.Jun 4 2015, 2:30 AM

@Xqt, could you explain why 'pip install requests' doesnt work for you.
You added that pip was working to the Version table.
And we have Win32 builds running on Appveyor, and those builds install requests without a problem. https://ci.appveyor.com/project/jayvdb/pywikibot-core/build/1.0.fix-appveyor.239

Another way to install requests is here:
http://www.lfd.uci.edu/~gohlke/pythonlibs/#requests
Maybe that works for you...?

In this specific situation, I think it might be advisable to actually have this change in 2.1, not in 2.0. Then current users can switch to the 2.0 branch safely, and we can work on 2.1 without worrying about changes such as these.

Users who are switching from compat to core are switching to a http library, either httplib2 or requests makes little difference.
If the benefit is that httplib2 was in externals, and therefore it runs without any dependencies needing to be installed, we need to evaluate whether that is necessary. What is causing the problems. If it is necessary to assist the switch to core, why not add externals/requests instead of externals/httplib2?

jayvdb added a comment.Jun 4 2015, 7:03 AM

On https://gerrit.wikimedia.org/r/#/c/215603/ PS6 , @Xqt says "John Vandenberg: Please give try to do pip install with py < 2.7.9 in Windows."

Which version are you using? I can test on any version which you are experiencing problems. The Appveyor builds (linked in previous comment; https://ci.appveyor.com/project/jayvdb/pywikibot-core/build/1.0.fix-appveyor.239)) include 2.7.8 .

Xqt added a comment.Jun 4 2015, 8:15 AM
C:\pwb\core>pip install requests
Der Befehl "pip" ist entweder falsch geschrieben oder
konnte nicht gefunden werden.

C:\pwb\core>

with py 2.7.3

XZise added a comment.Jun 4 2015, 10:02 AM

Which basically means that the command pip couldn't be found. So the ONLY problem is actually that requests isn't packaged together with pywikibot-core? Couldn't we make it similar to how we packaged httplib2 and use requests as a submodule? Alternatively mention how to to install requests in the README or so?

I agree it's highly needed that we add requests as a submodule for core. It makes life a lot easier for non-programmers who are using pywikibot

Betacommand added a comment.EditedJun 4 2015, 11:13 AM

If you're on windows PIP isnt added to the system path. Try:

cd /
cd  C:\Python27\Scripts
pip install requests
Xqt added a comment.EditedJun 4 2015, 5:06 PM

Thanks Betacommand but there isn't any pip.exe anywhere - it is part of py2.7.9 or newer.

Xqt added a comment.Jun 4 2015, 5:10 PM

Xzise, Ladsgroup maybe. But be aware all settings must be changed and documented for several access routes (git, tortoisegit, svn, tortoisesvn, nightly)

@Xqt try following https://pip.pypa.io/en/latest/installing.html makes adding and managing packages on windows so much easier

XZise added a comment.Jun 4 2015, 5:19 PM

Well if git submodules are working correctly then there should be no issue with adding requests as a submodule. Nightly is also using git itself so these three routes are covered. Now I know you had problems with svn when we wanted to separate compat's and core's i18n. But if git submodules are unusable via svn then using submodules at all (including httplib2) doesn't seem like a good idea because any changes to httplib2 won't be visible to svn users.

jayvdb added a comment.Jun 5 2015, 1:51 AM

The svn / submodules problem is that it relies on the user to update the submodules, including importing the submodules stored in .svnprops.
This should be fixable by using git subtree instead of git submodule .
However we would need to test what SVN does if the user has set up svn externals , then we change it from a submodule to a subtree. Does svn automatically adjust?

jayvdb added a comment.Jun 5 2015, 1:57 AM

git subtrees also allow symlinks into the sub repo , which would mean the nasty voodoo in externals/httplib/__init__.py wouldnt be necessary. see https://gerrit.wikimedia.org/r/#/c/203794/1/tests/README.rst,cm for example

We are building a stable branch '2.0' which bot operators can use for production bots which maintains backwards compatibility and does not introduce new mandatory dependencies.
https://github.com/wikimedia/pywikibot-core/tree/2.0

jayvdb updated the task description. (Show Details)Jun 6 2015, 6:57 AM
jayvdb set Security to None.
Xqt added a subscriber: Kolega2357.Jun 8 2015, 5:55 PM

I have a problem with httplib2 for pywikipedia. See picture http://prntscr.com/7ehyy2

XZise added a comment.Jun 8 2015, 9:51 PM

Did you try cloning recursively or updating the submodules? Are you using the 2.0 branch?

No. I dont use 2.0 branch.

XZise added a comment.Jun 8 2015, 10:13 PM

Okay either use a recent version which uses requests or use the 2.0 branch and retest it. If the error still occurs could you paste the output of 'python pwb.py version'? And where did you get the version from? Also you can copy the contents into text and not an image (which would allow others to search for it) by right-clicking and then selecting “Select” and drawing an rectangle to what you want to copy and after that pressing enter to actually copy it.

C:\Kolega2357core>python pwb.py login
ImportError: No module named httplib2
Python module httplib2 >= 0.6.0 is required.
Did you clone without --recursive?
Try running 'git submodule update --init' or 'pip install httplib2'.

XZise added a comment.Jun 8 2015, 10:43 PM

Okay where did you get it?

Which version of Python are you using? (could you run python --version)
Does it have pip installed? (try pip --version)
If you are using Python 2.6.8 or lower, I'd suggest upgrading to Python 2.7.10 , which includes pip.

Once you have pip, you can run

pip install httplib2

I use Python 2.7.6

C:\Kolega2357core>pip --version
'pip' is not recognized as an internal or external command,
operable program or batch file.

C:\Kolega2357core>pip install httplib2
'pip' is not recognized as an internal or external command,
operable program or batch file.

jayvdb added a comment.Jun 9 2015, 1:02 AM

Is it possible for you to upgrade to Python 2.7.10 ? https://www.python.org/ftp/python/2.7.10/python-2.7.10.msi

I use 64 bit OS

Change 215603 abandoned by Xqt:
[FEAT] Basic httplib2 support

Reason:
requests looks good with some fixes

https://gerrit.wikimedia.org/r/215603

Xqt lowered the priority of this task from High to Low.Mar 18 2016, 2:58 PM
Xqt removed a project: Patch-For-Review.
Xqt closed this task as Invalid.May 28 2017, 2:09 PM

release 3.0 is published and requests is mandatory now.