Page MenuHomePhabricator

login() with retry=True might delete correct password if unknown APIError occurs
Closed, InvalidPublicFeature

Description

To use pywikibot with a farm of wikis i try to set the login password via software. I am getting the password from an encrypted source.

The line

config2.authenticate[self.netloc] = (self.user,self.getPassword())

didn't seem to have any effect in my adapter code so i started debugging the login process. It looks like LoginManager has a password field which is not used in site.php:

class LoginManager(object):

    """Site login manager."""

    @deprecated_args(username='user', verbose=None)
    def __init__(self, password=None, sysop=False, site=None, user=None):
login_manager = api.LoginManager(
            site=self, sysop=sysop, user=self._username[sysop])

The login function in LoginManager does not have a password parameter at all. The current code structure makes it really awkward to supply a password via software.

if not self.password:
            # First check that the username exists,
            # to avoid asking for a password that will not work.
            if not autocreate:
                self.check_user_exists()

            # As we don't want the password to appear on the screen, we set
            # password = True
            self.password = pywikibot.input(

It would be great if the logic would be changed to making a password available via api would be made simpler e.g. by having password=None as an optional parameter for login.

Event Timeline

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptMar 25 2020, 1:53 PM
Xqt triaged this task as Low priority.Mar 25 2020, 2:11 PM
Xqt changed the subtype of this task from "Task" to "Feature Request".
Xqt added a subscriber: Mpaa.

I am currently experimenting with this patch:

RCS file: RCS/site.py,v
retrieving revision 1.1
diff -r1.1 site.py
2017c2017
<     def login(self, sysop=False, autocreate=False):
---
>     def login(self, sysop=False, autocreate=False,password=None):
2088c2088
<             site=self, sysop=sysop, user=self._username[sysop])
---
>             site=self, sysop=sysop, user=self._username[sysop],password=password)

would there be more to it than these two lines?

The patch seems to work for all 45 of my sites. It would be great if it would make it into the release.

Xqt added a subscriber: Xqt.Mar 25 2020, 5:44 PM

@Seppl2013: Could you please commit your patch to the Gerrit repository for reviewing. You may use "gerrit patch uploader" if you don't have a developer accout at gerrit, see https://www.mediawiki.org/wiki/Gerrit_patch_uploader

Since it's only two lines it looks very awkward to learn a new tool chain for this. WikiMedia still lives in its own development world so that it is hard for outsiders to contribute effectivly.

I am preparing a new python package at https://github.com/WolfgangFahl/py-3rdparty-mediawiki and would love to concentrate on this. This new package will depend on the patch above.

Since it's only two lines it looks very awkward to learn a new tool chain for this. WikiMedia still lives in its own development world so that it is hard for outsiders to contribute effectivly.

Gerrit patch uploader is simple web form, where you only upload .patch/.diff file and describe it and the tool makes a proper pull request itself for you. You don't have to learn anything from Wikimedia Gerrit infrastructure at all, you only upload one file and that's all.

Dvorapa added a comment.EditedMar 26 2020, 6:17 PM

@Seppl2013 Have you tried password file? From what I know authenticate is for server password, not for user password.

Please try and tell me if this helped to solve your issue.

@Dvorpa - i'd like to keep my passwords encrypted.

@Dvorpa - if you now how to handle the patch upload could you please do it for me - i don't know how to create a patchfile from an RCS diff.

Dvorapa added a comment.EditedMar 27 2020, 11:49 AM

@Seppl2013 I don't understand. What's the difference in encryption between user-config.py and passwordfile? Both are just text files and both accept the same values, only passwordfile is dedicated only to passwords and not to other settings. Also there is a pywikibot.lwp lwp storage for passwords, which is encrypted.

Anyway, I would like to make proper solution to this issue, not just a quick workaround you mention. What would you expect from Pywikibot or dream Pywikibot can do (in details, step by step)? We can make a proper functionality for your need.

Note: readPassword seems to be the method, that needs adjustments in this case or there should be created a new little brother/sister method to it.

From my point the proposed change is not a workaround. It's fine. The encryption in this case was motivated to be MediaWiki-Japi compatible which uses https://github.com/WolfgangFahl/py-3rdparty-mediawiki/blob/master/wikibot/crypt.py inspired by https://stackoverflow.com/questions/24168246/replicate-javas-pbewithmd5anddes-in-python-2-7 that's why it's important to be able to set the password via API and not from a configuration file. In fact https://test.pypi.org/project/py-3rdparty-mediawiki/0.0.2b0/ only uses

put_throttle = 0

as the content of user-config.py to simplify things as in mediawiki -japi down to

from wikibot.wikibot import WikiBot
wikibot=WikiBot.ofWikiId("test2")
wikibot.site ...

the necessary family files are generated and registered on the fly from the Mediawiki-Japi ini files.

https://github.com/WolfgangFahl/py-3rdparty-mediawiki/blob/master/wikibot/wikibot.py needs the patch at:

self.site.login(password=self.getPassword())

at this point. If you can supply a similar solution that's also fine.

Why don't you use LoginManager directly if you need to pass a password?

Seppl2013 added a comment.EditedMar 27 2020, 2:03 PM

How? I didn't see an example and proper doc for it and couldn't find a way to decipher the source code. Passing the password as the constructor suggests seems to be natural to me.

Dvorapa added a comment.EditedMar 27 2020, 2:14 PM

https://doc.wikimedia.org/pywikibot/master/api_ref/pywikibot.html?highlight=loginmanager#pywikibot.login.LoginManager

from pywikibot.api import LoginManager

lm = LoginManager(password='', site=pywikibot.Site('', ''), user='')
lm.login()

If site and/or user are missing, they are loaded from config
If password is missing, it is loaded from passwordfile

For regular passwords and botpasswords. If using OAuth, Similarly OauthLoginManager class is here to help.

Seppl2013 added a comment.EditedMar 27 2020, 2:24 PM

Tried it and it does not work as advertised. pywikbot.api does not seem to exist and for:

from pywikibot.login import LoginManager
lm = LoginManager(password=self.getPassword(), site=self.site, user=self.user)
lm.login()

the result is an error:

write() argument must be str, not None

from the middle of nowhere.

The patch looks so much simpler and i understand it ...

now my software is broken again and I need to repatch manually since pip uninstall removed the RCS directory ...

Seppl2013 added a comment.EditedMar 27 2020, 2:31 PM

It looks like site.py is gone now and code moved to site/__init__.py ... what is the motivation for this inorthodox coding style?

pip3 show pywikibot
Name: pywikibot
Version: 3.0.20200326

When i debug the code i end up in:

def getCookie(self):
       """
       Login to the site.

       @see: U{https://www.mediawiki.org/wiki/API:Login}

       @return: cookie data if successful, None otherwise.
       """
       # THIS IS OVERRIDDEN IN data/api.py
       return None

???

The new patch seems to be:

rcsdiff __init__.py 
===================================================================
RCS file: RCS/__init__.py,v
retrieving revision 1.1
diff -r1.1 __init__.py
2026c2026
<     def login(self, sysop=None, autocreate=False):
---
>     def login(self, sysop=None, autocreate=False,password=None):
2089c2089
<         login_manager = api.LoginManager(site=self, user=self.username())
---
>         login_manager = api.LoginManager(site=self, user=self.username(),password=password)

Things run again - only 4 x slower than with the 2019 version ...

I am trying to figure out how to create a patch file with https://cocoon.apache.org/2.0/howto/howto-patch.html - the unified diff there looks different than the rcsdiff patch above. What is needed for a proper patch file?

https://test.pypi.org/project/py-3rdparty-mediawiki/0.0.2rc0/ has the latest code that works with the patch. https://github.com/WolfgangFahl/py-3rdparty-mediawiki/blob/master/wikibot/wikibot.py#L135 has your proposal but commented it since it breaks the tests. You'll need to prepare at least one ini file with your credentials if you want to test your self so you might be the first to create the issue mentioned in http://mediawiki-japi.bitplan.com/index.php/Py-3rdparty-mediawiki#Features

Dvorapa added a comment.EditedMar 27 2020, 4:15 PM

Gerrit patch uploader: all steps to generate and upload patch are described:
https://tools.wmflabs.org/gerrit-patch-uploader/

pywikbot.api does not seem to exist and for:

Sorrry, my bad, it should be pywikibot.data.api, but this is easy to find out:

from pywikibot.data.api import LoginManager

lm = LoginManager(password='', site=pywikibot.Site('', ''), user='')
lm.login()

It looks like site.py is gone now and code moved to site/__init__.py ... what is the motivation for this inorthodox coding style?

8 500 lines file will be split into smaller files (the same for 6 000 lines page.py). Pretty useful orthodox style of coding. If you import site module properly like import pywikibot.site, nothing should break at all.

Thx the data.api LoginManager works. How do i get rid of the warning:

WARNING: API warning (login): Main-account login via action=login is deprecated and may stop working without warning. To continue login with action=login, see [[Special:BotPasswords]]. To safely continue using main-account login, see action=clientlogin.

I specifically want to make changes on behalf of main acount users.

Dvorapa added a comment.EditedMar 27 2020, 4:31 PM

This is a bug in Pywikibot if you use user password instead of bot password, see T137805. I am currently working on a fix for this.

You can avoid it by using bot password or OAuth authentication with Pywikibot, if you can.

Thx the data.api LoginManager works. How do i get rid of the warning:

WARNING: API warning (login): Main-account login via action=login is deprecated and may stop working without warning. To continue login with action=login, see [[Special:BotPasswords]]. To safely continue using main-account login, see action=clientlogin.

I specifically want to make changes on behalf of main acount users.

Resolved in master, will be in next release.

Very good. I think this task can be closed now.

Dvorapa closed this task as Invalid.Mar 28 2020, 12:31 PM

Anyway, feel free to ask about anything or suggest any other missing feature in Pywikibot you might need!

Thx. There is a followup problem. When trying to login to a Mediawiki 1.33.2 system I get:

WARNING: API error readapidenied: You need read permission to use this module.
WARNING: API error readapidenied: You need read permission to use this module.
ERROR: APIError: readapidenied: You need read permission to use this module.
[help: See http://.....bitplan.com/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce&gt; for notice of API deprecations and breaking changes.]
WARNING: API error readapidenied: You need read permission to use this module.
WARNING: API error readapidenied: You need read permission to use this module.
WARNING: Could not check user ... exists on ...:en
Password for user ... on ...:en (no characters will be shown): Warning: Password input may be echoed.

So there is a lot of activity the framework does on the API which still fails to deliver the login.

I'd prefer an API where login means login and not checking whether there is a user and other other stuff which eventually leads to not login in but asking for a password which was already supplied ...

Seppl2013 added a comment.EditedMar 30 2020, 10:42 AM

I suspect:

login_manager = api.LoginManager(site=self, user=self.username())
     if login_manager.login(retry=True, autocreate=autocreate):

as the culprit .. here it might be that the existing password seems to be forgotten again. And the getuserinfo has a force=True - why is that?

Dvorapa added a comment.EditedMar 30 2020, 10:51 AM

So there is a lot of activity the framework does on the API which still fails to deliver the login.

There is still a lot of bugs in Pywikibot login process. Nobody touched this for more than a year (or at least tests have failed due to login issues for 12 months). I'm working on it, watch T224712 for more details.

as the culprit .. here it might be that the existing password seems to be forgotten again.

Yes, it is cleared in one place where it shouldn't or should, but with some message indicating the real problem, I'll take care of it.

And the getuserinfo has a force=True - why is that?

This is needed to check actual login status, not cached one. Anyway readapidenied should correctly continue after this. I'll take a look into this too.

Seppl2013 added a comment.EditedMar 30 2020, 12:27 PM

Sigh - this is like a catch 22 see how https://phabricator.wikimedia.org/T248673 got closed while i am stuck with MW 1.27.3. So neither my old wiki nor my new wiki is useable in the process of semi-automatic migration ... and in the meantime my "new wiki" which i have been trying to migrate to for a year now is not new any more ...

After patch for T248768 will be merged and the task will be closed (if no issues occur), I will ask you for trying Pywikibot master with all latest patches to login. Your info about login issues helped me with debugging and I think with your response about how Pywikibot interacts with your private wiki we can fix both login issues and private wiki issues at once (hopefully)

This comment was removed by Dvorapa.
Seppl2013 added a comment.EditedMar 31 2020, 2:31 PM

Again i had to do a patch to make sure the password is not forgot with two lines:

login.py:

site.password=password

site.__init__.py:

self.password=None # in constructor

....

login_manager = api.LoginManager(site=self, user=self.username(), password=self.password)

i would laugh about how awkward the code is if it wasn't for how complicated things got in doing this open source multi developer multi -year patching style ...

If i supply:
mw_version
password
user
site

and what not why on earth would the API decide to fetch this via API - e.g. looking at a timeout value whether the version might have been changed. Boy i am telling the API the version to make sure it doesn't have to look it up ...

I always thought that is what API's are for - accepting commands and executing them not starting to think the developer who uses the API is probably not giving all the information and we doubly have to make sure the APIs fails while trying to fix the developers input ...

Wait for the patches that I made today. Hopefully this will work finally and no patch needs to be done.

Thx for the offer. I am fine as long as i have a workaround ... no need to hurry then.

Dvorapa added a comment.EditedMar 31 2020, 5:09 PM

Okay, current master seems no longer fail tests. Now please test and tell me, what to fix with login to private wikis.

Note to myself: T100965, T112578, T153891

password is deleted if there is ANY issue with login, which is not correct. But I hope all issues should be fixed now

Seppl2013 added a comment.EditedApr 1 2020, 10:05 AM

Could you please put a release on test.pypi.org so that I can install the master with pip ... or simply create a release candidate on normal pypi.org and put the link here.

https://pypi.org/project/pywikibot/ currently shows a release from March 27.

Dvorapa added a comment.EditedApr 1 2020, 12:30 PM

You can try https://test.pypi.org/project/pywikibot/ (but I created it in a rush, so no guarantee it will work properly)

Dvorapa added a comment.EditedApr 8 2020, 12:49 PM

Okay, if any issue with the new release occur, please create a new task with a new description of the problem. Many login issues were solved in the new release. The issue of retry=True deleting correct password if any login error might still occur, but we need to know more details about how it happens with the new release.

Dvorapa renamed this task from pywikibot password handling to login() with retry=True might delete correct password if unknown APIError occurs.Apr 8 2020, 12:54 PM

I have tried today with pywikibot>=3.0.20200508 and on my local machine the tests run. I am having some trouble in the travis environment of a project I am using things now - I'll create a separate issue for that.