Page MenuHomePhabricator

Travis jobs with OAuth wait for password input and get terminated
Closed, ResolvedPublic

Description

https://travis-ci.org/wikimedia/pywikibot-core/jobs/265424635#L4408
https://travis-ci.org/wikimedia/pywikibot-core/jobs/265424634#L1599

tests/site_tests.py::TestUsernameInUsers::test_username_in_users Password for user Pywikibot-oauth on wpbeta:en (no characters will be shown): 

The job exceeded the maximum time limit for jobs, and has been terminated.

Job #20 (with OAuth on test.wikpedia.org) has more than 1000 of the following warning:

WARNING: /home/travis/build/wikimedia/pywikibot-core/pywikibot/comms/http.py:345: UserWarning: Invalid authentication tokens for test.wikipedia.org set in `config.authenticate`

The failure has appeared after 4bb76aa, but I can't find any relation between that commit and the failure.
Also it only has affected jobs #15, #16, and #20. These jobs have OAUTH_DOMAIN set in their enviroment variable.

Event Timeline

Dalba updated the task description. (Show Details)
Dalba renamed this task from test_username_in_users is breaking travis jobs on wpbeta:zh and wpbeta:en by waiting for password to test_username_in_users is breaking travis jobs by waiting for password when OAuth is activated.Aug 17 2017, 2:02 PM
Dalba renamed this task from test_username_in_users is breaking travis jobs by waiting for password when OAuth is activated to Travis jobs with OAuth wait for password input and get terminated.Aug 18 2017, 1:21 AM
Dalba updated the task description. (Show Details)

That warning is shown [[ https://phabricator.wikimedia.org/diffusion/PWBC/browse/master/pywikibot/comms/http.py;d2af1bab80d49f99fe6a12fefb1e816ad1bea1b7$341 | only if len(config.authenticate[path]) not in [2, 4] ]].

It would be helpful if someone with access to travis enviroment could check that environment variable OAUTH_TOKENS_WIKIPEDIA_TEST has correct format and values.

Change 372499 had a related patch set uploaded (by Dalba; owner: Dalba):
[pywikibot/core@master] http.py: Show a more descriptive warning about invalid authentication tokens

https://gerrit.wikimedia.org/r/372499

Xqt triaged this task as High priority.Aug 18 2017, 6:29 AM

Change 372499 merged by jenkins-bot:
[pywikibot/core@master] http.py: Show a more descriptive warning about invalid authentication tokens

https://gerrit.wikimedia.org/r/372499

Xqt claimed this task.

The last patch wasn't really supposed to resolve this issue, it'll just show a clearer warning:

https://travis-ci.org/wikimedia/pywikibot-core/jobs/266262014#L6341

WARNING: /home/travis/build/wikimedia/pywikibot-core/pywikibot/comms/http.py:347: UserWarning: config.authenticate["zh.wikipedia.beta.wmflabs.org"] has invalid value.

It should contain 2 or 4 items, not 150.

See https://www.mediawiki.org/wiki/Manual:Pywikibot/OAuth for more info.

  .format(path=path, length=len(config.authenticate[path])))

Now we need to find out where do these 150 items are coming from.

Only two possibilities come to my mind, either someone has changed the travis enviroment variable, or we are mutating config.authenticate somewhere during the tests.

Dalba removed Xqt as the assignee of this task.Aug 19 2017, 3:40 PM
Dalba added a subscriber: Xqt.

Change 372778 had a related patch set uploaded (by Dalba; owner: Dalba):
[pywikibot/core@master] travis.yml: Print the length and type of config.authenticate values

https://gerrit.wikimedia.org/r/372778

Change 372778 merged by jenkins-bot:
[pywikibot/core@master] travis.yml: Print the length and type of config.authenticate values

https://gerrit.wikimedia.org/r/372778

The results of the above patch:

https://travis-ci.org/wikimedia/pywikibot-core/jobs/266461986#L1059

$ python -c "from __future__ import print_function;from pywikibot import config as c; a=c.authenticate; [(print('authenticate_key:', i), print('authenticate_value_type:', type(a[i])), print('authenticate_value_length:', len(a[i]))) for i in a]"

authenticate_key: en.wikipedia.beta.wmflabs.org

authenticate_value_type: <type 'unicode'>

authenticate_value_length: 150

authenticate_key: wiki.musicbrainz.org

authenticate_value_type: <type 'tuple'>

authenticate_value_length: 2

https://travis-ci.org/wikimedia/pywikibot-core/jobs/266461987

$ python -c "from __future__ import print_function;from pywikibot import config as c; a=c.authenticate; [(print('authenticate_key:', i), print('authenticate_value_type:', type(a[i])), print('authenticate_value_length:', len(a[i]))) for i in a]"

authenticate_key: wiki.musicbrainz.org

authenticate_value_type: <class 'tuple'>

authenticate_value_length: 2

authenticate_key: zh.wikipedia.beta.wmflabs.org

authenticate_value_type: <class 'str'>

authenticate_value_length: 150

This is immediately after creation of the config file.

Therefore I think the value of OAUTH_TOKENS_WIKIPEDIA_TEST has been changed and has the wrong format.

I don't know who has access to travis enviroment. I'm going to add Diffusion-Repository-Administrators, I hope they can help us by checking pywikibot's enviroment variables on travis-ci.

Specifically the value of the following variables needs to be checked:

  • OAUTH_TOKENS_WIKIPEDIA_TEST
  • OAUTH_TOKENS_WPBETA_EN
  • OAUTH_TOKENS_WPBETA_ZH

AFAIU the code, the expected format for them is:

consumer_token:consumer_secret:access_token:access_secret

I don't know how exactly travis is configured -- I vaguely remember @jayvdb set this up (T104306 is vaguely related).

Given that the Travis account is 'https://travis-ci.org/wikimedia/pywikibot-core', I suspect anyone with access to the wikimedia github account can also access the travis account (probably uses github oauth?).

The password for the user account is stored on toolforge (as mentioned in T100797#1321387). Anyone with access to the pywikibot toolforge account can read the contents of that file.

I propose to deactivate all tests which fail for that reason until we can reconfigure it and we can get them work again. Would be great to get the remaining tests passing.

I propose to deactivate all tests which fail for that reason until we can reconfigure it and we can get them work again. Would be great to get the remaining tests passing.

There may be lots of them. Looking at the log many of the other tests have ERRORed as well up the point where a password input is prompted, including most/all of the script_tests, some of the pagegenerators_tests, some of the site_tests, page_tests... We might as well completely deactivate OAuth jobs.

Change 376701 had a related patch set uploaded (by Dalba; owner: Dalba):
[pywikibot/core@master] travis.yml: Disable OAuth jobs

https://gerrit.wikimedia.org/r/376701

Change 376701 merged by jenkins-bot:
[pywikibot/core@master] travis.yml: Disable OAuth jobs

https://gerrit.wikimedia.org/r/376701

What is needed to do here?

Someone with access to Pywikibot-test account and Travis-CI environment should reset Travis environment variables for OAuth, then run the tests on a test branch to see if it fixes the issue.

If fixed, re-enable the test on the master branch, it not, more debugging will be needed. But I'm very hopeful that resetting OAuth tokens will fix it.

Access to Pywikibot-test account is written in the Pywikibot Toolforge project and access to Travis-CI was given to me and Xqt in T196810.

This comment was removed by Dvorapa.
This comment was removed by Dvorapa.

I confirm that I can log me in successfully using oauths creds of OAUTH_TOKENS_WIKIPEDIA_TEST found in passwd file.

Could you also try both OAUTH_TOKENS_WPBETA_**?

I fixed the keys on Travis (from passwd file). They should be in the form of key:key:key:key. Now we should test them using test branch as Dalba suggested (how?).

Could you also try both OAUTH_TOKENS_WPBETA_**?

I can also successfully log me in on beta:en and beta:zh with the other tokens.

I fixed the keys on Travis (from passwd file). They should be in the form of key:key:key:key. Now we should test them using test branch as Dalba suggested (how?).

By test branch, I meant pushing to a test repository branch on Github, integrated with Travis, to see if the newly configured credentials work as expected. We can skip that and directly re-enable the disabled jobs on master branch (and revert if the problem still exists).

Change 443545 had a related patch set uploaded (by Dalba; owner: dalba):
[pywikibot/core@master] Re-enable OAuth tests

https://gerrit.wikimedia.org/r/443545

Change 443545 merged by jenkins-bot:
[pywikibot/core@master] Re-enable OAuth tests

https://gerrit.wikimedia.org/r/443545

The error is still there, but I think I've figured out the wrong part.

Change 443581 had a related patch set uploaded (by Dalba; owner: dalba):
[pywikibot/core@master] .travis.yml: Escape single quotes when printing oauth_token_var

https://gerrit.wikimedia.org/r/443581

Change 443581 merged by jenkins-bot:
[pywikibot/core@master] .travis.yml: Escape single quotes when printing oauth_token_var

https://gerrit.wikimedia.org/r/443581

Multiple wbbeta specific errors occured (no new pages, no af language, main talk page is redirected), but no OAuth failure and the type is finally a tuple, so hopefully this is close to be solved.

travis still fails after the last patch

travis still fails after the last patch

As @Dvorapa mentioned earlier, the failures are not OAuth errors, we should fix them in separate tasks.

The only thing remaining here is to revert 60c301360f44d185a28a69094c7714b1fa14e01a.

Change 443635 had a related patch set uploaded (by Dvorapa; owner: Dvorapa):
[pywikibot/core@master] Revert "travis.yml: Print the length and type of config.authenticate values"

https://gerrit.wikimedia.org/r/443635

Change 443635 merged by jenkins-bot:
[pywikibot/core@master] Revert "travis.yml: Print the length and type of config.authenticate values"

https://gerrit.wikimedia.org/r/443635