Page MenuHomePhabricator

Run pywikibot test suite regularly on beta cluster as part of MediaWiki/Wikimedia CI
Open, NormalPublic

Description

In a wikitech-l thread, it was suggested that pywikibots test suite should be regularly run (on WMF infrastructure) against changes which will be deployed to Wikimedia production wikis. i.e. as part of the MediaWiki/Wikimedia CI.

http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/82753

A separate task (T100796) exists for running the pywikibot test suite against Wikimedia beta wikis using Travis-CI as part of the pywikibot CI. The blockers for that task are not exactly the same, but are similar. e.g. the account 'Pywikibot-test' does not need to be used by Wikimedia CI - it is an artefact of the Travis CI/GitHub system.

Running these tests will hopefully assist with T108322: prevent modules with broken paraminfo being deployed to production wikis.

Test framework is running at https://integration.wikimedia.org/ci/job/pywikibot-tests-beta-cluster/ ; however 0 tests are run due to a tox problem.

tl;dr: Test Pywikibot regularly against MediaWiki upcoming changes using Wikimedia-CI

Related Objects

Event Timeline

jayvdb created this task.May 30 2015, 5:20 PM
jayvdb raised the priority of this task from to Needs Triage.
jayvdb updated the task description. (Show Details)
jayvdb added subscribers: jayvdb, greg.
Restricted Application added subscribers: Aklapper, Unknown Object (MLST). · View Herald TranscriptMay 30 2015, 5:20 PM

The pywikibot master tree is frequently red for long periods of time for a variety of reasons. As a result these should be run using either the pywikibot 2.0 pre releases obtained from pip, or a new tag which would be updated periodically only when the pywikibot tree is green.

mmodell triaged this task as Normal priority.Jun 1 2015, 6:57 PM
mmodell moved this task from To Triage to Backlog on the Beta-Cluster-Infrastructure board.
mmodell added a subscriber: mmodell.
hashar added a subscriber: hashar.

@jayvdb have you implemented a family for the beta cluster?

We will need the exact command to execute so we can wrap them in a Jenkins job then run it daily.

jayvdb added a comment.Jun 2 2015, 2:45 PM

@hashar , this task is only waiting on the blocker task T69931 , which is a Beta-Cluster bug. The tests against betawiki are failing because of that bug. (On the sister task T100796 are links to the latest builds against betawiki with errors)

Pywikibot does not have a family class for the beta cluster, as our family class generator fails due to T69931, so needs to be generated for each build . (However, Pywikibot now has an AutoFamily class, which is able to be used instead)

Commands to run a test build against enwp.beta without logged in user tests:

python -m generate_family_file 'http://en.wikipedia.beta.wmflabs.org/' 'wpbeta' 'y' 
python -m generate_user_files -dir:~/.pywikibot/ -family:wpbeta -lang:en -v
nosetests

To add logged in user tests, the password file needs to be created and added to user-config.py, which requires a bit of shell tomfoolery. I can give precise instructions for that too if necessary, when we're closer to actually doing this.

Ricordisamoa added a subscriber: Ricordisamoa.

Change 220384 had a related patch set uploaded (by Legoktm):
[WIP] Add job to run pywikibot tests against beta cluster

https://gerrit.wikimedia.org/r/220384

The from_url error was a case of T85658 which I had not fixed properly, and now has a patch up for review which works with beta wiki (https://gerrit.wikimedia.org/r/#/c/220722/ ).

Restricted Application added a subscriber: Luke081515. · View Herald TranscriptJul 15 2015, 11:25 PM

Waiting for betawiki's IWM to be fixed was taking too long...so the Pywikibot tests are now skipping the two tests which rely on a sane IWM.
T75906: Check/update dumpInterwiki for beta usage is no longer a blocker. T100796: Run pywikibot tests on beta wikis is resolved.

Change 220384 merged by jenkins-bot:
Add job to run pywikibot tests against beta cluster

https://gerrit.wikimedia.org/r/220384

I configured the job to run "@midnight" which jenkins interpreted as 2:29AM, but okay...

Lets see if and how flaky this is over the next few days and then set up some sort of notifications upon failure.

jayvdb updated the task description. (Show Details)Aug 22 2015, 9:34 AM
jayvdb set Security to None.

https://integration.wikimedia.org/ci/job/pywikibot-tests-beta-cluster/ seems to be running fine right now. If there is any interest, we could get the test suite to generate a JUnit report file which can be interpreted by Jenkins.

Another few days - still looking good.

If there is any interest, we could get the test suite to generate a JUnit report file which can be interpreted by Jenkins.

I believe that is as simple as using nosetests --with-xunit.

As Pywikibot-OAuth is finished successfully, the test account could be an oauth user, which means user tests can be run while side stepping the security problems of T100797: Set up Pywikibot account on beta sites to run user tests.

jayvdb added a comment.Sep 3 2015, 2:50 AM

Bah. this isnt working now.
The logs show Ran 0 tests in 0.013s.

https://integration.wikimedia.org/ci/job/pywikibot-tests-beta-cluster/11/console - OK
..
https://integration.wikimedia.org/ci/job/pywikibot-tests-beta-cluster/10/console - OK
https://integration.wikimedia.org/ci/job/pywikibot-tests-beta-cluster/21/console - NOT OK.

build 21 included:

  1. 3e8555de (tox.ini : https://gerrit.wikimedia.org/r/#/c/233680/)
  2. 0f427c7df (ui_tests)
  3. 507c860 (translation update)

The tox.ini update is likely to be the culprit, but I am struggling to reproduce the problem.

jayvdb updated the task description. (Show Details)Sep 27 2015, 11:18 PM
greg added a comment.Sep 28 2015, 3:58 PM

I asked a similar question on a related bug I think and my confusion was clarified, so trying that again ;)

Is there a Beta-Cluster task within this task? The title/description read to me like it is really a CI-only task (ie: the work that needs to happen is in CI-config, and maybe in pywikibot).

If there is any Beta-Cluster specific task (eg, one of the two (closed) blockers listed), please do break it out as it's own (sub)task. (And I'll remove the Beta-Cluster project from this task.)

hashar added a comment.EditedNov 2 2016, 8:33 PM

It does not run any test!

+ tox -e venv -- python setup.py nosetests --tests tests --verbosity=2 -a '"family=wpbeta,code=en"'
...

----------------------------------------------------------------------
Ran 0 tests in 0.026s

OK
___________________________________ summary ____________________________________
  venv: commands succeeded

I reproduce it on my local machine using a fresh pywikibot/core repo:

export PYWIKIBOT2_DIR=`pwd`
tox -e venv -- python -m generate_family_file 'http://en.wikipedia.beta.wmflabs.org/' 'wpbeta' 'y'
tox -e venv -- python -m generate_user_files -dir:`pwd` -lang:en -v
echo "console_encoding='utf8'" >> `pwd`/user-config.py
tox -e venv -- pip install -rdev-requirements.txt
tox -e venv -- python setup.py nosetests --tests tests --verbosity=2 -a '"family=wpbeta,code=en"'

Seems none of the tests match family=wpbeta at all, can be checked by passing to nosetests --collect-only:

tox -e venv -- python setup.py nosetests --tests tests --verbosity=2 --collect-only -a family=wpbeta

0 tests

With -a code=en there are 710 tests found.

Instead of wpbeta maybe we can use wikipedia? That would gives us 682 tests.

hashar added a comment.Nov 2 2016, 8:54 PM

With -a code=en,family=wikipedia:

Ran 679 tests in 445.510s
FAILED (SKIP=7, failures=2)

Failures output being:

======================================================================
FAIL: Test formatting interwiki links using Page instances.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/hashar/projects/pywikibot/core/tests/textlib_tests.py", line 139, in test_interwiki_format_Link
    textlib.interwikiFormat(interwikis, self.site))
AssertionError: u'[[de:German]]\n[[fr:French]]\n' != u'[[wpbeta:de:German]]\n[[wpbeta:fr:French]]\n'
- [[de:German]]
+ [[wpbeta:de:German]]
?   +++++++
- [[fr:French]]
+ [[wpbeta:fr:French]]
?   +++++++

-------------------- >> begin captured logging << --------------------
pywiki: VERBOSE: Found candidate wpbeta:de
pywiki: VERBOSE: Found 1 wpbeta:de processes running, including this one.
requests.packages.urllib3.connectionpool: INFO: Starting new HTTPS connection (1): de.wikipedia.org
requests.packages.urllib3.connectionpool: DEBUG: "GET /w/api.php?meta=userinfo%7Csiteinfo&format=json&continue=&uiprop=blockinfo%7Chasmsg&siprop=namespaces%7Cnamespacealiases%7Cgeneral&maxlag=5&action=query HTTP/1.1" 200 1514
pywiki: VERBOSE: Found candidate wikipedia:de
pywiki: WARNING: Found multiple matches for URL "https://de.wikipedia.org/wiki/$1": ('de', Family("wpbeta")), (u'de', Family("wikipedia")) (use first)
pywiki: VERBOSE: Found candidate wpbeta:fr
pywiki: VERBOSE: Found 1 wpbeta:fr processes running, including this one.
requests.packages.urllib3.connectionpool: INFO: Starting new HTTPS connection (1): fr.wikipedia.org
requests.packages.urllib3.connectionpool: DEBUG: "GET /w/api.php?meta=userinfo%7Csiteinfo&format=json&continue=&uiprop=blockinfo%7Chasmsg&siprop=namespaces%7Cnamespacealiases%7Cgeneral&maxlag=5&action=query HTTP/1.1" 200 1674
pywiki: VERBOSE: Found candidate wikipedia:fr
pywiki: WARNING: Found multiple matches for URL "https://fr.wikipedia.org/wiki/$1": ('fr', Family("wpbeta")), (u'fr', Family("wikipedia")) (use first)
--------------------- >> end captured logging << ---------------------
======================================================================
FAIL: Test formatting interwiki links using Page instances.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/hashar/projects/pywikibot/core/tests/textlib_tests.py", line 129, in test_interwiki_format_Page
    textlib.interwikiFormat(interwikis, self.site))
AssertionError: u'[[de:German]]\n[[fr:French]]\n' != u'[[wpbeta:de:German]]\n[[wpbeta:fr:French]]\n'
- [[de:German]]
+ [[wpbeta:de:German]]
?   +++++++
- [[fr:French]]
+ [[wpbeta:fr:French]]
?   +++++++


----------------------------------------------------------------------

Does this also manage to run the wikibase tests on wikidata.beta.wmflabs.org or is it limited to just the wikipedia family?

The tests are tagged with familly = wikipedia but the Jenkins job runs them with family=wpbeta and hence no tests are found.

As a hack I replaced the 'wikipedia' familly with the generate one for 'wpbeta'

$ mv pywikibot/families/wpbeta_family.py pywikibot/families/wikipedia_family.py

Then running tests tagged family=wikipedia,code=en using:

PYWIKIBOT2_DIR=$(pwd) TOX_TESTENV_PASSENV=PYWIKIBOT2_DIR tox -e venv -- python setup.py nosetests --tests tests --verbosity=2 -a '"family=wikipedia,code=en"'

It has a bunch of failures though:

----------------------------------------------------------------------
Ran 113 tests in 10.778s

FAILED (SKIP=3, errors=5, failures=4)
Dvorapa added a subscriber: Dvorapa.EditedJun 9 2018, 12:53 PM

Probably the tests should be retagged then? (I mean the apropriate tags should be added then)

Or what is needed to do here?

Dvorapa updated the task description. (Show Details)Feb 23 2019, 10:23 PM
Dvorapa updated the task description. (Show Details)Feb 23 2019, 10:28 PM