The pywikibot master tree is frequently red for long periods of time for a variety of reasons. As a result these should be run using either the pywikibot 2.0 pre releases obtained from pip, or a new tag which would be updated periodically only when the pywikibot tree is green.

@jayvdb have you implemented a family for the beta cluster?

We will need the exact command to execute so we can wrap them in a Jenkins job then run it daily.

@hashar , this task is only waiting on the blocker task T69931 , which is a Beta-Cluster bug. The tests against betawiki are failing because of that bug. (On the sister task T100796 are links to the latest builds against betawiki with errors)

Pywikibot does not have a family class for the beta cluster, as our family class generator fails due to T69931, so needs to be generated for each build . (However, Pywikibot now has an AutoFamily class, which is able to be used instead)

Commands to run a test build against enwp.beta without logged in user tests:

python -m generate_family_file 'http://en.wikipedia.beta.wmflabs.org/' 'wpbeta' 'y' 
python -m generate_user_files -dir:~/.pywikibot/ -family:wpbeta -lang:en -v
nosetests

To add logged in user tests, the password file needs to be created and added to user-config.py, which requires a bit of shell tomfoolery. I can give precise instructions for that too if necessary, when we're closer to actually doing this.

Change 220384 had a related patch set uploaded (by Legoktm):
[WIP] Add job to run pywikibot tests against beta cluster

https://gerrit.wikimedia.org/r/220384

https://integration.wikimedia.org/ci/view/Default/job/pywikibot-tests-beta-cluster/1/consoleFull is an example run of the above job.

The from_url error was a case of T85658 which I had not fixed properly, and now has a patch up for review which works with beta wiki (https://gerrit.wikimedia.org/r/#/c/220722/ ).

Waiting for betawiki's IWM to be fixed was taking too long...so the Pywikibot tests are now skipping the two tests which rely on a sane IWM.
T75906: Check/update dumpInterwiki for beta usage is no longer a blocker. T100796: Run pywikibot tests on beta wikis is resolved.

Change 220384 merged by jenkins-bot:
Add job to run pywikibot tests against beta cluster

https://gerrit.wikimedia.org/r/220384

I configured the job to run "@midnight" which jenkins interpreted as 2:29AM, but okay...

Lets see if and how flaky this is over the next few days and then set up some sort of notifications upon failure.

https://integration.wikimedia.org/ci/job/pywikibot-tests-beta-cluster/ seems to be running fine right now. If there is any interest, we could get the test suite to generate a JUnit report file which can be interpreted by Jenkins.

In T100903#1566414, @hashar wrote:

https://integration.wikimedia.org/ci/job/pywikibot-tests-beta-cluster/ seems to be running fine right now.

Another few days - still looking good.

If there is any interest, we could get the test suite to generate a JUnit report file which can be interpreted by Jenkins.

I believe that is as simple as using nosetests --with-xunit.

As Pywikibot-OAuth is finished successfully, the test account could be an oauth user, which means user tests can be run while side stepping the security problems of T100797: Set up Pywikibot account on beta sites to run user tests.

Bah. this isnt working now.
The logs show Ran 0 tests in 0.013s.

https://integration.wikimedia.org/ci/job/pywikibot-tests-beta-cluster/11/console - OK
..
https://integration.wikimedia.org/ci/job/pywikibot-tests-beta-cluster/10/console - OK
https://integration.wikimedia.org/ci/job/pywikibot-tests-beta-cluster/21/console - NOT OK.

build 21 included:

1. 3e8555de (tox.ini : https://gerrit.wikimedia.org/r/#/c/233680/)
2. 0f427c7df (ui_tests)
3. 507c860 (translation update)

The tox.ini update is likely to be the culprit, but I am struggling to reproduce the problem.

I asked a similar question on a related bug I think and my confusion was clarified, so trying that again ;)

Is there a Beta-Cluster task within this task? The title/description read to me like it is really a CI-only task (ie: the work that needs to happen is in CI-config, and maybe in pywikibot).

If there is any Beta-Cluster specific task (eg, one of the two (closed) blockers listed), please do break it out as it's own (sub)task. (And I'll remove the Beta-Cluster project from this task.)

It does not run any test!

+ tox -e venv -- python setup.py nosetests --tests tests --verbosity=2 -a '"family=wpbeta,code=en"'
...

----------------------------------------------------------------------
Ran 0 tests in 0.026s

OK
___________________________________ summary ____________________________________
  venv: commands succeeded

I reproduce it on my local machine using a fresh pywikibot/core repo:

export PYWIKIBOT2_DIR=`pwd`
tox -e venv -- python -m generate_family_file 'http://en.wikipedia.beta.wmflabs.org/' 'wpbeta' 'y'
tox -e venv -- python -m generate_user_files -dir:`pwd` -lang:en -v
echo "console_encoding='utf8'" >> `pwd`/user-config.py
tox -e venv -- pip install -rdev-requirements.txt
tox -e venv -- python setup.py nosetests --tests tests --verbosity=2 -a '"family=wpbeta,code=en"'

Seems none of the tests match family=wpbeta at all, can be checked by passing to nosetests --collect-only:

tox -e venv -- python setup.py nosetests --tests tests --verbosity=2 --collect-only -a family=wpbeta

0 tests

With -a code=en there are 710 tests found.

Instead of wpbeta maybe we can use wikipedia? That would gives us 682 tests.

With -a code=en,family=wikipedia:

Ran 679 tests in 445.510s
FAILED (SKIP=7, failures=2)

Failures output being:

======================================================================
FAIL: Test formatting interwiki links using Page instances.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/hashar/projects/pywikibot/core/tests/textlib_tests.py", line 139, in test_interwiki_format_Link
    textlib.interwikiFormat(interwikis, self.site))
AssertionError: u'[[de:German]]\n[[fr:French]]\n' != u'[[wpbeta:de:German]]\n[[wpbeta:fr:French]]\n'
- [[de:German]]
+ [[wpbeta:de:German]]
?   +++++++
- [[fr:French]]
+ [[wpbeta:fr:French]]
?   +++++++

-------------------- >> begin captured logging << --------------------
pywiki: VERBOSE: Found candidate wpbeta:de
pywiki: VERBOSE: Found 1 wpbeta:de processes running, including this one.
requests.packages.urllib3.connectionpool: INFO: Starting new HTTPS connection (1): de.wikipedia.org
requests.packages.urllib3.connectionpool: DEBUG: "GET /w/api.php?meta=userinfo%7Csiteinfo&format=json&continue=&uiprop=blockinfo%7Chasmsg&siprop=namespaces%7Cnamespacealiases%7Cgeneral&maxlag=5&action=query HTTP/1.1" 200 1514
pywiki: VERBOSE: Found candidate wikipedia:de
pywiki: WARNING: Found multiple matches for URL "https://de.wikipedia.org/wiki/$1": ('de', Family("wpbeta")), (u'de', Family("wikipedia")) (use first)
pywiki: VERBOSE: Found candidate wpbeta:fr
pywiki: VERBOSE: Found 1 wpbeta:fr processes running, including this one.
requests.packages.urllib3.connectionpool: INFO: Starting new HTTPS connection (1): fr.wikipedia.org
requests.packages.urllib3.connectionpool: DEBUG: "GET /w/api.php?meta=userinfo%7Csiteinfo&format=json&continue=&uiprop=blockinfo%7Chasmsg&siprop=namespaces%7Cnamespacealiases%7Cgeneral&maxlag=5&action=query HTTP/1.1" 200 1674
pywiki: VERBOSE: Found candidate wikipedia:fr
pywiki: WARNING: Found multiple matches for URL "https://fr.wikipedia.org/wiki/$1": ('fr', Family("wpbeta")), (u'fr', Family("wikipedia")) (use first)
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: Test formatting interwiki links using Page instances.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/hashar/projects/pywikibot/core/tests/textlib_tests.py", line 129, in test_interwiki_format_Page
    textlib.interwikiFormat(interwikis, self.site))
AssertionError: u'[[de:German]]\n[[fr:French]]\n' != u'[[wpbeta:de:German]]\n[[wpbeta:fr:French]]\n'
- [[de:German]]
+ [[wpbeta:de:German]]
?   +++++++
- [[fr:French]]
+ [[wpbeta:fr:French]]
?   +++++++


----------------------------------------------------------------------

Does this also manage to run the wikibase tests on wikidata.beta.wmflabs.org or is it limited to just the wikipedia family?

The tests are tagged with familly = wikipedia but the Jenkins job runs them with family=wpbeta and hence no tests are found.

As a hack I replaced the 'wikipedia' familly with the generate one for 'wpbeta'

$ mv pywikibot/families/wpbeta_family.py pywikibot/families/wikipedia_family.py

Then running tests tagged family=wikipedia,code=en using:

PYWIKIBOT2_DIR=$(pwd) TOX_TESTENV_PASSENV=PYWIKIBOT2_DIR tox -e venv -- python setup.py nosetests --tests tests --verbosity=2 -a '"family=wikipedia,code=en"'

It has a bunch of failures though:

----------------------------------------------------------------------
Ran 113 tests in 10.778s

FAILED (SKIP=3, errors=5, failures=4)

Probably the tests should be retagged then? (I mean the apropriate tags should be added then)

Or what is needed to do here?