Page MenuHomePhabricator

Tests: allow for offline operation
Closed, ResolvedPublic

Description

https://integration.wikimedia.org/ci/job/pywikibot-core-tests/186/consoleFull failed due to a lack of network connectivity. It would be great to first do a quick check if enwiki is online/accessible, and if not, to skip all network-requiring tests.


Version: core-(2.0)
Severity: normal

Details

Reference
bz56959

Event Timeline

bzimport raised the priority of this task from to Normal.Nov 22 2014, 2:33 AM
bzimport added a project: Pywikibot-General.
bzimport set Reference to bz56959.
bzimport added a subscriber: Unknown Object (????).

Change 154469 had a related patch set uploaded by John Vandenberg:
Test metaclass

https://gerrit.wikimedia.org/r/154469

With change 154469, the loading of each site is moved into setUpClass. Additional code will be needed to nicely skip the tests if the site couldnt be contacted, however that needs to be another changeset as the API is very unfriendly for this, as it wants to retry the connection many (12?) times by default before giving up.

Change 154469 merged by XZise:
Test metaclass

https://gerrit.wikimedia.org/r/154469

jayvdb set Security to None.
jayvdb moved this task from Backlog to Framework on the Pywikibot-tests board.
jayvdb removed a subscriber: Unknown Object (????).

I think this is basicly done.

nosetests can be run in !net mode, and with https://gerrit.wikimedia.org/r/#/c/177985/ for T58963 tests classes are skipped after config.socket_timeout per site , however I've not tested it while offline. TestMe

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 16 2015, 8:52 PM

After some time it deletes the log (afaik 1 month, ping @hashar) unfortunately.

Yup the build results are discarded automatically after 15 days. Usually one can "just" comment "recheck" to have them run again.

The pywikibot-core-tests job has been deleted though :( I think the idea was to run pywikibot against beta cluster or the enwiki live site. Since the bug has been filled the test suite has been greatly improved though.

Maybe we can repurpose it to create a subset of tests that we would run every X hours again wikipedia.org (read only) and beta cluster (read/write) ?

Yeah this CI bug can't really be tested without disabling the network from the CI infrastructure, right?

The job https://integration.wikimedia.org/ci/job/pywikibot-core-tox-nose/ runs tox -enose, what it runs is defined in /tox.ini and roughly ends up being:

python -W error::UserWarning -m generate_user_files -family:test -lang:test -v

nosetests --with-doctest --with-detecthttp -v -a '!net' tests pywikibot --ignore-files=(gui\.py|botirc\.py|rcstream\.py|api\.py)

Note -a '!net' which excludes any test labelled with net i.e. requiring network.

So I guess running tox -enose is offline operation.

Invoking simply tox will run python setup.py test which might well runs everything.

XZise added a comment.Dec 16 2015, 9:21 PM

There are two things to point out though: A test may not be labeled net when we screw up and there is actually a plugin for nose which prevents at least certain connections.

jayvdb closed this task as Resolved.Dec 17 2015, 4:32 AM
jayvdb claimed this task.

...
Maybe we can repurpose it to create a subset of tests that we would run every X hours again wikipedia.org (read only) and beta cluster (read/write) ?

That is T100903: Run pywikibot test suite regularly on beta cluster as part of MediaWiki/Wikimedia CI.

The developer may now run no-network tests using !net, verify they are only running no-network tests with --with-detecthttp, and the metaclass now has the infrastructure to skip network tests if the network is not available. Not much left to do here.