Page MenuHomePhabricator

Set up unit test running for pywikibot/core
Closed, ResolvedPublic

Description

pywikibot/core has unit tests that are triggered by running "python setup.py test". If the script exits with a status code of 0, they all passed. Otherwise it's a failure.

A little bit of of setup needs to be done first:

A user-config.py needs to be placed in ~/.pywikibot/user-config.py (I will upload an example one). ~/.pywikibot will need to be writable by the script since it will create temporary cache files.

Is there any way we can provide a password that will be used to test functions only available to a logged in user?

Since anyone can write code to run on any wiki, these should only be run for whitelisted users (for now this can be the pywikibot gerrit group) or when being merged.

Also, this should be non-voting for now since all of our tests aren't passing :/


Version: wmf-deployment
Severity: enhancement
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=44220
https://bugzilla.wikimedia.org/show_bug.cgi?id=56961

Details

Reference
bz50344

Related Objects

StatusSubtypeAssignedTask
ResolvedNone
ResolvedNone
Resolvedjayvdb
ResolvedXqt
ResolvedNone
ResolvedNone
ResolvedNone
ResolvedNone
Resolvedjayvdb
ResolvedGeorggi1999
ResolvedNone
ResolvedNone
ResolvedRicordisamoa
ResolvedNone
ResolvedNone
ResolvedNone
ResolvedMpaa
Resolvedmurfel
ResolvedLadsgroup
ResolvedXZise
ResolvedNone
ResolvedNone
ResolvedNone
ResolvedRicordisamoa
ResolvedRicordisamoa
ResolvedXqt
ResolvedXqt
ResolvedXqt
DeclinedNone
ResolvedNone
ResolvedXqt
ResolvedXqt
ResolvedXZise
ResolvedXqt
ResolvedXqt
ResolvedXqt
DeclinedBUG REPORTNone
ResolvedBUG REPORTXqt

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 1:45 AM
bzimport set Reference to bz50344.
bzimport added a subscriber: Unknown Object (MLST).
Legoktm created this task.Jun 28 2013, 5:37 AM

Created attachment 12669
user-config.py file

The attached file is a working user-config.py file that needs to be in ~/.pywikibot before the unit tests are run.

Attached:

One more thing, if for some reason we can't use a ~/.pywikibot/ folder, we can use any other location by setting the PYWIKIBOT2_DIR environment variable.

If I understand correctly, the concern is that changesets are not trustable, so they can not be allowed to do anything crazy.

The jenkins job can be set to run only for known people, or after a known person has +1'd the change.

The test suite runs correctly with config.simulate enabled, which prevents a set of API actions specified in config.actions_to_block, which defaults to ['edit', 'watch', 'move', 'delete', 'undelete', 'protect','emailuser'].

There are several other actions that should be in that default set, including 'upload', and others added if the source is untrusted, such as 'createaccount'.

A more certain method is the bot user (Pywikibot-test) could be blocked - I dont think there are many tests which would fail because of that. We could skip any tests which dont like being a blocked user.

However, running any code means a changeset could alter config.*, which means the patch uploader could disable simulate, or they could add unblocked credentials.

There are almost perfect ways to lock down 'config', and we could prevent running the job if there are changes to pwb.py or api.py from an untrusted patch uploader.

Also there's just a general concern of executing malicious code.

There are currently plans to move test running to isolated VMs, but I think that also meant there would be no access to external servers (production servers). We could have jenkins set up a vanilla MediaWiki install, import a few pages and then run our tests against it?

jayvdb added a comment.Aug 1 2014, 7:58 AM

To avoid executing malicious code, the simplest approach is to do the tests only if someone trusted uploaded or +1'd the changeset.
I see that mediawiki-core has phpunit and qunit tests being run. Couldnt those tests run malicious code too?

The problems with _only_ running tests against a different mediawiki install is that a) the tests will nearly all need to be rewritten, and family files need to be built for the complex inter-language/wiki relationships, and b) it still must be isolated to prevent malicious code (using urllib2/httplib2) attacking non test servers.

Re my last post, rather than using config.actions_to_block, a HTTP proxy could be used to prevent any network traffic except whitelisted API actions.
Then the 'only' problem is local/filesystem tomfoolery, and an OS-enforced sandbox will do the trick, and is quite easy.

Some tests don't depend on a server at all, and it would be good to start with just running those tests, running the tests in a sandbox which isnt allowed to use sockets and threads, or files other than those on a whitelist (and apicache/*).

textlib for example, depends on family file information, and some siteinfo which could (and probably should) be permanently stored in the family file and only refreshed periodically. At worse we could build a MockEnWpSite to run those tests against. We've just had a massive build breakage that lasted about a week because of textlib changes which were not tested against the current tests, and one of the checked in 'fixes' included changes to wikibase_tests.py that didnt compile. It would be nice if code review didnt include checking the code compiles and unit tests pass - that wastes time, and the workpit cant be used for something else while the pywiki test suite runs, for at least 15 mins.

Thanks to John work on pywikibot test infrastructure, we now have a Jenkins job that runs tests that do not require network access.

Change-Id I6ee69f5b4e594a111d91bc0bafb59716900b478c tracks all three changes.

https://gerrit.wikimedia.org/r/#/q/I6ee69f5b4e594a111d91bc0bafb59716900b478c,n,z