Created attachment 12669
user-config.py file

The attached file is a working user-config.py file that needs to be in ~/.pywikibot before the unit tests are run.

Attached:

One more thing, if for some reason we can't use a ~/.pywikibot/ folder, we can use any other location by setting the PYWIKIBOT2_DIR environment variable.

If I understand correctly, the concern is that changesets are not trustable, so they can not be allowed to do anything crazy.

The jenkins job can be set to run only for known people, or after a known person has +1'd the change.

The test suite runs correctly with config.simulate enabled, which prevents a set of API actions specified in config.actions_to_block, which defaults to ['edit', 'watch', 'move', 'delete', 'undelete', 'protect','emailuser'].

There are several other actions that should be in that default set, including 'upload', and others added if the source is untrusted, such as 'createaccount'.

A more certain method is the bot user (Pywikibot-test) could be blocked - I dont think there are many tests which would fail because of that. We could skip any tests which dont like being a blocked user.

However, running any code means a changeset could alter config.*, which means the patch uploader could disable simulate, or they could add unblocked credentials.

There are almost perfect ways to lock down 'config', and we could prevent running the job if there are changes to pwb.py or api.py from an untrusted patch uploader.

Also there's just a general concern of executing malicious code.

There are currently plans to move test running to isolated VMs, but I think that also meant there would be no access to external servers (production servers). We could have jenkins set up a vanilla MediaWiki install, import a few pages and then run our tests against it?

To avoid executing malicious code, the simplest approach is to do the tests only if someone trusted uploaded or +1'd the changeset.
I see that mediawiki-core has phpunit and qunit tests being run. Couldnt those tests run malicious code too?

The problems with _only_ running tests against a different mediawiki install is that a) the tests will nearly all need to be rewritten, and family files need to be built for the complex inter-language/wiki relationships, and b) it still must be isolated to prevent malicious code (using urllib2/httplib2) attacking non test servers.

Re my last post, rather than using config.actions_to_block, a HTTP proxy could be used to prevent any network traffic except whitelisted API actions.
Then the 'only' problem is local/filesystem tomfoolery, and an OS-enforced sandbox will do the trick, and is quite easy.

Some tests don't depend on a server at all, and it would be good to start with just running those tests, running the tests in a sandbox which isnt allowed to use sockets and threads, or files other than those on a whitelist (and apicache/*).

textlib for example, depends on family file information, and some siteinfo which could (and probably should) be permanently stored in the family file and only refreshed periodically. At worse we could build a MockEnWpSite to run those tests against. We've just had a massive build breakage that lasted about a week because of textlib changes which were not tested against the current tests, and one of the checked in 'fixes' included changes to wikibase_tests.py that didnt compile. It would be nice if code review didnt include checking the code compiles and unit tests pass - that wastes time, and the workpit cant be used for something else while the pywiki test suite runs, for at least 15 mins.

Thanks to John work on pywikibot test infrastructure, we now have a Jenkins job that runs tests that do not require network access.

Change-Id I6ee69f5b4e594a111d91bc0bafb59716900b478c tracks all three changes.

https://gerrit.wikimedia.org/r/#/q/I6ee69f5b4e594a111d91bc0bafb59716900b478c,n,z