CC'ing two other people on http://meta.orain.org/wiki/Special:ListUsers/steward in case they can help.

Would be nice to have, what details do we need?

@Addshore , the main question is whether Orain is OK with Pywikibot including an Orain wiki on our travis-ci build matrix, which would involve a regular read-only load test of the server, for each checking which is probably averaging 10 times a day.

Pywikibot also has some write tests, which are disabled by default, but we could configure them so they can be manually run on test.orain.org.

There will no doubt be some differences between Orain and Wikimedia wikis, so some tests will probably fail, but that is good and we'll need to improve the tests or improve pywikibot to support Orain configuration.

you say 10 times per day, how many reads / writes are there in each of these 'times'? :)

We don't write anything on Travis builds by default but the number of reads is hard to count.

EDIT: Should've read John's comment before answering directly.

What exactly do these load tests and write tests entail? What's being tested?

The list of tests which will run on the site can be seen at https://travis-ci.org/wikimedia/pywikibot-core/jobs/69537685 , however about a third of them will never actually send a request to the server, either because they are 'dry' tests, or because the test is expected to not hit the server (e.g. all the 'Test running ...' tests)

Another way to get a feel for the types of tests which would run is to look though https://github.com/wikimedia/pywikibot-core/blob/master/tests/site_tests.py , specifically any test class that subclasses 'DefaultSiteTestCase' is configured to run on any site, and would be run on the Orain wiki if permitted.

The 'pain' will be felt if there are API queries that use SQL which isnt indexed properly, such as T101502: Slow API list=recentchanges query (DBError: Lost connection to MySQL server during query). The API layer has a retry limit set to 2 during tests, so it wont retry these expensive queries too many times before it gives up and fails the test.

I can do a sample test run, using a JVbot-test if you want to monitor the amount of server activity caused.

There are two types of edit tests:

• expected failures: https://github.com/wikimedia/pywikibot-core/blob/master/tests/edit_failure_tests.py
• real writes: https://github.com/wikimedia/pywikibot-core/blob/master/tests/edit_tests.py and https://github.com/wikimedia/pywikibot-core/blob/master/tests/wikibase_edit_tests.py (and a few others)

Real write tests are configured to only run on test.wikipedia.org (test:test) or test.wikidata.org (wikidata:test), and only when an envvar is enabled. That envvar isnt enabled on automated tests.

One last question, I promise (maybe) :P

Besides our approval, is there anything you need from us to make this happen?

Afaik it's not necessary or recommended but the only think I could imagine is to add the test account to the Bot group.

I've gone ahead and given the account the bot flag. I'll watch test.orain.org and watch bot edits. Note that this is still not approval, just.. Well, it is a bot :p

Hopefully fl4 approves after reading above comments

Test run done, with login occurring at
https://travis-ci.org/jayvdb/pywikibot-core/jobs/69887121#L431

Only one failure:

======================================================================
FAIL: Test running interwiki -simulate.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/jayvdb/pywikibot-core/tests/script_tests.py", line 263, in testScript
    self.assertIn(error, result['stderr'])
AssertionError: u'does not exist. Skipping.' not found in u'Which page to check: Retrieving 1 pages from oraintest:en.\nNOTE: [[en:Test page]] does not have any interwiki links\n======Post-processing [[en:Test page]]======\nNo changes needed on page [[en:Test page]]\n'
-------------------- >> begin captured logging << --------------------
pywiki: VERBOSE: START /home/travis/build/jayvdb/pywikibot-core/tests/script_tests.py:TestScript.test_interwiki_simulate
--------------------- >> end captured logging << ---------------------

Which is caused by http://test.orain.org/wiki/Test_page existing. I expect that https://test.wikipedia.org/wiki/Test_page would also cause the same failure. We can easily fix that by using a more ridiculous page title for that test.

Change 223292 had a related patch set uploaded (by John Vandenberg):
interwiki script test fails if 'Test page' exists

https://gerrit.wikimedia.org/r/223292

Change 223292 merged by jenkins-bot:
interwiki script test fails if 'Test page' exists

https://gerrit.wikimedia.org/r/223292

Change 223461 had a related patch set uploaded (by John Vandenberg):
Run SITE_ONLY tests on test.orain.org

https://gerrit.wikimedia.org/r/223461

@Addshore I'm okay with giving the go-ahead for both site tests and the expected failure tests if you have no issues with them.

Change 223461 merged by jenkins-bot:
Run SITE_ONLY tests on test.orain.org

https://gerrit.wikimedia.org/r/223461

Change 229617 had a related patch set uploaded (by John Vandenberg):
Generating https orain family fails on Python 2

https://gerrit.wikimedia.org/r/229617

Change 229617 merged by XZise:
Generating https orain family fails on Python 2.6

https://gerrit.wikimedia.org/r/229617