https://gist.github.com/mattflaschen/7904894 is not an availability test, but wikitools (https://code.google.com/p/python-wikitools/) (the library it uses) is a candidate to help write such availability tests (though there are of course other possible libraries).

Pywikibot has a decently large suite of tests that actually use the live API on WMF sites: https://github.com/wikimedia/pywikibot-core/tree/master/tests

That said, what's the goal? Most API modules should have proper unit tests that check their output.

Presumably it's intended as an end-to-end integration test.

It would apply to cases where:

• The commit itself is not broken, or at least Jenkins-run unit tests don't catch it..
• The basic "is the homepage up?" checks (e.g. http://status.wikimedia.org/) don't catch it, since the site itself still works.
• A particular API doesn't work as deployed, e.g. due to a specific production database outage.

This has probably happened in some cases.

(In reply to comment #2)

That said, what's the goal? Most API modules should have proper unit tests
that
check their output.

Two goals:

The general goal is to provide test coverage along the lines of Mike Cohn's 'test pyramid': http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid . We have at least some coverage at the top and bottom of the pyramid, but lack coverage at the middle.

The specific goal is also to monitor certain services in production in a reliable way using the WMF continuous integration tools. For example, some time ago Commons was configured incorrectly and media uploads stopped working, even though there was nothing wrong with the code and UploadWizard continued to work properly in test environments.

(made this a tracking bug for the sub-tasks like creating API level tests for UploadWizard. Please report those bugs and make them blocking this one. For clarify: those bugs should be something like "UploadWizard API test for some specific interesting/fragile part of the code".)

aarcos.wiki wrote:

Created bug 58555 to track the creation of basic API smoke tests for the Upload Wizard. Just updated that bug with my latest findings.

Chris (and others), we now have some tests that are using the API. Can this be resolved, or is there something else to be done?

Željko, which tests are you referring to?

Jenkins jobs:

https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/

https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.org/

Job configuration:

https://github.com/wikimedia/integration-jenkins-job-builder-config/blob/cloudbees/jobs.yaml#L255-L270

https://github.com/wikimedia/integration-jenkins-job-builder-config/blob/cloudbees/job_template.yaml#L81-L126

Okay, those are bug 58555's tests, right?

This is currently a tracking bug for APIs in general to test like this. I don't know if it should be closed now (and potentially reopened if tests are wanted for other APIs), or left open.

Yes, looks like I was thinking of bug 58555.

Anyway, this tracking bug does not track any other bugs, so I vote to resolve it, unless somebody adds a bug that needs to be tracked.

In T60353#594652, @Legoktm wrote:

Pywikibot has a decently large suite of tests that actually use the live API on WMF sites: https://github.com/wikimedia/pywikibot-core/tree/master/tests

That said, what's the goal? Most API modules should have proper unit tests that check their output.

These pywikibot tests are now quite extensive, are running against test.wikipedia/test.wikidata, and IMO T100903: Run pywikibot test suite regularly on beta cluster as part of MediaWiki/Wikimedia CI (and related T100796: Run pywikibot tests on beta wikis) would probably resolve this task.

The main 'gap' is that pywikibot 'edit' tests are not enabled by default. There are edit tests, including upload tests (for stashes also), but they need to be explicitly enabled with an env var.