Page MenuHomePhabricator

create API level tests to monitor services on beta/test2wiki and also on production (tracking)
Closed, InvalidPublic

Description

Moving discussion to here for the record:

Some time ago we noted that while we have some reasonable unit tests in some repos, and some reasonable browser tests in some repos, we lack any consistent tests at the API or services level.

Having such tests would be useful not only as regression tests in the test environments, but also in order to monitor the availability of key services in the production environments. UploadWizard on Commons is a case of particular note, and the upload function in particular for UW: http://commons.wikimedia.org/w/api.php?action=help&modules=upload with the "stash" parameter.

Matt Flaschen is also doing things along those lines: https://gist.github.com/mattflaschen/7904894


Version: unspecified
Severity: enhancement
Whiteboard: zfilipin

Details

Reference
bz58353

Event Timeline

bzimport raised the priority of this task from to Normal.Nov 22 2014, 2:13 AM
bzimport set Reference to bz58353.
bzimport added a subscriber: Unknown Object (MLST).

https://gist.github.com/mattflaschen/7904894 is not an availability test, but wikitools (https://code.google.com/p/python-wikitools/) (the library it uses) is a candidate to help write such availability tests (though there are of course other possible libraries).

Pywikibot has a decently large suite of tests that actually use the live API on WMF sites: https://github.com/wikimedia/pywikibot-core/tree/master/tests

That said, what's the goal? Most API modules should have proper unit tests that check their output.

Presumably it's intended as an end-to-end integration test.

It would apply to cases where:

  • The commit itself is not broken, or at least Jenkins-run unit tests don't catch it..
  • The basic "is the homepage up?" checks (e.g. http://status.wikimedia.org/) don't catch it, since the site itself still works.
  • A particular API doesn't work as deployed, e.g. due to a specific production database outage.

This has probably happened in some cases.

(In reply to comment #2)

That said, what's the goal? Most API modules should have proper unit tests
that
check their output.

Two goals:

The general goal is to provide test coverage along the lines of Mike Cohn's 'test pyramid': http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid . We have at least some coverage at the top and bottom of the pyramid, but lack coverage at the middle.

The specific goal is also to monitor certain services in production in a reliable way using the WMF continuous integration tools. For example, some time ago Commons was configured incorrectly and media uploads stopped working, even though there was nothing wrong with the code and UploadWizard continued to work properly in test environments.

greg added a comment.Dec 12 2013, 10:38 PM

(made this a tracking bug for the sub-tasks like creating API level tests for UploadWizard. Please report those bugs and make them blocking this one. For clarify: those bugs should be something like "UploadWizard API test for some specific interesting/fragile part of the code".)

aarcos.wiki wrote:

Created bug 58555 to track the creation of basic API smoke tests for the Upload Wizard. Just updated that bug with my latest findings.

Chris (and others), we now have some tests that are using the API. Can this be resolved, or is there something else to be done?

Željko, which tests are you referring to?

Okay, those are bug 58555's tests, right?

This is currently a tracking bug for APIs in general to test like this. I don't know if it should be closed now (and potentially reopened if tests are wanted for other APIs), or left open.

Yes, looks like I was thinking of bug 58555.

Anyway, this tracking bug does not track any other bugs, so I vote to resolve it, unless somebody adds a bug that needs to be tracked.

jayvdb added a subscriber: jayvdb.Jul 11 2015, 3:24 PM

Pywikibot has a decently large suite of tests that actually use the live API on WMF sites: https://github.com/wikimedia/pywikibot-core/tree/master/tests
That said, what's the goal? Most API modules should have proper unit tests that check their output.

These pywikibot tests are now quite extensive, are running against test.wikipedia/test.wikidata, and IMO T100903: Run pywikibot test suite regularly on beta cluster as part of MediaWiki/Wikimedia CI (and related T100796: Run pywikibot tests on beta wikis) would probably resolve this task.

The main 'gap' is that pywikibot 'edit' tests are not enabled by default. There are edit tests, including upload tests (for stashes also), but they need to be explicitly enabled with an env var.

Restricted Application added a subscriber: Luke081515. · View Herald TranscriptJul 11 2015, 3:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 9 2015, 11:42 PM
Aklapper closed this task as Invalid.Apr 7 2019, 3:57 PM

Anyway, this tracking bug does not track any other bugs, so I vote to resolve it, unless somebody adds a bug that needs to be tracked.

Still the case five years later, hence closing as invalid. (No need to have generic tasks around that don't 'track' anything.)