I've just written a few new, fragile jobs and was hoping to write tests to cover the bits that make me nervous. Ideally, there would be a small sample of input data copied to test tables, and the job would produce an expected output in the destination tables, which could be compared.
There's currently no WMF standard for writing tests like this, so this task is for investigating what other organizations do.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T209611 [Epic] Make ORES scores for wikidata available as a dump | |||
| Open | None | T209739 Produce dump files for ORES scores | |||
| Declined | None | T213496 Investigate formal test framework for Oozie jobs |
One alternative is a JUnit class meant for workflow and coordinator testing:
http://oozie.apache.org/docs/5.1.0/ENG_MiniOozie.html
An example usage:
https://github.com/rkanter/MiniOozieExample
Seems like this will do what we want. I was able to run nearly out of the box, with only this minor patch (which might be due to a misconfiguration in my MacOS).
https://github.com/rkanter/MiniOozieExample/pull/1
> mvn test .... Running embedded servlet container at: http://wmf-laptop.redacted.net:12345 JobId: 0000000-190110154410625-oozie-awig-W Status: RUNNING Status: RUNNING .... Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
removing Release-Engineering-Team as this seems like just an investigation task, please do let us know what you find out!