Page MenuHomePhabricator

Reading List REST Interface: testing plan
Closed, ResolvedPublic

Description

External callers should not need changes, or even be aware, when we replace the RESTBase "lists" endpoints with MediaWiki REST API endpoints. To help ensure this, create a testing plan to ensure compatibility.

Ideally, this will include tests that can be run against both the RESTBase endpoints and the new MediaWiki REST API endpoints, to ensure compatibility. This may mean an external testing suite, such as was used for AQS 2.0, is appropriate. But any approach that achieves the goal can be considered.

One complication compared to AQS 2.0 is that the Reading Lists functionality includes endpoints that modify data.

See the associated Miro board for endpoint map.
See parent task T336693: Re-implement reading lists REST interface outside RESTbase for context and details.

We have three levels of testing planned:

  1. unit tests: these are implemented within the extension. These exercise the contract of each individual handler and trait that are part of the new REST API implementation. Because these run on every git (gerrit) push, they are useful for discovering regressions vs expected behavior. But because they do not actively compare behavior to the existing production endpoints, they are susceptible to missing subtle incompatibilities. And because they focus on the internals, they could miss issues with the behavior of the system as a whole.
  2. mocha tests: these are implemented within the extension. They exercise the new REST endpoints from an outside-in perspective and test the behavior of the system as a whole. This is very valuable. However, they are less effective than unit tests at exercising the internals of the handlers in detail, and they also do not compare the behavior of the old and new systems.
  3. compatibility script: this is an external Python script designed to be executed against both the old and the new endpoints. As such, it is less effective than either the unit tests or the mocha tests at testing internals, and cannot be used to test any additions made in the new REST API. However, it is capable of comparing the behavior of the old and new system, helping ensure callers will not be surprised by any unexpected changes.

Event Timeline

Change 972408 had a related patch set uploaded (by Fgoodwin; author: Fgoodwin):

[mediawiki/extensions/ReadingLists@master] WIP: REST test suite

https://gerrit.wikimedia.org/r/972408

Change 976266 had a related patch set uploaded (by Fgoodwin; author: Fgoodwin):

[mediawiki/extensions/ReadingLists@master] Work in progress: Rest Get Lists

https://gerrit.wikimedia.org/r/976266

Change 980057 had a related patch set uploaded (by Fgoodwin; author: Fgoodwin):

[mediawiki/extensions/ReadingLists@master] Broken: Handler Tests

https://gerrit.wikimedia.org/r/980057

BPirkle triaged this task as Medium priority.
BPirkle added a subscriber: FGoodwin.

Change 1008069 had a related patch set uploaded (by BPirkle; author: BPirkle):

[mediawiki/extensions/ReadingLists@master] Adjust paths for REST setup and teardown handlers to match production

https://gerrit.wikimedia.org/r/1008069

Change 1008069 merged by jenkins-bot:

[mediawiki/extensions/ReadingLists@master] Adjust paths for REST setup and teardown handlers to match production

https://gerrit.wikimedia.org/r/1008069

FJoseph-WMF changed the task status from Open to In Progress.Apr 25 2024, 3:37 PM

Change #972408 abandoned by Fgoodwin:

[mediawiki/extensions/ReadingLists@master] WIP: REST test suite

Reason:

https://gerrit.wikimedia.org/r/972408

Change #976266 abandoned by Fgoodwin:

[mediawiki/extensions/ReadingLists@master] Work in progress: Rest Get Lists

Reason:

Work completed

https://gerrit.wikimedia.org/r/976266

Change #980057 abandoned by Fgoodwin:

[mediawiki/extensions/ReadingLists@master] Broken: Handler Tests

Reason:

Final tests completed

https://gerrit.wikimedia.org/r/980057

Progress update:

  • phpunit tests are complete and passing
  • mocha tests are complete and passing
  • compatibility tests are passing via this script, executed by me.

I'd prefer to do something a little better on the compatibility testing. The script I used is probably sufficient, but not as clean or convenient as I'd like if other people need to rerun this. I'll put a little more time into this under T360217: Reading List REST Interface: external compatibility testing before quitting to see if I can come up with something worthwhile.

Going to call this resolved. I created T373163: Extend http-response-diff-pipeline testing tool to handle more complicated pipelines for possible future improvements to http-response-diff-pipeline.