tl;dr: create a testing script that can be executed against both the existing production RESTBase Reading Lists endpoints, and the new production MW REST Reading Lists endpoints, to detect any incompatibilities. Execute this script and create separate tasks for any issues found.
The goal of the Reading Lists REST endpoints is to switch callers (WMF Mobile apps) away from RESTBase without needing any changes on the part of callers. The new endpoints were created by examining the contracts of the existing endpoints and attempting to mirror that. However, contracts do not necessarily accurately capture every minor details of an endpoint, and it is possible that something was overlooked or implemented incorrectly.
To avoid breakage for callers (and ultimately end users of the Reading Lists functionality) it would be helpful to compare the runtime behavior of the two APIs before attempting to switch over. In a previous project (AQS 2.), we did this via an external testing suite ({T319314}), which was a helpful approach.
In the case of Reading Lists, we have fewer endpoints, fewer callers, and no dedicated QA person. Also, the Reading Lists endpoints require authentication and cannot conveniently be executed against existing canned test data (external testing is more likely to require multiple API calls to set up reading lists and insert test data for the test user, all of which is itself testable behavior). These considerations all seem to make a dedicated testing script more suitable than the approach used in AQS 2.0. However, some of the general ideas of the AQS 2.0 approach (such as various specific things we could test) do apply.
The most direct path to resolving this task is a bespoke script (likely in Python, although choice of language is not dictated by this task). However, it would be more beneficial to future projects if the script could be as generic as possible, allowing it to be adapted to other similar projects that we may work on in the future.
There are several levels of automation that could be implemented (or not), including:
* hard-coded constants to point at either RESTBase or MW REST (requires manually executing the script twice, changing the code in between)
* interactive or command line parameters to dynamically determine which API to exercise (still requires manually executing the script twice)
* ability to enter configuration for both APIs, and then automatic execution against both
* command line output (which must then be manually diff'ed)
* output to file (which must then be manually diff'ed)
* automatic in-script diff with highlighting of any incompatibilities
While a minimal script implementation is suitable for resolving this task, the more ambitious goals should at least be considered and any learnings documented.
Completion criteria:
[] script is created and published somewhere convenient for future use
[] script is executed and any relevant incompatibilities are discovered
[] phab tasks are created for any issues that require code changes