Description
Currently implementation ranking is (potentially) done with every call to ApiPerformTest, but only when certain conditions are met. In particular, it happens when the call requests test results that cover all attached implementations and testers, and when no tester results come from the cache *, which only happens when the function has been updated.
- * UPDATE 2024-03-21: this was changed a while back, so we update when there's at least one tester result that's live (not from cache).
Concerns:
- A nontrivial change to an implementation or a tester could warrant re-ranking, but it won't happen.
- Currently the input values passed to ApiPerformTest specify a superset of attached implementations and testers, but it's possible the call could be changed in such a way that implementation ranking would never happen.
Instead, we might want to make ranking happen whenever there's a nontrivial change to a function, implementation or tester. At these times the "nocache" parameter could be used to force all results to be produced live. (Alternatively, we could change ApiPerformTest to allow ranking to happen when some of the tester results are retrieved from the cache.) Either way, ranking could be done by ApiPerformTest calls that are made specifically for the purpose of ranking, so they wouldn't ever need to be changed.
In addition, by reinstating the updateImplementationRanking parameter, we could turn off ranking in the existing call to ApiPerformTest, which potentially would eliminate some ranking that doesn't need to happen.
Desired behavior/Acceptance criteria
- We should have a more precise approach in which implementation ranking happens when it's clearly warranted, and only then.
Completion checklist
- Before closing this task, review one by one the checklist available here: https://www.mediawiki.org/wiki/Abstract_Wikipedia_team/Definition_of_Done#Front-end_Task/Bug_Completion_Checklist