Page MenuHomePhabricator

Provide details of result validation
Open, Needs TriagePublicFeature

Description

Feature summary (what you would like to be able to do and where):
The “Details” for a Test case should include (as a minimum) the Implementation used by the Function specified for the “result validation”.

Use case(s) (list the steps that you performed to discover that problem, and describe the actual underlying problem which you want to solve. Do not describe only a solution):
When a test fails, we need to distinguish between cases where the failure is in the evaluation of the function being tested and cases where the failure occurred in the “result validation”. In the latter case, we need to know which implementation was used in the outermost “result validation” function call. (If the “result validation” contains nested function calls, we should be able to see which implementation of which function produced an error, but this is assumed to be dependent on T363384 and T363396.)

Benefits (why should this be implemented?):
Contributors can easily believe that there is a problem with the Test case or the function being tested when there is no such problem.
In a recent example, a query was raised on Telegram about Z21468 after the very experienced contributor was unable to identify the problem. It turned out that the “result validation” function was using a “ghost” implementation (an implementation of a different function that was incorrectly listed in the tested function’s implementation list and not displayed in the “Implementations” box on the function page). Fixing this problem also resolved some other issues that had previously baffled other contributors.