Page MenuHomePhabricator

Review WikiLambda system’s criteria for updating a function’s implementation list
Open, LowPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

What happens?:
Two implementations with relatively poor performance were promoted above three implementations with better performance.

What should have happened instead?:
Disconnected test cases should be disregarded or have a lower weighting than connected test cases.

Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia):

Other information (browser name/version, screenshots, etc.):

The function had six implementations and seven test cases. Two test cases were not connected at the time, since they test with invalid Natural number objects (negative value or leading zero). The implementations promoted by WikiLambda system were successful for all seven tests. Three of the less favoured implementations were successful for six out of seven tests, failing for the disconnected case with a leading zero. The preferred implementations rely on Python (==) and have significantly worse performance than the other three, which rely on built-in Z803 and Z866. (The sixth implementation relies on JavaScript (===) and had a timeout failure or two.)

Because disconnecting Z13524 (on April 18th: https://www.wikifunctions.org/w/index.php?title=Z13522&diff=99728&oldid=99559 ) seems to have made no difference to WikiLambda system’s preferences, I considered it appropriate to disconnect the preferred implementations for the time being.

Please see T360385 for issues that I suspect are related to this problem, at least in part.

Disconnections: https://www.wikifunctions.org/w/index.php?search=T363908&title=Special:Search&profile=advanced&fulltext=1&ns1=1

Event Timeline

Let's timebox the investigation and fix for now in order to figure out what is going on. Say two days, and then write down the status. We should figure out why seemingly slower implementations are ranked higher. Another example where this happens is on Z16137: https://www.wikifunctions.org/wiki/Z16137

All of the testers against Z16142 are faster than Z16143, quite considerably so, and yet, WikiLambda system ranked the latter higher than the former: https://www.wikifunctions.org/wiki/Z16137?uselang=en&diff=prev&oldid=103958

This lead to the community disconnecting a perfectly valid implementation. We should figure out why the ranking is off here.