Change Details

[[ https://gerrit.wikimedia.org/g/wikimedia/discovery/autoreporter | autoreporter ]] was a tool that @chelsyx (no longer at the Foundation) wrote – based on our manually-created reports (e.g.// [[ https://github.com/wikimedia-research/Discovery-Search-Test-InterleavedLTR | First assessment of learning-to-rank ]]//) – to generate reports of Search Platform team's A/B tests. An example of such a report is //[[ https://analytics.wikimedia.org/published/datasets/discovery/reports/Second_MLR_Test_for_hewiki.html | Second MLR Test for hewiki ]]// Unfortunately, this tool has become profoundly outdated and no longer works. In the time since its creation we switched from storing EventLogging data in MySQL to exclusively in Hadoop/Hive, the [[ https://meta.wikimedia.org/wiki/Schema:TestSearchSatisfaction2 | TestSearchSatisfaction2 ]] schema was renamed to [[ https://meta.wikimedia.org/wiki/Schema:SearchSatisfaction | SearchSatisfaction ]], and was subsequently migrated to Modern Event Platform (https://schema.wikimedia.org/repositories/secondary/jsonschema/analytics/legacy/searchsatisfaction/latest.json). I have also continued to make updates to the underlying package [[ https://gerrit.wikimedia.org/g/wikimedia/discovery/wmf | wmf ]] (now [[ https://github.com/wikimedia/wmfdata-r | wmfdata ]]) and factored out the code for analysis of interleaved tests to [[ https://github.com/wikimedia/wmfastr | wmfastr ]] (formerly [[ https://gerrit.wikimedia.org/g/wikimedia/discovery/ortiz | ortiz ]]). Needless to say, this tool is in need of a massive overhaul to work with the new data pipeline. Also it would be much easier to use and maintain as an R package, so that should also be done as part of the overhaul.

[[ https://gerrit.wikimedia.org/g/wikimedia/discovery/autoreporter | autoreporter ]] was a tool that @chelsyx (no longer at the Foundation) wrote – based on our manually-created reports (e.g.// [[ https://github.com/wikimedia-research/Discovery-Search-Test-InterleavedLTR | First assessment of learning-to-rank ]]//) – to generate reports of Search Platform team's A/B tests. An example of such a report is //[[ https://analytics.wikimedia.org/published/datasets/discovery/reports/Second_MLR_Test_for_hewiki.html | Second MLR Test for hewiki ]]// Unfortunately, this tool has become profoundly outdated and no longer works. In the time since its creation we switched from storing EventLogging data in MySQL to exclusively in Hadoop/Hive, the [[ https://meta.wikimedia.org/wiki/Schema:TestSearchSatisfaction2 | TestSearchSatisfaction2 ]] schema was renamed to [[ https://meta.wikimedia.org/wiki/Schema:SearchSatisfaction | SearchSatisfaction ]], and was subsequently migrated to Modern Event Platform (https://schema.wikimedia.org/repositories/secondary/jsonschema/analytics/legacy/searchsatisfaction/latest.json). I have also continued to make updates to the underlying package [[ https://gerrit.wikimedia.org/g/wikimedia/discovery/wmf | wmf ]] (now [[ https://github.com/wikimedia/wmfdata-r | wmfdata ]]) and factored out the code for analysis of interleaved tests to [[ https://github.com/wikimedia/wmfastr | wmfastr ]] (formerly [[ https://gerrit.wikimedia.org/g/wikimedia/discovery/ortiz | ortiz ]]). Needless to say, this tool is in need of a massive overhaul to work with the new data pipeline. Also it would be much easier to use and maintain as an R package, so that should also be done as part of the overhaul. **CLARIFICATION**: This is to update the tool that generates a //very// comprehensive report and – while not a quarter's worth of work – is still a significant effort, perhaps 1-2 weeks of engineering while juggling other work. For quick & simple analysis of preference from interleaved A/B tests, see these notes on //[[ https://people.wikimedia.org/~bearloga/notes/interleaved.html | Estimating Preference For Ranking Functions With Clicks On Interleaved Search Results ]]//.

[[ https://gerrit.wikimedia.org/g/wikimedia/discovery/autoreporter | autoreporter ]] was a tool that @chelsyx (no longer at the Foundation) wrote – based on our manually-created reports (e.g.// [[ https://github.com/wikimedia-research/Discovery-Search-Test-InterleavedLTR | First assessment of learning-to-rank ]]//) – to generate reports of Search Platform team's A/B tests. An example of such a report is //[[ https://analytics.wikimedia.org/published/datasets/discovery/reports/Second_MLR_Test_for_hewiki.html | Second MLR Test for hewiki ]]// Unfortunately, this tool has become profoundly outdated and no longer works. In the time since its creation we switched from storing EventLogging data in MySQL to exclusively in Hadoop/Hive, the [[ https://meta.wikimedia.org/wiki/Schema:TestSearchSatisfaction2 | TestSearchSatisfaction2 ]] schema was renamed to [[ https://meta.wikimedia.org/wiki/Schema:SearchSatisfaction | SearchSatisfaction ]], and was subsequently migrated to Modern Event Platform (https://schema.wikimedia.org/repositories/secondary/jsonschema/analytics/legacy/searchsatisfaction/latest.json). I have also continued to make updates to the underlying package [[ https://gerrit.wikimedia.org/g/wikimedia/discovery/wmf | wmf ]] (now [[ https://github.com/wikimedia/wmfdata-r | wmfdata ]]) and factored out the code for analysis of interleaved tests to [[ https://github.com/wikimedia/wmfastr | wmfastr ]] (formerly [[ https://gerrit.wikimedia.org/g/wikimedia/discovery/ortiz | ortiz ]]). Needless to say, this tool is in need of a massive overhaul to work with the new data pipeline. Also it would be much easier to use and maintain as an R package, so that should also be done as part of the overhaul. **CLARIFICATION**: This is to update the tool that generates a //very// comprehensive report and – while not a quarter's worth of work – is still a significant effort, perhaps 1-2 weeks of engineering while juggling other work. For quick & simple analysis of preference from interleaved A/B tests, see these notes on //[[ https://people.wikimedia.org/~bearloga/notes/interleaved.html | Estimating Preference For Ranking Functions With Clicks On Interleaved Search Results ]]//.