autoreporter was a tool that @chelsyx (no longer at the Foundation) wrote – based on our manually-created reports (e.g. First assessment of learning-to-rank) – to generate reports of Search Platform team's A/B tests. An example of such a report is Second MLR Test for hewiki
Unfortunately, this tool has become profoundly outdated and no longer works. In the time since its creation we switched from storing EventLogging data in MySQL to exclusively in Hadoop/Hive, the TestSearchSatisfaction2 schema was renamed to SearchSatisfaction, and was subsequently migrated to Modern Event Platform (https://schema.wikimedia.org/repositories/secondary/jsonschema/analytics/legacy/searchsatisfaction/latest.json).
I have also continued to make updates to the underlying package wmf (now wmfdata) and factored out the code for analysis of interleaved tests to wmfastr (formerly ortiz). Needless to say, this tool is in need of a massive overhaul to work with the new data pipeline. Also it would be much easier to use and maintain as an R package, so that should also be done as part of the overhaul.
CLARIFICATION: This is to update the tool that generates a very comprehensive report and – while not a quarter's worth of work – is still a significant effort, perhaps 1-2 weeks of engineering while juggling other work. For quick & simple analysis of preference from interleaved A/B tests, see these notes on Estimating Preference For Ranking Functions With Clicks On Interleaved Search Results.