Story: As a PM/UX Designer we want to know if a new feature performs better/worse/different than an old one.
Context: To properly evaluate the impact, we need random assignment
TODO:
See also T170392: Create gadget that enables the use of the elastic search backend for the entity selector for an opt-in / opt-out variant.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Wikidata-bugs | T77898 query taking 10s: TermSqlIndex::getMatchingIDs | |||
| Open | None | T46529 Wikidata search problems (tracking) | |||
| Resolved | Smalyshev | T78157 [Story] Use ElasticSearch for entity search on wikidata.org | |||
| Open | None | T170547 Metrics to evaluate new search for item suggestor | |||
| Declined | None | T170549 Provide A/B test for item suggestor | |||
| Resolved | Smalyshev | T162292 Reindex wikidata to pick up labels/descriptions mappings | |||
| Resolved | dcausse | T160926 Make noop script be able replace whole fields with nested subfields | |||
| Resolved | dcausse | T166589 Update wikidata code to take advantage of nested fields noop script |
I collaborated with @Addshore the last time we created such a feature. We did the random assignment by userID % 2, I think
Whether we want to start with randomized A/B testing, or rather an opt-in or opt-out testing, needs to be discussed with @Lydia_Pintscher and @Lea_Lacroix_WMDE.
In any case, we should hive a hint to users to avoid confusion. Perhaps we can add an indicator to the popup showing the search result. Something like "this result was generated using experimental feature X" with a link to more information.
@daniel: My preferred way would be having two phases:
Since both should work at least OK, I would not say "experimental" but "new" or so, since "experimental" sounds kind of dangerous. I like the indicate-the-test idea, though.
Are we still doing this? I understood that we decided to enable Elastic option for production.
I understood that we decided to enable Elastic option for production.
Jep :)
Since there were concerns that the switch might cause trouble or be worse than the "old" algorithm: How did these concerns resolve? Was there research done? Did people feel it is actually not a problem anymore? etc.
People who complained the most about the old search said they think the new one is considerably better based on Stas' test page.
Great. Thanks for documenting this.
Am 25.10.2017 5:26 nachm. schrieb "Lydia_Pintscher" <
no-reply@phabricator.wikimedia.org>:
Lydia_Pintscher added a comment.
People who complained the most about the old search said they think the
new one is considerably better based on Stas' test page.*TASK DETAIL*
https://phabricator.wikimedia.org/T170549*EMAIL PREFERENCES*
https://phabricator.wikimedia.org/settings/panel/emailpreferences/*To: *Lydia_Pintscher
*Cc: *Smalyshev, PokestarFan, Lea_Lacroix_WMDE, Addshore, Aklapper,
daniel, Charlie_WMDE, James_Budday, Aleksey_WMDE, Jonas, Lydia_Pintscher,
Jan_Dittrich, GoranSMilovanovic, QZanden, Wikidata-bugs, aude, Mbch331