Page MenuHomePhabricator

Logging for AB testing recommender systems for CX Suggestions
Closed, ResolvedPublic

Description

We want to run an AB test comparing recommender systems. Our metric for comparison is the fraction of accepted/published suggestions. Although we can already track when a suggestion is accepted and published, we need to add the following logging tables to run the test:

suggestions_schown: one log event per rendered suggestion
suggestion_id:
source language
target language
source title
user_id
recommender_type
timestamp
rank: position in the list of suggestions at time of rendering

suggestion_actions: one log event for every interaction with a suggestion
suggestions_id
timestamp
action: remove, save, start
rank: position in the list of recommendations at time of click

Additionally, when a user starts translating a suggestion, store the suggestion_id in log.ContentTranslationCTA and wikishared.cx_translations.

Details

Related Gerrit Patches:
mediawiki/extensions/ContentTranslation : masterAdd event logging for actions on suggestions

Event Timeline

ellery updated the task description. (Show Details)Jan 27 2017, 8:08 PM
leila triaged this task as High priority.Jan 28 2017, 12:25 AM
schana removed schana as the assignee of this task.Jan 30 2017, 5:55 PM
schana added a subscriber: schana.

Will this logging be within CX?

It seems like this logging could be added to the existing schema, but it definitely would abuse the scope of a CTA. I'll provide two proposals: one that uses the existing schema and one with a new schema. I prefer the new schema since it provides a clear separation of concerns.

Using existing schema

  • Modify the ContentTranslationCTA schema
    • Add appropriate enum values to the action parameter
      • favorite when the user marks a suggestion as a favorite
      • discard when the user discards a suggestion
  • Modify the application
    • Add the following log events
      • suggestion shown
      • suggestion favorited
      • suggestion discarded
    • Add additional information to the cta
      • rank-{integer} the position a suggestion holds in a list of suggestions
      • id-{uuid} a unique identifier for a suggestion

Create a ContentTranslationSuggestion schema

  • Start with the ContentTranslationCTA schema
  • Remove cta
  • Change action enum
    • shown
    • accept
    • discard
    • favorite
  • Add suggestionId that will hold a uuid for keeping track of a suggestion
  • Add rank for what position a suggestion holds in the list of suggestions
  • Add type in accordance with the values present in the existing code
  • Add typeExtra for holding extra information associated with the suggestion type
    • This is where the info from what recommender was used would live (related_articles, morelike)

@Arrbee @santhosh What are your thoughts?

@ellery Would this data provide enough info for the metric?

schana moved this task from Next Up to in-progress on the GapFinder board.

I prefer the new schema since it provides a clear separation of concerns.

👍 New schema is better here

I've created the new schema and am going to start implementing the usage of it.

Change 337033 had a related patch set uploaded (by Nschaaf):
Add event logging for actions on suggestions

https://gerrit.wikimedia.org/r/337033

schana moved this task from in-progress to Review on the GapFinder board.Feb 10 2017, 3:21 PM

Added @Nikerabbit for attention. @schana Niklas will be reviewing this as Santhosh is on leave for the next few days. Please feel free to direct any comments or questions to him.

@Nikerabbit I've responded to your comment on the pull request. Let me know if I can be of further help.

Change 337033 merged by jenkins-bot:
Add event logging for actions on suggestions

https://gerrit.wikimedia.org/r/337033

Arrbee moved this task from QA to Done on the Language-2017 Sprint 3 board.Feb 23 2017, 8:06 AM

@schana @leila, anything more needed from us for this ticket? Thanks.

There's nothing else on my end. Thanks for all the help!

Arrbee closed this task as Resolved.Mar 6 2017, 9:29 AM

Great! I am closing this ticket. If you think it needs to stay open, please feel free to reopen. Thanks.