Page MenuHomePhabricator

Logging for AB testing recommender systems for CX Suggestions
Closed, ResolvedPublic


We want to run an AB test comparing recommender systems. Our metric for comparison is the fraction of accepted/published suggestions. Although we can already track when a suggestion is accepted and published, we need to add the following logging tables to run the test:

suggestions_schown: one log event per rendered suggestion
source language
target language
source title
rank: position in the list of suggestions at time of rendering

suggestion_actions: one log event for every interaction with a suggestion
action: remove, save, start
rank: position in the list of recommendations at time of click

Additionally, when a user starts translating a suggestion, store the suggestion_id in log.ContentTranslationCTA and wikishared.cx_translations.

Event Timeline

leila triaged this task as High priority.Jan 28 2017, 12:25 AM
schana added a subscriber: schana.

Will this logging be within CX?

It seems like this logging could be added to the existing schema, but it definitely would abuse the scope of a CTA. I'll provide two proposals: one that uses the existing schema and one with a new schema. I prefer the new schema since it provides a clear separation of concerns.

Using existing schema

  • Modify the [[ | ContentTranslationCTA]] schema
    • Add appropriate enum values to the action parameter
      • favorite when the user marks a suggestion as a favorite
      • discard when the user discards a suggestion
  • Modify the application
    • Add the following log events
      • suggestion shown
      • suggestion favorited
      • suggestion discarded
    • Add additional information to the cta
      • rank-{integer} the position a suggestion holds in a list of suggestions
      • id-{uuid} a unique identifier for a suggestion

Create a ContentTranslationSuggestion schema

  • Start with the ContentTranslationCTA schema
  • Remove cta
  • Change action enum
    • shown
    • accept
    • discard
    • favorite
  • Add suggestionId that will hold a uuid for keeping track of a suggestion
  • Add rank for what position a suggestion holds in the list of suggestions
  • Add type in accordance with the values present in the existing code
  • Add typeExtra for holding extra information associated with the suggestion type
    • This is where the info from what recommender was used would live (related_articles, morelike)

@Arrbee @santhosh What are your thoughts?

@ellery Would this data provide enough info for the metric?

I prefer the new schema since it provides a clear separation of concerns.

👍 New schema is better here

I've created the new schema and am going to start implementing the usage of it.

Change 337033 had a related patch set uploaded (by Nschaaf):
Add event logging for actions on suggestions

Added @Nikerabbit for attention. @schana Niklas will be reviewing this as Santhosh is on leave for the next few days. Please feel free to direct any comments or questions to him.

@Nikerabbit I've responded to your comment on the pull request. Let me know if I can be of further help.

Change 337033 merged by jenkins-bot:
Add event logging for actions on suggestions

@schana @leila, anything more needed from us for this ticket? Thanks.

There's nothing else on my end. Thanks for all the help!

Great! I am closing this ticket. If you think it needs to stay open, please feel free to reopen. Thanks.