Goal: Increase user engagement in Wikipedia Android app.

• Time buckets: 2 weeks (or more if possible) for A/B test, one week before (and one week after) as control group
• All results need to be checked for statistical significance

Performance measures:

• CTR per recommendation source
• Avg. session length, pages per recommendation source

• Detail evaluation: In order to investigate in depth differences of the recommender systems, article properties (length, category, inbound links, popularity, ...) should be evaluated as well.

Use data from EL schemas: https://meta.wikimedia.org/wiki/Research:Schemas

• Aside schema specific data (timeSpent, readMoreList, ...) the evaluation requires page data (pageTitle or pageId - for article-based evaluation) and user data (appInstallId - can be anonymized).
• Relevant schemas:
  • MobileWikiAppArticleSuggestions https://meta.wikimedia.org/wiki/Schema_talk:MobileWikiAppArticleSuggestions
    • Track when the user views or interacts with the 'read more' suggestions at the bottom of the current article.
    • Measure performance with CTR
  • MobileWikiAppPageScroll https://meta.wikimedia.org/wiki/Schema_talk:MobileWikiAppPageScroll
    • Measure how much of the page the user scrolls through.
    • Metrics per pageId + appInstallId: timeSpent, maxPercentViewed
    • Measure for user engagement (overall and after a recommendation-click)
  • MobileWikiAppSessions https://meta.wikimedia.org/wiki/Schema:MobileWikiAppSessions
    • Track user behavior during a session of using the app.
    • Metrics: length, totalPages, fromSearch, fromRandom, fromLanglink, fromInternal, fromExternal, fromHistory, fromReadingList, fromNearby, fromDisambig, fromBack
  • MobileAppShareAttempts https://meta.wikimedia.org/wiki/Schema_talk:MobileAppShareAttempts
    • Logs instances of people trying to share via Mobile app
    • General measure of user satisfaction (maybe biased by other factors)
• Combined schemas:
  • long-CTR: time spent or maxPercentViewed of recommended page (check for equal pageId and appInstallId)
  • recommended-shares: Shares of pages that have been recommended before (in a 2 day time frame)
  • recommended-session: number of totalPages per session, session length for user that clicked on a recommendations