**Goal: Increase user engagement in Wikipedia Android app.**
- Time buckets: 2 weeks (or more if possible) for A/B test, one week before (and one week after) as control group
- All results need to be checked for statistical significance
Performance measures:
- CTR per recommendation source
- Avg. session length, pages per recommendation source
- Detail evaluation: In order to investigate in depth differences of the recommender systems, article properties (length, category, inbound links, popularity, ...) should be evaluated as well.
Use data from EL schemas: https://meta.wikimedia.org/wiki/Research:Schemas
- Aside schema specific data (timeSpent, readMoreList, ...) the evaluation requires page data (pageTitle or pageId - for article-based evaluation) and user data (appInstallId - can be anonymized).
- Relevant schemas:
- MobileWikiAppArticleSuggestions https://meta.wikimedia.org/wiki/Schema_talk:MobileWikiAppArticleSuggestions
- Track when the user views or interacts with the 'read more' suggestions at the bottom of the current article.
- Measure performance with CTR
- MobileWikiAppPageScroll https://meta.wikimedia.org/wiki/Schema_talk:MobileWikiAppPageScroll
- Measure how much of the page the user scrolls through.
- Metrics per pageId + appInstallId: timeSpent, maxPercentViewed
- Measure for user engagement (overall and after a recommendation-click)
- MobileWikiAppSessions https://meta.wikimedia.org/wiki/Schema:MobileWikiAppSessions
- Track user behavior during a session of using the app.
- Metrics: length, totalPages, fromSearch, fromRandom, fromLanglink, fromInternal, fromExternal, fromHistory, fromReadingList, fromNearby, fromDisambig, fromBack
- MobileAppShareAttempts https://meta.wikimedia.org/wiki/Schema_talk:MobileAppShareAttempts
- Logs instances of people trying to share via Mobile app
- General measure of user satisfaction (maybe biased by other factors)
- Combined schemas:
- long-CTR: time spent or maxPercentViewed of recommended page (check for equal pageId and appInstallId)
- recommended-shares: Shares of pages that have been recommended before (in a 2 day time frame)
- recommended-session: number of totalPages per session, session length for user that clicked on a recommendations