Page MenuHomePhabricator

Analyze impact of appearance controls on usage
Closed, ResolvedPublic

Description

Appearance controls were launched with the 5.6.0 update. The theory was that making reading more customizable and comfortable we would encourage more reading, particularly longer reading sessions in the evening. That in turn would lead to more app re-use and retention (not for its own sake but as a measure).

Although we're limited in the available data, I'd like to understand if that theory proved true.

Specifically:

  • Do users using 5.6.0 have longer or more frequent sessions compared to users of earlier version during the same period? More than recent historical averages?
  • Was their any change in day 7 retention after this version or for new users downloading this version?

Bonus points:

  • Has their been any change in the diurnal usage curve of the app? That is, do things like brightness and color control actually result in more pageviews or longer/more sessions in the evening and night hours?
  • [Likely impossible, though maybe on Android?] Is there any relation between the appearance prefernces selected and the amount of use? That is do users of dark mode read more than people who leave the default settings in place?

Event Timeline

JMinor created this task.Sep 6 2017, 6:12 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 6 2017, 6:12 PM
JMinor updated the task description. (Show Details)Sep 6 2017, 6:13 PM
JMinor triaged this task as Medium priority.Sep 11 2017, 6:25 PM
JMinor moved this task from Needs Triage to Tracking on the Wikipedia-iOS-App-Backlog board.

Regarding the first question, I made an initial plot of the median session length with (recent) rollout dates:

(This comes from the general apps session metrics data, and the plot goes back to December 2015 - need to fix the x-axis.)

Regarding the second question: day-7 retention does not seem to have changed notably with the rollout of 5.6.0.

(I seem to recall we already looked at this earlier, but didn't document it on this task yet. Data via EL, as usual:

SELECT LEFT(timestamp, 8) AS date,
SUM(IF(event_appInstallAgeDays = 0, 1, 0)) AS day0_active,
SUM(IF(event_appInstallAgeDays = 7, 1, 0)) AS day7_active
FROM log.MobileWikiAppDailyStats_12637385
WHERE
timestamp LIKE '2017%'
AND (event_appInstallAgeDays = 0
OR event_appInstallAgeDays = 7)
AND (userAgent LIKE '%iPhone%'
OR userAgent LIKE '%iOS%')
GROUP BY date ORDER BY DATE;

)

Regarding bonus question 1, I took a quick look at that curve in Pivot (restricted to North America because timezones), but this doesn't seem to be something that can be eyeballed easily. Can discuss more in person.

Tbayer moved this task from Triage to In progress on the Reading-analysis board.
Tbayer closed this task as Resolved.Feb 26 2018, 10:49 PM

Closing this now, as discussed some weeks ago - it seems we have exhausted the reasonably promising avenues to find an impact of this change.

Note that there was a remarkable increase with 5.6.1 (instead of 5.6.0), for which I have made a new task: T188332