Are people using the new RC page tools? Which ones are they using? Has that increased their use of the page, and presumably their satisfaction with it? The ERI success metrics this task proposes are crude, but should provide some answers to these questions.
It's important we establish a baseline for RC Page tool usage before we release the beta. (Establishing the baseline after beta release would be bad, since many of our most active users will be in the beta.) This exercise will also flush out any issues with the tracking mechanisms we've put in place.
Proposed metrics
- Tool usage profile: What filters and other tools (e.g., highlighting) do reviewers use most often? Can we come up with a profile of current tool usage?
- Note that for Highlight tools, it will be important to know which filters users are highlighting. I.e., we can't just say the Highlight tool was used X times, though we will want to know that total too. But we'll especially want to know that users were, for example, highlighting Newcomers, etc. The combination, Highlight + Tool.
- How can the profile numbers best be expressed? I think best would be if each tool were expressed in terms of the % of total tool "settings" it represents (per week/mo?). So, if filters were selected 100K times in a week, and if Newcomer was selected 5K times, then its usage would be 5%. Conceptually, we're talking about a pie chart.
- Page popularity (sessions): Can we establish a valid metric for and then a baseline stat for something we might call "page sessions." (E.g., a period of RC page and other page use that could be said to be terminated if the user does not return to the RC page for 30 mins.)
- Tool engagement: What proportion of "page sessions" use only default settings vs .those that involve tool selections? (If this goes up with the new system, we can conclude the interface has made the tools more accessible).
- Session length: Another traditional measure of engagement is length of session, the theory being that if users like the tool they will use it longer.
Questions/Issues
- For sessions in which users have employed a saved URL that includes non-default filters, the non-default filters should be counted towards the tool usage profile and tool engagement stats. Otherwise we're liable to ignore the habits of our most prolific and savvy users.
- I'm suggesting "page sessions" instead of page views as a gross measure of page popularity for a number of reasons. E.g., page views might actually go down after beta release if users are finding what they need more easily.
- What is currently being counted as a "page view" now anyway? E.g., is it every time the page reloads? Or every time a user chooses a new tool? etc.
- I'm thinking a week or a month might be the relevant period for this type of analysis, to avoid normal weekly rhythms.
- While session length is indeed a traditional engagement metric, on the theory that people will spend more time doing things that they like, there is some question as to whether it's meaningful here. We just don't know enough about users' habits. E.g., it's possible that users simply have a certain, nonelastic amount of time they devote to edit review. Or it's possible they have certain fixed goals they seek to attain; if more efficient tools let them attain those goals more quickly, session time might decrease. So what I'd say is, this would be interesting if we can achieve it without a lot of work.
- Do we need to produce the baseline figures now for all wikis we will ever want to measure? Or will the data be available indefinitely?
Steps
- We don't need to build a graphing tool out of the gate. Our goal, I think, is to produce a spreadsheet from which we can extract meaningful conclusions. If we want to automate analysis, we can do that later.
My sense of the best way to proceed is this:
- Investigate the issues, determine what is possible and how involved the project will be, then report back.
- If required, put in place whatever tools are necessary to get the data we want.
- Make a trial run at producing analysis for two of the ORES wikis, one large and one small. Say en.wiki and pl.wiki?
- Refine methodology/technology as needed.
- Rerun the analysis of the two wikis above.
- [If baseline figures will not be available indefinitely (see above), determine a test set and run baselines]