Questions/Issues

• How can the tool usage numbers best be expressed? One nice way might be in the % of total tool "settings" a given setting represents. So, if filters were selected 100K times in a week, and if Newcomer was selected 5K times, then its usage would be 5%. Conceptually, we're talking about a pie chart.
• I'm suggesting "page sessions" instead of page views as a gross measure of page popularity for a number of reasons. E.g., page views might actually go down after beta release if users are finding what they need more easily.
  • What is currently being counted as a "page view" now anyway? E.g., is it every time the page reloads? Or every time a user chooses a new tool? etc.
• Do we need to produce the baseline figures now for all wikis we will ever want to measure? Or will the data be available indefinitely?

In T170214#3423056, @jmatazzoni wrote:

• What is currently being counted as a "page view" now anyway? E.g., is it every time the page reloads? Or every time a user chooses a new tool? etc.

If selecting a tool represents a page view we need to consider that some filters may be counted multiple times for the same final user intent. For example, if a user wants to filter by X, Y and Z, and activates the filters in that order this will result in one view for X, another for X+Y and another for X+Y+Z. This may result in a total of 3 views for X, 2 views for Y and 1 view for Z; and we may reach the conclusion that X is a much popular filter than Z.

I guess it is hard to distinguish which visits are part of selecting a set of filters, and which ones are used to do actual review work. Maybe we can just accept such noise in the data, or try to combine the numbers with tool engagement numbers or other de-duplication strategy depending on how much impact we think that noise could have.

In T170214#3424124, @Pginer-WMF wrote:

If selecting a tool represents a page view we need to consider that some filters may be counted multiple times for the same final user intent. For example, if a user wants to filter by X, Y and Z, and activates the filters in that order this will result in one view for X, another for X+Y and another for X+Y+Z. This may result in a total of 3 views for X, 2 views for Y and 1 view for Z; and we may reach the conclusion that X is a much popular filter than Z.

@Catrope is what Pau says here correct, in terms of the way the filters will be counted? That would throw off the whole exercise pretty significantly.

In T170214#3425639, @jmatazzoni wrote:

@Catrope is what Pau says here correct, in terms of the way the filters will be counted? That would throw off the whole exercise pretty significantly.

Yes, he is correct. We could mitigate that a bit by counting how many sessions a filter was used in.

In T170214#3428362, @Catrope wrote:

Yes, he is correct. We could mitigate that a bit by counting how many sessions a filter was used in.

Hmmm. So if I understand this, what you're saying is that without correcting as you suggest, we'd expect that filters at the top of the menu would always be more "popular," since they will be selected first and then counted again and again. Is that right? That's no good at all.

As to your suggested fix: when you say "sessions," are you talking about the hourly hack we discussed? I.e., where a period of RC Page use by an individual within a given clock hour counts as a "session". If so, I'm imagining that what we'd get from correcting with this method would be something like this:

• there were 1M "sessions" in the two-week period.
• Filters A, B, C were used in 90% of sessions.
• Filters D, E, F were used in 80% of sessions
• Etc.

Would it be like that? If so, what we're losing I guess is that if the reviewer used filter A in one search during a session and filter B in 100 searches, we wouldn't know. We'd count them the same. Is that right?

I think those results would still be useful. We can hypothesize that it all equals out. What do others think?

This is a bit of a radical suggestion, but we could try to do this part in the front end, and depend on sending the data on closing the menu.

So a user selecting A+B+C will likely open the filter popup, select A, then B, then C (without closing it) then close the popup -- and at that point, we'll send the data for A, B and C (and we can even send another schema for the combination?)

The only side effect is that users may close the popup and reopen multiple times (we can try to mitigate that by only resending the data if there was an actual change first) and the other problem is that users may close the popup to change view (not clicking on the view button, etc) in which case we will report back A+B+C (filters) and then again A+B+C+namespaceA+namsepaceB the next time the user closes the menu.

Does that make sense?

Highlight usage

This one is easy, because there's so little data, and it doesn't suffer from the double selection issue. We started collecting this data on 2017-06-14; between then and the present, only 75 distinct users have used the highlighting feature, and highlights were set a grand total of only 506 times. By far the most popular filters to highlight by are the ORES ones, as well as newcomers and unregistered users.

Because there is so little data, I didn't bother to slice it by wiki, by week, by session (user+hour) or anything like that, so the numbers below are highlight events (i.e. the number of times someone set a highlight for a given property). The wikis where highlighting was used the most were the Portuguese (166 events) and English Wikipedias (105), by far (#3 is Farsi with 29). I also included histograms of which days had more or fewer highlight events and unique highlight users at the bottom of the data below.

In T170214#3428591, @Mooeypoo wrote:

This is a bit of a radical suggestion, but we could try to do this part in the front end, and depend on sending the data on closing the menu.

Ingenious! Is there a way to also count Saved Filters with that, which, of course, don't require opening of the panel.

In T170214#3428606, @jmatazzoni wrote:

In T170214#3428591, @Mooeypoo wrote:

This is a bit of a radical suggestion, but we could try to do this part in the front end, and depend on sending the data on closing the menu.

Ingenious! Is there a way to also count Saved Filters with that, which, of course, don't require opening of the panel.

We could probably get away with that by putting the logging operation where we tell the system to load saved queries.

Also, just to emphasize, this should only be for this specific count; the front end isn't as robust in keeping data as the backend is, because it may in these cases not log things like "from" / "days" / "limit" and any other data that is not directly implemented in the menus. It's a good method to count this specific requirement, though. We should be able to do that.

Also, I'm suggesting this mainly because I think working with sessions is probably harder but I don't exactly know if that's the case, so ping @Catrope for his opinion.

In T170214#3428597, @Catrope wrote:
75 distinct users have used the highlighting feature, and highlights were set a grand total of only 506 times.

So that's 75 users on all wikis over the last month? That sounds mighty low for such a useful feature. But before we start figuring out how to make highlighting more visible, we'll want some context for these numbers, such as:

• How many RC Page users who have the beta have been active in that time overall? E.g., what % of users used Highlighting. I assume it's a small percentage, but we don't really know, do we?
• How many "sessions" with the beta did users engage in during that time?

In T170214#3428525, @jmatazzoni wrote:

In T170214#3428362, @Catrope wrote:

Yes, he is correct. We could mitigate that a bit by counting how many sessions a filter was used in.

Hmmm. So if I understand this, what you're saying is that without correcting as you suggest, we'd expect that filters at the top of the menu would always be more "popular," since they will be selected first and then counted again and again. Is that right? That's no good at all.

Yes, probably.

As to your suggested fix: when you say "sessions," are you talking about the hourly hack we discussed? I.e., where a period of RC Page use by an individual within a given clock hour counts as a "session".

Yes.

If so, I'm imagining that what we'd get from correcting with this method would be something like this:

• there were 1M "sessions" in the two-week period.
• Filters A, B, C were used in 90% of sessions.
• Filters D, E, F were used in 80% of sessions
• Etc.

Would it be like that? If so, what we're losing I guess is that if the reviewer used filter A in one search during a session and filter B in 100 searches, we wouldn't know. We'd count them the same. Is that right?

Yes, that's right.

I think those results would still be useful. We can hypothesize that it all equals out. What do others think?

To make it more meaningful we could use 1-minute sessions for this purpose (or some other convenient but relatively small number).

In T170214#3428636, @Mooeypoo wrote:

In T170214#3428606, @jmatazzoni wrote:

In T170214#3428591, @Mooeypoo wrote:

This is a bit of a radical suggestion, but we could try to do this part in the front end, and depend on sending the data on closing the menu.

Ingenious! Is there a way to also count Saved Filters with that, which, of course, don't require opening of the panel.

We could probably get away with that by putting the logging operation where we tell the system to load saved queries.

Also, just to emphasize, this should only be for this specific count; the front end isn't as robust in keeping data as the backend is, because it may in these cases not log things like "from" / "days" / "limit" and any other data that is not directly implemented in the menus. It's a good method to count this specific requirement, though. We should be able to do that.

Also, I'm suggesting this mainly because I think working with sessions is probably harder but I don't exactly know if that's the case, so ping @Catrope for his opinion.

Yes, this would be much better and easier. But we can only use that strategy to improve our data collection for the future, not for the past. So it's useful if you want numbers next week/month (which you probably do!) but not if you want numbers right now (which you also do).

In T170214#3428663, @jmatazzoni wrote:

In T170214#3428597, @Catrope wrote:
75 distinct users have used the highlighting feature, and highlights were set a grand total of only 506 times.

So that's 75 users on all wikis over the last month? That sounds mighty low for such a useful feature. But before we start figuring out how to make highlighting more visible, we'll want some context for these numbers, such as:

Yeah I was very surprised at how low it was. I looked at the data for traces of loss and didn't find any clear evidence of it. I did just try highlighting something and that hasn't shown up in the data yet, but I'm not sure how much this is supposed to lag by. I'll see if I can find validation error logs somewhere, those would tell us if we've been dropping lots of events somehow.

• How many RC Page users who have the beta have been active in that time overall? E.g., what % of users used Highlighting. I assume it's a small percentage, but we don't really know, do we?
• How many "sessions" with the beta did users engage in during that time?

I was going to get those numbers next as part of the "Tool usage profile" question, I just wanted to start with something simple.

Another data point that might be interesting, and that we could explore more (on other wikis) if we like: on enwiki, 217 users have enabled the highlight button at least once. A further 300 never enabled it but did open the dropdown at least once. So that's a total of 517 users who we can consider to have used the beta in the sense that they opened the filter dropdown (its main UI component). Compare that to 8602 users who have the beta feature enabled.

I completely forgot that this was tracked. It's not part of our metrics tracking really, it's stored so that we remember to stop showing the pulsating dot after they've tried it once.

For the people who ended up clicking the highlight button (the group of 217), the number of times they opened the popup before they clicked the highlight button breaks down as follows:

mysql:research@s3-analytics-slave [enwiki]> select count(*), cast(v as int) as num from (select up1.up_value as v from user_properties as up1 where up1.up_property='rcenhancedfilters-seen-highlight-button-counter' and up1.up_value >= 1 and up1.up_user in (select up2.up_user from user_properties as up2 where up2.up_property='rcenhancedfilters-tried-highlight' and up2.up_value=1) ) as x group by num order by num;
+----------+------+
| count(*) | num  |
+----------+------+
|       96 |    1 | -- 96 people clicked it the first time they opened it
|       44 |    2 | -- 44 people clicked it the second time, etc.
|       23 |    3 |
|       13 |    4 |
|        8 |    5 |
|       10 |    6 |
|        1 |    7 |
|        2 |    8 |
|        5 |    9 |
|        1 |   10 |
|        4 |   12 |
|        2 |   13 |
|        1 |   14 |
|        1 |   18 |
|        1 |   36 |
|        1 |  321 |
+----------+------+
16 rows in set (0.03 sec)

(strangely this doesn't add up to 217 but to 213; for 4 users, we did not record any attempts to open the dropdown but did record that they enabled highlighting; perhaps they followed a URL with ?highlight=1 in it)
We show a pulsating dot on the button the 6th, 9th, 12th, etc. time (the spec asked for this to happen the 5th, 8th, 11th, etc time but I think we have an off-by-one error in this code).

For people who never clicked the highlight button (the group of 300), the number of times they opened the popup breaks down as:

mysql:research@s3-analytics-slave [enwiki]> select count(*), cast(v as int) as num from (select up1.up_value as v from user_properties as up1 where up1.up_property='rcenhancedfilters-seen-highlight-button-counter' and up1.up_user not in (select up2.up_user from user_properties as up2 where up2.up_property='rcenhancedfilters-tried-highlight' and up2.up_value=1) ) as x group by num order by num;
+----------+------+
| count(*) | num  |
+----------+------+
|      146 |    1 |
|       63 |    2 |
|       24 |    3 |
|       18 |    4 |
|        6 |    5 |
|        8 |    6 |
|        6 |    7 |
|        5 |    8 |
|        3 |    9 |
|        2 |   10 |
|        2 |   11 |
|        2 |   12 |
|        2 |   13 |
|        1 |   14 |
|        2 |   15 |
|        3 |   17 |
|        2 |   18 |
|        1 |   22 |
|        1 |   46 |
|        1 |   70 |
|        1 |   92 |
|        1 |  113 |
+----------+------+
22 rows in set (0.01 sec)

@jmatazzoni What do you think of the data below? It's only enwiki for now and for the last full two weeks (July 17-30; a slightly different time frame might be better because of changes that happened on the 28th), but if you like it I'll rerun it for all the wikis you asked for.

Caveats: this is the simplest way of data gathering, so the things I'm counting are "events"/"hits", which has a lot of problems as we've previously discussed. Live update (if users find the magic flag, a few have) makes this worse. I'll work on counting "sessions" (i.e. groups of events by the same user in the same ten minutes) next, but the kinds of queries I'd do would be similar, so please look at that below (or at P5832 to view it in a larger area).