Interactions by wiki

• I think the current mean per open is useful to see (especially as a way to compare between wikis), but would appreciate also having the total numbers. Would add one more line graph below to show totals for each variable - this will help show us how often generally these things are happening, not just per session.
• Wanted to clarify: the success rate of adding params means that someone adds, puts a value in, and then saves the dialog - correct?
• Can we add the success rate for editing parameters as well? Is this significantly more work?
• This is another area where I think it would be helpful to have the total of all wikis, plus ability to filter by the indiv wikis we are tracking. Already, I think it's interesting how significant the differences between them are.

Open and close overview

• for "open byMethod All" I think the only graph needed here is the pie chart. This is not something any of our changes will be likely to influence, though it's interesting to display once since we have the data.
• Interesting to see such a high number of anonymous editors! I think breakdown by edit count is looking good. Will just need to be updated once this ticket is done: T269986: Add edit count bucketing to all metrics.
• It took me some time to understand what "close bySaved All" means. Can this be renamed? If I understand correctly, this means: the dialog was opened, any of the three changes above were done (edit param, add known param, add unknown param), then the dialog was saved. I would call this "Successful edit to existing template" or something like that. Then you can define 'success' at the top, like you have now and add a definition for 'edit'. I don't think the top two graphs are needed. The pie graph communicates this well. Though, it's good to have a line graph, because this is something that we may impact in the future.
• For all in this section, 'All' would ideally mean all wikis - not all four.

Open by method

• I don't think we need this section. The single graph above communicates it, and like I mentioned above, we are not likely to impact these numbers.

Open by edit count, Close by wiki

• These sections look good to me.

@ECohen_WMDE Thanks for your feedback and questions!

Interactions by wiki

• I added the mean number per day in a bar chart and a timeline. I guess we can discuss if we need all of them.
• The success rate describes the difference between the action with a successful save action and the action with an unsuccessful save action. It shows the percentage of the successful actions.
• Currently there is no dimension bySaveEdit for edit parameter (see table above). It would be possible to add it, I will check if there is a ticket and add it.
• Not sure if I understand your comment: do you want the sum of the action in all wikis together or the sum for each wiki? I added a overview for all interactions and for all (*) wikis to the overview section. The filter feature (select one of the variable at the top left) can be used wherever it makes sense across the complete board.

Open and close overview --> Overview

• Ok, I removed all graphs except the pie chart.
• Sure, if the dimension is added to all metrics, we can create the respective graphs.
• I added the description from the ticket to the top. I removed the top two graphs.
• We could display the data for all wikis with the current means, but then I would not be configurable with the dropdown. But we are already working on adding more wikis. So this should be solvable soon.

Open by method

• Ok, I removed this section.

Thanks!

I looked into adding the all (*) wiki variable. Setting it as default does not seem to be a good idea due to the very large number of wikis (more than 100). I can however change the overview graphs (the ones that are combining the data across wikis) to always display all (*) wikis. Then you would not be able to adapt them by using the variable wiki. @ECohen_WMDE do you need to display the overview for single wikis or a customized selection? If yes, I could simply copy the whole section, so there would be one version with a fixed wikis number (all = *) and one that can be adapted (select and combine among de, en, fa and tr).

Good point! Maybe it just makes sense to make the top overview/all selection actually all and leave it at that. I think it's ok if you cannot adapt them since the bottom graphs are broken down by wiki and show that information.

Ok, I changed the overview section to only display data for all (*) wikis.

Review notes:

• Definitions are great. I made some small changes, mostly rearranging.
• Maybe we can drop the definitions under "interactions by wiki", since they're explained in the overview section now?

• Info bubbles need to be more verbose. At the minimum, the reader should be able to derive the units. For example, the first graph "edit parameter by wiki" currently says "mean of edit parameter by <wiki filter value>". I suggest "Mean number of edit parameter value events per day, summed over <wiki filter value>" (units are "events per day").

• Opens, interaction, and close sections should appear in that order, the same as the columns in the Overview.

• I think the abolute-numbered interaction graphs should appear after the normalized ones.

• How about changing "Success rate of ..." to "Success rate *after ..." ?

Follow-up:

• Aggregation fails to compensate for sampling. All aggregation queries relying on VisualEditorFeatureUse need to be scaled by 1 / $wgWMESchemaEditAttemptStepSamplingRate (specified in WikimediaEvents/extension.json), or 1 / 0.0625 = 16. Also, any events with is_oversampled must be excluded.

• Didn't we also want "Success rate after edit parameter" and "Success rate after documentation click"? But I think this requires aggregation changes, so should be done in a follow-up?

Normalization: I know this is mostly specified in a follow-up ticket, so I'm just dumping my impressions here but not blocking.

• I like that the "in all (*) wikis" graphs don't respond to the wiki filter. However, I would personally like to see the following to give nuance and comparability across wikis. If these have already been discussed, please ignore.
  • Opens by wiki (for the 4 pilot wikis), normalized by visualeditor session count, as a line graph over time. (I think this is already specified in follow-up work?)
  • How can we compare opens by method across wikis? Maybe wiki filtering should affect the pie chart? Normalized open ratios segmented by edit count would be helpful.
  • Line graph to compare close success percentage across wikis.
  • Normalized proportion of successful closes line graph. The "close by saved" line graph with absolute numbers doesn't feel helpful, it can't be interpreted easily. I would just replace it with the normalized stats. (Already in follow-up task?)

Thank you for the review! I applied your comments:

• I dropped the duplicate definition.
• I changed the info bubble texts to be more lengthily and user friendly.
• I changed the order of the sections according to the overview order.
• I changed the order of the absolute and normalized graphs.
• I changed the wording in success rate graphs.

Follow-up:

• If the compensation for the sampling should be done at the aggregation step, maybe a separate ticket would be best (because this one deals with the Grafana board)?
• Right, we wanted to add byEditSaved to edit_parameter (I will check for a ticket and create one if necessary). I think for doc_click there is no save action.

Normalization:

• Yes, the number of open actions per session is already part of this ticket: https://phabricator.wikimedia.org/T271902
• If I understood correctly, we do not need detailed views for open by method, because we will not make changes there.
• There are graphs for each wikis close success percentage in the last section. Is that sufficient?
• I guess once we have the number of VE session we can use it also for the normalization of the close actions. Or could we also maybe use the number of opens to qualify the close numbers?

In T270242#6777222, @lilients_WMDE wrote:

• If the compensation for the sampling should be done at the aggregation step, maybe a separate ticket would be best (because this one deals with the Grafana board)?

Definitely—to be done in T273454: Compensate for sampling.

• There are graphs for each wikis close success percentage in the last section. Is that sufficient?

Yes, I think that's fine!

• I guess once we have the number of VE session we can use it also for the normalization of the close actions. Or could we also maybe use the number of opens to qualify the close numbers?

Probably just normalized by opens.

I forgot, but we already tested comparing open and close numbers. We found out, that the numbers are at the same level (except for this issue). The percentages would not change. So I guess we do not really need the normalization here.