@ppelberg

Here are the details regarding the reply tool usage on Arabic, Czech and Hungraian Wikipedias following deployment as an opt-out feature. Data below reflects events recorded from 24 September 2020 through 16 October 2020.

High-level observations/notes:

• There have been 619 reply tool events (45 events from logged out users). More than half of these events (58.8%) were made by Senior Contributors with over 1000 edits.
• Over half (330) of the reply tool events came from Arabic Wikipedia.
• There are 226 distinct registered users to date. The two largest edit count groups are Senior Contributors with over 1000 edits (39.38% of reply tool users) and Junior contributors with 1-4 edits (26.55% of reply tool users).
• There are a couple of limitations/caveats related to including logged-out users in this analysis. See below:
  • We currently do not have a unique identifier for logged-out users recorded for discussion tool-related events. As a result, we can determine how many and which events come from logged out users but unable to identify the number of distinct logged out users.
  • We do not know the edit count for logged out users. They are all recorded with an edit count of 0 in the data.

Number of Events[1] [2]

[1] Defined as reply tool edits that reach action = 'init'
[2] Includes events by logged out users. All these users are recorded as having 0 edits.

Number of Distinct Users [3]

[3] Only includes logged in users. We do not currently store a unique identifier to determine the number of distinct logged out users.

Recommendation
Based on the numbers above, I'd recommend either wait another 2 more weeks if possible to to ensure we have sufficient data to represent each group in the workflow engagement analysis. There are several edit count groups with under 50 events total, which will make it hard to confidently report on any metrics for those groups. It would be great if we could get at least 1000 events total with hopefully some more recorded for lower edit count groups and on Hungarian and Czech Wikis. Assuming we continue to see a similar rate of increase over the next 2 weeks that should be sufficient. Happy to discuss other options.

Thank you for pulling this together, @MNeisler. A few comments and clarifying questions in line, below...

• There have been 619 reply tool events (45 events from logged out users).

• To confirm, would it be accurate for me to understand the above as meaning the following "The Reply Tool has been initiated 619 times at the Arabic, Czech and Hungarian Wikipedias between 24-Sep and 16-Oct-2020?
• Is "619 reply tool events" strike you as surprising in any way or is it more or less in line with what you'd expect?

• Over half (330) of the reply tool events came from Arabic Wikipedia.

• Again, would it be accruate for me to understand the "reply tool events" counted as part of that "330" number as only action = 'init' events. Read: when someone clicks a [ reply ] link.

• There are a couple of limitations/caveats related to including logged-out users in this analysis. See below:
  • We currently do not have a unique identifier for logged-out users recorded for discussion tool-related events. As a result, we can determine how many and which events come from logged out users but unable to identify the number of distinct logged out users.
  • We do not know the edit count for logged out users. They are all recorded with an edit count of 0 in the data.

• Noted.

Recommendation
Based on the numbers above, I'd recommend either wait another 2 more weeks if possible to to ensure we have sufficient data to represent each group in the workflow engagement analysis.

• Doing what you are suggesting here and waiting another 2 weeks sounds like a good idea to me (read: using data gathered from 24-Sep through 30-Oct-2020 for the analysis). Let's do that.

There are several edit count groups with under 50 events total...

• To confirm, "50 events total" here means 50 events for a given user group, across wikis (e.g. there seem to have been 42 events triggered by the 100-999 edits group)?

In T263050#6562047, @ppelberg wrote:

Thank you for pulling this together, @MNeisler. A few comments and clarifying questions in line, below...

• There have been 619 reply tool events (45 events from logged out users).

• To confirm, would it be accurate for me to understand the above as meaning the following "The Reply Tool has been initiated 619 times at the Arabic, Czech and Hungarian Wikipedias between 24-Sep and 16-Oct-2020?

Yes that's correct. There have been 619 action = 'init reply tool events on Arabic, Czech and Hungarian Wikipedias between 24-Sep and 16-Oct-2020.

• Is "619 reply tool events" strike you as surprising in any way or is it more or less in line with what you'd expect?

I was expecting a slightly larger number of events over 2 weeks since it was deployed as opt-out but I think given the size of the wikis it's not too surprising. The proportion of events to users looks similar to what I expected. We're currently seeing multiple reply tool clicks from each Senior Contributor while Junior Contributors appear to typically make just 1 or 2 clicks each. There is also a fairly even distribution of reply tool users across all edit experience levels, which is great for the analysis.

• Over half (330) of the reply tool events came from Arabic Wikipedia.

• Again, would it be accruate for me to understand the "reply tool events" counted as part of that "330" number as only action = 'init' events. Read: when someone clicks a [ reply ] link.

Yes, that's correct. I only reviewed action= 'init events in this analysis.

Recommendation
Based on the numbers above, I'd recommend either wait another 2 more weeks if possible to to ensure we have sufficient data to represent each group in the workflow engagement analysis.

• Doing what you are suggesting here and waiting another 2 weeks sounds like a good idea to me (read: using data gathered from 24-Sep through 30-Oct-2020 for the analysis). Let's do that.

Sounds good. I will plan the start the workflow analysis the week of November 2nd.

There are several edit count groups with under 50 events total...

• To confirm, "50 events total" here means 50 events for a given user group, across wikis (e.g. there seem to have been 42 events triggered by the 100-999 edits group)?

Yes, correct. Since we're reviewing how each experience level group is engaging with the tool, we will ideally need a larger number of events recorded for each edit group in order to more accurately report on their success.