Page MenuHomePhabricator

Personalized Praise: Instrumentation
Closed, ResolvedPublic

Description

User Story:

As the Growth Product Manager,
I want to measure the impact of our features,
Because then I can make informed decisions about how to proceed.

As a Mentor,
I want to know the impact of this new feature,
Because I only want to invest time in sending personalized messages to mentees if I can see that it helps with newcomer retention or activity

Background:

Research shows that praise and encouragement from other users increases newcomer retention.
Growth documentation:

Mediawiki overview of Positive Reinforcement project.

Questions:

Personalized Praise questions:

  • How many mentees are surfaced to Mentors via this feature?
  • How many messages are posted to Mentee talk pages because of this feature?
  • Is there a difference in activity or retention for Mentees who are sent a personal message from their Mentor, and those who do not receive a message (assuming they meet the same criteria).
  • Do Mentors send more Thanks when they have this feature enabled?

For Spanish Wikipedia Mentorship comparison (comparing newcomers with Mentors vs. newcomers without Mentors):

  • What proportion of newcomers activate (edit within 24 hours of registration)?
  • What proportion of activated newcomers are retained as editors (meaning they return to edit again within 14 days).
  • What is the average number of edits made in the first two weeks after a user registers?
  • What is the proportion of constructive edits (i.e. not reverted within 48 hours) made across a newcomer’s first two weeks?
Measurement Specifications

Positive Reinforcement Measurement Specifications

Personalized praise experiment plan:

The Personalized Praise feature is focused on mentors. There is a limited number of mentors on every wiki, whereas when it comes to newcomers the number increases steadily every day as new users register on the wikis. While we could run experiments with the mentors, we are likely to run into two key challenges. First, the limited number of mentors could mean that the experiments would need to run for a long time. Second, and more importantly, mentors are well integrated into the community and communicate with each other, meaning they are likely to figure out if some have access to features that others do not. We will therefore give the Personalized Praise features to all mentors and examine activity and effects on newcomers pre/post deployment in order to understand the feature’s effectiveness.

In summary, this means we are looking to run two consecutive experiments with the Impact and Leveling up features, followed by a deployment of the Personalized Praise features to all mentors. These experiments will first run on the pilot wikis. We can extend this to additional wikis if we find a need to do that, but it would only happen after we have analyzed the leading indicators and found no concerns.

Each experiment will run for approximately one month, and for each experiment we will have an accompanying set of leading indicators that we will analyze two weeks after deployment. The list below shows what the planned experiments will be:

  • Impact: treatment group gets the updated Impact module.
  • Leveling up: treatment group gets both the updated Impact module and the Leveling up features.
  • Personalized praise: all mentors get the Personalized praise features.

Personalized praise instrumentation rules:

The instrumentation for the Personalized praise feature focuses on the Mentor dashboard and how mentors use the feature. We plan to modify the existing Mentor dashboard instrumentation to enable us to understand the paths mentors take to get to the dashboard, as some of the Personalized praise features link there. Secondly, we propose instrumenting the “Send appreciation” module to capture mentor interactions so as to understand more about their decisions and how that affects newcomers.

When a mentor visits the Mentor dashboard, we want to capture the following information in the mentor_dashboard/visit schema:

  1. The information currently captured by the schema (e.g. timestamp, mobile interface, etc).
  2. A randomised token set on page load to enable connecting events from multiple schemas (similar to homepage_pageview_token in the HomepageVisit and HomepageModule schemas).
  3. The database name of the wiki, because most analytics data uses wiki_db or similar rather than domain names to identify the wiki.
  4. The route the mentor took to reach the dashboard (similar to referer_route in HomepageVisit), to enable understanding whether the mentor clicked on a link in a notification, the post-edit message after sending appreciation, etc. This field does not need to be required.

We also want to capture the following events and information in a new “Send appreciation” module schema, which will be largely similar to how the HomepageModule schema captures events for the Suggested Edits module:

  1. Meta-information about the event: wiki (the database name, not the domain), timestamp, whether the mentor was on the desktop or mobile platform, etc…
  2. A randomised token set on page load to enable connecting events from multiple schemas. This token should match the one in mentor_dashboard/visit for a given page load of the dashboard.
  3. Module impression:
    1. A count of how many praiseworthy mentees were found, and the mentor’s settings of thresholds for determining “praiseworthy”.
    2. If information about a mentee is shown, then record a “mentee impression” event with that metadata: mentee user ID, number of edits made, number of thanks received, longest streak, and number of reverts.
  4. Click on the “more info” icon.
  5. Click on the “X” to close the “more info” overlay.
  6. Click on the “forward” or “back” arrows to move between mentees (if more than one is listed).
    1. Record metadata about the mentee similarly as described above for the initial module impression.
  7. Click on the “cog wheel” icon to open up the settings.
    1. Click on the “X” to close the settings overlay without saving any changes.
    2. Click on the “Done” button to close the overlay and save changes. We also record state changes (previous settings, new settings) for all the settings. For the message subject and the appreciation message itself, we record the length of them (in bytes). For all other settings, we record the previous and new values.
  8. Click on the “Send appreciation” button.
    1. Record the user ID of the mentee to enable connecting it with the displayed metadata.

If the mentor clicks on the “Send appreciation” button and is taken to the mentee’s talk page, we also want to record information about decisions made on the talk page. The functionality to send the message on the talk page is done through DiscussionTools, in order to make the process of sending messages easier. That tool is already instrumented through the EditAttempStep schema and the talk_page_edit schema. Because EditAttemptStep is no longer sampled (it was increased to capturing 100% of edit sessions in T312016), we should have instrumentation of all sessions where mentors are sending appreciation messages for up to 90 days. We therefore do not plan to add any instrumentation to the mentor interactions on the mentee’s talk page.

If the mentor sends appreciation, then the URL in the post-edit message that links back to the Mentor Dashboard should have a referer_route parameter that reflects that this was a post-appreciation message click so this is recorded in the mentor_dashboard/visit schema.

For the “Send appreciation” Echo notification, the URL in the notification should function similarly as the post-edit message in that it sets the referer_route parameter so that we learn if the mentor clicked on the link. This means the link will probably need to be different for a web and email notification.

Acceptance Criteria:

Given a Mentor visits their Mentor Dashboard or views a Personalized praise notification,
When the mentor interacts with a Personalized praise feature,
Then the features are instrumented in a way that will allow evaluation of the proposed hypotheses

Details

Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
KStoller-WMF triaged this task as Medium priority.
KStoller-WMF moved this task from Inbox to Triaged on the Growth-Team board.
KStoller-WMF removed a subscriber: Sdkb.

Change 891368 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[schemas/event/secondary@master] Add analytics/mediawiki/mentor_dashboard/personalized_praise

https://gerrit.wikimedia.org/r/891368

Change 900327 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] [WIP] Personalized praise: Implement instrumentation

https://gerrit.wikimedia.org/r/900327

@Urbanecm_WMF - sorry for the delay on this one! I've added the instrumentation rules from the Measurement Specification doc. I notice that @nettrom_WMF still has a few open questions in the notes section, so please just let us know if there is anything further you need to know here.

@Urbanecm_WMF and @KStoller-WMF : I've completed drafting out the instrumentation in the measurement specification and copied everything over to the task description here. I deleted the "notes" section because all of the notes should now be reflected in the events that we'd like to capture.

There are a couple of open questions:

  1. I'm unsure how the message sending feature on the mentee's talk page is implemented. If it reuses DiscussionTools and event data is already captured in mediawiki/talk_page_edit, we might want to not instrument anything else to reduce complexity. This means that we might have to infer a decision to not send a message by a mentor clicking "Send appreciation" on the dashboard, followed by no further action taken.
  2. I'm unsure whether we should store the mentee's user ID. The proposed instrumentation suggests not doing that, but it does make things a bit more complicated. For example, if we store the user ID then it's easy to connect the impression and "Send appreciation" button clicks using that, rather than a hash. And if the message is sent using DiscussionTools, then we can connect the edit using the mentee's user name.

Let me know if there are questions about anything, as I'm sure I've missed explaining something in the instrumentation because there was a detail I forgot to make explicit.

Thanks for the info @nettrom_WMF and @KStoller-WMF! To answer the questions:

I'm unsure how the message sending feature on the mentee's talk page is implemented. If it reuses DiscussionTools and event data is already captured in mediawiki/talk_page_edit, we might want to not instrument anything else to reduce complexity. This means that we might have to infer a decision to not send a message by a mentor clicking "Send appreciation" on the dashboard, followed by no further action taken.

The "Send appreciation" button only opens DiscussionTools with a message prefilled (regardless of user's DiscussionTools-related preferences). Once DiscussionTools's interface loads, we do not control the workflow anymore. Any instrumentation that normally works for topics added through DiscussionTools should work for Send appreciation button as well. Checking whether mentors tend to click the button without actually sending anything is indeed a good idea.

Should we revisit instrumentation for the talk page (ie. tracking of what happens on the talk page)? I'm afraid that hooking into DiscussionTools could be fairly tricky, and I'd prefer letting DiscussionTools work mostly independently on our features.

I'm unsure whether we should store the mentee's user ID. The proposed instrumentation suggests not doing that, but it does make things a bit more complicated. For example, if we store the user ID then it's easy to connect the impression and "Send appreciation" button clicks using that, rather than a hash. And if the message is sent using DiscussionTools, then we can connect the edit using the mentee's user name.

The proposed instrumentation only suggests to not store the mentee user ID for the notified action, which would be sent whenever mentor received a notification about new mentee(s) to praise. The reason for this is that there is no single mentee ID related to the notification (the notification informs mentors about all mentees currently listed in the Personalized praise module, which might mean more than one mentee).

Other two proposed actions (suggested, which would be sent whenever MediaWiki determined that a particular mentee meets the conditions set by the mentor and praised, which would be sent whenever mentor clicked the "Send appreciation" button) do include the mentee user ID. For all three actions (including the notified one), mentor's ID would be available.

Do you think we should store all relevant mentee IDs even for the notified action?

Questions from myself:

Click on the “Done” button to close the overlay and save changes. We also record state changes (previous settings, new settings) for all the settings except the default message subject. For the message itself, we record the length of the message (in bytes). For all other settings, we record the previous and new values.

That shouldn't be an issue. I thought about reusing prefupdate for this for a while, but I guess that the JSON blob property which includes the settings is a bit too long to include in that schema. Also, I'd like to note that the storage for the default message will be an on-wiki page (namely, an user subpage for the mentor, something like User:Mentor/Default praiseworthy message; the subpage suffix will be the same on all wikis [in English]). Technically, that subpage can be deleted/moved or edited manually (ie. outside of the Mentor dashboard). This is certainly an edge case, but I'm not sure if we need to account for this eventuality in the instrumentation (or perhaps, evaluation of the acquired data).

On a different note: It's not clear to me why we don't want history for the message subject. Out of curiosity, can that be clarified please?

Record the hashed user ID of the mentee to enable connecting it with the displayed metadata.

[minor] It feels there's not much value in the hasing of the mentee ID (since the appreciation takes form of a talk page message, which is publicly visible). I suggest to omit the hashing.

The "Send appreciation" button only opens DiscussionTools with a message prefilled (regardless of user's DiscussionTools-related preferences). Once DiscussionTools's interface loads, we do not control the workflow anymore. Any instrumentation that normally works for topics added through DiscussionTools should work for Send appreciation button as well. Checking whether mentors tend to click the button without actually sending anything is indeed a good idea.

Should we revisit instrumentation for the talk page (ie. tracking of what happens on the talk page)? I'm afraid that hooking into DiscussionTools could be fairly tricky, and I'd prefer letting DiscussionTools work mostly independently on our features.

Since the workflow loads up DiscussionTools and we know that DT is instrumented (with one task for me to follow up on noted below), I think we should not attempt to add more instrumentation to the talk page. I'm happy to go with easy solutions rather than try tricky things.

One question I have about these appreciation messages, will they have a specific edit tag to identify them as Personalised praise? One possible scenario we might run into is that a mentor clicks "Send appreciation" and then sees something on the mentee's talk page that makes them change their mind, but perhaps also makes them add a comment. Would we be able to tell the difference? I'm also not sure how likely we are to encounter that scenario?

One task for me to follow up on is to check in with the Web team to see if the DiscussionTools instrumentation samples or captures all edits.

The proposed instrumentation only suggests to not store the mentee user ID for the notified action, which would be sent whenever mentor received a notification about new mentee(s) to praise. The reason for this is that there is no single mentee ID related to the notification (the notification informs mentors about all mentees currently listed in the Personalized praise module, which might mean more than one mentee).

Other two proposed actions (suggested, which would be sent whenever MediaWiki determined that a particular mentee meets the conditions set by the mentor and praised, which would be sent whenever mentor clicked the "Send appreciation" button) do include the mentee user ID. For all three actions (including the notified one), mentor's ID would be available.

Do you think we should store all relevant mentee IDs even for the notified action?

No, I don't think we should store those. As you mention, the notified action relates to all mentees. For those notifications, I'm more focused on whether the mentor acts on them by going to the dashboard.

Questions from myself:

Click on the “Done” button to close the overlay and save changes. We also record state changes (previous settings, new settings) for all the settings except the default message subject. For the message itself, we record the length of the message (in bytes). For all other settings, we record the previous and new values.

That shouldn't be an issue. I thought about reusing prefupdate for this for a while, but I guess that the JSON blob property which includes the settings is a bit too long to include in that schema. Also, I'd like to note that the storage for the default message will be an on-wiki page (namely, an user subpage for the mentor, something like User:Mentor/Default praiseworthy message; the subpage suffix will be the same on all wikis [in English]). Technically, that subpage can be deleted/moved or edited manually (ie. outside of the Mentor dashboard). This is certainly an edge case, but I'm not sure if we need to account for this eventuality in the instrumentation (or perhaps, evaluation of the acquired data).

When it comes to the default message page being edited outside of the dashboard, I don't think we need to account for that as I also think that's an edge case. Since it's a page we can also tell when it's being edited, and so I'm not concerned that we'll miss something there.

On a different note: It's not clear to me why we don't want history for the message subject. Out of curiosity, can that be clarified please?

Sure, I'll make a note to add something to the task description and the Measurement specification!

[minor] It feels there's not much value in the hasing of the mentee ID (since the appreciation takes form of a talk page message, which is publicly visible). I suggest to omit the hashing.

That sounds reasonable to me! When I update the task description and the Measurement specification, I'll make sure to change that as well.

[...]
Since the workflow loads up DiscussionTools and we know that DT is instrumented (with one task for me to follow up on noted below), I think we should not attempt to add more instrumentation to the talk page. I'm happy to go with easy solutions rather than try tricky things.

Makes sense.

One question I have about these appreciation messages, will they have a specific edit tag to identify them as Personalised praise? One possible scenario we might run into is that a mentor clicks "Send appreciation" and then sees something on the mentee's talk page that makes them change their mind, but perhaps also makes them add a comment. Would we be able to tell the difference? I'm also not sure how likely we are to encounter that scenario?

Good question! T322449: Personalized praise talk page message doesn't ask for a tag, so I didn't plan for one, but having one would certainly make sense. That said, I'm not sure whether the potential tag would help us to tell the difference between a praising message and a non-praising message sent via Personalized praise, since mentors will have the ability to change the message content as they please. In other words: if a mentor closes the preloaded interface and then clicks "Add a new topic" again (opening DiscussionTools for the second time), we'd be able to detect that. However, if a mentor merely empties the preloaded message and replaces it with something totally different, that's more difficult to identify

[...]

The proposed instrumentation only suggests to not store the mentee user ID for the notified action, which would be sent whenever mentor received a notification about new mentee(s) to praise. The reason for this is that there is no single mentee ID related to the notification (the notification informs mentors about all mentees currently listed in the Personalized praise module, which might mean more than one mentee).

Other two proposed actions (suggested, which would be sent whenever MediaWiki determined that a particular mentee meets the conditions set by the mentor and praised, which would be sent whenever mentor clicked the "Send appreciation" button) do include the mentee user ID. For all three actions (including the notified one), mentor's ID would be available.

Do you think we should store all relevant mentee IDs even for the notified action?

No, I don't think we should store those. As you mention, the notified action relates to all mentees. For those notifications, I'm more focused on whether the mentor acts on them by going to the dashboard.

Sounds good!

[...]
Also, I'd like to note that the storage for the default message will be an on-wiki page (namely, an user subpage for the mentor, something like User:Mentor/Default praiseworthy message; the subpage suffix will be the same on all wikis [in English]). Technically, that subpage can be deleted/moved or edited manually (ie. outside of the Mentor dashboard). This is certainly an edge case, but I'm not sure if we need to account for this eventuality in the instrumentation (or perhaps, evaluation of the acquired data).

When it comes to the default message page being edited outside of the dashboard, I don't think we need to account for that as I also think that's an edge case. Since it's a page we can also tell when it's being edited, and so I'm not concerned that we'll miss something there.

Makes sense, thanks!

On a different note: It's not clear to me why we don't want history for the message subject. Out of curiosity, can that be clarified please?

Sure, I'll make a note to add something to the task description and the Measurement specification!

[minor] It feels there's not much value in the hasing of the mentee ID (since the appreciation takes form of a talk page message, which is publicly visible). I suggest to omit the hashing.

That sounds reasonable to me! When I update the task description and the Measurement specification, I'll make sure to change that as well.

Ty!

Change 901539 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[schemas/event/secondary@master] Add fields to analytics/mediawiki/mentor_dashboard/visit

https://gerrit.wikimedia.org/r/901539

Change 901540 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] Log more details on Mentor dashboard visits

https://gerrit.wikimedia.org/r/901540

One question I have about these appreciation messages, will they have a specific edit tag to identify them as Personalised praise? One possible scenario we might run into is that a mentor clicks "Send appreciation" and then sees something on the mentee's talk page that makes them change their mind, but perhaps also makes them add a comment. Would we be able to tell the difference? I'm also not sure how likely we are to encounter that scenario?

Good question! T322449: Personalized praise talk page message doesn't ask for a tag, so I didn't plan for one, but having one would certainly make sense. That said, I'm not sure whether the potential tag would help us to tell the difference between a praising message and a non-praising message sent via Personalized praise, since mentors will have the ability to change the message content as they please. In other words: if a mentor closes the preloaded interface and then clicks "Add a new topic" again (opening DiscussionTools for the second time), we'd be able to detect that. However, if a mentor merely empties the preloaded message and replaces it with something totally different, that's more difficult to identify

I think you bring up a good point that we might end up mislabelling messages if we have an edit tag, as the mentor can easily change the message. While changing the section of the measurement specification that talks about capturing the events on the talk page, it became clear to me that since EditAttemptStep is no longer sampled we will be able to understand the decision making more clearly (e.g. if the mentor cancels the edit and later brings it up again).

In conclusion: I think not having an edit tag will work well for us, so no changes needed there either.

I've updated the task description to reflect the following changes:

  1. We do not hash the mentee user ID but store it directly.
  2. When a mentor makes changes to their settings, we store the state changes of all values, including the length of the appreciation message subject. I could not find a good way to explain why we might want to exclude it, and learning whether mentors make small or large changes to it could be useful.
  3. I've rewritten the section about instrumentation of the mentee's talk page, where EditAttemptStep and the talk_page_edit schemas should be sufficient for our needs as they're capturing all events.

I've also synced the measurement specification on these changes.

Change 901539 merged by jenkins-bot:

[schemas/event/secondary@master] Add fields to analytics/mediawiki/mentor_dashboard/visit

https://gerrit.wikimedia.org/r/901539

Change 901540 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Log more details on Mentor dashboard visits

https://gerrit.wikimedia.org/r/901540

Change 891368 merged by jenkins-bot:

[schemas/event/secondary@master] Add analytics/mediawiki/mentor_dashboard/personalized_praise

https://gerrit.wikimedia.org/r/891368

Change 900327 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Personalized praise: Add server-side instrumentation

https://gerrit.wikimedia.org/r/900327

Change 917951 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[schemas/event/secondary@master] [Growth] Personalized praise: Add database

https://gerrit.wikimedia.org/r/917951

Change 917952 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] PersonalizedPraiseLogger: Update schema version

https://gerrit.wikimedia.org/r/917952

Change 918500 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] [Growth] Add mediawiki.mentor_dashboard.interaction

https://gerrit.wikimedia.org/r/918500

Change 917951 merged by jenkins-bot:

[schemas/event/secondary@master] [Growth] Personalized praise: Add database

https://gerrit.wikimedia.org/r/917951

Change 917952 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] PersonalizedPraiseLogger: Update schema version

https://gerrit.wikimedia.org/r/917952

Change 919218 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[schemas/event/secondary@master] Add pageviews_token to analytics/mediawiki/mentor_dashboard/visit

https://gerrit.wikimedia.org/r/919218

Change 919219 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] SpecialMentorDashboard: Generate pageview tokens

https://gerrit.wikimedia.org/r/919219

Change 919236 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[schemas/event/secondary@master] [WIP] Add analytics/mediawiki/mentor_dashboard/interaction

https://gerrit.wikimedia.org/r/919236

Change 919218 merged by jenkins-bot:

[schemas/event/secondary@master] Add pageviews_token to analytics/mediawiki/mentor_dashboard/visit

https://gerrit.wikimedia.org/r/919218

Change 919219 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] SpecialMentorDashboard: Generate pageview tokens

https://gerrit.wikimedia.org/r/919219

Change 919404 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] [WIP] Personalized praise: Add instrumentation

https://gerrit.wikimedia.org/r/919404

@Urbanecm_WMF : I've reviewed the existing and proposed schemas and compared them to the specification. This is great work! Almost everything looks according to spec from what I can tell and will enable us to answer the questions we have. Here are the two questions I've got:

  1. The Figma designs showed a custom post-edit dialogue when a mentor has completed posting a praise message. Is that part of the implementation? I saw that we discussed above that we hand things off to DiscussionTools once the mentor is on the mentee's talk page, so I'm not sure if that means the post-edit dialogue will be whatever DT does? The reason I ask is that the specification above mentions a post-edit dialogue link back to the Mentor Dashboard and capturing that in the mentor_dashboard/visit schema. This is not a blocker for me because I don't think it's critical to capture this, but I would want to update the measurement specification to reflect what we were able to do.
  2. I'm in favour of using user IDs instead of user names wherever possible, because the former is more stable than the latter. The proposed mentor_dashboard/interaction schema specifies the mentee user name on line 66. Can that be changed to the mentee user ID? I think that's the only place where I noticed it, looked to me like we consistently capture mentee user IDs everwhere else.

In the process of doing this review, I also noticed that the task description hadn't been updated properly to specify that we store the mentee user ID directly. That's what my latest description edit changed. From what I can tell, the schemas already implemented this the intended way so they don't need updating.

The suggested setup with the three schemas (visit, interaction, and personalized_praise) makes sense to me, and as mentioned above should enable us to answer the questions we have. I think having the four actions (suggested, notified, praised, skipped) in the personalized_praise schema collects all of that data neatly in one place. I wouldn't have thought of using that approach myself, so it shows that it's great to have other minds thinking about this too!

Hi @nettrom_WMF,

thanks for the review and feedback!

The Figma designs showed a custom post-edit dialogue when a mentor has completed posting a praise message. Is that part of the implementation? I saw that we discussed above that we hand things off to DiscussionTools once the mentor is on the mentee's talk page, so I'm not sure if that means the post-edit dialogue will be whatever DT does? The reason I ask is that the specification above mentions a post-edit dialogue link back to the Mentor Dashboard and capturing that in the mentor_dashboard/visit schema. This is not a blocker for me because I don't think it's critical to capture this, but I would want to update the measurement specification to reflect what we were able to do.

Unfortunately, this is not yet implemented (currently, the post edit dialog is fully in DT's hand). I do want to implement that though. If that happens, the link would point to Special:MentorDashboard?source=personalized-praise-post-edit (or something similar), which would be captured in referer_route of the analytics/mediawiki/mentor_dashboard/visit schema. Would that be sufficient to provide you with the relevant information?

I'm in favour of using user IDs instead of user names wherever possible, because the former is more stable than the latter. The proposed mentor_dashboard/interaction schema specifies the mentee user name on line 66. Can that be changed to the mentee user ID? I think that's the only place where I noticed it, looked to me like we consistently capture mentee user IDs everwhere else.

Sure thing! I'll change that. Thanks for noticing!

Change 919236 merged by jenkins-bot:

[schemas/event/secondary@master] Add analytics/mediawiki/mentor_dashboard/interaction

https://gerrit.wikimedia.org/r/919236

Change 918500 merged by jenkins-bot:

[operations/mediawiki-config@master] [Growth] Add mediawiki.mentor_dashboard.interaction

https://gerrit.wikimedia.org/r/918500

Mentioned in SAL (#wikimedia-operations) [2023-05-24T13:48:21Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:918500|[Growth] Add mediawiki.mentor_dashboard.interaction (T325117)]]

Change 919404 merged by Urbanecm:

[mediawiki/extensions/GrowthExperiments@master] Personalized praise: Add instrumentation

https://gerrit.wikimedia.org/r/919404

Mentioned in SAL (#wikimedia-operations) [2023-05-24T13:55:27Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:918500|[Growth] Add mediawiki.mentor_dashboard.interaction (T325117)]] (duration: 07m 06s)

Change 922851 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@wmf/1.41.0-wmf.10] Personalized praise: Add instrumentation

https://gerrit.wikimedia.org/r/922851

Change 922852 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@wmf/1.41.0-wmf.9] Personalized praise: Add instrumentation

https://gerrit.wikimedia.org/r/922852

Change 922852 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.41.0-wmf.9] Personalized praise: Add instrumentation

https://gerrit.wikimedia.org/r/922852

Change 922851 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.41.0-wmf.10] Personalized praise: Add instrumentation

https://gerrit.wikimedia.org/r/922851

Mentioned in SAL (#wikimedia-operations) [2023-05-24T15:52:46Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:922852|Personalized praise: Add instrumentation (T325117)]], [[gerrit:922851|Personalized praise: Add instrumentation (T325117)]]

Mentioned in SAL (#wikimedia-operations) [2023-05-24T15:54:19Z] <urbanecm@deploy1002> urbanecm: Backport for [[gerrit:922852|Personalized praise: Add instrumentation (T325117)]], [[gerrit:922851|Personalized praise: Add instrumentation (T325117)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-05-24T16:01:20Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:922852|Personalized praise: Add instrumentation (T325117)]], [[gerrit:922851|Personalized praise: Add instrumentation (T325117)]] (duration: 08m 33s)

Change 922936 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] Personalized praise: Improve instrumentation

https://gerrit.wikimedia.org/r/922936

Verified we're receiving some data in the data lake:

hive (event)> select dt, `database`, module, action from mediawiki_mentor_dashboard_interaction where year=2023 limit 10;
OK
dt      database        module  action
2023-05-24T15:55:20.404Z        testwiki        personalized-praise     impression
2023-05-24T15:55:15.058Z        testwiki        personalized-praise     impression
2023-05-24T15:55:30.651Z        testwiki        personalized-praise     impression
2023-05-24T15:55:49.483Z        testwiki        personalized-praise     impression
2023-05-24T16:19:23.653Z        testwiki        personalized-praise     impression
2023-05-24T16:19:18.322Z        testwiki        personalized-praise     impression
2023-05-24T17:27:44.772Z        testwiki        personalized-praise     impression
2023-05-24T17:41:47.440Z        testwiki        personalized-praise     impression
2023-05-24T17:27:32.358Z        testwiki        personalized-praise     impression
Time taken: 0.362 seconds, Fetched: 9 row(s)
hive (event)>

So far nothing from outside of testwiki. I'll check again in a while, as the interaction schema was only enabled today.

Change 923309 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] MentorDashboardLogger: Exit quietly if EventLogging is unavailable

https://gerrit.wikimedia.org/r/923309

So far nothing from outside of testwiki. I'll check again in a while, as the interaction schema was only enabled today.

Events from other wikis are now available in the data lake (both from today and yesterday), so instrumentation appears to work. Moving to Code Review for few non-critical improvements, then this can be likely closed.

Change 922936 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Personalized praise: Improve instrumentation

https://gerrit.wikimedia.org/r/922936

Change 923309 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] MentorDashboardLogger: Exit quietly if EventLogging is unavailable

https://gerrit.wikimedia.org/r/923309

Unfortunately, this is not yet implemented (currently, the post edit dialog is fully in DT's hand). I do want to implement that though. If that happens, the link would point to Special:MentorDashboard?source=personalized-praise-post-edit (or something similar), which would be captured in referer_route of the analytics/mediawiki/mentor_dashboard/visit schema. Would that be sufficient to provide you with the relevant information?

Appreciate the patience as it took me a while to get back to this. This proposed way of capturing is exactly what I was looking for, so yes, that would be perfectly sufficient! And as I previously mentioned, this isn't a blocker for anything on my end.

@Urbanecm_WMF : From what I can tell, the notifications are being sent out by MediaWiki but not captured as events in event.mediawiki_mentor_dashboard_personalized_praise. The first query below is the Hive/Spark query of the event data, which returns a count of zero. The second I've run for each of the four pilot wiki MariaDB replicas (e.g. analytics-mysql eswiki --use-x1 on a stat* machine), and they all return a non-zero result.

SELECT
    count(1) AS num_notifications
FROM event.mediawiki_mentor_dashboard_personalized_praise
WHERE year = 2023
AND (month = 6
     OR (month = 5 AND day >= 24))
AND `database` IN ("arwiki", "bnwiki", "cswiki", "eswiki")
AND action = "notified";
SELECT
    count(1) AS num_notifications
FROM echo_event
WHERE event_type = 'new-praiseworthy-mentees';

Since the notifications are available in MediaWiki this isn't blocking evaluating the leading indicators in T337069, but I figured it's a bug we'd like to fix as time allows.

Thanks for the ping @nettrom_WMF. You are correct; the notified events are unfortunately emitted invalid and don't make it into the data lake. Elena filled T338078: eventgate_validation_error - 'mentee_id' should be integer and I plan to deploy the fix earlier today. Glad to hear this is not a blocker for leading indicators.

Hi @nettrom_WMF, FYI, the bug you mentioned above should now be fixed.
Notified events should start flowing in soon. Thanks again!