In T325117#8860148, @Urbanecm_WMF wrote:

Unfortunately, this is not yet implemented (currently, the post edit dialog is fully in DT's hand). I do want to implement that though. If that happens, the link would point to Special:MentorDashboard?source=personalized-praise-post-edit (or something similar), which would be captured in referer_route of the analytics/mediawiki/mentor_dashboard/visit schema. Would that be sufficient to provide you with the relevant information?

In T337069#8878424, @KStoller-WMF wrote:

@nettrom_WMF Just FYI, I'm moving this out of unblocked since the remainder of the Personalized praise work was just released: T334630: Personalized praise: Deployment of the new mentor dashboard module to Growth Pilot Wikis.

Hopefully we can look at some initial leading indicators as soon as next week?

With the publication of the results for the Levelling Up leading indicators, as well as the planned leading indicators for Personalized praise on the project page, this task can be resolved.

The findings for all indicators have been posted on wiki here: https://www.mediawiki.org/wiki/Growth/Positive_reinforcement#Levelling_up Closing this task as resolved.

@Urbanecm_WMF : I've reviewed the existing and proposed schemas and compared them to the specification. This is great work! Almost everything looks according to spec from what I can tell and will enable us to answer the questions we have. Here are the two questions I've got:

Reducing the priority of this for now and moving it to "Needs Investigation" while I follow up with @VirginiaPoundstone on what API Platform needs.

I've chatted with @cchen about what the Structured Data team needs so that we're not duplicating efforts, and learned that there isn't currently any overlap between us. Adding this comment mainly to document that the SD team will send out notifications about section-level images to experienced editors. If the Growth team at some point chooses to measure something like edits, we might want to make sure we take that into account.

In T335716#8833119, @Tgr wrote:

@nettrom_WMF would you like separate schemas (where it makes sense, e.g. mediawiki/page/image-suggestions-feedback) or should we just plug in a section ID into the schema, and set it to null for the old image recommendation task type? Do you want a specific format for the section ID (ordinal, section title text etc)?

Adding the Product Analytics tag and resetting the priority so that Product Analytics can triage this tomorrow. Since this is planned for late May/early June the work needed on this at the moment is more of a planning nature, which to me fits into PA's "medium priority".

We've examined this phenomenon in several different ways looking for patterns, and not found any specific pattern that provides clues as to why this is happening. This analysis started out by looking at the three breakdowns suggested in the task, as well as by wiki since it was trivial to split the data that way too. Here's what we found:

In T334432#8773117, @BCornwall wrote:

@nettrom_WMF Are you the approving party (manager) of @KMorgan-WMF?

I've gathered all the data from English Wikipedia and in the process fixed a bug in our processing so that we also correctly count mobile page views after the desktop to mobile redirect. We do not report data for the French and Dutch campaigns because of low numbers.

Reducing the priority of this to Medium as the majority of this work is done and not as urgent. I'll also remove the due date.

The leading indicators have been defined and I've created a notebook to aggregate data and set it up to run once a day. We also have a Superset dashboard to monitor the indicators. What remains is:

In T331495#8760478, @Pcoombe wrote:

Those seeing the Thank You page in English do get a link to create an account on the English Wikipedia e.g. in Sweden the English language Thank You page links to https://en.wikipedia.org/w/index.php?title=Special:CreateAccount&campaign=typage-nwe-sv-2023.

This seemed to make more sense for the user, and is consistent with what we previously did in Latin America. I mentioned this in Slack, but I don't think the links in T321370 got updated sorry.

Editing this to remove the posted data because the more recent table posted below is the authoritative one.

@Ottomata : as you've discovered, @Trizek-WMF is already assigned to the wmf group and is WMF staff (and also subscribed to this task).

In T331495#8735607, @HNordeenWMF wrote:

To make sure I understand your point "we'll have to look at the combined metrics since there's no language data". Does this mean we won't be able to know for example English-France registrations, we'll only be able to know total France registrations (en, fr) as a portion of total France TY Page views? That seems ok.
I'm assuming we'd still be able to look at total English registrations (all countries) as a portion of total English TY Page views (all countries) too? Or do we also lose that ability.

In T331495#8730997, @KStoller-WMF wrote:

@nettrom_WMF - My sense is that we did more in-depth analysis for the initial experiment, and at this point it should be OK to simplify and just look at combined metrics.
But if you or @HNordeenWMF think splitting the data by language is definitely needed, feel free to speak up.

This work has been done multiple times over since Newcomer Tasks was deployed, although not documented here. Closing as resolved.

@HNordeenWMF : I'm currently working on gathering these statistics and noticed that T321370 specifies two languages for each campaign: the local language of the wiki as well as English. The language is then reflected in the uselang parameter in the URL. When grabbing the data I noticed this parameter doesn't show up (except for a few test visits), and from playing around with the Thank You page I did not see the uselang parameter in the link to account creation. I'm not sure if this is a bug or not?

This analysis has been completed. We created a slide deck presenting the findings and decided to upload that to Commons for wider accessibility: https://commons.wikimedia.org/wiki/File:T332441_-_Thanks_Usage_-_Introduction.pdf The notebooks used for data gathering and visualization are in this GitHub repository

I've updated the task description to reflect the following changes:

In T325117#8706515, @Urbanecm_WMF wrote:

In T325117#8706061, @nettrom_WMF wrote:

One question I have about these appreciation messages, will they have a specific edit tag to identify them as Personalised praise? One possible scenario we might run into is that a mentor clicks "Send appreciation" and then sees something on the mentee's talk page that makes them change their mind, but perhaps also makes them add a comment. Would we be able to tell the difference? I'm also not sure how likely we are to encounter that scenario?

Good question! T322449: Personalized praise talk page message doesn't ask for a tag, so I didn't plan for one, but having one would certainly make sense. That said, I'm not sure whether the potential tag would help us to tell the difference between a praising message and a non-praising message sent via Personalized praise, since mentors will have the ability to change the message content as they please. In other words: if a mentor closes the preloaded interface and then clicks "Add a new topic" again (opening DiscussionTools for the second time), we'd be able to detect that. However, if a mentor merely empties the preloaded message and replaces it with something totally different, that's more difficult to identify

Found this after my curiosity made me check if log_search is available on the Wiki replicas. A couple of fields that might be considered non-private are:

I got curious and checked whether log_search is available on the standard replicas, and it isn't because it contains private data. T85756 exists to also make it available more widely with private data redacted. If it's sqooped it would need to be marked as private, similarly as some of the other tables we have.

Moving this to the Product Analytics kanban board and the Growth Team's current sprint board as this work will need to start either this week or next.

Reopening this because this regular expression:

Hi and thank you for your interest! Please check thoroughly https://www.mediawiki.org/wiki/New_Developers (and all of its communication section!). The page covers how to get started, assigning tasks, task status, how to find a codebase, how to create patches, where to ask general development questions and where to get help with setup problems, and how to ask good questions. Thanks a lot! :)

Adding this to the Product Analytics kanban board, keeping the High priority. The plan is to start this work on March 20 to get an initial perspective on the usage of Thanks as it relates to edits made in the most recent months, then go on from there based on what our needs are.

In T325117#8704159, @Urbanecm_WMF wrote:

The "Send appreciation" button only opens DiscussionTools with a message prefilled (regardless of user's DiscussionTools-related preferences). Once DiscussionTools's interface loads, we do not control the workflow anymore. Any instrumentation that normally works for topics added through DiscussionTools should work for Send appreciation button as well. Checking whether mentors tend to click the button without actually sending anything is indeed a good idea.

Should we revisit instrumentation for the talk page (ie. tracking of what happens on the talk page)? I'm afraid that hooking into DiscussionTools could be fairly tricky, and I'd prefer letting DiscussionTools work mostly independently on our features.

@Urbanecm_WMF and @KStoller-WMF : I've completed drafting out the instrumentation in the measurement specification and copied everything over to the task description here. I deleted the "notes" section because all of the notes should now be reflected in the events that we'd like to capture.

In T330637#8698121, @kostajh wrote:

It seems there's another problem, though, because I see multiple impression events for the post-edit dialog after dismissing the try new task panel:

Adding myself as a subscriber in case we do decide to create a new table. That would require the Welcome Survey aggregation script to be updated to use the new data source. It should not be seen as an argument for or against any particular solution, instead it's just a change in data that I'll need to adapt to. Carry on.

In T330637#8687140, @kostajh wrote:

I had just completed my 5th copyedit when I received this panel. The next task type I should try is "links". The edit-count-for-task-type is 4 because that is the value shown when I opened the page. I should probably revise that to show 5.

I've dug into this phenomenon and found as follows:

In T322587#8679879, @kostajh wrote:

@KStoller-WMF @nettrom_WMF this one does not need to be gated behind the LevelingUp feature flag, nor hidden from users in the oldimpact experiment group, is that correct? (Put another way, it can be merged and released to everyone when it's finished, correct?)

In T330637#8660470, @kostajh wrote:

@nettrom_WMF I didn't implement the feature in this way in T322386: Leveling up: "Try a new task" dialog. The user will only be prompted to try a new task if there is at least one available task matching the new task type and their existing topic filters. I'll ask in T322386 if that's acceptable for this iteration.

In T330727#8655402, @Tgr wrote:

@nettrom_WMF FWIW it doesn't seem like events right before navigation (such as button click events when the button is a link or JS redirect) get logged. This is not specific to this dialog (I can see that the dialog triggers the mw.track event but then EventLogging doesn't react to it) - maybe caused by the EventLogging library rewrite a while ago? From skimming the metric library code the handling of this scenario seems solid, so not sure what's going on.

I assume in this specific case it is not super important as there will be a homepagevisit event right afterwards anyway.

In T330727#8651445, @Tgr wrote:

@nettrom_WMF does that sound OK? Especially the type field in action_data that differentiates post-edit notices after suggested vs. non-suggested edits, so far we only had it in the impression action - do you want to stick with that and use the session ID to match further events, or put type into the action data for all events to differentiate them from normal post-edit? (Or maybe use a new context value like postedit-nonsuggested? That seems the cleanest to me.) Currently for link-click actions the action data is just the link name, without any key-value structure; we should probably change that (here and elsewhere for consistency) if we want to add another piece of information for link click events.

This task came up in the Product Analytics team's board refinement meeting, and I wanted to chime in and confirm that the Growth team doesn't use this instrumentation in our analyses. Or in other words, from the perspective of the Growth team this instrumentation is safe to decommission.

The team decided first to stop experimenting on smaller wikis, then removed the control group altogether in T320876. No need to keep this task open.

The Levelling up instrumentation section in the measurement specification is now complete and ready for review and/or implementation. I'll be in charge of updating the measurement specification based on any changes that come out of the development process.

One thing I noticed when I was just checking the mocks is that the proposal is to show the dialogue on every k edits (with k=5) as the default. So if I've made 10 add a link edits, I should see the dialogue again (provided I haven't checked "don't show me this again"). Are we still planning on doing that or have these changed to milestones that trigger the dialogue and then never again?

We've done a first pass of this analysis by looking at users who exit the topic selection dialogue on wikis where this feature has been available. While that doesn't capture users who turn the feature on during the module initialization process, our findings suggested changes to the UX was needed (ref some of the related tasks from December and January) to improve the user experience. This conclusion wouldn't change if we extended the dataset, and so we decided to instead prioritize other work.

I dug into this as part of T322433 and found that the main limiting factor here is the threshold, and not time. The figure below exemplifies this, where the X-axis is time since first edit tagged "newcomer task" in hours, the Y-axis is the proportion of editors making a threshold, and each line is a different threshold (this figure is also on Commons as: Newcomer_task_milestones_by_hour_Feb_2023.png) It doesn't matter if we change the scale of the X-axis to minutes, hours, or days, the graph looks basically the same.

This analysis is largely complete. We have an internal googledoc report, and I've gone ahead and created this on-wiki summary that goes through the main findings and has a couple of graphs to support it.

@KStoller-WMF @RHo : I noticed one thing about this notification when I was working on investigating baseline for this and checked the Figma designs: if we define this as written then the longest possible interval a newcomer can go between making their first suggested edit and the notification being sent is 9 x 120 hours, which is 1080 hours or 45 days. Or say a newcomer activates by making a suggested edit on the first day, then they can wait 4 days between each of the 8 subsequent edits and get the notification 5 days after that for like 37 days total.

In T327849#8568496, @kostajh wrote:

@nettrom_WMF do you think we should log something in homepagemodule schema to indicate if the user's browser has Intl support?