Page MenuHomePhabricator

Newcomer tasks: instrumentation
Open, MediumPublic

Description

We will need to instrument the newcomer tasks feature to facilitate a controlled experiment and to answer our high-level research questions about the feature.

The measurement plan for newcomer tasks is here: https://docs.google.com/document/d/1dEE_XicyBWOKPu8op4t6neIHncSqlxubUgFa82bZQxI/edit#


An important element is to assemble our written measurement plan. We will go through these steps in this order:

  • @nettrom_WMF starts a draft in a Google Doc.
  • @MMiller_WMF collaborate to refine the draft.
  • @JTannerWMF circulates this draft to various approvers to begin to inform them and receive initial feedback.
  • At the same time, the team starts to comment on the draft and @nettrom_WMF and @MMiller_WMF will incorporate changes. We will set a cutoff date when all changes should be complete.
  • Implementation begins. Engineers may begin implementation before this step if there is sufficient confidence about parts of the plan.
  • @JTannerWMF completes the approval process.

Related Objects

Event Timeline

A note from a conversation with @nettrom_WMF -- in analyzing the newcomer tasks impact, we may want to look at non-reverted edits, as opposed to just edits. The feature directly causes edits, but the question is whether they are valuable edits for the wiki.

kostajh moved this task from Inbox to Q1 2019-20 on the Growth-Team board.Aug 19 2019, 12:12 PM

Seems like something for us to sort out in Q1 but @MMiller_WMF feel free to move.

nettrom_WMF triaged this task as Medium priority.
nettrom_WMF moved this task from Incoming to In Progress on the Growth-Team (Current Sprint) board.
nettrom_WMF moved this task from Triage to Doing on the Product-Analytics board.
nettrom_WMF updated the task description. (Show Details)

The draft measurement plan is being worked on, so I've claimed this task, updated tags, and moved it to the right columns.

nettrom_WMF moved this task from Doing to Review on the Product-Analytics board.

Draft measurement plan has been created, reassigning to @MMiller_WMF for reviews and revisions.

@MMiller_WMF let me know when I can circulate for the instrumentation DACI or if you want to revise it first.

@JTannerWMF -- @nettrom_WMF and I are working on the measurement plan this week, and it will be ready to go to our team for review at the beginning of next week. At the same time, it will be ready for the DACI process.

MMiller_WMF updated the task description. (Show Details)Oct 18 2019, 10:21 PM

The measurement plan is now ready for implementation, and has been circulated for approvals. It is here: https://docs.google.com/document/d/1dEE_XicyBWOKPu8op4t6neIHncSqlxubUgFa82bZQxI/edit#

MMiller_WMF added a project: Epic.

Moving to the Epics in Progress column because all work will happen on subtasks.

Tgr added a comment.Nov 26 2019, 12:07 AM

On chat, we discussed whether the current approach to identifying newcomer tasks related actions outside the homepage is the right one. (This affects instrumentation, edit tagging, and in the future showing in-context help.) I'll try to summarize it here:

  • Currently we are doing client-side tracking: add an identifier to the task card link, and then pass that along through various edit-related client-side logic (VisualEditor, MobileFrontend editor) and finally pass it along with edit API calls (this part is not implemented yet). The identifier could be a key that can be looked up on the server side to fetch task details, or the task details could be encoded into it, that's a relatively minor implementation issue, but the way we preserve our knowledge that what's currently happening in the browser tab is part of a workflow that was initiated by the user clicking on a given task card would be by preserving some state in the browser window across various changes (like switching between the different editors, switching between edit / read, maybe doing multiple consecutive edits).
  • The alternative would be server-side tracking (in a somewhat weaker sense of tracking): keep track of what task card the user has clicked on, but don't try to follow their flow of actions (not really possible on the server side), just detect when they do something to a page the card of which they have clicked on before (within some time limit, as we have to expire the information from the server at some point). When the information is needed on the client side (e.g. for showing a task-specific help panel), propagate it from the server by JS globals or some dedicated API.

Implementation-wise, the second approach seems far more efficient as the first would involve changing a component (or multiple components, especially for things involving multiple alternative editor interfaces) not owned by us every time we want to extend what kind of workflows we are tracking. Do we want to handle the user switching to editing and back to reading, or visiting the talk page and coming back, or visiting their mentor's talk page and coming back, without losing track of the fact that this is part of a suggested edit workflow? Each of those would be a dedicated code change with client-side tracking. With server-side tracking, it would be handled naturally (we wouldn't learn the fact that they visited the talk page etc., but the edit done at the end would be identified as being the result of a suggested edit workflow.)

That ease of use would come with some loss off accuracy: on one hand there would be an expiry (while with client-side tracking we can preserve the state as long as the browser tab is kept open) so if we set that to one hour and the user spends 70 minutes rearranging sentences before clicking save, we are out of luck. On the other hand, we would have false positives for users finding their way to the task editing interface via some different means (maybe the click a task card, then switch to another browser tab, go to recent changes, and click the same article from there) - not really sure if that's a bad thing or a good thing. More generally, we'd lose the ability to differentiate between interactions flows happening in different browser tabs.

Overall, switching to a server-side approach seems like a good trade-off to me; it would make handling suggested edits related actions outside the home page easier and faster to implement, while the differences are mostly corner cases. But it's fundamentally a product/analytics call.

I think this is partly a design issue, which @RHo should chime in on, and partly a measurement issue. With regards to the design part, I'm trying to think ahead to how things might work with guidance. If the user is recommended the Egg tart article, clicks through to it, goes somewhere else, and then later returns to it, should they again see the guidance (meaning we treat them as if they came through from the Homepage)? I think the answer to that affects how we connect the Homepage schema to the Help Panel and EditAttemptStep schemas in that situation.

With regards to the current state, from what I can tell there are two key parts in the measurement plan where this workflow comes in:

  1. The user clicked "Edit" on the recommended article.
  2. The user saved an edit on the recommended article.

We use the EditAttemptStep schema for the first part (joined with HomepageModule through the tokens), and since it's a leading indicator as well it's a key measurement to keep track of. Here I'm mainly interested in users opening the editor after coming directly from the Homepage, anything happening later (e.g. because they click edit->view->edit is less of a concern to me).

For the second part, the measurement plan isn't particularly specific on what editing a recommended article means in terms of temporality. I'm also not particularly worried about what users might do in the meantime, similarly to what @MMiller_WMF mentions in T236885#5691823. It's convenient if it's possible to join with EditAttemptStep again, but if what we end up with is an edit tag, that's also fine.

If we end up implementing this then the 60 minute threshold is fine, it's the commonly identified between-edit session threshold on Wikipedia anyway.

MMiller_WMF updated the task description. (Show Details)Nov 27 2019, 2:04 AM
RHo added a comment.Nov 27 2019, 11:39 AM

I think this is partly a design issue, which @RHo should chime in on, and partly a measurement issue. With regards to the design part, I'm trying to think ahead to how things might work with guidance. If the user is recommended the Egg tart article, clicks through to it, goes somewhere else, and then later returns to it, should they again see the guidance (meaning we treat them as if they came through from the Homepage)? I think the answer to that affects how we connect the Homepage schema to the Help Panel and EditAttemptStep schemas in that situation.

My expectation/hope would be that we should show the guidance to a user if they return to a suggested article within a given timeframe. One use case that comes to mind is a user browses through suggestions on their mobile device and decides to work on the article on Desktop. Another is a user opening up a number of suggestions to by going back and forth on their homepage, then deciding to edit one of the suggestions by navigating there via the global search. All this to say, I would prefer the method that allows more accuracy and detail on the actions users got to the suggested edit, since this will help with improving the design (e.g., if we see people opening multiple suggestions on multiple tabs to review, the design might shift from being one suggestion card to multiple suggestions per 'page' in the module).

If we end up implementing this then the 60 minute threshold is fine, it's the commonly identified between-edit session threshold on Wikipedia anyway.
Another is a user opening up a number of suggestions to by going back and forth on their homepage, then deciding to edit one of the suggestions by navigating there via the global search.

Maybe it should be a much longer window, like 1 week?

Another is a user opening up a number of suggestions to by going back and forth on their homepage, then deciding to edit one of the suggestions by navigating there via the global search.

A related scenario is that a user lands organically on an article that has maintenance templates that would include it in the suggested edits module, but that user didn't navigate to the article via suggested – they never saw it there. Would the guidance still show in this case? And if so, would an edit made on this also be tagged as a "suggested edit"? I think the answer is "yes" to both but just wanted to double check.

RHo added a comment.Nov 27 2019, 12:20 PM

A related scenario is that a user lands organically on an article that has maintenance templates that would include it in the suggested edits module, but that user didn't navigate to the article via suggested – they never saw it there. Would the guidance still show in this case? And if so, would an edit made on this also be tagged as a "suggested edit"? I think the answer is "yes" to both but just wanted to double check.

This is one of the open questions in the v1.2 planning doc – I agree that the answers should probably be yes, but would want us to be able to show source as via the homepage/SE module vs organically. Another consideration is this would affect the design and copy a little – we could either provide more context and explanation if it is detected that a user got to the article not from Suggested edits, or make the task guidance like this for everyone (re-explaining a lot more about what they are seeing).

I agree that the answers should probably be yes, but would want us to be able to show source as via the homepage/SE module vs organically.

For the purposes of guiding, yes we could definitely distinguish between the two. But I'm wondering specifically if you'd want the "newcomer tasks" edit tag to appear, or if there would be a second tag (like "organic 🍎 newcomer tasks") for when the user happened upon a an article by chance, and followed the guidance to successfully do the newcomer task, but did not start their journey from Special:Homepage.

RHo added a comment.Dec 3 2019, 6:10 PM

I agree that the answers should probably be yes, but would want us to be able to show source as via the homepage/SE module vs organically.

For the purposes of guiding, yes we could definitely distinguish between the two. But I'm wondering specifically if you'd want the "newcomer tasks" edit tag to appear, or if there would be a second tag (like "organic 🍎 newcomer tasks") for when the user happened upon a an article by chance, and followed the guidance to successfully do the newcomer task, but did not start their journey from Special:Homepage.

Will defer to @MMiller_WMF and @nettrom_WMF, but maybe a separate second tag is easier for analysis?

@kostajh @Tgr @RHo @nettrom_WMF -- thanks for your patience while it took me time to get to this conversation. My first question is: this proposal to switch from client-side to server-side, this is sort of a potential brainstorm discussion, right? There isn't a current urgent reason we need to pursue it on any specific timeframe, right?

Here are my thoughts:

  • I think we should err on the side of being more inclusive about what "counts" as a suggested edit. If a user edits an article that they've seen as a suggestion, I am pretty confident that those two things are related (especially if they happen within a certain time of each other, like a day). All the wikis we're working with have tens of thousands of articles, so the chances of coincidence are low.
  • I don't think I'm equipped to specify whether we should stick with client-side or switch to server-side, but I hope that this conversation has given the engineers enough information to decide.
  • Some of this discussion started becoming about feature specifications, instead of instrumentation needs. I feel like we will probably need to migrate these thoughts elsewhere. @RHo, how about in the "Newcomer tasks (guidance) -- planning" document?
    • If a user is looking at an article that has a maintenance template on it, but did not arrive there from the suggested edits module and did not recently see that article as a suggested edit, it would probably be annoying for the help panel to pop up in read mode, assuming the user wanted to edit the article. They could easily just be reading it. In some of our target wikis (like Arabic), a solid quarter or third of articles have some maintenance templates on them. So if we don't want to be intrusive, in this scenario we could let the help panel open up in "suggested edits mode" only once the user clicks edit on the article.
    • On the other hand, this could be a great opportunity to encourage serendipitous edits to articles that users are actually interested in, because they're reading them! Maybe we could design an experience where the help panel has some in-between affordance that conveys, "Hey, open me up, because you can edit this very article!"
    • Then, depending on how we want this experience to work, we could make the edit tags fit it.
MMiller_WMF closed subtask Restricted Task as Invalid.Jan 24 2020, 9:44 PM
MMiller_WMF removed a subtask: Restricted Task.