Page MenuHomePhabricator

Move the "copyedit" template-based task to medium
Open, MediumPublic

Description

Gated by the feature flag from T401192, we want to move the copy-edit template-based task to medium difficulty. It should then also no longer be enabled by default.

Open Questions:

  • what happens to existing newcomers who had that task enabled?
  • should we change the description of the task to NOT include "tone" so there is less overlap with Revise tone?

Event Timeline

KStoller-WMF lowered the priority of this task from High to Medium.

In my opinion, this requires a wider discussion/clarification. As far as I know, we will want to deploy the Improve Tone task as an experiment. If we also move copyedits one level up, we'd need to decide whether:

  1. we move the task only for the treatment group (resulting in two groups of users having the same task in two difficulty levels), or
  2. we move the task for everyone on the wiki (resulting in the control group no longer representing the status quo, which might mislead the experiment results)

In addition to that, reclassifying the task for the experiment would also mean we'd be pushing it back to Easy once the experiment concludes. Given our experiments are relatively short (and we want them to be even shorter), this would be noticeable from the user perspective as well.

I believe it would make more sense to do this change only when we start treating the new task as a new feature rather than an experiment. If we want to focus user attention on Improve Tone rather than copyediting, it might make more sense to change our default task type filters (so the task type wouldn't be checked by default, only when a user selects it intentionally).

This topic was briefly discussed in todays growth meeting as well. Given its uncertainty, I am moving this to the Needs Product column for @KStoller-WMF to reconsider/respecify as appropriate.

Thanks for bringing up this complexity!

Given the fact that we aim to experiment quickly and we don't want to mislead experiment results I think in the short term we should NOT move Copyedit to a Medium difficulty task. (Although after the experiment we may want to revisit this decision).
FYI @AAlhazwani-WMF

What I'm less certain about is if for the experiment we should show Copyedit tasks to both the treatment and control or if we want to "hide" the Copyedit task from the treatment who will have access to the Improve Tone task instead.

I'll chat more with @Iflorez and @Trizek-WMF since this decision has both experiment and community experience implications.

Several points to consider:

  • We received feedback in the past regarding Copyedit, where users asked to have it moved to the medium level. Should we act on this and move the task for everyone, no matter how the edit is provided (template or model)?
  • If we move that task, what will populate the Easy section? Add a link is not available everywhere (not deployed, not activated).
  • Again: is the easy /medium / hard hierarchy the best?

After discussion with @Iflorez, I suggest we proceed with simply "hiding" the Copy edit task for users who land in the experiment treatment group while we A/B test this. This would mean:

  • The Control group receives the existing status quo experience
  • The Treatment group receives "Add a Link" and "Improve Tone" as Easy tasks, which most closely mirrors the future user experience we have planned if this experiment proves to be successful.

I've started to outline this and other known requirements in this experiment set-up task: T402707: Set up A/B test for “Revise Tone” Structured Task
(But please chime in if you think a separate task or approach is needed).

If we move that task, what will populate the Easy section? Add a link is not available everywhere (not deployed, not activated).

The goal is that, over time, only Structured Tasks will make up the Easy category, ensuring that newcomers’ first Suggested Edits are truly “easy” to complete. If Improve Tone proves successful in helping new editors get started, then Improve Tone and Add a Link would become the default Easy tasks in the languages where they are available.

Again: is the easy /medium / hard hierarchy the best?

Amin and I discussed this "leveling up" framework recently. In discussions with other designers there was general agreement that the current easy / medium / hard hierarchy is the clearest option. Changing it would also require substantial copy and translation updates across multiple places, and the potential benefits did not seem to outweigh the effort and disruption involved.