Page MenuHomePhabricator

Allow quicksurvey to target based on edit count
Closed, ResolvedPublic5 Story Points

Description

Various teams (Wikidata, Wikimedia Execs, Audiences, Research) are interested in running surveys that could target:

  • Editors only
  • Editors with certain edit counts
    • Bucketed in the same way that we do with the Popups schema (see the editCountBucket property)? @Jdlrobson points out that this is already the case.
  • Non-editors only

AC

A survey can be configured to:

  • Target anonymous users (wgEditCount === null) and logged-in users without edits (wgEditCount === 0)
  • Target a non-editor (wgEditCount === 0)
  • Target an editor (wgEditCount > 0)
  • Target a user with an edit count that falls into a given range -- e.g., 5-20 edits

Descoped

  • The user's bucketed edit count will be recorded when the user answers the survey.
    • Right now this can only happen for InternalSurvey's. I'm not sure if it's needed for ExternalSurvey's but if it is, this is quite a significant change and requires its own task.
      • Note from @Isaac : The QuickSurveysResponses schema does record a bucketed edit count for all surveys. For InternalSurveys, this edit count is directly associated with a respondent's answer to the survey. For ExternalSurveys, this edit count is just associated with a respondent selecting "Yes" to take the survey. The information can then be joined post-hoc via pageview tokens that are passed along to the survey.

Notes

  1. Answers to "internal" surveys are recorded via the QuickSurveysResponses schema.
  2. There are really three classes of users that can be sampled via this functionality: anonymous users (not logged in), non-editors (logged in but no edits associated with their account), and editors (logged in and at least one edit). Anonymous users are treated as having zero edits unless minEdits is set, in which case anonymous users are excluded. See T186737 for functionality aimed more directly at logged-in vs. not logged-in.

Developer notes

The users bucketed edit count is already inspected and recorded in the schema, so this change would only need to update the behavior for showing the survey.

To support this kind of surveys we'd need to add an optional field to a survey definition inside QuickSurveysConfig that sets a minimum edit count.

This field would then need to be inspected inside the showSurvey method.

Given the team hasn't worked on this code, there is likely to be an added tax as we regain familiarity with the codebase and how it works including setup/testing which may bump the estimation.

While working on this code, it might be a good opportunity to modernise some of the code - for instance replace $.each usages with Array.prototype.forEach.

Comments

@JKatzWMF: If it's trivial, being able to target by edit bucket would also be very useful for future surveys (on T197631: Add binary editor targeting to quick surveys tool).

QA steps

Make sure to test these on desktop site. There is a bug with mobile (T218243)!

A survey has been setup to only show for users with edit count between 1 and 22

Visit https://reading-web-staging.wmflabs.org/wiki/Test_the_diff in an incognito window

  • Verify the survey doesn't show if you are logged out
  • Verify the survey doesn't show if you are logged in and have an edit count of 0
  • Verify the survey DOES show if you are logged in and have an edit count of 1
  • Verify the survey DOES show if you are logged in and have an edit count of 20
  • Verify the survey doesn't show if you are logged in and have an edit count of 23

Survey should look like

QA Results

StatusDetails
✅ PassedT139317#5023065
NOTE: This only works with the desktop site. It fails with mobile.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ovasileva moved this task from Needs Analysis to Upcoming on the Readers-Web-Backlog board.

Change 493130 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/extensions/QuickSurveys@master] POC: Allow surveys to define an audience

https://gerrit.wikimedia.org/r/493130

I'm looking for some feedback on my pseudo code and approach before fleshing out the contents of the code.

Jdlrobson added a subscriber: Isaac.

Edit Counts:

  • Right now, you can do the equivalent of sampling on users with edits less than some maximum (e.g., <5 edits), between a minimum and a maximum (e.g., 5-10 edits), and above a minimum (e.g., <10 edits), which uses the actual edit count as opposed to the pre-set buckets like "1-4 edits" reported in EventLogging (thanks for this!)

Correct. I did consider using the preset buckets but figured using minEdit and maxEdit would give the most flexibility to survey editors.

  • I don't see a way right now to exclude anonymous users through the edit count though (looking at the isInAudience function at https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/QuickSurveys/+/493130/5/resources/ext.quicksurveys.lib/lib.js@133). Anonymous users seem to be treated as having zero edits: mw.config.get( 'wgUserEditCount' ) returns null when not logged in but null is treated as 0 when doing greater than / less than comparisons and there is no check for null values of the editCount variable. Is there plans to have a separate check for logged-in vs. not-logged-in? Or am I missing some way of excluding them through the editCount variable?

I didn't consider this as part of this task as when we talk about edit count we are talking about logged in users (since we have no way of knowing an anons account). I've setup T217415 - take a look!

I didn't consider this as part of this task as when we talk about edit count we are talking about logged in users (since we have no way of knowing an anons account). I've setup T217415 - take a look!

It seems we got a bit confused in terms of the bucketing. My interpretation of "target a non-editor" was to target exclusively anons, but you're right, as-is it's everyone with no edits. In this case, let's get T186737: Let me choose whether to present a survey to logged-in or logged-out editors estimated and pull it in next grooming.

The user's bucketed edit count will be recorded when the user answers the survey.

@Isaac Is this needed? As written it seems to be thinking about internal surveys, however, it looks like you are planning to run an external survey and external surveys have no way (currently) to gain information about the user. If we want to copy that to the survey we'll need to work something out using query strings and I suggest we open a task to define that in detail.

Jdlrobson updated the task description. (Show Details)Mar 4 2019, 7:12 PM
Jdlrobson updated the task description. (Show Details)
Isaac added a comment.Mar 4 2019, 8:38 PM

@Jdlrobson: I see what you mean and I didn't write the statement but I don't think the intent was to pass the edit count to the survey in the same way we do the pageviewToken via instanceTokenParameterName but to just ensure that it was recorded in the eventlogging. I really can't think of any scenario in which it would be useful to pass the edit count to the survey and just have different surveys each targeting a different range of edit counts should solve that problem if the edit count really was needed.

Regarding recording edit count: from my standpoint, between the edit count bucket that is already logged (https://meta.wikimedia.org/wiki/Schema:QuickSurveysResponses) and the min/max edit count values that would be part of the survey config, no additional work would be needed.

regarding recording edit count: from my standpoint, between the edit count bucket that is already logged

you are correct. THanks for clarifying!
Okay, so I think https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/QuickSurveys/+/493130/ is ready to go in my opinion.

Change 493130 merged by jenkins-bot:
[mediawiki/extensions/QuickSurveys@master] Allow surveys to define an audience of editor buckets

https://gerrit.wikimedia.org/r/493130

Please ping me for QA steps after you've QAed T216746 and I'll set you up with a testing environment.

Please ping me for QA steps after you've QAed T216746 and I'll set you up with a testing environment.

@Jdlrobson Ping!

Jdlrobson updated the task description. (Show Details)Mar 7 2019, 7:22 PM

(This is ready to test now!)

Edtadros reassigned this task from Edtadros to Jdlrobson.EditedMar 11 2019, 3:22 AM
Edtadros added a subscriber: Edtadros.

Test Result

Status: ⬜ Not Completed
OS: macOS Mojave
Browser: Chrome DevTools Device Emulator (iPhone X)

Test Artifact(s):

  • Verify the survey doesn't show if you are logged out

✅ PASS

  • Verify the survey doesn't show if you are logged in and have an edit count of 0

✅ PASS

  • Verify the survey DOES show if you are logged in and have an edit count of 1

✅ PASS
03/10/2019 - Jdlrobson, I created a new user and made 1 edit. I do not see the survey. It would be helpful to know what to look for. Please provide a screenshot/description of what the survey looks like. This is what I see. I'm holding off on failing this step until I'm sure that it's not there and I'm missing it due to user error on my part.

03/13/2019 - This passes now. The survey shows up once the beta opt-in question is dismissed, either by clicking OK or Cancel.

  • Verify the survey DOES show if you are logged in and have an edit count of 20

❌ Fail

3/13/2019 - Tested with user Edtadros with 20 edits.

  • Verify the survey doesn't show if you are logged in and have an edit count of 23

⬜ Not Run

I just ran throught the QA steps and this should be working. Make sure to use an incognito window to avoid issues with localStorage, but let me know if this is still not showing and I'll debug some more.

@Jdlrobson the survey didn't show up when I had 20 edits. I logged the user (Edtadros) out and back in. Please see the failed step in T139317#5014245

Let's do another sync today! Have sent you a calendar invite for after standup.

Test Result

Status: ✅ PASS
OS: macOS Mojave
Browser: Chrome (desktop site)

Test Artifact(s):

  • Verify the survey doesn't show if you are logged out

✅ PASS

  • Verify the survey doesn't show if you are logged in and have an edit count of 0

✅ PASS

  • Verify the survey DOES show if you are logged in and have an edit count of 1

✅ PASS

  • Verify the survey DOES show if you are logged in and have an edit count of 20

✅ PASS

  • Verify the survey doesn't show if you are logged in and have an edit count of 23

✅ PASS

@Jdlrobson / @ovasileva - I'm passing this based on the QA Test note that indicates this should be tested on Desktop and references a bug for the mobile site. We will retest this for mobile once that bug fix is in.

ovasileva reassigned this task from ovasileva to Isaac.Mar 20 2019, 9:37 AM
Isaac added a comment.Mar 21 2019, 1:33 PM

Looks good to me. Documentation of each potential criteria:

  • Target anonymous users (wgEditCount === null) and logged-in users without edits (wgEditCount === 0)
  • minEdits undefined and maxEdits set to 0
  • Target a non-editor (wgEditCount === 0)
  • minEdits and maxEdits set to 0. Alternatively, setting maxEdits set to 0 and anons to false (T186737) would also lead to just targeting logged-in users without edits.
  • Target an editor
  • minEdits set to 1, which would sample all users with at least one edit. This is based on the definition used in this task that a non-editor has zero edits and therefore an editor has at least one edit.
  • Target a user with an edit count that falls into a given range -- e.g., 5-20 edits
  • For this example, minEdits set to 5 and maxEdits set to 20. Notably, the configuration allows for flexible ranges to be set (as opposed to limiting the edit ranges to pre-defined buckets as recorded in the EventLogging schema)
Isaac closed this task as Resolved.Mar 21 2019, 1:42 PM
Isaac updated the task description. (Show Details)