Page MenuHomePhabricator

Add edit buckets to search satisfaction schema
Closed, ResolvedPublic3 Estimated Story Points

Description

Background

The results of the A/B test on the first search change led to some interesting conclusions. For our next A/B test, we'd like to explore adding search buckets to the schema so that we can gain a bit more insights in patterns between new and more veteran editors

Acceptance criteria

  • Add edit count buckets to search satisfaction schema

QA Results - Prod

ACStatusDetails
1T272991#6874948

Event Timeline

This can be done in two steps:

  1. Add a userEditBucket optional property to the SearchSatisfaction schema in the schemas/event/secondary repository (here):
allOf:
    properties:
        properties:
            userEditBucket:
                type: string
                enum:
                  - 0 edits
                  - 1-4 edits
                  - 5-99 edits
                  - 100-999 edits
                  - 1000+ edits
                description: ...
  1. Add the code that sets the property to the instrument. Since this data isn't specific to any specific user action, it should be added near where we added the skin and skinVersion properties, i.e. near here, and bump the version of the schema in extension.json.

We might want to revisit reusing the user edit count bucketing code again. However, code reuse and testing in possible but non-trivial in the WikimediaEvents, which is geared towards isolated modules that do everything that they need.

ovasileva set the point value for this task to 3.Jan 27 2021, 5:24 PM

We might want to revisit reusing the user edit count bucketing code again. However, code reuse and testing in possible but non-trivial in the WikimediaEvents, which is geared towards isolated modules that do everything that they need.

Actually, this doesn't need to be revisited. mw.wikimediaEvents.getUserEditCountBucket( editCount: number ) is a thing!

Change 665317 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[schemas/event/secondary@master] SearchSatisfaction: Add editBucketCount

https://gerrit.wikimedia.org/r/665317

Change 665320 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[mediawiki/extensions/WikimediaEvents@master] Add userEditBucket to searchSatisfaction

https://gerrit.wikimedia.org/r/665320

Jdrewniak subscribed.
ovasileva raised the priority of this task from Medium to High.Feb 22 2021, 10:50 AM

Change 665317 merged by Phuedx:
[schemas/event/secondary@master] SearchSatisfaction: Add editBucketCount

https://gerrit.wikimedia.org/r/665317

Change 665320 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Add userEditBucket to searchSatisfaction

https://gerrit.wikimedia.org/r/665320

Regrettably, this will have to tested by checking data in the event.searchsatisfaction Hive table once the change has gone live – it'll be deployed to the group0 wikis tomorrow – as there's no way to override the sampling code that enables the SearchSatisfaction instrument.

@Edtadros: If you have access to Hue (i.e. you've signed an NDA and you're in the nda LDAP group, then you should be able to test this. Alternatively, @MNeisler or I could.

Change 666406 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[schemas/event/secondary@master] Rematerialize searchsatisfaction/1.4.0 for enforced numeric bounds and for examples $id

https://gerrit.wikimedia.org/r/666406

Change 666406 merged by Ottomata:
[schemas/event/secondary@master] Rematerialize searchsatisfaction/1.4.0 for enforced numeric bounds and for examples $id

https://gerrit.wikimedia.org/r/666406

@ovasileva @phuedx I checked data logged in the event.searchsatisfaction hive table and unfortunately have not yet seen any events recorded for the event.usereditbucket field on any of the group0 wikis. See hive query and results for testwiki below.

The new field has been added to the schema in WikimediaEvents but only NULL values have been recorded as of 23 Feb 2021.

isanonusereditbucketnum_events
falseNULL25
trueNULL1

Data via:

SELECT
  event.isAnon,
  event.userEditBucket,
  COUNT(*) As num_events
FROM
event.searchsatisfaction
WHERE
  YEAR = 2021
  AND MONTH = 02
  AND DAY >= 23
  AND wiki = 'testwiki'
GROUP BY
  event.userEditBucket,
  event.isAnon;

Is there any way to check if these events are firing correctly in the browser? Let me know if there are any other checks or queries I can help run.

I checked data logged in the event.searchsatisfaction hive table and unfortunately have not yet seen any events recorded for the event.usereditbucket field on any of the group0 wikis. See hive query and results for testwiki below.

Update: It looks like was due to a regression in WikimediaEvents. A fix was merged and backported today (24 February 2021) to address. I rechecked the data and confirmed we are now receiving events and they are logging as expected.

QA check and notes documented in T274869#6857856

Edtadros subscribed.

Test Result - Prod

Status: ✅ PASS
Environment: production
OS: NA
Browser: NA
Device: NA
Emulated Device: NA

Test Artifact(s):

QA Steps

✅ AC1: Add edit count buckets to search satisfaction schema
Megan tested this here: T272991#6857931. I'm just adding a pretty table.