Page MenuHomePhabricator

Confirm the bucketing for the languages A/B tests worked as expected
Closed, ResolvedPublic2 Estimated Story Points

Description

While running the A/B test it wasn't 100% clear if the bucketing was working correctly.

We'll add a statsd counter that runs the bucketing code that was used for the A/B test.

$bucket = 'vector.language-test.' . ( $inGroupA ? 'A' : 'B' );

// TODO: Inject this dependency using DI!!1
MediaWikiServices::getInstance()
  ->getStatsdDataFactory()
  ->increment( $bucket )
  ;

QA

In beta cluster

With the help of a developer, we will need to setup the graph at https://grafana-labs.wikimedia.org/d/000000051/reading-web-beta-dashboard?viewPanel=7&orgId=1

QA should involve checking the 2 buckets are of roughly equal size.

In production

With the help of a dev eloperwe will need to setup a graph on https://grafana.wikimedia.org/d/000000566/overview?orgId=1

QA should involve checking the 2 buckets are of roughly equal size.

QA Results - Beta

ACStatusDetails
1T286932#7306510

QA Results - Prod

ACStatusDetails
1T286932#7306512

Event Timeline

Sam points out this could be done on the server side and might make more sense there.

Change 708318 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[mediawiki/skins/Vector@master] Add statsd counter for languages A/B test

https://gerrit.wikimedia.org/r/708318

cjming added a subscriber: cjming.

Requesting some of Sam's 30% time..

Jdlrobson updated the task description. (Show Details)
Jdlrobson added a subscriber: phuedx.

Change 708318 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Add statsd counter for languages A/B test

https://gerrit.wikimedia.org/r/708318

@phuedx / @Jdlrobson , this is what I'm seeing. Doesn't look 50/50. Per instructions I'm invoking the help of a developer.

Screen Shot 2021-08-05 at 7.14.22 AM.png (518×1 px, 47 KB)

LGoto added a subscriber: Edtadros.

Change 710274 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/skins/Vector@master] Fix language treatment A/B test bucket counting

https://gerrit.wikimedia.org/r/710274

Please wait until at least 12th August before beginning QA to allow the existing data on beta cluster to stabilize.

Change 710274 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Fix language treatment A/B test bucket counting

https://gerrit.wikimedia.org/r/710274

Change 710717 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/skins/Vector@wmf/1.37.0-wmf.18] Fix language treatment A/B test bucket counting

https://gerrit.wikimedia.org/r/710717

Change 710717 abandoned by Phuedx:

[mediawiki/skins/Vector@wmf/1.37.0-wmf.18] Fix language treatment A/B test bucket counting

Reason:

Err… I cherry picked the wrong patch…

https://gerrit.wikimedia.org/r/710717

@Jdlrobson /@phuedx I still see the same graph as T286932#7263384. I tried changing the time duration to the last 7 days or even from 8/12 until now and I get nothing and the totals are zero. Thoughts?

Test Result - Beta

Status: ✅ PASS
Environment: beta
OS: macOS Big Sur
Browser: Chrome
Device: MBP
Emulated Device: NA

Test Artifact(s):

QA Steps

QA
In beta cluster
With the help of a developer, we will need to setup the graph at https://grafana-labs.wikimedia.org/d/000000051/reading-web-beta-dashboard?viewPanel=7&orgId=1

QA should involve checking the 2 buckets are of roughly equal size.

✅ AC1: Language buckets are roughly equal.

Screen Shot 2021-08-24 at 2.11.44 PM.png (721×769 px, 54 KB)

Test Result - Prod

Status: ✅ PASS
Environment: Prod
OS: macOS Big Sur
Browser: Chrome
Device: MBP
Emulated Device: NA

Test Artifact(s):

QA Steps

In production
With the help of a dev eloperwe will need to setup a graph on https://grafana.wikimedia.org/d/000000566/overview?orgId=1

QA should involve checking the 2 buckets are of roughly equal size.

✅ AC1: Language buckets are roughly equal.

Screen Shot 2021-08-24 at 2.12.17 PM.png (767×769 px, 62 KB)

ovasileva added subscribers: jwang, ovasileva.

@jwang - could you confirm that this is sufficient proof for the buckets?

Yes, it seems 2 buckets are of roughly equal size.