Page MenuHomePhabricator
Create Task
Maniphest T379682

Growth KPI Grafana dashboard claims no A/B testing happens at pilot wikis
Open, HighPublicBUG REPORT
Actions

Description

According to the Growth KPI Grafana dashboard, no A/B testing is currently happening at arwiki/eswiki/frwiki/cswiki. This does not match the reality, see P71060 for cswiki queries. This task tracks fixing the charts to match reality.

Original description

Originally reported as WMF Slack post by @Michael.

According to the Growth KPI Grafana dashboard for enwiki (screenshot below), the control variant is not assigned to any newcomers at enwiki:

image.png (580×788 px, 88 KB)

This is not expected. The control variant should be assigned to 100% of enwiki newcomers. Due to the T379681: community-updates-module variant is assigned outside of Growth pilot wikis bug, the split is currently 50/50, but even in that case, the dashboard has no reason to show zeros here.

Here is a tabular view:

image.png (504×3 px, 112 KB)

image.png (1×3 px, 177 KB)

Details

SubjectRepoBranchLines +/-
Revert "ExperimentUserDefaultsManager: use read latest when retrieving central id"mediawiki/extensions/GrowthExperimentsmaster+8 -17
ExperimentUserDefaultsManager: use read latest when retrieving central idmediawiki/extensions/GrowthExperimentsmaster+17 -8
ExperimentsHooks: avoid computing a bucket for non-registeredmediawiki/extensions/GrowthExperimentsmaster+48 -6
ExperimentUserDefaultsManager: use read latest when retrieving central idmediawiki/extensions/GrowthExperimentswmf/1.44.0-wmf.4+17 -8
Log an error when central user ID cannot be retrievedmediawiki/extensions/GrowthExperimentsmaster+20 -5
HomepageHooks: run metrics increment in deferred updatemediawiki/extensions/GrowthExperimentswmf/1.44.0-wmf.3+18 -12
HomepageHooks: run metrics increment in deferred updatemediawiki/extensions/GrowthExperimentsmaster+18 -12
Customize query in gerrit

Related Objects
Search...

Event Timeline

Urbanecm_WMF created this task.Tue, Nov 12, 9:05 PM
Restricted Application added a project: Growth-Team. · View Herald TranscriptTue, Nov 12, 9:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
KStoller-WMF subscribed.Tue, Nov 12, 9:37 PM
Urbanecm_WMF added a subscriber: Sgs.Wed, Nov 13, 12:58 PM

@Sgs asked me how did I assert the distribution is working. I selected few random users at https://en.wikipedia.org/wiki/Special:Log/newusers and called UserOptionsManager::getOption() on them in a shell.php session. I saw both variants, so I concluded the charts are wrong and distribution is correct"working as configured".

Here is a copy from my shell.php session (paste protected to WMF-NDA, as it contains private data):

{P71031}

Urbanecm_WMF added a comment.Wed, Nov 13, 1:00 PM

I re-did my tests with more users, still seem to be working as configured:

{P71032}

Urbanecm_WMF assigned this task to Sgs.Wed, Nov 13, 3:37 PM

@Sgs and me debugged this today. The issue is that by the time the instrumentation runs (the $this->perDbNameStatsdDataFactory->increment( 'GrowthExperiments.UserVariant.' . $variant ) call), CentralAuth does not have an ID available yet. This means we are consistently hashing a zero (the placeholder ID), and receiving a consistent variant. The ID is assigned in CentralAuth's UserCreationHookHandler::onLocalUserCreated, so it may run before or after our hook (the order is not guaranteed). As such, the ID may be missing (but doesn't need to). This does not impact the evaluation of the variant, because that runs much later in the process (after onLocalUserCreated), and the CentralAuth ID is definitely assigned by then.

@Sgs and I agreed on resolving this by creating a deferred update. That should ensure we have central ID available, and should still run before data are passed over to statsd. Assigning to @Sgs based on that.

Sgs added a comment.EditedWed, Nov 13, 3:40 PM
In T379682#10317328, @Urbanecm_WMF wrote:

@Sgs and I agreed on resolving this by creating a deferred update. That should ensure we have central ID available, and should still run before data are passed over to statsd. Assigning to @Sgs based on that.

Working on it, thanks for the summary

gerritbot added a comment.Wed, Nov 13, 4:04 PM

Change #1090884 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] HomepageHooks: run metrics increment in deferred update

https://gerrit.wikimedia.org/r/1090884

gerritbot added a project: Patch-For-Review.Wed, Nov 13, 4:04 PM
Sgs edited projects, added Growth-Team (Current Sprint); removed Growth-Team.Wed, Nov 13, 4:05 PM
Sgs moved this task from Incoming to Code Review on the Growth-Team (Current Sprint) board.
gerritbot added a comment.Thu, Nov 14, 12:34 PM

Change #1090884 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] HomepageHooks: run metrics increment in deferred update

https://gerrit.wikimedia.org/r/1090884

Urbanecm_WMF mentioned this in T379909: Define where to add code that needs to run after a new central user has been created.Thu, Nov 14, 12:36 PM
ReleaseTaggerBot added a project: MW-1.44-notes (1.44.0-wmf.4; 2024-11-19).Thu, Nov 14, 1:00 PM
Urbanecm_WMF triaged this task as High priority.Thu, Nov 14, 1:07 PM
Urbanecm_WMF changed the subtype of this task from "Task" to "Bug Report".
Urbanecm_WMF moved this task from Code Review to QA on the Growth-Team (Current Sprint) board.
gerritbot added a comment.Thu, Nov 14, 1:08 PM

Change #1091231 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.3] HomepageHooks: run metrics increment in deferred update

https://gerrit.wikimedia.org/r/1091231

gerritbot added a comment.Thu, Nov 14, 2:54 PM

Change #1091231 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.3] HomepageHooks: run metrics increment in deferred update

https://gerrit.wikimedia.org/r/1091231

Stashbot added a comment.Thu, Nov 14, 2:55 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-14T14:55:25Z] <sgimeno@deploy2002> Started scap sync-world: Backport for [[gerrit:1091231|HomepageHooks: run metrics increment in deferred update (T379682)]]

Stashbot added a comment.Thu, Nov 14, 2:59 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-14T14:59:21Z] <sgimeno@deploy2002> sgimeno: Backport for [[gerrit:1091231|HomepageHooks: run metrics increment in deferred update (T379682)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

ReleaseTaggerBot edited projects, added MW-1.44-notes (1.44.0-wmf.3; 2024-11-12); removed MW-1.44-notes (1.44.0-wmf.4; 2024-11-19).Thu, Nov 14, 3:00 PM
Stashbot added a comment.Thu, Nov 14, 3:06 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-14T15:06:41Z] <sgimeno@deploy2002> Finished scap sync-world: Backport for [[gerrit:1091231|HomepageHooks: run metrics increment in deferred update (T379682)]] (duration: 11m 15s)

Maintenance_bot removed a project: Patch-For-Review.Thu, Nov 14, 3:30 PM
gerritbot added a comment.Thu, Nov 14, 5:17 PM

Change #1091303 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] Log an error when central user ID cannot be retrieved

https://gerrit.wikimedia.org/r/1091303

gerritbot added a project: Patch-For-Review.Thu, Nov 14, 5:17 PM
gerritbot added a comment.Fri, Nov 15, 10:08 AM

Change #1091303 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Log an error when central user ID cannot be retrieved

https://gerrit.wikimedia.org/r/1091303

Maintenance_bot removed a project: Patch-For-Review.Fri, Nov 15, 10:30 AM
ReleaseTaggerBot edited projects, added MW-1.44-notes (1.44.0-wmf.4; 2024-11-19); removed MW-1.44-notes (1.44.0-wmf.3; 2024-11-12).Fri, Nov 15, 11:00 AM
Sgs moved this task from QA to Blocked / Needs Work on the Growth-Team (Current Sprint) board.Mon, Nov 18, 8:50 AM
Urbanecm_WMF added a comment.Mon, Nov 18, 8:54 AM

I looked at variants for cswiki:

{P71060}

Seems the charts are still wrong.

Urbanecm_WMF attached a referenced file: F57696566: image.png. (Show Details)Mon, Nov 18, 8:57 AM
Urbanecm_WMF added a comment.Mon, Nov 18, 11:26 AM

Some of the approaches listed at T379909: Define where to add code that needs to run after a new central user has been created might be worth taking a look into.

Dreamy_Jazz subscribed.Tue, Nov 19, 2:43 PM

Caused loads of error logs. Filed as a train blocker T380271.

Urbanecm_WMF mentioned this in T380294: Investigate suspicious calls to ExperimentUserDefaultsManager.Tue, Nov 19, 4:53 PM
Urbanecm_WMF renamed this task from Growth KPI Grafana dashboard claims control is not assigned to any users at enwiki to Growth KPI Grafana dashboard claims no A/B testing happens at pilot wikis.Tue, Nov 19, 4:58 PM
Urbanecm_WMF updated the task description. (Show Details)
gerritbot added a comment.Tue, Nov 19, 5:57 PM

Change #1092881 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] ExperimentUserDefaultsManager: use read latest when retrieving central id

https://gerrit.wikimedia.org/r/1092881

gerritbot added a project: Patch-For-Review.Tue, Nov 19, 5:57 PM
Sgs moved this task from Blocked / Needs Work to Code Review on the Growth-Team (Current Sprint) board.Tue, Nov 19, 6:01 PM
gerritbot added a comment.Tue, Nov 19, 6:09 PM

Change #1092886 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] ExperimentsHooks: avoid computing a bucket for non-registered

https://gerrit.wikimedia.org/r/1092886

gerritbot added a comment.Wed, Nov 20, 1:04 PM

Change #1092881 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] ExperimentUserDefaultsManager: use read latest when retrieving central id

https://gerrit.wikimedia.org/r/1092881

ReleaseTaggerBot edited projects, added MW-1.44-notes (1.44.0-wmf.5; 2024-11-25); removed MW-1.44-notes (1.44.0-wmf.4; 2024-11-19).Wed, Nov 20, 2:00 PM
gerritbot added a comment.Thu, Nov 21, 12:12 PM

Change #1093889 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.4] ExperimentUserDefaultsManager: use read latest when retrieving central id

https://gerrit.wikimedia.org/r/1093889

gerritbot added a comment.Thu, Nov 21, 2:34 PM

Change #1093889 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.4] ExperimentUserDefaultsManager: use read latest when retrieving central id

https://gerrit.wikimedia.org/r/1093889

Stashbot added a comment.Thu, Nov 21, 2:35 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-21T14:35:39Z] <sgimeno@deploy2002> Started scap sync-world: Backport for [[gerrit:1093889|ExperimentUserDefaultsManager: use read latest when retrieving central id (T379682)]]

Stashbot added a comment.Thu, Nov 21, 2:41 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-21T14:41:14Z] <sgimeno@deploy2002> sgimeno: Backport for [[gerrit:1093889|ExperimentUserDefaultsManager: use read latest when retrieving central id (T379682)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Sgs moved this task from Code Review to Blocked / Needs Work on the Growth-Team (Current Sprint) board.Thu, Nov 21, 2:52 PM
ReleaseTaggerBot edited projects, added MW-1.44-notes (1.44.0-wmf.4; 2024-11-19); removed MW-1.44-notes (1.44.0-wmf.5; 2024-11-25).Thu, Nov 21, 3:00 PM
gerritbot added a comment.Thu, Nov 21, 3:00 PM

Change #1092886 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] ExperimentsHooks: avoid computing a bucket for non-registered

https://gerrit.wikimedia.org/r/1092886

Maintenance_bot removed a project: Patch-For-Review.Thu, Nov 21, 3:31 PM
ReleaseTaggerBot edited projects, added MW-1.44-notes (1.44.0-wmf.5; 2024-11-25); removed MW-1.44-notes (1.44.0-wmf.4; 2024-11-19).Thu, Nov 21, 4:00 PM
gerritbot added a comment.Thu, Nov 21, 5:39 PM

Change #1093933 had a related patch set uploaded (by Urbanecm; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] Revert "ExperimentUserDefaultsManager: use read latest when retrieving central id"

https://gerrit.wikimedia.org/r/1093933

gerritbot added a project: Patch-For-Review.Thu, Nov 21, 5:39 PM
gerritbot added a comment.Thu, Nov 21, 6:11 PM

Change #1093933 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Revert "ExperimentUserDefaultsManager: use read latest when retrieving central id"

https://gerrit.wikimedia.org/r/1093933

Maintenance_bot removed a project: Patch-For-Review.Thu, Nov 21, 6:31 PM
Urbanecm_WMF added a subtask: T380500: CentralAuthUser returning outdated data after user creation.Thu, Nov 21, 7:56 PM

The underlying problem appears to be in CentralAuth (T380500); I added it as a subtask.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits