Page MenuHomePhabricator
Create Task
Maniphest T422146

Experiment config and schema registration (Article Guidance initial intervention)
Closed, ResolvedPublic4 Estimated Story Points
Actions

Description

As part of the Article Guidance experiment (T396029), the experiment needs to be registered and configured so that it is ready for automated analysis once the feature code is deployed.

Work required

  1. Register the experiment in Test Kitchen UI once experiment code is deployed
  2. Create and submit the MR to register the experiment for automated analysis
  3. @MNeisler or @mpopov to review and merge the MR

Dependencies

  • Instrumentation spec complete and QA'd (T417046)

Resources

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Instrumentation: log new articles namespace and sourcemediawiki/extensions/ArticleGuidancewmf/1.47.0-wmf.3+6 -2
Instrumentation: log new articles namespace and sourcemediawiki/extensions/ArticleGuidancemaster+6 -2
Log editing_start and article_saved events for control groupmediawiki/extensions/ArticleGuidancewmf/1.47.0-wmf.3+73 -54
Log editing_start and article_saved events for control groupmediawiki/extensions/ArticleGuidancemaster+73 -54
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Article Guidance funnel metricsrepos/product-analytics/test-kitchen/experiment-analytics-configs!68sbissonT422146-4main
Improve Article Guidance metricsrepos/product-analytics/test-kitchen/experiment-analytics-configs!67sbissonT422146-3main
Fix article guidance queryrepos/product-analytics/test-kitchen/experiment-analytics-configs!66bearlogaarticle-guidance-query-fixmain
Customize query in GitLab

Related Objects
Search...

Event Timeline

GGalofre-WMF created this task.Apr 2 2026, 11:58 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 2 2026, 11:58 AM
GGalofre-WMF added a parent task: T417046: Instrument the steps on the Article Guidance workflow.Apr 2 2026, 11:59 AM
GGalofre-WMF moved this task from Backlog to Experimentation and Analytics on the Article-Guidance board.
GGalofre-WMF mentioned this in T396029: [Epic] Guidance for Article and Section creation.Apr 2 2026, 12:01 PM
GGalofre-WMF added a subscriber: SBisson.

@SBisson I created this task for the schema and experiment registration.

SBisson added a comment.Apr 16 2026, 2:54 PM

It turns out there is no schema to create, we're using web/base, but there will be an experiment and an instrument to create in the Test Kitchen UI. Who should be doing that?

GGalofre-WMF added a comment.Apr 24 2026, 11:14 AM

Following up with clarification on who owns each step, based on a conversation with @MNeisler:

Both steps are typically owned by the engineers, with support from Product Analytics (@MNeisler and @mpopov) as needed:

1. Configuring the experiment in Test Kitchen UI
Owned by the engineer. @MNeisler and @mpopov can help answer questions as needed.

2. Registering the experiment for automated analysis
Also owned by the engineer. This involves:

  • Adding the metrics for the experiment to metrics_catalog if not already available
  • Adding an entry to experiments_registry.yaml
  • A guide for this process is available here
  • The guide and current metrics catalog include example queries which may be useful for defining the metrics for this experiment
  • Once the MR is ready, add both @MNeisler and @mpopov as reviewers — they will review and merge with any suggested changes
SBisson claimed this task.Apr 24 2026, 1:54 PM
SBisson triaged this task as High priority.
SBisson moved this task from Backlog to In-progress on the LPL Hypothesis board.
eamedina subscribed.Apr 24 2026, 5:20 PM
SBisson mentioned this in T417046: Instrument the steps on the Article Guidance workflow.Apr 28 2026, 3:18 PM
SBisson moved this task from In-progress to Needs Code Review on the LPL Hypothesis board.Apr 28 2026, 9:34 PM
Nikerabbit set the point value for this task to 4.Apr 29 2026, 10:24 AM
Nikerabbit added a project: LPL Projects (Guided article creation - FY26 WE 1.10).
gerritbot added a comment.Tue, May 19, 6:36 PM

Change #1289406 had a related patch set uploaded (by Sbisson; author: Sbisson):

[mediawiki/extensions/ArticleGuidance@master] Log editing_start and article_saved events for control group

https://gerrit.wikimedia.org/r/1289406

gerritbot added a project: Patch-For-Review.Tue, May 19, 6:36 PM
gerritbot added a comment.Wed, May 20, 12:00 AM

Change #1289406 merged by jenkins-bot:

[mediawiki/extensions/ArticleGuidance@master] Log editing_start and article_saved events for control group

https://gerrit.wikimedia.org/r/1289406

gerritbot added a comment.Wed, May 20, 12:06 AM

Change #1289446 had a related patch set uploaded (by Sbisson; author: Sbisson):

[mediawiki/extensions/ArticleGuidance@wmf/1.47.0-wmf.3] Log editing_start and article_saved events for control group

https://gerrit.wikimedia.org/r/1289446

ReleaseTaggerBot added a project: MW-1.47-notes (1.47.0-wmf.4; 2026-05-26).Wed, May 20, 1:00 AM
gerritbot added a comment.Wed, May 20, 1:07 PM

Change #1289446 merged by jenkins-bot:

[mediawiki/extensions/ArticleGuidance@wmf/1.47.0-wmf.3] Log editing_start and article_saved events for control group

https://gerrit.wikimedia.org/r/1289446

Stashbot added a comment.Wed, May 20, 1:07 PM

Mentioned in SAL (#wikimedia-operations) [2026-05-20T13:07:44Z] <sbisson@deploy1003> Started scap sync-world: Backport for [[gerrit:1289446|Log editing_start and article_saved events for control group (T422146)]]

Stashbot added a comment.Wed, May 20, 1:09 PM

Mentioned in SAL (#wikimedia-operations) [2026-05-20T13:09:40Z] <sbisson@deploy1003> sbisson: Backport for [[gerrit:1289446|Log editing_start and article_saved events for control group (T422146)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Wed, May 20, 1:15 PM

Mentioned in SAL (#wikimedia-operations) [2026-05-20T13:15:40Z] <sbisson@deploy1003> Finished scap sync-world: Backport for [[gerrit:1289446|Log editing_start and article_saved events for control group (T422146)]] (duration: 07m 55s)

Maintenance_bot removed a project: Patch-For-Review.Wed, May 20, 1:31 PM
ReleaseTaggerBot edited projects, added: MW-1.47-notes (1.47.0-wmf.3; 2026-05-19); removed: MW-1.47-notes (1.47.0-wmf.4; 2026-05-26).Wed, May 20, 2:00 PM
CodeReviewBot added a comment.Wed, May 20, 10:27 PM

mneisler merged https://gitlab.wikimedia.org/repos/product-analytics/test-kitchen/experiment-analytics-configs/-/merge_requests/61

Article Guidance A/B test automated analysis metrics

CodeReviewBot added a comment.Thu, May 21, 2:56 AM

bearloga updated https://gitlab.wikimedia.org/repos/product-analytics/test-kitchen/experiment-analytics-configs/-/merge_requests/66

Fix article guidance query

mpopov added a comment.Thu, May 21, 3:34 AM

Experiment results now available: https://superset.wikimedia.org/superset/dashboard/p/ZEXOPgzLrKb/ (permalink to this specific experiment's results in https://superset.wikimedia.org/superset/dashboard/experiment-analytics/)

Nikerabbit moved this task from Needs Code Review to Product Signoff on the LPL Hypothesis board.Thu, May 21, 11:43 AM
SBisson moved this task from Product Signoff to In-progress on the LPL Hypothesis board.Mon, May 25, 2:18 PM

Some rework on the metrics needed now that I have a good way to test them locally.

gerritbot added a comment.Mon, May 25, 2:20 PM

Change #1293128 had a related patch set uploaded (by Sbisson; author: Sbisson):

[mediawiki/extensions/ArticleGuidance@master] Instrumentation: log new articles namespace and source

https://gerrit.wikimedia.org/r/1293128

gerritbot added a project: Patch-For-Review.Mon, May 25, 2:20 PM
gerritbot added a comment.Mon, May 25, 7:43 PM

Change #1293128 merged by jenkins-bot:

[mediawiki/extensions/ArticleGuidance@master] Instrumentation: log new articles namespace and source

https://gerrit.wikimedia.org/r/1293128

gerritbot added a comment.Mon, May 25, 11:50 PM

Change #1293177 had a related patch set uploaded (by Sbisson; author: Sbisson):

[mediawiki/extensions/ArticleGuidance@wmf/1.47.0-wmf.3] Instrumentation: log new articles namespace and source

https://gerrit.wikimedia.org/r/1293177

gerritbot added a comment.Tue, May 26, 1:04 PM

Change #1293177 merged by jenkins-bot:

[mediawiki/extensions/ArticleGuidance@wmf/1.47.0-wmf.3] Instrumentation: log new articles namespace and source

https://gerrit.wikimedia.org/r/1293177

Stashbot added a comment.Tue, May 26, 1:05 PM

Mentioned in SAL (#wikimedia-operations) [2026-05-26T13:05:07Z] <sbisson@deploy1003> Started scap sync-world: Backport for [[gerrit:1293177|Instrumentation: log new articles namespace and source (T422146)]]

Stashbot added a comment.Tue, May 26, 1:07 PM

Mentioned in SAL (#wikimedia-operations) [2026-05-26T13:06:59Z] <sbisson@deploy1003> sbisson: Backport for [[gerrit:1293177|Instrumentation: log new articles namespace and source (T422146)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Tue, May 26, 1:12 PM

Mentioned in SAL (#wikimedia-operations) [2026-05-26T13:12:16Z] <sbisson@deploy1003> Finished scap sync-world: Backport for [[gerrit:1293177|Instrumentation: log new articles namespace and source (T422146)]] (duration: 07m 09s)

CodeReviewBot added a comment.Tue, May 26, 4:56 PM

bearloga updated https://gitlab.wikimedia.org/repos/product-analytics/test-kitchen/experiment-analytics-configs/-/merge_requests/67

Improve Article Guidance metrics

CodeReviewBot added a comment.Tue, May 26, 7:31 PM

sbisson opened https://gitlab.wikimedia.org/repos/product-analytics/test-kitchen/experiment-analytics-configs/-/merge_requests/68

Article Guidance funnel metrics

SBisson moved this task from In-progress to Needs Code Review on the LPL Hypothesis board.Tue, May 26, 7:33 PM
CodeReviewBot added a comment.Thu, May 28, 2:23 PM

bearloga merged https://gitlab.wikimedia.org/repos/product-analytics/test-kitchen/experiment-analytics-configs/-/merge_requests/67

Improve Article Guidance metrics

CodeReviewBot added a comment.Wed, Jun 3, 3:35 PM

bearloga merged https://gitlab.wikimedia.org/repos/product-analytics/test-kitchen/experiment-analytics-configs/-/merge_requests/68

Article Guidance funnel metrics

SBisson moved this task from Needs Code Review to Product Signoff on the LPL Hypothesis board.Thu, Jun 4, 7:05 PM
GGalofre-WMF closed this task as Resolved.Tue, Jun 9, 12:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits