Launch A/B test for sameAs property
Closed, ResolvedPublic5 Story Points

Description

Rollout schedule

Please report status regularly. Maximum caching of page ~7 days. The A/B test is anticipated to run ~2 months.

Initial launch checklist

  • Verify blocking tasks are concluded (see notes below)
  • Verify blocking patches appear on branch (see notes below)
  • Deploy T208755, Prod: Enable Schema.org page split test at 1% sampling, https://gerrit.wikimedia.org/r/473079
  • Verify deployment was successful (see notes below)

Deployment verification

  • Check Logstash for new message:wikibase logs periodically.

Use the pages identified in T208789 when checking the treatment. The treatment should look like a new application/ld+json script tag added towards the bottom of the page:

<script type="application/ld+json">{"@context":"https:\/\/schema.org","@type":"Article","name":"Kitten","url":"https:\/\/en.wikipedia.org\/wiki\/Kitten","sameAs":"https:\/\/www.wikidata.org\/entity\/Q147","mainEntity":"https:\/\/www.wikidata.org\/entity\/Q147","author":{"@type":"Organization","name":"Contributors to Wikimedia projects"},"publisher":{"@type":"Organization","name":"Wikimedia Foundation, Inc.","logo":{"@type":"ImageObject","url":"https:\/\/www.wikimedia.org\/static\/images\/wmf-hor-googpub.png"}},"datePublished":"2002-07-31T13:37:08Z","dateModified":"2018-10-08T14:18:31Z","image":"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/thumb\/0\/06\/Kitten_in_Rizal_Park%2C_Manila.jpg\/1200px-Kitten_in_Rizal_Park%2C_Manila.jpg","headline":"young of a cat"}</script>

So, for a few pages on a few wikis:

Background

In T206868: [Spike 24hrs] How do we measure the effects of the sameAs property on pageviews using an A/B test we determined and built a way to add the sameAs property to certain articles. We would now like to do the actual A/B test.

Note: double check that all previous patches have been deployed (and if not swatted). For example, the changes in T207790 must be live prior to launching.

The initial rollout is at 1% sampling with 50% A/B bucketing split across wikis:

$wgWBClientSettings[ 'pageSchemaNamespaces' ] = [ 0 ];
$wgWBClientSettings[ 'pageSchemaSplitTestSamplingRatio' ] = 0.01;
$wgWBClientSettings[ 'pageSchemaSplitTestBuckets' ] = [ 'control', 'treatment' ];

The patch for the above configuration is here. Note that the order of the bucket config matters and affects the query as mentioned in T208763.

Blocking tasks that must be completed before deployment

Blocking patches that must appear on branch before deployment

  • T206868, Update: add split testing to page schemas, wmf/1.33.0-wmf.3, I6eb59214835054fbaa42145da716e59f911e58df
  • T198946, Update: add Article schema to pages for SEO, wmf/1.33.0-wmf.1, wmf/1.33.0-wmf.2, wmf/1.33.0-wmf.3, I02bc43f16cc5308b2613d1cc441df8232a055dbb
  • T198946, Hygiene: move SEO asset to mediawiki-config, wmf/1.33.0-wmf.1, wmf/1.33.0-wmf.2, wmf/1.33.0-wmf.3, I2cfcdf02fe9fe99352067c0b8753353971b405a3
  • T208796, Update: use wikibase-debug logger instead of "PageRandomLookup", NONE, I7a5b9e9cedff13a1f1e4bcb9efbd5a5a9f27f3a4 (slotted for @Niedzielski)
  • T207790, Define WikimediaMessages for Wikibase SEO change, NONE, I8345d95a774531ff62bda4a8766f848a847921ee
  • T207790, Update: add Wikimedia logo for SEO, NONE, Ie082fd8cdad623048a4be0bd8f026218ee0f3da5

Related patches that do not block or affect launch:

  • T208909, Provide a script to reset the page_random column, I914ff44d1212c565e08ff17effbd682dfe7f70fb
  • T208789, Hygiene: add PageSplitTester test to be used, I176a52a862239ed7781621950a15f0585d5adbc7
  • T94948, Doc: pageterms query response type for wbptterms, Ia50296965d76837b06706ec15b517fa52febd4de
  • Doc: result domain for GlobalFunctions::wfRandom(), Ib179d70902e231eaeeafe6449f505464eb25204d

Bugs found during QA that don't block the initial launch

Acceptance criteria

  • Ensure all subtasks have been resolved. If not, and this is in the sprint, point this out to your product owner or tech lead.
  • Launch sameAs A/B test on all projects but the ones listed below at 1% sampling and 50% [ 'control', 'treatment' ] bucketing for main namespace pages. This means that .005 of pages will receive the new schema.
    • Do not launch on: Indonesian, Portuguese, Punjabi, Dutch, Korean, Bhojpuri, Cherokee, Kazakh, Catalan, French, Yoruba, Kalmyk, Ukrainian
  • Verify all is well by performing the QA steps identified in T208772 on the pages bucketed in T208789, checking the Google Developer console for mobile and nonmobile issues across wikis, and checking in on outside reports with @ovasileva.
  • Increase the rollout gradually to 100% (50% of main namespace pages will receive the treatment).
  • Make a task to remove the A/B test and enable 100% of main namespace pages to receive the treatment.

A/B test analysis consideration

  • Discard the most popular pages when evaluating the results
  • Confirm cache lifespan with @phuedx
  • Page bucketing does not change during the test, but sampling and caching will affect when schemas are visible
  • The main page and redirects are not treated specially by the code

Related Objects

There are a very large number of changes, so older changes are hidden. Show Older Changes

@pmiazga, @ovasileva, @phuedx, @Tbayer please review this task thoroughly and edit as needed.

Niedzielski updated the task description. (Show Details)Mon, Nov 12, 11:06 PM
Niedzielski updated the task description. (Show Details)Mon, Nov 12, 11:09 PM
Legoktm changed the edit policy from "Custom Policy" to "All Users".Tue, Nov 13, 8:56 AM
Legoktm added a subscriber: Legoktm.

I didn't see any reason for this to have a restrictive edit policy, so I've reset it to "all users".

phuedx updated the task description. (Show Details)Tue, Nov 13, 11:31 AM

Mentioned in SAL (#wikimedia-operations) [2018-11-13T12:22:52Z] <phuedx@deploy1001> Synchronized php-1.33.0-wmf.3/extensions/WikimediaMessages/: SWAT: [[gerrit:473164|Define WikimediaMessages for Wikibase SEO change (T208755)]] (duration: 00m 56s)

phuedx updated the task description. (Show Details)Tue, Nov 13, 12:52 PM

I8345d95a774531ff62bda4a8766f848a847921ee has been deployed: https://en.wikipedia.org/w/index.php?title=Special%3AAllMessages&prefix=wikibase-page-schema&filter=all&lang=en&limit=50

Because the above was an l18n change, it required a scap sync (which includes rebuilding the l18n caches for core and extensions), which took ~22 minutes and delayed the deployment of I7a5b9e9cedff13a1f1e4bcb9efbd5a5a9f27f3a4. I've rescheduled the latter to be deployed later in the day.

Change 473221 had a related patch set uploaded (by Niedzielski; owner: Stephen Niedzielski):
[operations/mediawiki-config@master] Prod: increase Schema.org page split test to 5% sampling

https://gerrit.wikimedia.org/r/473221

Niedzielski updated the task description. (Show Details)Tue, Nov 13, 2:47 PM

Change 473224 had a related patch set uploaded (by Niedzielski; owner: Stephen Niedzielski):
[operations/mediawiki-config@master] Prod: increase Schema.org page split test to 25% sampling

https://gerrit.wikimedia.org/r/473224

Change 473225 had a related patch set uploaded (by Niedzielski; owner: Stephen Niedzielski):
[operations/mediawiki-config@master] Prod: increase Schema.org page split test to 50% sampling

https://gerrit.wikimedia.org/r/473225

Change 473227 had a related patch set uploaded (by Niedzielski; owner: Stephen Niedzielski):
[operations/mediawiki-config@master] Prod: increase Schema.org page split test to 100% sampling

https://gerrit.wikimedia.org/r/473227

Niedzielski updated the task description. (Show Details)Tue, Nov 13, 3:02 PM
Niedzielski updated the task description. (Show Details)Tue, Nov 13, 3:34 PM
Niedzielski updated the task description. (Show Details)Tue, Nov 13, 5:10 PM
Niedzielski updated the task description. (Show Details)Tue, Nov 13, 5:24 PM
ovasileva set the point value for this task to 5.Tue, Nov 13, 6:26 PM
Legoktm removed a subscriber: Legoktm.Tue, Nov 13, 7:26 PM
Niedzielski updated the task description. (Show Details)Tue, Nov 13, 7:53 PM
Niedzielski updated the task description. (Show Details)

I think @phuedx is doing the initial deploy so assigning to him.

Niedzielski updated the task description. (Show Details)Tue, Nov 13, 8:14 PM
pmiazga updated the task description. (Show Details)Tue, Nov 13, 8:28 PM

T209352 is not a blocker. Working as intended.

Niedzielski updated the task description. (Show Details)Wed, Nov 14, 1:05 AM
Niedzielski updated the task description. (Show Details)Wed, Nov 14, 4:21 AM
Tbayer updated the task description. (Show Details)Wed, Nov 14, 7:30 AM
phuedx reassigned this task from phuedx to pmiazga.Wed, Nov 14, 9:33 AM

Per the task description (@pmiazga is deploying the first change in the chain and I will be available for QA).

pmiazga added a comment.EditedWed, Nov 14, 11:00 AM

Feature will be disabled for wikis:

  • Indonesian: idwiki
  • Portuguese: ptwiki
  • Punjabi: pawiki, pnbwiki
  • Dutch: nlwiki, nds_nlwiki
  • Korean: kowiki
  • Bhojpuri: bhwiki
  • Cherokee: chrwiki
  • Kazakh: kkwiki
  • Catalan: cawiki
  • French: frwiki
  • Yoruba: yowiki
  • Kalmyk: xalwiki

Change 473079 merged by jenkins-bot:
[operations/mediawiki-config@master] Prod: Enable Schema.org page split test at 1% sampling

https://gerrit.wikimedia.org/r/473079

Mentioned in SAL (#wikimedia-operations) [2018-11-14T12:19:11Z] <pmiazga@deploy1001> Synchronized wmf-config: SWAT: [[gerrit:473079]|Enable Schema.org page split test at 1% sampling (T208755)]] (duration: 00m 54s)

Change 473221 merged by jenkins-bot:
[operations/mediawiki-config@master] Prod: increase Schema.org page split test to 5% sampling

https://gerrit.wikimedia.org/r/473221

Mentioned in SAL (#wikimedia-operations) [2018-11-15T00:50:02Z] <niharika29@deploy1001> Synchronized wmf-config/InitialiseSettings.php: increase Schema.org page split test to 5% sampling T208755 (duration: 00m 54s)

Change 473224 merged by jenkins-bot:
[operations/mediawiki-config@master] Prod: increase Schema.org page split test to 25% sampling

https://gerrit.wikimedia.org/r/473224

Mentioned in SAL (#wikimedia-operations) [2018-11-15T12:42:07Z] <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:473224|Prod: increase Schema.org page split test to 25% sampling]] T208755 (duration: 00m 53s)

phuedx updated the task description. (Show Details)Thu, Nov 15, 1:12 PM
phuedx updated the task description. (Show Details)Thu, Nov 15, 1:18 PM

☝️ The task description was slightly out of date and was bold.

Per T208755#4749857 we're currently at 25% sampling, which means:

  • The initial launch is complete and has been verified, and
  • The bug(s) that @pmiazga found didn't block the initial launch.

The T208772 is defined as blocking task (Q/A the sameAs property). Task is still open but I decided to roll-up the A/B test sameAs property because:

  • I tested most of the stuff on the BetaCluster and it worked properly
  • I also identified couple issues/unknowns which didn't block the launch
  • once the task went live (1%, 5%, 25%) I was using this task (T208772) to recheck all the possible test scenarios on the production environment.

Every time we deploy something, we re-check stuff on production environment. Instead of creating a new task I decided to re-use the same task as it had pretty nice acceptance criteria that I could apply to both beta env and prod env.

pmiazga updated the task description. (Show Details)Thu, Nov 15, 3:30 PM
pmiazga updated the task description. (Show Details)

A patch for T206497 is in the works for several wikis to point mobile to desktop. Will this impact the a/b test in any way if enabled?

mpopov updated the task description. (Show Details)Fri, Nov 16, 5:53 PM
Restricted Application added a subscriber: Base. · View Herald TranscriptFri, Nov 16, 5:53 PM

Feature will be disabled for wikis:

  • Indonesian: idwiki
  • Portuguese: ptwiki
  • Punjabi: pawiki, pnbwiki
  • Dutch: nlwiki, nds_nlwiki
  • Korean: kowiki
  • Bhojpuri: bhwiki
  • Cherokee: chrwiki
  • Kazakh: kkwiki
  • Catalan: cawiki
  • French: frwiki
  • Yoruba: yowiki
  • Kalmyk: xalwiki

By the way!!! Per discussion with @ovasileva, we decided to not to deploy on Ukrainian as well. I just added that to the task description too, if that's not too late. We were hoping to use it as a "control" instead of Spanish for the test of sitemaps on Indonesian & Portuguese Wikipedias (T209720).

Feature will be disabled for wikis:

  • Indonesian: idwiki
  • Portuguese: ptwiki
  • Punjabi: pawiki, pnbwiki
  • Dutch: nlwiki, nds_nlwiki
  • Korean: kowiki
  • Bhojpuri: bhwiki
  • Cherokee: chrwiki
  • Kazakh: kkwiki
  • Catalan: cawiki
  • French: frwiki
  • Yoruba: yowiki
  • Kalmyk: xalwiki

I think it's too late :| We've already rolled out / sampled to 25% on Ukrainian, Spanish, and everywhere except ^

Perhaps related? The overlap with T206497 and T206496 is arwiki, hiwiki, itwiki, zhwiki if I understand correctly.

pmiazga updated the task description. (Show Details)Mon, Nov 19, 9:50 AM

Change 473225 merged by jenkins-bot:
[operations/mediawiki-config@master] Prod: increase Schema.org page split test to 50% sampling

https://gerrit.wikimedia.org/r/473225

Mentioned in SAL (#wikimedia-operations) [2018-11-19T12:10:45Z] <pmiazga@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:473225]|Enable Schema.org page split test at 50% sampling (T208755)]] (duration: 00m 46s)

Change 473227 merged by jenkins-bot:
[operations/mediawiki-config@master] Prod: increase Schema.org page split test to 100% sampling

https://gerrit.wikimedia.org/r/473227

Mentioned in SAL (#wikimedia-operations) [2018-11-20T00:15:24Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Increase Schema.org page split test to 100% sampling (T208755) (duration: 00m 48s)

pmiazga removed pmiazga as the assignee of this task.
pmiazga claimed this task.
pmiazga updated the task description. (Show Details)
pmiazga updated the task description. (Show Details)
pmiazga updated the task description. (Show Details)Tue, Nov 20, 2:57 PM
pmiazga removed pmiazga as the assignee of this task.Tue, Nov 20, 3:02 PM
pmiazga removed a project: Patch-For-Review.
ovasileva closed this task as Resolved.EditedTue, Nov 20, 3:32 PM
ovasileva claimed this task.

Test is running at 50%, all acceptance criteria are met cc @mpopov. Checking off and resolving. Thank you @Niedzielski and @pmiazga for all your hard work here!

ovasileva updated the task description. (Show Details)Tue, Nov 20, 3:32 PM

Thank you @Niedzielski and @pmiazga for all your hard work here.

Seconded!