Rollout schedule
- Nov 14 Wednesday, European morning: 1% sampling, 50/50% control/treatment ( slotted for @pmiazga )
- Nov 14 Wednesday, US evening: 5% sampling, 50/50% control/treatment (slotted for @Niedzielski)
- Nov 15 Thursday, European morning: 25% sampling, 50/50% control/treatment (slotted for @pmiazga)
- Nov 19 Monday, European morning: 50% sampling, 50/50% control/treatment (slotted for @pmiazga)
- Nov 19 Monday, US evening: 100% sampling, 50/50% control/treatment (slotted for @Niedzielski)
- TBD (post-AB): 100% sampling, 100% treatment (T209377)
Please report status regularly. Maximum caching of page ~7 days. The A/B test is anticipated to run ~2 months.
Initial launch checklist
- Verify blocking tasks are concluded (see notes below)
- Verify blocking patches appear on branch (see notes below)
- Deploy T208755, Prod: Enable Schema.org page split test at 1% sampling, https://gerrit.wikimedia.org/r/473079
- Verify deployment was successful (see notes below)
Deployment verification
- Check Logstash for new message:wikibase logs periodically.
Use the pages identified in T208789 when checking the treatment. The treatment should look like a new application/ld+json script tag added towards the bottom of the page:
<script type="application/ld+json">{"@context":"https:\/\/schema.org","@type":"Article","name":"Kitten","url":"https:\/\/en.wikipedia.org\/wiki\/Kitten","sameAs":"https:\/\/www.wikidata.org\/entity\/Q147","mainEntity":"https:\/\/www.wikidata.org\/entity\/Q147","author":{"@type":"Organization","name":"Contributors to Wikimedia projects"},"publisher":{"@type":"Organization","name":"Wikimedia Foundation, Inc.","logo":{"@type":"ImageObject","url":"https:\/\/www.wikimedia.org\/static\/images\/wmf-hor-googpub.png"}},"datePublished":"2002-07-31T13:37:08Z","dateModified":"2018-10-08T14:18:31Z","image":"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/thumb\/0\/06\/Kitten_in_Rizal_Park%2C_Manila.jpg\/1200px-Kitten_in_Rizal_Park%2C_Manila.jpg","headline":"young of a cat"}</script>So, for a few pages on a few wikis:
- Check treatment and control distribution; sampled pages in the treatment (.5%) should have a new script tag and 99.5% should have not changes. I.e., most pages should have be in control and unaffected.
- Make sure the content appears correct.
- Check that non-main namespace pages are unaffected.
- The main page could be affected. Check it.
- Check mobile and desktop; it should have no effect on treatment or control.
- For pages with images, the image is referenced in the schema and it matches the page image (show under page info and the og:image meta HTML tag property).
- Schemas are understood by https://search.google.com/structured-data/testing-tool/u/0/. When all data is available for a given page, there should be no errors reported. When data is unavailable, such as short description or page image, the linkage should still be presented but with missing data omitted.
- Spot check pages and watch for new trends on https://www.google.com/webmasters/tools/structured-data?hl=en&authuser=1&siteUrl=https://en.wikipedia.org/ (note: mobile and desktop sites are distinct)
- The data is valid JSON-LD as verified by https://json-ld.org/playground-dev/.
- The data is linked properly as verified by http://linter.structured-data.org. E.g. http://linter.structured-data.org/?url=https:%2F%2Fde.wikipedia.org%2Fapi%2Frest_v1%2Fpage%2Fhtml%2FDouglas_Adams.
Background
In T206868: [Spike 24hrs] How do we measure the effects of the sameAs property on pageviews using an A/B test we determined and built a way to add the sameAs property to certain articles. We would now like to do the actual A/B test.
Note: double check that all previous patches have been deployed (and if not swatted). For example, the changes in T207790 must be live prior to launching.
The initial rollout is at 1% sampling with 50% A/B bucketing split across wikis:
$wgWBClientSettings[ 'pageSchemaNamespaces' ] = [ 0 ]; $wgWBClientSettings[ 'pageSchemaSplitTestSamplingRatio' ] = 0.01; $wgWBClientSettings[ 'pageSchemaSplitTestBuckets' ] = [ 'control', 'treatment' ];
The patch for the above configuration is here. Note that the order of the bucket config matters and affects the query as mentioned in T208763.
Blocking tasks that must be completed before deployment
- T208789 (results are in! this is good to go but would benefit from review by @Tbayer)
- T209315 (help requested from @Tbayer)
- T209309 (in sign off)
- T208772 (in sign off)
- T208909 (must be QA'd by @Tbayer)
Blocking patches that must appear on branch before deployment
- T206868, Update: add split testing to page schemas, wmf/1.33.0-wmf.3, I6eb59214835054fbaa42145da716e59f911e58df
- T198946, Update: add Article schema to pages for SEO, wmf/1.33.0-wmf.1, wmf/1.33.0-wmf.2, wmf/1.33.0-wmf.3, I02bc43f16cc5308b2613d1cc441df8232a055dbb
- T198946, Hygiene: move SEO asset to mediawiki-config, wmf/1.33.0-wmf.1, wmf/1.33.0-wmf.2, wmf/1.33.0-wmf.3, I2cfcdf02fe9fe99352067c0b8753353971b405a3
- T208796, Update: use wikibase-debug logger instead of "PageRandomLookup", NONE, I7a5b9e9cedff13a1f1e4bcb9efbd5a5a9f27f3a4 (slotted for @Niedzielski)
- T207790, Define WikimediaMessages for Wikibase SEO change, NONE, I8345d95a774531ff62bda4a8766f848a847921ee
- T207790, Update: add Wikimedia logo for SEO, NONE, Ie082fd8cdad623048a4be0bd8f026218ee0f3da5
Related patches that do not block or affect launch:
- T208909, Provide a script to reset the page_random column, I914ff44d1212c565e08ff17effbd682dfe7f70fb
- T208789, Hygiene: add PageSplitTester test to be used, I176a52a862239ed7781621950a15f0585d5adbc7
- T94948, Doc: pageterms query response type for wbptterms, Ia50296965d76837b06706ec15b517fa52febd4de
- Doc: result domain for GlobalFunctions::wfRandom(), Ib179d70902e231eaeeafe6449f505464eb25204d
Bugs found during QA that don't block the initial launch
Acceptance criteria
- Ensure all subtasks have been resolved. If not, and this is in the sprint, point this out to your product owner or tech lead.
- Launch sameAs A/B test on all projects but the ones listed below at 1% sampling and 50% [ 'control', 'treatment' ] bucketing for main namespace pages. This means that .005 of pages will receive the new schema.
- Do not launch on: Indonesian, Portuguese, Punjabi, Dutch, Korean, Bhojpuri, Cherokee, Kazakh, Catalan, French, Yoruba, Kalmyk, Ukrainian
- Verify all is well by performing the QA steps identified in T208772 on the pages bucketed in T208789, checking the Google Developer console for mobile and nonmobile issues across wikis, and checking in on outside reports with @ovasileva.
- Increase the rollout gradually to 100% (50% of main namespace pages will receive the treatment).
- Make a task to remove the A/B test and enable 100% of main namespace pages to receive the treatment.
A/B test analysis consideration
- Discard the most popular pages when evaluating the results
- Confirm cache lifespan with @phuedx
- Page bucketing does not change during the test, but sampling and caching will affect when schemas are visible
- The main page and redirects are not treated specially by the code