== Rollout schedule
- Nov 14 Wednesday, European morning: [[ https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/473079/ | 1% sampling, 50/50% control/treatment ]] ( [[https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181114T1200 | slotted for ]] @pmiazga )
- Nov 14 Wednesday, US evening: [[ https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/473221/ | 5% sampling, 50/50% control/treatment ]] ([[ https://wikitech.wikimedia.org/wiki/Deployments#Thursday,_November_15 | slotted for ]] @niedzielski)
- Nov 15 Thursday, European morning: [[ https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/473224/ | 25% sampling, 50/50% control/treatment ]]
- Nov 19 Monday, European morning: [[ https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/473225/ | 50% sampling, 50/50% control/treatment ]]
- Nov 19 Monday, US evening: [[ https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/473227/ | 100% sampling, 50/50% control/treatment ]] ([[ https://wikitech.wikimedia.org/wiki/Deployments#Tuesday,_November_20 | slotted for ]] @niedzielski)
- TBD (post-AB): 100% sampling, 100% treatment (T209377)
Please report status regularly. Maximum caching of page ~7 days. The A/B test is anticipated to run ~2 months.
== Initial launch checklist
[] Verify blocking tasks are concluded (see notes below)
[] Verify blocking patches appear on branch (see notes below)
[] Deploy T208755, Prod: Enable Schema.org page split test at 1% sampling, https://gerrit.wikimedia.org/r/473079
[] Verify deployment was successful (see notes below)
== Deployment verification
[] Check [[ https://logstash.wikimedia.org | Logstash ]] for new `message:wikibase` logs periodically.
Use the pages identified in T208789 when checking the treatment. The treatment should look like a new `application/ld+json` script tag added towards the bottom of the page:
```
<script type="application/ld+json">{"@context":"https:\/\/schema.org","@type":"Article","name":"Kitten","url":"https:\/\/en.wikipedia.org\/wiki\/Kitten","sameAs":"https:\/\/www.wikidata.org\/entity\/Q147","mainEntity":"https:\/\/www.wikidata.org\/entity\/Q147","author":{"@type":"Organization","name":"Contributors to Wikimedia projects"},"publisher":{"@type":"Organization","name":"Wikimedia Foundation, Inc.","logo":{"@type":"ImageObject","url":"https:\/\/www.wikimedia.org\/static\/images\/wmf-hor-googpub.png"}},"datePublished":"2002-07-31T13:37:08Z","dateModified":"2018-10-08T14:18:31Z","image":"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/thumb\/0\/06\/Kitten_in_Rizal_Park%2C_Manila.jpg\/1200px-Kitten_in_Rizal_Park%2C_Manila.jpg","headline":"young of a cat"}</script>
```
So, for a few pages on a few wikis:
[] Check treatment and control distribution; sampled pages in the treatment (.5%) should have a new script tag and 99.5% should have not changes. I.e., most pages should have be in control and unaffected.
[] Make sure the content appears correct.
[] Check that non-main namespace pages are unaffected.
[] The main page could be affected. Check it.
[] Check mobile and desktop; it should have no effect on treatment or control.
[] For pages with images, the image is referenced in the schema and it matches the page image (show under page info and the og:image meta HTML tag property).
[] Schemas are understood by https://search.google.com/structured-data/testing-tool/u/0/. When all data is available for a given page, there should be no errors reported. When data is unavailable, such as short description or page image, the linkage should still be presented but with missing data omitted.
[] Spot check pages and watch for new trends on https://www.google.com/webmasters/tools/structured-data?hl=en&authuser=1&siteUrl=https://en.wikipedia.org/ (note: mobile and desktop sites are distinct)
[] The data is valid JSON-LD as verified by https://json-ld.org/playground-dev/.
[] The data is linked properly as verified by http://linter.structured-data.org. E.g. http://linter.structured-data.org/?url=https:%2F%2Fde.wikipedia.org%2Fapi%2Frest_v1%2Fpage%2Fhtml%2FDouglas_Adams.
== background
In {T206868} we determined and built a way to add the sameAs property to certain articles. We would now like to do the actual A/B test.
Note: double check that all previous patches have been deployed (and if not swatted). For example, the changes in T207790 must be live prior to launching.
The initial rollout is at **1% sampling** with 50% A/B bucketing split across wikis:
```lang=php
$wgWBClientSettings[ 'pageSchemaNamespaces' ] = [ 0 ];
$wgWBClientSettings[ 'pageSchemaSplitTestSamplingRatio' ] = 0.01;
$wgWBClientSettings[ 'pageSchemaSplitTestBuckets' ] = [ 'control', 'treatment' ];
```
[[ https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/473079/ | The patch for the above configuration is here. ]] Note that the order of the bucket config matters and affects the query as mentioned in T208763.
== Blocking tasks that must be completed before deployment ==
[x] T208789 (results are in! this is good to go but would benefit from review by @tbayer)
[x] T209315 (help requested from @tbayer)
[] T209309
[] T208772 (in progress by @pmiazga)
[] T208909 (must be QA'd by @tbayer)
== Blocking patches that must appear on branch before deployment ==
[x] T206868, Update: add split testing to page schemas, **wmf/1.33.0-wmf.3**, I6eb59214835054fbaa42145da716e59f911e58df
[x] T198946, Update: add Article schema to pages for SEO, **wmf/1.33.0-wmf.1, wmf/1.33.0-wmf.2, wmf/1.33.0-wmf.3**, I02bc43f16cc5308b2613d1cc441df8232a055dbb
[x] T198946, Hygiene: move SEO asset to mediawiki-config, **wmf/1.33.0-wmf.1, wmf/1.33.0-wmf.2, wmf/1.33.0-wmf.3**, I2cfcdf02fe9fe99352067c0b8753353971b405a3
[x] T208796, Update: use wikibase-debug logger instead of "PageRandomLookup", **NONE**, I7a5b9e9cedff13a1f1e4bcb9efbd5a5a9f27f3a4 ([[ https://wikitech.wikimedia.org/wiki/Deployments#Wednesday,_November_14 | slotted for ]] @niedzielski)
[x] T207790, Define WikimediaMessages for Wikibase SEO change, **NONE**, I8345d95a774531ff62bda4a8766f848a847921ee
[x] T207790, Update: add Wikimedia logo for SEO, **NONE**, Ie082fd8cdad623048a4be0bd8f026218ee0f3da5
== Related patches that **do not** block or affect launch:
[x] T208909, Provide a script to reset the page_random column, I914ff44d1212c565e08ff17effbd682dfe7f70fb
[x] T208789, Hygiene: add PageSplitTester test to be used, I176a52a862239ed7781621950a15f0585d5adbc7
[x] T94948, Doc: pageterms query response type for wbptterms, Ia50296965d76837b06706ec15b517fa52febd4de
[x] Doc: result domain for GlobalFunctions::wfRandom(), Ib179d70902e231eaeeafe6449f505464eb25204d
== Acceptance criteria
[] Ensure all subtasks have been resolved. If not, and this is in the sprint, point this out to your product owner or tech lead.
[] Launch sameAs A/B test on all projects but the ones listed below at 1% sampling and 50% `[ 'control', 'treatment' ]` bucketing for main namespace pages. This means that .005 of pages will receive the new schema.
-- Do not launch on: Indonesian, Portuguese, Punjabi, Dutch, Korean, Bhojpuri, Cherokee, Kazakh, Catalan, French, Yoruba, Kalmyk
[] Verify all is well by performing the QA steps identified in T208772 on the pages bucketed in T208789, checking the [[ https://www.google.com/webmasters/tools/structured-data?hl=en&authuser=1&siteUrl=https://en.wikipedia.org/ | Google Developer console ]] for mobile and nonmobile issues across wikis, and checking in on outside reports with @ovasileva.
[] Increase the rollout gradually to 100% (50% of main namespace pages will receive the treatment).
[] Make a task to remove the A/B test and enable 100% of main namespace pages to receive the treatment.
== A/B test analysis consideration
- Discard the most popular pages when evaluating the results
- Confirm cache lifespan with @phuedx
- Page bucketing does not change during the test but sampling and caching will affect when schemas are visible
- The main page and redirects are not treated specially by the code