Maniphest T209377

Remove A/B test and launch to 100%
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

NOTE: tentative deployment date April 30, 2019

Background

Based on the results from our A/B test, we estimated the effect of sameAs to be a 1.4% increase (95% CI: 0.7-2.1) in average page views per day.

Acceptance Criteria

Delete PageSplitTester and PageSplitTesterTest from Wikibase (this will move to Core in T209319)
Delete PageRandomLookup and PageRandomLookupTest
Delete related configuration variables from Wikibase
Rollout directly to 100% on projects that were not a part of the A/B test

The effect should be to both eliminate some cruft introduced to Wikibase for "sameAs" and to rollout to 100%. If this is too dicey, submit an initial mediawiki-config patch that removes the "control" bucket and then do this task.

QA

Pick a group 0 or 1 wiki such as Commons.
Go to a random main namespace page on the wiki such as Charaxes jasius.
Examine the page source and search for application/ld+json. You should see something like:

<script type="application/ld+json">{"@context":"https:\/\/schema.org","@type":"Article","name":"Charaxes jasius","url":"https:\/\/commons.wikimedia.org\/wiki\/Charaxes_jasius","sameAs":"http:\/\/www.wikidata.org\/entity\/Q1320946","mainEntity":"http:\/\/www.wikidata.org\/entity\/Q1320946","author":{"@type":"Organization","name":"Contributors to Wikimedia projects"},"publisher":{"@type":"Organization","name":"Wikimedia Foundation, Inc.","logo":{"@type":"ImageObject","url":"https:\/\/www.wikimedia.org\/static\/images\/wmf-hor-googpub.png"}},"datePublished":"2005-10-14T21:53:21Z","dateModified":"2019-03-08T00:43:01Z","image":"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/c\/cc\/Charaxes_jasius_MHNT_CUT_2013_3_12_male_Roquebrun_34_Dorsal.jpg","headline":"species of insect"}</script>

If you don't find the LD+JSON blob, purge the cache and check again.
Repeat the process 10 times.

QA Results

AC	Status	Details
1	✅	T209377#5183353

Details

	Subject	Repo	Branch	Lines +/-
	Remove wikibase sameAs A/B test config	operations/mediawiki-config	master	+0 -57
	Remove sameAs A/B test logic	mediawiki/extensions/Wikibase	master	+3 -567

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		Jdlrobson	T209319 SEO: Post-AB test cleanup
Resolved		• nray	T211264 Remove Wikibase page split test configuration code from mediawiki-config
Resolved		Krinkle	T198970 Epic: Implement SEO improvements suggested by Go Fish Digital
Resolved		ovasileva	T209306 [Epic] [SEO] Enable Schema.org Article linked data for all main namespace pages
Resolved		ovasileva	T209377 Remove A/B test and launch to 100%
Resolved		mpopov	T209891 Analyze results of sameAs A/B test
Resolved		ovasileva	T208755 Launch A/B test for sameAs property
Resolved		ovasileva	T206868 [Spike 24hrs] How do we measure the effects of the sameAs property on pageviews using an A/B test
Resolved		None	T198946 Add Schema property 'sameAs' pointing to Wikidata entries
Resolved		Jdlrobson	T204070 [Spike, 8hrs] Where should Schema property 'sameAs' live?
Resolved		None	T207790 Add Wikibase page schema system messages
Resolved		None	T208772 QA page schemas
Resolved		None	T208763 Enable page schemas on the beta cluster
Resolved		Jdforrester-WMF	T208809 All pages on Beta Cluster Wikidata and Commons do not load, "Error: invalid magic word 'translation'"
Resolved		• Tbayer	T208789 Identify pages to be bucketed in page schema linked data A/B test
Resolved		• Tbayer	T208909 [Bug] Update old nonuniformly distributed page_random values
Resolved		None	T208796 Use wikibase-debug Logstash channel to log unexpected page_random values
Resolved		ovasileva	T209309 [Spike, 1hrs] Evaluate approximate page size increase
Invalid		• Tbayer	T209315 Enable Google Developer Access for SEO deployers
Resolved	Jan 13 2019	mpopov	T211191 Check in sameAs A/B test results
Resolved		mpopov	T211190 SameAs A/B test preliminary analysis

Event Timeline

• Niedzielski triaged this task as Medium priority.Nov 13 2018, 3:27 PM

• Niedzielski created this task.

• Niedzielski added a subtask: T208755: Launch A/B test for sameAs property.

• Niedzielski mentioned this in T209319: SEO: Post-AB test cleanup.

• Niedzielski mentioned this in T208755: Launch A/B test for sameAs property.Nov 13 2018, 3:34 PM

ovasileva closed subtask T208755: Launch A/B test for sameAs property as Resolved.Nov 20 2018, 3:32 PM

• Niedzielski added a parent task: T209319: SEO: Post-AB test cleanup.Dec 6 2018, 12:03 AM

• Niedzielski removed a subtask: T208755: Launch A/B test for sameAs property.Dec 6 2018, 12:06 AM

• Niedzielski removed a parent task: T209306: [Epic] [SEO] Enable Schema.org Article linked data for all main namespace pages.

• Niedzielski added a parent task: T211264: Remove Wikibase page split test configuration code from mediawiki-config.Dec 6 2018, 12:14 AM

• Niedzielski removed a parent task: T209319: SEO: Post-AB test cleanup.

Based on the results from T209891#5008750 (1.4% increase in traffic on average), we've decided to roll out the change to all wikipedias

ovasileva moved this task from Incoming to Upcoming on the Web-Team-Backlog board.Apr 12 2019, 10:20 AM

ovasileva updated the task description. (Show Details)

ovasileva updated the task description. (Show Details)Apr 12 2019, 10:23 AM

ovasileva set the point value for this task to 3.Apr 16 2019, 4:19 PM

ovasileva edited projects, added Web-Team-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4); removed Web-Team-Backlog.

ovasileva moved this task from To Do to Blocked on Others on the Web-Team-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4) board.

ovasileva updated the task description. (Show Details)

NOTE: tentative deployment date April 30, 2019

@Tbayer, @cscott, @Addshore, @Gilles, @WMDE-leszek, @Smalyshev, @Tarrow, @CKoerner_WMF, @mpopov we're planning to complete the roll out of the "sameAs" page schema changes soon. Please comment if you have any concerns.

Thanks for the ping. I've added a note to the next issue of Tech News to let folks know of the change. It will go out on 2019-04-22

@Niedzielski all fine from WMDE side

ovasileva moved this task from Blocked on Others to To Do on the Web-Team-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4) board.Apr 17 2019, 5:19 PM

• nray claimed this task.Apr 18 2019, 4:55 PM

• nray moved this task from To Do to Doing on the Web-Team-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4) board.

Change 505050 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/extensions/Wikibase@master] Remove sameAs A/B test logic

https://gerrit.wikimedia.org/r/505050

gerritbot added a project: Patch-For-Review.Apr 19 2019, 12:14 AM

• nray moved this task from Doing to Needs Code Review on the Web-Team-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4) board.Apr 19 2019, 12:47 AM

• nray moved this task from Needs Code Review to Doing on the Web-Team-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4) board.Apr 19 2019, 1:39 AM

• nray moved this task from Doing to Needs Code Review on the Web-Team-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4) board.Apr 19 2019, 1:55 AM

• nray moved this task from Needs Code Review to Doing on the Web-Team-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4) board.Apr 23 2019, 5:16 PM

@nray's patch looks good to me I think we can merge it this week which would allow the train to carry the changes starting on Tuesday (April 30th) as per usual.

@WMDE-leszek, if it is possible, would someone from WMDE review and +2 if it's ok? After deployment, @nray is going to follow up with a mediawiki-config clean up patch to undo the configuration variables added.

Change 505999 had a related patch set uploaded (by Nray; owner: Nray):
[operations/mediawiki-config@master] Remove wikibase sameAs A/B test config

https://gerrit.wikimedia.org/r/505999

In T209377#5132593, @Niedzielski wrote:

@nray's patch looks good to me I think we can merge it this week which would allow the train to carry the changes starting on Tuesday (April 30th) as per usual.

@WMDE-leszek, if it is possible, would someone from WMDE review and +2 if it's ok? After deployment, @nray is going to follow up with a mediawiki-config clean up patch to undo the configuration variables added.

That mediawiki-config patch is at https://gerrit.wikimedia.org/r/505999 if anyone has time to review it

@ovasileva, @mpopov, FYI in reviewing @nray's config patch, I noticed that the following wikis were at 0% and will jump to 100%:

	// Do not test on the Indonesian, Portuguese, Punjabi, Dutch, Korean, Bhojpuri, Cherokee,
	// Kazakh, Catalan, French, Yoruba, and Kalmyk Wikipedias. These Wikipedias have been identified
	// as good control candidates or are part of another ongoing test.
	'idwiki' => 0,
	'ptwiki' => 0,
	'pawiki' => 0,
	'pnbwiki' => 0,
	'nlwiki' => 0,
	'nds_nlwiki' => 0,
	'kowiki' => 0,
	'bhwiki' => 0,
	'chrwiki' => 0,
	'kkwiki' => 0,
	'cawiki' => 0,
	'frwiki' => 0,
	'yowiki' => 0,
	'xalwiki' => 0,

I believe all other wikis are at 50% and will jump to 100%.

• nray moved this task from Needs Code Review to Blocked on Others on the Web-Team-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4) board.Apr 24 2019, 4:53 PM

Friendly ping to the WMDE folks: we're hoping to see this merged before Tuesday's train. If it's not, we'll have to write a special config patch in mediawiki-config to remove the control group from PageSplitTester which would not be ideal.

Change 505050 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Remove sameAs A/B test logic

https://gerrit.wikimedia.org/r/505050

ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.3; 2019-04-30).Apr 26 2019, 2:00 PM

In T209377#5140506, @Niedzielski wrote:

Friendly ping to the WMDE folks: we're hoping to see this merged before Tuesday's train. If it's not, we'll have to write a special config patch in mediawiki-config to remove the control group from PageSplitTester which would not be ideal.

Apologies for sleeping on this one. Merged now.

Thank you so much @WMDE-leszek!!!

ovasileva moved this task from Blocked on Others to To Do on the Web-Team-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4) board.Apr 30 2019, 9:36 AM

ovasileva moved this task from To Do to Blocked on Others on the Web-Team-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4) board.Apr 30 2019, 5:16 PM

Moving to QA as we'd like to QA this as soon as it hits production. We'll move back to to-do for necessary cleanup after this

• Niedzielski updated the task description. (Show Details)May 1 2019, 5:44 PM

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptMay 1 2019, 5:44 PM

@Edtadros, I've added QA steps.

Jdlrobson added a parent task: T209306: [Epic] [SEO] Enable Schema.org Article linked data for all main namespace pages.May 2 2019, 4:09 PM

Test Result

Status: ✅ PASS
OS: macOS Mojave
Browser: Chrome
Device: MBP
Emulated Device: iPhoneX

Test Artifact(s):

QA Steps

Pick a group 0 or 1 wiki such as Commons.
Go to a random main namespace page on the wiki such as Charaxes jasius.
Examine the page source and search for application/ld+json.
If you don't find the LD+JSON blob, purge the cache and check again.
Repeat the process 10 times.

Iteration 1

<script type="application/ld+json">{"@context":"https:\/\/schema.org","@type":"Article","name":"Ana Catarina Silva Pereira","url":"https:\/\/es.wikipedia.org\/wiki\/Ana_Catarina_Silva_Pereira","sameAs":"http:\/\/www.wikidata.org\/entity\/Q28973803","mainEntity":"http:\/\/www.wikidata.org\/entity\/Q28973803","author":{"@type":"Organization","name":"Contributors to Wikimedia projects"},"publisher":{"@type":"Organization","name":"Wikimedia Foundation, Inc.","logo":{"@type":"ImageObject","url":"https:\/\/www.wikimedia.org\/static\/images\/wmf-hor-googpub.png"}},"datePublished":"2019-01-10T15:48:35Z","dateModified":"2019-03-22T13:43:26Z","headline":"jugadora portuguesa de f\u00fatbol sala"}</script>

Iteration 2

<script type="application/ld+json">{"@context":"https:\/\/schema.org","@type":"Article","name":"Estaci\u00f3n de La Plana-Picamoixons","url":"https:\/\/es.wikipedia.org\/wiki\/Estaci%C3%B3n_de_La_Plana-Picamoixons","sameAs":"http:\/\/www.wikidata.org\/entity\/Q8779980","mainEntity":"http:\/\/www.wikidata.org\/entity\/Q8779980","author":{"@type":"Organization","name":"Contributors to Wikimedia projects"},"publisher":{"@type":"Organization","name":"Wikimedia Foundation, Inc.","logo":{"@type":"ImageObject","url":"https:\/\/www.wikimedia.org\/static\/images\/wmf-hor-googpub.png"}},"datePublished":"2009-08-26T17:17:17Z","dateModified":"2019-05-12T22:02:43Z"}</script>

Iteration 3

<script type="application/ld+json">{"@context":"https:\/\/schema.org","@type":"Article","name":"Terapia de choque (cortometraje)","url":"https:\/\/es.wikipedia.org\/wiki\/Terapia_de_choque_(cortometraje)","sameAs":"http:\/\/www.wikidata.org\/entity\/Q19245768","mainEntity":"http:\/\/www.wikidata.org\/entity\/Q19245768","author":{"@type":"Organization","name":"Contributors to Wikimedia projects"},"publisher":{"@type":"Organization","name":"Wikimedia Foundation, Inc.","logo":{"@type":"ImageObject","url":"https:\/\/www.wikimedia.org\/static\/images\/wmf-hor-googpub.png"}},"datePublished":"2015-02-13T16:15:31Z","dateModified":"2015-08-15T20:32:14Z","headline":"Cortometraje espa\u00f1ol de 2010."}</script>

Iteration 4

<script type="application/ld+json">{"@context":"https:\/\/schema.org","@type":"Article","name":"Galcasa","url":"https:\/\/es.wikipedia.org\/wiki\/Galcasa","sameAs":"http:\/\/www.wikidata.org\/entity\/Q16570508","mainEntity":"http:\/\/www.wikidata.org\/entity\/Q16570508","author":{"@type":"Organization","name":"Contributors to Wikimedia projects"},"publisher":{"@type":"Organization","name":"Wikimedia Foundation, Inc.","logo":{"@type":"ImageObject","url":"https:\/\/www.wikimedia.org\/static\/images\/wmf-hor-googpub.png"}},"datePublished":"2014-04-21T05:30:32Z","dateModified":"2019-03-31T20:15:57Z"}</script>

Iteration 5

<script type="application/ld+json">{"@context":"https:\/\/schema.org","@type":"Article","name":"Triforillonia","url":"https:\/\/es.wikipedia.org\/wiki\/Triforillonia","sameAs":"http:\/\/www.wikidata.org\/entity\/Q21447890","mainEntity":"http:\/\/www.wikidata.org\/entity\/Q21447890","author":{"@type":"Organization","name":"Contributors to Wikimedia projects"},"publisher":{"@type":"Organization","name":"Wikimedia Foundation, Inc.","logo":{"@type":"ImageObject","url":"https:\/\/www.wikimedia.org\/static\/images\/wmf-hor-googpub.png"}},"datePublished":"2017-03-26T22:15:15Z","dateModified":"2018-02-26T08:07:41Z","image":"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/4\/48\/Tribrachiidae.JPG"}</script>

Iteration 6

<script type="application/ld+json">{"@context":"https:\/\/schema.org","@type":"Article","name":"Arteria cerebral media","url":"https:\/\/es.wikipedia.org\/wiki\/Arteria_cerebral_media","sameAs":"http:\/\/www.wikidata.org\/entity\/Q498179","mainEntity":"http:\/\/www.wikidata.org\/entity\/Q498179","author":{"@type":"Organization","name":"Contributors to Wikimedia projects"},"publisher":{"@type":"Organization","name":"Wikimedia Foundation, Inc.","logo":{"@type":"ImageObject","url":"https:\/\/www.wikimedia.org\/static\/images\/wmf-hor-googpub.png"}},"datePublished":"2007-11-21T15:45:59Z","dateModified":"2018-11-07T13:20:58Z","image":"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/c\/ca\/Circle_of_Willis_es.svg"}</script>

Iteration 7

<script type="application/ld+json">{"@context":"https:\/\/schema.org","@type":"Article","name":"He Touched Me","url":"https:\/\/es.wikipedia.org\/wiki\/He_Touched_Me","sameAs":"http:\/\/www.wikidata.org\/entity\/Q1754918","mainEntity":"http:\/\/www.wikidata.org\/entity\/Q1754918","author":{"@type":"Organization","name":"Contributors to Wikimedia projects"},"publisher":{"@type":"Organization","name":"Wikimedia Foundation, Inc.","logo":{"@type":"ImageObject","url":"https:\/\/www.wikimedia.org\/static\/images\/wmf-hor-googpub.png"}},"datePublished":"2014-11-08T12:47:41Z","dateModified":"2019-04-04T09:27:35Z","headline":"\u00e1lbum de Elvis Presley"}</script>

Iteration 8

<script type="application/ld+json">{"@context":"https:\/\/schema.org","@type":"Article","name":"Jristina Ioannidi","url":"https:\/\/es.wikipedia.org\/wiki\/Jristina_Ioannidi","sameAs":"http:\/\/www.wikidata.org\/entity\/Q27050157","mainEntity":"http:\/\/www.wikidata.org\/entity\/Q27050157","author":{"@type":"Organization","name":"Contributors to Wikimedia projects"},"publisher":{"@type":"Organization","name":"Wikimedia Foundation, Inc.","logo":{"@type":"ImageObject","url":"https:\/\/www.wikimedia.org\/static\/images\/wmf-hor-googpub.png"}},"datePublished":"2019-05-02T03:09:15Z","dateModified":"2019-05-02T03:09:15Z","headline":"halter\u00f3fila griega"}</script>

Iteration 9

<script type="application/ld+json">{"@context":"https:\/\/schema.org","@type":"Article","name":"Ingeborg Drewitz","url":"https:\/\/es.wikipedia.org\/wiki\/Ingeborg_Drewitz","sameAs":"http:\/\/www.wikidata.org\/entity\/Q71381","mainEntity":"http:\/\/www.wikidata.org\/entity\/Q71381","author":{"@type":"Organization","name":"Contributors to Wikimedia projects"},"publisher":{"@type":"Organization","name":"Wikimedia Foundation, Inc.","logo":{"@type":"ImageObject","url":"https:\/\/www.wikimedia.org\/static\/images\/wmf-hor-googpub.png"}},"datePublished":"2006-06-18T17:27:05Z","dateModified":"2018-10-13T17:08:19Z","headline":"escritora alemana"}</script>

Iteration 10

<script type="application/ld+json">{"@context":"https:\/\/schema.org","@type":"Article","name":"Artern","url":"https:\/\/es.wikipedia.org\/wiki\/Artern","sameAs":"http:\/\/www.wikidata.org\/entity\/Q163346","mainEntity":"http:\/\/www.wikidata.org\/entity\/Q163346","author":{"@type":"Organization","name":"Contributors to Wikimedia projects"},"publisher":{"@type":"Organization","name":"Wikimedia Foundation, Inc.","logo":{"@type":"ImageObject","url":"https:\/\/www.wikimedia.org\/static\/images\/wmf-hor-googpub.png"}},"datePublished":"2018-06-12T12:08:47Z","dateModified":"2018-06-12T12:08:47Z","image":"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/4\/4a\/Bundesarchiv_Bild_146-2006-0013%2C_Luftbild_Artern.jpg"}</script>

Edtadros updated the task description. (Show Details)May 15 2019, 2:22 AM

Looks good, leaving for technical signoff for the remainder of the acceptance criteria

ovasileva updated the task description. (Show Details)May 20 2019, 9:26 AM

Delete PageSplitTester and PageSplitTesterTest from Wikibase (this will move to Core in T209319)

Delete PageRandomLookup and PageRandomLookupTest

Delete related configuration variables from Wikibase

All three were done in 5d62583e179eb35bd406d669474094e6da02e246.