Page MenuHomePhabricator

[Spike 24hrs] How do we measure the effects of the sameAs property on pageviews using an A/B test
Closed, ResolvedPublic

Description

Background

As we are not sure about the propagation time of adding the sameAs property, the most reliable way to determine whether any changes are observed is adding the property to a subset of pages and looking at any changes using an A/B test

Hypothesis

Adding the Schema.org sameAs property will increase pageviews coming from search engines

Acceptance criteria

  • Determine how to set up an A/B test for the sameAs property
  • Provide an estimate for how long the change will take (is a mid-November deployment viable)
  • No deviation in behavior for logout / login state.

Notes and thoughts so far

  1. We can use page id to bucket pages
  2. We would like to target 50% of traffic by assigning the test or control group to a particular number of the page id (for example, even and odd page ID's)
  3. We would like the ability to avoid the top 100 articles from this test
  4. We don't know if we can do this client-side or if we have to do it server-side
  5. We should look into whether we should analyze the results on wikidata as well. From Tilman's commen: "I have to say that I don't know enough about the topic yet to fully understand the theory of change, i.e. the assumed mechanism that causes the addition of sameAs to increase views. E.g. would we also assume that the linked Wikidata benefits and will be ranked higher? If yes, should the analysis take Wikidata views into account as well?"

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 12 2018, 5:19 PM
ovasileva renamed this task from [Spike] How do we measure the effects of sameAs using an A/B test to [Spike] How do we measure the effects of the sameAs property on pageviews using an A/B test.Oct 12 2018, 5:22 PM
ovasileva triaged this task as High priority.
ovasileva updated the task description. (Show Details)
ovasileva added subscribers: Jdlrobson, Tbayer, mpopov and 3 others.
ovasileva renamed this task from [Spike] How do we measure the effects of the sameAs property on pageviews using an A/B test to [Spike 24hrs] How do we measure the effects of the sameAs property on pageviews using an A/B test.Oct 16 2018, 4:53 PM

Pulling into the sprint for investigation on the viability and timing of an A/B test. After discussing with @JKatzWMF and @Tnegrin, we have settled on the following:

  • deploying sameAs in Mid-November as an A/B test to run throughout the fundraiser (if possible)
  • the test group will be 50% of articles

Updating the description to reflect this as well.

ovasileva updated the task description. (Show Details)Oct 18 2018, 3:21 PM

Just to confirm, that means 50% of articles across all projects?

Just to confirm, that means 50% of articles across all projects?

50% for enwiki in particular. I think we should deploy on a couple of smaller wikis first to take a look at effects and then to 50% on the remaining projects.

Not all articles have wikidata entries, I don't know what is the wikidata entry <-> wiki article coverage.
Do we want to test 50% of all articles, or 50% of articles that have the wikidata entry?

Not all articles have wikidata entries, I don't know what is the wikidata entry <-> wiki article coverage.
Do we want to test 50% of all articles, or 50% of articles that have the wikidata entry?

The former as we want to look at overall change, but being able to check whether the article has a wikidata entry would also be helpful. So long as there's no connection with page id's and wikidata entries, we should be okay. @mpopov - does this make sense for your analysis?

Niedzielski updated the task description. (Show Details)Oct 23 2018, 8:49 PM

Change 469926 had a related patch set uploaded (by Niedzielski; owner: Stephen Niedzielski):
[mediawiki/extensions/Wikibase@master] WIP: add page split testing

https://gerrit.wikimedia.org/r/469926

@Addshore, @WMDE-leszek, the new patch for rolling out page schema A/B tests is up and ready for your review. Please let me know if you have any questions and thank you so much for your continued help!

Niedzielski removed Niedzielski as the assignee of this task.Oct 29 2018, 8:19 AM

WMDE friends seem ready to merge. @pmiazga, please review the new PageRandomLookupTest. @Tbayer, please sign off on A/B testing approach.

atgo added a subscriber: atgo.Nov 1 2018, 9:05 PM

Change 469926 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Update: add split testing to page schemas

https://gerrit.wikimedia.org/r/469926

This task being something of a hot potato didn't go through our standard process and actually encompassed the work of both spiking and doing. Special thanks are deserved by many for its resolution but most especially to @Tbayer, @Addshore, and @WMDE-leszek.

ovasileva assigned this task to pmiazga.Nov 7 2018, 6:08 PM

Just to clarify, currently we will show the sameAs property when:

  • wikidata entity exists for given article
  • article is in enabled namespace
  • the article is bucketed for new treatment.

We bucket articles by page_random property, which means we bucket by the article (not the wikidata entry). If my thinking is right it doesn't fully match
your comment @ovasileva (T206868#4677736). We check if the article is bucketed, not the wikidata entry.

If you're ok with that, please sign off the task, I did all other checks and it looks ok.

pmiazga reassigned this task from pmiazga to ovasileva.Nov 13 2018, 3:54 PM
pmiazga updated the task description. (Show Details)
pmiazga removed a project: Patch-For-Review.

Just to clarify, currently we will show the sameAs property when:

  • wikidata entity exists for given article
  • article is in enabled namespace
  • the article is bucketed for new treatment.

We bucket articles by page_random property, which means we bucket by the article (not the wikidata entry). If my thinking is right it doesn't fully match
your comment @ovasileva (T206868#4677736). We check if the article is bucketed, not the wikidata entry.
If you're ok with that, please sign off the task, I did all other checks and it looks ok.

Confirming that this is correct behavior

ovasileva closed this task as Resolved.Nov 13 2018, 3:59 PM

Resolving. @mpopov, @Tbayer - feel free to reopen if any concerns arise from your side.