Wikipedia.org dashboard: Determine if new layout change caused any decreases
Closed, ResolvedPublic4 Story Points

Description

Per some comments in a thread on mediawiki.org, there is a suspicion that the deployment of the collapsed language links has decreased the page views to the Ukrainian Wikipedia as a result of some users browsers being configured to Russian, thereby burying the link to the Ukrainian Wikipedia. Our previous analysis suggests that this suspicion is incorrect, and the page views dashboard also does not seem to indicate there has been a drop, but we should confirm whether this is really the case using statistical analysis.

Hypothesis: deployment of collapsed links to wikipedia.org has significantly decrease page views to https://uk.wikipedia.org. Let's perform whatever analysis we need to to decide whether to reject or accept this hypothesis.

Note: We recently released this new dashboard page to view any changes in the language clickthroughs from the Wikipedia.org portal page (ticket T140816)

Deskana created this task.Aug 25 2016, 4:43 AM
Restricted Application added subscribers: Base, Aklapper. · View Herald TranscriptAug 25 2016, 4:43 AM
Ata added a subscriber: Ata.Aug 25 2016, 6:07 AM
Ijon awarded a token.Aug 25 2016, 6:23 AM
Deskana updated the task description. (Show Details)Aug 25 2016, 4:40 PM
Deskana triaged this task as High priority.
Deskana added a subscriber: debt.

Setting to high priority; this relates to a recent deployment, so should be done as soon as is possible. @debt, please triage accordingly. :-)

debt assigned this task to mpopov.Aug 25 2016, 8:29 PM
debt lowered the priority of this task from High to Normal.

We'll take a look but please be aware, that we only have about a week's worth of data to look at right not, as we only pushed the new portal page layout into production last week (Aug 16, 2016).

Ijon added a subscriber: Ijon.Aug 25 2016, 8:45 PM
mpopov set the point value for this task to 4.Aug 25 2016, 8:45 PM
debt renamed this task from Determine if recent deployment to wikipedia.org caused decrease to Wikipedia.org dashboard: Determine if new layout change caused any decreases.Aug 26 2016, 4:25 PM
debt updated the task description. (Show Details)
mpopov added a subscriber: chelsyx.Sep 7 2016, 6:23 PM

First draft up at: http://wikimedia-research.github.io/Discovery-Research-Portal/ukrainian/

@chelsyx Please review it when you have a chance. Thanks!

Thanks, @mpopov!

In the executive summary, I assume that the "we do have sufficient evidence" is meant to be "we do not have sufficient evidence"? :-)

mpopov added a comment.Sep 7 2016, 6:39 PM

Thanks, @mpopov!

In the executive summary, I assume that the "we do have sufficient evidence" is meant to be "we do not have sufficient evidence"? :-)

Whoops! Will fix. Thanks!

debt added a comment.Sep 8 2016, 5:59 PM

There's a table (Table 2) that has missing data (at least it looks like it) in the description of the table:

  1. The following control time series:
  2. The following control time series:

...and slightly disappointed that you used the dynamic time warping-matched pageview time series but didn't reference The B-52's Time Warp song. :-P

mpopov added a subscriber: TJones.Sep 9 2016, 8:32 PM

@chelsyx & @TJones: please review the new 2nd draft: http://wikimedia-research.github.io/Discovery-Research-Portal/ukrainian/

@Deskana & @debt: Thanks for pointing those mistakes out! I've fixed them in 2nd draft.

TJones added a comment.Sep 9 2016, 9:38 PM

@mpopov: comments sent in email.

@mpopov Looks great! Sent you my comments via email :)

3rd draft: http://wikimedia-research.github.io/Discovery-Research-Portal/ukrainian/

PDF version: https://github.com/wikimedia-research/Discovery-Research-Portal/blob/master/Analyses/Ukrainian%20Wikipedia/report.pdf

@TJones: getting the reference linking right like you suggested took some tinkering but I did it :D you were right, it is better this way.

@chelsyx, @debt, @Deskana, @TJones What do you think? Is this the final draft or does there need to be a 4th?

(I will remove the "3rd draft" subtitle if we agree that it's done, and I will make "Download PDF" into an actual link once the final draft PDF is uploaded on Commons.)

Overall, it looks good. I like the breakout by Ukrainian/non-Ukrainian IPs in fig 2. The ref links were worth it! :)

Did Table 1 disappear? It seems to only be a header.

debt added a comment.Sep 15 2016, 11:34 PM

I've been struggling with this sentence:

"On 16 August 2016, the team deployed the final major patch (for the foreseeable future) wherein the links to Wikipedia..."

Can it be changed to (something like) this:

On 16 August 2016, the team deployed a major patch wherein the links to Wikipedia

I also can't find Table 1 and am also curious as to why the images that are referenced aren't near where we referenced them at (embedded rather than all at the bottom of the report)?

debt added a comment.Sep 19 2016, 11:05 PM

Thanks! I've added the links to the Portal page for Analysis work.

debt closed this task as Resolved.Sep 23 2016, 9:12 PM