Page MenuHomePhabricator

Investigate search behavior changes resulting from changes to the mobile Hindi Wikipedia main page
Closed, ResolvedPublic

Description

How has search been affected by changes to mobile main page which rolled out March 26 (T190101)?

See T185584 for additional context.

Event Timeline

Tbayer renamed this task from Investigate search behavior for people coming from Hindi campaign to Investigate search behavior changes resulting from changes to the mobile Hindi Wikipedia main page.Apr 1 2018, 4:54 AM
Tbayer updated the task description. (Show Details)
Tbayer added subscribers: chelsyx, mpopov.

@atgo Per the earlier discussion at T185584, I have clarified the scope of this task as assessing the impact of the design change, rather than assessing the impact of the video campaign. (Question #3 in T185584 still covers the latter regarding search.)

MBinder_WMF moved this task from Triage to Next Up on the Product-Analytics board.Apr 12 2018, 8:22 PM
Restricted Application added a project: Product-Analytics. · View Herald TranscriptApr 19 2018, 12:19 AM

Hi @mpopov @chelsyx when do you think you'll be able to look into this and T191354? Ideally I'd like to have some numbers before the QCi in 2 weeks.

Hi @mpopov @chelsyx when do you think you'll be able to look into this and T191354? Ideally I'd like to have some numbers before the QCi in 2 weeks.

Checking again

Hi @mpopov @chelsyx when do you think you'll be able to look into this and T191354? Ideally I'd like to have some numbers before the QCi in 2 weeks.

Hi @atgo , I should be able to look at it this week

chelsyx moved this task from Next Up to Doing on the Product-Analytics board.

Thanks @chelsyx

I'll be OOO starting on Thursday and @ZMcCune will be taking over on tracking this for me.

Copy from T191354#4176890

We don't see any spike or dip in the total number of search, zero result rate and clickthrough rate, when the Hindi mobile main page change (March 26th), or when the video campaign launch (April 2nd):



In sum, we don't see any impact on search from mobile main page change or video campaign on Hindi Wikipedia.

@atgo Please let me know if you have any question.

chelsyx closed this task as Resolved.May 10 2018, 10:26 PM

Mark this task as resolved. Feel free to reopen it if you have any question.

Thanks @chelsyx . Apologies for the very slow response time on this, I've been OOO on sabbatical and just back now.

There are a couple of things I'd like to understand more here.

  1. There is a steady drop in app searches across this time. What's that all about?
  2. App searches seem incorrectly high as compared to desktop or mobile web. In your first screenshot, they look like 2-4x more searches from the app, but the web's traffic overall looks like about 100x that of the app, so I'm finding this hard to believe. Can you clarify what's going on here?
atgo reopened this task as Open.Jul 13 2018, 6:46 PM

Thanks @chelsyx . Apologies for the very slow response time on this, I've been OOO on sabbatical and just back now.

There are a couple of things I'd like to understand more here.

[...]

App searches seem incorrectly high as compared to desktop or mobile web. In your first screenshot, they look like 2-4x more searches from the app, but the web's traffic overall looks like about 100x that of the app, so I'm finding this hard to believe. Can you clarify what's going on here?

Agree that it would be good to know the answer to this question. (Also, are the data source and queries used for this analysis documented somewhere? Does it have to do with the dataset mentioned at https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Cirrus ?)

Another thing: Since this task was about changes to the mobile web version specifically, it could have been worth separating mobile web and desktop searches in F17605560. Considering the outcome here, it may not yield more insight in this specific case, but it may be worth keeping in mind for T199726: Understand if ru.wikipedia main page changes had impact on mobile users.

chelsyx closed this task as Resolved.Jun 18 2019, 11:01 PM

@atgo Sorry about the late response to your question:

Given that we're not able to recover the raw data now, the following answers are based on my best guesses:

  1. There is a steady drop in app searches across this time. What's that all about?

This looks more like a seasonal change to me.

  1. App searches seem incorrectly high as compared to desktop or mobile web. In your first screenshot, they look like 2-4x more searches from the app, but the web's traffic overall looks like about 100x that of the app, so I'm finding this hard to believe. Can you clarify what's going on here?

This is likely to be the result of different sampling rate on different platform. So the scale of this metric is not providing a lot of useful information (since it is sampled data), but the trend of the metric is.