The search feature is now built so we can start thinking about this.

I don't know if logging the exact search terms is going to be acceptable.

As a first step we could log a search action with how many results are returned as well as a link-click with something like "search-result-2" (referring to the second result on the list). These don't seem too controversial and would already tell us if the feature is being used at all.

In T211166#4873741, @SBisson wrote:

The search feature is now built so we can start thinking about this.

I don't know if logging the exact search terms is going to be acceptable.

As a first step we could log a search action with how many results are returned as well as a link-click with something like "search-result-2" (referring to the second result on the list). These don't seem too controversial and would already tell us if the feature is being used at all.

+1 - can we consider splitting the instrumentation task to first just track usage of search at all before considering whether logging specific terms is feasible? I.e., can we start with logging the following:

• # times search is tapped overall
• # times search is tapped per session as well as overall (ie., are people searching multiple times in a single session?)
• # times someone clicks on a search result (without tracking specific links)
• Activity in the help panel before and after search is used (e.g., to help answer how many people go on to ask a question after searching and presumably failing to find a result)

@SBisson -- would we be able to log these simple things (whether search is clicked into and used) in the existing HelpPanel schema? Or would we be making a new schema?

In T211166#4879210, @MMiller_WMF wrote:

@SBisson -- would we be able to log these simple things (whether search is clicked into and used) in the existing HelpPanel schema? Or would we be making a new schema?

It fits in the current schema. We just have to update it to accept a few more action types. It's minor.

Next step: we are going to talk with the Search team to see how they approach this. Then we're going to come up with an approach for our purposes that fits our privacy commitments.

Update: we have asked for privacy advice on whether a simple approach to instrumentation would work, before we even have that conversation with the Search team. If the simple approach looks good, I will make a separate task for the "simple" and "advanced" versions.

Change 488935 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[mediawiki/extensions/GrowthExperiments@master] Help panel: search instrumentation

https://gerrit.wikimedia.org/r/488935

@SBisson -- @nettrom_WMF and I discussed, and developed specific instrumentation criteria for the basic version of instrumenting search.

High level research question: How useful is search as a feature?

More specifically, we want to answer these questions:

• How often do people click into the search bar?
• How often do they type something?
• How many characters do they type?
• How many results are returned?
• Do they click any of the links returned?
• Are higher results more likely to be clicked?
• How often do users run multiple searches?
• Do users who search for things also ask questions or click other links?

This part is up to you, but here's how we were imagining the instrumentation would change:

• Log an event when a user clicks into the search bar.
• Log an event hen a user clicks the back button to get out of the search context.
• Log an event when a search is run, storing how many characters are in the search string, and the number of results -- not the actual search string.
• Log an event when a user clicks on a search result, storing the ranking of the link they clicked, but not the specific link.

In T211166#4936040, @MMiller_WMF wrote:

This part is up to you, but here's how we were imagining the instrumentation would change:

• Log an event when a user clicks into the search bar.

This will be an action of type 'search-focus'. For technically reasons this one is a little trigger happy. It fires a little more than we would expect. It is possible to identify the extra events and filter them out if the exact count per session is going to be looked at.

• Log an event hen a user clicks the back button to get out of the search context.

This will be action: 'back-home' with action_data: { from: 'search' }

A similar event will occur when the user exits the search context by clearing the search input but with action_data: { from: 'blank-search-input' }

• Log an event when a search is run, storing how many characters are in the search string, and the number of results -- not the actual search string.

This will be action: 'search' with action_data: { queryLength: 5, resultCount: 10 } Result count is often going to be 10 because that's the current limit we are using.

• Log an event when a user clicks on a search result, storing the ranking of the link they clicked, but not the specific link.

This will be action: 'link-click' with action_data: 'search-result-N' Where N is the rank of the result, starting at 1

@SBisson -- thanks. This looks good to me. I think it would be good to start moving this toward testing in Test Wiki.

@kostajh and @Catrope -- can we move this toward being tested as we await approval? We want to test in Test Wiki so we can look at the events in Hadoop.

can we move this toward being tested as we await approval? We want to test in Test Wiki so we can look at the events in Hadoop.

I'll review the patch tomorrow if Roan doesn't get to it today, but we won't have it in test wiki until Monday.

Change 488935 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Help panel: search instrumentation

https://gerrit.wikimedia.org/r/488935

Change 489757 had a related patch set uploaded (by Kosta Harlan; owner: Sbisson):
[mediawiki/extensions/GrowthExperiments@wmf/1.33.0-wmf.16] Help panel: search instrumentation

https://gerrit.wikimedia.org/r/489757

Change 489757 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.33.0-wmf.16] Help panel: search instrumentation

https://gerrit.wikimedia.org/r/489757

Mentioned in SAL (#wikimedia-operations) [2019-02-11T19:16:54Z] <catrope@deploy1001> Synchronized php-1.33.0-wmf.16/extensions/GrowthExperiments/: Help panel search instrumentation (T211166) (duration: 00m 47s)