Page MenuHomePhabricator

Add code for tracking the usage of keywords and the overall usage of the extension
Closed, ResolvedPublic

Description

Create the necessary schema and add necessary tracking code to the extension in order to track

  • how often people executed a search from Special:Search with AdvancedSearch enabled
  • how often people used the AdvancedSearch for their search
  • which keyword combination was used how many times

For details see the ticket about the Grafana board: T180571

Details

Related Gerrit Patches:
mediawiki/extensions/AdvancedSearch : masterUpdate AdvancedSearchRequest Logging Schema version
mediawiki/extensions/AdvancedSearch : wmf/1.31.0-wmf.12Also track filetype option
mediawiki/extensions/AdvancedSearch : masterAlso track filetype option
mediawiki/extensions/AdvancedSearch : masterTrack usage of features

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

I assume this includes:

  • how many users enabled the extension as a beta feature, how many disabled it (similar to what we have for TwoColConflict)
  • number of how many searches are submitted using the new form
  • separate tracking for each supported keyword: how many times was each keyword used?

WMDE-Analytics-Engineering WMDE-Design @Jan_Dittrich @Lea_WMDE @Bmueller could you confirm this?

Pablo-WMDE added a subscriber: Pablo-WMDE.EditedOct 11 2017, 12:15 PM

number of how many searches are submitted using the new form

Idea is something along the lines of

mw.track( 'advancedSearch.submit', { 'intitle': true, 'incategory': false, ... } );

@Addshore Do you think this is feasible? Can the second parameter be an object?

Tobi_WMDE_SW updated the task description. (Show Details)Oct 11 2017, 1:30 PM
  • number of how many searches are submitted using the new form

After a quick chat with @Jan_Dittrich, we think it makes also sense to track how many users are actually visiting the new advanced search interface to compare that number with the number of users that actually submitted a search. I've added this to the list of things to track in the task description.

Adding to Tobi’s

visiting the new advanced search interface to compare

…this is basically the same principle we use for checking the turning on/off of the beta features: How many try, how many actually used it.

Addshore added a comment.EditedOct 13 2017, 9:15 AM

number of how many searches are submitted using the new form

If you simply want to count you can use a counter, example @ https://github.com/wikimedia/mediawiki-extensions-RevisionSlider/blob/master/modules/ext.RevisionSlider.init.js#L16

Idea is something along the lines of

mw.track( 'advancedSearch.submit', { 'intitle': true, 'incategory': false, ... } );

@Addshore Do you think this is feasible? Can the second parameter be an object?

This looks like you are trying to go down the event logging route.
An example can be found @ https://github.com/wikimedia/mediawiki-extensions-Flow/blob/71e04ff364650c61c5089e6eff448bece58e0b88/modules/engine/misc/flow-eventlog.js#L20 with the event. prefix.
And an example schema definition @ https://meta.wikimedia.org/wiki/Schema:FlowReplies
For further docs see https://www.mediawiki.org/wiki/Extension:EventLogging/Programming

After a quick chat with @Jan_Dittrich, we think it makes also sense to track how many users are actually visiting the new advanced search interface to compare that number with the number of users that actually submitted a search. I've added this to the list of things to track in the task description.

Is the URL exactly the same as the regular searchpage url?
If there were a difference we could simply look at the page view count for this.

…this is basically the same principle we use for checking the turning on/off of the beta features: How many try, how many actually used it.

In practise the way it is done is very different though, as counting the number of beta feature turn ofns / offs is actually done using a daily DB query and diffing the results.

Is the URL exactly the same as the regular searchpage url?
If there were a difference we could simply look at the page view count for this.

AFAIK it is exactly the same URL, Special:Search. So I guess we would add tracking to the extension entrypoint and just count how often the extension is loaded. Does that make sense?

Is the URL exactly the same as the regular searchpage url?
If there were a difference we could simply look at the page view count for this.

AFAIK it is exactly the same URL, Special:Search. So I guess we would add tracking to the extension entrypoint and just count how often the extension is loaded. Does that make sense?

Yup

JeroenDeDauw moved this task from Todo to Doing on the WMDE-Fundraising-Sprint-13 board.
JeroenDeDauw removed JeroenDeDauw as the assignee of this task.Oct 19 2017, 11:45 AM
JeroenDeDauw added a subscriber: JeroenDeDauw.
kai.nissen moved this task from Todo to Doing on the WMDE-Fundraising-Sprint-13 board.

Change 388131 had a related patch set uploaded (by Kai Nissen (WMDE); owner: Kai Nissen (WMDE)):
[mediawiki/extensions/AdvancedSearch@master] Track usage of features

https://gerrit.wikimedia.org/r/388131

This requires Extension:EventLogging. The schema is hosted on meta can be retrieved from there on any other wiki.

If you want to run this in your local environment:

  • refer to the extensions's documentation on how to setup a dummy event consumer
  • make sure to disable ad blocking plugins and the do-not-track feature in your browser
kai.nissen removed kai.nissen as the assignee of this task.Nov 2 2017, 6:19 PM
kai.nissen moved this task from Doing to Review on the WMDE-Fundraising-Funban-2 board.
kai.nissen added a subscriber: kai.nissen.
Lea_WMDE moved this task from Proposed to Todo on the WMDE-QWERTY-Team board.Nov 9 2017, 12:47 PM
GoranSMilovanovic added a comment.EditedNov 10 2017, 9:52 PM

The new Grafana Dashboard is: Mediawiki Advanced Search.

@Tobi from Grafana, defining a query similar to the one that tracks the number of global users for the TwoColConflict extension:

keepLastValue(daily.betafeatures.global_user_counts.totals.advancedSearch,100)

the "advancedSearch" metric - nor anything similar to it - can be found from the default source (which should be Graphite).

Q1. Are the data from the extension sent to Graphite already?

Q2. This scheme is (a) stored where (database, table), and (2) used for what? @Pablo-WMDE

Q1. Are the data from the extension sent to Graphite already?

The extension is not currently deployed in production so you won't find any data in the production graphite.

Lea_WMDE removed GoranSMilovanovic as the assignee of this task.Nov 15 2017, 9:57 AM
Lea_WMDE triaged this task as Medium priority.
Lea_WMDE updated the task description. (Show Details)
Lea_WMDE added a subscriber: GoranSMilovanovic.
Tobi_WMDE_SW renamed this task from track usage of keywords / general use of the extension to Add code for tracking the usage of keywords.Nov 17 2017, 1:12 PM
Tobi_WMDE_SW renamed this task from Add code for tracking the usage of keywords to Add code for tracking the usage of keywords and the overall usage of the extension.
Tobi_WMDE_SW updated the task description. (Show Details)

Change 388131 merged by jenkins-bot:
[mediawiki/extensions/AdvancedSearch@master] Track usage of features

https://gerrit.wikimedia.org/r/388131

Tobi_WMDE_SW closed this task as Resolved.Nov 28 2017, 8:42 AM
Tobi_WMDE_SW claimed this task.
Tobi_WMDE_SW moved this task from Currently in sprint to Done on the WMDE-QWERTY-Team board.

Data is now availible:

mysql:research@analytics-slave.eqiad.wmnet [log]> select count(*) from AdvancedSearchRequest_17379859 limit 10;
+----------+
| count(*) |
+----------+
|      113 |
+----------+
1 row in set (0.00 sec)
Addshore reopened this task as Open.Dec 21 2017, 11:03 AM

This is missing the file_type keywords

Change 399602 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/AdvancedSearch@master] Also track filetype option

https://gerrit.wikimedia.org/r/399602

Change 399602 merged by jenkins-bot:
[mediawiki/extensions/AdvancedSearch@master] Also track filetype option

https://gerrit.wikimedia.org/r/399602

Change 399628 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/AdvancedSearch@wmf/1.31.0-wmf.12] Also track filetype option

https://gerrit.wikimedia.org/r/399628

Change 399628 abandoned by Addshore:
Also track filetype option

Reason:
Will just wait for the train to run this week.

https://gerrit.wikimedia.org/r/399628

Change 403113 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/AdvancedSearch@master] Update AdvancedSearchRequest Logging Schema version

https://gerrit.wikimedia.org/r/403113

Change 403113 merged by jenkins-bot:
[mediawiki/extensions/AdvancedSearch@master] Update AdvancedSearchRequest Logging Schema version

https://gerrit.wikimedia.org/r/403113

Addshore closed this task as Resolved.Jan 12 2018, 12:08 PM
Addshore moved this task from Active 🚁 to Closing ✔️ on the User-Addshore board.
Tobi_WMDE_SW moved this task from Done to Demoed on the WMDE-QWERTY-Team board.Feb 20 2018, 4:54 PM