Page MenuHomePhabricator

Implement (anonymous) tracking mechanisms to determine how frequently tools are used
Closed, DeclinedPublic

Description

We have a bot, Reports bot, that generates reports for WikiProjects. I would like for there to be some way to track when these reports are used. This helps us measure our impact by seeing where clicks on reports lead to improved articles. It also helps us determine which tools are used more frequently; this will help us decide which to prioritize. This is high priority because I want to begin collecting this data as soon as possible.

VERY IMPORTANT: I do not want tracking of *users*. I just want aggregate numbers of clicks not otherwise associated with individuals.

Event Timeline

Harej assigned this task to XZise.
Harej raised the priority of this task from to High.
Harej updated the task description. (Show Details)
Harej added a project: WikiProject-X.
Harej added subscribers: Aklapper, Harej, Halfak.
Harej renamed this task from Implement (anonymous) tracking mechanisms to determine who uses our tools to Implement (anonymous) tracking mechanisms to determine how frequently tools are used.Oct 26 2015, 11:01 PM
Harej set Security to None.

Discussed this matter with @Legoktm @Bawolff and @yuvipanda.

Routing clicks through an external tracking service (even one hosted by Labs) is off the table. EventLogging requires JavaScript, which is not readily deployed on a per-page basis (and we are not likely to get a change approved in common.js any time soon).

Yuvipanda offered a good middle of the road option where we track referrer data. This way, we can see that people went from a WikiProject page to an article. This doesn't necessarily tell us through which *report* people clicked the link, but by keeping track of which links are in each report, we can infer this. For example, referrer logs can tell us that people went to the Tripoli article through WikiProject Libya, and our own internal record keeping can tell us that a link to the Tripoli page was included in Report Foo posted to WikiProject Libya at a certain time and date. (I expect that most of the time, the same article won't be linked to twice on the same page; when that does happen, it won't be included in per-tool metrics but for WikiProject-wide metrics.)

If we agree this is a good direction to go in I will ask for an NDA so that I will be able to analyze referrer data. (Again, all information will be collected and analyzed in the aggregate.)

Discussed further with @Capt_Swing.

Using referrer logs may not be feasible. However, we do have options for inferring activity based on (a) pageviews of WikiProject pages; (b) edit activity by members on WikiProject-affiliated pages; and (c) the presence of specific links on a worklist as discussed above. I still would like to have access to referrer data to most concretely demonstrate the links between WikiProject pages and other pages, and am happy to sign relevant non-disclosure agreements to do so, but we have other tools at our disposal.

I imagine this working as some kind of script with the following data inputs:

  • WikiProject member list (as indicated on a project's /Members subpage) and the contribution logs of these members
  • stats.grok.se for page views, at least until the official pageview restbase is online
  • A database we would maintain, keeping track of whenever a page link appears in a given worklist.

The third thing would require the most work and it would require a central WPX Analytics database that other scripts would hook into, including third-party ones; SuggestBot lists are a current example of this. (Or, we just periodic scans of the page for page links and try to associate a link with a report, though this is essentially scraping and likely to be error-prone.)

With this data combined we can have periodic reports prepared.

A point of consideration: the CollaborationKit extension T123028 may do this more directly through EventLogging, which may eliminate the need to implement it through a DB table on our Labs instance and other stuff.

Harej removed XZise as the assignee of this task.Jun 9 2016, 8:34 PM
Harej added a subscriber: XZise.