Page MenuHomePhabricator

Implement metrics for tool labs (under NDA?)
Open, NormalPublic

Description

To prevent/foresee issues such as T121104, we should have metrics. Because there are privacy implications to this, I think the best way to handle this is to put this behind an interface that is only accessible to people who have signed an NDA.

What are we allowed within the confinements of the privacy policy? I don't think IP is useful, but the refer(r)er could be considered private information -- but that definitely /is/ useful to have.

Just setting up e.g. piwik might be the easiest solution, but there are of course others.

Event Timeline

valhallasw raised the priority of this task from to Normal.
valhallasw updated the task description. (Show Details)

After looking around for alternatives, http://goaccess.io/man seems to be a reasonable option. It allows for both console-mode and html output, and is available through apt.

Most relevant take-aways, based on the logs that were available:

Referring sites statistics:

Referring Sites
Top Referring Sites sorted by hits

Hits	%	URL◀
400514	35.51%	tools.wmflabs.org
26710	2.37%	de.wikipedia.org
19478	1.73%	dmowiki.com
10769	0.95%	en.wikipedia.org
4262	 0.38%	es.wikipedia.org
3715	 0.33%	pl.wikipedia.org
3667	 0.33%	wma.wmflabs.org
3188	 0.28%	he.wikipedia.org
2952	 0.26%	ru.wikipedia.org
2007	 0.18%	www.wikidata.org
1967	 0.17%	en.m.wikipedia.org

I have contacted dmowiki to get their incorrect configuration if ImageMapEdit fixed.

As for actual monitoring, it might be more effective to just monitor the access log growth rate, or possibly just size (not sure how often it is rotated).

Change 260566 had a related patch set uploaded (by Merlijn van Deen):
toollabs: install goaccess on webproxy hosts

https://gerrit.wikimedia.org/r/260566

Change 260566 merged by Yuvipanda:
toollabs: install goaccess on webproxy hosts

https://gerrit.wikimedia.org/r/260566

valhallasw moved this task from Triage to Backlog on the Toolforge board.May 27 2016, 12:23 PM
Harej added a subscriber: Harej.Jan 24 2018, 11:30 PM