Implement metrics for tool labs (under NDA?)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	valhallasw
	Dec 11 2015, 2:08 PM

Description

To prevent/foresee issues such as T121104, we should have metrics. Because there are privacy implications to this, I think the best way to handle this is to put this behind an interface that is only accessible to people who have signed an NDA.

What are we allowed within the confinements of the privacy policy? I don't think IP is useful, but the refer(r)er could be considered private information -- but that definitely /is/ useful to have.

Just setting up e.g. piwik might be the easiest solution, but there are of course others.

Details

	Project	Branch	Lines +/-	Subject
	operations/puppet	production	+1 -1	toollabs: install goaccess on webproxy hosts

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	bd808	T166406 Program 10 Outcome 3: Outreach
Resolved	Quiddity	T176677 Promote Toolforge Tools and their maintainers within Wikimedia communities
Open	None	T178834 Provide any rough metrics for tool and project usage
Open	None	T129630 Collect and display basic metrics for all tools (service groups)
Open	None	T87001 Provide basic page view metrics for individual tools on toollabs
Resolved	valhallasw	T121233 Implement metrics for tool labs (under NDA?)

Event Timeline

valhallasw created this task.Dec 11 2015, 2:08 PM

valhallasw raised the priority of this task from to Medium.

valhallasw updated the task description. (Show Details)

valhallasw added projects: Cloud-Services, Toolforge.

valhallasw added subscribers: Halfak, Aklapper, yuvipanda, valhallasw.

For reference, this happened earlier with wdsearch on enwiki: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-tech/20131204.txt / http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs/20131204.txt

After looking around for alternatives, http://goaccess.io/man seems to be a reasonable option. It allows for both console-mode and html output, and is available through apt.

Most relevant take-aways, based on the logs that were available:

Yahoo Slurp! uses 20% of our bandwidth. What's our crawler policy?
Geohack is super popular: 12% of all hits are for https://tools.wmflabs.org/geohack/geohack.php. https://tools.wmflabs.org/para/geo/worldadmin98 takes a second place with (6.5%)

Referring sites statistics:

Referring Sites
Top Referring Sites sorted by hits

Hits	%	URL◀
400514	35.51%	tools.wmflabs.org
26710	2.37%	de.wikipedia.org
19478	1.73%	dmowiki.com
10769	0.95%	en.wikipedia.org
4262	 0.38%	es.wikipedia.org
3715	 0.33%	pl.wikipedia.org
3667	 0.33%	wma.wmflabs.org
3188	 0.28%	he.wikipedia.org
2952	 0.26%	ru.wikipedia.org
2007	 0.18%	www.wikidata.org
1967	 0.17%	en.m.wikipedia.org

I have contacted dmowiki to get their incorrect configuration if ImageMapEdit fixed.

As for actual monitoring, it might be more effective to just monitor the access log growth rate, or possibly just size (not sure how often it is rotated).

Change 260566 had a related patch set uploaded (by Merlijn van Deen):
toollabs: install goaccess on webproxy hosts

https://gerrit.wikimedia.org/r/260566

gerritbot added a project: Patch-For-Review.Dec 22 2015, 1:29 PM

See https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin#Local_package_management for documentation.

Ricordisamoa added a subscriber: Ricordisamoa.Dec 29 2015, 5:35 PM

Change 260566 merged by Yuvipanda:
toollabs: install goaccess on webproxy hosts

https://gerrit.wikimedia.org/r/260566

valhallasw moved this task from Triage to Backlog on the Toolforge board.May 27 2016, 12:23 PM

scfc removed a project: Patch-For-Review.Nov 27 2016, 1:34 AM

Nemo_bis added a subscriber: Nemo_bis.Mar 29 2017, 2:23 PM

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:45 PM

Quiddity mentioned this in T178834: Provide any rough metrics for tool and project usage.Oct 23 2017, 7:26 PM