We should use the Varnish request logs, passed into Hadoop, for this; EventLogging can be used when we have more specific questions.

Query runs is just a simply COUNT(*); for unique users, let's use unique permutations of (ip, user_agent, accept_language)

Ori suggested another option: logstash as described in http://www.bravo-kernel.com/2014/12/setting-up-logstash-1-4-2-to-forward-nginx-logs-to-elasticsearch/ and then get it to logstash-beta.wmflabs.org

As said in the email thread, I'd like an idea of what the ability of researchers to access that machine easily and conveniently is. If the answer is "there isn't one, we'd have to invent it" please let's look for other options.

@Ironholds so far it's on http://logstash-beta.wmflabs.org but I'm not sure what other options do we have for now.

How does one automatically get data out of logstash?

@Ironholds ok, I'll look into this and update here

In T98030#1347031, @Ironholds wrote:

How does one automatically get data out of logstash?

Cron and curl, mostly. There is an Elasticsearch feature called percolator but I don't know that it integrates with logstash.

@Ironholds Now that the service is running in production, and is hooked up to varnish, do we need to do anything here or is it all automatically taken care of? Do you know?

Is it hooked up to the same varnishes? Which varnishes is it hooked up to? What do the requests look like? What's the URL it lives at?

@Ironholds @Smalyshev Sounds like you two need to have a quick hangout while I'm out to find answers to these questions. :-)

Indeed. @Smalyshev can you poke me Monday?

@Ironholds Sure, will do.