Page MenuHomePhabricator

On beta cluster varnish stats process points to production statsd
Closed, ResolvedPublic

Description

Spotted on deployment-cache-parsoid05 but probably is the same on all beta cluster varnishes:

python /usr/local/bin/varnishstatsd --statsd-server=statsd.eqiad.wmnet --key-prefix=varnish.eqiad.backends
python /usr/local/bin/varnishxcps --statsd-server=statsd.eqiad.wmnet
python /usr/local/bin/varnishrls --statsd-server=statsd.eqiad.wmnet

So varnishstatsd, varnishxcps, varnishrls all point to statsd.eqiad.wmnet which is the production entry point. They should be made to point to the labs statsd.

Event Timeline

hashar created this task.Oct 28 2015, 10:08 AM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added a subscriber: hashar.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 28 2015, 10:08 AM
chasemp set Security to None.
hashar added a subscriber: ori.Oct 28 2015, 8:01 PM

The reason is the role classes in modules/role/manifests/cache/statsd.pp all have:

statsd_server => 'statsd.eqiad.wmnet',

If we could make vary with hiera that would let us change the value on Beta-Cluster-Infrastructure.

Change 249490 had a related patch set uploaded (by Hashar):
cache: vary statsd_server with hiera

https://gerrit.wikimedia.org/r/249490

hashar claimed this task.Oct 28 2015, 8:09 PM
hashar moved this task from To Triage to In-progress on the Beta-Cluster-Infrastructure board.
hashar added a project: WorkType-Maintenance.
hashar triaged this task as Normal priority.Nov 2 2015, 8:35 PM
Restricted Application added a project: Operations. · View Herald TranscriptMay 4 2016, 9:13 AM

I have rebased the Puppet patch https://gerrit.wikimedia.org/r/#/c/249490/ that let us vary the statsd server to send metrics to. It has impact on production though.

Originally, I thought about also changing the metric prefix to use BetaMediaWiki. to align with operations/mediawiki-config. That would require a lot of tweaks in puppet for little benefit, so lets drop the idea.

Stats can be seen on https://graphite-labs.wikimedia.org/ under varnish. as varnish.clients.*or varnish.eqiad.*. That clash with the varnish labs project and the metrics reported by Diamond, but that is not a big deal.

The patch has been cherry picked on beta cluster for quite a while already so it is essentially fixed. What is left to do is to review / polish up the puppet patch and get it deployed to production.

Change 249490 merged by BBlack:
cache: vary statsd_server with hiera

https://gerrit.wikimedia.org/r/249490

hashar closed this task as Resolved.Aug 29 2016, 1:56 PM

Has been kindly reviewed and merged in by @BBlack . One less cherry pick!