Page MenuHomePhabricator
Create Task
Maniphest T188749

Deprecation of mw.errors.* metrics
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

While working on the migration of eventlog1001 to eventlog1002 (T185667) I discovered the following puppet role applied to eventlog1001:

role::logging::mediawiki::errors

The functionality seems to be to get udp logs from mwlog1001, parse them and send some metrics to statsd/graphite via the following script:

https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/mediawiki/files/monitoring/mwerrors.py

From what I gathered https://grafana.wikimedia.org/dashboard/db/production-logging uses metrics grabbed from logstash, so I am wondering if anybody uses anymore mw.errors.* somewhere. If not, I'd stop collecting these metrics as soon as possibile.

elukey@neodymium:~$ sudo cumin 'R:class = mediawiki::monitoring::errors' 'uname -a' --dry-run
1 hosts will be targeted:
eventlog1001.eqiad.wmnet

elukey@neodymium:~$ sudo cumin 'R:class = role::logging::mediawiki::errors' 'uname -a' --dry-run
1 hosts will be targeted:
eventlog1001.eqiad.wmnet

elukey@neodymium:~$ sudo cumin 'R:File = /usr/local/bin/mwerrors' 'uname -a' --dry-run
1 hosts will be targeted:
eventlog1001.eqiad.wmnet

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
role::eventlogging::analytics: deprecate mw.errors.* metricsoperations/puppetproduction+2 -125
Customize query in gerrit

Related Objects
Search...

Event Timeline

elukey created this task.Mar 2 2018, 4:36 PM
elukey triaged this task as Medium priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 2 2018, 4:36 PM
gerritbot subscribed.Mar 2 2018, 4:44 PM

Change 415887 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::eventlogging::analytics: deprecate mw.errors.* metrics

https://gerrit.wikimedia.org/r/415887

gerritbot added a project: Patch-For-Review.Mar 2 2018, 4:44 PM
elukey updated the task description. (Show Details)Mar 2 2018, 4:48 PM
fgiunchedi added a comment.Mar 2 2018, 4:57 PM

Sounds good to me, we'd also need to audit dashboards in case we're using it somewhere and replace with logstash metrics.

Krinkle added a comment.Mar 2 2018, 6:00 PM

I’ve got a dashboard search tool we could use to easily check that:
https://gist.github.com/Krinkle/b5ceff5156c1f4cf3568e373cc135bad

Krinkle added a comment.Mar 2 2018, 7:29 PM

It seems that aside from mw.errors.exception, the mw.errors.* metrics were last written to in 2016.

[19:20 UTC] krinkle at graphite1001.eqiad.wmnet in /var/lib/carbon/whisper/mw/errors

 - fatal: Jan 31  2016 rate.wsp ..
 - catchable: May 20  2016 rate.wsp ..
 - query: Sep 22  2016 rate.wsp ..

 - exception: Mar  2 19:20 rate.wsp ..

Found 0 matches for mw.errors in any of our current dashboards.

Just in case the way it queried it was more complex or with templates/wildcards etc, also ran a search just for exception:

Searching for: /[("'{,]exception[^"')}]+/g
...
... checking db/aqs-cassandra-storage (AQS :: Cassandra :: Storage)
[ '"exceptions' ]
...
... checking db/production-logging (Production Logging)
[ '{exception,fatal', '{exception,fatal', '{exception,fatal' ]
...
... checking db/restbase-cassandra-storage (RESTBase :: Cassandra :: Storage)
[ '"exceptions' ]
...
... checking db/restbase-staging-cassandra-storage (RESTBase staging :: Cassandra :: Storage)
[ '"exceptions' ]
...

The only related one here is db/production-logging which uses the logstash metrics instead.

Krinkle edited projects, added Performance-Team (Radar); removed Performance-Team.Mar 2 2018, 7:29 PM
Krinkle moved this task from Limbo to Perf issue on the Performance-Team (Radar) board.
greg edited projects, added Release-Engineering-Team (Watching / External); removed Release-Engineering-Team.Mar 2 2018, 8:29 PM
elukey added a comment.Mar 3 2018, 6:33 PM

Thanks @Krinkle! @fgiunchedi I think we are ready to go, what do you think?

elukey added a parent task: T114199: Upgrade eventlogging servers to Stretch.Mar 5 2018, 8:17 AM
elukey moved this task from Backlog to In Progress on the User-Elukey board.Mar 5 2018, 2:19 PM
fgiunchedi added a comment.Mar 5 2018, 2:22 PM
In T188749#4021055, @elukey wrote:

Thanks @Krinkle! @fgiunchedi I think we are ready to go, what do you think?

Sounds good! Thanks to you and @Krinkle for the audit.

gerritbot added a comment.Mar 5 2018, 2:26 PM

Change 415887 merged by Elukey:
[operations/puppet@production] role::eventlogging::analytics: deprecate mw.errors.* metrics

https://gerrit.wikimedia.org/r/415887

Stashbot subscribed.Mar 5 2018, 2:34 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-05T14:34:35Z] <elukey> graphite metrics mw.error.* deprecated in T188749

elukey set the point value for this task to 5.Mar 5 2018, 2:34 PM
elukey moved this task from Next Up to Done on the Analytics-Kanban board.
Krinkle unsubscribed.Mar 6 2018, 2:25 AM
elukey moved this task from In Progress to Done on the User-Elukey board.Mar 6 2018, 8:40 AM
• Nuria closed this task as Resolved.Mar 26 2018, 9:29 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits