Page MenuHomePhabricator
Create Task
Maniphest T75881

Send MediaWiki profiling on beta cluster to graphite.wmflabs.org
Closed, ResolvedPublic
Actions

Description

We have a graphite instance on labs at http://graphite.wmflabs.org/ . T54357 was to request the MediaWiki profiling to be sent to a central Graphite, though the task got hijacked in favor of setting up Diamond/monitoring. Thus opening a new task.

Will have to:

  • figure out how the profiles metrics are sent to statsd
  • find the IP address to send statsd metric to configure the emitter (labmon1001.eqiad.wmnet)
  • change the profiling sampling rate ( wmf-config/StartProfiler.php has ( mt_rand() % 50 ) == 0

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
beta: Send stats to graphite.wmflabs.orgoperations/mediawiki-configmaster+1 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT53494 Use Beta cluster as a true canary for code deployments (epic)
 StalledNoneT53497 Setup monitoring for Beta Cluster (tracking)
 ResolvedyuvipandaT54357 Set up graphite monitoring for the beta cluster
 ResolvedyuvipandaT54867 monitor that application servers are responding
 ResolvedyuvipandaT72141 Determine first pass list of icinga-alerting data from graphite.wmflabs
 ResolvedyuvipandaT69333 Yell loudly of failed puppet runs on Beta Cluster instances
 DeclinedNoneT65296 puppet labsstatus not reported when using role::puppet::self
 DeclinedyuvipandaT72862 monitor unsigned salt keys
 ResolvedyuvipandaT87063 Monitor the Parsoid backend service on beta cluster
 ResolvedyuvipandaT87087 Monitor mathoid, cxserver, ciotid and apertium services on betacluster
 OpenNoneT87093 Setup monitoring for database servers in beta cluster
 ResolvedfgiunchediT144502 deploy prometheus node_exporter and server to deployment-prep
 InvalidNoneT128357 Beta cluster job queue is unmonitored / potentially not running all jobs
 ResolvedhasharT130184 beta cluster 'labswiki' not referenced in all-labs.dblist causing jobrunner to error out
 ResolvedKrinkleT125976 Run mediawiki::maintenance scripts in Beta Cluster
 OpenNoneT295241 Make icinga-wm complain if beta CI jobs haven't run in some period of time
 ResolvedFeaturebd808T393926 Setup mailing list for automated monitoring reports from Beta Cluster project
 ResolvedFeaturetaaviT393975 Route deployment-prep Prometheus alerts to the betacluster-alerts@lists.wikimedia.org mailing list
 OpenNoneT315695 Add basic MediaWiki/web site up alerting to the Beta Cluster
 ResolvedhasharT75882 Setup Grafana for beta cluster
 ResolvedhasharT75881 Send MediaWiki profiling on beta cluster to graphite.wmflabs.org

Event Timeline

hashar created this task.Nov 25 2014, 2:16 PM
hashar assigned this task to yuvipanda.
hashar removed yuvipanda as the assignee of this task.
hashar raised the priority of this task from to Medium.
hashar updated the task description. (Show Details)
hashar added a project: Beta-Cluster-Infrastructure.
hashar changed Security from none to None.
hashar mentioned this in T75882: Setup Grafana for beta cluster.
hashar added subscribers: Unknown Object (MLST), greg, • demon and 6 others.
hashar moved this task from To Triage to Next: Feature on the Beta-Cluster-Infrastructure board.Nov 25 2014, 2:19 PM
yuvipanda added a comment.Nov 25 2014, 2:20 PM

Statsd runs on labmon1001.eqiad.wmnet for labs.

hashar added a comment.Nov 27 2014, 8:36 AM
In T75881#784336, @yuvipanda wrote:

Statsd runs on labmon1001.eqiad.wmnet for labs.

Can you confirm labmon1001 is an OK receiver for the Beta cluster MediaWiki profiling?

Krinkle added a comment.Dec 2 2014, 5:22 AM

There's at least one consumer of graphite.wmflabs.org (Nagf; https://github.com/wikimedia/nagf; https://tools.wmflabs.org/nagf/) that assumes the root level to be Nova projects with instances inside of that.

Perhaps we should add a level of nesting to prevent it from becoming a chaos? Like instances..

Alternatively, to remain backwards-compatible, we could prefix the non-instance data to allow filtering (e.g. _)

yuvipanda added a comment.Mar 5 2015, 1:35 PM
In T75881#790730, @hashar wrote:

Can you confirm labmon1001 is an OK receiver for the Beta cluster MediaWiki profiling?

Yup. Please prefix all stats with project name (in this case, deployment-prep) and I guess we should be good to go.

@Krinkle: We can query LDAP / wikitech API to get lists of instances, so that should be able to fix this?

gerritbot subscribed.Mar 5 2015, 1:38 PM

Change 194508 had a related patch set uploaded (by Yuvipanda):
beta: Send stats to graphite.wmflabs.org

https://gerrit.wikimedia.org/r/194508

gerritbot added a project: Patch-For-Review.Mar 5 2015, 1:38 PM
yuvipanda added a comment.Mar 5 2015, 1:51 PM

^ might or might not work, I don't actually know enough about how / what statsd stats come from in MW to be sure.

gerritbot added a comment.Mar 5 2015, 2:19 PM

Change 194508 merged by Yuvipanda:
beta: Send stats to graphite.wmflabs.org

https://gerrit.wikimedia.org/r/194508

yuvipanda mentioned this in rOMWC0fe73968537c: beta: Send stats to graphite.wmflabs.org.Mar 5 2015, 2:19 PM
yuvipanda added a comment.Mar 5 2015, 2:30 PM

Alright, that didn't actually change / do anything.

yuvipanda updated the task description. (Show Details)Mar 5 2015, 2:32 PM
hashar closed this task as Resolved.Nov 2 2015, 9:07 PM
hashar claimed this task.

This has been done as part of the Monolog refactoring and we have:

wmf-config/CommonSettings.php:  $wgStatsdMetricPrefix = 'BetaMediaWiki';

Profiling is sent to labmon1001.eqiad.wmnet and profiling is available on graphite.wmflabs.org under BetaMediaWiki.

hashar removed a subtask: T75882: Setup Grafana for beta cluster.Nov 2 2015, 9:08 PM
hashar added a parent task: T75882: Setup Grafana for beta cluster.
hashar moved this task from Next: Feature to Done on the Beta-Cluster-Infrastructure board.Dec 8 2015, 12:13 PM
greg added a project: Essential-Work.Jan 13 2016, 8:38 PM
• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:57 PM
Restricted Application added a project: Release-Engineering-Team (Kanban). · View Herald TranscriptJun 7 2017, 6:57 PM
• Phabricator_maintenance edited projects, added RelEng-Archive-FY201718-Q1; removed Release-Engineering-Team (Kanban).Sep 26 2017, 11:48 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits