Page MenuHomePhabricator
Create Task
Maniphest T205862

Expand modern metrics infrastructure coverage (2018-19 Q2 goal)
Closed, ResolvedPublic
Actions

Description

See also goals page for TEC6

  • Plan and execute a new organization scheme for SRE Grafana dashboards
  • Retire >= 80% of production Diamond collectors
  • Provision >= 50% of statsd/Graphite-only metrics in Prometheus

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+56 -0rsyslog: add prometheus-rsyslog-exporter support
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedfgiunchediT205862 Expand modern metrics infrastructure coverage (2018-19 Q2 goal)
 OpenNoneT178690 Better organization for SRE grafana dashboards
 Resolved• Mathew.onipeT212839 Remove "prometheus" from elasticsearch grafana dashboard names
 ResolvedcolewhiteT183454 Deprovision Diamond collectors no longer in use
 DeclinedNoneT206327 Setup metrics monitoring for OpenLDAP/corp
 ResolvedfgiunchediT207041 Adapt Kafka dashboards to use metrics from prometheus-node-exporter
 DuplicateNoneT210991 Deprecate Diamond collectors in Tool Labs / Tool Forge
 ResolvedMoritzMuehlenhoffT210993 Deprecate Diamond collectors in Cloud VPS
 ResolvedMoritzMuehlenhoffT228238 Remove nfsiostat collector for diamond if possible, which may be broken on tools workers
 OpenNoneT264920 Grafana "cloud-vps-project-board" needs to be migrated from Graphite to Prometheus
 OpentaaviT266050 Build Prometheus service for use by all Cloud VPS projects and their instances
 Resolved• JHeddenT250206 Deploy a proof of concept prometheus server in cloudvps to replace shinken
 Resolved• JHeddenT250210 Request creation of metricsinfra VPS project
 ResolvedBstormT252260 Try out removing the nfs client probes from the node exporter on VMs
 Resolvedbd808T256134 wikistats.analytics.eqiad.wmflabs blocking Prometheus scraping from metricsinfra
 ResolvedAndrewT128615 Get rid of Toolforge home page check from shinken
 DeclinedNoneT128716 Make icinga-wm report Tools homepage check at #wikimedia-cloud, too
 OpenNoneT186552 Create alerts for bastion hosts - Usage and latency
 ResolvednskaggsT284973 Request increased quota for metricsinfra Cloud VPS project
 OpentaaviT284993 Enable self-service Prometheus configuration management for project administrators
 OpenNoneT286302 Enable external API access to metricsinfra Prometheus configuration
 ResolvedtaaviT286300 Add a database-backed backend for prometheus-configurator
 ResolvedtaaviT286299 Create initial scaffolding for Prometheus configuration automation
 OpenNoneT288052 Figure out how to deploy metricsinfra Prometheus configuration tooling
 OpenNoneT288058 Add support for more scrape targets configuration types
 ResolvedtaaviT288059 metricsinfra: Add support for static service discovery
 OpenNoneT288060 metricsinfra: Add support for Kubernetes service discovery
 OpenNoneT288061 metricsinfra: Add client TLS support for scrapes
 OpenNoneT288067 metricsinfra: Add hosted Prometheus blackbox exporter
 OpenNoneT288166 metricsinfra: Alert routing configuration
 OpenNoneT128715 Add all Cloud VPS project administrators to the Prometheus notification group for each project
 ResolvedtaaviT285055 Allow project members/admins to ack/silence alerts of that project
 ResolvedjbondT286716 Create solution for developer account authentication for services hosted in Cloud VPS
 ResolvedtaaviT286301 Scale up metricsinfra prometheus beyond one Prometheus instance (Thanos/Cortex/similar)
 ResolvedtaaviT286335 Split metricsinfra alertmanager to separate hosts from prometheus
 ResolvedtaaviT287148 Set up IRC relay for metricsinfra alerts
 OpenNoneT287349 Deploy prometheus pushgateway to metricsinfra
 OpenNoneT288053 Add external meta-monitoring for metricsinfra
 OpenNoneT288168 metricsinfra: Build out default alert rules
 ResolvedtaaviT310799 Upgrade metricsinfra prometheus to bullseye
 ResolveddcaroT310802 Request increased quota for metricsinfra Cloud VPS project
 ResolvedAndrewT288108 Figure out how to deal with security groups when rolling out metricsinfra scraping
 ResolvedcolewhiteT211094 Port DirectorySize diamond collector to a Prometheus exporter

Event Timeline

fgiunchedi created this task.Oct 1 2018, 1:41 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 1 2018, 1:41 PM
fgiunchedi added subtasks: T178690: Better organization for SRE grafana dashboards, T183454: Deprovision Diamond collectors no longer in use, T177196: Port non-deprecated Diamond collectors to Prometheus.Oct 1 2018, 1:42 PM
bd808 added a subscriber: bd808.Oct 1 2018, 4:16 PM
fgiunchedi moved this task from Inbox to In progress on the observability board.Oct 2 2018, 8:47 AM
herron triaged this task as Medium priority.Oct 2 2018, 5:25 PM
fgiunchedi added a project: User-fgiunchedi.Oct 3 2018, 1:15 PM
fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.
fgiunchedi removed a subtask: T177196: Port non-deprecated Diamond collectors to Prometheus.Oct 3 2018, 1:23 PM
gerritbot added a subscriber: gerritbot.Oct 29 2018, 10:03 AM

Change 470345 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] rsyslog: add prometheus-rsyslog-exporter support

https://gerrit.wikimedia.org/r/470345

gerritbot added a project: Patch-For-Review.Oct 29 2018, 10:03 AM
gerritbot added a comment.Nov 5 2018, 2:52 PM

Change 470345 merged by Filippo Giunchedi:
[operations/puppet@production] rsyslog: add prometheus-rsyslog-exporter support

https://gerrit.wikimedia.org/r/470345

CDanis added a subscriber: CDanis.Nov 19 2018, 5:20 PM
MoritzMuehlenhoff closed subtask T183454: Deprovision Diamond collectors no longer in use as Resolved.Dec 18 2018, 4:19 PM
fgiunchedi updated the task description. (Show Details)Jan 16 2019, 1:34 PM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:27 PM
fgiunchedi moved this task from Doing to Backlog on the User-fgiunchedi board.Apr 4 2019, 2:55 PM
fgiunchedi moved this task from In progress to Inbox on the observability board.Apr 15 2019, 2:39 PM
Maintenance_bot removed a project: Patch-For-Review.May 22 2019, 3:52 PM
fgiunchedi closed this task as Resolved.Jul 16 2019, 1:22 PM
fgiunchedi claimed this task.

Resolving, quarter is over, some subtasks still TODO

fgiunchedi removed a subtask: T205870: Fully migrate producers off statsd.Aug 13 2019, 1:12 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL