Page MenuHomePhabricator
Create Task
Maniphest T326266

Remove the WMCS statsd/Graphite service
Closed, ResolvedPublic
Actions

Description

Looks like we're fairly close to being able to being finally able to remove Diamond from our stack. Once that happens, we can start thinking about getting rid of the statsd/Graphite service entirely.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
scap: remove references to cloudmetrics1003/4operations/puppetproduction+0 -4
wmnet: drop cloudmetrics CNAMEsoperations/dnsmaster+1 -5
hieradata: remove more cloudmetrics1003 referencesoperations/puppetproduction+3 -9
cr-cloud: Drop cloudmetrics excemptionsoperations/homer/publicmaster+0 -21
hieradata: drop dmz_cidr excemptions for cloudmetrics1003/4operations/puppetproduction+0 -5
P:wmcs::graphite: disable incoming metricsoperations/puppetproduction+0 -31
wmcs: Disable Graphite query accessoperations/puppetproduction+6 -8
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedtaaviT336774 Current status of cloudmetrics and its components
 ResolvedtaaviT326266 Remove the WMCS statsd/Graphite service
 ResolvedtaaviT317032 Remove Diamond?
 ResolvedtaaviT264920 Grafana "cloud-vps-project-board" needs to be migrated from Graphite to Prometheus
 ResolvedtaaviT266050 Build Prometheus service for use by all Cloud VPS projects and their instances
 Resolved• JHeddenT250206 Deploy a proof of concept prometheus server in cloudvps to replace shinken
 Resolved• JHeddenT250210 Request creation of metricsinfra VPS project
 Resolved• BstormT252260 Try out removing the nfs client probes from the node exporter on VMs
 Resolvedbd808T256134 wikistats.analytics.eqiad.wmflabs blocking Prometheus scraping from metricsinfra
 ResolvedAndrewT128615 Get rid of Toolforge home page check from shinken
 DeclinedNoneT128716 Make icinga-wm report Tools homepage check at #wikimedia-cloud, too
 Resolved• nskaggsT284973 Request increased quota for metricsinfra Cloud VPS project
 ResolvedtaaviT285055 Allow project members/admins to ack/silence alerts of that project
 ResolvedjbondT286716 Create solution for developer account authentication for services hosted in Cloud VPS
 ResolvedtaaviT286301 Scale up metricsinfra prometheus beyond one Prometheus instance (Thanos/Cortex/similar)
 ResolvedtaaviT286335 Split metricsinfra alertmanager to separate hosts from prometheus
 ResolvedtaaviT287148 Set up IRC relay for metricsinfra alerts
 ResolvedtaaviT310799 Upgrade metricsinfra prometheus to bullseye
 ResolveddcaroT310802 Request increased quota for metricsinfra Cloud VPS project
 ResolvedAndrewT288108 Figure out how to deal with security groups when rolling out metricsinfra scraping
 ResolvedhasharT348830 Puppet failure on deploy-1004.devtools.eqiad1.wikimedia.cloud

Event Timeline

taavi created this task.Jan 4 2023, 7:20 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 4 2023, 7:20 PM
taavi renamed this task from Remove the WMCS statsd service to Remove the WMCS statsd/Graphite service.Jan 4 2023, 7:20 PM
taavi added a subtask: T317032: Remove Diamond?.
taavi updated the task description. (Show Details)
bd808 mentioned this in T241284: Service name needed for things wanting to write data to WMCS's statsd service.Jan 4 2023, 11:32 PM
fnegri subscribed.Jan 13 2023, 7:16 PM
fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 6:22 PM
fnegri moved this task from Kanban to Inbox on the cloud-services-team board.
taavi mentioned this in T336774: Current status of cloudmetrics and its components.May 16 2023, 3:27 PM
taavi added a parent task: T336774: Current status of cloudmetrics and its components.May 17 2023, 12:20 PM
fgiunchedi subscribed.May 17 2023, 1:33 PM
taavi claimed this task.Jul 3 2023, 4:16 PM
taavi closed subtask T317032: Remove Diamond? as Resolved.Jul 17 2023, 1:15 PM
gerritbot added a comment.Jul 28 2023, 4:41 PM

Change 942691 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:wmcs::graphite: disable incoming metrics

https://gerrit.wikimedia.org/r/942691

gerritbot added a project: Patch-For-Review.Jul 28 2023, 4:41 PM

Change 942692 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] wmcs: Disable Graphite query access

https://gerrit.wikimedia.org/r/942692

gerritbot added a comment.Aug 7 2023, 9:36 AM

Change 942692 merged by Filippo Giunchedi:

[operations/puppet@production] wmcs: Disable Graphite query access

https://gerrit.wikimedia.org/r/942692

gerritbot added a comment.Aug 7 2023, 9:36 AM

Change 942691 merged by Filippo Giunchedi:

[operations/puppet@production] P:wmcs::graphite: disable incoming metrics

https://gerrit.wikimedia.org/r/942691

Stashbot added a comment.Aug 7 2023, 9:39 AM

Mentioned in SAL (#wikimedia-cloud) [2023-08-07T09:39:21Z] <taavi> cloud vps graphite service was disabled: https://wikitech.wikimedia.org/wiki/News/2023_Cloud_VPS_metrics_changes T326266

Maintenance_bot removed a project: Patch-For-Review.Aug 7 2023, 10:10 AM
gerritbot added a comment.Sep 22 2023, 11:49 AM

Change 960027 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/homer/public@master] cr-cloud: Drop cloudmetrics excemptions

https://gerrit.wikimedia.org/r/960027

gerritbot added a project: Patch-For-Review.Sep 22 2023, 11:49 AM

Change 960028 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: drop dmz_cidr excemptions for cloudmetrics1003/4

https://gerrit.wikimedia.org/r/960028

gerritbot added a comment.Sep 25 2023, 6:55 AM

Change 960028 merged by Majavah:

[operations/puppet@production] hieradata: drop dmz_cidr excemptions for cloudmetrics1003/4

https://gerrit.wikimedia.org/r/960028

gerritbot added a comment.Sep 25 2023, 7:37 AM

Change 960027 merged by jenkins-bot:

[operations/homer/public@master] cr-cloud: Drop cloudmetrics excemptions

https://gerrit.wikimedia.org/r/960027

Stashbot added a comment.Sep 25 2023, 7:49 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-25T07:49:08Z] <taavi> drop cloudmetrics exceptions from cr firewall ACLs https://gerrit.wikimedia.org/r/c/operations/homer/public/+/960027 T326266

Maintenance_bot removed a project: Patch-For-Review.Sep 25 2023, 8:10 AM
gerritbot added a comment.Sep 25 2023, 11:13 AM

Change 960576 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: remove more cloudmetrics1003 references

https://gerrit.wikimedia.org/r/960576

gerritbot added a project: Patch-For-Review.Sep 25 2023, 11:13 AM
gerritbot added a comment.Sep 25 2023, 12:01 PM

Change 960576 merged by Majavah:

[operations/puppet@production] hieradata: remove more cloudmetrics1003 references

https://gerrit.wikimedia.org/r/960576

Maintenance_bot removed a project: Patch-For-Review.Sep 25 2023, 12:10 PM
dancy subscribed.Sep 27 2023, 3:36 PM

https://gerrit.wikimedia.org/r/960576 broke puppet on deploy-1004.devtools.eqiad1.wikimedia.cloud.

Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Class[Profile::Mediawiki::Deployment::Server]: parameter 'statsd' expects a String value, got Undef (file: /etc/puppet/modules/role/manifests/deployment_server.pp, line: 15, column: 5)

class profile::mediawiki::deployment::server in operations/puppet/modules/profile/manifests/mediawiki/deployment/server.pp has a lookup('statsd') call.

hashar mentioned this in T348830: Puppet failure on deploy-1004.devtools.eqiad1.wikimedia.cloud .Oct 13 2023, 6:37 AM
hashar subscribed.
In T326266#9203951, @dancy wrote:

https://gerrit.wikimedia.org/r/960576 broke puppet on deploy-1004.devtools.eqiad1.wikimedia.cloud.

Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Class[Profile::Mediawiki::Deployment::Server]: parameter 'statsd' expects a String value, got Undef (file: /etc/puppet/modules/role/manifests/deployment_server.pp, line: 15, column: 5)

class profile::mediawiki::deployment::server in operations/puppet/modules/profile/manifests/mediawiki/deployment/server.pp has a lookup('statsd') call.

I have independently filed it that this morning as T348830: Puppet failure on deploy-1004.devtools.eqiad1.wikimedia.cloud

hashar added a subtask: T348830: Puppet failure on deploy-1004.devtools.eqiad1.wikimedia.cloud .Oct 13 2023, 6:38 AM
hashar closed subtask T348830: Puppet failure on deploy-1004.devtools.eqiad1.wikimedia.cloud as Resolved.Oct 13 2023, 6:56 AM
gerritbot added a comment.Oct 23 2023, 12:18 PM

Change 967898 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/dns@master] wmnet: drop cloudmetrics CNAMEs

https://gerrit.wikimedia.org/r/967898

gerritbot added a project: Patch-For-Review.Oct 23 2023, 12:18 PM
gerritbot added a comment.Oct 24 2023, 8:52 AM

Change 967898 merged by Majavah:

[operations/dns@master] wmnet: drop cloudmetrics CNAMEs

https://gerrit.wikimedia.org/r/967898

Maintenance_bot removed a project: Patch-For-Review.Oct 24 2023, 9:10 AM
dancy unsubscribed.Oct 24 2023, 2:40 PM
TheresNoTime mentioned this in T349648: Wikifunctions Beta services down, alerting, blocking use and CI.Oct 24 2023, 4:31 PM
hashar unsubscribed.Oct 25 2023, 10:25 AM
taavi closed this task as Resolved.Oct 27 2023, 9:32 AM
gerritbot added a comment.Nov 13 2023, 10:48 AM

Change 973735 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] scap: remove references to cloudmetrics1003/4

https://gerrit.wikimedia.org/r/973735

gerritbot added a project: Patch-For-Review.Nov 13 2023, 10:48 AM
gerritbot added a comment.Nov 13 2023, 11:03 AM

Change 973735 merged by Majavah:

[operations/puppet@production] scap: remove references to cloudmetrics1003/4

https://gerrit.wikimedia.org/r/973735

Maintenance_bot removed a project: Patch-For-Review.Nov 13 2023, 11:10 AM
JeanFred mentioned this in T351574: Internal Server Error on InteGraality.Nov 21 2023, 8:02 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits