Decommission 'coal' and 'coal-web' services
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Krinkle
	Apr 23 2023, 2:39 PM

Description

As of change 908220 last week, we no longer display Coal metrics on the https://performance.wikimedia.org/ homepage.

For monitoring, we now use Grafana alerts (done).
For the homepage, we now embed Grafana panels (done).
For long-term stability we are transitioning to Prometheus (not a blocker, in progress: T175087).

Work

De-provision coal and coal-web services in Puppet from webperf hosts in Beta Cluster, Eqiad, and Codfw.
De-provision coal configuration in Puppet for Scap deployments.
Remove unused Puppet code relating to coal, coal-web, and the coal Scap target.
Remove unused Hieradata in Puppet (for prod) and in Horizon (for Beta Cluster) relating to coal.
Archive the repository at https://gerrit.wikimedia.org/g/performance/coal . This means it will be closed for code changes, and no longer indexed by Codesearch.
Update Wikitech runbooks and documentation. https://wikitech.wikimedia.org/wiki/Performance/Runbook/Webperf-processor_services

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	webperf: Remove remnants of coal and coal-web	operations/puppet	production	+2 -11
	coal: Uninstall from webperf role and start decom	operations/puppet	production	+7 -245

Customize query in gerrit

Related Objects

Mentioned In: T228380: Tech debt: sunsetting of Graphite
T222874: service::uwsgi should mask uwsgi.service
T225740: Create Prometheus check for Coal service health
Mentioned Here: T175087: Create a navtiming processor for Prometheus

Event Timeline

Krinkle created this task.Apr 23 2023, 2:39 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 23 2023, 2:39 PM

Krinkle mentioned this in T225740: Create Prometheus check for Coal service health.Apr 23 2023, 2:39 PM

Krinkle mentioned this in T222874: service::uwsgi should mask uwsgi.service.Apr 23 2023, 2:41 PM

Krinkle mentioned this in T228380: Tech debt: sunsetting of Graphite.Apr 23 2023, 2:43 PM

Aklapper renamed this task from Decomission 'coal' and 'coal-web' services to Decommission 'coal' and 'coal-web' services.Apr 23 2023, 3:29 PM

Change 910889 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/puppet@production] coal: Uninstall from webperf role and start decom

https://gerrit.wikimedia.org/r/910889

gerritbot added a project: Patch-For-Review.Apr 23 2023, 4:23 PM

We should update https://wikitech.wikimedia.org/wiki/Performance#/media/File:WMF_Performance_Team_infrastructure_2022.png once it's done :)

• larissagaulia moved this task from Inbox, needs triage to Doing: Goals on the Performance-Team board.Apr 24 2023, 6:19 PM

Krinkle triaged this task as Low priority.Apr 24 2023, 6:19 PM

Krinkle claimed this task.May 1 2023, 7:20 PM

Krinkle moved this task from Doing: Goals to Backlog: Maintenance, non-prioritized on the Performance-Team board.May 1 2023, 7:36 PM

Change 910889 merged by Filippo Giunchedi:

[operations/puppet@production] coal: Uninstall from webperf role and start decom

https://gerrit.wikimedia.org/r/910889

Maintenance_bot removed a project: Patch-For-Review.May 9 2023, 3:11 PM

Krinkle closed this task as Resolved.May 9 2023, 3:48 PM

Krinkle updated the task description. (Show Details)

Krinkle updated the task description. (Show Details)May 9 2023, 4:24 PM

Change 925918 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/puppet@production] webperf: Remove remnants of coal and coal-web

https://gerrit.wikimedia.org/r/925918

gerritbot added a project: Patch-For-Review.Jun 1 2023, 7:44 PM

Change 925918 merged by Filippo Giunchedi:

[operations/puppet@production] webperf: Remove remnants of coal and coal-web

https://gerrit.wikimedia.org/r/925918

Maintenance_bot removed a project: Patch-For-Review.Jun 5 2023, 8:10 AM

21:34:16 <Krinkle> cwhite: it seems 'coal' is still running on webperf1003. I guess we didn't absent it and/or intentionally removed it simply with intention to remove by hand but haven't yet?

• lmata added a project: SRE Observability (FY2023/2024-Q2).Sep 27 2023, 2:15 PM

• lmata moved this task from Inbox to Radar on the observability board.

Krinkle reassigned this task from Krinkle to colewhite.Oct 2 2023, 11:34 PM

Krinkle edited projects, added MediaWiki-Platform-Team (Radar); removed Performance-Team.