Page MenuHomePhabricator
Create Task
Maniphest T298246

Disable unused services on maps nodes
Closed, ResolvedPublic
Actions

Description

Since we are now serving maps only using tegola and the tile pregeneration in tegola looks like its working fine i think its OK to disable cassandra and tilerator from maps masters.

  • Remove cassandra storage from kartotherian sources
  • Disable cassandra service
  • Disable tilerator service
  • Disable redis
  • Disable triggering tile pregeneration after OSM syncs

After we are confident with the changes running from both clusters for a couple of weeks with services disabled, we can completely decommission them from maps masters.

Details

SubjectRepoBranchLines +/-
maps: remove tilerator and cassandraoperations/puppetproduction+54 -400
maps: remove redisoperations/puppetproduction+3 -54
maps: remove tilerator::regenoperations/puppetproduction+0 -92
maps: remove Cassandra and Tilerator serviceoperations/puppetproduction+2 -202
Disable triggering tile pregeneration on OSM syncsoperations/puppetproduction+1 -15
maps: increase postgres resources across eqiadoperations/puppetproduction+14 -7
maps: tweak postgres configuration settings to use more resourcesoperations/puppetproduction+21 -13
maps: add cassandra toggle, disable cassandra on maps hostsoperations/puppetproduction+15 -6
Disable cassandra dependency from sourcesmaps/kartotherian/deployimposm+0 -18
Disable tilerator in all envs maps are deployedoperations/puppetproduction+3 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

Jgiannelos created this task.Dec 23 2021, 2:58 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptDec 23 2021, 2:58 PM
gerritbot added a comment.Jan 7 2022, 12:37 PM

Change 752145 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[operations/puppet@production] Disable tilerator in all envs maps are deployed

https://gerrit.wikimedia.org/r/752145

gerritbot added a project: Patch-For-Review.Jan 7 2022, 12:37 PM
Jgiannelos renamed this task from Disable unused services on maps{1009,2009} nodes to Disable unused services on maps nodes.Jan 7 2022, 1:36 PM
gerritbot added a comment.Jan 11 2022, 1:18 PM

Change 752145 merged by Hnowlan:

[operations/puppet@production] Disable tilerator in all envs maps are deployed

https://gerrit.wikimedia.org/r/752145

Jgiannelos updated the task description. (Show Details)Jan 11 2022, 2:17 PM
Jgiannelos updated the task description. (Show Details)
Jgiannelos updated the task description. (Show Details)Jan 11 2022, 2:34 PM
gerritbot added a comment.Jan 11 2022, 3:05 PM

Change 753057 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: add cassandra toggle, disable cassandra on maps hosts

https://gerrit.wikimedia.org/r/753057

gerritbot added a comment.Jan 11 2022, 3:18 PM

Change 753058 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[maps/kartotherian/deploy@imposm] Disable cassandra dependency from sources

https://gerrit.wikimedia.org/r/753058

gerritbot added a comment.Jan 11 2022, 4:01 PM

Change 753058 merged by jenkins-bot:

[maps/kartotherian/deploy@imposm] Disable cassandra dependency from sources

https://gerrit.wikimedia.org/r/753058

Jgiannelos updated the task description. (Show Details)Jan 11 2022, 5:14 PM
gerritbot added a comment.Jan 11 2022, 6:13 PM

Change 753111 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[operations/puppet@production] Disable triggering tile pregeneration on OSM syncs

https://gerrit.wikimedia.org/r/753111

gerritbot added a comment.Jan 25 2022, 10:49 AM

Change 753057 merged by Hnowlan:

[operations/puppet@production] maps: add cassandra toggle, disable cassandra on maps hosts

https://gerrit.wikimedia.org/r/753057

gerritbot added a comment.Jan 26 2022, 11:53 AM

Change 757424 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: tweak postgres configuration settings to use more resources

https://gerrit.wikimedia.org/r/757424

gerritbot added a comment.Feb 1 2022, 11:21 AM

Change 757424 merged by Hnowlan:

[operations/puppet@production] maps: tweak postgres configuration settings to use more resources

https://gerrit.wikimedia.org/r/757424

gerritbot added a comment.Feb 1 2022, 12:23 PM

Change 758829 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: increase postgres resources across cluster

https://gerrit.wikimedia.org/r/758829

gerritbot added a comment.Feb 1 2022, 3:39 PM

Change 758829 merged by Hnowlan:

[operations/puppet@production] maps: increase postgres resources across eqiad

https://gerrit.wikimedia.org/r/758829

gerritbot added a comment.Feb 7 2022, 4:55 PM

Change 760619 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: remove tilerator and cassandra

https://gerrit.wikimedia.org/r/760619

gerritbot added a comment.Apr 7 2022, 4:22 PM

Change 753111 abandoned by Jgiannelos:

[operations/puppet@production] Disable triggering tile pregeneration on OSM syncs

Reason:

Tilerator to be removed completely

https://gerrit.wikimedia.org/r/753111

Jgiannelos moved this task from Needs triage to Kanban on the Product-Infrastructure-Team-Backlog-Deprecated board.Apr 29 2022, 12:15 PM
Jgiannelos edited projects, added Product-Infrastructure-Team-Backlog-Deprecated (Kanban); removed Product-Infrastructure-Team-Backlog-Deprecated.
jijiki moved this task from Incoming 🐫 to 🙈🙉🙊Backlog on the serviceops board.Sep 28 2022, 2:19 PM
jijiki moved this task from 🙈🙉🙊Backlog to this.quarter 🍕 on the serviceops board.Nov 15 2022, 8:58 AM
TheDJ added a comment.Nov 15 2022, 12:38 PM

Shall we ask Elon ?

Macro carltondance:

gerritbot added a comment.Nov 24 2022, 5:32 PM

Change 860634 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: remove Cassandra and Tilerator service

https://gerrit.wikimedia.org/r/860634

hnowlan added a comment.Nov 24 2022, 5:44 PM

https://gerrit.wikimedia.org/r/c/operations/puppet/+/860634/ is a simpler attempt at this work - there are many tilerator-related variables that are used in configuration of still-desired services, so rather than strip everything out at once, remove Cassandra, Tilerator and TileratorUI to get the ball rolling (and most importantly, clean up a bunch of space from removing Cassandra to give us more breathing room during replication lag and planet imports).

gerritbot added a comment.Nov 29 2022, 10:49 AM

Change 860634 merged by Hnowlan:

[operations/puppet@production] maps: remove Cassandra and Tilerator service

https://gerrit.wikimedia.org/r/860634

hnowlan updated the task description. (Show Details)Nov 29 2022, 10:54 AM
gerritbot added a comment.Dec 6 2022, 11:54 AM

Change 865053 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: remove tilerator::regen

https://gerrit.wikimedia.org/r/865053

gerritbot added a comment.Dec 6 2022, 12:05 PM

Change 865053 merged by Hnowlan:

[operations/puppet@production] maps: remove tilerator::regen

https://gerrit.wikimedia.org/r/865053

gerritbot added a comment.Dec 6 2022, 12:23 PM

Change 865056 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: remove redis

https://gerrit.wikimedia.org/r/865056

hnowlan added a project: Platform Team Workboards (Platform Engineering Reliability).Dec 12 2022, 2:38 PM
hnowlan moved this task from Backlog to In review on the Platform Team Workboards (Platform Engineering Reliability) board.
gerritbot added a comment.Jan 9 2023, 3:18 PM

Change 865056 merged by Hnowlan:

[operations/puppet@production] maps: remove redis

https://gerrit.wikimedia.org/r/865056

hnowlan changed the task status from Open to In Progress.Jan 9 2023, 3:28 PM
hnowlan claimed this task.
hnowlan triaged this task as Medium priority.
hnowlan updated the task description. (Show Details)
MoritzMuehlenhoff mentioned this in T327513: Upgrade maps servers to bullseye.Mar 20 2023, 10:50 AM
gerritbot added a comment.Mar 21 2023, 2:46 PM

Change 760619 merged by Hnowlan:

[operations/puppet@production] maps: remove tilerator and cassandra

https://gerrit.wikimedia.org/r/760619

hnowlan closed this task as Resolved.Mar 21 2023, 3:22 PM
hnowlan updated the task description. (Show Details)
MoritzMuehlenhoff reopened this task as Open.Mar 22 2023, 1:21 PM
MoritzMuehlenhoff subscribed.

I think we should still remove Cassandra and Java packages on these hosts, otherwise it'll be confusing for all further Cassandra/Java updates if we have stale packages which are no longer referenecd by an apt source (and thus don't flag pending updates either)

hnowlan added a comment.Mar 22 2023, 3:23 PM

Cassandra and Java packages removed!

hnowlan moved this task from In review to Done on the Platform Team Workboards (Platform Engineering Reliability) board.Mar 23 2023, 3:25 PM
hnowlan closed this task as Resolved.Oct 18 2023, 3:24 PM
MSantos awarded a token.Oct 19 2023, 7:02 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits