Page MenuHomePhabricator

Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases
Closed, ResolvedPublic

Description

Per our conversation with @jcrespo:

modules/mysql isn't used by core databases (in favor of modules/mariadb) and is unmaintained. It perhaps could be fully removed and replaced by the mariadb module), but it currently has these users in the tree:

  • hieradata/labs.yaml:mysql::server::use_apparmor: false # The images we have for debian don't support it
  • modules/mediawiki_singlenode/manifests/init.pp: class { '::mysql::server':
  • modules/quarry/manifests/database.pp: class { '::mysql::server':
  • modules/role/manifests/requesttracker/labs.pp: class { 'mysql::server':
  • modules/role/manifests/requesttracker/labs.pp: require => Class['::requesttracker', 'mysql::server'],
  • modules/role/manifests/simplelamp.pp: class { '::mysql::server': # affects multiple projects (!) -> https://tools.wmflabs.org/openstack-browser/puppetclass/role::simplelamp T215662
  • modules/role/manifests/statistics/private.pp: mysql::config::client { 'statistics-private': # Has moved to other places?
  • modules/role/manifests/wikimetrics/staging.pp: class { '::mysql::server':
  • modules/statistics/manifests/compute.pp: mysql::config::client { 'stats-research':
  • modules/statistics/manifests/discovery.pp: ::mysql::config::client { 'discovery-stats':
  • modules/statistics/manifests/mysql_credentials.pp: mysql::config::client { $title:
  • modules/statistics/manifests/wmde.pp: mysql::config::client { 'research-wmde': ( # moved to other file
  • modules/statistics/manifests/wmde/graphite.pp: mysql::config::client { 'research-wmde':

Plus, any other cleanups that we need to do while (if?) we're repurposing the mariadb module to cover misc use cases :)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 391849 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] Remove mysql module from WMF

https://gerrit.wikimedia.org/r/391849

Dzahn added a subscriber: Dzahn.

Created subtask to make quarry use the mariadb module since that is one of the few things still using it.

1978Gage2001 moved this task from Triage to In progress on the DBA board.Dec 11 2017, 9:45 AM
Marostegui moved this task from In progress to Triage on the DBA board.Dec 11 2017, 11:07 AM
jcrespo moved this task from Triage to Meta/Epic on the DBA board.Feb 7 2018, 3:40 PM
Dzahn added a comment.Sep 5 2018, 7:51 PM

T202588 exists for the quarry migration. That will unblock a lot of this.

Mysql module is also used in puppet/modules/profile/manifests/icinga.pp. This should be removed once the transition to stretch is completed.

quarry doesn't use the mysql module anymore since https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/454481/ was merged. i want to confirm on T181205 though

Dzahn added a comment.Sep 13 2018, 9:10 PM

another place where the mysql module is used is "m" in "role(simplelamp)"

modules/role/manifests/simplelamp.pp: class { '::mysql::server':

That class is used by the following instances/projects:

https://tools.wmflabs.org/openstack-browser/puppetclass/role::simplelamp

They would all have to coordinate or switch one by one to a new class "simplelamp2" (with a better name) while simplelamp stays for the transition. Same is true about replacing apache module in them with httpd module.

quarry switched now to mariadb, subtask resolved. updating the list in the ticket description here. thanks to @zhuyifei1999 for that

Dzahn updated the task description. (Show Details)Sep 13 2018, 9:15 PM
Dzahn changed Risk Rating from N/A to default.
Dzahn updated the task description. (Show Details)
Dzahn updated the task description. (Show Details)Sep 13 2018, 9:17 PM
Dzahn updated the task description. (Show Details)Sep 13 2018, 9:26 PM
Dzahn updated the task description. (Show Details)
Dzahn added a comment.Sep 13 2018, 9:30 PM

i think for the ones in the statistics and wikimetrics we should ask Analytics how they feel about converting that to mariadb. doe it make sense to add a tag for their team?

Dzahn added a comment.Oct 18 2018, 3:12 PM

icinga will stop using the mysql module once on stretch. Once T202782 is resolved and einsteinium isn't the prod Icinga server anymore this part will be done.

   if os_version('debian >= stretch') {
        require_package('mariadb-client')
    } else {
        class { 'mysql': }
}

Change 470658 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] delete mediawiki_singlenode module and mediawiki:::install role

https://gerrit.wikimedia.org/r/470658

Change 470726 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] hieradata/labs: remove mysql::server::use_apparmor: false

https://gerrit.wikimedia.org/r/470726

Change 470658 merged by Dzahn:
[operations/puppet@production] delete mediawiki_singlenode module and mediawiki:::install role

https://gerrit.wikimedia.org/r/470658

Dzahn updated the task description. (Show Details)Nov 6 2018, 8:44 PM

Deleted mediawiki_singlenode module. This checked off one more box on the list.

Change 475876 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] profile::icinga: stop using mysql module, rm jessie support

https://gerrit.wikimedia.org/r/475876

Change 475876 merged by Dzahn:
[operations/puppet@production] profile::icinga: stop using mysql module, rm jessie support

https://gerrit.wikimedia.org/r/475876

Change 470726 merged by Dzahn:
[operations/puppet@production] hieradata/labs: remove mysql::server::use_apparmor: false

https://gerrit.wikimedia.org/r/470726

Dzahn updated the task description. (Show Details)Jan 2 2019, 4:53 PM
Dzahn added a comment.Jan 2 2019, 7:43 PM

The only remaining uses of the mysql module are now limited to the statistics module and wikimetrics which i think are maintained by analytics and the simplelamp role (used on a lot of cloud vps instances, needs wmcs involvement?). Suggest pinging/adding those subteams to get this completed.

Restricted Application added a project: Analytics. · View Herald TranscriptJan 2 2019, 7:43 PM

@Dzahn wikimetrics is going to be sunset this quarter, so you won't have to worry about that any more.

OK FINE I'LL DO IT

Milimetric raised the priority of this task from Medium to High.
Milimetric added a project: Analytics-Kanban.
Milimetric moved this task from Incoming to Operational Excellence on the Analytics board.

Change 482693 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Refactor mysql::config::client to mariadb::config::client

https://gerrit.wikimedia.org/r/482693

Change 482693 merged by Ottomata:
[operations/puppet@production] Refactor mysql::config::client to mariadb::config::client

https://gerrit.wikimedia.org/r/482693

Ottomata updated the task description. (Show Details)Feb 8 2019, 8:32 PM

Change 489314 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove role::wikimetrics::staging

https://gerrit.wikimedia.org/r/489314

Change 489314 merged by Ottomata:
[operations/puppet@production] Remove role::wikimetrics::staging

https://gerrit.wikimedia.org/r/489314

Ottomata reassigned this task from Ottomata to Dzahn.Feb 8 2019, 8:45 PM
Ottomata updated the task description. (Show Details)

Thanks Daniel! The Analytics usages are gone. I'm assigning back to you to figure the final WMCS piece.

Change 489328 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] convert simplelamp from mysql to mariadb

https://gerrit.wikimedia.org/r/489328

Dzahn updated the task description. (Show Details)Feb 8 2019, 10:27 PM
Dzahn updated the task description. (Show Details)
Dzahn moved this task from Ready to Deploy to Done on the Analytics-Kanban board.
Dzahn added a comment.Feb 8 2019, 10:30 PM

Thanks Daniel! The Analytics usages are gone. I'm assigning back to you to figure the final WMCS piece.

Yep, thank you too Andrew. I moved it to the "Done" column on your Kanban, i hope that was right. Started with the simplelamp stuff which i kept for last because of the affected users.

Second, mariadb::packages_wmf and mariadb::packages should probably be merged into one.

While I completely agree, that should be a different ticket?

Ottomata removed a subscriber: Ottomata.Feb 11 2019, 2:38 PM

Change 489328 abandoned by Dzahn:
convert simplelamp from mysql to mariadb

Reason:
different approach to make a new role in parallel to allow per-project switching https://gerrit.wikimedia.org/r/c/operations/puppet/ /489339

https://gerrit.wikimedia.org/r/489328

Change 495864 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wmcs/simplelamp: remove mysql puppetization

https://gerrit.wikimedia.org/r/495864

Change 545289 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] CI rspec: update puppet version used in spec tests

https://gerrit.wikimedia.org/r/545289

Change 545289 merged by Jbond:
[operations/puppet@production] CI rspec: update puppet version used in spec tests

https://gerrit.wikimedia.org/r/545289

Dzahn added a comment.Apr 27 2020, 9:15 AM

Projects currently still using the "simplelamp" class:

Project: glampipe

All project instances

Project: gratitude

gratsync.gratitude.eqiad.wmflabs

Project: signwriting

signwriting-swis.signwriting.eqiad.wmflabs
signwriting-swserver.signwriting.eqiad.wmflabs

Project: wildcat

dannyb.wildcat.eqiad.wmflabs

I will make an effort to contact these people and give them a heads-up and then eventually merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/495864 which will remove puppetization of their existing mysql setup.

Change 592653 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] create a role like simplelamp but using mariadb, not mysql

https://gerrit.wikimedia.org/r/592653

Change 592653 merged by Dzahn:
[operations/puppet@production] create simplelamp2, replace mysql->mariadb, apache->httpd

https://gerrit.wikimedia.org/r/592653

Dzahn added a comment.May 8 2020, 8:59 AM

I have sent email to all administrators of the projects using it and told them I created a new role simplelamp2 to replace simplelamp and that we would like to replace it.

Dzahn added a comment.EditedMay 15 2020, 11:07 AM

I have since gotten replies from 2 project owners and was able to remove the role from their instances. 2 are still missing but soon i will just remove them myself if there is no reply. This is very close now, just a tiny bit more patience please and we will be able to resolve this.

Also there is now "role:simplelamp2" which uses httpd instead of apache and mariadb instead of mysql module which projects can use in cloud VPS instead.

Change 596682 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] delete role::simplelamp

https://gerrit.wikimedia.org/r/596682

Dzahn added a comment.May 15 2020, 5:34 PM

Removed simplelamp from the last 2 remaining projects listed in openstack-browser.

I think it's good to go now. Commented on https://gerrit.wikimedia.org/r/c/operations/puppet/+/391849

Change 495864 abandoned by Dzahn:
wmcs/simplelamp: remove mysql puppetization

Reason:
not needed anymore, role::simplelamp2 exists and can be used instead

https://gerrit.wikimedia.org/r/495864

Change 596682 merged by Dzahn:
[operations/puppet@production] delete role::simplelamp

https://gerrit.wikimedia.org/r/596682

jcrespo updated the task description. (Show Details)May 18 2020, 7:58 AM

Change 391849 had a related patch set uploaded (by Dzahn; owner: Jcrespo):
[operations/puppet@production] Remove mysql module from WMF

https://gerrit.wikimedia.org/r/391849

Change 391849 merged by Dzahn:
[operations/puppet@production] Remove mysql module from WMF

https://gerrit.wikimedia.org/r/391849

Dzahn closed this task as Resolved.May 19 2020, 11:03 AM

done! the module has been removed