Page MenuHomePhabricator

Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases
Open, HighPublic

Description

Per our conversation with @jcrespo:

First, modules/mysql isn't used by core databases (in favor of modules/mariadb) and is unmaintained. It perhaps could be fully removed and replaced by the mariadb module), but it currently has these users in the tree:

  • hieradata/labs.yaml:mysql::server::use_apparmor: false # The images we have for debian don't support it
  • modules/mediawiki_singlenode/manifests/init.pp: class { '::mysql::server':
  • modules/quarry/manifests/database.pp: class { '::mysql::server':
  • modules/role/manifests/requesttracker/labs.pp: class { 'mysql::server':
  • modules/role/manifests/requesttracker/labs.pp: require => Class['::requesttracker', 'mysql::server'],
  • modules/role/manifests/simplelamp.pp: class { '::mysql::server': # affects multiple projects (!) -> https://tools.wmflabs.org/openstack-browser/puppetclass/role::simplelamp T215662
  • modules/role/manifests/statistics/private.pp: mysql::config::client { 'statistics-private': # Has moved to other places?
  • modules/role/manifests/wikimetrics/staging.pp: class { '::mysql::server':
  • modules/statistics/manifests/compute.pp: mysql::config::client { 'stats-research':
  • modules/statistics/manifests/discovery.pp: ::mysql::config::client { 'discovery-stats':
  • modules/statistics/manifests/mysql_credentials.pp: mysql::config::client { $title:
  • modules/statistics/manifests/wmde.pp: mysql::config::client { 'research-wmde': ( # moved to other file
  • modules/statistics/manifests/wmde/graphite.pp: mysql::config::client { 'research-wmde':
  • Second, mariadb::packages_wmf and mariadb::packages should probably be merged into one.

Plus, any other cleanups that we need to do while (if?) we're repurposing the mariadb module to cover misc use cases :)

Event Timeline

faidon created this task.Apr 3 2017, 5:07 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 3 2017, 5:07 PM

Change 391849 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] Remove mysql module from WMF

https://gerrit.wikimedia.org/r/391849

Dzahn added a subscriber: Dzahn.

Created subtask to make quarry use the mariadb module since that is one of the few things still using it.

1978Gage2001 moved this task from Triage to In progress on the DBA board.Dec 11 2017, 9:45 AM
Marostegui moved this task from In progress to Triage on the DBA board.Dec 11 2017, 11:07 AM
jcrespo moved this task from Triage to Meta/Epic on the DBA board.Feb 7 2018, 3:40 PM
Dzahn added a comment.Sep 5 2018, 7:51 PM

T202588 exists for the quarry migration. That will unblock a lot of this.

Mysql module is also used in puppet/modules/profile/manifests/icinga.pp. This should be removed once the transition to stretch is completed.

quarry doesn't use the mysql module anymore since https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/454481/ was merged. i want to confirm on T181205 though

Dzahn added a comment.Sep 13 2018, 9:10 PM

another place where the mysql module is used is "m" in "role(simplelamp)"

modules/role/manifests/simplelamp.pp: class { '::mysql::server':

That class is used by the following instances/projects:

https://tools.wmflabs.org/openstack-browser/puppetclass/role::simplelamp

They would all have to coordinate or switch one by one to a new class "simplelamp2" (with a better name) while simplelamp stays for the transition. Same is true about replacing apache module in them with httpd module.

quarry switched now to mariadb, subtask resolved. updating the list in the ticket description here. thanks to @zhuyifei1999 for that

Dzahn updated the task description. (Show Details)Sep 13 2018, 9:15 PM
Dzahn changed Risk Rating from N/A to default.
Dzahn updated the task description. (Show Details)
Dzahn updated the task description. (Show Details)Sep 13 2018, 9:17 PM
Dzahn updated the task description. (Show Details)Sep 13 2018, 9:26 PM
Dzahn updated the task description. (Show Details)
Dzahn added a comment.Sep 13 2018, 9:30 PM

i think for the ones in the statistics and wikimetrics we should ask Analytics how they feel about converting that to mariadb. doe it make sense to add a tag for their team?

Dzahn added a comment.Oct 18 2018, 3:12 PM

icinga will stop using the mysql module once on stretch. Once T202782 is resolved and einsteinium isn't the prod Icinga server anymore this part will be done.

   if os_version('debian >= stretch') {
        require_package('mariadb-client')
    } else {
        class { 'mysql': }
}

Change 470658 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] delete mediawiki_singlenode module and mediawiki:::install role

https://gerrit.wikimedia.org/r/470658

Change 470726 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] hieradata/labs: remove mysql::server::use_apparmor: false

https://gerrit.wikimedia.org/r/470726

Change 470658 merged by Dzahn:
[operations/puppet@production] delete mediawiki_singlenode module and mediawiki:::install role

https://gerrit.wikimedia.org/r/470658

Dzahn updated the task description. (Show Details)Nov 6 2018, 8:44 PM

Deleted mediawiki_singlenode module. This checked off one more box on the list.

Change 475876 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] profile::icinga: stop using mysql module, rm jessie support

https://gerrit.wikimedia.org/r/475876

Change 475876 merged by Dzahn:
[operations/puppet@production] profile::icinga: stop using mysql module, rm jessie support

https://gerrit.wikimedia.org/r/475876

Change 470726 merged by Dzahn:
[operations/puppet@production] hieradata/labs: remove mysql::server::use_apparmor: false

https://gerrit.wikimedia.org/r/470726

Dzahn updated the task description. (Show Details)Jan 2 2019, 4:53 PM
Dzahn added a comment.Jan 2 2019, 7:43 PM

The only remaining uses of the mysql module are now limited to the statistics module and wikimetrics which i think are maintained by analytics and the simplelamp role (used on a lot of cloud vps instances, needs wmcs involvement?). Suggest pinging/adding those subteams to get this completed.

Restricted Application added a project: Analytics. · View Herald TranscriptJan 2 2019, 7:43 PM

@Dzahn wikimetrics is going to be sunset this quarter, so you won't have to worry about that any more.

OK FINE I'LL DO IT

Milimetric raised the priority of this task from Normal to High.
Milimetric added a project: Analytics-Kanban.
Milimetric moved this task from Incoming to Operational Excellence on the Analytics board.

Change 482693 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Refactor mysql::config::client to mariadb::config::client

https://gerrit.wikimedia.org/r/482693

Change 482693 merged by Ottomata:
[operations/puppet@production] Refactor mysql::config::client to mariadb::config::client

https://gerrit.wikimedia.org/r/482693

Ottomata updated the task description. (Show Details)Feb 8 2019, 8:32 PM

Change 489314 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove role::wikimetrics::staging

https://gerrit.wikimedia.org/r/489314

Change 489314 merged by Ottomata:
[operations/puppet@production] Remove role::wikimetrics::staging

https://gerrit.wikimedia.org/r/489314

Ottomata reassigned this task from Ottomata to Dzahn.Feb 8 2019, 8:45 PM
Ottomata updated the task description. (Show Details)

Thanks Daniel! The Analytics usages are gone. I'm assigning back to you to figure the final WMCS piece.

Change 489328 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] convert simplelamp from mysql to mariadb

https://gerrit.wikimedia.org/r/489328

Dzahn updated the task description. (Show Details)Feb 8 2019, 10:27 PM
Dzahn updated the task description. (Show Details)
Dzahn moved this task from Ready to Deploy to Done on the Analytics-Kanban board.
Dzahn added a comment.Feb 8 2019, 10:30 PM

Thanks Daniel! The Analytics usages are gone. I'm assigning back to you to figure the final WMCS piece.

Yep, thank you too Andrew. I moved it to the "Done" column on your Kanban, i hope that was right. Started with the simplelamp stuff which i kept for last because of the affected users.

Second, mariadb::packages_wmf and mariadb::packages should probably be merged into one.

While I completely agree, that should be a different ticket?

Ottomata removed a subscriber: Ottomata.Feb 11 2019, 2:38 PM

Change 489328 abandoned by Dzahn:
convert simplelamp from mysql to mariadb

Reason:
different approach to make a new role in parallel to allow per-project switching https://gerrit.wikimedia.org/r/c/operations/puppet/ /489339

https://gerrit.wikimedia.org/r/489328

Change 495864 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wmcs/simplelamp: remove mysql puppetization

https://gerrit.wikimedia.org/r/495864

Change 545289 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] CI rspec: update puppet version used in spec tests

https://gerrit.wikimedia.org/r/545289