Page MenuHomePhabricator

mariadb puppet module doesn't start mysql service in labs (possibly anywhere)
Closed, ResolvedPublic

Description

Adding these classes to an instance in labs:

standard
mariadb::packages
mariadb::config

With this hiera configuration:

mariadb::config::prompt: 'STAGING'
mariadb::config::config: 'mariadb/beta.my.cnf.erb'
mariadb::config::password: 'somedumbpassword'
mariadb::config::datadir: '/srv/sqldata'
mariadb::config::tmpdir: '/srv/tmp'

results in an instance with mariadb installed, but unable to start until mysql_install_db is run.

Event Timeline

thcipriani raised the priority of this task from to Needs Triage.
thcipriani updated the task description. (Show Details)
thcipriani added projects: Staging, acl*sre-team.
thcipriani added subscribers: thcipriani, Springle.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 6 2015, 7:14 PM
coren triaged this task as Medium priority.Mar 6 2015, 8:55 PM
coren added a subscriber: coren.

I'm not sure that should be considered a bug - mysql_install_db is a very, very destructive operation that is probably unwise to automate in the puppet class intended to have the packages installed, especially since you might want to point it at a directory where a database already exists.

I'm not sure that should be considered a bug - mysql_install_db is a very, very destructive operation that is probably unwise to automate in the puppet class intended to have the packages installed, especially since you might want to point it at a directory where a database already exists.

I may have mis-phrased the bug description: I don't necessarily think automating mysql_install_db is the solution—it's just the only way I got mariadb to start.

Maybe it would be a better to say that I expected that adding role::mariadb to a fresh server would result in a server with a running mariadb instance or one that could be started.

Maybe it would be a better to say that I expected that adding role::mariadb to a fresh server would result in a server with a running mariadb instance or one that could be started.

While I sympathize with the request, if this happens it needs to default to off or be put into a separate role for labs. Puppet being responsible for starting/restarting services (or at least databases) is not wise. Plus production never wants mysql_install_db.

While I sympathize with the request, if this happens it needs to default to off or be put into a separate role for labs. Puppet being responsible for starting/restarting services (or at least databases) is not wise. Plus production never wants mysql_install_db.

Considering that we want staging to exactly match prod, can you list the steps involved in getting a new db server up for prod?

  1. Provision box, sign puppet, first run, etc
  2. xtrabackup clone & prepare data from another server
  3. Start MariaDB service, wait for replication to catch up
  4. mediawiki-config commit -- pool server with low load for warm up
  5. mediawiki-config commit -- raise server to normal load

https://wikitech.wikimedia.org/wiki/Setting_up_a_MySQL_replica

The catch is that step #2 should really be from a depooled slave so it completes in reasonable time without saturating the network. Ie, an extra depool/repool step for another node in there.

fgiunchedi edited projects, added DBA; removed Staging.Dec 1 2016, 12:15 AM
fgiunchedi added a subscriber: fgiunchedi.

I don't think this applies anymore but moving on to DBA's radar for confimation

This is quite old indeed and we do not start MySQL everywhere (apart from labs) on purpose. We do not really want Puppet to handle the MySQL service at this point as it can be dangerous.

fgiunchedi closed this task as Invalid.Dec 1 2016, 5:55 PM

Thanks @Marostegui, tentatively resolving

Restricted Application removed a subscriber: Liuxinyu970226. · View Herald TranscriptDec 1 2016, 5:55 PM
jcrespo changed the task status from Invalid to Resolved.Dec 1 2016, 5:58 PM
jcrespo added a subscriber: jcrespo.

In fact Mariadb can start automatically for non production hosts (right now beta, dns-labs, and analytics-labs), so this is actually resolved: https://phabricator.wikimedia.org/diffusion/OPMD/browse/master/manifests/service.pp;057540db856000b8904ce7d30b4e59535c4ea019$45

Krenair added a subscriber: Krenair.Dec 1 2016, 6:26 PM

In fact Mariadb can start automatically for non production hosts (right now beta, dns-labs, and analytics-labs)

I did some digging, 'dns-labs' seems to refer to role::labs::dns which actually runs on labservices1001.wikimedia.org, labservices1002.wikimedia.org, and labtestservices2001.wikimedia.org, and 'analytics-labs' might refer to role::analytics_cluster::database::meta but this runs on analytics1003.eqiad.wmnet (and probably others)