Page MenuHomePhabricator

Initial setup and provision of labsdb1009, labsdb1010 and labsdb1011
Closed, ResolvedPublic

Description

This is to include OS, Puppet, and database server running.

Details

Related Gerrit Patches:
operations/puppet : productionlabsdb: enable socket authentication
operations/puppet : productionInstall MariaDB 10.1 on New labsdb replicas
operations/puppet : productionSetup the new labsdb hosts with a new role
operations/puppet : productionPrepare hosts for labsdb1009, -10 and -11

Related Objects

Event Timeline

jcrespo created this task.Jul 15 2016, 10:02 AM
Restricted Application added a project: Cloud-Services. · View Herald TranscriptJul 15 2016, 10:02 AM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald Transcript

Change 299121 had a related patch set uploaded (by Jcrespo):
Prepare hosts for labsdb1009, -10 and -11

https://gerrit.wikimedia.org/r/299121

Change 299121 merged by Jcrespo:
Prepare hosts for labsdb1009, -10 and -11

https://gerrit.wikimedia.org/r/299121

Change 299127 had a related patch set uploaded (by Jcrespo):
[WIP] Setup the new labsdb hosts with a new role

https://gerrit.wikimedia.org/r/299127

@chasemp @yuvipanda This already needs your input, as I am editing "your" class hierarchy. I am creating a different role for the replicas as it will probably need very different config than a db::slave. Although maybe db::slave is now a bad name.

This is the smallest of the things pending here- I would like to discuss architecture, versions, user handling, proxies. Please meet with me soon.

chasemp triaged this task as High priority.Jul 25 2016, 2:11 PM

Change 299127 merged by Jcrespo:
Setup the new labsdb hosts with a new role

https://gerrit.wikimedia.org/r/299127

chasemp renamed this task from Setup and provision labsdb1009, labsdb1010 and labsdb1011 to Initial setup and provision of labsdb1009, labsdb1010 and labsdb1011.Sep 29 2016, 1:01 PM
chasemp updated the task description. (Show Details)

@jcrespo one problem w/ T142807 being a subtask here is we have associated this task as a key result for the goal, and we have said T142807 is not part of that. Depending on how strictly you want to interpret the task/subtask relationship it may be better for T142807 to be a subtask of the main parent task itself at T140788

My interpretation of parent/child is blocker/bloquee, not "part of", but feel free to move thing around if that helps you, I am a bit lost with so many tickets about the same. Obviouly T142807 is blocked by, but not part of the goal.

Change 319558 had a related patch set uploaded (by Jcrespo):
Install MariaDB 10.1 on New labsdb replicas

https://gerrit.wikimedia.org/r/319558

Marostegui added a subscriber: Marostegui.EditedNov 3 2016, 12:41 PM

How are you planning to migrate (or populate data) in these new servers?

How are you planning to migrate (or populate data) in these new servers?

I am also curious!

One small thought -- I think we should strive to contain that convo to T147052 if we can. Not meant to be pedantic or annoying :) I think this task can be closed as scoped in the description actually. Since we did tie in key results with specific tasks (down to exact titles) I would like to try to keep their scopes locked down.

Change 319558 merged by Jcrespo:
Install MariaDB 10.1 on New labsdb replicas

https://gerrit.wikimedia.org/r/319558

Change 319618 had a related patch set uploaded (by Jcrespo):
labsdb: enable socket authentication

https://gerrit.wikimedia.org/r/319618

Change 319618 merged by Jcrespo:
labsdb: enable socket authentication

https://gerrit.wikimedia.org/r/319618

Socket authentication allow for easier administration:

$ sudo mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 80
Server version: 10.1.18-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
jcrespo closed this task as Resolved.Nov 3 2016, 6:13 PM

Monitoring of the new hosts can be seen at:

https://grafana.wikimedia.org/dashboard/db/mysql?var-dc=eqiad%20prometheus%2Fops&var-server=labsdb1009
https://grafana.wikimedia.org/dashboard/db/mysql?var-dc=eqiad%20prometheus%2Fops&var-server=labsdb1010
https://grafana.wikimedia.org/dashboard/db/mysql?var-dc=eqiad%20prometheus%2Fops&var-server=labsdb1011

And private monitoring (query monitoring) at:

https://tendril.wikimedia.org/host

Slow queries are identified with no problem.

We will have to do some tuning in the future of the monitoring, but the original scope (OS, Puppet, and database server running) is done.

Mentioned in SAL (#wikimedia-operations) [2016-12-14T16:17:01Z] <chasemp> tcp mysql addition in core fw for labsdb1009/10/11 from labs instances T140452

Mentioned in SAL (#wikimedia-operations) [2016-12-14T16:17:01Z] <chasemp> tcp mysql addition in core fw for labsdb1009/10/11 from labs instances T140452

We'll remove this and all direct labsdb rules here in favor of the proxies only when the the time comes

Just one minor comment- you want to comment this on the open T141097 or T147051 or T147052. This is an old ticket just for the initial os installation/reimage.