This is to include OS, Puppet, and database server running.
Description
Details
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | jcrespo | T140788 Labs databases rearchitecture (tracking) | |||
Resolved | jcrespo | T141097 Implement a frontend failover solution for labsdb replicas | |||
Resolved | Cmjohnson | T136860 rack/setup/install/deploy labsdb1009-labsdb1011 | |||
Resolved | jcrespo | T140452 Initial setup and provision of labsdb1009, labsdb1010 and labsdb1011 | |||
Resolved | jcrespo | T149422 Prepare for mariadb 10.1 |
Event Timeline
Change 299121 had a related patch set uploaded (by Jcrespo):
Prepare hosts for labsdb1009, -10 and -11
Change 299127 had a related patch set uploaded (by Jcrespo):
[WIP] Setup the new labsdb hosts with a new role
@chasemp @yuvipanda This already needs your input, as I am editing "your" class hierarchy. I am creating a different role for the replicas as it will probably need very different config than a db::slave. Although maybe db::slave is now a bad name.
This is the smallest of the things pending here- I would like to discuss architecture, versions, user handling, proxies. Please meet with me soon.
@jcrespo one problem w/ T142807 being a subtask here is we have associated this task as a key result for the goal, and we have said T142807 is not part of that. Depending on how strictly you want to interpret the task/subtask relationship it may be better for T142807 to be a subtask of the main parent task itself at T140788
My interpretation of parent/child is blocker/bloquee, not "part of", but feel free to move thing around if that helps you, I am a bit lost with so many tickets about the same. Obviouly T142807 is blocked by, but not part of the goal.
Change 319558 had a related patch set uploaded (by Jcrespo):
Install MariaDB 10.1 on New labsdb replicas
I am also curious!
One small thought -- I think we should strive to contain that convo to T147052 if we can. Not meant to be pedantic or annoying :) I think this task can be closed as scoped in the description actually. Since we did tie in key results with specific tasks (down to exact titles) I would like to try to keep their scopes locked down.
Change 319618 had a related patch set uploaded (by Jcrespo):
labsdb: enable socket authentication
Socket authentication allow for easier administration:
$ sudo mysql Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 80 Server version: 10.1.18-MariaDB MariaDB Server Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
Monitoring of the new hosts can be seen at:
https://grafana.wikimedia.org/dashboard/db/mysql?var-dc=eqiad%20prometheus%2Fops&var-server=labsdb1009
https://grafana.wikimedia.org/dashboard/db/mysql?var-dc=eqiad%20prometheus%2Fops&var-server=labsdb1010
https://grafana.wikimedia.org/dashboard/db/mysql?var-dc=eqiad%20prometheus%2Fops&var-server=labsdb1011
And private monitoring (query monitoring) at:
https://tendril.wikimedia.org/host
Slow queries are identified with no problem.
We will have to do some tuning in the future of the monitoring, but the original scope (OS, Puppet, and database server running) is done.
Mentioned in SAL (#wikimedia-operations) [2016-12-14T16:17:01Z] <chasemp> tcp mysql addition in core fw for labsdb1009/10/11 from labs instances T140452
We'll remove this and all direct labsdb rules here in favor of the proxies only when the the time comes