| Marostegui | |
| Jan 20 2025, 2:26 PM |
| F58924654: Screenshot 2025-03-26 at 13-00-54 Dash.png | |
| Mar 26 2025, 12:02 PM |
Per my IRC chat with @FCeratto-WMF , let's try to create a dashboard to visualize hosts that are depooled.
Right now to check if a host is depooled we have to check:
https://noc.wikimedia.org/dbconfig/eqiad.json
https://noc.wikimedia.org/dbconfig/codfw.json
Or else do it via dbctl get (or with the wrapped), but it would be better if we have a dashboard where we can get that list (and why not even depool and repool from there).
Let's start the discussion on how we want this to look like.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| In Progress | FCeratto-WMF | T384810 MariaDB lifetime management system | |||
| Open | FCeratto-WMF | T384212 Create a dashboard to show depooled hosts | |||
| Open | None | T257814 Use zarcillo as an authoritative inventory of db instances/roles | |||
| Resolved | • Kormat | T257816 Set up replication for zarcillo | |||
| Declined | None | T257819 Add haproxy config for zarcillo | |||
| Open | None | T257821 Add monitoring to ensure consistency between puppet and zarcillo | |||
| Resolved | • Kormat | T257822 Add monitoring to ensure consistency between tendril and zarcillo | |||
| Resolved | • Kormat | T258566 Refactor tendril+zarcillo roles/profiles | |||
| Open | None | T389663 Fetch DB-related data from puppet |
I'm not sure you'd need puppet data to do this. dbctl provides the list of pooled/depooled dbs: https://noc.wikimedia.org/dbconfig/eqiad.json you can get the list of depooled ones by removing anything that's pooled from hostsByName. It's not super clean but I think we can even provide a better list via dbctl (I don't know how to setup grafana dashboard for it though)
Puppet could be used for T257814, see comment https://phabricator.wikimedia.org/T257814#10666643 - for an initial "MVP" implementation of a dashboard indeed we can get away without it.
Change #1135382 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):
[operations/puppet@production] hiera: Add zarcillo service to k8s
Change #1135387 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):
[operations/puppet@production] hiera: Add zarcillo k8s service on traffic server
Change #1135414 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):
[operations/deployment-charts@master] Add zarcillo k8s service
Change #1135432 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/deployment-charts@master] python-webapp: Update modules
Change #1135438 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):
[operations/dns@master] Add zarcillo (aux k8s) CNAME
Change #1135382 merged by Federico Ceratto:
[operations/puppet@production] hiera: Add zarcillo service to k8s
Change #1135696 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):
[operations/deployment-charts@master] Add namespace for zarcillo
Change #1135696 merged by Federico Ceratto:
[operations/deployment-charts@master] Add namespace for zarcillo
Change #1135432 merged by jenkins-bot:
[operations/deployment-charts@master] python-webapp: Update modules
Change #1135438 merged by Federico Ceratto:
[operations/dns@master] Add zarcillo (aux k8s) CNAME
Change #1135414 merged by Federico Ceratto:
[operations/deployment-charts@master] Add zarcillo k8s service
Change #1137314 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):
[operations/deployment-charts@master] values.yaml: Update chart for zarcillo in aux-k8s
Change #1137314 merged by jenkins-bot:
[operations/deployment-charts@master] values.yaml: Update deployment for zarcillo in aux-k8s
Change #1138688 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):
[operations/deployment-charts@master] zarcillo: values.yaml: Update container path
Change #1138688 merged by Federico Ceratto:
[operations/deployment-charts@master] zarcillo: values.yaml: Update container path
Change #1145127 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):
[operations/deployment-charts@master] zarcillo: values.yaml: Fix typo, remove comment
Change #1145127 merged by Clément Goubert:
[operations/deployment-charts@master] zarcillo: values.yaml: Fix typo, remove comment
Change #1146018 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):
[operations/deployment-charts@master] zarcillo: values.yaml: Add FQDN for SNI
Change #1146018 merged by jenkins-bot:
[operations/deployment-charts@master] zarcillo: values.yaml: Add FQDN for SNI
Change #1135387 merged by Federico Ceratto:
[operations/puppet@production] hiera: Add zarcillo k8s service on traffic server
The initial development version is up: https://zarcillo.wikimedia.org/
It is just a prototype at the moment but any early feedback could be beneficial.
Nice work!
Some initial comments/ideas
@Marostegui thanks. I added colors to highlight master hosts, the currently primary datacenter and removed Zone. Do we want to expose all weights or just one row per host or one per instance?
To fetch depooled hosts from the zarcillo db I'm going to need a user with access rights. Ideally a dedicated user with subnet filtering.
I think having the weights there is useful. For the user, that's fine, we can create a user with just SELECT to that DB from the DB you'd be doing it. Which host will it fetch data from? Right now we normally have 10.64.% for eqiad, would that be enough?
I'm told by @Clement_Goubert 10.194.0.0/16 and 10.67.0.0/16 is required for k8s.
Regarding DB users, ideally we could start with a read-only user to access different tables on zarcillo db for the dashboard
Then, the process fetching data from the prod databases (independent from the dashboard) would need a read-only user to run SHOW REPLICA STATUS and accessing the heartbeats, plus an user that can write the output into zarcillo db.
That's fine, please create a username and a random password, commit it on the private puppet repo and I will grab it from there. Once done I will create the read only user.
@Marostegui I created the entries in the private puppet repo, also created a copy for you in your home directory as discussed on IRC.
| username | permissions | host |
| zarcillo_ids_rw_preprod | GRANT SELECT, INSERT, UPDATE, DELETE, CREATE ON zarcillo_preprod | db1215 |
| zarcillo_ids_rw_prod | GRANT SELECT, INSERT, UPDATE, DELETE, CREATE ON zarcillo | db1215 |
The networks to allow are: 10.67.80.0/255.255.248.0
Edit: we discussed using Prometheus and Orchestrator instead of scraping replication data from the databases and review this in future.
Commands for db2230 for the test database:
CREATE DATABASE `zarcillo_preprod`; CREATE USER 'zarcillo_ids_rw_preprod'@'10.67.80.0/255.255.248.0' IDENTIFIED BY 'REPLACEME'; GRANT SELECT, INSERT, UPDATE, DELETE, CREATE ON `zarcillo_preprod`.* TO 'zarcillo_ids_rw_preprod'@'10.67.80.0/255.255.248.0'; FLUSH PRIVILEGES;
I'm adding documentation for the Web UI at https://doc.wikimedia.org/data_persistence/zarcillo/README.html#_web_ui as a way to share progress here. I can also paste the documentation here if desired.
Thank you - can you use https://wikitech.wikimedia.org/wiki/MariaDB ? We tend to publish everything under there. Please complete it all under https://wikitech.wikimedia.org/wiki/Zarcillo