Performance schema statistics were not deployed to public prometheus/grafana because of privacy reasons (it could contain sensitive information AND could reveal user information for wikis with low load, we were told). The alternative was going to be to deploy and aggregate that to tendril or to a private prometheus instance, but that is not that easy/quick.
For short term, we realized all deployers did not have access to that (only to wiki dbs/those with root access). Please check performance schema/sys utility metrics on each database, see if they are helpful, and if there is something exposed that shouldn't be to deployers. It is available on the production, but not pooled with active use db2083. If everythings is correct, we will provide access to all mediawiki servers to all deployers.
Log in to db2083 (sql wikidatawiki -h db2083, use sys), and query some starts/query digest/index stats.
I have added some people to this ticket that may be interested on this- but I require no actionable from you (except test it/break it), also feel free to unubscribe if this is not interesting for you.
For a quick documentation, check my mediawiki query optimization slides at: https://www.slideshare.net/jynus/query-optimization-from-0-to-10-and-up-to-57/158 / https://github.com/jynus/query-optimization
For Security-Team, the request is:
we want your ok to expose low level internal mw database performance metrics to mw deployers for debugging purposes, through the existing mediawiki admin database account. Think of logs, but real time and database layer. Metrics will be accessed, for now, through the SQL interface only. There is a lot of metrics involved (literally billions of them) but it includes file, thread, user, transaction, query and query digest, and indexes individual metrics and counters (aggregations). Currently that access is reserved for mysql root users only.
For DBA s, the actual changes to be done are:
set sql_log_bin=0; GRANT SELECT, EXECUTE ON `sys`.* TO <mw_admin_account>; GRANT SELECT ON `performance_schema`.* TO <mw_admin_account>;
on all mw core dbs.