Page MenuHomePhabricator

Evaluate security concerns of logging beta cluster db queries on tendril
Closed, ResolvedPublic

Description

Release-Engineering-Team wants to log slow queries for debugging and better monitor mysqls there (see parent task). Soon there will be more tools for monitoring mysqls thanks to performance schema (T99485), but using it now has certain blockers and it is not ready to be rolled in now.

I would like from netops evaluating the security concerns of making available the mysql port (3306) on "deployment-db1.deployment-prep.eqiad.wmflabs" and "deployment-db2.deployment-prep.eqiad.wmflabs" for access from tendril (db1011, on the production cluster) to integrate in the current database monitoring system, or proposing another architecture solution.

While, at first, beta cluster should be isolated from the the rest of the network, the same could be said about labsdb instances, and those are monitored using the same tool.

Event Timeline

jcrespo raised the priority of this task from to Medium.
jcrespo updated the task description. (Show Details)
jcrespo added subscribers: jcrespo, hashar.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

This isn't a strictly netops concern but I'll give my opinion anyway: I think it's a very bad idea to have a production monitoring tool monitor hosts in a labs project (or more). They are, by design, entirely different administrative domains with different configuration, gurantees etc. In fact, we're talking about two administrative domains away (prod -> labs -> beta, all three being different). labsdb are under the production realm and so this is not a proper comparison.

As for proposing another architecture… I think this should ultimately be coming from you but… how about deploying the "current database monitoring system" (tendril?) in beta? This also gives us the added benefit of giving us a playing ground/staging area for that system as well.

jcrespo claimed this task.