In T340780, linkrecommendation issued errors for a period of time due to being unable to contact the MariaDB master database its data is in. This happened due to T337812 and the egress rules needed to be updated to follow the move. We should investigate if we can find a way to abstract away from the applications, and the helm charts, those failovers/moves.
There's a couple of paths that we can take. Some that come to mind are in broad strokes:
Drop egress rules altogether and allow all applications to reach out to any resources they want.
Having egress rules that make sure that services don't get to connect to anything they haven't been explicitly configured for has been a requirement (in part set by the then Services team, and in part set by SRE). The logic behind the requirement was Principle_of_least_privilege. The practical gains are both to mitigate RCE attacks that compromise a service as well as avoiding non easily discoverable communication patterns between various applications and services. In essence, we are talking about patterns that are not clearly set in a single easily searchable repo. The anti-pattern that this move tries to avoid is having services talk to each other through either non sanctioned pathways (e.g. via the edge cache CDN instead of via the service proxy) or via a configuration that is set solely in the deployment repo of the application (which doesn't scale if you have tens of services/applications)
In other environments, the same problem is solved by having the recipient of a communication explicitly whitelist it on their end. Unfortunately, this approach isn't/wasn't so each to achieve in our environment due to the sheer size of historical code. It also has the drawback of adding the onus of updates to the receiving side, which might not be necessarily aware that the initiating side's need might not even exist anymore(e.g. it has migrated away from needing that dependency, it has been undeployed, removed the powered functionality etc), leading eventually to a violation of the Principle_of_least_privilege.
For the above reasons, it's probably prudent that we don't forego entirely our egress limitations.
Allow a predefined set of dst ports, representing MariaDB clusters to be reached by all apps in a kubernetes clusters. (chosen solution)
In this setup, we 'd allow all outgoing traffic for a set predefined dst ports (e.g. 3306-3320). Those ports would be agreed up with Data-Persistence, we would add rules to allow access towards those ports for applications that define in their values.yaml files the MariaDB sections they want access to.
This is essentially just a trimmed down version of the above. It substantially trims down the attack surface by allowing only a few TCP ports, selectable by the application. The port range is derived from MariaDB Puppet manifests, so the source of truth is in the hands of Data-Persistence, allowing to add sections as they please with the proper mappings. Developers/Deployers only need to define the sections they want, without caring about the actual TCP ports.
Depending on the original range of those ports, it might also be sustainable long term, meaning we won't be needing to update that port range often. The MariaDB hosts have anyway password authentication and presumably a compromised service wouldn't have access to the passwords.
Note that this is the path we followed for MW-on-K8s. See https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/charts/mediawiki/templates/_mw_helpers.tpl#28
Make overall the WikiKube environment (if not the other k8s environments as well) aware of the moves of MariaDB masters.
What we are talking here about is essentially a more automated and stricter version of the above. Applications set a tunable (that we have to invent and implement), that lists the clusters they want access to, e.g. mariadb_cluster_access: ["m1", "x1", "s1", "etc"], and using some puppetdb/netbox automation we collect all IP addresses and allow, possibly per application, access. The difference from the above one is that we don't end up needing a "contract" of sorts (the port list) that all sides need to honor, but rather always the entirety of the information required (the IPs+ports of all destinations needed by the application). The big drawback here is that we need deployments for rules to be updated. If we go with per applications rules, multiple deployments, if we go with a global rule, at least 1 deployment.
This idea would need some work to sketch out more clearly and implement and I am not entirely sold on the value we 'd derive to be honest.
Ossify MariaDB master/follower endpoints
This is essentially what we 've done with all LVS services. Their IPs are kinda set in stone, probably never to change after their initial deployment, moving the updates to the DNS layer (and even then just for switching DCs or maintenance). Given that we already got dbproxy hosts for the Misc clusters, abstracting away the MariaDB hosts themselves, could we just put those behind LVS, giving them statically assigned, never to change virtual service IPs? The pros would be consistency with everything else and knowing that this that works. Cons are that we might need some LVS updates if we even want to add more ports.