Both sessionstore & echostore require that we update a list of egress rules when the makeup of the cluster changes (hosts provisioned and/or decommissioned). As was noted in r1030175, we could make use of external-services-networkpolicies to drop the list of IPs and allow Puppet to populate the related files when something changes. This would eliminate a lot of toil.
Additionally, we might be able to use the Cassandra service name to replace the list of client seed nodes.
brouberol@deploy1002:~$ host cassandra-restbase-a-codfw.external-services.svc.cluster.local 10.194.61.161 Using domain server: Name: 10.194.61.161 Address: 10.194.61.161#53 Aliases: cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.32.229 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.32.226 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.16.111 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.48.237 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.48.234 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.48.142 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.48.182 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.48.170 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.16.240 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.32.73 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.32.232 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.16.153 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.16.237 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.16.243 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.32.191 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.16.237 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.48.234 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.16.111 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.16.240 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.16.243 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.16.153 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.48.142 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.32.191 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.48.182 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.32.229 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.32.226 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.32.73 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.48.170 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.48.237 cassandra-restbase-a-codfw.external-services.svc.cluster.local has address 10.192.32.232 brouberol@deploy1002:~$
NOTE: The behavior here might be client-dependent, and will need to be tested.
See also:
- https://wikitech.wikimedia.org/wiki/Kubernetes/Deployment_Charts#Enabling_egress_to_services_external_to_Kubernetes
- T331894: Improve how we address outside k8s infrastructure from within charts (e.g. network policies)
- https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1030175/comments/65a80fed_6049059a