We need to migrate all of the services that use the aqs-http-gateway chart to make use of the new external-services mechanism, which will make it easier to specify the correct network policies to enable access to their Cassandra and Druid data sources.
The following six are AQS services:
| Service | External Services |
| device-analytics | cassandra aqs |
| edit-analytics | cassandra aqs, druid-public |
| editor-analytics | cassandra aqs, druid-public |
| geo-analytics | cassandra aqs |
| media-analytics | cassandra aqs |
| page-analytics | cassandra aqs |
In addition, we have one more service that uses the same chart:
| Service | External Services |
| image-suggestions | cassandra aqs |
Image-suggestions is not technically an AQS service, which is why it is listed separately.
Currently, we use symlinks to share a set of common network policies in an _aqs2-common_ folder for the cassandra hosts, specific to each data centre:
btullis@deploy1002:/srv/deployment-charts/helmfile.d/services$ find *-analytics -type l -exec ls -o {} \;
lrwxrwxrwx 1 root 34 Nov 1 2023 device-analytics/global-staging.yaml -> ../_aqs2-common_/global-eqiad.yaml
lrwxrwxrwx 1 root 34 Nov 1 2023 device-analytics/global-eqiad.yaml -> ../_aqs2-common_/global-eqiad.yaml
lrwxrwxrwx 1 root 34 Nov 1 2023 device-analytics/global-codfw.yaml -> ../_aqs2-common_/global-codfw.yaml
lrwxrwxrwx 1 root 34 May 2 17:43 editor-analytics/global-staging.yaml -> ../_aqs2-common_/global-eqiad.yaml
lrwxrwxrwx 1 root 34 May 2 17:43 editor-analytics/global-eqiad.yaml -> ../_aqs2-common_/global-eqiad.yaml
lrwxrwxrwx 1 root 34 May 2 17:43 editor-analytics/global-codfw.yaml -> ../_aqs2-common_/global-codfw.yaml
lrwxrwxrwx 1 root 34 Nov 1 2023 geo-analytics/global-staging.yaml -> ../_aqs2-common_/global-eqiad.yaml
lrwxrwxrwx 1 root 34 Nov 1 2023 geo-analytics/global-eqiad.yaml -> ../_aqs2-common_/global-eqiad.yaml
lrwxrwxrwx 1 root 34 Nov 1 2023 geo-analytics/global-codfw.yaml -> ../_aqs2-common_/global-codfw.yaml
lrwxrwxrwx 1 root 34 Nov 1 2023 media-analytics/global-staging.yaml -> ../_aqs2-common_/global-codfw.yaml
lrwxrwxrwx 1 root 34 Nov 1 2023 media-analytics/global-eqiad.yaml -> ../_aqs2-common_/global-eqiad.yaml
lrwxrwxrwx 1 root 34 Nov 1 2023 media-analytics/global-codfw.yaml -> ../_aqs2-common_/global-codfw.yaml
lrwxrwxrwx 1 root 34 Nov 1 2023 page-analytics/global-staging.yaml -> ../_aqs2-common_/global-codfw.yaml
lrwxrwxrwx 1 root 34 Nov 1 2023 page-analytics/global-eqiad.yaml -> ../_aqs2-common_/global-eqiad.yaml
lrwxrwxrwx 1 root 34 Nov 1 2023 page-analytics/global-codfw.yaml -> ../_aqs2-common_/global-codfw.yamlWe noticed when modifying editor-analytics to use both Cassandra and Druid that this mechanism would no longer be the most suitable, so we should migrate to Calico Network Policies.
In addition the druid-public extry in the external-services contains all of the brokers, but it doesn't contain the LVS service IP, which is what clients currently use.
We may want to consider adding this IP address to the network policy.
btullis@puppetmaster1001:~$ host 10.2.2.38 38.2.2.10.in-addr.arpa domain name pointer druid-public-broker.svc.eqiad.wmnet.
The druid analytics cluster doesn't use LVS, so it doesn't need modifying.