The Analytics VLAN's ACLs are currently missing:
- Whitelist for the new webproxy IP (install1002.w.o)
- IPv6 filtering
- Removal of dangling IPs
The Analytics VLAN's ACLs are currently missing:
Adding install1002's IP to the whitelist should be:
edit set firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.86/32
In the same term, I can see:
First two don't have a PTR, the last one is dataset1001.wikimedia.org. So overall we might do:
edit delete firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.160/32 delete firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.90/32 set firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.86/32 show | compare commit comment "Added new webproxy IP to analytics-publicIP-v4 and cleaned up stale configs"
Stale things found while reviewing:
Most urgent fixes:
delete firewall family inet filter analytics-in4 term aqs from destination-address 10.64.0.123/32 delete firewall family inet filter analytics-in4 term aqs from destination-address 10.64.32.175/32 delete firewall family inet filter analytics-in4 term aqs from destination-address 10.64.48.117/32
Add new AQS IPs, not sure how to add comments like /* aqs100{4,5,6}-a */ though:
/* aqs100{7,8,9} */ set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.0.199/32 set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.16.14/32 set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.48.119/32 /* aqs100{7,8,9}-a */ set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.0.213/32 set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.16.74/32 set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.48.122/32 /* aqs100{7,8,9}-b */ set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.0.237/32 set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.16.78/32 set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.48.123/32
Add the new Archiva IP and remove the old one:
set firewall family inet filter analytics-in4 term archiva from destination-address 208.80.154.73/32 delete firewall family inet filter analytics-in4 term archiva from destination-address 208.80.154.154/32
Add kafka2003 IP:
set firewall family inet filter analytics-in4 term kafka from destination-address 10.192.32.150/32
+1 to all of these. But, seeing as there has been an IPv6 with the ACLs for a while, maybe we should ask Ops about the use of continuing to support this VLAN. Not sure!
For now though, ja +1 :)
Old/New elastic search IP from Discovery: https://etherpad.wikimedia.org/p/analytics-acls
Completed the AQS work due to T157533 (under Brandon's supervision). I am going to keep working on this task during the next days to fix the remaining items.
Caveat: the Kafka Analytics cluster is still in the Analytics VLAN, so extra care must be taken.
Other batches:
Fix logstash IPs:
set firewall family inet filter analytics-in4 term logstash from destination-address 10.64.0.122 set firewall family inet filter analytics-in4 term logstash from destination-address 10.64.48.113 delete firewall family inet filter analytics-in4 term logstash from destination-address 10.64.32.136 delete firewall family inet filter analytics-in4 term logstash from destination-address 10.64.32.138
Add Webproxy and remove old IPs not used anymore (afaics):
delete firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.160/32 delete firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.90/32 set firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.86/32
Fixed logstash IPs, added install1002 (208.80.154.86/32) but not removed the other ones (for the moment).
Remove elastic1001 -> 1016:
delete firewall family inet filter analytics-in4 term es from destination-address 10.64.0.108/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.0.109/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.0.110/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.0.111/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.48.118/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.48.119/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.32.139/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.32.140/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.32.141/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.32.142/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.32.143/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.32.144/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.48.10/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.48.11/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.48.12/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.48.13/32
Add elastic1032 -> 1047:
set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.233/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.234/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.235/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.236/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.45/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.46/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.47/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.48/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.32.108/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.32.109/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.32.110/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.32.111/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.85/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.86/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.70/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.71/32
New Elastic servers being provisioned elastic1048->1052 (https://gerrit.wikimedia.org/r/#/c/336872/2/templates/wmnet):
set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.238/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.111/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.112/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.32.21/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.32.22/32
Add elastic2025 -> 2036:
set firewall family inet filter analytics-in4 term es from destination-address 10.192.0.77/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.0.78/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.0.79/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.16.191/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.16.192/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.16.193/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.32.156/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.32.157/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.32.158/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.48.73/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.48.74/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.48.75/32
Add Elastic SSL port:
set firewall family inet filter analytics-in4 term es from destination-port 9243
Fixed elastic IPs (not added annotations to analytics-in4).
Next ones:
term udplog { from { destination-address { 233.58.59.1/32; } protocol udp; destination-port 8420; } then accept; }
elukey@stat1004:~$ cat << EOF | egrep -o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" | while read ip; do dig -x $ip +short; done term analytics-publicIP-v4 { from { destination-address { 208.80.154.160/32; 208.80.154.90/32; 208.80.154.11/32; 208.80.154.86/32; } } then accept; } EOF dataset1001.wikimedia.org. install1002.wikimedia.org.
elukey@stat1004:~$ cat << EOF | egrep -o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" | while read ip; do dig -x $ip +short; done term ssh { > from { > destination-address { > 208.80.154.15/32; > 208.80.154.73/32; > 10.64.32.135/32; > 208.80.154.80/32; > 10.64.48.18/32; > 208.80.154.81/32; > } > protocol tcp; > destination-port ssh; > } > then accept; > } > EOF sodium.wikimedia.org. meitnerium.wikimedia.org. hassium.eqiad.wmnet. aluminium.wikimedia.org. dbstore1002.eqiad.wmnet. cobalt.wikimedia.org.
elukey@stat1004:~$ cat << EOF | egrep -o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" | while read ip; do dig -x $ip +short; done term rsync-http-https { > from { > destination-address { > 208.80.154.15/32; > 208.80.154.73/32; > 10.64.32.135/32; > 208.80.154.80/32; > 10.64.48.18/32; > 208.80.154.81/32; > 10.64.32.167/32; > 10.64.0.21/32; > } > protocol tcp; > destination-port [ 873 80 443 ]; > } > then accept; > } > EOF sodium.wikimedia.org. meitnerium.wikimedia.org. hassium.eqiad.wmnet. aluminium.wikimedia.org. dbstore1002.eqiad.wmnet. cobalt.wikimedia.org. eventlog1001.eqiad.wmnet. fluorine.eqiad.wmnet.
elukey@stat1004:~$ cat << EOF | egrep -o "10\.[0-9]+\.[0-9]+\.[0-9]+" | while read ip; do dig -x $ip +short; done term prelabsdb-mysql { > from { > destination-address { > 10.64.32.23/32; > 10.64.32.24/32; > 10.64.32.27/32; > } > protocol tcp; > destination-port [ 3306 3307 3308 ]; > } > then accept; > } > EOF kubernetes1003.eqiad.wmnet. db1057.eqiad.wmnet.
These ones should be the last ones to fix the IPv4 rules. @Ottomata, can we review them together?
term udplog {
+ 1
Remove IPs the term analytics-publicIP-v4:
+1
Review the IPs in term ssh
Don't know anything about this, but also not sure why we have special ssh acl rules.
Review the term rsync-http-https
I think we need some of these. I'm pretty sure some logs are rsynced from eventlog1001 and maybe also from fluorine to stat1002/3.
Review the term prelabsdb-mysql:
+1 as far as I know. Maybe Madhu or Yuvi know about kubernetes1003.eqiad.wmnet ?
Proposed fixes:
delete firewall family inet filter analytics-in4 term udplog delete firewall family inet filter analytics-in4 term prelabsdb-mysql delete firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.160/32 delete firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.90/32
After these the rest should be fine.
The above fixes have been all committed to cr1/cr2.
After running tcpdump ip6 on a couple of hosts I realized that the puppet agent contacts puppetmaster1001 via IPv6. I added a special term called puppet to analytics-in4 to whitelist it. The puppet agent still uses IPv6 with this change, probably it will work better after the analytics-in6 filter will be in place.
The next step is to design and add the analytics-in6 filter to cr1/cr2 eqiad, but I would wait for kafka1012-1022 to be decommissioned before that. Those host (the analytics kafka cluster) are in the analytics VLAN and any disruption of service for IPv6 traffic (for example due to the wrong rule pushed) might affect other important systems outside the VLAN. In T167992 we are creating the new kafka-jumbo cluster (not in the analytics vlan) and we'll slowly migrate all the analytics kafka clients to it (still no task created yet).
Since this task has been open for a long time, I'll open a new one when we'll be ready to create the analytics-in6 filter.