The Analytics VLAN's ACLs are currently missing:
- Whitelist for the new webproxy IP (install1002.w.o)
- IPv6 filtering
- Removal of dangling IPs
Description
Description
Related Objects
Related Objects
- Mentioned In
- T163002: Spread eqiad analytics Kafka nodes to multiple racks ans rows
T157943: Analytics cluster should connect to elasticsearch over SSL
T157806: Review the Analytics Firewall rules on cr1/cr2
T157533: Cassandra loading job are causing Pageview stale data - Mentioned Here
- T167992: rack/setup/install new kafka nodes kafka-jumbo100[1-6]
T157533: Cassandra loading job are causing Pageview stale data
T120281: Make elasticsearch cluster accessible from analytics hadoop workers
Event Timeline
Comment Actions
Adding install1002's IP to the whitelist should be:
edit set firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.86/32
In the same term, I can see:
- 208.80.154.160/32;
- 208.80.154.90/32;
- 208.80.154.11/32;
First two don't have a PTR, the last one is dataset1001.wikimedia.org. So overall we might do:
edit delete firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.160/32 delete firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.90/32 set firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.86/32 show | compare commit comment "Added new webproxy IP to analytics-publicIP-v4 and cleaned up stale configs"
Comment Actions
Stale things found while reviewing:
- term udplog is probably not worth to keep
- term kafka is missing kafka2003's IP
- term archiva should contain meitnerium's IP, 208.80.154.73
- term logstash contains stale IPs (10.64.32.136 is db1078.eqiad.wmnet, 10.64.32.138 is aqs1005.eqiad.wmnet). We'd need to add 10.64.0.122 and 10.64.48.113 (logstash100[13])
- term aqs contains stale IPs, we'd need to remove aqs100[123] and aqs100[123]-[a,b] and add the equivalents for aqs100[789]
- term es (T120281), if still needed might need some review.
Comment Actions
Most urgent fixes:
- Remove old AQS IPs
delete firewall family inet filter analytics-in4 term aqs from destination-address 10.64.0.123/32 delete firewall family inet filter analytics-in4 term aqs from destination-address 10.64.32.175/32 delete firewall family inet filter analytics-in4 term aqs from destination-address 10.64.48.117/32
Add new AQS IPs, not sure how to add comments like /* aqs100{4,5,6}-a */ though:
/* aqs100{7,8,9} */
set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.0.199/32
set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.16.14/32
set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.48.119/32
/* aqs100{7,8,9}-a */
set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.0.213/32
set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.16.74/32
set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.48.122/32
/* aqs100{7,8,9}-b */
set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.0.237/32
set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.16.78/32
set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.48.123/32Add the new Archiva IP and remove the old one:
set firewall family inet filter analytics-in4 term archiva from destination-address 208.80.154.73/32 delete firewall family inet filter analytics-in4 term archiva from destination-address 208.80.154.154/32
Add kafka2003 IP:
set firewall family inet filter analytics-in4 term kafka from destination-address 10.192.32.150/32
Comment Actions
+1 to all of these. But, seeing as there has been an IPv6 with the ACLs for a while, maybe we should ask Ops about the use of continuing to support this VLAN. Not sure!
For now though, ja +1 :)
Comment Actions
Old/New elastic search IP from Discovery: https://etherpad.wikimedia.org/p/analytics-acls
Comment Actions
Completed the AQS work due to T157533 (under Brandon's supervision). I am going to keep working on this task during the next days to fix the remaining items.
Caveat: the Kafka Analytics cluster is still in the Analytics VLAN, so extra care must be taken.
Comment Actions
Other batches:
Fix logstash IPs:
set firewall family inet filter analytics-in4 term logstash from destination-address 10.64.0.122 set firewall family inet filter analytics-in4 term logstash from destination-address 10.64.48.113 delete firewall family inet filter analytics-in4 term logstash from destination-address 10.64.32.136 delete firewall family inet filter analytics-in4 term logstash from destination-address 10.64.32.138
Add Webproxy and remove old IPs not used anymore (afaics):
delete firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.160/32 delete firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.90/32 set firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.86/32
Comment Actions
Fixed logstash IPs, added install1002 (208.80.154.86/32) but not removed the other ones (for the moment).
Comment Actions
Remove elastic1001 -> 1016:
delete firewall family inet filter analytics-in4 term es from destination-address 10.64.0.108/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.0.109/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.0.110/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.0.111/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.48.118/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.48.119/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.32.139/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.32.140/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.32.141/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.32.142/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.32.143/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.32.144/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.48.10/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.48.11/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.48.12/32 delete firewall family inet filter analytics-in4 term es from destination-address 10.64.48.13/32
Add elastic1032 -> 1047:
set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.233/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.234/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.235/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.236/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.45/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.46/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.47/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.48/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.32.108/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.32.109/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.32.110/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.32.111/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.85/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.86/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.70/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.71/32
New Elastic servers being provisioned elastic1048->1052 (https://gerrit.wikimedia.org/r/#/c/336872/2/templates/wmnet):
set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.238/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.111/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.112/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.32.21/32 set firewall family inet filter analytics-in4 term es from destination-address 10.64.32.22/32
Add elastic2025 -> 2036:
set firewall family inet filter analytics-in4 term es from destination-address 10.192.0.77/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.0.78/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.0.79/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.16.191/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.16.192/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.16.193/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.32.156/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.32.157/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.32.158/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.48.73/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.48.74/32 set firewall family inet filter analytics-in4 term es from destination-address 10.192.48.75/32
Add Elastic SSL port:
set firewall family inet filter analytics-in4 term es from destination-port 9243
Comment Actions
Fixed elastic IPs (not added annotations to analytics-in4).
Next ones:
- Remove udplog ?
term udplog {
from {
destination-address {
233.58.59.1/32;
}
protocol udp;
destination-port 8420;
}
then accept;
}- Remove IPs the term analytics-publicIP-v4:
elukey@stat1004:~$ cat << EOF | egrep -o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" | while read ip; do dig -x $ip +short; done
term analytics-publicIP-v4 {
from {
destination-address {
208.80.154.160/32;
208.80.154.90/32;
208.80.154.11/32;
208.80.154.86/32;
}
}
then accept;
}
EOF
dataset1001.wikimedia.org.
install1002.wikimedia.org.- Review the IPs in term ssh
elukey@stat1004:~$ cat << EOF | egrep -o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" | while read ip; do dig -x $ip +short; done
term ssh {
> from {
> destination-address {
> 208.80.154.15/32;
> 208.80.154.73/32;
> 10.64.32.135/32;
> 208.80.154.80/32;
> 10.64.48.18/32;
> 208.80.154.81/32;
> }
> protocol tcp;
> destination-port ssh;
> }
> then accept;
> }
> EOF
sodium.wikimedia.org.
meitnerium.wikimedia.org.
hassium.eqiad.wmnet.
aluminium.wikimedia.org.
dbstore1002.eqiad.wmnet.
cobalt.wikimedia.org.- Review the term rsync-http-https
elukey@stat1004:~$ cat << EOF | egrep -o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" | while read ip; do dig -x $ip +short; done
term rsync-http-https {
> from {
> destination-address {
> 208.80.154.15/32;
> 208.80.154.73/32;
> 10.64.32.135/32;
> 208.80.154.80/32;
> 10.64.48.18/32;
> 208.80.154.81/32;
> 10.64.32.167/32;
> 10.64.0.21/32;
> }
> protocol tcp;
> destination-port [ 873 80 443 ];
> }
> then accept;
> }
> EOF
sodium.wikimedia.org.
meitnerium.wikimedia.org.
hassium.eqiad.wmnet.
aluminium.wikimedia.org.
dbstore1002.eqiad.wmnet.
cobalt.wikimedia.org.
eventlog1001.eqiad.wmnet.
fluorine.eqiad.wmnet.- Review the term prelabsdb-mysql:
elukey@stat1004:~$ cat << EOF | egrep -o "10\.[0-9]+\.[0-9]+\.[0-9]+" | while read ip; do dig -x $ip +short; done
term prelabsdb-mysql {
> from {
> destination-address {
> 10.64.32.23/32;
> 10.64.32.24/32;
> 10.64.32.27/32;
> }
> protocol tcp;
> destination-port [ 3306 3307 3308 ];
> }
> then accept;
> }
> EOF
kubernetes1003.eqiad.wmnet.
db1057.eqiad.wmnet.These ones should be the last ones to fix the IPv4 rules. @Ottomata, can we review them together?
Comment Actions
term udplog {
+ 1
Remove IPs the term analytics-publicIP-v4:
+1
Review the IPs in term ssh
Don't know anything about this, but also not sure why we have special ssh acl rules.
Review the term rsync-http-https
I think we need some of these. I'm pretty sure some logs are rsynced from eventlog1001 and maybe also from fluorine to stat1002/3.
Review the term prelabsdb-mysql:
+1 as far as I know. Maybe Madhu or Yuvi know about kubernetes1003.eqiad.wmnet ?
Comment Actions
Proposed fixes:
delete firewall family inet filter analytics-in4 term udplog delete firewall family inet filter analytics-in4 term prelabsdb-mysql delete firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.160/32 delete firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.90/32
After these the rest should be fine.
Comment Actions
The above fixes have been all committed to cr1/cr2.
After running tcpdump ip6 on a couple of hosts I realized that the puppet agent contacts puppetmaster1001 via IPv6. I added a special term called puppet to analytics-in4 to whitelist it. The puppet agent still uses IPv6 with this change, probably it will work better after the analytics-in6 filter will be in place.
Comment Actions
The next step is to design and add the analytics-in6 filter to cr1/cr2 eqiad, but I would wait for kafka1012-1022 to be decommissioned before that. Those host (the analytics kafka cluster) are in the analytics VLAN and any disruption of service for IPv6 traffic (for example due to the wrong rule pushed) might affect other important systems outside the VLAN. In T167992 we are creating the new kafka-jumbo cluster (not in the analytics vlan) and we'll slowly migrate all the analytics kafka clients to it (still no task created yet).
Comment Actions
Since this task has been open for a long time, I'll open a new one when we'll be ready to create the analytics-in6 filter.