Page MenuHomePhabricator
Create Task
Maniphest T157435

Review ACLs for the Analytics VLAN
Closed, ResolvedPublic
Actions

Description

The Analytics VLAN's ACLs are currently missing:

  1. Whitelist for the new webproxy IP (install1002.w.o)
  2. IPv6 filtering
  3. Removal of dangling IPs

Related Objects

Event Timeline

elukey created this task.Feb 7 2017, 12:30 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 7 2017, 12:31 PM
elukey triaged this task as Medium priority.Feb 7 2017, 12:31 PM
elukey added a project: Analytics-Kanban.
elukey added subscribers: mark, faidon.Feb 7 2017, 12:46 PM

Adding install1002's IP to the whitelist should be:

edit
set firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.86/32

In the same term, I can see:

  • 208.80.154.160/32;
  • 208.80.154.90/32;
  • 208.80.154.11/32;

First two don't have a PTR, the last one is dataset1001.wikimedia.org. So overall we might do:

edit
delete firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.160/32
delete firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.90/32
set firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.86/32
show | compare
commit comment "Added new webproxy IP to analytics-publicIP-v4 and cleaned up stale configs"

@mark, @faidon to review :)

elukey added a comment.EditedFeb 7 2017, 1:25 PM

Stale things found while reviewing:

  • term udplog is probably not worth to keep
  • term kafka is missing kafka2003's IP
  • term archiva should contain meitnerium's IP, 208.80.154.73
  • term logstash contains stale IPs (10.64.32.136 is db1078.eqiad.wmnet, 10.64.32.138 is aqs1005.eqiad.wmnet). We'd need to add 10.64.0.122 and 10.64.48.113 (logstash100[13])
  • term aqs contains stale IPs, we'd need to remove aqs100[123] and aqs100[123]-[a,b] and add the equivalents for aqs100[789]
  • term es (T120281), if still needed might need some review.
elukey added a subscriber: Ottomata.Feb 7 2017, 1:27 PM
elukey added a comment.Feb 7 2017, 2:02 PM

Most urgent fixes:

  • Remove old AQS IPs
delete firewall family inet filter analytics-in4 term aqs from destination-address 10.64.0.123/32
delete firewall family inet filter analytics-in4 term aqs from destination-address 10.64.32.175/32
delete firewall family inet filter analytics-in4 term aqs from destination-address 10.64.48.117/32

Add new AQS IPs, not sure how to add comments like /* aqs100{4,5,6}-a */ though:

/* aqs100{7,8,9} */
set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.0.199/32
set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.16.14/32
set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.48.119/32
/* aqs100{7,8,9}-a */
set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.0.213/32
set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.16.74/32
set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.48.122/32
/* aqs100{7,8,9}-b */
set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.0.237/32
set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.16.78/32
set firewall family inet filter analytics-in4 term aqs from destination-address 10.64.48.123/32

Add the new Archiva IP and remove the old one:

set firewall family inet filter analytics-in4 term archiva from destination-address 208.80.154.73/32
delete firewall family inet filter analytics-in4 term archiva from destination-address 208.80.154.154/32

Add kafka2003 IP:

set firewall family inet filter analytics-in4 term kafka from destination-address 10.192.32.150/32
Ottomata added a comment.Feb 7 2017, 2:23 PM

+1 to all of these. But, seeing as there has been an IPv6 with the ACLs for a while, maybe we should ask Ops about the use of continuing to support this VLAN. Not sure!

For now though, ja +1 :)

elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.Feb 7 2017, 2:35 PM
elukey added a comment.Feb 7 2017, 2:41 PM

Old/New elastic search IP from Discovery: https://etherpad.wikimedia.org/p/analytics-acls

elukey added a comment.Feb 8 2017, 5:51 PM

Completed the AQS work due to T157533 (under Brandon's supervision). I am going to keep working on this task during the next days to fix the remaining items.

Caveat: the Kafka Analytics cluster is still in the Analytics VLAN, so extra care must be taken.

elukey added a comment.Feb 10 2017, 12:29 PM

Added kafka2003, fixed Archiva.

elukey added a comment.Feb 10 2017, 12:36 PM

Other batches:

Fix logstash IPs:

set firewall family inet filter analytics-in4 term logstash from destination-address 10.64.0.122
set firewall family inet filter analytics-in4 term logstash from destination-address 10.64.48.113
delete firewall family inet filter analytics-in4 term logstash from destination-address 10.64.32.136
delete firewall family inet filter analytics-in4 term logstash from destination-address 10.64.32.138

Add Webproxy and remove old IPs not used anymore (afaics):

delete firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.160/32
delete firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.90/32
set firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.86/32
elukey added a comment.Feb 10 2017, 1:21 PM

Fixed logstash IPs, added install1002 (208.80.154.86/32) but not removed the other ones (for the moment).

elukey added a comment.EditedFeb 13 2017, 9:16 AM

Remove elastic1001 -> 1016:

delete firewall family inet filter analytics-in4 term es from destination-address 10.64.0.108/32
delete firewall family inet filter analytics-in4 term es from destination-address 10.64.0.109/32
delete firewall family inet filter analytics-in4 term es from destination-address 10.64.0.110/32
delete firewall family inet filter analytics-in4 term es from destination-address 10.64.0.111/32
delete firewall family inet filter analytics-in4 term es from destination-address 10.64.48.118/32
delete firewall family inet filter analytics-in4 term es from destination-address 10.64.48.119/32
delete firewall family inet filter analytics-in4 term es from destination-address 10.64.32.139/32
delete firewall family inet filter analytics-in4 term es from destination-address 10.64.32.140/32
delete firewall family inet filter analytics-in4 term es from destination-address 10.64.32.141/32
delete firewall family inet filter analytics-in4 term es from destination-address 10.64.32.142/32
delete firewall family inet filter analytics-in4 term es from destination-address 10.64.32.143/32
delete firewall family inet filter analytics-in4 term es from destination-address 10.64.32.144/32
delete firewall family inet filter analytics-in4 term es from destination-address 10.64.48.10/32
delete firewall family inet filter analytics-in4 term es from destination-address 10.64.48.11/32
delete firewall family inet filter analytics-in4 term es from destination-address 10.64.48.12/32
delete firewall family inet filter analytics-in4 term es from destination-address 10.64.48.13/32

Add elastic1032 -> 1047:

set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.233/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.234/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.235/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.236/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.45/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.46/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.47/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.48/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.32.108/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.32.109/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.32.110/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.32.111/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.85/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.86/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.70/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.71/32

New Elastic servers being provisioned elastic1048->1052 (https://gerrit.wikimedia.org/r/#/c/336872/2/templates/wmnet):

set firewall family inet filter analytics-in4 term es from destination-address 10.64.0.238/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.111/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.16.112/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.32.21/32
set firewall family inet filter analytics-in4 term es from destination-address 10.64.32.22/32

Add elastic2025 -> 2036:

set firewall family inet filter analytics-in4 term es from destination-address 10.192.0.77/32
set firewall family inet filter analytics-in4 term es from destination-address 10.192.0.78/32
set firewall family inet filter analytics-in4 term es from destination-address 10.192.0.79/32
set firewall family inet filter analytics-in4 term es from destination-address 10.192.16.191/32
set firewall family inet filter analytics-in4 term es from destination-address 10.192.16.192/32
set firewall family inet filter analytics-in4 term es from destination-address 10.192.16.193/32
set firewall family inet filter analytics-in4 term es from destination-address 10.192.32.156/32
set firewall family inet filter analytics-in4 term es from destination-address 10.192.32.157/32
set firewall family inet filter analytics-in4 term es from destination-address 10.192.32.158/32
set firewall family inet filter analytics-in4 term es from destination-address 10.192.48.73/32
set firewall family inet filter analytics-in4 term es from destination-address 10.192.48.74/32
set firewall family inet filter analytics-in4 term es from destination-address 10.192.48.75/32

Add Elastic SSL port:

set firewall family inet filter analytics-in4 term es from destination-port 9243
elukey added a comment.Feb 13 2017, 1:13 PM

Fixed elastic IPs (not added annotations to analytics-in4).

Next ones:

  1. Remove udplog ?
term udplog {
    from {
        destination-address {
            233.58.59.1/32;
        }
        protocol udp;
        destination-port 8420;
    }
    then accept;
}
  1. Remove IPs the term analytics-publicIP-v4:
elukey@stat1004:~$ cat << EOF | egrep -o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" | while read ip; do dig -x $ip +short; done
term analytics-publicIP-v4 {
    from {
        destination-address {
            208.80.154.160/32;
            208.80.154.90/32;
            208.80.154.11/32;
            208.80.154.86/32;
        }
    }
    then accept;
}
EOF

dataset1001.wikimedia.org.
install1002.wikimedia.org.
  1. Review the IPs in term ssh
elukey@stat1004:~$ cat << EOF | egrep -o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" | while read ip; do dig -x $ip +short; done
term ssh {
>     from {
>         destination-address {
>             208.80.154.15/32;
>             208.80.154.73/32;
>             10.64.32.135/32;
>             208.80.154.80/32;
>             10.64.48.18/32;
>             208.80.154.81/32;
>         }
>         protocol tcp;
>         destination-port ssh;
>     }
>     then accept;
> }
> EOF
sodium.wikimedia.org.
meitnerium.wikimedia.org.
hassium.eqiad.wmnet.
aluminium.wikimedia.org.
dbstore1002.eqiad.wmnet.
cobalt.wikimedia.org.
  1. Review the term rsync-http-https
elukey@stat1004:~$ cat << EOF | egrep -o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" | while read ip; do dig -x $ip +short; done
term rsync-http-https {
>     from {
>         destination-address {
>             208.80.154.15/32;
>             208.80.154.73/32;
>             10.64.32.135/32;
>             208.80.154.80/32;
>             10.64.48.18/32;
>             208.80.154.81/32;
>             10.64.32.167/32;
>             10.64.0.21/32;
>         }
>         protocol tcp;
>         destination-port [ 873 80 443 ];
>     }
>     then accept;
> }
> EOF
sodium.wikimedia.org.
meitnerium.wikimedia.org.
hassium.eqiad.wmnet.
aluminium.wikimedia.org.
dbstore1002.eqiad.wmnet.
cobalt.wikimedia.org.
eventlog1001.eqiad.wmnet.
fluorine.eqiad.wmnet.
  1. Review the term prelabsdb-mysql:
elukey@stat1004:~$ cat << EOF | egrep -o "10\.[0-9]+\.[0-9]+\.[0-9]+" | while read ip; do dig -x $ip +short; done
term prelabsdb-mysql {
>     from {
>         destination-address {
>             10.64.32.23/32;
>             10.64.32.24/32;
>             10.64.32.27/32;
>         }
>         protocol tcp;
>         destination-port [ 3306 3307 3308 ];
>     }
>     then accept;
> }
> EOF
kubernetes1003.eqiad.wmnet.
db1057.eqiad.wmnet.

These ones should be the last ones to fix the IPv4 rules. @Ottomata, can we review them together?

Ottomata added a comment.Feb 14 2017, 5:07 PM

term udplog {

+ 1

Remove IPs the term analytics-publicIP-v4:

+1

Review the IPs in term ssh

Don't know anything about this, but also not sure why we have special ssh acl rules.

Review the term rsync-http-https

I think we need some of these. I'm pretty sure some logs are rsynced from eventlog1001 and maybe also from fluorine to stat1002/3.

Review the term prelabsdb-mysql:

+1 as far as I know. Maybe Madhu or Yuvi know about kubernetes1003.eqiad.wmnet ?

elukey added a comment.Feb 15 2017, 11:05 AM

Proposed fixes:

delete firewall family inet filter analytics-in4 term udplog
delete firewall family inet filter analytics-in4 term prelabsdb-mysql
delete firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.160/32
delete firewall family inet filter analytics-in4 term analytics-publicIP-v4 from destination-address 208.80.154.90/32

After these the rest should be fine.

elukey added a comment.EditedFeb 16 2017, 1:25 PM

The above fixes have been all committed to cr1/cr2.

After running tcpdump ip6 on a couple of hosts I realized that the puppet agent contacts puppetmaster1001 via IPv6. I added a special term called puppet to analytics-in4 to whitelist it. The puppet agent still uses IPv6 with this change, probably it will work better after the analytics-in6 filter will be in place.

elukey moved this task from In Progress to Paused on the Analytics-Kanban board.Mar 2 2017, 12:26 PM
elukey moved this task from Backlog to Analytics Backlog on the User-Elukey board.Mar 8 2017, 1:15 PM
elukey merged a task: Restricted Task.Apr 7 2017, 1:04 PM
ayounsi added a subscriber: ayounsi.Apr 7 2017, 2:31 PM
elukey moved this task from Analytics Backlog to Stalled on the User-Elukey board.May 2 2017, 4:35 PM
ayounsi moved this task from Backlog to Audit on the netops board.Jun 27 2017, 2:41 PM
elukey edited projects, added Analytics; removed Analytics-Kanban.Jul 28 2017, 2:50 PM
mforns moved this task from Incoming to Backlog (Later) on the Analytics board.Jul 31 2017, 3:36 PM
elukey moved this task from Stalled to Ops Backlog on the User-Elukey board.Aug 11 2017, 9:12 AM
elukey added a comment.Sep 8 2017, 7:42 AM

The next step is to design and add the analytics-in6 filter to cr1/cr2 eqiad, but I would wait for kafka1012-1022 to be decommissioned before that. Those host (the analytics kafka cluster) are in the analytics VLAN and any disruption of service for IPv6 traffic (for example due to the wrong rule pushed) might affect other important systems outside the VLAN. In T167992 we are creating the new kafka-jumbo cluster (not in the analytics vlan) and we'll slowly migrate all the analytics kafka clients to it (still no task created yet).

elukey changed the task status from Open to Stalled.Oct 6 2017, 1:21 PM
elukey moved this task from Ops Backlog to Stalled on the User-Elukey board.
fdans moved this task from Backlog (Later) to Dashiki on the Analytics board.Jan 8 2018, 4:56 PM
Milimetric moved this task from Dashiki to Incoming on the Analytics board.Apr 2 2018, 3:32 PM
Milimetric moved this task from Dashiki to Incoming on the Analytics board.
Milimetric moved this task from Dashiki to Incoming on the Analytics board.
Milimetric moved this task from Dashiki to Incoming on the Analytics board.
Milimetric moved this task from Dashiki to Incoming on the Analytics board.
Milimetric moved this task from Dashiki to Incoming on the Analytics board.
Nuria closed this task as Resolved.Apr 5 2018, 5:02 PM
elukey added a comment.Apr 5 2018, 5:05 PM

Since this task has been open for a long time, I'll open a new one when we'll be ready to create the analytics-in6 filter.

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL