We 've had part of this discussion in #wikimedia-netops IRC channel. I can post a backlog (we don't have a bot yet archiving that channel) but some first (and partial) consensus seems to be to go for B. The premise is that a service meant to be public and destined to be used by end-users should not be hosted in the analytics cluster but rather the production cluster. thorium right now being an implicit exception to that "rule". The entire "rule" should be discussed more thoroughly at some point in the future (I don't think it makes sense to sidetrack the druid task for the larger discussion).

AFAICT, and based on the above, A doesn't really make that much sense, since production cluster LVS servers should not be routing to services in the analytics cluster . C could possibly be implemented for internal use by the analytics cluster but should not be used to power services crossing the production-analytics clusters boundary. I can not calculate the usefulness of it, but if there is little to be gained from it, maybe postpone it ?

FWIW, I am not using the terminology analytics VLAN but rather cluster as there are in reality 4 VLANs (one per eqiad rack row)

In T177511#3671178, @akosiaris wrote:

We 've had part of this discussion in #wikimedia-netops IRC channel. I can post a backlog (we don't have a bot yet archiving that channel) but some first (and partial) consensus seems to be to go for B. The premise is that a service meant to be public and destined to be used by end-users should not be hosted in the analytics cluster but rather the production cluster. thorium right now being an implicit exception to that "rule". The entire "rule" should be discussed more thoroughly at some point in the future (I don't think it makes sense to sidetrack the druid task for the larger discussion).

It seems indeed the best course of action, even though the caveat is that all the Hadoop nodes ferm rules will need to be updated to allow the new public cluster (outside of the analytics cluster boundaries) to communicate with them.
The second use case, namely having a load balancing solution in front of the private druid cluster, can surely wait and it is not pressing enough in my opinion to justify the effort for C.

@akosiaris what are the steps to take to move druid100[345] outside the analytics vlan(s)? From what I know we'd need to:

1. Properly remove the hosts from service (already done by Andrew, they have now role spare)
2. Assign new IPs via operations/dns to druid100[456]
3. Configure the host ports to not apply the related Analytics VLAN id
4. Reimage the hosts

Is the above right or am I missing something ?

Thanks!

In T177511#3671213, @elukey wrote:

@akosiaris what are the steps to take to move druid100[345] outside the analytics vlan(s)? From what I know we'd need to:

1. Properly remove the hosts from service (already done by Andrew, they have now role spare)
2. Assign new IPs via operations/dns to druid100[456]
3. Configure the host ports to not apply the related Analytics VLAN id
4. Reimage the hosts

Is the above right or am I missing something ?

Thanks!

5. Remove druid100[456] from any router ACL entry (Garbage collection).

I 'll do 5, the rest all LGTM

Change 383318 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Reassign IPs to druid100[456] to move them out of the Analytics VLAN

https://gerrit.wikimedia.org/r/383318

5. Remove druid100[456] from any router ACL entry (Garbage collection).

I 'll do 5, the rest all LGTM

Done (I had nothing to do actually, there are no ACL entries for druid hosts)

In T177511#3671257, @akosiaris wrote:

5. Remove druid100[456] from any router ACL entry (Garbage collection).

I 'll do 5, the rest all LGTM

Done (I had nothing to do actually, there are no ACL entries for druid hosts)

I am going to take the 5. spot to add the new druid100[456] IPs to the analytics-in4 filter :)

Mentioned in SAL (#wikimedia-operations) [2017-10-10T10:33:02Z] <akosiaris> T177511 switch druid100[456] to private1-x-eqiad VLANs

Change 383318 merged by Elukey:
[operations/dns@master] Reassign IPs to druid100[456] to move them out of the Analytics VLAN

https://gerrit.wikimedia.org/r/383318

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['druid1004.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201710101118_elukey_19905.log.

Completed auto-reimage of hosts:

['druid1004.eqiad.wmnet']

Of which those FAILED:

['druid1004.eqiad.wmnet']

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['druid1004.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201710101159_elukey_14560.log.

Completed auto-reimage of hosts:

['druid1004.eqiad.wmnet']

Of which those FAILED:

['druid1004.eqiad.wmnet']

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['druid1004.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201710101244_elukey_14320.log.

Completed auto-reimage of hosts:

['druid1004.eqiad.wmnet']

Of which those FAILED:

['druid1004.eqiad.wmnet']

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['druid1004.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201710101326_elukey_31466.log.

Completed auto-reimage of hosts:

['druid1004.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['druid1005.eqiad.wmnet', 'druid1006.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201710101405_elukey_8347.log.

Updated cr1/cr2 eqiad with the following:

elukey@re0.cr2-eqiad> show system rollback compare 1 0
[edit firewall family inet filter analytics-in4]
       term default { ... }
+      term druid {
+          from {
+              destination-address {
+                  10.64.0.35/32;
+                  10.64.16.172/32;
+                  10.64.48.171/32;
+              }
+              protocol tcp;
+              port [ 8090 8082 ];
+          }
+          then accept;
+      }

Mentioned in SAL (#wikimedia-operations) [2017-10-10T14:22:26Z] <elukey> add druid public cluster's IPs to analytics-in4 on cr1/cr2 - T177511

Also added the following:

elukey@re0.cr2-eqiad# show | compare
[edit firewall family inet filter analytics-in4]
      term puppet { ... }
!      term druid { ... }
[edit firewall family inet filter analytics-in4 term druid from destination-address]
+        /* druid1004 */
         10.64.0.35/32 { ... }
[edit firewall family inet filter analytics-in4 term druid from destination-address]
+        /* druid1005 */
         10.64.16.172/32 { ... }
[edit firewall family inet filter analytics-in4 term druid from destination-address]
+        /* druid1006 */
         10.64.48.171/32 { ... }

I forgot to insert the new term before the default one that rejects everything, so the new rules were not applied. I also added annotations to the IPs.