Page MenuHomePhabricator

Replacement of network::constant's special_hosts
Closed, ResolvedPublic

Description

This is something I've been doing with the help of @akosiaris recently, based on discussions in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/498796/ and some other commits, but it essentially boils down to this:
Let's abolish any concept of a special_hosts variable for multiple different types of host, and have all uses of it get the data through their parameters -> profiles -> hiera etc. It's data that we should be able to customise but its current location in network::constants makes that difficult.

There's also the secondary thing of the special hosts macros in modules/base/templates/firewall/defs.erb which is being dealt with in some of these patches where convenient.

It was rather easy for the first few:

But as Alexandros pointed out, it gets more difficult when it comes to the remaining ones:

labs only:

  • cumin_real_masters - this one might go away naturally with T219421, though we could probably get rid of it now given the cumin_masters patch above. Theoretically I could make it unused myself, but I wouldn't be able to test it directly myself on bastion-restricted due to the login restriction there which obviously I'm not going to remove. - https://gerrit.wikimedia.org/r/510712

prod only stuff, mostly analytics stuff or things that haven't been necessary within labs yet:

Event Timeline

Krenair created this task.Apr 13 2019, 11:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 13 2019, 11:45 PM
Krenair updated the task description. (Show Details)Apr 13 2019, 11:48 PM
Krenair updated the task description. (Show Details)Apr 13 2019, 11:50 PM

Help from anyone with answering the questions around monitoring_hosts and deployment_hosts would be appreciated.

Krenair updated the task description. (Show Details)Apr 14 2019, 2:37 PM
akosiaris triaged this task as Low priority.Apr 15 2019, 4:22 PM
akosiaris updated the task description. (Show Details)

Change 505356 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] network::constants: Move puppet_frontends to using existing data and resources

https://gerrit.wikimedia.org/r/505356

Krenair updated the task description. (Show Details)Apr 20 2019, 1:22 AM

Change 505366 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] network::constants: Remove seemingly unused druid_analytics_hosts

https://gerrit.wikimedia.org/r/505366

Change 505373 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] network::constants: Move various analytics special_hosts to hiera

https://gerrit.wikimedia.org/r/505373

Krenair updated the task description. (Show Details)Apr 20 2019, 4:23 AM

Change 505407 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] network::constants: Move mysql_root_clients from special_hosts to hieradata

https://gerrit.wikimedia.org/r/505407

Krenair updated the task description. (Show Details)Apr 20 2019, 3:54 PM

Change 505366 merged by Alexandros Kosiaris:
[operations/puppet@production] network::constants: Remove seemingly unused druid_analytics_hosts

https://gerrit.wikimedia.org/r/505366

Change 505373 merged by Alexandros Kosiaris:
[operations/puppet@production] network::constants: Move various analytics special_hosts to hiera

https://gerrit.wikimedia.org/r/505373

Change 505356 merged by Alexandros Kosiaris:
[operations/puppet@production] network::constants: Move puppet_frontends to using existing data instead

https://gerrit.wikimedia.org/r/505356

Krenair updated the task description. (Show Details)Apr 22 2019, 12:57 PM

I think it is better to hardcode the constants on modules/profile/manifests/mariadb/ferm.pp (for now, not as an ideal situation) than to go on a multi-file refactoring commit without even notifying or searching input from the code maintainer (note I also inherited that code).

Please don't think I am happy with the current situation of role::mariadb as it is now but there is a reason we have not yet started the work of fully refactoring it- as it will take weeks of dedication (and we cannot stop other work by keeping it on the fly for more than a few days). There is a plan for it, I am not forgetting- I am just proposing the above as a temporary measure to unblock you. In particular, the mariadb refactoring should change one class at a time or it has the risk to be outdated by the time it can be deployed (DBAs are actually doing that, slowly, if you look at profile::mariadb.

As a separate comment, note that use of hieradata for certain things has been at times objected (not from me, I am quite agnostic about that topic)- not saying this is one of those cases, just suggesting to comment to a wider audience for large refactorings.

Krenair updated the task description. (Show Details)Fri, Apr 26, 1:21 PM
Krenair updated the task description. (Show Details)
Krenair added a subscriber: aborrero.

Change 506672 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Move monitoring_hosts out of network::constants into hieradata

https://gerrit.wikimedia.org/r/506672

Krenair updated the task description. (Show Details)Mon, Apr 29, 5:37 PM

Change 507116 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Move deployment_hosts out of network::constants

https://gerrit.wikimedia.org/r/507116

Krenair updated the task description. (Show Details)Mon, Apr 29, 8:13 PM
Krenair added a comment.EditedMon, Apr 29, 9:37 PM

I think it is better to hardcode the constants on modules/profile/manifests/mariadb/ferm.pp (for now, not as an ideal situation) than to go on a multi-file refactoring commit

Yeah, I bit off a bit more than I could chew in terms of refactoring all that in one go... I've reduced the commit to just dealing with network::constants, leaving the MYSQL_ROOT_CLIENTS define in place for now until role::mariadb::ferm/profile::mariadb::ferm can be sorted out. We've done that with several other cases in this ticket, it isn't critical to getting this closed - in some other cases it was convenient enough to take care of it in the same commit.

Krenair updated the task description. (Show Details)Mon, Apr 29, 9:40 PM

Change 505407 merged by Jcrespo:
[operations/puppet@production] network::constants: Move mysql_root_clients from special_hosts

https://gerrit.wikimedia.org/r/505407

Krenair updated the task description. (Show Details)Wed, May 15, 5:29 PM

Change 506672 merged by Alexandros Kosiaris:
[operations/puppet@production] Move monitoring_hosts out of network::constants into hieradata

https://gerrit.wikimedia.org/r/506672

akosiaris updated the task description. (Show Details)Thu, May 16, 9:05 AM

Change 507116 merged by Alexandros Kosiaris:
[operations/puppet@production] Move deployment_hosts out of network::constants

https://gerrit.wikimedia.org/r/507116

akosiaris updated the task description. (Show Details)Thu, May 16, 11:37 AM
jcrespo removed a subscriber: jcrespo.Thu, May 16, 11:49 AM

Change 510712 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] profile::openstack::eqiad1::cumin::target: kill auth_group

https://gerrit.wikimedia.org/r/510712

Change 510713 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] network::constants: kill special_hosts

https://gerrit.wikimedia.org/r/510713

Krenair updated the task description. (Show Details)Thu, May 16, 12:07 PM

Change 510712 merged by Andrew Bogott:
[operations/puppet@production] profile::openstack::eqiad1::cumin::target: kill auth_group

https://gerrit.wikimedia.org/r/510712

Krenair updated the task description. (Show Details)Thu, May 16, 3:02 PM

Change 510713 merged by Alexandros Kosiaris:
[operations/puppet@production] network::constants: kill special_hosts

https://gerrit.wikimedia.org/r/510713

Krenair closed this task as Resolved.Fri, May 17, 2:22 PM