Page MenuHomePhabricator

Improve the user experience adding new nodes to puppet
Open, LowPublic

Description

In puppet, we declare every node in manifests/site.pp; give our style guide, this generally means we have something like node <node selector> { role(<puppet role> }, as puppet looks up which classes to apply to a node by node name.

These selectors, while simple in the case of services in small clusters (like, say, the VMs serving lists.wikimedia.org or phabricator) , can become regexes from hell for larger clusters. As a couple examples:

node /^db2(145|146|153|170|174|176|188|203|212|216)\.codfw\./ {
    role(mariadb::core)
}

node /^druid10(0[7-9]|1[0-1])\.eqiad\./ {
    role(druid::public::worker)
}

node /^elastic(205[5-9]|20[6-9][0-9])\.codfw\./ {
    role(elasticsearch::cirrus)
}

Editing a regexp is truly error prone, and it's not particularly handy. The consequences of a mistake here can be truly dire, even if we can always check what happens using the puppet compiler, there's no reason to do that.

Given the 1:1 correspondence, my proposal is to invert the relationships and declare nodes in such large clusters in a node/data.yaml file, like follows:

mariadb::core:
 - db2145.codfw.wmnet
 - db2146.codfw.wmnet
...

and import the definitions in site.pp either via an external node classifier or by running some simple puppet function for the node default declaration to look up in that table.

Allowing this would make it much easier to manage systems that are in large clusters, while we would retain the current system for smaller clusters where it's not convenient.

As a strawman, based on the structure above, would look like:

node default {
    $fqdn = $facts['networking']['fqdn']
    $node_roles = loadyaml('node.yaml').map | String $role, Array $nodes | {
        $nodes.map | $node | { [$node, $role] }
    }.reduce({}) | $memo, $value | {
        # check for collisions
        $value.each | $v | {
            if ($v[0] in $memo) {
                fail("Node ${v[0]} is declared twice, for role ${v[1]} and role ${memo[$v[0]]}")
            }
        }
        $memo + Hash($value)
    }

    if $fqdn in $node_roles {
        role($node_roles[$fqdn])
    }
}

We can also add CI checks around all this, of course, to make it even safer. We would ofc also need to modify the puppet compiler to make use of such a file in its heuristics, which should work much better as we don't need to "parse" puppet code in python.

Event Timeline

Joe updated the task description. (Show Details)
Joe updated the task description. (Show Details)

Related to https://phabricator.wikimedia.org/T388127
Avoiding the spicy regexps is really good and on top of that if we list one host per row in the yaml file it becames git-diff friendly

One lower tech option, is to use multiple simple regexes, e.g.

node /^sretest1002\.eqiad\./,
    /^sretest1004\.eqiad\./,
    /^sretest1006\.eqiad\./,
    /^sretest1001\.eqiad\./,
    /^sretest1003\.eqiad\./ {
    role(sretest)
}

Though a bit ugly, this should work with all our systems at present. One issue with using just the FQDN is that is breaks tools which rely on matching other hostnames, for instance the dcl tool reuses the site.pp defs for matching hosts of the form sretest1002.eqiad.default.svc.k8s.lan. I'm not aware of other tooling that is dependent on the regexes. On the other hand, if we moved the logic into an ENC script we could add some facility to return the correct role for dcl's hostnames as well.

One issue with using just the FQDN is that is breaks tools which rely on matching other hostnames, for instance the dcl tool reuses the site.pp defs for matching hosts of the form sretest1002.eqiad.default.svc.k8s.lan. I'm not aware of other tooling that is dependent on the regexes. On the other hand, if we moved the logic into an ENC script we could add some facility to return the correct role for dcl's hostnames as well.

Maybe I'm missing something, but one of the reasons I wanted to go with that format was also to make it easier for the puppet-compiler and dcl to parse which hosts belong to a node, without parsing regexes in site.pp. There's many ways to get a match without the TLD, starting from .startswith() in python.

Alternatively, we can ofc remove the TLD from the matching expression in the pseudocode I posted. I don't think that having long regexes is really feasible when we're talking about 20-30 hosts in a stanza.

FWIW I would suggest prioritizing readability and safety for prod configuration, and git-diff friendliness.
When converting the existing configuration for testbeds like dcl and pontoon perhaps a small script could update data.yaml replacing .wmnet with a custom TLD as required.

With my pontoon hat on: what I did is basically the same as the suggested node/data.yaml, i.e. map roles to (cloud) hostnames and pontoon-enc takes care of serving the data

With my pontoon hat on: what I did is basically the same as the suggested node/data.yaml, i.e. map roles to (cloud) hostnames and pontoon-enc takes care of serving the data

I completely forgot where I got the idea of the file format from :D We could indeed also make pontoon-enc (which I guess is a small script) used in production instead of that awful bowl of puppet I wrote above.

One issue with using just the FQDN is that is breaks tools which rely on matching other hostnames, for instance the dcl tool reuses the site.pp defs for matching hosts of the form sretest1002.eqiad.default.svc.k8s.lan. I'm not aware of other tooling that is dependent on the regexes. On the other hand, if we moved the logic into an ENC script we could add some facility to return the correct role for dcl's hostnames as well.

Maybe I'm missing something, but one of the reasons I wanted to go with that format was also to make it easier for the puppet-compiler and dcl to parse which hosts belong to a node, without parsing regexes in site.pp. There's many ways to get a match without the TLD, starting from .startswith() in python.

Sorry I didn't realize that was on of the aims. I was focused on the regex complexity piece, which I thought was the primary concern of this task. In the case of dcl, the tool stands up a puppetserver, so the parsing of site.pp is done via puppet, the same as in production. For the puppet-compiler, I wasn't aware there were issues with the current python parsing breaking?

In proposing possible solutions, I would love to understand a bit more why our site.pp uses complex regexes. From looking through the git log it appears that the main use case is moving servers in and and out of their insetup role. What are the other reasons people use complex regexes?

FWIW my (simple) use case is typically "every node that looks like e.g. ms-be* should be swift::storage. I'd like to still be able to do that (since it's safe to add new nodes to the role straight away without an intermediate insetup state)

Edit: @Volans pointed me to profile::netbox::host in the puppet repo . It sounds like we could use this instead of a Puppet fact.

@jhathaway in addition to site.pp (which everyone uses), we are also using it to add row/rack awareness to our Elastic (soon-to-be-called Cirrussearch) clusters, ref this CR .

I'm not sure if it helps anyone else, but would be nice to pull that particular information from the source of truth (Netbox) instead of config management. Would it be possible to create row/rack Puppet facts from Netbox info? That would help us get rid of gross regexes, and maybe it could be useful to other teams?

In proposing possible solutions, I would love to understand a bit more why our site.pp uses complex regexes. From looking through the git log it appears that the main use case is moving servers in and and out of their insetup role. What are the other reasons people use complex regexes?

Because sometimes a class of servers is composed of several sequential machines with different roles / functions. Databases are a perfect example; the old appservers as well, for example. Same for analytics hosts.

One issue with using just the FQDN is that is breaks tools which rely on matching other hostnames, for instance the dcl tool reuses the site.pp defs for matching hosts of the form sretest1002.eqiad.default.svc.k8s.lan. I'm not aware of other tooling that is dependent on the regexes. On the other hand, if we moved the logic into an ENC script we could add some facility to return the correct role for dcl's hostnames as well.

Maybe I'm missing something, but one of the reasons I wanted to go with that format was also to make it easier for the puppet-compiler and dcl to parse which hosts belong to a node, without parsing regexes in site.pp. There's many ways to get a match without the TLD, starting from .startswith() in python.

Sorry I didn't realize that was on of the aims. I was focused on the regex complexity piece, which I thought was the primary concern of this task. In the case of dcl, the tool stands up a puppetserver, so the parsing of site.pp is done via puppet, the same as in production. For the puppet-compiler, I wasn't aware there were issues with the current python parsing breaking?

Well the idea would be to have the server-role correspondence in a format that is both easy to read for humans, and not really as error prone as regexes, and at the same time allow machine access to the information without the need to query e.g. puppetdb for the info every time.

Also, once we've introduced an ENC, nothing prevents us from modifying it to receive information about servers from a remote API, in case it's needed or we want to (I've maintaned for quite some time that I'd like puppet roles to be tied to netbox given how much it's our actual source of truth at the moment.

jhathaway subscribed.