If I'm not mistaken, this mechanism is how all classes are applied across Cloud to all instances?

That is quite possible in the case of using Horizon! I think the issue here was more specifically the class applications in Wikitech. I might vote to make this more about deprecating the wikitech hiera because too many places to look == places things get missed.

However, I'm pretty sure there's already a task for deprecating wikitech hiera.

@chasemp it seems to be the way certain classes are applied on top of the default ones for some projects. The hiera pages I've listed are the only ones I could find in Wikitech that used the "classes" list but maybe there are others (used Google).

often described as "ugly", "awful", etc

From what point of view? This mechanism is 100% what makes it possible for project maintainers to choose Puppet modules to apply to any instance in Cloud VPS projects. The "classes" hiera setting may come from the Horizon backend storage, from Wikitech's Hiera namespace, or from files in ops/puppet.git repo which is probably the part that is ugliest bit of this.

I should probably have left those words out of the ticket description but it's what I've been told. I think it's the same arguments for wanting to drop Wikitech from being a hiera backend? I can't speak for others but having "classes" in wikitech got me by surprise as well. I think it's just another gotcha in the infrastructure, that you wouldn't normally expect from previous Puppet experience but maybe my experience is limited in this regard.

However, in light of the discussion that happened on IRC about hiera backends, and your comment about "classes" in Wikitech being the way that some projects choose to re-use the ops/puppet.git code, it seems this is a feature we can't drop.

I could only find 4 projects that are using this feature:

• tools
  • role::aptly::client (listed in Wikitech and also in hieradata/labs/tools/common.yaml)
  • role::prometheus::node_exporter (included in ::standard)
  • role::toollabs::logging::sender (can't find it included anywhere but in wikitech, candidate for profile::toolforge::base?)
  • profile::toolforge::clush::target (will be required in the new toolforge cluster by profile::toolforge::base)

• toolsbeta
  • role::aptly::client
  • profile::toolforge::clush::target (will be required in the new toolforge cluster by profile::toolforge::base)

• staging
  • apt::unattendedupgrades (already applied from profile::base::labs?)

• deployment-prep
  • apt::unattendedupgrades (already applied from profile::base::labs?)
  • role::aptly::client
  • role::prometheus::node_exporter (included in ::standard)

I'm not clear why Tools/Toolsbeta specifically are making use of this feature except for allowing external contributors with access to Wikitech to make changes to the infrastructure without having access to ops/puppet.git. Those classes could very well be included directly in puppet code, if I understood things right.

I don't know what staging is, looks like a competitor to deployment-prep that didn't move forward much? And for deployment-prep, it seems they make heavy use of storing this information in Horizon so maybe they can apply it all from there?

I understand it's a feature that has its use so, especially for external contributors, so I'm fine with closing this ticket. Maybe we can re-evaluate once we have the user personas and use cases to have the discussion about our long-term plans (including our hiera backend discussion).

In T210614#4782555, @GTirloni wrote:

I could only find 4 projects that are using this feature

I do not think that the ability to add this key to any Hiera config is well documented (or really that it should be) so the low number of usages is not surprising. I would not be surprised to learn that the existing usage you have found are largely from a point in time before the Horizon Puppet interface was available.

This exact functionality from the point of view of the Puppet master is provided by the "Other Classes" section of our Horizon Puppet dashboard. Its literally what powers the point-and-click adding of classes via Horizon as well. You can prove this to yourself by playing with the config for an instance and seeing that if you add a class into "Other Classes" that is actually listed in the giant list of possible classes and saving the change that the refreshed page will show things as though you had clicked the "Apply Class" button for the row.

Again, I think the real point of confusion is that we source Hiera configuration from at least 3 places (Horizon, Wikitech, ops/puppet) and not that we use hiera_include('classes', []) to select the classes that are applied to a given instance.

Very good point, Horizon! I did use "other classes" to test new code and it felt very useful indeed. Maybe that's the killer use case.

Thanks for the historical perspective too.