Page MenuHomePhabricator

Horizon dashboard for managing instance puppet config
Closed, ResolvedPublic

Description

In Horizon, in the instance details tab, I want a tab called 'Puppet Roles.' That will display a structured gui:

[] First role
    param1:  [ text field ]
    param2:  [ text field ]
    param3:  [ text field ]
    param4:  [ text field ]
[] Second role
    param1:  [ text field ]
    param2:  [ text field ]
...

> Advanced roles (puppet experts only)

===========

-- THESE ROLLS ARE SET BUT NO LONGER EXIST IN PUPPET --
[x] deprecated role 1
   param1: label
...

===========

> Free-form hiera (for puppet experts only)

The set of available roles will come from a direct query on the puppetmaster. A select set of designate roles will appear expanded, the rest of the roles will be under the 'expert' disclosure. A free-form hiera .yaml field will also be available, also undisclosed by default.

Getting/setting node-specific roles will happen via a to-be-written REST API. See https://etherpad.wikimedia.org/p/puppet-enc-labs for details on the API.

That API will be implemented via a custom ENC and some hiera magic.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+4 -9
operations/puppetproduction+50 -12
operations/puppetproduction+33 -12
operations/puppetproduction+1 -0
operations/puppetproduction+4 -0
operations/puppetproduction+2 -1
operations/puppetproduction+2 -6
operations/puppetproduction+0 -14
operations/puppetproduction+49 -15
operations/puppetproduction+23 -40
operations/puppetproduction+3 -14
operations/puppetproduction+135 -9
operations/puppetproduction+1 -1
operations/puppetproduction+4 -0
operations/puppetproduction+73 -43
operations/puppetproduction+243 -4
operations/puppetproduction+114 -51
operations/puppetproduction+13 -5
operations/puppetproduction+937 -0
operations/puppetproduction+11 -0
operations/puppetproduction+3 -0
operations/puppetproduction+1 -1
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
yuvipanda set Security to None.

I think generally speaking there are two related but different use cases for Puppet management of Labs instances:

  1. "I want to make a Labs instance that does well defined task X." (e.g. runs MediaWiki-Vagrant)
  2. "I'm a smart sysadmin who understands Puppet and Heira and I want to manage a complex configuration for my project."

What I don't know is how many different "well defined task X" sub cases we have other than the MediaWiki-Vagrant one. The current default list of classes in wikitech is:

  • role::simplelamp
  • role::aptly
  • role::labs::lvm::srv
  • role::labs::mediawiki_vagrant
  • role::puppet::self

Said magic is:

For ENC

Write a simple shell script / python script that hits this API and returns the list of classes *only* for the given node

For Hiera

This will replace the mwyaml hiera provider, calling to this backend instead. The hiera backend will just make a http request for given node name to this interface, and give back to puppet whatever parameters get returned.

Added T101447 as blocking task since this would eliminate the ability to set ldap global variables (or unset / modify them)

With more discussion between me, @chasemp and @bd808, I think https://etherpad.wikimedia.org/p/puppet-enc-labs is at a fairly steady solid state.

Change 291969 had a related patch set uploaded (by Andrew Bogott):
Allow horizon hosts to contact the labs puppetmaster.

https://gerrit.wikimedia.org/r/291969

Change 291969 merged by Andrew Bogott:
Allow horizon hosts to contact the labs puppetmaster.

https://gerrit.wikimedia.org/r/291969

First pass of a (still somewhat ugly) 'apply role' dialog.

rolemodal.png (1×2 px, 605 KB)

Questions:

  • How to distinguish between 'unset' and '' for params?
  • How to make the in-line comment formatting look less stupid?

Change 294342 had a related patch set uploaded (by Andrew Bogott):
WIP: Horizon tab for modifying instance puppet config

https://gerrit.wikimedia.org/r/294342

Krenair requests a (?) icon for mouseover docs in the overview screen. I'd like that too, but so far it seems to be hard without rewriting the base class.

Cool!

With respect to the overview screen: I'm inclined to think of roles like 'tags' rather than a list of true/false values, and the following feels more natural to me:

pasted_file (166×614 px, 13 KB)

I'm not sure how easy that is to implement, though. The Django admin panel can do this relatively easily.

As for the edit panel, I think the current option is already pretty nice. You could consider just exposing a YAML edit field, as that makes it easier to copy-paste configuration between hosts, and also immediately clarifies 'not set' and 'set to be empty'.

Change 294716 had a related patch set uploaded (by Andrew Bogott):
Add the instance tld (e.g. 'wmflabs') to designate and horizon config.

https://gerrit.wikimedia.org/r/294716

Change 294716 merged by Andrew Bogott:
Add the instance tld (e.g. 'wmflabs') to designate and horizon config.

https://gerrit.wikimedia.org/r/294716

Change 306981 had a related patch set uploaded (by Andrew Bogott):
Added filtertags to labs role descriptions.

https://gerrit.wikimedia.org/r/306981

Change 306981 merged by Andrew Bogott:
Added filtertags to labs role descriptions.

https://gerrit.wikimedia.org/r/306981

Change 294342 merged by Andrew Bogott:
Horizon tab for modifying instance puppet config

https://gerrit.wikimedia.org/r/294342

Change 308188 had a related patch set uploaded (by Alex Monk):
openstack horizon puppettab: Put docs on a question mark next to the role name

https://gerrit.wikimedia.org/r/308188

Change 308188 merged by Andrew Bogott:
openstack horizon puppettab: Put docs on a question mark next to the role name

https://gerrit.wikimedia.org/r/308188

Change 310325 had a related patch set uploaded (by Andrew Bogott):
puppetpanel: Add a 'Project Puppet' panel

https://gerrit.wikimedia.org/r/310325

Change 310325 merged by Andrew Bogott:
puppetpanel: Add a 'Project Puppet' panel

https://gerrit.wikimedia.org/r/310325

Change 310683 had a related patch set uploaded (by Andrew Bogott):
Puppet panel: Add per-prefix config tabs

https://gerrit.wikimedia.org/r/310683

Change 310683 merged by Andrew Bogott:
Puppet panel: Add per-prefix config tabs

https://gerrit.wikimedia.org/r/310683

Change 310718 had a related patch set uploaded (by Andrew Bogott):
Puppet Panel: Actually populate the prefix panel with prefixes.

https://gerrit.wikimedia.org/r/310718

Change 310718 merged by Andrew Bogott:
Puppet Panel: Actually populate the prefix panel with prefixes.

https://gerrit.wikimedia.org/r/310718

This interface has turned into a bit of a monster... it now has has:

  • A panel with one tab for project-wide settings
  • A panel with N tabs for prefix-based settings
  • Tabs in the instance detail group for instance-based settings

I don't love it, but it lets us do what we need. Remaining tasks:

  • Fix the dumb question-mark icon that doesn't render properly
  • Add a way to remove prefixes
  • Figure out what takes so damn long to render a tab
  • (probably) aesthetic improvements

When you create a new prefix, you don't get taken to the prefix' tab but to the first tab. Same happens after any actions - if you edit hiera, it resets you to the first tab.

Change 310938 had a related patch set uploaded (by Yuvipanda):
labs: Add httpyaml hiera backend to labs

https://gerrit.wikimedia.org/r/310938

Change 310938 merged by Yuvipanda:
labs: Add httpyaml hiera backend to labs

https://gerrit.wikimedia.org/r/310938

Change 310945 had a related patch set uploaded (by Yuvipanda):
labs: Use puppetmaster service URL for hiera backend

https://gerrit.wikimedia.org/r/310945

Change 310945 merged by Yuvipanda:
labs: Use puppetmaster service URL for hiera backend

https://gerrit.wikimedia.org/r/310945

Change 310952 had a related patch set uploaded (by Yuvipanda):
puppet: Add option to use newer ENC

https://gerrit.wikimedia.org/r/310952

Change 310962 had a related patch set uploaded (by Andrew Bogott):
puppet panel: Add a button to remove prefixes.

https://gerrit.wikimedia.org/r/310962

Change 310962 merged by Andrew Bogott:
puppet panel: Add a button to remove prefixes.

https://gerrit.wikimedia.org/r/310962

Performance of the puppet pannels is still pretty bad - takes about 20-30s to load on my laptop (firefox). We shouldn't enable it for labs like this, need to bring it down at least an order of magnitude.

Change 311041 had a related patch set uploaded (by Andrew Bogott):
Puppet Panel: Speed up loading of role documentation

https://gerrit.wikimedia.org/r/311041

Change 311057 had a related patch set uploaded (by Andrew Bogott):
Puppet Panel: Speed up role formatting

https://gerrit.wikimedia.org/r/311057

Change 311041 abandoned by Andrew Bogott:
Puppet Panel: Speed up loading of role documentation

https://gerrit.wikimedia.org/r/311041

Change 311057 merged by Andrew Bogott:
Puppet Panel: Speed up role formatting

https://gerrit.wikimedia.org/r/311057

Change 311160 had a related patch set uploaded (by Andrew Bogott):
Puppet Panel: Add captions to instance tabs

https://gerrit.wikimedia.org/r/311160

Change 311160 merged by Andrew Bogott:
Puppet Panel: Add captions to instance tabs

https://gerrit.wikimedia.org/r/311160

Change 311164 had a related patch set uploaded (by Andrew Bogott):
Puppet Panel: Remove use of breadcrumb_nav

https://gerrit.wikimedia.org/r/311164

Change 311164 merged by Andrew Bogott:
Puppet Panel: Remove use of breadcrumb_nav

https://gerrit.wikimedia.org/r/311164

Change 311449 had a related patch set uploaded (by Andrew Bogott):
Puppet Horizon Panel: Deploy on Labs

https://gerrit.wikimedia.org/r/311449

Change 311449 merged by Andrew Bogott:
Puppet Horizon Panel: Deploy on Labs

https://gerrit.wikimedia.org/r/311449

Change 311486 had a related patch set uploaded (by Andrew Bogott):
Pupept panel: Use "{}" to represent an empty hiera config

https://gerrit.wikimedia.org/r/311486

Change 311488 had a related patch set uploaded (by Andrew Bogott):
Puppet Panel: Don't choke on a (user-provided) "" hiera string.

https://gerrit.wikimedia.org/r/311488

Change 311486 merged by Andrew Bogott:
Puppet panel: Use "{}" to represent an empty hiera config

https://gerrit.wikimedia.org/r/311486

Change 311488 merged by Andrew Bogott:
Puppet Panel: Don't choke on a (user-provided) "" hiera string.

https://gerrit.wikimedia.org/r/311488

Change 310952 merged by Yuvipanda:
puppet: Add option to use newer ENC

https://gerrit.wikimedia.org/r/310952

Change 311762 had a related patch set uploaded (by Andrew Bogott):
Puppet Panel: No checkboxes!

https://gerrit.wikimedia.org/r/311762

Change 311762 merged by Andrew Bogott:
Puppet Panel: No checkboxes!

https://gerrit.wikimedia.org/r/311762

Change 311767 had a related patch set uploaded (by Yuvipanda):
puppet: Use newer ENC

https://gerrit.wikimedia.org/r/311767

Change 311767 merged by Yuvipanda:
puppet: Use newer ENC

https://gerrit.wikimedia.org/r/311767

Change 312044 had a related patch set uploaded (by Yuvipanda):
puppet: Enable ENC on trusty modules too

https://gerrit.wikimedia.org/r/312044

Change 312044 merged by Yuvipanda:
puppet: Enable ENC on trusty nodes too

https://gerrit.wikimedia.org/r/312044