Page MenuHomePhabricator
Create Task
Maniphest T91990

Horizon dashboard for managing instance puppet config
Closed, ResolvedPublic
Actions

Description

In Horizon, in the instance details tab, I want a tab called 'Puppet Roles.' That will display a structured gui:

[] First role
    param1:  [ text field ]
    param2:  [ text field ]
    param3:  [ text field ]
    param4:  [ text field ]
[] Second role
    param1:  [ text field ]
    param2:  [ text field ]
...

> Advanced roles (puppet experts only)

===========

-- THESE ROLLS ARE SET BUT NO LONGER EXIST IN PUPPET --
[x] deprecated role 1
   param1: label
...

===========

> Free-form hiera (for puppet experts only)

The set of available roles will come from a direct query on the puppetmaster. A select set of designate roles will appear expanded, the rest of the roles will be under the 'expert' disclosure. A free-form hiera .yaml field will also be available, also undisclosed by default.

Getting/setting node-specific roles will happen via a to-be-written REST API. See https://etherpad.wikimedia.org/p/puppet-enc-labs for details on the API.

That API will be implemented via a custom ENC and some hiera magic.

Details

SubjectRepoBranchLines +/-
puppet: Enable ENC on trusty nodes toooperations/puppetproduction+4 -9
puppet: Use newer ENCoperations/puppetproduction+50 -12
puppet: Add option to use newer ENCoperations/puppetproduction+33 -12
Puppet Panel: No checkboxes!operations/puppetproduction+1 -0
Puppet Panel: Don't choke on a (user-provided) "" hiera string.operations/puppetproduction+4 -0
Puppet panel: Use "{}" to represent an empty hiera configoperations/puppetproduction+2 -1
Puppet Horizon Panel: Deploy on Labsoperations/puppetproduction+2 -6
Puppet Panel: Remove use of breadcrumb_navoperations/puppetproduction+0 -14
Puppet Panel: Add captions to instance tabsoperations/puppetproduction+49 -15
Puppet Panel: Speed up role formattingoperations/puppetproduction+23 -40
Puppet Panel: Speed up loading of role documentationoperations/puppetproduction+3 -14
puppet panel: Add a button to remove prefixes.operations/puppetproduction+135 -9
labs: Use puppetmaster service URL for hiera backendoperations/puppetproduction+1 -1
labs: Add httpyaml hiera backend to labsoperations/puppetproduction+4 -0
Puppet Panel: Actually populate the prefix panel with prefixes.operations/puppetproduction+73 -43
Puppet panel: Add per-prefix config tabsoperations/puppetproduction+243 -4
puppetpanel: Add a 'Project Puppet' paneloperations/puppetproduction+114 -51
openstack horizon puppettab: Put docs on a question mark next to the role nameoperations/puppetproduction+13 -5
Horizon tab for modifying instance puppet configoperations/puppetproduction+937 -0
Added filtertags to labs role descriptions.operations/puppetproduction+11 -0
Add the instance tld (e.g. 'wmflabs') to designate and horizon config.operations/puppetproduction+3 -0
Allow horizon hosts to contact the labs puppetmaster.operations/puppetproduction+1 -1
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Andrew assigned this task to yuvipanda.Mar 9 2015, 4:54 PM
Andrew added a project: Cloud-Services.
Andrew added subscribers: Aklapper, Andrew.
yuvipanda removed yuvipanda as the assignee of this task.Jun 7 2015, 5:14 PM
yuvipanda set Security to None.
Luke081515 added a project: Horizon.Mar 3 2016, 8:32 PM
Andrew updated the task description. (Show Details)Apr 19 2016, 9:12 PM
Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptApr 19 2016, 9:12 PM
Andrew updated the task description. (Show Details)Apr 19 2016, 9:12 PM
bd808 subscribed.Apr 19 2016, 9:26 PM

I think generally speaking there are two related but different use cases for Puppet management of Labs instances:

  1. "I want to make a Labs instance that does well defined task X." (e.g. runs MediaWiki-Vagrant)
  2. "I'm a smart sysadmin who understands Puppet and Heira and I want to manage a complex configuration for my project."

What I don't know is how many different "well defined task X" sub cases we have other than the MediaWiki-Vagrant one. The current default list of classes in wikitech is:

  • role::simplelamp
  • role::aptly
  • role::labs::lvm::srv
  • role::labs::mediawiki_vagrant
  • role::puppet::self
Andrew updated the task description. (Show Details)Apr 19 2016, 10:09 PM
Andrew added a comment.Apr 19 2016, 10:13 PM

Current class usage: https://phabricator.wikimedia.org/P2929

• chasemp subscribed.EditedApr 19 2016, 10:16 PM

Some notes that exist on the possible ENC

https://etherpad.wikimedia.org/p/puppet-enc-labs

Roles and their application in Labs:

https://phab.wmfusercontent.org/file/data/2pvdd4gd2zb6cvwgsky6/PHID-FILE-slfudtsimgcfedyhmx7q/abmahmr5sozugehc/Class_usage_in_labs

yuvipanda subscribed.Apr 19 2016, 10:18 PM

Said magic is:

For ENC

Write a simple shell script / python script that hits this API and returns the list of classes *only* for the given node

For Hiera

This will replace the mwyaml hiera provider, calling to this backend instead. The hiera backend will just make a http request for given node name to this interface, and give back to puppet whatever parameters get returned.

yuvipanda added a subtask: T101447: Convert all ldap globals into hiera variables instead.Apr 19 2016, 11:49 PM

Added T101447 as blocking task since this would eliminate the ability to set ldap global variables (or unset / modify them)

yuvipanda added a comment.Apr 20 2016, 12:02 AM

With more discussion between me, @chasemp and @bd808, I think https://etherpad.wikimedia.org/p/puppet-enc-labs is at a fairly steady solid state.

Andrew created subtask T133412: Create puppet backend with REST api for labs instance configuration.Apr 22 2016, 5:58 PM
yuvipanda updated the task description. (Show Details)Apr 23 2016, 12:36 PM
yuvipanda updated the task description. (Show Details)Apr 23 2016, 1:30 PM
gerritbot subscribed.May 31 2016, 7:16 PM

Change 291969 had a related patch set uploaded (by Andrew Bogott):
Allow horizon hosts to contact the labs puppetmaster.

https://gerrit.wikimedia.org/r/291969

gerritbot added a project: Patch-For-Review.May 31 2016, 7:16 PM

Change 291969 merged by Andrew Bogott:
Allow horizon hosts to contact the labs puppetmaster.

https://gerrit.wikimedia.org/r/291969

Andrew mentioned this in rOPUPc56b9eab85c3: Allow horizon hosts to contact the labs puppetmaster..May 31 2016, 7:22 PM
Andrew added a comment.Jun 13 2016, 10:24 PM

First pass of a (still somewhat ugly) 'apply role' dialog.

rolemodal.png (1×2 px, 605 KB)

Questions:

  • How to distinguish between 'unset' and '' for params?
  • How to make the in-line comment formatting look less stupid?
Andrew claimed this task.Jun 14 2016, 3:00 PM
gerritbot added a comment.Jun 14 2016, 4:05 PM

Change 294342 had a related patch set uploaded (by Andrew Bogott):
WIP: Horizon tab for modifying instance puppet config

https://gerrit.wikimedia.org/r/294342

Andrew added a comment.Jun 14 2016, 6:02 PM

Krenair requests a (?) icon for mouseover docs in the overview screen. I'd like that too, but so far it seems to be hard without rewriting the base class.

yuvipanda created subtask T137835: Figure out type coercion rules for puppet parameter config in horizon UI.Jun 14 2016, 6:13 PM
valhallasw subscribed.Jun 14 2016, 6:13 PM

Cool!

With respect to the overview screen: I'm inclined to think of roles like 'tags' rather than a list of true/false values, and the following feels more natural to me:

pasted_file (166×614 px, 13 KB)

I'm not sure how easy that is to implement, though. The Django admin panel can do this relatively easily.

As for the edit panel, I think the current option is already pretty nice. You could consider just exposing a YAML edit field, as that makes it easier to copy-paste configuration between hosts, and also immediately clarifies 'not set' and 'set to be empty'.

gerritbot added a comment.Jun 16 2016, 2:23 PM

Change 294716 had a related patch set uploaded (by Andrew Bogott):
Add the instance tld (e.g. 'wmflabs') to designate and horizon config.

https://gerrit.wikimedia.org/r/294716

gerritbot added a comment.Jun 16 2016, 2:34 PM

Change 294716 merged by Andrew Bogott:
Add the instance tld (e.g. 'wmflabs') to designate and horizon config.

https://gerrit.wikimedia.org/r/294716

Andrew mentioned this in rOPUP99d042fa974b: Add the instance tld (e.g. 'wmflabs') to designate and horizon config..Jun 16 2016, 2:38 PM
Andrew mentioned this in rOPUP65e677d2998a: Include eqiad/codfw in INSTANCE_TLD.Jun 16 2016, 2:45 PM
Andrew mentioned this in rOPUPfb41ed0d1ea9: Add the instance tld (e.g. 'wmflabs') to designate and horizon config..Jun 17 2016, 6:05 PM
Andrew mentioned this in rOPUP4b507fba268e: Add the instance tld (e.g. 'wmflabs') to designate and horizon config..
Andrew mentioned this in rOPUPae5fc0a1ad0d: WIP: Horizon tab for modifying instance puppet config.
Andrew mentioned this in rOPUPc3ffb43276b3: WIP: Horizon tab for modifying instance puppet config.
Andrew mentioned this in rOPUP8386d3708bb1: WIP: Horizon tab for modifying instance puppet config.
Andrew mentioned this in rOPUPd7b7517a7e3f: WIP: Horizon tab for modifying instance puppet config.
yuvipanda mentioned this in T85279: Use a Puppet ENC to define which classes are included in which nodes (in Labs).Jul 5 2016, 1:33 PM
Andrew mentioned this in rOPUPa8e4f7a401cd: WIP: Horizon tab for modifying instance puppet config.Jul 19 2016, 10:03 PM
yuvipanda updated the task description. (Show Details)Jul 29 2016, 7:51 PM
Andrew mentioned this in rOPUPd76ac4603649: WIP: Horizon tab for modifying instance puppet config.Jul 29 2016, 9:34 PM
Andrew mentioned this in rOPUP8d090beada96: WIP: Horizon tab for modifying instance puppet config.Aug 25 2016, 5:58 PM
gerritbot added a comment.Aug 26 2016, 8:12 PM

Change 306981 had a related patch set uploaded (by Andrew Bogott):
Added filtertags to labs role descriptions.

https://gerrit.wikimedia.org/r/306981

gerritbot added a comment.Aug 26 2016, 8:46 PM

Change 306981 merged by Andrew Bogott:
Added filtertags to labs role descriptions.

https://gerrit.wikimedia.org/r/306981

• AlexMonk-WMF added a subtask: T143960: horizon.tables.FixedFilterAction failing in puppet gui.Aug 26 2016, 9:32 PM
Luke081515 updated the task description. (Show Details)Aug 26 2016, 10:11 PM
gerritbot added a comment.Aug 29 2016, 8:37 PM

Change 294342 merged by Andrew Bogott:
Horizon tab for modifying instance puppet config

https://gerrit.wikimedia.org/r/294342

gerritbot added a comment.Sep 2 2016, 4:00 PM

Change 308188 had a related patch set uploaded (by Alex Monk):
openstack horizon puppettab: Put docs on a question mark next to the role name

https://gerrit.wikimedia.org/r/308188

gerritbot added a comment.Sep 5 2016, 3:32 PM

Change 308188 merged by Andrew Bogott:
openstack horizon puppettab: Put docs on a question mark next to the role name

https://gerrit.wikimedia.org/r/308188

Andrew closed subtask T101447: Convert all ldap globals into hiera variables instead as Resolved.Sep 7 2016, 2:33 PM
gerritbot added a comment.Sep 13 2016, 3:21 PM

Change 310325 had a related patch set uploaded (by Andrew Bogott):
puppetpanel: Add a 'Project Puppet' panel

https://gerrit.wikimedia.org/r/310325

gerritbot added a comment.Sep 13 2016, 5:24 PM

Change 310325 merged by Andrew Bogott:
puppetpanel: Add a 'Project Puppet' panel

https://gerrit.wikimedia.org/r/310325

gerritbot added a comment.Sep 14 2016, 10:31 PM

Change 310683 had a related patch set uploaded (by Andrew Bogott):
Puppet panel: Add per-prefix config tabs

https://gerrit.wikimedia.org/r/310683

gerritbot added a comment.Sep 14 2016, 10:46 PM

Change 310683 merged by Andrew Bogott:
Puppet panel: Add per-prefix config tabs

https://gerrit.wikimedia.org/r/310683

gerritbot added a comment.Sep 15 2016, 1:17 AM

Change 310718 had a related patch set uploaded (by Andrew Bogott):
Puppet Panel: Actually populate the prefix panel with prefixes.

https://gerrit.wikimedia.org/r/310718

gerritbot added a comment.Sep 15 2016, 2:24 PM

Change 310718 merged by Andrew Bogott:
Puppet Panel: Actually populate the prefix panel with prefixes.

https://gerrit.wikimedia.org/r/310718

Andrew added a comment.Sep 15 2016, 2:35 PM

This interface has turned into a bit of a monster... it now has has:

  • A panel with one tab for project-wide settings
  • A panel with N tabs for prefix-based settings
  • Tabs in the instance detail group for instance-based settings

I don't love it, but it lets us do what we need. Remaining tasks:

  • Fix the dumb question-mark icon that doesn't render properly
  • Add a way to remove prefixes
  • Figure out what takes so damn long to render a tab
  • (probably) aesthetic improvements
yuvipanda added a comment.Sep 15 2016, 7:40 PM

When you create a new prefix, you don't get taken to the prefix' tab but to the first tab. Same happens after any actions - if you edit hiera, it resets you to the first tab.

gerritbot added a comment.Sep 15 2016, 7:55 PM

Change 310938 had a related patch set uploaded (by Yuvipanda):
labs: Add httpyaml hiera backend to labs

https://gerrit.wikimedia.org/r/310938

gerritbot added a comment.Sep 15 2016, 7:58 PM

Change 310938 merged by Yuvipanda:
labs: Add httpyaml hiera backend to labs

https://gerrit.wikimedia.org/r/310938

gerritbot added a comment.Sep 15 2016, 8:05 PM

Change 310945 had a related patch set uploaded (by Yuvipanda):
labs: Use puppetmaster service URL for hiera backend

https://gerrit.wikimedia.org/r/310945

gerritbot added a comment.Sep 15 2016, 8:06 PM

Change 310945 merged by Yuvipanda:
labs: Use puppetmaster service URL for hiera backend

https://gerrit.wikimedia.org/r/310945

gerritbot added a comment.Sep 15 2016, 8:24 PM

Change 310952 had a related patch set uploaded (by Yuvipanda):
puppet: Add option to use newer ENC

https://gerrit.wikimedia.org/r/310952

gerritbot added a comment.Sep 15 2016, 8:45 PM

Change 310962 had a related patch set uploaded (by Andrew Bogott):
puppet panel: Add a button to remove prefixes.

https://gerrit.wikimedia.org/r/310962

gerritbot added a comment.Sep 15 2016, 8:47 PM

Change 310962 merged by Andrew Bogott:
puppet panel: Add a button to remove prefixes.

https://gerrit.wikimedia.org/r/310962

yuvipanda added a comment.Sep 15 2016, 9:19 PM

Performance of the puppet pannels is still pretty bad - takes about 20-30s to load on my laptop (firefox). We shouldn't enable it for labs like this, need to bring it down at least an order of magnitude.

gerritbot added a comment.Sep 15 2016, 10:11 PM

Change 311041 had a related patch set uploaded (by Andrew Bogott):
Puppet Panel: Speed up loading of role documentation

https://gerrit.wikimedia.org/r/311041

gerritbot added a comment.Sep 15 2016, 10:52 PM

Change 311057 had a related patch set uploaded (by Andrew Bogott):
Puppet Panel: Speed up role formatting

https://gerrit.wikimedia.org/r/311057

gerritbot added a comment.Sep 15 2016, 10:54 PM

Change 311041 abandoned by Andrew Bogott:
Puppet Panel: Speed up loading of role documentation

https://gerrit.wikimedia.org/r/311041

gerritbot added a comment.Sep 15 2016, 11:22 PM

Change 311057 merged by Andrew Bogott:
Puppet Panel: Speed up role formatting

https://gerrit.wikimedia.org/r/311057

gerritbot added a comment.Sep 16 2016, 5:19 PM

Change 311160 had a related patch set uploaded (by Andrew Bogott):
Puppet Panel: Add captions to instance tabs

https://gerrit.wikimedia.org/r/311160

gerritbot added a comment.Sep 16 2016, 5:21 PM

Change 311160 merged by Andrew Bogott:
Puppet Panel: Add captions to instance tabs

https://gerrit.wikimedia.org/r/311160

gerritbot added a comment.Sep 16 2016, 5:42 PM

Change 311164 had a related patch set uploaded (by Andrew Bogott):
Puppet Panel: Remove use of breadcrumb_nav

https://gerrit.wikimedia.org/r/311164

gerritbot added a comment.Sep 16 2016, 5:44 PM

Change 311164 merged by Andrew Bogott:
Puppet Panel: Remove use of breadcrumb_nav

https://gerrit.wikimedia.org/r/311164

gerritbot added a comment.Sep 19 2016, 4:08 PM

Change 311449 had a related patch set uploaded (by Andrew Bogott):
Puppet Horizon Panel: Deploy on Labs

https://gerrit.wikimedia.org/r/311449

gerritbot added a comment.Sep 19 2016, 4:47 PM

Change 311449 merged by Andrew Bogott:
Puppet Horizon Panel: Deploy on Labs

https://gerrit.wikimedia.org/r/311449

gerritbot added a comment.Sep 19 2016, 7:07 PM

Change 311486 had a related patch set uploaded (by Andrew Bogott):
Pupept panel: Use "{}" to represent an empty hiera config

https://gerrit.wikimedia.org/r/311486

gerritbot added a comment.Sep 19 2016, 7:12 PM

Change 311488 had a related patch set uploaded (by Andrew Bogott):
Puppet Panel: Don't choke on a (user-provided) "" hiera string.

https://gerrit.wikimedia.org/r/311488

gerritbot added a comment.Sep 19 2016, 7:16 PM

Change 311486 merged by Andrew Bogott:
Puppet panel: Use "{}" to represent an empty hiera config

https://gerrit.wikimedia.org/r/311486

gerritbot added a comment.Sep 19 2016, 7:18 PM

Change 311488 merged by Andrew Bogott:
Puppet Panel: Don't choke on a (user-provided) "" hiera string.

https://gerrit.wikimedia.org/r/311488

gerritbot added a comment.Sep 20 2016, 6:51 PM

Change 310952 merged by Yuvipanda:
puppet: Add option to use newer ENC

https://gerrit.wikimedia.org/r/310952

gerritbot added a comment.Sep 20 2016, 7:05 PM

Change 311762 had a related patch set uploaded (by Andrew Bogott):
Puppet Panel: No checkboxes!

https://gerrit.wikimedia.org/r/311762

gerritbot added a comment.Sep 20 2016, 7:13 PM

Change 311762 merged by Andrew Bogott:
Puppet Panel: No checkboxes!

https://gerrit.wikimedia.org/r/311762

gerritbot added a comment.Sep 20 2016, 7:38 PM

Change 311767 had a related patch set uploaded (by Yuvipanda):
puppet: Use newer ENC

https://gerrit.wikimedia.org/r/311767

gerritbot added a comment.Sep 20 2016, 7:40 PM

Change 311767 merged by Yuvipanda:
puppet: Use newer ENC

https://gerrit.wikimedia.org/r/311767

gerritbot added a comment.Sep 21 2016, 5:00 PM

Change 312044 had a related patch set uploaded (by Yuvipanda):
puppet: Enable ENC on trusty modules too

https://gerrit.wikimedia.org/r/312044

gerritbot added a comment.Sep 21 2016, 6:00 PM

Change 312044 merged by Yuvipanda:
puppet: Enable ENC on trusty nodes too

https://gerrit.wikimedia.org/r/312044

Andrew closed this task as Resolved.Oct 3 2016, 6:23 PM
Andrew closed subtask T133412: Create puppet backend with REST api for labs instance configuration as Resolved.
Andrew closed subtask T137835: Figure out type coercion rules for puppet parameter config in horizon UI as Resolved.Oct 28 2016, 7:26 PM
• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:53 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL