Page MenuHomePhabricator
Create Task
Maniphest T129092

Hiera is not properly configured on Nodepool instances
Closed, ResolvedPublic
Actions

Description

As part of magically provisioning dependencies on Nodepool instances (eg https://gerrit.wikimedia.org/r/#/c/274675/ ) I have found out Hiera was falling due to empty files (fixed: T128846: DIB images have /puppet/hieradata files that are empty).

While debugging, I also found that Hiera lookup is broken on Nodepool instances. Namely it is missing the fact for labsproject:

$ /puppet/utils/hiera_lookup --fqdn=`hostname --fqdn` labs project

$ facter labsproject

$

The hiera configuration file references ::labsproject and since it is empty, a wild range of Hiera hierarchy is not traversed. Relevant conf from hiera.yaml

:hierarchy:
  - "labs/hosts/%{::hostname}"
  - "labs/%{::labsproject}/host/%{::hostname}"
  - "labs/%{::labsproject}/common"
  - "labs"
  - "secret/%{::labsproject}"
  - "private/%{::labsproject}"
  - common
  - "secret/common"
  - "private/common"

Using hiera_lookup in verbose mode (-v):

jenkins@ci-jessie-wikimedia-45448:~$ /puppet/utils/hiera_lookup --fqdn=`hostname --fqdn` labsproject -v
DEBUG: 2016-03-07 16:38:25 +0000: Looking up labsproject

DEBUG: 2016-03-07 16:38:25 +0000: Loading info from hosts/ci-jessie-wikimedia-45448 for labsproject
DEBUG: 2016-03-07 16:38:25 +0000: The source is: hosts/ci-jessie-wikimedia-45448
DEBUG: 2016-03-07 16:38:25 +0000: Cannot find datafile /puppet/hieradata/hosts/ci-jessie-wikimedia-45448.yaml, skipping
DEBUG: 2016-03-07 16:38:25 +0000: Searching for labsproject in 
DEBUG: 2016-03-07 16:38:25 +0000: Loading info from regex/ci-jessie-wikimedia-45448.eqiad.wmflabs for labsproject
DEBUG: 2016-03-07 16:38:25 +0000: Regex match going on - using regex.yaml
DEBUG: 2016-03-07 16:38:25 +0000: Searching for labsproject in /puppet/hieradata/regex.yaml
DEBUG: 2016-03-07 16:38:25 +0000: Loading file /puppet/hieradata/regex.yaml
<snip Scanning label foo>
DEBUG: 2016-03-07 16:38:25 +0000: Loading info from eqiad/labs for labsproject
DEBUG: 2016-03-07 16:38:25 +0000: The source is: eqiad/labs
DEBUG: 2016-03-07 16:38:25 +0000: Cannot find datafile /puppet/hieradata/eqiad/labs.yaml, skipping
DEBUG: 2016-03-07 16:38:25 +0000: Searching for labsproject in 
DEBUG: 2016-03-07 16:38:25 +0000: Loading info from eqiad for labsproject
DEBUG: 2016-03-07 16:38:25 +0000: The source is: eqiad
DEBUG: 2016-03-07 16:38:25 +0000: Searching for labsproject in /puppet/hieradata/eqiad.yaml
DEBUG: 2016-03-07 16:38:25 +0000: Loading file /puppet/hieradata/eqiad.yaml
DEBUG: 2016-03-07 16:38:25 +0000: Loading info from private/eqiad/labs for labsproject
DEBUG: 2016-03-07 16:38:25 +0000: The source is: eqiad/labs
DEBUG: 2016-03-07 16:38:25 +0000: Cannot find datafile /puppet/hieradata/eqiad/labs.yaml, skipping
DEBUG: 2016-03-07 16:38:25 +0000: Searching for labsproject in 
DEBUG: 2016-03-07 16:38:25 +0000: Loading info from private/eqiad for labsproject
DEBUG: 2016-03-07 16:38:25 +0000: The source is: eqiad
DEBUG: 2016-03-07 16:38:25 +0000: Searching for labsproject in /puppet/hieradata/eqiad.yaml
DEBUG: 2016-03-07 16:38:25 +0000: Loading file /puppet/hieradata/eqiad.yaml
DEBUG: 2016-03-07 16:38:25 +0000: Loading info from common for labsproject
DEBUG: 2016-03-07 16:38:25 +0000: The source is: common
DEBUG: 2016-03-07 16:38:25 +0000: Searching for labsproject in /puppet/hieradata/common.yaml
DEBUG: 2016-03-07 16:38:25 +0000: Loading file /puppet/hieradata/common.yaml
DEBUG: 2016-03-07 16:38:25 +0000: Loading info from private/common for labsproject
DEBUG: 2016-03-07 16:38:25 +0000: The source is: common
DEBUG: 2016-03-07 16:38:25 +0000: Searching for labsproject in /puppet/hieradata/common.yaml
DEBUG: 2016-03-07 16:38:25 +0000: Loading file /puppet/hieradata/common.yaml

jenkins@ci-jessie-wikimedia-45448:~$

Facter is unahppy:

$ facter hostname realm labsproject labsprojectfrommetadata
hostname => ci-jessie-wikimedia-45448
labsproject => nil
labsprojectfrommetadata => nil
realm => nil

It seems labsproject and realm are populated from puppet.git manifests/realm.pp which we might not include.

The labsprojectfrommetadata would be shipped via modules/base/lib/facter/labsprojectfrommetadata.rb:

Facter.add(:labsprojectfrommetadata) do

And we can't really include ::base because it ships too much unrelated stuff that ends up breaking puppet provisioning on the very lightweight image.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
hiera_lookup: recognize labs project and siteoperations/puppetproduction+19 -3
hiera_lookup: support 'labs' realmoperations/puppetproduction+9 -3
dib: set $labsprojectintegration/configmaster+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Mar 7 2016, 4:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 7 2016, 4:46 PM
• mobrovac added a comment.Mar 7 2016, 5:49 PM

You can add facts by passing in environment variables:

$ FACTER_MYVARNAME=blah puppet apply

This will cause Facter to load MYVARNAME as a facter variable.

hashar added a comment.Mar 7 2016, 5:57 PM
$ facter labsproject

$ FACTER_LABSPROJECT=foobar facter labsproject
foobar
$

Super nice hack @mobrovac ! I guess that is a fair workaround to populate facts :-} Thank you

gerritbot subscribed.Mar 8 2016, 11:20 AM

Change 275786 had a related patch set uploaded (by Hashar):
dib: set $labsproject

https://gerrit.wikimedia.org/r/275786

gerritbot added a project: Patch-For-Review.Mar 8 2016, 11:20 AM

Change 275786 merged by jenkins-bot:
dib: set $labsproject

https://gerrit.wikimedia.org/r/275786

hashar triaged this task as High priority.Mar 8 2016, 11:56 AM
hashar moved this task from Backlog to In-progress on the Continuous-Integration-Scaling board.
hashar removed a project: Continuous-Integration-Infrastructure.
hashar claimed this task.Mar 8 2016, 1:53 PM
hashar added a comment.Mar 9 2016, 10:06 PM

operations/puppet.git has a hiera utility /utils/hiera_lookup but it does not support labs anymore so I had to fix it:

https://gerrit.wikimedia.org/r/276345
https://gerrit.wikimedia.org/r/276346

gerritbot added a comment.Mar 9 2016, 10:07 PM

Change 276346 had a related patch set uploaded (by Hashar):
hiera_lookup: recognize labs project and site

https://gerrit.wikimedia.org/r/276346

gerritbot added a comment.Mar 9 2016, 10:07 PM

Change 276345 had a related patch set uploaded (by Hashar):
hiera_lookup: support 'labs' realm

https://gerrit.wikimedia.org/r/276345

Paladox subscribed.Mar 9 2016, 10:08 PM
hashar added a parent task: T128280: Enable librsvg-dev on jenkins.Mar 9 2016, 10:08 PM
hashar added a parent task: T119693: Extract Graphoid Debian packages dependencies to a standalone puppet class so they can be installed on CI slaves.
hashar closed this task as Resolved.Mar 10 2016, 12:11 PM

I have used the updated /utils/hiera_lookup on a Nodepool instance and it seems to work.

$ cat labs/contintcloud/common.yaml 
---
# install dev packages for services
service::configuration::use_dev_pkgs: true
ci-jessie-wikimedia-48049:~$ PUPPETDIR=/puppet ./hiera_lookup --fqdn=ci-jessie-wikimedia-48049.contintcloud.eqiad.wmflabs service::configuration::use_dev_pkgs
true

Been fixed by setting $::labsproject in puppet via https://gerrit.wikimedia.org/r/#/c/275786/

gerritbot added a comment.Mar 26 2016, 11:08 AM

Change 276345 merged by Alexandros Kosiaris:
hiera_lookup: support 'labs' realm

https://gerrit.wikimedia.org/r/276345

Krenair mentioned this in T135427: Beta puppetmaster cherry-pick process.May 16 2016, 8:02 PM
hashar mentioned this in rOPUP72b4cea7d1d4: hiera_lookup: recognize labs project and site.Jun 30 2016, 12:47 PM
hashar mentioned this in rOPUP63aae1fead4c: hiera_lookup: recognize labs project and site.Jul 20 2016, 1:02 PM
gerritbot added a comment.Aug 30 2016, 4:39 PM

Change 276346 merged by Filippo Giunchedi:
hiera_lookup: recognize labs project and site

https://gerrit.wikimedia.org/r/276346

hashar added a comment.Aug 30 2016, 4:42 PM

Last patch landed in puppet.git so that is definitely fixed.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits